Hi Guys, i’m having a problem with googlebots or any bots for that matter, accessing my website. During troubleshooting, i have disabled the x-theme theme and google bots can access the site. But when the x-theme is active, no bots are able to crawl. I’ve tried everything and all out of ideas. Are you guys able to help please? My website is callandcreative.com
I’ve done some more trouble shooting. I have delete all themes and installed a fresh copy of x theme pro. When pro is active, Google page insights says:
Lighthouse returned error: ERRORED_DOCUMENT_REQUEST. Lighthouse was unable to reliably load the page you requested. Make sure you are testing the correct URL and that the server is properly responding to all requests. (Status code: 403)Dismiss
When i set the current theme to Twenty Twenty, google page speed insights works fine.
Hi @callandcreative,
Thank you for writing in, I’ve tested it here and I don’t see Googlebot is being blocked on your site. Please login to your Cloudflare dashboard and Purge all cache.
Purging cached resources from Cloudflare
Hope it helps,
Cheers!
Hi @friech ,
Thanks for your reply and i hope you are well.
I’m still getting the error when using Page Speed Insights. I have tried other tools which crawl and they are blocked also giving a 403 error. I’ve had cloudflare on development mode so cache is bypassed.
I’m completely stuck on this one, if anyone has any ideas?
Thanks
Hello @callandcreative,
To better assist you with your issue, kindly provide us access to your site so that we can check your settings. Please create a secure note with the following info:
– Link to your site
– WordPress Admin username / password
To know how to create a secure note, please check this out: https://theme.co/docs/getting-support
Regards.
Please find the website access below:
Hi @callandcreative,
Based on the browser console, yes, returns 403 but it’s not due to the theme since it has no capability to change header statuses. It’s not blocked, the PageInsights won’t simply analyze it due to 403 status even there is active content.
Even by disabling CloudFlare plugin, it still serves the resources from CloudFlare CDN. All requests are first handled with them and that includes redirects and HTTP statuses. Would you mind providing your CloudFlare’s acount login credentials as well? So I could disable it and test it while I’m checking this issue.
And please provide your FTP login credentials as well in case we need to install a fresh copy of Pro theme.
Thanks!
Hi @rad thanks for your reply.
Can you provide your email address so i can add you as a user to cloudflare please?
Feel free to try what you like, i’ve come to a dead end with my testing!
Hello @callandcreative,
Thanks for providing the access. I have logged in and I found out that you have a robots.txt file in your directory and it contains:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Allow: /callandcreative.com
This could be the reason while you are having Error 403 in Page Insights.
Please remove your robots.txt file and test your site again in Page Insights.
Kindly let us know how it goes.
This robots.txt file is not causing the issue. Something within the x-theme directory is causing the problem. When i enable the theme ‘twenty twenty’ bots are able to crawl the site. As soon as i enable x theme pro or the pro child theme, bots are blocked.
Hello @callandcreative,
I have logged in to your site and I temporarily switch to Twenty Twenty theme. When I tested your site in PageSpeed Insights, I am still seeing the same error message:
This error has nothing to do with the theme. It could be because of your server settings. Please ask further assistance from your hosting provider.
Please let us know how it goes.
This was not the case on my end. Did you disable cloudflare caching when you where testing? I can guarantee, 100% that twenty twenty allows bots to crawl the site and i can confirm that X-theme is 100% the problem.
Just an update on the situation, i configured a new server droplet on digital ocean and installed a fresh copy of Wordpress. After migrating the database and theme files to the new server, i was still left with the same issue of bots unable to crawl the website. It is definitely something within the theme files causing this problem and it is not the hosting provider.
There is no evidence in the website front-end code that i can see is blocking all bots from crawling. The rules in the allow bots.
The solution to this problem is unfortunately resulting in a site rebuild, probably without using X-theme as this situation sucks.
Hi @callandcreative,
Again, nothing is blocked, the PageInsight only refuses to load or analyze it due to 403 header response from the site. Please check this https://support.google.com/webmasters/thread/12564090?hl=en. Perhaps the server don’t like something from the theme and returns 403, it’s because status are server headers, not theme. Although it can be triggered by PHP codes, I can assure you that the theme has no PHP code that triggers server 403 status.
Unfortunately, I can’t provide my email, would you mind providing a temporary login for Cloudflare? The problem with the plugin is its limitation of what we can test.
And please provide the latest credentials and installation if you moved it to another server. It’s currently inaccessible from my end as of this moment.
Thanks!
Hi,
The credentials should work again now. I’m afraid i cannot give you access to my cloudflare account as there is access to client accounts on there. If you want me to remove cloudflare from the site, let me know.
I have installed a fresh copy of X Theme Pro with no child theme.
I can see when i inspect the page in Chrome that there is a 403 error:
When i inspect the page in Safari, there is a 404 error pointing to a file.
Any ideas?
Here is a screen shot of both the pro theme and twenty twenty. you will see that only the pro theme is causing the error in page speed insights.
I’d like to chime in a sec and say that Lighthouse is generating proper reports for all my X and Pro sites. It is definitely not Pro’s fault.
Hi @callandcreative,
Sorry that you’re still having that issue, have you tried contacting the Cloudflare support? If not I advise doing so, Blocking Legitimate Google/Bing bots is a common issue with Cloudflare.
- https://community.cloudflare.com/t/cloudflare-firewall-blocking-legitimate-google-bing-bots/144763
- https://community.cloudflare.com/t/cloudflare-managed-special-rules-are-blocking-googlebot/82911
As Rad said above, the Cloudflare plugin is limited, please work from the Cloudflare’s dashboard directly (e.g. purging cache, changing settings, etc.).
Thanks,
Hi Guys,
I removed the site from Cloudflare today so the domain is pointing directly at the server. I tested Page Speed Insights again and it still gives the same result. There is still a 403 error in the browser console.
I have also contacted the guys who manage the server but they have not replied yet. I’ll post their response here when they get back to me.
Any ideas what this might be? Could there be something in the theme files causing that error in the console.
Thanks for your help.
Hi @callandcreative,
I’m sorry I can’t point where is that 403 error is coming from, but have you check your SSL certificate recently, GTmetrix says its expired, please check and address that first, that might be related to the blocking issue.
https://gtmetrix.com/?job_error=FqPwaSqh
Thanks,
Hi @callandcreative,
It looks like you deleted you previous post.
Please let us know in case you have any other questions and post a new thread.
Thank you.