Forum | Taekwondo: ATA Western Australia

robots.txt unreachable – sitemap error

Reply to Post

Page: 1

User

Post

3:37 am
June 16, 2008

Roy Khoh

Canning Vale, Western Australia

Admin

posts 163

Quote and Reply

Print this Post

Google Webmaster Tools reported sitemap errors. The details showed that robots.txt was unreachable and/or network unreachable. This was roughly from Jun 6 through to Jun 14.

Researching online revealed that it could be a DNS lookup failure – so I have since changed nameserver DNS records. I am now using “everydns”.

I'm not sure if that just took time to propogate, I did wait the 48 hours – still to no avail. Looking at packet headers, when I access the robots.txt myself – it returned 200 code (successful and working properly).

I checked with the host (3ix) whether the ip address/range for googlebot had been banned. Their response didn't answer my question, they just said "everything is fine" … "it is not our end" type of mentality.

Finally I deactivated “WP Super Cache”. A couple hours after that, the robots.txt file was reachable again by Google Webmaster Tools (googlebot crawler). Thinking back it was roughly the time I updated to the latest 0.6.4 version did this problem roughly begin. Albeit I also updated some other plugins around the same time.

7:26 am
July 1, 2008

Roy Khoh

Canning Vale, Western Australia

Admin

posts 163

Quote and Reply

Print this Post

It's only been half a month since, and I have had not problems with the sitemap or robots.txt unreachable error so far.

"WP Super Cache" is still deactivated, I had not switched it on – even after upgrading to WordPress 2.5.0 or 2.5.1

The nameservers used for DNS are still with "everydns" and the original nameservers from 3ix.

Ultimately, because of a mis-timing for about 24 hours between the 48 hour DNS propogation and the "fix", I am pin-pointing the solution to either "WP Super Cache" or the host finally realising and removing googlebot from their blacklist/firewall.

I am leaning towards the host firewall as the problem, because the technical support I chatted to didn't really seem too knowledgeable. Especially since they did not answer my direct question of "is googlebot blacklisted from the host?" They just kind of side-stepped around it and pointed at other things.

Page: 1

Reply to Post