UPDATE #2: September 6, 2010
At noontime Eastern, I heard from Naoko again and she confirmed robots.txt has been upgraded site-wide on WordPress.com!
I had to change the privacy settings on all my blogs — and then back again to “public” — to force the new robots.txt file to update. The plan worked.
All 13 public blogs are now set and updated and ready for Google and the rest of the indexed search world to remove our proprietary Movable Type search results.
Here’s a screenshot of the new robots.txt file disallowing the “/cgi-bin/” directory. I highlighted the new addition:
THANK YOU from prying us from the rock!
The moment I published this article today at 12:24pm Easter time, I followed up with WordPress.com support and gave them the link to this article in an attempt to better explain — with screenshots — the problem I was trying to solve.
At 2:12pm — less than two hours after I wrote to WordPress.com — Naoko replied:
I was waiting for this to actually go live, but a change has been made in our code.
Will be added to robots.txt (not visible yet, I need to check back with the developer).
Fantastic news! That solves my proprietary Movable type search results problem across all 13 of my public WordPress.com blogs! Here is my reply:
Oh, that’s great news! Is this change on a per-blog basis, or is it site wide?
If it’s side wide, are there plans to include robots.txt proprietary search disallows for the other blogging services?
I will update my article to reflect the information you provide.
I will keep you updated!
I don’t see the “/cgi-bin/” disallow yet on any of my blogs in robots.txt, but the moment it goes live, I will go back to Webmaster Tools and specifically ask that the “/cgi-bin/” directory be removed now and forever from all my blogs.
As well, because of this robots.txt disallow addition, I will now be able to effectively venture into Yahoo! and Bing to see if I can get the same directory deleted in those services for all my blogs.
Thank you WordPress.com Gods!
I recently discovered a terrible Movable Type artifact that still remains festering and alive within me — via Google Search Returns — six months after I became a Six Apart refugee and gave up my expensive, self-hosted, standalone, blog hosting and returned to my first blogging home: WordPress.com. You can see an example of the problem below in the third search return in the screenshot. That “Memeingful: Search Results” link takes you to a proprietary Movable Type search return that has been dead for six months. Click on that link, and you’ll be taken to a “Not Found” error page on WordPress.com.