Whitehouse site blocks indexing of files about Iraq

By at 12:34 pm Monday, Oct 27

Whitehouse.gov has a robots.txt file that keeps directories containing the word "iraq" from being indexed by search-engines.
Sometime between April 2003 and late October, 2003, hundreds of instances of the term "iraq" were added to the whitehouse.gov robots.txt file.

On a quick look, It appears that the google cached version of the robots.txt file very nearly could be created by searching for the string "text" in the April 16 file and replacing it with the string "iraq" then adding the newly changed files ("text" to "iraq") to the original file, keeping the lines with "text" in them as well. That's from a quick look so I'm not sure it would hold up, but it appears it could explain the bulk of the "iraq" appearances.

Link (Thanks, Kurt!)

Cory Doctorow

Upcoming appearances

* Feb 9, 2012, DeKalb, IL: Day of Doctorow, NIU
* Feb 10-12, 2012, Chicago, IL: Capricon 32
* Feb 13, 2012, Arlington, TX: UT Arlington College of Engineering Distinguished Speaker Series
* Feb 16, 2012, Victoria, BC: 13th Annual Privacy and Security Conference

Recent books:
* Context (essays)
* With a Little Help (short stories)
* For the Win (YA novel)
* Makers (adult novel)

Comments are closed.

Where not otherwise specified, this work is licensed under a Creative Commons License permitting non-commercial sharing with attribution. Boing Boing is a trademark of Happy Mutants LLC in the United States and other countries.