• Yacybot has become one of our most active bots recently. We now have 90 variations of this bot, and we have seen a fair amount of traffic from them. These bots are from Yacy.net, a "distributed web search" that can be downloaded and run locally as "a scalable personal web crawler and web search engine". related...
• Sensis Web Crawler dropped in - according to their site, "Sensis Corporation's Purpose is to provide distinctively elegant, innovative technical solutions in the service of humanity." related...
• Over the past two months, we've had many flavors of Nutch pass by - below is a taste of what we've seen related...

  Manav/Nutch-0.9 */Nutch-0.9-dev weivel/Nutch-0.9 TestSpider/Nutch-1.0-dev Nutch/Nutch-1.0-dev SimilarPages/Nutch-1.0-dev sphsearch.org/Nutch-1.0-dev

  Webcrawler/Nutch-1.0-dev NetSeer/Nutch-0.9 SapphireWebCrawler/1.0 Test crawler Nutch/Nutch-1.0-dev Ayna/Nutch-0.9 nutch_princeton/Nutch-1.0-dev beast/Nutch-0.9

• Where do all the bad bots go? We found out that there is in fact a "Bot Hell" (Bothell, WA) located about 12 miles southwest of Seattle, Washington. We're pretty sure it's *not* pronounced "bot-hell", but it's still funny enough. Maybe bots like devil in disguise (v. 2.0.1a) and HellBoundHackerOS reside there...
• Here are a few random bots from the last month that interested us:
  
  • mixi-crawler/2.00 is from Japan. I spent some time on their site, but my Japanese is non-existent, and even with the help of Google Translate, I could not figure out what the site does related...
  • ZONGOLBOT the Web Spider is a bot from the UK search engine of the same name related...
  • flatlandbot/allspark is a spider from Flatland Industries. To their credit, they provide not only robots.txt support and an email to contact them at, but they also have a phone number to dial if you are having trouble with one of their bots. related...
  • Two new versions of Zontirbot showed up recently related...

Blog Archives: