Bots vs Browsers : Public Bot / User Agent Database & Commentary

Home | Archives | About User Agents | User Agent Test Track | Recent Additions | Contributors | Donate | Top Searches | User Agent IP Directory

Bots vs Browsers - database of 311,536 user agents and growing

The Latest:
2009.5.31 - Summer Bots make a splash!

With summer beginning, we saw a big splash from some new bots, and some new versions of old familar bots. This puts us dangerously close to the 300,000 mark with 297,455 user agents and 3,233 bots.



2009.4.18 - Where did Palm go?

Recently I was helping a client setup their phone for Google Apps' mail (Just GMail with a different face). I had gone through some very helpful support articles which included instructions for phones running Android, Windows Mobile 5 and 6, iPhone, Blackberry, and the list goes on. The client had told me they were running a Treo 700, which I assumed was running Windows Mobile 5 or 6. After some research, I found that their version was actually a Treo 700p running Palm's OS, not the Treo 700w running Windows Mobile. In looking for instructions for setting this up, help was scarce, even from Google's extensive mobile help knowledgebase.

Considering their former prominence in the mobile market, one would think that legacy support for Palm's operating systems would still be strong. Granted, I have not used a Palm in over 10 years, and back then it was a basic Palm Pilot PDA in black and white, but for many years they were the prevailing mobile device manufacturer.

In light of their market domination in handheld devices between 2000 and 2005, I became curious about how they disappeared so quickly. In doing a little research, it seems that Palm just missed the boat on the smartphone industry. They had an early competitive advantage in the market, but were late to market and slow to innovate in the smartphone revolution. The idea of a PDA without a phone built-in is redundant these days, with the only exception being the iPod Touch.

Palm is all but dead in the phone market and smaller PDA market these days. The strangest part about Palm's lingering PDA presence is their online Palm shop. They still list three models of phone-less PDA's, but when viewing their availability, all three report that "This product is not currently available on the Palm Store".

In memory of Palm's former market presence, I reviewed our user agent logs to get an idea of what we've seen over the years. Over the last 4 years, we have captured about 70 variations of Palm user agents, few of which ran Palm OS.

2009.2.18 - Happy Belated 3rd Birthday to Us!

So it's been a while since we've posted our progress, so here goes:
We've been busy, and so have our web traffic logs - we've discovered 320 bots since our last post, and found over 31,000 other new user agents since then. There's alot of noise in those logs, so here are the high points that we hope will interest you: This brings us to 252,674 user agents and 2,828 bots.

2008.11.23 - Bots should be thankful

This thanksgiving, all the bots out there should give thanks for the masses of bandwidth they chew each day at the expense of webmasters around the world. If our logs are any indication, few bots actually bring enough human traffic to make up for their crawls, so there's alot of free-loading bots out there. Anyways, we had over 200 new faces, and here are the ones we found interesting: This month brings us to 221,187 user agents and 2,508 bots. Thanks for visiting!
2008.10.22 - Would you say I have a plethora of new bots?

We have a new category and over 100 new bots this month. Let's cut to the chase: This installment brings us to 212,592 User Agents and 2,376 bots.
2008.9.6 - 200,000 User Agents!

The last month was very exciting, since we broke 200,000 user agents! It was almost exactly a year ago that we broke 100,000 user agents, and almost 2 years since we broke 50,000 user agents. How the time flies by. Here are some bots that seemed interesting to us this month as we head further into September: We are now at 200,000 user agents and 2,194 bots.
2008.8.3 - Here comes August...

The last couple of weeks have been pretty routine around here. In our logs, we've sifted through a growing number of script injection hacks via user agent, some if which are getting quite creative with their HTML markup and JavaScript technique. We've also seen some new bots, and some old bots that are very active as of late.

Here's what we've seen:

  • In close relation to the robot WALL-E, his cousin swish-e turned up in our logs this week. Instead of cleaning up 700 years of trash on what is left of Earth, swish-e is an open source system for indexing web pages. The acronym "swish-e" stands for "Simple Web Indexing System for Humans - Enhanced". related...
  • More and more user agents for the Nintendo Wii have been showing up lately, so we decided to create a new category for this. The Nintendo Wii category is our latest category tracking user agents, so check it out to see any WII-related user agents that we've encountered in our logs. related...
  • We've noticed a great deal of activity recently from Russian search engine bot Yandex - in particular, from IP 77.88.25.28. They hit our sites over 15,000 times today, but the concentration per domain name and over time intervals was not quite enough to put them in range of being banned. In the two years that we've been tracking this bot, we've never seen the traffic rates this high. We'll keep a close eye on them over the next few weeks, as this trend may affect our readers as well. related...
  • Several of our consulting clients as of late have needed help ridding their sites of SQL injections. One thing that we have noticed in common when running our cleanup tools on their databases is that most of them end up with a table on their database called "t_jiaozhu". After Googling the term, we have found countless others with the same story - SQL injection hack, table created with weird name "t_jiaozhu". The point is, make sure your site is SQL injection attack-proof. If you think you may have been hacked or just aren't sure, check your database for the table "t_jiaozhu".

    Depending on your web architecture, be sure that all SQL calls are scrubbed either through common framework level cleansing or by home-grown means. Once you have secured your database from SQL injection, make sure you don't forget to protect yourself from script injection attempts as well. We've documented many of these script injection attempts that appear in user agents here.

    Anyways, off the soap box and back to the bots! related...
  • BobCrawl/Nutch-0.9 is a new form of Nutch that appeared in our logs, claiming to be a "Test/Development crawler". On a side note, in an effort to inform us that its URL and email are not available, they mis-spelled this in the user agent and put in "notavalable".

    Leave it to us to get caught up in the details. related...
  • Flatland Industries sent their web spider flatlandbot. Their website claims that the bot follows robots.txt exclusion standards, so if you don't want them around, be sure to let them know. related...
  • Here's a strange one - Blubberlutsch/1.0. There is absolutely no information currently on Google for this user agent. Results vary from site to site from "Donald Duck" to "Star Wars Attack of the Clones". My best guess is that it's German slang for something, so in the meantime, we'll call it a bot, and check back on it later. related...
  • We had our first close encounter / UFO sighting this week - UFO/77.7 (CoSMoS; Z; Pearl 256; peep) F!R3F0>< P\/\/NS y0!. related...
  • Another new search engine bot appeared from Isidorus/2.0. related...
  • Quite possibly the strangest user agent of the year, Nintendo64/1.0 (SuperMarioOS with Cray-II Y-MP Emulation) paid us a visit recently. related...
Opening up August, we have 192,894 user agents and 2,070 bots. Thanks for dropping by, and remember - only you can prevent injection hacks!
2008.7.13 - 2,000 bots!

We set a new milestone this week for our user agent database - we broke 2,000 bots!

This week's logs were comprised mostly of new versions of bots we know about. There are a few new faces mixed in there too - here's the latest:
  • On the browser side, one new face in our logs is from iGetter. This is a new download utility that looks pretty interesting. related...
  • Thumbshots is another new bot on our site - we witnessed 3 unique bot user agents from them this week: thumbshots-de-bot, KFSW/1.0.0.0.4 thumbshot-de-bot, and KFSW/1.0.0.0.1 ThumbShots-de-bot. related...
  • Exooba crawler/0.5.1 debuted this version this week in our server logs. According to their site, they are a testing crawler that uses gathered data strictly for testing purposes only. Their site also informed us that they fully comply with robots.txt exclusion standards. related...
  • Here's an interesting one - Daruma/1.08 (Windows 98; U) [en]. We were unable to find anything about this user agent in our research. When googling this bot, we are the only result on google as of yet. However, according to Wikipedia, the word "Daruma" is commonly used to describe "Daruma Dolls", also known as dharma dolls. These are hollow, round Japanese wish dolls, modelled after Zen patriarch Bodhidharma. This just goes to show that you can learn alot more than techie facts on this blog. related...
  • CatchBot/1.0 is another new face on the site. related...
  • Metaspinner/0.01 is a bot from MetaSpinner - yet another search engine crawling the web for data. related...
  • OmniExplorer_Bot/1.0x Job Crawler is a new version of the OmniExplorer bot family. OmniExplorer has a long standing pedigree of bots on our site, and here you can see more of them: related...
  • Yacybot is another long-standing bot family on our site, and they turned up a new version this week as well related...
  • We posted about our first OOZBOT sighting last week, and now we have a new user agent from them as well: OOZBOT related...
Mid July is approaching fast, and we've got 187,746 user agents and 2,018 bots - thanks for visiting!
2008.7.6 - Dog Days of Summer Bots

We had quite a few new faces in the logs last week, from established sites like Yahoo! all the way to some new startup search engines that are sending their bots out into the wild to harvest searchable content. Here's the high points of our week:
  • Yahoo! has a new YahooSeeker-Testing/v3.9 bot that dropped in for a visit. related...
  • TheRarestParser/0.4b is a new version sent from TheRarestBlog. This bot crawls web pages to find the rarest words used on the web. related...
  • Watch out Google - here's two startup search companies that are crawling the web -
    OOZBOT/0.17 from Setooz - their site is currently "Under Construction", but a search engine will likely appear there someday. related...

    Zook.in looks like a startup that sent their Knight/0.2 bot by for a visit. From visiting their site, it looks like they aren't quite ready for prime time yet since there isn't really a site there yet. related...
  • Slurpem Search sent a bot informing us that they are "Sucken Up The Web". I hope Slurpem is thirsty, because there's alot of web out there to suck up. related...
  • Firebat 2.7.13 is a new bot on our site as well. Firebat does not have a good reputation on the web from what our research has turned up. They are mentioned in many places with large denial of service attacks and several other shady activities. related...
  • EmailWolf 1.00 prowled our sites looking for emails to chew on, but we don't post emails, so the email wolf went hungry. related...
  • Someone or something at IP 84.0.233.218 was kind enough to attempt a script injection to call a javascript alert telling the calling browser of XSS (Cross Site Scripting) vulnerabilities on the site. Of course, we are not vulnerable to XSS, but nice try. This puts us at 75 unique script injection attempts. Make sure when you are viewing your web stats that you do not allow scripts to be executed from the user agent, or they could easily snipe your cookie and hack your web stats portal. related...
We're heading into the second week in July now at 185,694 user agents and 1,993 bots - thanks for visiting!
2008.6.23 - Another one bites the dust

Just a quick update - we had a few new interesting bots, and another menacing crawl from Wget...
  • Wget/1.11.3 returned with a vengeance today, this time from IP 131.107.0.106. They burned through 66,000 page views in just a few hours. As usual, we banned this IP and life will go on. related...
  • SearchHub Bot is a bot sent from inSynchro, a Malaysian Business Intelligence company established in 1999 that specializes in developing web-based enterprise solutions specifically in the area of Project Management. related...
  • Thanks to the Google "Translate this page" feature, I was able to scout out Semager/1.2. Their site is in German, and from what I can tell in the English translation, the Semager bot feeds the Semager Search Engine. Nothing sinister here - just another search engine bot. We do have a few Semager user agents in stock - related...
Bottom line: 182,499 user agents and 1,959 bots.
More...
User Agent Categories:
All Bots (3,341)
Google Chrome (343)
iPhone Browsers (351)
iPod Browsers (119)
iTunes Browsers (18)
Nintendo WII (34)
Sitemap related (6)
RSS Related (45)
Google Earth Related (44)
Facebook User Agents (8)
Google Related (6,548)
Yahoo! Related (144)
MSN Related (11,858)
Script Injections (113)
Ask Jeeves / Teoma (95)
Ichiro (bots) (6)
Nutch Bots (187)
Avant Browsers (2,363)
Blackberry Browsers (817)
Android Related (32)
Konqueror Browsers (4,868)
MacIntosh Browsers (7,622)
Maxthon Browsers (2,503)
Netscape Browsers (867)
Opera Browsers (5,198)
Safari Browsers (4,614)
Proxy User Agents (337)
Unidentified (10,357)
Search our User Agents:
Top Searches:   [view all]
"iPhone"
"Android"
"iPod"
"mobile"
"Blackberry"
"<script"
"motorola"
"google earth"
"spyware"
"facebook"
"psycheclone"
"iTunes"
"Tablet"
"Spider"
"google image"
"mediapartners google"
"grub client"
"web products"
"spam"
"linux"
"bot"
"media center"
"robot"
"alawar"
"Red Hat"
Supported by Aptitude Technology Limited