Bots vs Browsers - database of 1,421,386 user agents and growing
Are the Bots beating the Browsers???
After years of observing Bots vs Browsers by analyzing user agent logs, the struggle appears to have swung heavily in favor of the bots in 2013. As late as 2012, the browsers (humans) were still within 1% of the majority of website traffic, but 2013 saw robot traffic increase with a vengeance.
According to an article published by Incapsula, robots are now responsible for 61.5% of all website traffic.
The study last year measured much closer, at 51% of all website traffic coming from bots.
This pushes human traffic down from 2012 at 49% down to 38.5% in 2013.
As far as the breakdown of what type of robots are responsible for all of this traffic, here is a quick list:
You can read more about the findings here.
- 31% Search Engines (i.e. the "good" bots)
- 5% Scrapers
- 4.5% Hacking tools
- 0.5% Spammers
- 20.5% Other Impersonators (basically all other non-humans)
Tablets, Slates, and Touch-enabled Devices - Part 1 : The iPad
New form factors for computers have changed the way we consume the web, and along the way we have left behind a trail of breadcrumbs in our user agents that we can now look back on to see this progression.
This week we'll focus on Apple's contribution to this space with the release of the iPad.
Here are some popular variations of the iPad user agent across all versions.
We hope this post is helpful and informative to you, and we enjoyed revisiting the history of the iPad user agent. Thanks for stopping by!
Remember that time when Apple decided the tablets of the time sucked and we needed something more practical?
In January of 2010, Apple announced the 1st generation iPad, which was released in April of 2010.
Not only did this mark the birth of the iPad, but also the birth of a slimmer, more portable and stripped-down personal computing experience.
Here are some variations of the first generation iPad user agent.
In March 2011, Apple went on to improve in the first generation iPad.
The second generation iPad was announced March 2, 2011, and released on March 11, 2011.
With the second iPad came double the memory of the original device, from 256 MB to 512 MB.
Second generation iPads also introduced a 0.7 MP camera, added a gyroscope sensor to the device, and upgraded the iOS version 5.1.1 to 6.1.
Here are some second generation iPad user agents that we have seen over the years.
Almost exactly one year after the iPad 2, Apple announced the iPad 3.
The third generation iPad was announced March 7, 2012, and released on March 16, 2012.
Once again the new iPad 3 doubled the memory of the device, this time from 512 MB to 1 GB.
The display of the iPad 3 sported the new Retina Display technology that Apple has become known for.
Other upgrades included Bluetooth 4.0 (from version 2.1 on previous iPads), and major upgrades to the camera to bring it up to 5 megapixel and 1080p HD video recording.
Third generation iPad user agents began showing up, and we were listening.
In the interests of breaking tradition, Apple announced the iPad 4 only 8 months after the iPad 3's release.
On October 23, 2012 Apple announced the new iPad 4, which would be released to consumers on November 2, 2012.
While many criticized the iPad 4 of being too similar to the iPad 3, and accused Apple of releasing a new iPad only to combat the release of the Microsoft Surface RT, there were some areas that were upgraded.
The fourth generation iPad improved the front-facing camera from .3 megapixel to 1.2 megapixel, and changed the device's processor from a 1 GHz dual-core ARM processor to a 1.4 GHz dual-core Apple Swift processor.
Most importantly, the iPad 4 added a new storage size of 128 GB - all previous iPad versions were available with 3 storage sizes: 16, 32, or 64 GB.
iPad 4 user agents have been appearing ever since, and the rest is history.
In the same announcement as the iPad 4, Apple announced an iPad Mini as well.
As far as user agents go, the iPad Mini is a bit of a mystery.
Apple does not identify an iPad Mini separately from other iPads, so developers trying to target this model had to use other means to detect the devices' capabilities.
Our growing total of user agents is now at 1,021,791, with 20,089 bots included in the mix.
7 Years Old, One Million User Agents!
Wow - 7 years old. What started out as a goofy idea of narrating user agents in our web traffic logs has grown so much in 7 years.
Not to mention that four our seventh birthday, we found our one-millionth user agent to put us into seven-digits!
We are now sitting at over 1 million user agents, of which 20,000+ are identified as bots.
The most notable change in the last seven years has to be the nature of devices that we see from our sites and contributing sites.
In the early days, most user agents were either Internet Explorer, Netscape, Opera, or some form of Mozilla, and everything else was pretty much bots or proxies.
Now looking over our categories, mobile and non-PC user agents seem to be the most diverse and rapid changing.
In our first two or three years we never saw advertising, spamming, or script injection attempts in user agents.
Since the Script Injections category appeared, it has grown rapidly, and the number of user agents containing spam has also been a challenge to weed out from useful data.
As always, thanks to our site contributors who take the time to send us their data to mine through, and to our readers - we hope that you continue to find our site useful in the coming years!
Over the next few weeks and months, we'll focus on posting about different areas that have changed in the User Agents realm over the past few years.
Look for posts to focus on different user agents from phones, tablet and touch enabled devices, gaming consoles, and last but not least, bots!
At seven years of operation, we now have 1,000,047 user agents and 20,014 bots tracked to date!
Happy 4th Birthday, Bots vs Browsers!
Today we turn 4 years old!
Thanks to everyone who has contributed to the site, and to our readers - you are the reason that we are here in the first place!
Here is the latest news from our user agent community:
Google AppEngine user agents have been on the rise, so we have started to tag them in their own category.
Macrothela bot variation Rassler has been steadily advancing its bot.
We have seen versions 0.17 through 0.24 in the last few months.
In addition, many new flavors of the Macrothela bot have been spotted, including atlantic, mars, martin, and zeger.
In hacker news, we have seen some new SQL injection attempts -
The first is an attempt access database schema information from an unsecure site -
SELECT * FROM information_schema.tables WHERE information_schema=database().
The second one tries to UNION the numbers 1 through 5 to an existing SELECT statement -
<?php 'UNION ALL SELECT 1,2,3,4,5' php?>
Here are two other non-SQL related script injection attempts against PHP sites - this one tries to include a remote txt file,
and this one is intended to execute the injected PHP code on the server.
We also saw 3 new Sheen web crawlers this month - SheenBot-1.0.0, 1.0.1, and 1.0.2.
In our four years of operation, we have added 421,718 user agents and 4,030 bots to our community database!
This was a big month for the site - we finally broke 300,000 user agents!
To celebrate, we also added some features to our IP Directory - here are the details:
This installment brings us to 316,779 user agents and 3,483 total bots.
We recently added a new feature to our IP Directory.
We now provide best-guess geographical information about each IP address including country, city, region, and even Latitude and longitude.
Just think - now you can find out the hometown of all your favorite bots!
We saw our Rassler collection grow again this month - we had seen versions 0.6 through 0.13, and this month we added
We also saw a surprise visit from the
O'Reilly Link Checker (links from our published works) crawler.
Maybe we are referenced in one of their published works - hopefully in a positive light!
The SaladSpoon/ShopSalad 1.0 bot dropped in as well.
This bot crawls for the online shopping aggregator site known as ShopSalad.com.
New Nutch versions were rampant as usual, with these 11 debuting this month:
Another crafty script injection attempt - this attack was geared towards SQL injection, not cross site scripting.
This is the part of the post where I get on my soapbox and tell you that you should protect any and all sites you work on by adding protections from the web application layer all the way down to proper database permissions.
Here are the failed attempts we saw this month:
'; DROP TABLE `member`--
'; DROP TABLE `member`-- ">'>
Yacybot was extremely busy this month, dropping 23 new versions of their bots in our logs.
We won't bother listing them all, just follow this link to see our Yacybot collection.
Summer Bots make a splash!
With summer beginning, we saw a big splash from some new bots, and some new versions of old familar bots.
This puts us dangerously close to the 300,000 mark with 297,455 user agents and 3,233 bots.
One interesting new bot this month was from the search engine Caret Byte (a.k.a. ^Byte).
The Caret Byte search engine is yet another search engine setting out to serve the global community by crawling free data and re-presenting it for a profit.
According to their site, they respect robots.txt files, so if you do not want to participate in serving the global community, at least you have a choice.
In one of the most astounding feats of software development known to man, the rassler bot advanced from v0.6 to v0.13 in the span of one month.
We saw a new version every few days, so we assume the rassler development crew had a busy month putting out so many builds.
Here are links to the respective versions with their debut dates as well:
2009-05-04 - rassler/0.6
2009-05-08 - rassler/0.7
2009-05-13 - rassler/0.8
2009-05-14 - rassler/0.9
2009-05-24 - rassler/0.10
2009-05-25 - rassler/0.11
2009-05-26 - rassler/0.12
2009-05-29 - rassler/0.13
Heeii/0.3.2 is another new bot this month.
This one comes from the Heeii toolbar that is downloadable as a browser add-on that allows users to recommend links to others.
Yandex is still one of the most active bots in our logs, this month debuting a new bot by the name of
Nutch made the second biggest splash this month as far as new bots - here are the new versions we saw:
By far and away the biggest splash of the month came from bots related to tdmsic.org. We saw 23 new bots this month.
Tdmsic stands for "The Danish Main Securtiy Intelligence Network", a new community which hit beta this year.
According to their site, "TDMSIC engages in research and development of high-leverage computersystem-technology for public and non-public purposes."
Here are a few of the bots that showed up this month:
Recently I was helping a client setup their phone for Google Apps' mail (Just GMail with a different face).
I had gone through some very helpful support articles which included instructions for phones running Android,
Windows Mobile 5 and 6,
Blackberry, and the list goes on.
The client had told me they were running a Treo 700, which I assumed was running Windows Mobile 5 or 6.
After some research, I found that their version was actually a Treo 700p running Palm's OS, not the Treo 700w running Windows Mobile.
In looking for instructions for setting this up, help was scarce, even from Google's extensive mobile help knowledgebase.
Considering their former prominence in the mobile market, one would think that legacy support for Palm's operating systems would still be strong.
Granted, I have not used a Palm in over 10 years, and back then it was a basic Palm Pilot PDA in black and white, but for many years they were the prevailing mobile device manufacturer.
In light of their market domination in handheld devices between 2000 and 2005, I became curious about how they disappeared so quickly.
In doing a little research, it seems that Palm just missed the boat on the smartphone industry.
They had an early competitive advantage in the market, but were late to market and slow to innovate in the smartphone revolution.
The idea of a PDA without a phone built-in is redundant these days, with the only exception being the iPod Touch.
Palm is all but dead in the phone market and smaller PDA market these days.
The strangest part about Palm's lingering PDA presence is their online Palm shop.
They still list three models of phone-less PDA's,
but when viewing their availability, all three report that "This product is not currently available on the Palm Store".
In memory of Palm's former market presence, I reviewed our user agent logs to get an idea of what we've seen over the years.
Over the last 4 years, we have captured about 70 variations of Palm user agents,
few of which ran Palm OS.
Happy Belated 3rd Birthday to Us!
So it's been a while since we've posted our progress, so here goes:
We've been busy, and so have our web traffic logs - we've discovered 320 bots since our last post, and found over 31,000 other new user agents since then.
There's alot of noise in those logs, so here are the high points that we hope will interest you:
This brings us to 252,674 user agents and 2,828 bots.
Yacybot has become one of our most active bots recently.
We now have 90 variations of this bot, and we have seen a fair amount of traffic from them.
These bots are from Yacy.net, a "distributed web search" that can be downloaded and run locally as "a scalable personal web crawler and web search engine".
Sensis Web Crawler
dropped in - according to their site, "Sensis Corporation's Purpose is to provide distinctively elegant, innovative technical solutions in the service of humanity."
Over the past two months, we've had many flavors of Nutch pass by - below is a taste of what we've seen
Where do all the bad bots go? We found out that there is in fact a "Bot Hell"
located about 12 miles southwest of Seattle, Washington.
We're pretty sure it's *not* pronounced "bot-hell", but it's still funny enough. Maybe bots like
devil in disguise (v. 2.0.1a)
Here are a few random bots from the last month that interested us:
is from Japan. I spent some time on their site, but my Japanese is non-existent, and even with the help of Google Translate, I could not figure out what the site does
ZONGOLBOT the Web Spider is a bot from the UK search engine of the same name
is a spider from Flatland Industries. To their credit, they provide not only robots.txt support and an email to contact them at, but they also have a phone number to dial if you are having trouble with one of their bots.
Two new versions of Zontirbot showed up recently
This thanksgiving, all the bots out there should give thanks for the masses of bandwidth they chew each day at the expense of webmasters around the world.
If our logs are any indication, few bots actually bring enough human traffic to make up for their crawls, so there's alot of free-loading bots out there.
Anyways, we had over 200 new faces, and here are the ones we found interesting:
This month brings us to 221,187 user agents and 2,508 bots. Thanks for visiting!
Would you say I have a plethora of new bots?
We have a new category and over 100 new bots this month. Let's cut to the chase:
This installment brings us to 212,592 User Agents and 2,376 bots.
In honor of Google's new browser "Google Chrome", we have created a new category to classify user agents from this browser.
To date, we have 47 user agents in this category, and growing. Be sure to bring your Google Chrome user agent by for a spin to add it to our category.
got busy on our web logs this month showing us 13 new user agents. Variations include OS details, version numbers, and processor architectures.
got busy with our logs, DomainCrawler/1.0
got busier, bringing in 34 new user agents this month.
The variations on this bot are not as interesting - the user agent just varies from each domain by adding the domain to the user agent.
got busy on our web logs this month showing us 13 new user agents. Variations include OS details, version numbers, and processor architectures.
We recorded some of the most interesting Nutch bots ever this month -
Some students at some university unleashed some bot on the web, but we may never know who / what / why this bot even exists:
University_Bot_Beta_1 (compatible; MSIE 6.0;)
Here's a helpful hint: put more information on your user agent if you want notoriety for your bot.
A couple of Pathtraq bots dropped in, Pathtraq/0.1
and Pathtraq/0.1 Gungho/0.09007.
It's worth noting that the second one is Gungho.
Finally, BitvoUserAgent made its debut in our logs.
This bot is a media crawler that is downloadable and available to the public.