Bots vs Browsers - Public Bot / User Agent Database & Commentary

Bots vs Browsers - database of 1,421,386 user agents and growing


The Latest:


DEC 13

2013

Are the Bots beating the Browsers???


After years of observing Bots vs Browsers by analyzing user agent logs, the struggle appears to have swung heavily in favor of the bots in 2013. As late as 2012, the browsers (humans) were still within 1% of the majority of website traffic, but 2013 saw robot traffic increase with a vengeance.

According to an article published by Incapsula, robots are now responsible for 61.5% of all website traffic. The study last year measured much closer, at 51% of all website traffic coming from bots. This pushes human traffic down from 2012 at 49% down to 38.5% in 2013.

As far as the breakdown of what type of robots are responsible for all of this traffic, here is a quick list:
  • 31% Search Engines (i.e. the "good" bots)
  • 5% Scrapers
  • 4.5% Hacking tools
  • 0.5% Spammers
  • 20.5% Other Impersonators (basically all other non-humans)
You can read more about the findings here.
FEB 3

2013

Tablets, Slates, and Touch-enabled Devices - Part 1 : The iPad


New form factors for computers have changed the way we consume the web, and along the way we have left behind a trail of breadcrumbs in our user agents that we can now look back on to see this progression. This week we'll focus on Apple's contribution to this space with the release of the iPad. Here are some popular variations of the iPad user agent across all versions.

  • Remember that time when Apple decided the tablets of the time sucked and we needed something more practical? In January of 2010, Apple announced the 1st generation iPad, which was released in April of 2010. Not only did this mark the birth of the iPad, but also the birth of a slimmer, more portable and stripped-down personal computing experience. Here are some variations of the first generation iPad user agent.
  • In March 2011, Apple went on to improve in the first generation iPad. The second generation iPad was announced March 2, 2011, and released on March 11, 2011. With the second iPad came double the memory of the original device, from 256 MB to 512 MB. Second generation iPads also introduced a 0.7 MP camera, added a gyroscope sensor to the device, and upgraded the iOS version 5.1.1 to 6.1. Here are some second generation iPad user agents that we have seen over the years.
  • Almost exactly one year after the iPad 2, Apple announced the iPad 3. The third generation iPad was announced March 7, 2012, and released on March 16, 2012. Once again the new iPad 3 doubled the memory of the device, this time from 512 MB to 1 GB. The display of the iPad 3 sported the new Retina Display technology that Apple has become known for. Other upgrades included Bluetooth 4.0 (from version 2.1 on previous iPads), and major upgrades to the camera to bring it up to 5 megapixel and 1080p HD video recording. Third generation iPad user agents began showing up, and we were listening.
  • In the interests of breaking tradition, Apple announced the iPad 4 only 8 months after the iPad 3's release. On October 23, 2012 Apple announced the new iPad 4, which would be released to consumers on November 2, 2012. While many criticized the iPad 4 of being too similar to the iPad 3, and accused Apple of releasing a new iPad only to combat the release of the Microsoft Surface RT, there were some areas that were upgraded. The fourth generation iPad improved the front-facing camera from .3 megapixel to 1.2 megapixel, and changed the device's processor from a 1 GHz dual-core ARM processor to a 1.4 GHz dual-core Apple Swift processor. Most importantly, the iPad 4 added a new storage size of 128 GB - all previous iPad versions were available with 3 storage sizes: 16, 32, or 64 GB. iPad 4 user agents have been appearing ever since, and the rest is history.
  • In the same announcement as the iPad 4, Apple announced an iPad Mini as well. As far as user agents go, the iPad Mini is a bit of a mystery. Apple does not identify an iPad Mini separately from other iPads, so developers trying to target this model had to use other means to detect the devices' capabilities.
We hope this post is helpful and informative to you, and we enjoyed revisiting the history of the iPad user agent. Thanks for stopping by!

Our growing total of user agents is now at 1,021,791, with 20,089 bots included in the mix.
JAN 15

2013

7 Years Old, One Million User Agents!


Wow - 7 years old. What started out as a goofy idea of narrating user agents in our web traffic logs has grown so much in 7 years. Not to mention that four our seventh birthday, we found our one-millionth user agent to put us into seven-digits! We are now sitting at over 1 million user agents, of which 20,000+ are identified as bots.

The most notable change in the last seven years has to be the nature of devices that we see from our sites and contributing sites. In the early days, most user agents were either Internet Explorer, Netscape, Opera, or some form of Mozilla, and everything else was pretty much bots or proxies. Now looking over our categories, mobile and non-PC user agents seem to be the most diverse and rapid changing. In our first two or three years we never saw advertising, spamming, or script injection attempts in user agents. Since the Script Injections category appeared, it has grown rapidly, and the number of user agents containing spam has also been a challenge to weed out from useful data.

We've come a long way from the days where we just wanted to know if you were Netscape Gold 3 or Netscape Navigator 4, and if your browser was capable of handling that new-fangled concept of client-side JavaScript. As always, thanks to our site contributors who take the time to send us their data to mine through, and to our readers - we hope that you continue to find our site useful in the coming years!

Over the next few weeks and months, we'll focus on posting about different areas that have changed in the User Agents realm over the past few years. Look for posts to focus on different user agents from phones, tablet and touch enabled devices, gaming consoles, and last but not least, bots!

At seven years of operation, we now have 1,000,047 user agents and 20,014 bots tracked to date!
JAN 12

2010

Happy 4th Birthday, Bots vs Browsers!


Today we turn 4 years old! Thanks to everyone who has contributed to the site, and to our readers - you are the reason that we are here in the first place!

Here is the latest news from our user agent community:

In our four years of operation, we have added 421,718 user agents and 4,030 bots to our community database!
JUL 12

2009

300,000 User Agents!


This was a big month for the site - we finally broke 300,000 user agents! To celebrate, we also added some features to our IP Directory - here are the details: This installment brings us to 316,779 user agents and 3,483 total bots.
MAY 31

2009

Summer Bots make a splash!


With summer beginning, we saw a big splash from some new bots, and some new versions of old familar bots. This puts us dangerously close to the 300,000 mark with 297,455 user agents and 3,233 bots.
APR 18

2009

Where did Palm go?


Recently I was helping a client setup their phone for Google Apps' mail (Just GMail with a different face). I had gone through some very helpful support articles which included instructions for phones running Android, Windows Mobile 5 and 6, iPhone, Blackberry, and the list goes on. The client had told me they were running a Treo 700, which I assumed was running Windows Mobile 5 or 6. After some research, I found that their version was actually a Treo 700p running Palm's OS, not the Treo 700w running Windows Mobile. In looking for instructions for setting this up, help was scarce, even from Google's extensive mobile help knowledgebase.

Considering their former prominence in the mobile market, one would think that legacy support for Palm's operating systems would still be strong. Granted, I have not used a Palm in over 10 years, and back then it was a basic Palm Pilot PDA in black and white, but for many years they were the prevailing mobile device manufacturer.

In light of their market domination in handheld devices between 2000 and 2005, I became curious about how they disappeared so quickly. In doing a little research, it seems that Palm just missed the boat on the smartphone industry. They had an early competitive advantage in the market, but were late to market and slow to innovate in the smartphone revolution. The idea of a PDA without a phone built-in is redundant these days, with the only exception being the iPod Touch.

Palm is all but dead in the phone market and smaller PDA market these days. The strangest part about Palm's lingering PDA presence is their online Palm shop. They still list three models of phone-less PDA's, but when viewing their availability, all three report that "This product is not currently available on the Palm Store".

In memory of Palm's former market presence, I reviewed our user agent logs to get an idea of what we've seen over the years. Over the last 4 years, we have captured about 70 variations of Palm user agents, few of which ran Palm OS.

FEB 18

2009

Happy Belated 3rd Birthday to Us!


So it's been a while since we've posted our progress, so here goes:
We've been busy, and so have our web traffic logs - we've discovered 320 bots since our last post, and found over 31,000 other new user agents since then. There's alot of noise in those logs, so here are the high points that we hope will interest you: This brings us to 252,674 user agents and 2,828 bots.

NOV 23

2008

Bots should be thankful


This thanksgiving, all the bots out there should give thanks for the masses of bandwidth they chew each day at the expense of webmasters around the world. If our logs are any indication, few bots actually bring enough human traffic to make up for their crawls, so there's alot of free-loading bots out there. Anyways, we had over 200 new faces, and here are the ones we found interesting: This month brings us to 221,187 user agents and 2,508 bots. Thanks for visiting!
OCT 22

2008

Would you say I have a plethora of new bots?


We have a new category and over 100 new bots this month. Let's cut to the chase: This installment brings us to 212,592 User Agents and 2,376 bots.
More...