Bots vs Browsers - Public Bot / User Agent Database & Commentary
Bots vs Browsers - database of 1,485,082 user agents and growing
The Latest:
MAR 17
2016
User Agent Debuts for Brave and Servo Browsers
As far as new browsers go, Microsoft's Edge browser release was the only major news last year.
However, over the last few weeks, two other browsers-in-the-making have caught our attention in the news.
Thanks to our users, we also have sample user agents for both that we are happy to share.
A Brave new Browser
Brave is an ambitious new browser that intends to revolutionize users' online experience by making browsing faster and safer.
To be specific, this would include blocking ads and trackers that are involved with gathering user information.
The basic idea is that Brave would insert their own non-intrusive ads into the blocked ads' slots, and publishers would be paid via a new revenue share model that is described in more detail here.
The company was founded by Brendan Eich, who is best known as the creator of JavaScript.
Read more about Brave by visiting their official site at Brave.com
and check out our new Brave user agents
Servo, the Parallel Browser Engine Project
Mozilla is building a new browser engine called Servo using their own programming language Rust.
Using their own words, "Servo is a modern, performant browser engine designed to be appropriate for applications including embedded use".
Servo has received a great deal of support in the industry beyond just Mozilla and individual contributors - Samsung has also contributed heavily to the efforts.
To celebrate leap day we are excited to announce the launch of "Can I Crawl?".
This is a new tool that we've launched to help webmasters and robots with interpreting and understanding robots.txt files.
The tool features color coding for each type of directive, in-line links for URL's in the robots.txt file, descriptions for each directive, and a way to switch from an interactive view to a raw view.
It even divides the file into sections by User-Agent directive to help readability.
If you're setting up a robots.txt file to instruct bots on how to crawl (or not crawl) your site, this tool is great for checking your instructions for accuracy.
Or maybe you have a bot and need to check a site's robots.txt file for instructions and sitemaps to help your crawl - this tool is perfect for that as well.
We hope you find the tool useful - Enjoy!
JAN 20
2016
10 Years Old, and Big Changes for Bots vs Browsers!
To celebrate the site's 10th year in operation, we're working hard to make sure Bots vs Browsers becomes more useful than ever.
We recently released a responsive mobile version of the site to accomodate our users on smaller screens.
We also implemented additional features for IP address details, including new geographical info and ASN details.
In addition, we have changed our user agent categories menu structure to handle extensive new classifications.
As our categories and classifications have grown organically over the years, some need to be broken down into smaller categories.
Older categories need to be set aside for users doing historical research, and more relevant newer categories need to be promoted into higher prominenence for users that are interested in more cutting-edge research.
To accomodate this, we have built a new classification structure and built new pattern-matching sequences to place user agents into their proper categories.
This will be an ongoing process, and as our site grows, we have built a new menu structure to hold large numbers of categories and ease navigation around the site.
We will also be addressing our search indexing, as the database has grown considerably since our first version of the site.
We did not anticipate searching over 1 million user agents when we first released, so it's time for an update!
At the 10 year mark, Bots vs Browsers now has 1,453,272 user agents, and we have identified 24,128 bots out of the mix.
JAN 6
2016
Internet Explorer : Out with the old, in with the new!
Just in time for the new year, Microsoft has announced that they will be pulling the plug on Internet Explorer versions 8, 9, and 10 next Tuesday (January 12).
The final patch for these browser versions will be released, and there will be no more security updates or technical support after that.
As with any legacy version retirement, it's never that simple.
Microsoft has several exclusions for older versions of IE on certain operating systems and under certain conditions.
If you are curious if your system or users will be affected, then you should read the details on Microsoft's full announcement here.
Microsoft released an "accidental" preview site for the upcoming Edge browser extensions.
The release was a preview for a developer site which announced Edge browser extensions support.
Indications from the preview site are that developers will have the chance to write extensions ahead of the general release, and Windows 10 testers would have the ability to try the extenstions out ahead of time as well.
Microsoft had originally planned to release extensions for the Edge browser in a patch release of the Windows 10 operating system, but delays caused the extensions release to be moved to early 2016.
It is likely that the extensions would leverage the Windows Store, which is integrated with Windows 10.
The Edge browser was released with Windows 10 initially, and is not available for previous versions of Windows.
From the screenshot of the extensions site, it appears that extensions for Reddit and Pinterest are available for download.
Details of the extensions preview site from Microsoft can be found on Twitter here.
After years of observing Bots vs Browsers by analyzing user agent logs, the struggle appears to have swung heavily in favor of the bots in 2013. As late as 2012, the browsers (humans) were still within 1% of the majority of website traffic, but 2013 saw robot traffic increase with a vengeance.
According to an article published by Incapsula, robots are now responsible for 61.5% of all website traffic.
The study last year measured much closer, at 51% of all website traffic coming from bots.
This pushes human traffic down from 2012 at 49% down to 38.5% in 2013.
As far as the breakdown of what type of robots are responsible for all of this traffic, here is a quick list:
31% Search Engines (i.e. the "good" bots)
5% Scrapers
4.5% Hacking tools
0.5% Spammers
20.5% Other Impersonators (basically all other non-humans)
Tablets, Slates, and Touch-enabled Devices - Part 1 : The iPad
New form factors for computers have changed the way we consume the web, and along the way we have left behind a trail of breadcrumbs in our user agents that we can now look back on to see this progression.
This week we'll focus on Apple's contribution to this space with the release of the iPad.
Here are some popular variations of the iPad user agent across all versions.
Remember that time when Apple decided the tablets of the time sucked and we needed something more practical?
In January of 2010, Apple announced the 1st generation iPad, which was released in April of 2010.
Not only did this mark the birth of the iPad, but also the birth of a slimmer, more portable and stripped-down personal computing experience.
Here are some variations of the first generation iPad user agent.
In March 2011, Apple went on to improve in the first generation iPad.
The second generation iPad was announced March 2, 2011, and released on March 11, 2011.
With the second iPad came double the memory of the original device, from 256 MB to 512 MB.
Second generation iPads also introduced a 0.7 MP camera, added a gyroscope sensor to the device, and upgraded the iOS version 5.1.1 to 6.1.
Here are some second generation iPad user agents that we have seen over the years.
Almost exactly one year after the iPad 2, Apple announced the iPad 3.
The third generation iPad was announced March 7, 2012, and released on March 16, 2012.
Once again the new iPad 3 doubled the memory of the device, this time from 512 MB to 1 GB.
The display of the iPad 3 sported the new Retina Display technology that Apple has become known for.
Other upgrades included Bluetooth 4.0 (from version 2.1 on previous iPads), and major upgrades to the camera to bring it up to 5 megapixel and 1080p HD video recording.
Third generation iPad user agents began showing up, and we were listening.
In the interests of breaking tradition, Apple announced the iPad 4 only 8 months after the iPad 3's release.
On October 23, 2012 Apple announced the new iPad 4, which would be released to consumers on November 2, 2012.
While many criticized the iPad 4 of being too similar to the iPad 3, and accused Apple of releasing a new iPad only to combat the release of the Microsoft Surface RT, there were some areas that were upgraded.
The fourth generation iPad improved the front-facing camera from .3 megapixel to 1.2 megapixel, and changed the device's processor from a 1 GHz dual-core ARM processor to a 1.4 GHz dual-core Apple Swift processor.
Most importantly, the iPad 4 added a new storage size of 128 GB - all previous iPad versions were available with 3 storage sizes: 16, 32, or 64 GB.
iPad 4 user agents have been appearing ever since, and the rest is history.
In the same announcement as the iPad 4, Apple announced an iPad Mini as well.
As far as user agents go, the iPad Mini is a bit of a mystery.
Apple does not identify an iPad Mini separately from other iPads, so developers trying to target this model had to use other means to detect the devices' capabilities.
We hope this post is helpful and informative to you, and we enjoyed revisiting the history of the iPad user agent. Thanks for stopping by!
Our growing total of user agents is now at 1,021,791, with 20,089 bots included in the mix.
JAN 15
2013
7 Years Old, One Million User Agents!
Wow - 7 years old. What started out as a goofy idea of narrating user agents in our web traffic logs has grown so much in 7 years.
Not to mention that four our seventh birthday, we found our one-millionth user agent to put us into seven-digits!
We are now sitting at over 1 million user agents, of which 20,000+ are identified as bots.
The most notable change in the last seven years has to be the nature of devices that we see from our sites and contributing sites.
In the early days, most user agents were either Internet Explorer, Netscape, Opera, or some form of Mozilla, and everything else was pretty much bots or proxies.
Now looking over our categories, mobile and non-PC user agents seem to be the most diverse and rapid changing.
In our first two or three years we never saw advertising, spamming, or script injection attempts in user agents.
Since the Script Injections category appeared, it has grown rapidly, and the number of user agents containing spam has also been a challenge to weed out from useful data.
We've come a long way from the days where we just wanted to know if you were Netscape Gold 3 or Netscape Navigator 4, and if your browser was capable of handling that new-fangled concept of client-side JavaScript.
As always, thanks to our site contributors who take the time to send us their data to mine through, and to our readers - we hope that you continue to find our site useful in the coming years!
Over the next few weeks and months, we'll focus on posting about different areas that have changed in the User Agents realm over the past few years.
Look for posts to focus on different user agents from phones, tablet and touch enabled devices, gaming consoles, and last but not least, bots!
At seven years of operation, we now have 1,000,047 user agents and 20,014 bots tracked to date!
JAN 12
2010
Happy 4th Birthday, Bots vs Browsers!
Today we turn 4 years old!
Thanks to everyone who has contributed to the site, and to our readers - you are the reason that we are here in the first place!
Here is the latest news from our user agent community:
Macrothela bot variation Rassler has been steadily advancing its bot.
We have seen versions 0.17 through 0.24 in the last few months.
In addition, many new flavors of the Macrothela bot have been spotted, including atlantic, mars, martin, and zeger.
related...
In our four years of operation, we have added 421,718 user agents and 4,030 bots to our community database!
JUL 12
2009
300,000 User Agents!
This was a big month for the site - we finally broke 300,000 user agents!
To celebrate, we also added some features to our IP Directory - here are the details:
We recently added a new feature to our IP Directory.
We now provide best-guess geographical information about each IP address including country, city, region, and even Latitude and longitude.
Just think - now you can find out the hometown of all your favorite bots!
Another crafty script injection attempt - this attack was geared towards SQL injection, not cross site scripting.
This is the part of the post where I get on my soapbox and tell you that you should protect any and all sites you work on by adding protections from the web application layer all the way down to proper database permissions.
related...