|
|
|
|
Bots vs Browsers - database of 234,004 user agents and growing
|
2008.6.22 - Bots don't run out of gas
Gas prices don't bother bots - they just want content. Or just email addresses, if they are that kind of bot. Anyways, that was my attempt at relating bots to current events.
This has been an interesting week. Another batch of new bots, email correspondence with one of them, and a direct attempt to manipulate our site's JavaScript from another:
-
We'll start with the attempt to call a JavaScript function on our site.
After doing some homework, someone realized that we have a JavaScript function called setUserAgent, which is used on our
User Agent Test Track.
Of course, the script didn't get a chance to execute because we plan for this sort of stuff.
The attempt looked like this - "javascript:void(setUserAgent('bwh3_user_agent'));".
related...
-
While monitoring this week, a bot from page-store.com got on my radar from their volume / rate of crawl.
Nothing worth banning, but it got my attention.
Luckily, they were kind enough to provide an email address in the user agent.
I emailed them, and the problem was very quickly resolved.
I also asked them to tell us a little about what they do.
From their reply, Page-store contracts with search-engine startup companies to supply them with web crawl data.
The value added includes site-aggregation, character encoding handling, language filtering, porn filtering, spam filtering, on-demand depth-k crawl, site-level and URL-level web-link data.
Unlike many of the bots / scrapers we run across, this one actually has a legitimate purpose, and responded to our request to conserve bandwidth.
related...
-
BebopBot/2.5.1, which apparently has a passion for Jazz, stopped in this week.
related...
-
Here's a list of some other interesting bots that made their debut in our logs this week:
After this week, our database is up to 182,223 user agents and 1,957 bots.
|
Blog Archives: |
2008.11.23
|
Bots should be thankful
|
2008.10.22
|
Would you say I have a plethora of new bots?
|
2008.9.6
|
200,000 User Agents!
|
2008.8.3
|
Here comes August...
|
2008.7.13
|
2,000 bots!
|
2008.7.6
|
Dog Days of Summer Bots
|
2008.6.23
|
Another one bites the dust
|
2008.6.22
|
Bots don't run out of gas
|
2008.6.19
|
And we're back...
|
2008.1.21
|
Happy 2nd Birthday!
|
2007.10.25
|
The Creature Post
|
2007.9.7
|
iTunes, Facebook, Google Earth, and more
|
2007.7.22
|
Hot Summer Bots
|
2007.7.6
|
The Apple iPhones have landed!
|
2007.6.13
|
Twiceler calls off the bots
|
2007.6.13
|
Twiceler Strikes!
|
2007.6.3
|
Land of 1,000 bots
|
2007.4.22
|
April Bots Bring May Traffic... (we hope)
|
2007.4.16
|
Spring Cleaning in the Web Traffic Logs
|
2007.3.24
|
March Bot Madness
|
2007.3.10
|
Got Spiders?
|
2007.2.24
|
Some New Faces (bots, of course)
|
2007.2.10
|
Winks, DataFountains, Hijacks, and Tool - what's the connection?
|
2007.2.3
|
February Stars
|
2007.1.12
|
Happy 1st Birthday!
|
2007.1.1
|
The 2007 bots are in!
|
2006.12.23
|
Have a Merry-Bot-Christmas
|
2006.12.12
|
50,000 User Agents!
|
2006.11.4
|
2 weeks, 19 new bots
|
2006.10.17
|
2 weeks, 30 bots
|
2006.10.2
|
4 weeks, 23 bots
|
2006.9.9
|
Playing catchup with the bots
|
2006.8.26
|
500 bots!
|
2006.8.20
|
5 days, 8 bots
|
2006.8.15
|
Two weeks, 15 new bots
|
2006.8.2
|
Another week, another gang of new bots
|
2006.7.25
|
40,000 User Agents!!!
|
2006.7.4
|
New category, bot revisions, and 6 new bots for July 4th
|
2006.6.25
|
New batch of imports!
|
2006.6.18
|
5 new bots, just in time for Father's Day!
|
2006.6.4
|
8 new bots for the database
|
2006.5.19
|
A couple new sightings
|
2006.5.9
|
Quiet week, one new bot
|
2006.5.2
|
A few new bots
|
2006.4.20
|
A quick bot-sighting update
|
2006.4.12
|
More user agents, more bots
|
2006.3.19
|
5,000+ New User Agents
|
2006.3.9
|
42 more bots, lots of new user agents
|
2006.2.19
|
New Bots and Browsers just addded!
|
2006.1.25
|
Welcome to Bots vs Browsers!
|
2006.1.17
|
Welcome to the User Agents Database that has no name yet!
|
2006.1.12
|
Day One
|
|
|
|
|