|
|
|
|
Bots vs Browsers - database of 189,416 user agents and growing
|
2007.6.13 - Twiceler calls off the bots
Twiceler was more than helpful with my request to slow down on our sites. Within half an hour of my email, here is their reply:
Sorry for any inconvenience. We should not be crawling
your sites that fast. I will stop crawling everything in your
IP space immediately.
I will check that we understand your robots.txt, but robots.txt
often has significant time delay as the standard says that you
should cache the robots.txt file for seven days.
Thanks, twiceler! That was surprisingly helpful, and a very fast response.
related...
|
Blog Archives: |
2008.7.13
|
2,000 bots!
|
2008.7.6
|
Dog Days of Summer Bots
|
2008.6.23
|
Another one bites the dust
|
2008.6.22
|
Bots don't run out of gas
|
2008.6.19
|
And we're back...
|
2008.1.21
|
Happy 2nd Birthday!
|
2007.10.25
|
The Creature Post
|
2007.9.7
|
iTunes, Facebook, Google Earth, and more
|
2007.7.22
|
Hot Summer Bots
|
2007.7.6
|
The Apple iPhones have landed!
|
2007.6.13
|
Twiceler calls off the bots
|
2007.6.13
|
Twiceler Strikes!
|
2007.6.3
|
Land of 1,000 bots
|
2007.4.22
|
April Bots Bring May Traffic... (we hope)
|
2007.4.16
|
Spring Cleaning in the Web Traffic Logs
|
2007.3.24
|
March Bot Madness
|
2007.3.10
|
Got Spiders?
|
2007.2.24
|
Some New Faces (bots, of course)
|
2007.2.10
|
Winks, DataFountains, Hijacks, and Tool - what's the connection?
|
2007.2.3
|
February Stars
|
2007.1.12
|
Happy 1st Birthday!
|
2007.1.1
|
The 2007 bots are in!
|
2006.12.23
|
Have a Merry-Bot-Christmas
|
2006.12.12
|
50,000 User Agents!
|
2006.11.4
|
2 weeks, 19 new bots
|
2006.10.17
|
2 weeks, 30 bots
|
2006.10.2
|
4 weeks, 23 bots
|
2006.9.9
|
Playing catchup with the bots
|
2006.8.26
|
500 bots!
|
2006.8.20
|
5 days, 8 bots
|
2006.8.15
|
Two weeks, 15 new bots
|
2006.8.2
|
Another week, another gang of new bots
|
2006.7.25
|
40,000 User Agents!!!
|
2006.7.4
|
New category, bot revisions, and 6 new bots for July 4th
|
2006.6.25
|
New batch of imports!
|
2006.6.18
|
5 new bots, just in time for Father's Day!
|
2006.6.4
|
8 new bots for the database
|
2006.5.19
|
A couple new sightings
|
2006.5.9
|
Quiet week, one new bot
|
2006.5.2
|
A few new bots
|
2006.4.20
|
A quick bot-sighting update
|
2006.4.12
|
More user agents, more bots
|
2006.3.19
|
5,000+ New User Agents
|
2006.3.9
|
42 more bots, lots of new user agents
|
2006.2.19
|
New Bots and Browsers just addded!
|
2006.1.25
|
Welcome to Bots vs Browsers!
|
2006.1.17
|
Welcome to the User Agents Database that has no name yet!
|
2006.1.12
|
Day One
|
|
|
|
|