|
|
|
|
Bots vs Browsers - database of 190,184 user agents and growing
|
2006.12.23 - Have a Merry-Bot-Christmas
After moving to the new server, we synchronized our logs and found 69 new bots over the last few weeks, which pushed us past the 700 bot milestone!
And without further ado, the highlights:
-
Next up, YodaoBot/1.0
dropped in for a visit. We couldn't tell much about this bot, as it's home site appears to be in Japanese. However, it looks like they honor the robots.txt standard.
related...
-
We ran across Twiceler in our logs, and visited their site.
Their informational page was brief and to the point, saying "Twiceler is an experimental robot. Please contact costello@cuill.com if you have any problems.
Twiceler should obey robots.txt.". I guess that says it all...
related...
-
Dries Buytaert sent his personal bot over.
According to his blog, he is a PHD student at the University of Ghent (Belgium), and lead of the
Drupal project.
related...
-
DepSpid/5.07, "The Dependency Spider", dropped in and provided a URL for it's site.
Upon visiting, we found out (in verbatim) that "DepSpid is a distributed kind of a web crawler. The DepSpid spider visits domains, analyses links and finally calculates scores about the link dependencies between individual domains. Each spider job starts at the main page of a domain and then follows each link on that page retrieving more pages and analysing them, too. The spider stays within one domain. If it finds an external link it only checks if the linked domain is reachable but doesn't continue crawling into the external domain. Every unknown domain will be visited from another spider job at a later time.
The DepSpid spider is currently under devlopment. Once it's running in production mode, the data collected by the spider will be publically available and will give webmasters a new kind of sight into their own or foreign domains."
related...
We now have 51,490 user agents and 720 bots
|
Blog Archives: |
2008.7.13
|
2,000 bots!
|
2008.7.6
|
Dog Days of Summer Bots
|
2008.6.23
|
Another one bites the dust
|
2008.6.22
|
Bots don't run out of gas
|
2008.6.19
|
And we're back...
|
2008.1.21
|
Happy 2nd Birthday!
|
2007.10.25
|
The Creature Post
|
2007.9.7
|
iTunes, Facebook, Google Earth, and more
|
2007.7.22
|
Hot Summer Bots
|
2007.7.6
|
The Apple iPhones have landed!
|
2007.6.13
|
Twiceler calls off the bots
|
2007.6.13
|
Twiceler Strikes!
|
2007.6.3
|
Land of 1,000 bots
|
2007.4.22
|
April Bots Bring May Traffic... (we hope)
|
2007.4.16
|
Spring Cleaning in the Web Traffic Logs
|
2007.3.24
|
March Bot Madness
|
2007.3.10
|
Got Spiders?
|
2007.2.24
|
Some New Faces (bots, of course)
|
2007.2.10
|
Winks, DataFountains, Hijacks, and Tool - what's the connection?
|
2007.2.3
|
February Stars
|
2007.1.12
|
Happy 1st Birthday!
|
2007.1.1
|
The 2007 bots are in!
|
2006.12.23
|
Have a Merry-Bot-Christmas
|
2006.12.12
|
50,000 User Agents!
|
2006.11.4
|
2 weeks, 19 new bots
|
2006.10.17
|
2 weeks, 30 bots
|
2006.10.2
|
4 weeks, 23 bots
|
2006.9.9
|
Playing catchup with the bots
|
2006.8.26
|
500 bots!
|
2006.8.20
|
5 days, 8 bots
|
2006.8.15
|
Two weeks, 15 new bots
|
2006.8.2
|
Another week, another gang of new bots
|
2006.7.25
|
40,000 User Agents!!!
|
2006.7.4
|
New category, bot revisions, and 6 new bots for July 4th
|
2006.6.25
|
New batch of imports!
|
2006.6.18
|
5 new bots, just in time for Father's Day!
|
2006.6.4
|
8 new bots for the database
|
2006.5.19
|
A couple new sightings
|
2006.5.9
|
Quiet week, one new bot
|
2006.5.2
|
A few new bots
|
2006.4.20
|
A quick bot-sighting update
|
2006.4.12
|
More user agents, more bots
|
2006.3.19
|
5,000+ New User Agents
|
2006.3.9
|
42 more bots, lots of new user agents
|
2006.2.19
|
New Bots and Browsers just addded!
|
2006.1.25
|
Welcome to Bots vs Browsers!
|
2006.1.17
|
Welcome to the User Agents Database that has no name yet!
|
2006.1.12
|
Day One
|
|
|
|
|