Path: www-robot :

WWW Robots  Web robots...

  title-bot.tar.gz - World Wide Web robot that collects all root document titles for all sites on the Internet.
  webspider.tar.gz - a simple web robot useing Java.
  webspider.zip
  bot.tar.gz - a simple web robot using C++.