Spidr : Ruby開發的Web爬蟲

• Follows:
  • a tags.
  • iframe tags.
  • frame tags.
  • Cookie protected links.
  • HTTP 300, 301, 302, 303 and 307 Redirects.
  • HTTP Basic Auth protected links.
• Black-list or white-list URLs based upon:
  • URL scheme
  • Host name
  • Port number
  • Full link
  • URL extension
• Provides call-backs for:
  • Every visited Page.
  • Every visited URL.
  • Every visited URL that matches a specified pattern.
  • Every URL that failed to be visited.
• Provides action methods to:
  • Pause spidering.
  • Skip processing of pages.
  • Skip processing of links.
• Restore the spidering queue and history from a previous session.
• Custom User-Agent strings.
• Custom proxy settings.
• HTTPS support.

項目主頁：http://www.baiduhome.net/lib/view/home/1349945908431