網頁抓取工具：PyRailgun

這是一個非常簡單易用的抓取工具

怎么使用? 首先你需要創建一個對應站點的規則文件 比如testsite.yaml

action: main
name: "vc動漫"
subaction:
- action: fetcher
  url: http://www.verycd.com/base/cartoon/page${1,1}${0,9}
  subaction:
  - action: parser
    rule: .entry_cover_list li
    subaction:
    - action: shell
      group: default
      subaction:
        - {action: parser, rule: '.entry_cover .cover_img', setField: img}
        - {action: parser, rule: 'a', pos: 0, attr: href, setField: src}
        - {action: parser, strip: 'true', rule: '.entry_cover .score', setField: score}
        - {action: parser, rule: '.bio a', setField: dest}
        - action: fetcher
          url: http://www.verycd.com${#src}
          subaction:
          - {action: parser,strip: 'true', rule: '#contents_more', setField: description}

然后在代碼里面把它作為一個任務加入到railgun

from railgun import RailGun

railgun = RailGun()
railgun.setTask(file("testsite.yaml"));
railgun.fire();
nodes = railgun.getShells('default')
print nodes

然后你就可以得到一個包含了所有解析后數據的節點列表 [{img:xxx,src:xxx,score:xxx,dest:xxx,description:xxx},{img:xxx,src:xxx,score:xxx,dest:xxx,description:xxx}]

項目主頁：http://www.baiduhome.net/lib/view/home/1362052372584