Scrapyd 1.5.0b1¶
Scrapyd is a service for deploying and running Scrapy spiders.
It allows you to upload Scrapy projects and control their spiders using a JSON API.
(If you are viewing this on GitHub, open the full documentation for additional details.)
Quickstart¶
Install Scrapyd¶
pip install scrapyd
Start Scrapyd¶
scrapyd
See Overview and Configuration for more details.
Upload a project¶
This involves building a Python egg and uploading it to Scrapyd via the addversion.json webservice.
Do this easily with the scrapyd-deploy command from the scrapyd-client package. Once configured:
scrapyd-deploy
Schedule a crawl¶
$ curl http://localhost:6800/schedule.json -d project=myproject -d spider=spider2
{"status": "ok", "jobid": "26d1b1a6d6f111e0be5c001e648c57f8"}
See API for more details.