Scrapyd

Scrapyd is an application for deploying and running Scrapy spiders. It enables you to deploy (upload) your projects and control their spiders using a JSON API.

Contents

• Overview
  • Projects and versions
  • How Scrapyd works
  • Starting Scrapyd
  • Scheduling a spider run
  • Web Interface
• Installation
  • Requirements
  • Installing Scrapyd (generic way)
• Deploying your project
• API
  • daemonstatus.json
  • addversion.json
  • schedule.json
  • cancel.json
  • listprojects.json
  • listversions.json
  • listspiders.json
  • listjobs.json
  • delversion.json
  • delproject.json
• Configuration file
  • http_port
  • bind_address
  • username
  • password
  • max_proc
  • max_proc_per_cpu
  • debug
  • eggs_dir
  • dbs_dir
  • logs_dir
  • items_dir
  • jobs_to_keep
  • finished_to_keep
  • poll_interval
  • runner
  • application
  • webroot
  • jobstorage
  • eggstorage
  • node_name
  • Example configuration file
• Contributing
  • Reporting Issues & Bugs
  • Tests
  • Running tests
  • Writing tests
  • Installing Locally
• Release notes
  • 1.3.0
  • 1.2.1
  • 1.2.0
  • 1.1.1
  • 1.1.0
  • 1.0.2
  • 1.0.1
  • 1.0.0