Scrapyd

Latest version: v1.5.0

Safety actively analyzes 682404 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 2 of 3

1.3.0

------------------

Added
~~~~~

- Add :ref:`username` and :ref:`password` settings, for HTTP authentication.
- Add :ref:`jobstorage` and :ref:`eggstorage` settings.
- Add a ``priority`` argument to the :ref:`schedule.json` webservice.
- Add ``project`` to all jobs in the response from the :ref:`listjobs.json` webservice.
- Add shortcut to jobs page to cancel a job using the :ref:`cancel.json` webservice.
- Python 3.7, 3.8, 3.9, 3.10 support.

Changed
~~~~~~~

- Make optional the ``project`` argument to the :ref:`listjobs.json` webservice, to easily query for all jobs.
- Improve HTTP headers across webservices.

Removed
~~~~~~~

- Python 2, 3.3, 3.4, 3.5 support.
- PyPy 2 support.
- Documentation for Ubuntu installs (Zyte no longer maintains the Ubuntu package).

Fixed
~~~~~

- Respect Scrapy's ``TWISTED_REACTOR`` setting.
- Replace deprecated ``SafeConfigParser`` with ``ConfigParser``.

1.2.1

------------------

Fixed
~~~~~

- Fix HTTP header types for newer Twisted versions.
- ``DeferredQueue`` no longer hides a pending job when reaching :ref:`max_proc`.
- The :ref:`addversion.json` webservice now works on Windows.
- test: Update binary eggs to be compatible with Scrapy 1.x.

Removed
~~~~~~~

- Remove deprecated SQLite utilities.

1.2.0

------------------

Added
~~~~~

- Webservice

- Add the :ref:`daemonstatus.json` webservice.
- Add a ``_version`` argument to the :ref:`schedule.json` and :ref:`listspiders.json` webservices.
- Add a ``jobid`` argument to the :ref:`schedule.json` webservice.
- Add ``pid`` to the running jobs in the response from the :ref:`listjobs.json` webservice.
- Include full tracebacks from Scrapy when failing to get spider list.
This makes debugging deployment problems easier, but webservice output noisier.

- Website

- Add a :ref:`webroot` setting for website root class.
- Add start and finish times to jobs page.

- Make console script executable.
- Add contributing documentation.
- Twisted 16 support.
- Python 3 support.

Changed
~~~~~~~

- Change :ref:`bind_address` default to 127.0.0.1, instead of 0.0.0.0, to listen only for connections from localhost.

Removed
~~~~~~~

- Deprecate unused SQLite utilities in the ``scrapyd.sqlite`` module.

- ``SqliteDict``
- ``SqlitePickleDict``
- ``SqlitePriorityQueue``
- ``PickleSqlitePriorityQueue``

- Scrapy 0.x support.
- Python 2.6 support.

Fixed
~~~~~

- Poller race condition for concurrently accessed queues.

1.1.1

------------------

Added
~~~~~

- Document and include missing settings in ``default_scrapyd.conf``.
- Document the spider queue's ``priority`` argument.
- Enable some missing tests for the SQLite queues.

Removed
~~~~~~~

- Disable bdist_wheel command in setup to define dynamic requirements, despite pip-7 wheel caching bug.

Fixed
~~~~~

- Use correct type adapter for sqlite3 blobs. In some systems, a wrong type adapter leads to incorrect buffer reads/writes.
- ``FEED_URI`` was always overridden by Scrapyd.
- Specify maximum versions for requirements that became incompatible.
- Mark package as zip-unsafe because Twistd requires a plain ``txapp.py``.

1.1.0

------------------

Added
~~~~~

- Add ``node_name`` (hostname) to webservice responses. (:commit:`fac3a5c`, :commit:`4aebe1c`)
- Add ``start_time`` to the running jobs in the response from the :ref:`listjobs.json` webservice. (:commit:`6712af9`, :commit:`acd460b`)

Changed
~~~~~~~

- Move ``scrapyd-deploy`` command to `scrapyd-client <https://pypi.org/project/scrapyd-client/>`__ package. (:commit:`c1358dc`, :commit:`c9d66ca`, :commit:`191353e`)
- Allow the :ref:`items_dir` setting to be a URL. (:commit:`e261591`, :commit:`35a21db`)
- Look for a ``~/.scrapyd.conf`` file in the user's home directory. (:commit:`1fce99b`)

Fixed
~~~~~

- Check if a spider exists before scheduling it. (:issue:`8`, :commit:`288afef`, :commit:`a185ff2`)
- Sanitize version names when creating egg paths. (:commit:`8023720`)
- Generate correct feed URIs, using w3lib. (:commit:`9a88ea5`)
- Fix git versioning for projects without annotated tags. (:issue:`34`, :commit:`e91dcf4`)
- Use valid HTML markup on website pages. (:commit:`da5664f`, :commit:`26089cd`)
- Use ``file`` protocol for ``SCRAPY_FEED_URI`` environment variable on Windows. (:commit:`4f0060a`)
- Copy ``JsonResource`` class from Scrapy, which no longer provides it. (:commit:`99ea920`)
- Lowercase ``scrapyd`` package name. (:commit:`1adfc31`).
- Mark package as zip-unsafe, because Twisted requires a plain ``txapp.py``. (:commit:`f27c054`)
- Install scripts using ``entry_points`` instead of ``scripts``. (:commit:`b670f5e`)

1.0.2

------------------

Fixed
~~~~~

- Mark package as zip-unsafe, because Twisted requires a plain ``txapp.py``.
- Specify maximum versions for compatible requirements.

Page 2 of 3

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.