Wpull

==================

* Changed: when using ``--youtube-dl`` and ``--warc-file``, the JSON metedata file is now saved in the WARC file compatible with pywb.
* Changed: logging and progress meter to say "unspecified" instead of "none" when no content length is provided by server to match Wget.

================

* Security: Updated certificate bundle.
* Fixed: ``--regex-type`` to accept ``pcre`` instead of ``posix``. Regular expressions always use Python's regex library. Posix regex is not supported.
* Fixed: when using ``--warc-max-size`` and ``--warc-append``, it wrote to existing sequential WARC files unnecessarily.
* Fixed: input URLs stored in memory instead of saved on disk. This issue was notable if there were many URLs provided by the ``--input-file`` option.
* Changed: when using ``--warc-max-size`` and ``--warc-append``, the next sequential WARC file is created to avoid appending to corrupt files.
* Changed: WARC file writing to use journal files and refuse to start program if any journals exist. This avoids corrupting files through naive use of ``--warc-append`` and allow for future automated recovery.
* Added: Open Graph and Twitter Card element links extraction.

================

* Fixed: a ``--database`` path with a question mark (``?``) truncated the path, did not use a on-disk database, or causes ``TypeError``. The question mark is automatically replaced with an underscore.
* Fixed: HTTP proxy support broken since version 0.1001.
* Added: ``no_proxy`` environment variable support.
* Added: ``--proxy-domains``, ``--proxy-exclude-domains``, ``--proxy-hostnames``, ``--proxy-exclude-hostnames``
* Removed: ``--no-secure-proxy-tunnel``.

===================

* Added: ``--preserve-permissions``.
* Fixed: exit code returned as 2 instead of 1 on generic errors.
* Changed: exception tracebacks are printed only on generic errors.
* Changed: temporary WARC log file is now compressed to save space.

Scripting Hook API:

* Added: Version 3 API
* Added: ``wait_time`` to version 3 which provides useful context including response or error infos.

===================

* Security: updated certificate bundle.
* Fixed: TypeError crash on bad Meta Refresh HTML element.
* Fixed: unable to fetch FTP files with spaces and other special characters.
* Fixed: AssertionError fetching URLs with trailing dot not properly removed.
* Added: ``--no-cache``.
* Added: ``--report-speed``.
* Added: ``--monitor-disk`` and ``--monitor-memory``.

===================

* Fixed malformed URLs printed to logs without sanitation.
* Fixed AttributeError crash on FTP servers that support MLSD.
* Improved link recursion heuristics when extracting from JavaScript and HTML.
* Added ``--retr-symlinks``.
* Added ``--session-timeout``.