Perceval

** New features and improvements: **

* DockerHub added as new backend.
* Fetch the latest commits added in a Git repository using
the argument `latest-items`.

** Bugs fixed: **

* In Slack, comment messages were not processed raising an error
when their UUIDs were computed. These messages do not include a
'user' field on the top layer, which made the backed to fail.
This field can be found inside 'comment' key.
* Some versions of gerrit return number review as an integer.
This value must be converted to string because UUIDs can only be
generated using string values. (144)

** New features and improvements: **

* Common functions used across GrimoireLab projects have been moved
to their own package. This package was named `grimoirelab-toolkit`.
From this version, Perceval depends on this package.

** Backend improvements: **

* **askbot**
- support new URLs schema for comment queries
* **bugzilla**
- set `User-Agent` header in HTTP clients
* **confluence**
- add content URL to each item
* **gerrit**
- add option to disable SSH host keys checks
* **nntp**
- raise `ParseError` exceptions when an encoding error is found
* **rss**
- set `User-Agent` header in HTTP clients

** New features and improvements: **

* New set of backends added:
- Hyperkitty
- NNTP
- Slack
* `RateLimitError` exception added for handling rate limit errors.
* Code was cleaned to follow most of the PEP8 style guidelines.

** Backend improvements: **

* **git**
- retry calls on SSH commands were added to avoid temporal server errors
* **github**
- HHTP 404 errors are managed when user's organizations are fetched
- generic `RateLimitError` exception is used

** Bugs fixed: **

* In Mediawiki backend, the log messages written when a revision is not
found were set to ERROR when the real level should have been WARNING.
* The URL used to fetch jobs in Jenkins was not common to all servers.
* When UUIDs are generated with some input data, some errors may be raised
due to problems encoding invalid characters on the input. To avoid these
problems, a surrogate escape control error has been set when data is
encoded to UTF8. (123)
* Handle Meetup requests rate limit. (126)

** Backend improvements: **

* **bugzillarest**
- messages in client errors were improved
* **git**
- new method `is_detached()`
* **mbox**
- ignore messages with invalid dates on `Date` header
* **phabricator**
- retry requests on HTTP 502 and 503 errors

** Bugs fixed: **

* The `mbox` class from Python's `mailbox` module fails when it tries to
decode non-ascii unix-from headers. This header is used as a separator
between messages. When this error is found, the class stops reading messages
from the mbox. Wrapping `mbox` class to override the way messages are read
was needed to catch the exception and decode the header using UTF-8.
* When a user does not exists on Phabricator, the API does not return an
error. It returns an empty list. The case where an empty list is returned
was not managed by the parser, which raised exceptions.
* In gerrit, the identifier of the change, `Change-Id` (or `id`), is not unique.
What it is unique in a gerrit sever is the number of each change and review.
This `number` is used now instead of `id` as the identifier of a review
* When Git repositories are reset to the current status on upstream, some of
them cannot deal with `origin` reference because it is ambiguous. Replacing
it by `FETCH_HEAD` works on those repositories with defined branches on
the origin.
* Git repositories in detached state do not need to be reset after `git fetch`
is called. This call is now skipped when a repository in this state
is in use. (105)

** New features and improvements: **

* New set of backends added:
- Askbot
- Meetup
- RSS
* Definition of `perceval.backends` namespace and dynamic loading of backends.
These two features allow to have third party backends or packages of
backends that can be imported and used at runtime.
* Mozilla's backends were moved to their own package: `perceval-mozilla`.
* Commands were refactorized generalizing their usage into `BackendCommand`
class which can run any type of backend. This was possible thanks to the
creation of `BackendCommandArgumentParser` class, that defines, manages and
parses those arguments needed to run a command; the definition of `pre_init()`
and `post_init()` methods during the initialization of the instance; and to
the implementation of `setup_cache()` as a public function of the `cache`
module.

** Backend improvements: **

* **bugzilla**
- set maximum number of bugs requested on CSV queries
* **git**
- parse commit trailers
- new methods `is_empty()` and `count_objects()`
- set missing encodings for the command output
- cleaning up of the module
* **jenkins**
- ignore invalid job builds
* **supybot**
- parse action and bot empty lines
- parse user actions with the format `*nick msg`
- generate item ids using the body of the message

** Bugs fixed: **

* The field 'timestamp' on metadata was not generated in UTC. The call
to `datetime.now()` does not generate a timestamp in UTC. It does using
the timezone of the system. The right way is to call to `datetime.utcnow()`
method. (92)
* The docker image for Perceval purged the git package after installing
`perceval`. This made impossible to run the backend for Git because
Perceval needs of `git` command under the hood. (95)
* Git empty repositories threw errors while fetching commits. Those were
raised because on empty repositories, those which do not have any history
or are only initialized, there are some commands that cannot be run, like
a pull or log. If any of this commands is called an error is be returned.
It was fixed checking whether the repository is empty and returning
an empty list of commits for those cases. (102 and 107)

** New features and improvements: **

* `category` field was added to items metadata to classify the type of
the item generated with each backend.
* The `tag` attribute added to the backends allows to mark the items
with a custom label.
* Two class methods, `has_caching` and `has_resuming`, are part now
of `Backend` class interface to notify whether a backend supports
caching and/or resuming of items.

** Backend improvements: **

* **jenkins**
- support blacklist of jobs
* **mediawiki**
- use API pages methods by default
* **phabricator**
- fetch and include projects data assigned to each task
* **redmine**
- fetch and include users data
* **remo**
- support new version of the API
* **supybot**
- parse messages written by special bots

** Bugs fixed: **

* Filepaths on merge commits were not captured on Git backend. This was
neccesary in those cases where merge commits only include data about
lines added and removed because the filepaths were not parsed and
included on the item data. (63)
* The `url` argument on the Gerrit backend was set to optional. It is
mandatory. Thus, it was set to positional on the argument parser. (60)
* Newer versions of Phabricator fixed a bug on API Conduit regarding
'constraints' parameter. The Phabricator client was modified to fix
this bug, too. (80)
* Python's `requests` library decompresses gzip-encoded responses, but
in some cases is only able to decompress some parts due to encoding
issues or to mixed contents. This problem was fixed downloading and
storing the orinal/raw data (compressed or decompressed) for furthed
processing.
* Jira backend did not return items in order, from oldest to newest. (89)
* Dates with invalid timezones were not parsed. In those cases, the
the dates will be converted usin UTC by default. (73)