Advertools

-------------------

* Added
- New function ``knowledge_graph`` for querying Google's API
- Faster ``sitemap_to_df`` with threads
- New parameter `max_workers` for ``sitemap_to_df`` to determine how fast
it could go
- New parameter `capitalize_adgroups` for ``kw_generate`` to determine
whether or not to keep ad groups as is, or set them to title case (the
default)

* Fixed
- Remove restrictions on the number of URLs provided to ``crawl``,
assuming `follow_links` is set to `False` (list mode)
- JSON-LD issue breaking crawls when it's invalid (now skipped)

* Removed
- Deprecate the ``youtube.guide_categories_list`` (no longer supported by
the API)

-------------------

* Added
- JSON-LD support in crawling. If available on a page, JSON-LD items will
have special columns, and multiple JSON-LD snippets will be numbered for
easy filtering
* Changed
- Stricter parsing for rel attributes, making sure they are in link
elements as well
- Date column names for ``robotstxt_to_df`` and ``sitemap_to_df`` unified
as "download_date"
- Numbering OG, Twitter, and JSON-LD where multiple elements are present in
the same page, follows a unified approach: no numbering for the first
element, and numbers start with "1" from the second element on. "element",
"element_1", "element_2" etc.

-------------------

* Added
- New features for the ``crawl`` function:
* Extract canonical tags if available
* Extract alternate `href` and `hreflang` tags if available
* Open Graph data "og:title", "og:type", "og:image", etc.
* Twitter cards data "twitter:site", "twitter:title", etc.

* Fixed
- Minor fixes to ``robotstxt_to_df``:
* Allow whitespace in fields
* Allow case-insensitive fields

* Changed
- ``crawl`` now only supports `output_file` with the extension ".jl"
- ``word_frequency`` drops `wtd_freq` and `rel_value` columns if `num_list`
is not provided

-------------------

* Added
- New function ``url_to_df``, splitting URLs into their components and to a
DataFrame
- Slight speed up for ``robotstxt_test``

-------------------

* Added
- New function ``robotstxt_test``, testing URLs and whether they can be
fetched by certain user-agents

* Changed
- Documentation main page relayout, grouping of topics, & sidebar captions
- Various documentation clarifications and new tests

-------------------

* Added
- User-Agent info to requests getting sitemaps and robotstxt files
- CSS/XPath selectors support for the crawl function
- Support for custom spider settings with a new parameter ``custom_settings``

* Fixed
- Update changed supported search operators and values for CSE