* Added - User-Agent info to requests getting sitemaps and robotstxt files - CSS/XPath selectors support for the crawl function - Support for custom spider settings with a new parameter ``custom_settings``
* Fixed - Update changed supported search operators and values for CSE
0.10.1
-------------------
* Changed - Links are better handled, and new output columns are available: ``links_url``, ``links_text``, ``links_fragment``, ``links_nofollow`` - ``body_text`` extraction is improved by containing <p>, <li>, and <span> elements
0.10.0
-------------------
* Added - New function ``crawl`` for crawling and parsing websites - New function ``robotstxt_to_df`` downloading robots.txt files into DataFrames
0.9.1
------------------
* Added - Ability to specify robots.txt file for ``sitemap_to_df`` - Ability to retreive any kind of sitemap (news, video, or images) - Errors column to the returnd DataFrame if any errors occur - A new ``sitemap_downloaded`` column showing datetime of getting the sitemap
* Fixed - Logging issue causing ``sitemap_to_df`` to log the same action twice - Issue preventing URLs not ending with xml or gz from being retreived - Correct sitemap URL showing in the ``sitemap`` column
0.9.0
------------------
* Added - New function ``sitemap_to_df`` imports an XML sitemap into a ``DataFrame``
0.8.1
------------------
* Changed - Column `query_time` is now named `queryTime` in the `youtube` functions - Handle json_normalize import from pandas based on pandas version