* Convert file to str in helper `split_by_paragraph` for `partition_text`
0.5.8
Enhancements
* Update `elements_to_json` to return string when filename is not specified * `elements_from_json` may take a string instead of a filename with the `text` kwarg * `detect_filetype` now does a final fallback to file extension. * Empty tags are now skipped during the depth check for HTML processing.
Features
* Add local file system to `unstructured-ingest` * Add `--max-docs` parameter to `unstructured-ingest` * Added `partition_msg` for processing MSFT Outlook .msg files.
Fixes
* `convert_file_to_text` now passes through the `source_format` and `target_format` kwargs. Previously they were hard coded. * Partitioning functions that accept a `text` kwarg no longer raise an error if an empty string is passed (and empty list of elements is returned instead). * `partition_json` no longer fails if the input is an empty list. * Fixed bug in `chunk_by_attention_window` that caused the last word in segments to be cut-off in some cases.
BREAKING CHANGES
* `stage_for_transformers` now returns a list of elements, making it consistent with other staging bricks
0.5.7
Enhancements
* Refactored codebase using `exactly_one` * Adds ability to pass headers when passing a url in partition_html() * Added optional `content_type` and `file_filename` parameters to `partition()` to bypass file detection
Features
* Add `--flatten-metadata` parameter to `unstructured-ingest` * Add `--fields-include` parameter to `unstructured-ingest`
Fixes
0.5.6
Enhancements
* `contains_english_word()`, used heavily in text processing, is 10x faster.
Features
* Add `--metadata-include` and `--metadata-exclude` parameters to `unstructured-ingest` * Add `clean_non_ascii_chars` to remove non-ascii characters from unicode string
Fixes
* Fix problem with PDF partition (duplicated test)
0.5.4
Enhancements
* Added Biomedical literature connector for ingest cli. * Add `FsspecConnector` to easily integrate any existing `fsspec` filesystem as a connector. * Rename `s3_connector.py` to `s3.py` for readability and consistency with the rest of the connectors. * Now `S3Connector` relies on `s3fs` instead of on `boto3`, and it inherits from `FsspecConnector`. * Adds an `UNSTRUCTURED_LANGUAGE_CHECKS` environment variable to control whether or not language specific checks like vocabulary and POS tagging are applied. Set to `"true"` for higher resolution partitioning and `"false"` for faster processing. * Improves `detect_filetype` warning to include filename when provided. * Adds a "fast" strategy for partitioning PDFs with PDFMiner. Also falls back to the "fast" strategy if detectron2 is not available. * Start deprecation life cycle for `unstructured-ingest --s3-url` option, to be deprecated in favor of `--remote-url`.
Features
* Add `AzureBlobStorageConnector` based on its `fsspec` implementation inheriting from `FsspecConnector` * Add `partition_epub` for partitioning e-books in EPUB3 format.
Fixes
* Fixes processing for text files with `message/rfc822` MIME type. * Open xml files in read-only mode when reading contents to construct an XMLDocument.
0.5.3
Enhancements
* `auto.partition()` can now load Unstructured ISD json documents. * Simplify partitioning functions. * Improve logging for ingest CLI.
Features
* Add `--wikipedia-auto-suggest` argument to the ingest CLI to disable automatic redirection to pages with similar names. * Add setup script for Amazon Linux 2 * Add optional `encoding` argument to the `partition_(text/email/html)` functions. * Added Google Drive connector for ingest cli. * Added Gitlab connector for ingest cli.