Scribe-data

<!-- - The translation process has been updated to allow for translations from non-English languages.
- Scribe-Data now outputs an SQLite table that has keys for target languages for each base language. -->
<!-- - English has been added to the data ETL process. -->

- Minor fixes to documentation index and file docstrings to fix errors.
- Revert change to package path definition to hopefully register the resources directory.

♻️ Code Refactoring

- The docs and tests were grafted into the package using `MANIFEST.in`.
- Minor fixes to file and function docstrings and documentation files.
- `include_package_data=True` is used in `setup.py` to hopefully include all files in the package distribution.

✨ Features

- The data and process needed for an English keyboard has been added ([39](https://github.com/scribe-org/Scribe-Data/issues/39)).
- The Wikidata queries for English have been updated to get all nouns and verbs.
- Formatting scripts have been written to prepare the queried data and load it into an SQLite database.
- The data update process has been cleaned up in preparation for future changes to Scribe-Data and to implement better practices.
- Language data was extracted into a JSON file for more succinct referencing ([52](https://github.com/scribe-org/Scribe-Data/issues/52)).
- Language codes are now checked with the package [langcodes](https://github.com/rspeer/langcodes) for easier expansion.
- A process has been created to check and update words that can be translated for each Scribe language ([44](https://github.com/scribe-org/Scribe-Data/issues/44)).
- The baseline data returned from Wikidata queries is now removed once a formatted data file is created.

✅ Tests

- A full testing suite has been added to run on GitHub Actions ([37](https://github.com/scribe-org/Scribe-Data/issues/37)).
- Unit tests have been added for Wikidata queries ([48](https://github.com/scribe-org/Scribe-Data/issues/48)) and utility functions ([#50](https://github.com/scribe-org/Scribe-Data/issues/50)).

🐞 Bug Fixes

- Tensorflow was removed from the download wiki process to fix build problems on Macs.

♻️ Code Refactoring

- The Anaconda based virtual environment was removed and documentation was updated to reflect this.
- Language data processes were moved into the `src/scribe_data/extract_transform/languages` directory to clean up the structure.
- Code formatting processes were defined with common structures based on language and word type variables defined at the top of files.

✨ Features

- The word "Scribe" is now added to language database nouns files if it's not already present ([35](https://github.com/scribe-org/Scribe-Data/issues/35)).
- German contracted prepositions have been added to the German prepositions formatting process ([34](https://github.com/scribe-org/Scribe-Data/issues/34)).
- Words that are upper case are now better included in the autocomplete lexicon with their lower case equivalents being removed.
- Words with apostrophes have been removed from the autocomplete lexicon.

♻️ Code Refactoring

- Database output column names are now zero indexed to better align with Python and other language standards.

✨ Features

- Scribe-Data now has the ability to generate SQLite databases from formatted language data.
- `data_to_sqlite.py` is used to read available JSON files and input their information into the databases.
- These databases are now sent to Scribe apps via defined paths.
- `send_dbs_to_scribe.py` finds all available language databases and copies them.
- Separating this step from the data update is in preparation for data import in the future where this will be an individual step.
- Scribe-Data now also creates autocomplete lexicons for each language within `data_to_sqlite.py`.
- JSON data is no longer able to be uploaded to Scribe app directories directly, with the SQLite directories now being exported instead.
- Emojis of singular nouns are now also linked to their plural counterparts if the plural isn't present in the emoji keyword outputs.
- The emoji process also now updates a column to the `data_table.txt` file for sharing on readmes with `update_data.py` maintaining it in the data update process.

♻️ Code Refactoring

- The Jupyter notebooks for autosuggestions and emojis as well as `update_data.py` were moved to the `extract_transform` directory given that they're not used to load data anymore.
- Their code was refactored to reflect their new locations.
- Massive amounts of refactoring happened to achieve the shift in the data export method:
- `format_WORD_TYPE.py` files export to a `formatted_data` directory within `extract_transform`.
- Copies of all data JSONs that were originally in Scribe apps are now in the `formatted_data` directories.
- Functions in `update_utils.py` were switched given that data is no longer uploaded into a `Data` directory within the language keyboard directories within Scribe apps.
- Lots of functions and variables were renamed to make them more understandable.
- Code to derive appropriate export locations within `format_WORD_TYPE.py` files was removed in favor of a language `formatted_data` directory.
- regex was added as a dependency.
- pylint comments were removed.
- Verb SPARQL query scripts for Spanish and Italian were simplified to remove unneeded repeat conditions ([7](https://github.com/scribe-org/Scribe-Data/issues/7)).

🐞 Bug Fixes

- The statements in translation files have been fixed as they were improperly defined after a file was moved.