Pyfunceble

:tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2:

Hello, World!

I hope that everything is going right for you and your loved one.
This release just provides some improvements and a few new features.

There are a few important changes to notice. The first one is the switch from raw SQL to SQLAlchemy. The reason behind the switch is a change in the development process. I cannot endorse nor want to maintain raw SQL anymore.
To understand the matter, before the switch to SQLAlchemy I had code blocks that handled MySQL connections and statements and another one to handle MariaDB connections and statements. Most of the time it was 1 block for both but in other - not so peaceful - parts there were 2. Meaning that I was handling every use case by myself. With SQLAlchemy, we have 1 line/block for multiple database types. Of course we only support MariaDB and MySQL but it simplifies the integration of other database types in the future. The other reason which convinced me that it's the right way is that it forced me to review and simplify the table structure. Meaning that:

- We minimize the total size,
- We optimize the SQL flow,
- We minimize the number of SQL queries being made.

To simplify the switch and keep it as smooth as possible, a migrator is integrated into PyFunceble. That way, you don't have to take any special action, PyFunceble will handle everything.

The second important change is the sorting algorithms that were improved. Meaning that I simplified them to be efficient but before anything
correct. This change has been made because the sorting of digits was not human friendly. So I took advantage of the issue to completely improve the sorting algorithms.

The third important change is the introduction of new special rules which I let you discover through the documentation.

The fourth and last important change is the new `--store-whois` argument and logic. Before the introduction of this logic, we were storing the whois WHOIS record into the database. From now, it's only if it is needed by you. The idea behind this is to reduce the size of our database. Please note that the database migrator will follow this argument too.

There is a lot more but that's it for the most important changes!

Let me know what you think or how you use PyFunceble with [PyFunceble](https://twitter.com/hashtag/pyfunceble) or [PyFunceble](https://twitter.com/pyfunceble) on Twitter :smile:!

I wish you all the best.
Stay safe and healthy!

:tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2:

What was fixed ?

- Typos.
- Badges URL.
- An issue under macOS while using the `--multiprocess` argument.

What was removed ?

- The `db_types` directory as it is now obsolete.

What was improved ?

- Some sections of the documentation.
- The standard sorting of file content.
- The hierarchical sorting of the file content.
- The cleanup logic behind the `--clean` and `--clean-all` arguments.
- The way we process the mining of information.
- The way we open and close a database (MariaDB/MySQL) connection.

What is new ?

- The usage of SQLAlchemy instead of raw SQL.
- With an automatic migration routine if the old layout is detected.
- The `*.wix.com` special rule.
- The `*.hpg.com.br` special rule.
- The `*.000webhostapp.com` special rule.
- The `*.angelfire.com` special rule.
- Introduction of the `--store-whois` argument and logic.
- A file position tracker which improve the continuation from an interrupted test.
- The mining capabilities are now available through the `--domain` and `--url` arguments.
- The multiprocessing mode is automatically deactivated if less than 2 processes are allocated/allowed.
- The `--all` argument prints everything.
- We now retry 3 times with a sleep of 3 seconds between each while downloading vital files. If the URL is still not reachable after 3 retries, the system will stop.
- CI tests with Python 3.6.11, 3.7.8, and 3.8.5.
- The `PYFUNCEBLE_DEBUG` and `PYFUNCEBLE_DEBUG_ONSCREEN` environment variables.

What is coming ?

- Fix of reported issues.
- Improvement of the documentation (help welcome!).
- Introduction of possible and interesting features.

---

Contributors

Thanks to those awesome people who contributed to this release!

- spirillen - [spirillen](https://github.com/spirillen)

:tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2:

Hello, World!

I hope that everything is going right for you and your loved one.
This release just provides a :fire: issue which was provided by my last release.

Indeed, as I automate everything, I did not notice that the input file was being deleted.

The reason for the `3.2.2` instead of the `3.2.1` is because I did a mistake in the versioning a few days ago which resulted in me being forced to release `3.2.1` to PyPi and all other platforms.

Let me know what you think or how you use PyFunceble with [PyFunceble](https://twitter.com/hashtag/pyfunceble) or [PyFunceble](https://twitter.com/pyfunceble) on Twitter :smile:!

I wish you all the best.
Stay safe and healthy!

:tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2:

What was fixed ?

- An issue with the deletion of the input source at the end of a test.
What is coming ?

- Fix of reported issues.
- Improvement of the documentation (help welcome!).
- Introduction of possible and interesting features.

---

Contributors

Thanks to those awesome people who contributed to this release!

- spirillen - [spirillen](https://github.com/spirillen)

:tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2:

Hello, World!

I hope that everything is going right for you and your loved one. This release just provides some fixes and very few new features.

This release includes one important change. Indeed, in the previous version, I included the notion of "shadow file" and made it default.
I now consider that decision as a bad one because it may have created unexpected behaviors especially while talking about execution time for some huge lists.
Therefore, from now, if you want to use that feature, you should use or append the `--shadow` argument or its configuration equivalent to access it.

The other important thing which is not a change is the deployment and provision of docker images and conda packages to the grand-public.
That means that from now, you can install PyFunceble (or its `-dev` counterpart) from PyPi, AUR, Docker Hub, and Anaconda Cloud. I spend a huge amount of time to automate the process. So, let me know if something is wrong with the two new repositories.

Let me know what you think or how you use PyFunceble with [PyFunceble](https://twitter.com/hashtag/pyfunceble) or [PyFunceble](https://twitter.com/pyfunceble) on Twitter :smile:!

I wish you all the best.
Stay safe and healthy!

:tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2:

What was fixed ?

- An issue with the creation of files when the `output/` directory is missing.
- An issue with some analytic files which were named `NoneNone`.
- An issue with the quiet mode which was not completely quiet while using the `-m` argument.

What was removed ?

- The direct dependency on `urllib3`.

What was improved ?

- The documentation.
- The tests of the AdBlock decoder.
- The way we handle subdomains which are behind a public suffix.
- The way we generate the "shadow file". It can now take advantage of multiprocessing.

What is new ?

- Deployment to Docker Hub.
- Deployment to Anaconda Cloud.
- While testing domains, we convert them to lowercase.
- CI tests with Python 3.6.10, 3.7.7, and 3.8.3.
- A way to define the location of the `output/` directory. It can be done through the `PYFUNCEBLE_OUTPUT_LOCATION` environment variable.
- The "shadow file" is not generated, nor used by default. You should explicitly give us permission to generate and use it through the `--shadow` argument or its configuration counterpart.

What is coming ?

- Fix of reported issues.
- Improvement of the documentation (help welcome!).
- Introduction of possible and interesting features.

---

Contributors

Thanks to those awesome people who contributed to this release!

- NeolithEra - [NeolithEra](https://github.com/NeolithEra)
- spirillen - [spirillen](https://github.com/spirillen)
- ZeroDot1 - [jawz101](https://github.com/ZeroDot1)

:tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2:

Hello, World!

I hope that everything is going right for you and your loved one. This release took a bit of time because it was hard to find some time to give it some love. Anyway, here we are!

This release includes 2 important changes. The first one, is the fix of the way we connect to your MariaDB or MySQL database. There was a bug because I assumed that PyMySQL will close the connection by itself while using the with-statement. It turns out that it was not the case.

The second one and probably the most important one, is the inclusion of the self-cleaning of the inactive database. After collecting data in your local storage, this feature was highly wanted and here it is. Indeed, before today, the system was collecting and saving information into the inactive database but it never cleaned the unneeded one. From now, and after 28 days sitting in there, a subject will be automatically removed.

I spend most of my time restructuring the documentation so that we could export it into PDF without having to lose the structure. I was a bit shocked by the generated PDF, that I restructured the whole documentation for it to be rendered correctly. As you know, using PyFunceble requires an internet connection so PDF won't be a big deal. But on my side, I would like to be able (and to start) to download the PDF, print it, and correct what I wrote. That's how I will try to work in the coming month in order to complete the documentation.

By the way, I would like to thank all of those who contributed to the documentation! A lot of work is still waiting but we will come that far one day.

Let me know what you think or how you use PyFunceble with [PyFunceble](https://twitter.com/hashtag/pyfunceble) on Twitter :smile:!

I wish you all the best.
Stay safe and healthy!

:tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2:

What was fixed ?

- A MySQL/MariaDB connection and possibly communication issue.
- The multiprocessing logic.
- The length of the first line of all output files.
- The coloration of the PyFunceble ASCII-Art at the end which may not have been correct while using the MySQL or MariaDB database.

What was improved?

- The configuration loader (completely rewritten).
- The behavior behind the generation of the `.pyfunceble-env` file for those who use the MySQL or MariaDB database.
- The help messages.
- The documentation's grammar, spelling, and structure.
- README's grammar and spelling.
- The structure of the inactive database (JSON).

What is new?

- A section in the documentation about some limitation which we may meet while using the MySQL or MariaDB database.
- The `--dots` argument and logic.
- The self-cleaning of the inactive database after 28 days (configurable).

What is coming?

- Fix of reported issues.
- Improvement of the documentation (help welcome!).

---

Contributors

Thanks to those awesome people who contributed to this release!

- AnonymousPoster - [AnonymousPoster](https://github.com/AnonymousPoster)
- jawz101 - [jawz101](https://github.com/jawz101)
- Mitchell Krog - [mitchellkrogza](https://github.com/mitchellkrogza)
- spirillen - [spirillen](https://github.com/spirillen)

:tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2:

Hello, World!

I'm glad to announce this new major version of PyFunceble!

It took me some time, but it's time to release this new version. This version provides a complete different tool. Indeed, the structure and the components of the project are totally different.
The idea of this major version was mainly to review the structure, but it went beyond my expectation thanks to the awesome users and contributors who gave some amazing feedback about PyFunceble(-dev).

One of the most fun I had, among the fix and development of new features, is the implementation of a total automation behind the generation and distribution of the IANA and Public Suffix databases file and the User-Agents. Indeed, before this version, I had to generate them and your own instance of PyFunceble had to fetch it after each release. From now, they are totally automated, and your own instance of PyFunceble will fetch and get the latest one when it judge that it's necessary to update your local version (generally after 24 hours).

This version is also the beginning of a new adventure, some new features and ideas.

Between what you may discover by yourself in this version, there is the support of GitLab CI/CD, IPv6 and even DNS over TCP to just name them.
As a list maintainer, this version provides you a switch which will checks the reputation of the subjects in your list with the help of the AlienVault's public reputation data. I highly invite you to check that as I'm thinking about the inclusion or improvement of reputation data in the future.


As a lot has been fixed, therefore, I'm not going to list them here, but if there are questions about the changes in details or a behavior which changed, please let me know on [Keybase](https://keybase.io/team/pyfunceble)!

Let me know what you think or how you use PyFunceble with [PyFunceble](https://twitter.com/hashtag/pyfunceble) on Twitter :smile:!

I wish you all the best!

:tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2:

What was improved?

- The project structure and source code.
- The documentation.
- The help menu/message.

What is new?

- The support of GitLab CI/CD.
- The support of Python 3.8.
- A way to check the reputation of a given subject (domain and IP).
- Then `.onion` TDL is not `INVALID` anymore.
- The support of the port when you give a custom DNS server.
- The possibility to give a MySQL/MariaDB Unix Socket file as hosts.
- The possibility to work from 2 branches (distribution and test branch) with the CI mode.
- `*.github.io` which return `404` are returned as `INACTIVE`.
- The possibility to fork the CLI behavior (file generation) from the API.
- A proper logging mechanism for better debugging.
- PyFunceble now initiate and configure the GIT repository for you under the CI mode.
- The tests where completely rewritten.
- You will get some warning and message from the CLI if your configuration are a bit extreme.
- The possibility to set a cooldown time between each test in order to reduce stress and pressure to DNS, WHOIS and target servers.
- We now look for CNAME, DNAME, A, and AAAA when the NS record is not given.
- Automation of a lot of administrative tasks that I had to do manually. Indeed, from now, the IANA, Public Suffix and even the User-Agent are automated in order for you to always get the latest one.
- The doubleclick SPECIAL rule is removed.
- The AdBlock decoder now decodes everything possible.
- ... and a lot more to discover!


What is coming?

- Fix of reported issues.
- Improvement of the behavior of the Inactive DB.
- Improvement of the documentation.

Migration from 2.x

The migration to 3.x is normally done automatically through the configuration checker.

In some rare case (and before submitting a new issue), you may delete your configuration files (`.PyFunceble_production.yaml` and `.PyFunceble.yaml`) first.

---

Contributors

Thanks to those awesome people who contributed to this release!

- Daniel - [dnmTX](https://github.com/dnmTX)
- Imre Kristoffer Eilertsen - [DandelionSprout](https://github.com/DandelionSprout)
- jawz101 - [jawz101](https://github.com/jawz101)
- Kowith Singkornkeeree - [kowith337](https://github.com/kowith337)
- Mitchell Krog - [mitchellkrogza](https://github.com/mitchellkrogza)
- opav - [opav](https://github.com/opav)
- ScriptTiger - [ScriptTiger](https://github.com/ScriptTiger)
- sjhgvr - [sjhgvr](https://github.com/sjhgvr)
- spirillen - [spirillen](https://github.com/spirillen)
- ZeroDot1 - [ZeroDot1](https://github.com/ZeroDot1)

:tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2:

Hello, World!

Here is a quick fix of the `--clean-all` argument.
Indeed before this release when `--clean-all` was used along with the MySQL, SQLite or MariaDB database type, everything was literally cleaned.

From now, everything except the WHOIS database table is deleted. Why not the WHOIS database?
It simple, it's actually a static file which saves information about domains and is used every time you retest something with PyFunceble to
reduce test time.

Let me know what you think or how you use PyFunceble with [PyFunceble](https://twitter.com/hashtag/pyfunceble) on Twitter :smile:!

I wish you all the best!

:tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2::tada: :star2:

What was improved?

- The `--clean-all` logic while using a database type different from JSON.

What is coming?

- Fix of reported issues.
- Review of the tests.

---

Contributors

Thanks to those awesome people who contributed to this release!

- Mitchell Krog - [mitchellkrogza](https://github.com/mitchellkrogza)