Credentialdigger

DB

The db now includes a new table, i.e., `embeddings`. More on this below.


Similarity model

- The similarity model is fully integrated into all the interfaces of Credential Digger (both the python library, the CLI, and the UI). In order to avoid a performance bottleneck during the manual assessment of the discoveries (present in v3.3.3), we added a table in the db such that the embeddings can be computed once, at scan time.
- The UI now shows (activated by default) a flag *"update similar discoveries"* to boost the manual review operations by calling the `update_similar_snippets function`.

Scan

- The `exclude` parameter has been deprecated to make the scan process more clear: either (all) the rules of a specific category are picked for a scan, or all the rules in the db are taken. No single rules can be excluded anymore.

UI

- The re-scan button has been moved to the homepage, where the repositories are listed (instead of keeping it in the discovery views)
- The choice of how many discoveries to list per page has been made persistent
- The button *"Show on GitHub"* has been fixed
- The count of the discoveries has been fixed


AoB

- Several updates to the documentation
- The db diagram has been updated to include the new table
- Several bugfixes

New Features

- Implement the similarity model feature in the python library
- Implement the `scan_snapshot` feature
- Integration in the python library (add a new scanner and a new method in the client)
- Integration in the CLI
- Integration in the UI
- Add integration tests for `scan_snapshot`
- Improve installation experience for MacOS (now less painful)
- Bugfixes

Add new features for the UI:
- possibility to lock the UI with a master password
- possibility to enable HTTPS

Bugfixes

New features implemented in the package

File Scanner
This new scanners offers the possibility to scan local git repositories (i.e., git repositories stored on the file system of the user) and local files/folders.

CI/CD
We implemented CI/CD that helps maintain the project. The tests are run [over an external repository](https://github.com/sap/credential-digger-tests).

AOB
- fix `force` parameter in scan: it now deletes previous scan results
- other minor fixes and improvements

Release a new major package for Credential Digger

Client Package

- Keep the line number of a discovery
- Replace the last commit scanned with the timestamp of the last scan (solves the bug of duplicates in another branch when scanning again a repo)

Command line interface

The CLI has been complemented with more features

- scan all the repositories of a user
- scan the wiki pages of a project
- fix bugs

UI

The UI has been totally re-written. Now it provides major improvements, both in performance and in functionality

- Git Token support
- File/Snippet views
- Pagination
- Scan in progress icon
- Extend client classes for adapting queries to the ui (performance)

Fix bugs when using a personal git token to scan repositories, users, and wiki pages.
Python 3.8 is now supported. Version 3.9 not yet (due to hyperscan not supported in Python3.9 yet).
Update README with new descriptions.