Dpulse

ADDED

1) Added some checks on empty inputs in scanning section of main menu
2) Added e-mails return from PageSearch to use them in reports
3) Added new function (PageSearch Sitemap Inspection) to inspect sitemap links
4) Added .json, SQL and different config variants files extraction in PageSearch
5) Added post-scan summary in PageSearch and PageSearch Sitemap Inspection
6) Added "documents with keywords" counter in PageSearch
7) Added custom KeyboardInterrupt handler (60)
8) Added exception handler for situations when organization name cannot be extracted (61, thanks to jdenoy)
9) Added pages separators for PDF report
10) Added 3 additional web elements to search (PageSearch)
11) Added counter of inspected links (PageSearch Sitemap nspection)
12) Added txt log of gathered data (PageSearch Sitemap Inspection)
13) Added sitemap_links.txt file generation when choosing XLSX report (64), which allows you to use PageSearch Sitemap Inspection mode
14) Added checks on invalid menu items
15) Added logs_processing.py to handle scan logging (58)
16) Added logging for crawl_processor.py functions (58)
17) Added logging for networking_processor.py functions (58)
18) Added appropriate log file name field in XLSX and PDF reports
19) Added API keys and cookies gathering in PageSearch

REMOVED

1) Removed "=>" symbol when writing parsed sitemap links in file
2) Removed "Sitemap links" sheet in XLSX report (64)

CHANGED

1) Fixed bug when function waits for e-mails return but PS was not conducted
2) Fixed bug when PageSearch ends program because subdomains weren't found
3) Fixed CLI typo in DB menu
4) Edited DPULSE wiki link
5) Updated "Help" menu content
6) Renamed "Scanning process" to "Basic scanning process"
7) Minor CLI visual improvements

ADDED

1) Added PageSearch module (currently in BETA testing stage)
2) Created reporting_modules folder with all modules which handle report creation
3) Added sys link to reporting_modules folder in dpulse.py
4) Added PageSearch basic support
5) Added PageSearch basic functionality with modules in pagesearch folder
6) Added crawl_processor.py text fillers when DPULSE cant find certain domain element
7) Added networking_processor.py text fillers when DPULSE cant find element
8) Added exception handlers from issues 44, 45
9) Added text filler if domain doesn't have contact e-mails (46)
10) Added text fillers to subdomain's PDF report part (47)
11) Added PyMuPDF requirement
12) Added PDF keywords search function for PageSearch
13) Added possibility to decline keywords search but continue docs extraction (53)
14) Added "separate by comma" remark in keywords input
15) Added check on corrupted PDF files downloaded by PageSearch
16) Added exception handler to subdomains gathering module (55)
17) Added check for PageSearch if subdomains were not found
18) Added info about PageSearch in general information (part of 43)
19) Added support of transfering PageSearch status message to html template (part of 43)
20) Filled empty gaps when DPULSE can't find some elements of domain (34)
21) Added more exception handlers (44, 45)

CHANGED

1) Moved pdf_report_creation.py from datagather_modules folder
2) Moved xlsx_report_creation.py from datagather_modules folder
3) Fixed dependabot alerts in requirements.txt, pyproject.toml and poetry.lock
4) Fixed xlsx report filling issue (49)
5) Fixed contact e-mails representation in PDF report (part of 42)
6) Fixed MX records not displayed in PDF report (30)
7) Changed color style of main menu
8) Changed date and time format to international
9) Normalized subdomains emails output for PDF report (part of 42)
10) Normalized output of InternetDB results (part of 42)
11) Fixed bug when 1 letter takes 1 string in PDF report (40)
12) Code clean-ups, more cosmetical improvements (42), minor interface changes and so on

ADDED

1) Added new data_assembler.py module which calls information gathering functions and assembles their results into one list, which then is used by report creation modules
2) Added xhtml2pdf requirement
3) Created .bat installer for Windows systems
4) Created .sh installer for Linux systems
5) Created run function in dpulse.py to support Poetry entry point

REMOVED

1) Removed DPULSE dependence on wkhtmltopdf third-party program, so you don't need to install it in DPULSE root anymore
2) Removed code part which checks wkhtmltopdf.exe presence in DPULSE root
3) Removed pdfkit requirement and imports from modules

CHANGED

1) Some cosmetical changes in report_template.html
2) Changed both xlsx and pdf report creation modules to correctly interact with data_assembler.py module
3) Changed some other modules to correctly interact with data_assembler.py module

So, release is here. We've also released on PyPI (https://pypi.org/project/dpulse/), you can download it from here also.

Many bugs were fixed, some things were redacted, but there's a lot of work to do further. Hope you'll support us and DPULSE with stars and enjoy the time with our program

ADDED

1) Added programm check if there is some data in report storage database. If there's nothing in DB - program won't allow you to use report recreation function and print informational message that DB is empty
2) Added exception handler for DB report recreation in case you'll try to recreate report ID which folder already exists in DPULSE root

DELETED

1) Deleted read_config function in pdf_report_creation.py

CHANGED

1) Changed program structure by adding service folder with new db_processing.py and files_processing.py modules
2) Some CLI changes and improvements
3) Moved DB processing functions to service/db_processing.py
4) Moved some parts of code from pdf_report_creation.py to db_processing.py and files_processing.py in service folder

ADDED

1) Added colorama requirement in README
2) Added function to write additional 4 report text files into report_storage.db
3) Added some new string in DB CLI menu to work with new DB features (indicators whether the new files are in DB)

DELETED

1) Some points were deleted from Help menu
2) Deleted some points from GitHub Wiki

CHANGED

1) Changed figlet font to slant
2) whois was replaced with python-whois in requirements.txt
3) report_creation.py module was renamed in pdf_report_creation.py, all imports were also renamed and since that this module will handle only report creation with PDF file included (preparation to xlsx report processing module)
4) Database report extraction now creates folder with report_extracted_IDN name and puts PDF report and 4 text files in it