Dpulse

PageSearch mode

For complex OSINT, it's not enough to find only vectors for their further use. Sensitive or other specific information that interests investigator is also important for more accurate approaches. We've added PageSearch feature, which allows you to find more e-mail addresses, API keys, exposed passwords, cookies, hidden forms of data and other web page elements, documents, config files, databases files and specified words by user in PDF files. Although PageSearch will continue to develop, you can use it to get some interesting results in v1.1

XLSX and HTML report support

We decided to add a little variety when you choose the type of report which DPULSE generates. XLSX is an extremely common and no less convenient format than PDF for creating reports. In our XLSX report file you can see several sheets, each of which contains some information

Also, HTML reports support was added. We had some problems with encodings in xhtml2pdf library and PDF reports, so HTML reports are more like an alternative way to get correct output in final report file. In further releases PDF report most likely will be removed, but if solution will appear - PDF reports remain

No more wkhtmltopdf installation requirement

Installing various third-party programs while working with another software is extremely inconvenient. We decided to rework the process of PDF reports creation using a different set of libraries that do not require installation of wkhtmltopdf. This will allow you to start using DPULSE much faster right after downloading it

Scan journal

In situations when errors appear, user wants to know what caused it. With this idea, scan log was introduced. It collects all successful scanning phase and various errors in a separate journal.log file. You can find it in DPULSE root folder after conducting at least one scan

Rolling releases branch

When v1.1 will be released, we will create new branch in GitHub repository. It will contain rolling releases which are being developed in live mode.

ADDED

1) Added some checks on empty inputs in scanning section of main menu
2) Added e-mails return from PageSearch to use them in reports
3) Added new function (PageSearch Sitemap Inspection) to inspect sitemap links
4) Added .json, SQL and different config variants files extraction in PageSearch
5) Added post-scan summary in PageSearch and PageSearch Sitemap Inspection
6) Added "documents with keywords" counter in PageSearch
7) Added custom KeyboardInterrupt handler (60)
8) Added exception handler for situations when organization name cannot be extracted (61, thanks to jdenoy)
9) Added pages separators for PDF report
10) Added 3 additional web elements to search (PageSearch)
11) Added counter of inspected links (PageSearch Sitemap nspection)
12) Added txt log of gathered data (PageSearch Sitemap Inspection)
13) Added sitemap_links.txt file generation when choosing XLSX report (64), which allows you to use PageSearch Sitemap Inspection mode
14) Added checks on invalid menu items
15) Added logs_processing.py to handle scan logging (58)
16) Added logging for crawl_processor.py functions (58)
17) Added logging for networking_processor.py functions (58)
18) Added appropriate log file name field in XLSX and PDF reports
19) Added API keys and cookies gathering in PageSearch

REMOVED

1) Removed "=>" symbol when writing parsed sitemap links in file
2) Removed "Sitemap links" sheet in XLSX report (64)

CHANGED

1) Fixed bug when function waits for e-mails return but PS was not conducted
2) Fixed bug when PageSearch ends program because subdomains weren't found
3) Fixed CLI typo in DB menu
4) Edited DPULSE wiki link
5) Updated "Help" menu content
6) Renamed "Scanning process" to "Basic scanning process"
7) Minor CLI visual improvements

ADDED

1) Added PageSearch module (currently in BETA testing stage)
2) Created reporting_modules folder with all modules which handle report creation
3) Added sys link to reporting_modules folder in dpulse.py
4) Added PageSearch basic support
5) Added PageSearch basic functionality with modules in pagesearch folder
6) Added crawl_processor.py text fillers when DPULSE cant find certain domain element
7) Added networking_processor.py text fillers when DPULSE cant find element
8) Added exception handlers from issues 44, 45
9) Added text filler if domain doesn't have contact e-mails (46)
10) Added text fillers to subdomain's PDF report part (47)
11) Added PyMuPDF requirement
12) Added PDF keywords search function for PageSearch
13) Added possibility to decline keywords search but continue docs extraction (53)
14) Added "separate by comma" remark in keywords input
15) Added check on corrupted PDF files downloaded by PageSearch
16) Added exception handler to subdomains gathering module (55)
17) Added check for PageSearch if subdomains were not found
18) Added info about PageSearch in general information (part of 43)
19) Added support of transfering PageSearch status message to html template (part of 43)
20) Filled empty gaps when DPULSE can't find some elements of domain (34)
21) Added more exception handlers (44, 45)

CHANGED

1) Moved pdf_report_creation.py from datagather_modules folder
2) Moved xlsx_report_creation.py from datagather_modules folder
3) Fixed dependabot alerts in requirements.txt, pyproject.toml and poetry.lock
4) Fixed xlsx report filling issue (49)
5) Fixed contact e-mails representation in PDF report (part of 42)
6) Fixed MX records not displayed in PDF report (30)
7) Changed color style of main menu
8) Changed date and time format to international
9) Normalized subdomains emails output for PDF report (part of 42)
10) Normalized output of InternetDB results (part of 42)
11) Fixed bug when 1 letter takes 1 string in PDF report (40)
12) Code clean-ups, more cosmetical improvements (42), minor interface changes and so on

ADDED

1) Added new data_assembler.py module which calls information gathering functions and assembles their results into one list, which then is used by report creation modules
2) Added xhtml2pdf requirement
3) Created .bat installer for Windows systems
4) Created .sh installer for Linux systems
5) Created run function in dpulse.py to support Poetry entry point

REMOVED

1) Removed DPULSE dependence on wkhtmltopdf third-party program, so you don't need to install it in DPULSE root anymore
2) Removed code part which checks wkhtmltopdf.exe presence in DPULSE root
3) Removed pdfkit requirement and imports from modules

CHANGED

1) Some cosmetical changes in report_template.html
2) Changed both xlsx and pdf report creation modules to correctly interact with data_assembler.py module
3) Changed some other modules to correctly interact with data_assembler.py module

So, release is here. We've also released on PyPI (https://pypi.org/project/dpulse/), you can download it from here also.

Many bugs were fixed, some things were redacted, but there's a lot of work to do further. Hope you'll support us and DPULSE with stars and enjoy the time with our program

ADDED

1) Added programm check if there is some data in report storage database. If there's nothing in DB - program won't allow you to use report recreation function and print informational message that DB is empty
2) Added exception handler for DB report recreation in case you'll try to recreate report ID which folder already exists in DPULSE root

DELETED

1) Deleted read_config function in pdf_report_creation.py

CHANGED

1) Changed program structure by adding service folder with new db_processing.py and files_processing.py modules
2) Some CLI changes and improvements
3) Moved DB processing functions to service/db_processing.py
4) Moved some parts of code from pdf_report_creation.py to db_processing.py and files_processing.py in service folder