Codechecker

New features
Git blame integration (3398, 3423, 3425, 3430)
With this feature it will be possible for a developer to check who modified the source line last where a CodeChecker error appears.

- If the project which was analyzed is a git repository `CodeChecker store` command will store blame information for every source files which are not stored yet.
- The GUI will have a button on the report detail view to show blame information alongside the source file.
- Hovering the mouse over a blame line, commit details will be shown in a pop-up window. Clicking on the hash will jump to the remote url of the repository and shows the commit which related to a blame line.


Cleanup plans (3419)
Cleanup plans can be used to track progress of reports in your product. The conception is similar to the github Milestones.

You can do the following:
- **Managing cleanup plans**: you can create cleanup plans by clicking on the *pencil icon* at the *Cleanup plan* filter on the *Reports* page. A pop-up window will be opened where you can *add*, *edit*, *close* or *remove* existing cleanup plans.
- **Add reports to a cleanup plan**: you can add multiple reports to a cleanup plan on the *Reports* page or on the *Report detail* page by clicking to the *Set cleanup plan* button and selecting a cleanup plan.
*Note*: you can remove reports from a cleanup plan the same way by clicking on the cleanup plan name.
- **Filter reports by cleanup plans**: you can filter reports by a cleanup plan by using the *Cleanup plan* filter on the *Reports* page. Using this filter with other filters (*Detection status*, *Review status* etc.) you will be able to filter active / resolved reports in you cleanup plan.


Local diff workflow support (3388)
If you want to use CodeChecker in your project but you don't want to run a CodeChecker server and to fix every reports found by CodeChecker for the first time (legacy findings) with this feature you can do the following:
1. Analyze your project to a report directory as usual (e.g.: `./reports`).
2. Create a baseline file from the reports which contains the legacy findings: `CodeChecker parse ./reports -e baseline -o reports.baseline`. Note: it is recommended to store this baseline file (`reports.baseline`) in your repository.
3. On source code changes after your project is re-analyzed use the `CodeChecker diff` command to get the new reports:
`CodeChecker cmd diff -b ./reports.baseline -n ./reports --new`
4. On configuration changes (new checkers / options are enabled / disabled, new CodeChecker / clang version is used, etc.) re-generate the baseline file (step 1-2).

LeakSanitizer Parser (3368, 3375)
The `report-converter` tool is extended with [LeakSanitizer](https://clang.llvm.org/docs/LeakSanitizer.html) which is a run-time memory leak detector for C programs.

sh
Compile your program.
clang -fsanitize=address -g lsan.c

Run your program and redirect the output to a file.
ASAN_OPTIONS=detect_leaks=1 ./a.out > lsan.output 2>&1

Generate plist files from the output.
report-converter -t lsan -o ./lsan_results lsan.output

Store reports.
CodeChecker store ./lsan_results -n lsan


For more information [see](https://github.com/Ericsson/codechecker/blob/master/docs/tools/report-converter.md).

Checker label (3233, 3413, 3414, 3415, 3432)
Previously the properties of checkers (*severity*, *profile*, *guideline*) are read from several JSON files. The goal was to handle all these and future properties of checkers in a common manner. This new solution uses labels which can be added to checkers.

The collection of labels is found in [config/labels](https://github.com/Ericsson/codechecker/tree/master/config/labels) directory. The goal of these labels is that you can enable or disable checkers by these labels.

sh
List checkers in "sensitive" profile.
CodeChecker checkers --label profile:sensitive

List checkers in "HIGH" severity.
CodeChecker checkers --label severity:HIGH

List checkers covering str34-c SEI-CERT rule.
CodeChecker checkers --label sei-cert:str-34-c

List checkers covering all SEI-CERT rules.
CodeChecker checkers --label guideline:sei-cert

List available profiles, guidelines and severities.
CodeChecker checkers --profile
CodeChecker checkers --guideline
CodeChecker checkers --severity

List labels and their available values.
CodeChecker checkers --label
CodeChecker checkers --label severity

Enable HIGH checkers during analysis.
CodeChecker analyze \
./compile_commands.json \
-o ./reports
-e severity:HIGH


Note: with this new feature we also added severity levels for pylint (3414) and cppcheck (3415) analyzers.

Analyze improvements / fixes
- Allow to override checker list (3203).
- Handle clang binary without installed dir (3186).
- Don't hardcode GCC in build-logger Makefile (3352).
- Improve debug log messages (3361).
- Remove the MallocOverflow checker from the sensitive profile (3392).
- Add the MallocOverflow checker to the extreme profile (3400).
- Create new diagnostic message hash (3402).
- Build log transformer: also ignore -fno-reorder-functions (3411).
- Don't run ClangSA checkers from clang-tidy (3417).

CLI (parse, diff, etc.) improvements / fixes
- Parse command exits with error in case of duplicated suppress comment (3253).
- Make parse subcommand to work with --skip option correctly (3328).
- Log options from the configuration file (3341).
- Do not print sensitive information when exception happens (3355).
- Add severity to CodeClimate export (3356).
- Improve log messages for gerrit output (3374).
- Fix gerrit output (3378).
- Fix check command config file support (3385).

Server improvements / fixes
- Use processes instead of threads (3349).
- Product View Permission (3332).
- Add index for report and run history id columns (3351).
- Unzip storage zip file to workspace directory (3347).
- Log run id when storing a run (3358).
- Comment date collision (3360).
- Fix exporting checker statistics to CSV (3362).
- Rephrase "report not found" error message (3376)
- Create columns for product details (3382).
- Fix setting analysis_info_id_seq (3383).
- Add 'thrift==0.13.0' dependency explicitly (3389, 3394).
- Show edit option only for admins (3426).

Other improvements fixes.
- Add local package to git automatically and refactore the doc (3319).
- Fix pypi package github action (3344).
- include package data files in python package (3357).
- Remove doxygen requirement (3346).
- Update checker_and_analyzer_configuration.md (3350).
- Web docker image hooks (3359).
- Add wait-for script to the docker image (3364).
- Change permission of helper script in docker image (3365).
- Usage of skip list handler is not optional anymore (3366).
- Fix broken alembic urls (3390).
- Documentation for Pypi package (3391).
- Add the severity for "readability-identifier-length" (3403).
- Override argparse error code (3408).
- Extend documentation with multi storage feature (3420).
- Test workspace is not necessarily under HOME (3421).
- Add the license file to the pypi package (3422).
- Add new features for 6.16.0 and 6.17.0 releases (3427).

New features
`PyPI` package support (3251, 3301).
PyPI is the most commonly used central repository for Python packages. For this reason from this release we will provide an official [PyPI package](https://pypi.org/project/codechecker/) for CodeChecker. This PyPi package can be easily installed on both Unix and Windows based systems easily by using the `pip` command: `pip install codechecker`.
Add compilation database generator for Bazel (3226, 3284).
CodeChecker was extended with a tool that can capture compilation database of a `Bazel` built product without actually performing compilation. For more information [see](https://github.com/Ericsson/codechecker/blob/master/docs/tools/bazel.md).
Exporter/importer command for `CodeChecker cmd` (3116)
New command line options are introduced (`CodeChecker cmd export` and `CodeChecker cmd import`) which can be used to export comments and review status for a particular run in a JSON based format from a running CodeChecker server and import it to another server.
sh
Export data from one server.
CodeChecker cmd export -n myrun \
--url https://first-server.codechecker.com:443 2>/dev/null | python -m json.tool > myrun_export.json

Import data to another server.
CodeChecker cmd import -i myrun_export.json --url https://second-server.codechecker.com:443


*Sparse* and *Cpplint* analyzers support (3160, 3248).
The `report-converter` tool was extend with two more analyzers:
- [`Sparse`](https://git.kernel.org/pub/scm/devel/sparse/sparse.git) which is a semantic checker for C programs; it can be used to find a number of potential problems with kernel code.
- [`CppLint`](https://github.com/cpplint/cpplint) which is a lint-like tool which checks C++ code against Google C++ Style Guide.

For more information [see](https://github.com/Ericsson/codechecker/blob/master/docs/tools/report-converter.md).

Analyze improvements / fixes
- Set parse subcommand exit code to 2 when any report exist (3313).
- Use `maximum CPU` resources by default during analysis (3249).
- Generate reproducer (3324).
- Enable the build logger fix for CR and LF by default (3310).
- Fix `ccache` compiler detection (3204).
- Adding severities for checkers (3218, 3337).
- Remove some code duplication from CodeChecker check command (3217).
- Add `altera-unroll-loops` to the list of checkers (3266).
- Adding cert checkers to sensitive profile (3338).
- Relative include paths to --sysroot (3259).
- Handle getting options for old analyzer version (3297).
- Fix logger compilation warnings (3305).
- Fix yaml dumper (3331).

CLI (parse, diff, etc.) improvements / fixes
- Fix storage of multiple report directory (3263, 3281, 3339).
- Fix creating session file (3212).
- Handle no mandatory env var when using gerrit output (3196).
- Handle invalid proxy settings (3198).
- Fix for SpotBugs Report Conveter with Plugins (3262).
- Use codechecker_report_hash module (3270, 3317).

Server improvements / fixes
- Cleanup unused data (comments, review statuses) (3243).
- Add analyzer commands for reports (3320, 3336).
- Add documentation link to the bug report (3330).
- Fix failed files uniqueing on the statistics page (3285).
- Allow to change the outstanding reports chart resolution (3179).
- Change granularity for Number of outstanding reports chart (3036).
- Faster query for reports (3316).
- Fix quotes in system comments (3094).
- Add button to copy file path (3176).
- Close filter settings on apply (3178).
- Run filter is not working for the new reports at the Product overview statistics (3035).
- Use textarea at source component description (3190).
- Show review status selector even if status change is disabled (3195).
- Highlight row in code editor on hover event (3224).
- Highlight report on the scrollbar (3225).
- Get CodeChecker API version automatically in webpack (3265).
- Move generated API stubs to the repo (3268, 3288).
- Workaround for SQLite limitation in severity change (3282).
- Permission checking compares auth. names in case insensitive… (3279)

Other improvements fixes.
- Scrollable sidebar at plist2html (3327).
- Version upgrades (3211, 3034, 3252, 3333).
- Add coverage for unit tests (3315).
- Add type hints (3215, 3216, 3214, 3280).
- Use Python3 enums (3291).
- Use python3 new style classes (3290).
- Documentation updates (3222, 3246, 3261, 3292, 3295, 3302).
- Performance test improvement (3278, 3287, 3289, 3325).
- Use singleton when creating context objects (3193).
- Fix non existen report directory test (3250).
- Fix unused import (3264).
- Compile test project with c++11 explicitly (3283).
- Add semicolon to web Makefile (3298).
- Ignore errors when removing workspace directories (3300, 3329).
- Refactor process runner function (3307).
- Enable `cyclic-import` and `consider-iterating-dictionary` checks (3314).
- Fix running tu_collector test target (3334).

:exclamation: :exclamation: :exclamation: Non-backward compatible changes :exclamation: :exclamation: :exclamation:
- When a checker name and the alias of this checker is turned on, `Clang Tidy (>=v11)` will generate only one report where the checker names are concatenated with `,` mark (e.g.: *cppcoreguidelines-avoid-magic-numbers,readability-magic-numbers*). Unfortunately in previous CodeChecker releases we didn't handle this use case properly and we generated only one report from it. We changed this behaviour in 3238 so multiple reports will be generated for each checker name / alias if both are enabled.
- From this release, the `CodeChecker analyze` command will indicate only the success and failure of analysis by zero and non-zero exit codes respectively. Before, the analysis subcommand returned with `2`, if there was any report in the analysis. Form this release, it will return with `0`, if the analysis was successful irrespectively of the number of reports.
The `CodeChecker parse` and `CodeChecker cmd diff` subcommand will return with value `2` if there is at least one (not suppressed) report in the result set (3232, 3255).

The return values of the subcommands is as follows:
- **CodeChecker analyze**:
0 - Successful analysis
1 - CodeChecker error
3 - Analysis of at least one translation unit failed
128+signum - Terminating on a fatal signal whose number is signum

- **CodeChecker parse**
0 - No report
1 - CodeChecker error
2 - At least one report emitted by an analyzer

- **CodeChecker check**
0 - No report
1 - CodeChecker error
2 - At least one report emitted by an analyzer
3 - Analysis of at least one translation unit failed
128+signum - Terminating on a fatal signal whose number is signum

- **CodeChecker cmd diff**
0 - No difference between baseline and newrun
1 - CodeChecker error
2 - There is at least one report difference between baseline and newrun

Analyze improvements / fixes
- Fix target attribute of the log parser (3184).
- Fix parsing clangsa analyze help (3206).
- Fix ccache compiler detection (3204).
- Handle no analyzer use cases (3194).
- Fix cleanup metadata (3192).
- Expose --supress option to the check sub-command (3231).
- Fix duplication warning when creating failed zip (3213).
- Handle Clang Tidy aliases in plist files (3238).
- Removing noisy checkers from the sei-cert guideline (3256).

CLI (parse, diff, etc.) improvements / fixes
- Change exit codes (3232, 3255).
- Fix file path in codeclimate output (3202).
- Fix source content change error when diffing remote runs (3191).
- Handle suppression properly in diff command (3189).
- Fix for the SpotBugs report converter (3237, 3247).
- Improve error message when cmd diff fails on user input (3240).

Server improvements / fixes
- Configure keepalive (3167).
- Wrap bugstep messages (3177).
- Fix database status in product name column (3185).
- Set filter properly when clicking on the diff count in the statistics page (3230).
- Fix getting analysis statistics (3229).

Other fixes
- Fix context of the docker github action (3181).
- fix run_codechecker.sh (3234).

News
- CodeChecker is now available in the [Snap Store](https://snapcraft.io/codechecker) and can be installed easily with the following command: `sudo snap install codechecker --classic`.
- We have enabled `Github Discussions` in our repository. Now if you have any `question` or an `idea` you have to create a new discussion instead of an issue. Bug reports still have to be created as an `Issue`.
- Our `Roadmap for 2021` is available here: https://github.com/Ericsson/codechecker/projects/15
- We moved from `Travis CI` to `Github Actions` (3066, 3086, 3131).
- Unfortunately one of our core team member gyorb left the project due to getting busy with other tasks. We wish you all the best and thank you very much for your hard work in the CodeChecker project 😊. Nevertheless, we hope that one day you return contributing! :smirk:

CLI related improvements/fixes
- `Sphinx` documentation generator tool parser (3017).
- Show comments when using `CodeChecker cmd results --details` command (3005).
- Using tags names in diff commands (3144).
- Fix `json` and `html` output when both formats are selected (3059).
- Unique lines when collecting statistics (3028).
- Diff `resolved` reports of remote to local (3129).
- Collect CTU-involved files in the report directory (3029).
- Fix double clang-tidy config flags (3157).
- Do not allow ctu-ast-mode in non-CTU mode (3146).
- Handle duplication warning at store (3159).
- `CodeChecker log` debug logs go to report dir (3166).
- Opt-in fix escape in logger in case of backslash, CR, LF (3169).

Server improvements/fixes
- Configure keepalive (3167).
- LDAP authenticated users default permissions (3072).
- Source code comment parsing with trim path (3078).
- Change review status date only if necessary (3123).
- Print API function name in server logs (3105).
- Fix multiple negative file path in source component filter (3051).
- Fix LDAP authentication exception (3073).
- Verify TLS certificate in LDAPS connection (3083).
- Log unsuccessful authentication requests (3148).
- Add more info log to the store API function (3165).
Fix quotes in system comments (3094).

GUI improvements/fixes
- Add shortcut links to the product page (3100).
- Show a progress bar while removing a run is in progress (3046).
- Sort runs in ascending order by the latest storage date by default (3071).
- Show both *review* and *detection status* icons in the `Report Tree` view (3037).
- Outstanding reports char day view (3054).
- Set required field for product config form (3056).
- Handle missing report (3102).
- Fix *undefined* `getRunIds` API function in Baseline Run filter (3043).
- Fix checker statistics difference (3130).
- Fix links in statistics tables (3067).
- Fix getting analyzer statistics for runs (3152).
- Refactoring product overview page (3147).
- Usability improvements (1522, 3041, 3042).

Documentation updates
- Documentation for report identification (3070).
- Extend documentation for detection status (3038).
- Describe the usage flow in the main readme (3069).
- Show command line features in the main readme (3068).
- Full GUI userguide rewrite (3080).
- Other fixes (3040, 3101, 3154).

Package updates
- We changed our `thrift` requirements from `0.11.0` to `0.13.0` (3032).
- We upgraded `lxml` requirements from `4.5.0` to `4.6.2` (3127).

Milestone
For more detailed information check the [milestone](https://github.com/Ericsson/codechecker/milestone/71) of this release.

Contributors
Big thanks to everyone who helped us creating this release: jay24rajput, rasjani, jimis, engr-basit, startergo.

New features
Web UI
* There is a brand new product statistics overview page with the information about the recently introduced or resolved reports or about the distribution of the reports in the product. 2986
* The run history list was moved from a separate tab to an expandable list under each run at the run list. This makes easier to find the relevant
run history entries for each run. 2953
* New report info button to show more information about a report at the report details page (run name, detection/fix date ...) 2961
* Source components can be used to create and save file path filters with a name to show results only from those parts of the analyzed project. With the newly introduced other component every report which does not belong to any other component can be filtered. 2989

Command line interface (CLI)
* New exit status numbers for the CodeChecker `analyze` and `check` commands for better CI integration 2943:
* 0 - Successful analysis and no new reports
* 1 - CodeChecker error
* 2 - At least one report emitted by an analyzer and there is no analyzer failure
* 3 - Analysis of at least one translation unit failed
* Gerrit output format is available for the parse subcommand. This output format was only available for the `CodeChecker cmd diff` command in the previous releases. With this change the parse command can be used for the gerrit integration too 2745
`CodeChecker parse analyzer_reports -e gerrit`

Report storage support for new source code analyzers
* Report conversion and storage support is available for multiple new source code analyzer tools ([Coccinelle](https://github.com/coccinelle/coccinelle) #2949, [Smatch](https://repo.or.cz/w/smatch.git) #2968, [Kernel-Doc](https://github.com/torvalds/linux/blob/master/scripts/kernel-doc) #2981). The [report-converter](https://github.com/Ericsson/codechecker/blob/master/tools/report-converter/README.md) tool can be used to convert the output of these analyzers to a format which can be stored to the web server or processed by other CodeChecker commands (`parse`, `cmd diff` ...).
For more information about the tool configuration and usage check out the user documentation:
* Coccinelle [user documentation](https://github.com/Ericsson/codechecker/tree/master/tools/report-converter#coccinelle)
* Smatch [user documentation](https://github.com/Ericsson/codechecker/tree/master/tools/report-converter#smatch)
* Kernel-Doc [user documentation](https://github.com/Ericsson/codechecker/tree/master/tools/report-converter#kernel-doc)

Changes

* Open reports date filter was renamed to "Outstanding reports on a given date" on the web UI. 2990
Also a new --outstanding-reports-date CLI filter argument was introduced as a filter option.
* Less code styling related checker groups are enabled by --enable-all flag.
The --enable-all flag enabled a lot of style checkers which could generate a lot of styling reports. 3013

Further improvements worth mentioning

* Allow users to overwrite location of the session file 2976
* Show how many filter items are visible at the filter tool tip if there are more items 2862
* Show selected filter items at Review status filter 2940
* Improve component statistics page load performance 3018
* Enable search and highlight occurrences of the selected text at the source code view 3011
* Set analyzer name for clang-diagnostic checkers when the reports are stored 2956
* Reintroduce skipfile script for gerrit integration to be able to analyze only the changed files. 3008
* New severity levels for cppcoreguidelines-prefer-member-initializer, altera-struct-pack-align and bugprone-redundant-branch-condition checkers 2954, 2948

Other improvements and bugfixes
For the full list of changes and improvements checkout the [milestone](https://github.com/Ericsson/codechecker/milestone/69?closed=1)

Contributors

Big thanks to everyone who helped us creating this release:
bulwahn, gargaroff, jay24rajput, sudipm-mukherjee, meghajain-1711, dl9pf, sylvestre, jimis, jgalenson,

New features

New statistics page in the Web UI
The statistics page got a new design with a lot of new features:
- statistics shown in separate tabs instead of one page for better visibility
- new, component statistics page, where reports are distributed per statistics, components can represent a part of a repository (directory, files)
- statistics comparison mode: you will be able compare the report statistics of two different analysis runs or time snapshot
- diff and review status filters are available on the statistics page
2897

Redesigned date selectors for the web UI filter and CLI
You will be able to list the open reports of your project for any date.
Open reports at a date are which were detected BEFORE the given date and NOT FIXED BEFORE the given date.
From the CLI the open reports can be queried like this:

CodeChecker cmd results --open-reports-date 2020:09:11:12:20 --url ...


Remember filters when navigate between pages
Filters are remembered during navigating between the pages. The report list and statistics related filters are saved separately.
2913

Show analyzer name alongside the reports
Analysis results from multiple static analyzers can be stored to the database, with this change for each report the analyzer
name can be viewed which produced the result.
2717

Always show similar reports
Reports with the same hash can be seen in a drop down list for each report without uniqueing 2896



Enable and disable checker profiles and guidelines (like sei-cert) in the analyzer CLI.
There is a new syntax extended with guideline support which can be used to enable checker sets.
With the new syntax the checkers, profiles and guideline can be enabled or disabled even if there is a conflict in their name.
The arguments may start with `profile:` of `guideline:` prefix which makes the choice explicit.
Without prefix it means a profile name, a guideline name or a checker group/name in this priority order.


CodeChecker analyze -o reports -e profile:sensitive -e guideline:sei-cert compile_command.json

Use these commands to list the available profiles
CodeChecker checkers --profile list
or guidelines:
CodeChecker checkers --guideline

New report converter for Markdownlint results
The reports from [Markdownlint](https://github.com/DavidAnson/markdownlint) can be converted and stored
to the report serve like this:

sh
Run Markdownlint.
mdl /path/to/your/project > ./mdl_reports.out
Use 'report-converter' to create a CodeChecker report directory from the
analyzer result of Markdownlint.
report-converter -t mdl -o ./codechecker_mdl_reports ./mdl_reports.out
Store Markdownlint reports with CodeChecker.
CodeChecker store ./codechecker_mdl_reports -n mdl

2829

The codechecker config file was extended with a parse section which can be used by the parse subcommand.
It can be used to set the path prefixes in the CodeChecker config file which should be trimmed by the parse subcommand when the reports are printed:

{
"parse": [
"--trim-path-prefix",
"/$HOME/workspace"
]
}

The config file for the parse command can be set like this:
`CodeChecker parse report --config codechecker_cfg.json`
2885

Environment variables can be used in the CodeChecker config file, they will be expanded automatically
json
{
"analyzer": [
"--skip=$HOME/project/skip.txt"
]
}

2877

Changes

On-demand Cross Translation Unit Analysis will be the default CTU analysis mode
The On-demand CTU analysis support introduced in the previous release is enabled by default now if the used
clang static analyzer supports it. CTU analysis will be performed without the huge temporary disc space allocation.

With the `--ctu-ast-mode` the analysis mode can be switched back to the old behavior if the new consumes too much memory:

CodeChecker analyze --ctu-ast-mode lod-from-pch ....


Further improvements worth mentioning
* Collect compiler information in case of clang mismatch 2872
* Log enabled checkers at the beginning of analysis 2858
* Add severity to statistics html page 2899
* Improve source component filter performance 2857
* Options specified on the command line after the --config option will override options specified in the config file 2883
* Compile command logging for `make` versions newer than 4.3 is fixed 2689

Full list of changes
View the [milestone](https://github.com/Ericsson/codechecker/milestone/67?closed=1) for the complete list of changes in this release.

Contributors
Big thanks to everyone who helped us creating this release: sylvestre gocarlos