Bigsdb

Hotfix release to fix the isolate query modification trigger not appearing on the isolate query page (broken in commit d4b5439).

**Full Changelog**: https://github.com/kjolley/BIGSdb/compare/v_1.43.0...v_1.43.1

This version adds optional web logging to record user access. A new table `log` has been added to the bigsdb_auth database that records IP address, username, and page called with a timestamp. In addition, the REST API logging to the bigsdb_rest database has been improved by the addition of client and username information for when authenticated access is used.

To enable logging, you need to set the following in bigsdb.conf:

web_log_to_db=1
rest_log_to_db=1

Logging requires writing to the database on each page access so there is a very small performance penalty to enabling this. The tables are however unlogged (i.e. data are not written to the PosgreSQL write-ahead log) which makes them considerably faster than ordinary tables but data in them will be lost in the event of a database crash or unclean shutdown.

As every page access is recorded the log tables will grow in size over time. It is recommended that they are pruned regularly to remove records older than a specified period of time - this may also be required by GDPR! The easiest way to do this is to set up a scheduled CRON job by adding the following to /etc/crontab:

0 * * * * postgres psql -c "DELETE FROM log WHERE timestamp < NOW() - INTERVAL '7 days'" bigsdb_rest > /dev/null
10 * * * * postgres psql -c "DELETE FROM log WHERE timestamp < NOW() - INTERVAL '7 days'" bigsdb_auth > /dev/null

Additionally, there is a fix to the locus stats function in the sequence typing database necessitated by the introduction of schemes that can include locus presence/absence in their definitions. You therefore need to update the bigsdb_rest, bigsdb_auth, and any sequence definition database using the rest_v1.43.sql, auth_v1.43.sql, and seqdefdb_v1.43.sql scripts respectively. See https://github.com/kjolley/BIGSdb/blob/develop/Upgrading_README.txt.

Note that in the unlikely event that you have defined schemes that utilize locus presence/absence (where profiles can include the 'P' designation to indicate 'presence'), rather than just sequence variation, you should update the locus stats within the typing database with the following commands, otherwise the reported minimum allele length may be wrong.

DELETE FROM locus_stats;
INSERT INTO locus_stats(locus,datestamp,allele_count,min_length,max_length)
SELECT loci.id,MAX(sequences.datestamp),COUNT(sequences.allele_id),MIN(LENGTH(sequence)),MAX(LENGTH(sequence))
FROM loci LEFT JOIN sequences ON loci.id=sequences.locus
WHERE allele_id NOT IN ('N','0','P') OR allele_id IS NULL
GROUP BY loci.id;


**Full Changelog**: https://github.com/kjolley/BIGSdb/compare/v_1.42.7...v_1.43.0

This version:

- Adds a new Reports plugin - this is used to generate customized genome reports. See https://github.com/kjolley/Klebsiella_reports for
examples of how to initiate and modify report templates.
- Adds an option to silently reject alleles that fail similarity check when batch uploading.
- Adds an option to make some fields expected for genome assemblies - set `required="genome_expected"` in the field attribute in config.xml.
- Makes the 'expand width' view persistent and allow its use for the front-end dashboard. This expands the width of page to that of the web browser and is useful if your results tables are very wide and you have a large monitor.
- Adds an option to return partial sequence matches for scheme queries using the API.

**Full Changelog**: https://github.com/kjolley/BIGSdb/compare/v_1.42.6...v_1.42.7

This release adds the following functionality and improvements:

* All tables can now be searched by a list of attributes. This has been possible for isolates, profiles and alleles (when using the specific-locus query) for a long time now, but it is now possible for example, to search for allele designations or tags based on a list of isolates or a list of loci.
* N50 gridlines have been added to the sequence bin cumulative length chart (https://github.com/kjolley/BIGSdb/commit/ff62aca47a4fea345b0852a224e3399f5107a8db).
* Multiple optimizations have been made to improve page load performance. This has largely focussed on the following:
* Reducing the number of database calls required by taking calls out of loops and instead retrieving batched results from the database in a single call and then caching these results (https://github.com/kjolley/BIGSdb/commit/b5998a43953ca259919155f0e1f9cb8b0d94fb72, https://github.com/kjolley/BIGSdb/commit/6131878f2c0dad49b8367c25a65e8591cce973fe, https://github.com/kjolley/BIGSdb/commit/e1ee2b64d2cf5ca3a7d5a0df21e969fef939d659, https://github.com/kjolley/BIGSdb/commit/96cc32127ebb3fe3817ccfb112864b6275dec10e).
* A more efficient SQL query has been made when generating the database view that filters the isolate table to include only records accessible by the current user, depending on their status and access to private records (https://github.com/kjolley/BIGSdb/commit/797a45a1d558b4f834833cb53203fdf4763e5fff).
* Removed the plugin manager isolate count check used to decide whether to list the plugin on the contents page (https://github.com/kjolley/BIGSdb/commit/e399d07055d3d96e33c70657e53124f3a18afd2a)
* Selected fields are now included in the ReporTree partitions summary. Any date field defined in the isolate table can now also be selected and used for the temporal component of cluster definitions.

**Full Changelog**: https://github.com/kjolley/BIGSdb/compare/v_1.42.5...v_1.42.6

This release introduces a new plugin that provides a wrapper for ReporTree ([Mixão et al. 2023 Genome Med 15:43](https://pubmed.ncbi.nlm.nih.gov/37322495/)).

See https://bigsdb.readthedocs.io/en/latest/data_analysis/reportree.html for details.

This release improves web-based sequence scanning by:

- showing the position of the first in-frame stop codon for loci defined as 'complete CDS'. The position has a coloured background that changes from red to green based on the position (red when it is at the 5' end, green when at the 3' end). This makes it easier for a curator to identify which matches are likely to be assignable as complete CDS alleles and/or whether they have internal stop codons other than near the 3' end of the sequence.


- adds an option to separate the search for start and stop codons. You can now choose to just look for in-frame stop codons that are within a specified percentage of the length of the best matching allele - either shorter or longer.