Resolwe

=================

Changed
-------
- **BACKWARD INCOMPATIBLE:** Remove obsolete processes: ``bsmap``,
``mcall``, ``coverage-garvan``, ``igv``, ``jbrowse-bed``,
``jbrowse-gff3``, ``jbrowse-gtf``, ``jbrowse-bam-coverage``,
``jbrowse-bam-coverage-normalized``, ``jbrowse-refseq``,
``fastq-mcf-single``, ``fastq-mcf-paired``, ``hsqutils-trim``,
``prinseq-lite-single``, ``prinseq-lite-paired``,
``sortmerna-single``, ``sortmerna-paired``, ``bam-coverage``,
``hsqutils-dedup``, ``vc-samtools``, ``workflow-heat-seq`` and
``alignment-tophat2``
- **BACKWARD INCOMPATIBLE:** Remove ``jbrowse-bam-coverage`` process
step from the ``workflow-accel`` workflow. The bigwig coverage track
is computed in ``align-bwa-trim`` process instead.
- **BACKWARD INCOMPATIBLE:** Remove ``resolwebio/utils`` Docker image.
This image is replaced by the ``resolwebio/common`` image.
- **BACKWARD INCOMPATIBLE:** Use ``resolwebio/common`` Docker image
as a base image for the ``resolwebio/biox``, ``resolwebio/chipseq``,
``resolwebio/dnaseq`` and ``resolwebio/rnaseq`` images
- **BACKWARD INCOMPATIBLE:** Remove ``resolwebio/legacy`` Docker image.
- Use sample name as the name of the data object in:

- ``alignment-bwa-aln``
- ``alignment-bowtie2``
- ``qc-prepeak``
- ``macs2-callpeak``
- Attach ``macs2-callpeak``, ``macs14`` and ``rose2`` process data to
the case/treatment sample
- Use ``resolwebio/dnaseq:4.0.0`` docker image in ``align-bwa-trim``
process
- Use ``resolwebio/rnaseq:4.0.0`` docker image in aligners:
``alignment-bowtie``, ``alignment-bowtie2``, ``alignment-bwa-mem``,
``alignment-bwa-sw``, ``alignment-bwa-aln``, ``alignment-hisat2``,
``alignment-star`` and ``alignment-subread``.
- Set memory limits in ``upload-genome``, ``trimmomatic-single`` and
``trimmomatic-paired`` processes
- Improve error messages in differential expression process ``DESeq2``

Added
-----
- Add ``makedb (WALT 1.01)`` - callable as ``makedb-walt``, tool to
create genome index for WALT aligner, to ``resolwebio/rnaseq`` docker
image
- Add ``resolwebio/wgbs`` docker image including the following tools:

- ``MethPipe (3.4.3)``
- ``WALT (1.01)``
- ``wigToBigWig (kent-v365)``
- Add ``resolwebio/common`` Docker image. This image includes common
bioinformatics utilities and can serve as a base image for other,
specialized ``resolwebio`` Docker images: ``resolwebio/biox``,
``resolwebio/chipseq``, ``resolwebio/dnaseq``
and ``resolwebio/rnaseq``.
- Add ``shift`` (user-defined cross-correlation peak strandshift) input
to ``qc-prepeak`` process
- Add ATAC-seq workflow
- Compute index for ``WALT`` aligner on genome upload and support
uploading the index together with the genome
- Add ``Whole genome bisulfite sequencing`` workflow and related WGBS
processes:

- ``WALT``
- ``methcounts``
- ``HMR``
- Add bedClip to `resolwebio/chipseq:3.1.0` docker image
- Add ``resolwebio/biox`` Docker image. This image is based on the
``resolwebio/common`` image and includes Biox Python library for
Dictyostelium RNA-Seq analysis support.
- Add ``resolwebio/snpeff`` Docker image. The image includes
SnpEff (4.3K) tool.
- Add spike-in names, rRNA and globin RNA cromosome names in
``resolwebio/common`` image
- Add UCSC bedGraphtoBigWig tool for calculating BigWig in
``bamtobigwig.sh`` script. In ``align-bwa-trim`` processor set this
option (that BigWig is calculated by UCSC tool instead of deepTools),
because it is much faster for amplicon files. In other processors update
the input parameters for ``bamtobigwig.sh``: ``alignment-bowtie``,
``alignment-bowtie2``, ``alignment-bwa-mem``, ``alignment-bwa-sw``,
``alignment-bwa-aln``, ``alignment-hisat2``, ``alignment-star``
``alignment-subread``, ``upload-bam``, ``upload-bam-indexed`` and
``upload-bam-secondary``.
- In ``bamtobigwig.sh`` don't create BigWig when bam file was aligned on
globin RNA or rRNA (this are QC steps and BigWig is not needed)

Fixed
-----
- **BACKWARD INCOMPATIBLE:** Use user-specificed distance metric in
hierarchical clustering
- Handle integer expression values in hierarchical clustering
- Fix Amplicon table gene hyperlinks for cases where multiple genes
are associated with detected variant
- Handle empty gene name in expression files in PCA
- Fix PBC QC reporting in ``qc-prepeak`` process for a case where
there are no duplicates in the input bam
- Fix ``macs2-callpeak`` process so that user defined fragment lenth
has priority over the ``qc-prepeak`` estimated fragment length when
shifting reads for post-peakcall QC
- Fix ``macs2-callpeak`` to prevent the extension of intervals beyond
chromosome boundaries in MACS2 bedgraph outputs
- Fix warning message in hierarchical clustering of genes to display gene
names


=================

=================

Fixed
-----
- Fix ``htseq-count-raw`` process to correctly map features with associated
feature symbols.


=================

=================

Fixed
-----
- Handle missing gene expression in hierarchical clustering of genes. If one or
more genes requested in gene filter are missing in selected expression files
a warning is issued and hierarchical clustering of genes is computed with the
rest of the genes instead of failing.
- Fix PCA computation for single sample case


=================

=================

Changed
-------
- **BACKWARD INCOMPATIBLE:** Require Resolwe 13.x
- **BACKWARD INCOMPATIBLE:** Remove ``gsize`` input from
``macs2-callpeak`` process and automate genome size selection
- **BACKWARD INCOMPATIBLE:** Set a new default ``sample`` and ``reads``
descriptor schema. Change slug from ``sample2`` to ``sample``, modify group
names, add ``cell_type`` field to the new ``sample`` descriptor schema, and
remove the original ``sample``, ``sample-detailed``, and ``reads-detailed``
descriptor schemas.
- **BACKWARD INCOMPATIBLE:** Unify types of ``macs14`` and
``macs2-callpeak`` processes and make ``rose2`` work with both
- **BACKWARD INCOMPATIBLE:** Remove ``replicates`` input in ``cuffnorm``
process. Use sample relation information instead.
- Use ``resolwebio/chipseq:3.0.0`` docker image in the following processes:

- ``macs14``
- ``macs2-callpeak``
- ``rose2``
- Downgrade primerclip to old version (v171018) in ``resolwebio/dnaseq:3.3.0``
docker image and move it to google drive.
- Move ``bam-split`` process to ``resolwebio/rnaseq:3.7.1`` docker image
- Count unique and multimmaping reads in ``regtools-junctions-annotate``
process

Added
-----
- Add ``qc-prepeak`` process that reports ENCODE3 accepted ChIP-seq and
ATAC-seq QC metrics
- Add QC report to ``macs2-callpeak`` process
- Add combining ChIP-seq QC reports in ``archive-samples`` process
- Add detection of globin-derived reads as an additional QC step in the
``workflow-bbduk-star-featurecounts-qc-single`` and
``workflow-bbduk-star-featurecounts-qc-paired`` processes.
- Add mappings from ENSEMBL or NCBI to UCSC chromosome names and deepTools
(v3.1.0) to ``resolwebio/dnaseq:3.3.0`` docker image
- Add BigWig output field to following processors:

- ``align-bwa-trim``
- ``upload-bam``
- ``upload-bam-indexed``
- ``upload-bam-secondary``
- Add ``replicate_groups`` Jinja expressions filter that accepts a list of
data objects and returns a list of labels determining replicate groups.
- Add 'Novel splice junctions in BED format' output to
``regtools-junctions-annotate`` process, so that user can visualize only
novel splice juntions in genome browsers.

Fixed
-----
- Fix handling of numerical feature_ids (NCBI source) in
``create_expression_set.py`` script
- Make ``chipseq-peakscore`` work with gzipped narrowPeak input from
``macs2-callpeak``
- Use uncompressed FASTQ files as input to STAR aligner to prevent
issues on (network) filesystems without FIFO support


=================

=================

Changed
-------
- **BACKWARD INCOMPATIBLE:** Require Resolwe 12.x
- **BACKWARD INCOMPATIBLE:** Remove obsolete processes: ``assembler-abyss``,
``cutadapt-amplicon``, ``feature_location``, ``microarray-affy-qc``,
``reads-merge``, ``reference_compatibility``, ``transmart-expressions``,
``upload-hmmer-db``, ``upload-mappability-bigwig``,
``upload-microarray-affy``.
- **BACKWARD INCOMPATIBLE:** Remove obsolete descriptor schema: ``transmart``.
- **BACKWARD INCOMPATIBLE:** Remove tools which are not used by any process:
``clustering_leaf_ordering.py``, ``go_genesets.py``, ``VCF_ad_extract.py``,
``volcanoplot.py``, ``xgff.py``, ``xgtf2gff.py``.
- **BACKWARD INCOMPATIBLE:** Management command for inserting features and
mappings requires PostgreSQL version 9.5 or newer
- Update the meta data like name, description, category, etc. of most of the
processes
- Speed-up management command for inserting mappings
- Change location of cufflinks to Google Drive for resolwebio/rnaseq Docker
build
- Calculate alignment statistics for the uploaded alignment (.bam) file in the
``upload-bam``, ``upload-bam-indexed`` and ``upload-bam-secondary`` processes.
- Annotation (GTF/GFF3) file input is now optional for the creation of the
STAR genome index files. Annotation file can be used at the alignment stage
to supplement the genome indices with the set of known features.
- Trigger process warning instead of process error in the case when
``bamtobigwig.sh`` scripts detects an empty .bam file.
- Set the default reads length filtering parameter to 30 bp in the
``rna-seq-bbduk-star-featurecounts`` and ``kapa-rna-seq-bbduk-star-featurecounts``
experiment descriptor schema. Expand the kit selection choice options in the
latter descriptor schema.

Added
-----
- Add ``MultiQC (1.6.0)`` and ``Seqtk (1.2-r94)`` to the
``resolwebio/utils:1.5.0`` Docker image
- Add ``sample2`` descriptor schema which is the successor of the original
``sample`` and ``reads`` descriptor schemas
- Add bedToBigBed and Tabix to resolwebio/rnaseq:3.7.0 docker image
- Add ``HS Panel`` choice option to the ``amplicon-master-file`` descriptor
schema
- Add MultiQC process
- Add process for the Seqtk tool ``sample`` sub-command. This process allows
sub-sampling of ``.fastq`` files using either a fixed number of reads or the
ratio of the input file.
- Add MultiQC analysis step to the ``workflow-bbduk-star-featurecounts-single``
and ``workflow-bbduk-star-featurecounts-single`` processes.
- Add ``workflow-bbduk-star-featurecounts-qc-single`` and
``workflow-bbduk-star-featurecounts-qc-paired`` processes which support
MultiQC analysis, input reads down-sampling (using Seqtk) and rRNA
sequence detection using STAR aligner.
- Add to ``resolwebio/chipseq`` Docker image:

- ``bedtools (2.25.0-1)``
- ``gawk (1:4.1.3+dfsg-0.1)``
- ``picard-tools (1.113-2)``
- ``run_spp.R (1.2) (as spp)``
- ``SPP (1.14)``
- Add ``regtools-junctions-annotate`` process that annotates novel splice
junctions.
- Add ``background`` relation type to fixtures

Fixed
-----
- Track ``source`` information in the ``upload-fasta-nucl`` process.
- When STAR aligner produces an empty alignment file, re-sort the alignment
file to allow successful indexing of the output ``.bam`` file.
- Create a symbolic link to the alignment file in the ``feature_counts`` process,
so that relative path is used in the quantification results. This prevent the
FeatureCounts output to be listed as a separate sample in the MultiQC reports.
- Fix handling of expression objects in ``archive-samples`` process


===================

===================

Changed
-------
- **BACKWARD INCOMPATIBLE:** Require Resolwe 11.x
- **BACKWARD INCOMPATIBLE:** Use read count instead of sampling rate
in strandedness detection
- **BACKWARD INCOMPATIBLE:** Remove ``genome`` input from ``rose2``
process and automate its selection
- **BACKWARD INCOMPATIBLE:** Refactor ``cutadapt-paired`` process
- **BACKWARD INCOMPATIBLE:** Improve leaf ordering performance in gene and
sample hierarchical clustering. We now use exact leaf ordering which has
been recently added to ``scipy`` instead of an approximate in-house
solution based on nearest neighbor algorithm. Add informative warning
and error messages to simplify troubleshooting with degenerate datasets.
- Remove ``igvtools`` from ``resolwebio/utils`` Docker image
- Improve helper text and labels in processes used for sequencing data upload
- Allow using custom adapter sequences in the
``workflow-bbduk-star-featurecounts-single`` and
``workflow-bbduk-star-featurecounts-paired`` processes
- Change chromosome names from ENSEMBL / NCBI to UCSC (example: "1" to
"chr1") in BigWig files. The purpose of this is to enable viewing BigWig
files in UCSC genome browsers for files aligned with ENSEBML or NCBI genome.
This change is done by adding script bigwig_chroms_to_ucsc.py to
bamtobigwig.sh script.
- Reduce RAM requirement in SRA import processes

Added
-----
- Add two-pass mode to ``alignment-star`` process
- Add ``regtools (0.5.0)`` to ``resolwebio/rnaseq`` Docker image
- Add KAPA experiment descriptor schema
- Add ``resdk`` Python 3 package to ``resolwebio/utils`` Docker image
- Add to ``cutadapt-single`` process an option to discard reads having more
'N' bases than specified.
- Add workflows for single-end ``workflow-cutadapt-star-featurecounts-single``
and paired-end reads ``workflow-cutadapt-star-featurecounts-paired``.
Both workflows consist of preprocessing with Cutadapt, alignment
with STAR two pass mode and quantification with featureCounts.
- Add descriptor schema ``rna-seq-cutadapt-star-featurecounts``

Fixed
-----
- **BACKWARD INCOMPATIBLE:** Fix the ``stitch`` parameter handling in
``rose2``
- fix ``upload-gtf`` to create JBrowse track only if GTF file is ok
- Pin ``sra-toolkit`` version to 2.9.0 in ``resolwebio/utils`` Docker image.
- Fix and improve ``rose2`` error messages
- Fail gracefully if bam file is empty when producing bigwig files
- Fail gracefully if there are no matches when mapping chromosome names


===================