Resolwe

===================

Changed
-------
- **BACKWARD INCOMPATIBLE:** Remove management command module
- **BACKWARD INCOMPATIBLE:** Remove filtering of genes with low expression
in PCA analysis
- **BACKWARD INCOMPATIBLE:** Remove obsolete RNA-seq DSS process
- Expand error messages in ``rose2`` process
- Check for errors during download of FASTQ files and use
``resolwebio/utils:1.3.0`` Docker image in import SRA process
- Increase Feature's full name's max length to 350 to support a long full
name of "Complement C3 Complement C3 beta chain C3-beta-c Complement C3
alpha chain C3a anaphylatoxin Acylation stimulating protein Complement C3b
alpha' chain Complement C3c alpha' chain fragment 1 Complement C3dg
fragment Complement C3g fragment Complement C3d fragment Complement C3f
fragment Complement C3c alpha' chain fragment 2" in Ensembl

Added
-----
- Add `exp_set` and `exp_set_json` output fields to expression processes:

- ``feature_counts``
- ``htseq-count``
- ``htseq-count-raw``
- ``rsem``
- ``upload-expression``
- ``upload-expression-cuffnorm``
- ``upload-expression-star``
- Add 'Masking BED file' input to ``rose2`` process which allows
masking reagions from the analysis
- Add ``filtering.outFilterMismatchNoverReadLmax`` input to
``alignment-star`` process
- Add mappings from ENSEMBL or NCBI to UCSC chromosome names to
``resolwebio/rnaseq:3.5.0`` docker image

Fixed
-----
- Fix peaks BigBed output in ``macs14`` process
- Remove duplicated forward of ``alignIntronMax`` input field in
BBDuk - STAR - featureCounts workflow
- Make ``cuffnorm`` process attach correct expression data objects to
samples
- Fix ``upload-gtf`` in a way that GTF can be shown in JBrowse. Because
JBrowse works only with GFF files, input GTF is converted to GFF from
which JBrowse track is created.


===================

===================

Fixed
-----
- Fix ``bamtobigwig.sh`` to timeout the ``bamCoverage`` calculation after
defined time


===================

===================

Added
-----
- Add to ``resolwebio/chipseq`` Docker image:

- ``Bedops (v2.4.32)``
- ``Tabix (v1.8)``
- ``python3-pandas``
- ``bedGraphToBigWig (kent-v365)``
- ``bedToBigBed (kent-v365)``
- Add to ``resolwebio/rnaseq:3.2.0`` Docker image:

- ``genometools (1.5.9)``
- ``igvtools (v2.3.98)``
- ``jbrowse (v1.12.0)``
- ``Bowtie (v1.2.2)``
- ``Bowtie2 (v2.3.4.1)``
- ``BWA (0.7.17-r1188)``
- ``TopHat (v2.1.1)``
- ``Picard Tools (v2.18.5)``
- ``bedGraphToBigWig (kent-v365)``
- Add Debian package ``file`` to ``resolwebio/rnaseq:3.3.0`` Docker image
- Support filtering by type on feature API endpoint
- Add BigWig output field to following processes:

- ``alignment-bowtie``
- ``alignment-bowtie2``
- ``alignment-tophat2``
- ``alignment-bwa-mem``
- ``alignment-bwa-sw``
- ``alignment-bwa-aln``
- ``alignment-hisat2``
- ``alignment-star``
- Add Jbrowse track output field to ``upload-genome`` processor.
- Use ``reslowebio/rnaseq`` Docker image and add Jbrowse track and IGV
sorting and indexing to following processes:

- ``upload-gff3``
- ``upload-gtf``
- ``gff-to-gtf``
- Add Tabix index for Jbrowse to ``upload-bed`` processor and use
``reslowebio/rnaseq`` Docker image
- Add BigWig, BigBed and JBrowse track outputs to ``macs14`` process
- Add Species and Build outputs to ``rose2`` process
- Add Species, Build, BigWig, BigBed and JBrowse track outputs to ``macs2``
process
- Add ``scipy`` (v1.1.0) Python 3 package to ``resolwebio/utils`` Docker image

Changed
-------
- **BACKWARD INCOMPATIBLE:** Drop support for Python 3.4 and 3.5
- **BACKWARD INCOMPATIBLE:** Require Resolwe 10.x
- **BACKWARD INCOMPATIBLE:** Upgrade to Django Channels 2
- **BACKWARD INCOMPATIBLE:** Count fragments (or templates) instead of reads
by default in ``featureCounts`` process and
``BBDuk - STAR - featureCounts`` pipeline. The change applies only to
paired-end data.
- **BACKWARD INCOMPATIBLE:** Use ``resolwebio/rnaseq:3.2.0`` Docker image
in the following processes that output reads:

- ``upload-fastq-single``
- ``upload-fastq-paired``
- ``files-to-fastq-single``
- ``files-to-fastq-paired``
- ``reads-merge``
- ``bbduk-single``
- ``bbduk-paired``
- ``cutadapt-single``
- ``cutadapt-paired``
- ``cutadapt-custom-single``
- ``cutadapt-custom-paired``
- ``trimmomatic-single``
- ``trimmomatic-paired``.

This change unifies the version of ``FastQC`` tool (0.11.7) used for
quality control of reads in the aforementioned processes. The new Docker
image comes with an updated version of Cutadapt (1.16) which affects
the following processes:

- ``cutadapt-single``
- ``cutadapt-paired``
- ``cutadapt-custom-single``
- ``cutadapt-custom-paired``.

The new Docker image includes also an updated version of Trimmomatic (0.36)
used in the following processes:

- ``upload-fastq-single``
- ``upload-fastq-paired``
- ``files-to-fastq-single``
- ``files-to-fastq-paired``
- ``trimmomatic-single``
- ``trimmomatic-paired``.
- **BACKWARD INCOMPATIBLE:** Change Docker image in ``alignment-subread``
from ``resolwebio/legacy:1.0.0`` with Subread (v1.5.1) to
``resolwebio/rnaseq:3.2.0`` with Subread (v1.6.0). ``--multiMapping`` option
was added instead of ``--unique_reads``. By default aligner report uniquely
mapped reads only.
- Update ``wigToBigWig`` to kent-v365 version in ``resolwebio/chipseq``
Docker image
- Change paths in HTML amplicon report template in ``resolwebio/dnaseq``
Docker image
- Move assay type input in BBDuk - STAR - featureCounts pipeline descriptor
schema to advanced options
- Use ``resolwebio/rnaseq:3.2.0`` Docker image with updated versions of tools
instead of ``resolwebio/legacy:1.0.0`` Docker image in following processes:

- ``alignment-bowtie`` with Bowtie (v1.2.2) instead of Bowtie (v1.1.2)
- ``alignment-bowtie2`` with Bowtie2 (v2.3.4.1) instead of Bowtie2 (v2.2.6)
- ``alignment-tophat2`` with TopHat (v2.1.1) instead of TopHat (v2.1.0)
- ``alignment-bwa-mem``, ``alignment-bwa-sw` and ``alignment-bwa-aln``
with BWA (v0.7.17-r1188) instead of BWA (v0.7.12-r1039)
- ``alignment-hisat2`` with HISAT2 (v2.1.0) instead of HISAT2 (v2.0.3-beta)
- ``upload-genome``
- Use ``resolwebio/base:ubuntu-18.04`` Docker image as a base image in
``resolwebio/utils`` Docker image
- Update Python 3 packages in ``resolwebio/utils`` Docker image:

- ``numpy`` (v1.14.4)
- ``pandas`` (v0.23.0)
- Replace ``bedgraphtobigwig`` with ``deepTools`` in ``resolwebio/rnaseq``
Docker image, due to faster performance
- Use ``resolwebio/rnaseq:3.3.0`` Docker image in ``alignment-star-index``
with STAR (v2.5.4b)

Fixed
-----
- Make management commands use a private random generator instance
- Fix output ``covplot_html`` of ``coveragebed`` process
- Fix process ``archive-samples`` and ``amplicon-archive-multi-report`` to
correctly handle nested file paths
- Change ``rose2`` and ``chipseq-peakscore`` to work with ``.bed`` or
``.bed.gz`` input files
- Fix the ``expression-aggregator`` process so that it tracks the
``species`` of the input expression data
- Fix ``bamtobigwig.sh`` to use ``deepTools`` instead of ``bedtools`` with
``bedgraphToBigWig`` due to better time performance


==================

==================

Changed
-------
- **BACKWARD INCOMPATIBLE:** Simplify the ``amplicon-report`` process inputs
by using Latex report template from the ``resolwebio/latex`` Docker image assets
- **BACKWARD INCOMPATIBLE:** Simplify the ``coveragebed`` process inputs
by using Bokeh assets from the ``resolwebio/dnaseq`` Docker image
- **BACKWARD INCOMPATIBLE:** Require Resolwe 9.x
- Update ``wigToBigWig`` tool in ``resolwebio/chipseq`` Docker image
- Use ``resolwebio/rnaseq:3.1.0`` Docker image in the following
processes:

- ``cufflinks``
- ``cuffnorm``
- ``cuffquant``
- Remove ``differentialexpression-limma`` process
- Use ``resolwebio/rnaseq:3.1.0`` docker image and expand error
messages in:

- ``cuffdiff``
- ``differentialexpression-deseq2``
- ``differentialexpression-edger``
- Update ``workflow-bbduk-star-htseq``
- Update ``quantseq`` descriptor schema
- Assert species and build in ``htseq-count-normalized`` process
- Set amplicon report template in ``resolwebio/latex`` Docker image to
landscape mode

Added
-----
- Support Python 3.6
- Add ``template_amplicon_report.tex`` to ``resolwebio/latex`` Docker image
assets
- Add SnpEff tool and bokeh assets to ``resolwebio/dnaseq`` Docker image
- Add automated library strand detection to ``feature_counts`` quantification process
- Add FastQC option ``nogroup`` to ``bbduk-single`` and ``bbduk-paired`` processes
- Add CPM normalization to ``htseq-count-raw`` process
- Add ``workflow-bbduk-star-htseq-paired``
- Add legend to amplicon report template in ``resolwebio/latex`` Docker image

Fixed
-----
- Fix manual installation of packages in Docker images to handle dots and
spaces in file names correctly
- Fix COSMIC url template in ``amplicon-table`` process
- Fix Create IGV session in Archive samples process
- Fix ``source`` tracking in ``cufflinks`` and ``cuffquant`` processes
- Fix amplicon master file validation script. Check and report error if
duplicated amplicon names are included. Validation will now pass also
for primer sequences in lowercase.
- Fix allele frequency (AF) calculation in ``snpeff`` process
- Fix bug in script for calculating FPKM. Because genes of raw counts from
``featureCounts`` were not lexicographically sorted, division of normalized counts
was done with values from other, incorrect, genes. Results from ``featureCounts``,
but not ``HTSeq-count`` process, were affected.


==================

==================

Changed
-------
- Use the latest versions of the following Python packages in
``resolwebio/rnaseq`` docker image: Cutadapt 1.16, Apache Arrow 0.9.0, pysam
0.14.1, requests 2.18.4, appdirs 1.4.3, wrapt 1.10.11, PyYAML 3.12
- Bump tools version in ``resolwebio/rnaseq`` docker image:

- Salmon to 0.9.1
- FastQC to 0.11.7
- Generalize the no-extraction-needed use-case in ``resolwebio/base`` Docker
image ``download_and_verify`` script

Added
-----
- Add the following Python packages to ``resolwebio/rnaseq`` docker image: six
1.11.0, chardet 3.0.4, urllib3 1.22, idna 2.6, and certifi 2018.1.18
- Add ``edgeR`` R library to ``resolwebio/rnaseq`` docker image
- Add Bedtools to ``resolwebio/rnaseq`` docker image

Fixed
-----
- Handle filenames with spaces in the following processes:

- ``alignment-star-index``
- ``alignment-tophat2``
- ``cuffmerge``
- ``index-fasta-nucl``
- ``upload-fasta-nucl``
- Fix COSMIC url template in (multisample) amplicon reports


==================

==================

Changed
-------
- **BACKWARD INCOMPATIBLE:** Refactor ``trimmomatic-single``,
``trimmomatic-paired``, ``bbduk-single``, and ``bbduk-paired`` processes
- **BACKWARD INCOMPATIBLE:** Merge ``align-bwa-trim`` and ``align-bwa-trim2``
process functionality. Retain only the refactored process under slug
``align-bwa-trim``
- **BACKWARD INCOMPATIBLE:** In processes handling VCF files, the output
VCF files are stored in bgzip-compressed form. Tabix index is not referenced
to an original VCF file anymore, but stored in a separate ``tbi`` output
field
- **BACKWARD INCOMPATIBLE:** Remove an obsolete ``workflow-accel-2`` workflow
- **BACKWARD INCOMPATIBLE:** Use Elasticsearch version 5.x
- **BACKWARD INCOMPATIBLE:** Parallelize execution of the following processes:

- ``alignment-bowtie2``
- ``alignment-bwa-mem``
- ``alignment-hisat2``
- ``alignment-star``
- ``alignment-tophat2``
- ``cuffdiff``
- ``cufflinks``
- ``cuffquant``
- Require Resolwe 8.x
- Bump STAR aligner version in ``resolwebio/rnaseq`` docker image to 2.5.4b
- Bump Primerclip version in ``resolwebio/dnaseq`` docker image
- Use ``resolwebio/dnaseq`` Docker image in ``picard-pcrmetrics`` process
- Run ``vc-realign-recalibrate`` process using multiple cpu cores to optimize
the processing time
- Use ``resolwebio/rnaseq`` Docker image in ``alignment-star`` process

Added
-----
- Add CNVKit, LoFreq and GATK to ``resolwebio/dnaseq`` docker image
- Add BaseSpace files download tool
- Add process to import a file from BaseSpace
- Add process to convert files to single-end reads
- Add process to convert files to paired-end reads
- Add ``vc-gatk4-hc`` process which implements GATK4 HaplotypeCaller variant
calling tool
- Add ``workflow-accel-gatk4`` pipeline that uses GATK4 HaplotypeCaller as an
alternative to GATK3 used in ``workflow-accel`` pipeline
- Add ``amplicon-master-file`` descriptor schema
- Add ``workflow-bbduk-star-featurecounts`` pipeline
- Add ``rna-seq-bbduk-star-featurecounts`` RNA-seq descriptor schema

Fixed
-----
- Fix iterative trimming in ``bowtie`` and ``bowtie2`` processes
- Fix ``archive-samples`` to use sample names for headers when merging
expressions
- Improve ``goea.py`` tool to handle duplicated mapping results
- Handle filenames with spaces in the following processes:

- ``alignment-hisat2``
- ``alignment-bowtie``
- ``prepare-geo-chipseq``
- ``prepare-geo-rnaseq``
- ``cufflinks``
- ``cuffquant``


==================