Indra

Python compatibility
- This release of INDRA is compatible with Python 3.5+ only and 2.x is not supported anymore.

Package structure, dependencies
- The `protmapper` package is a new dependency which takes over interfacing with UniProt (including file-based access and web service access). Consequently, almost all functionalities of `indra.databases.uniprot_client` and `indra.databases.phosphosite_client`, as well as the core of `indra.preassembler.site_mapper` are now provided by `protmapper`. The overall API of these INDRA modules has not changed, however.
- INDRA is now also available as a Docker image on Dockerhub under
`labsyspharm/indra`, further instructions in the README
- Installation instructions for the Python-Java bridge pyjnius have been updated,
INDRA's Java/Scala based dependencies can now work with versions of Java up to 11.
- The `statements.py` file has been refactored into its own module in
`indra.statements`, the overall API has not changed.
- INDRA's logging is now fully hierarchical and follows the submodule
structure.

Input processors
- CWMS processor in `indra.sources.cwms` can extract temporal and geo-spatial
context.
- Major overhaul of Hume processor in `indra.sources.hume`, supporting latest
JSON-LD format and extracting temporal and geo-spatial context
- An additional JSON-based API and processor is now available for SOFIA in
`indra.sources.sofia`, supporting text reading.
- New input processor for the RLIMS-P reading system in `indra.sources.rlimsp`.
- The INDRA DB REST source in `indra.sources.indra_db_rest` has been refactored
into multiple files and now provides a Processor class-based interface similar
to other input sources.
- More robust handling of invalid identifiers in PyBEL inputs in
`indra.sources.bel`.
- BioPAX input processor in `indra.sources.biopax` now eliminates spurious
duplicate extractions by patterns
- New analysis script is available in `indra.sources.trips` for TRIPS/CWMS
extractions.

Core assembly modules
- Belief engine makes a new Bayesian scorer class available in `indra.belief`
which allows integrating curation directly into belief scores.
- Preassembler in `indra.preassembler` preserves original grounding in addition
to the original entity texts upon Statement deduplication.
- The SiteMapper has been substantially refactored to take its core functionality from the `protmapper` package (https://github.com/indralab/protmapper).
- A new disambiguation tool, DEFT (https://github.com/indralab/deft) is now
integrated into the grounding mapping process in
`indra.preassembler.grounding_mapper`.
- Bug fixes and performance improvements in HierarchyManager in
`indra.preassembler.hierarchy_manager`.
- Bug fixes and performance improvements in OntologyMapper in
`indra.preassembler.ontology_mapper`.
- Assembly pipeline in `indra.tools.assemble_corpus` now supports merging
grounding from multiple evidences at the Statement level, as well as additional
preasssembly functionalities such as evidence flattening.

Tools
- New web service added for real-time curation of Statements with belief
updates, ontology extension, and re-grounding/re-assembly in
`indra.tools.live_curation`.

Output assemblers
- Major overhaul of the PySB Assembler in `indra.assemblers.pysb` to refactor
into multiple files, support individual statement-level policies, and passing
in parameterization from outside.
- CX assembler in `indra.assemblers.cx` supports setting custom network visual
styles.
- Major improvements and many new features in the HTML assembler for curation in
`indra.assemblers.html`, including grouping and ranking of Statements
- Fixes to the SBGN assembly format in `indra.assemblers.sbgn`.

Literature and database clients
- PMC client in `indra.literature.pmc` now supports getting text from NXML
- Protein lengths now available via the UniProt client in
`indra.databases.uniprot_client`

Resources
- All resource files updated to latest version as of 12/1/2018
- New ontology mapping resources for Hume and CWMS

REST API
- Multiple new input and assembly endpoints added to the REST API for Eidos,
CWMS, Sofia, and Hume, ontology mapping, and belief filtering.

**This is the last release of INDRA that is compatible with both Python 2.7 and Python 3.5+. The next release will drop Python 2 compatibility and maintain compatibility with Python 3.5+ only.**

Package structure, dependencies
- INDRA is now compatible with ndex2 version >= 2.0

Representation
- New `text_refs` attribute in Evidence which allows representing text
references other than PMID, e.g. DOI, URL, etc.
- Evidence str/repr display improved and extended to support pretty printing
large Evidence objects

Input processors
- Geographical location context extracted from Eidos in `indra.sources.eidos`
- EKB file-based processing added to TRIPS and CWMS APIs in
`indra.sources.trips` and `indra.sources.cwms`
- REACH processor now extracts and propagate entity coordinates from
REACH output in `indra.sources.reach`
- Sparser API has more robust timeout and failure handling, and multiple
improvements in the processor in `indra.sources.sparser`
- New function to submit curation via INDRA DB REST API in
`indra.sources.indra_db_rest`
- Improvements and extensions in INDRA DB REST API including the ability
to limit number of returned Statements in `indra.sources.indra_db_rest`
- Bug fixes in MedScan processor in `indra.sources.medscan`

Core assembly modules
- Belief engine refactored to pass evidences instead of scores when propagating
evidence on the refinements graph in `indra.belief`
- The assembly of large Complexes and Associations has been significantly
optimized in `indra.statements` and `indra.preassembler`

Output assemblers
- New HTML Statement browser and curation interface added in
`indra.assemblers.html`
- English assembler extended to handle Influence, Association, and Conversion
Statements in `indra.assemblers.english`
- Bug fixes in CX assembly and hub layout in `indra.assemblers.cx`

Literature and database clients
- New GO client added in `indra.databases.go_client` which allows normalizing
biological processes and cellular components to standardized names
- UniProt client can now retreive descriptions of proteins in
`indra.databases.uniprot_client`

Resources
- All resource files updated to latest version as of 10/30/2018
- Ontologies for SOFIA, Eidos and Hume, and mapping between them updated

Package structure
- The DB is no longer part of the INDRA package, and `indra_db_rest`, `indra.db` and `indra.tools.reading.db_reading` have been moved to https://github.com/indralab/indra_db
- Most of the models and applications in the `models` folder have been moved to https://github.com/sorgerlab/indra_apps with just some core examples and models remaining
- Files in the submodules in `indra.sources` are now following a standardized naming of `api.py` and `processor.py`. The corresponding APIs haven't changed.
- The `indra.assemblers` module is now divided into further submodules, one for each output assembler, e.g. `indra.assemblers.pysb`. This changes the way these modules need to be imported to e.g. `from indra.assemblers.pysb import PysbAssembler`.

Representation
- Object model for `Context` added, including `BioContext` and `WorldContext` and further classes. `Evidence` now has a `context` attribute to carry this information (which previously appeared in `annotations`).
- New `Association` Statement class to represent non-causal and non-directional relationships
- SBO annotations in JSON serialization are now obtional and off by default

Input processors
- New input processor and API for LINCS drug-target relationships (`indra.sources.lincs_drug`)
- New input processor and API for Target Affinity Spectrum data for drug-target relationships (`indra.sources.tas`)
- The Eidos input processor has numerous improvements and extenstions including producing `WorldContext` for each `Evidence`, carrying negation and hedging, and picking up more meta-information and annotations (`indra.sources.eidos.processor`)
- The Eidos reader has been updated to work with Eidos version 0.2.2 (`indra.sources.eidos.reader`)
- The BEL input processor now produces BioContext for each Evidence (`indra.sources.bel.processor`)
- Many new features in the INDRA DB REST client (`indra.sources.indra_db_rest`) including ordering, timeouts, various size limits, retries, etc.
- REACH input processor now produces BioContext for each Evidence (`indra.sources.reach.processor`)
- SIGNOR input processor now produces BioContext for each Evidence (`indra.sources.signor.processor`)
- SOFIA input processor now produces WorldContext for each Evidence (`indra.sources.sofia.processor`)
- TEES API and processor refactored to support batch reading (`indra.sources.tees`)


Core assembly modules
- The BeliefEngine in `indra.belief` has been refactored to allow subclassing the BeliefScorer to implement custom scoring approaches. The default scores have been moved to a resource file in `indra.resources`.
- The BeliefEngine can now handle negative as well as positive evidence for a given Statement.
- The Preassembler now adds annotations for raw texts corresponding to Agents that would otherwise be lost upon duplicate resolution.

Output assemblers
- New hub layout generator for CX networks in `indra.assemblers.cx`
- Improvements and extensions to English assembler in `indra.assemblers.english`

Literature and database client
- New client to the MeSH database to allow name lookup and standardization in `indra.databases.mesh_client`
- New client to the LINCS small molecule and protein tables in `indra.databases.lincs_client`
- NDEx client can now be used without explicitly passing credentials in `indra.databases.ndex_client`
- Improvements to Elsevier client to be able to extract text from a wider range of XML formats in `indra.literature.elsevier_client`

Resources
- Resource files for new sources and database clients added
- Statements JSON schema updated with Context object model

Setup and dependencies:
- INDRA now supports networkx 2.x
- The Cython version restriction is now removed, and the recommended installation method for pyjnius is through Github
- Extra dependencies for various modules are now given explicitly in setup.py
- Separate configuration module (`indra.config`) to manage `config.ini` and `db_config.ini` files
- Contribution guidelines added

Representation:
- Extensions to keys and hashes used to identify related Statements (`indra.statements`)
- Statements now implement a `contradicts` method to find contradictions

Input processors:
- Major improvements and adaptation to changes in Eidos (`indra.sources.eidos`), Hume (`indra.sources.hume`), CWMS (`indra.sources.cwms`), and SOFIA (`indra.sources.sofia`) reader inputs
- BBN reader renamed to Hume (`indra.sources.hume`)
- Ontologies for the Eidos, Hume, and CWMS readers are integrated, and grounding information from these sources is represented
- New ISI reader API and processor added (`indra.sources.isi`)
- New Medscan reader API and processor added (`indra.sources.medscan`)
- Generalized and refactored SIGNOR input API and processor added (`indra.sources.signor`)
- BEL resources are now PyBEL compatible and processing defaults to PyBEL, rather than BEL/RDF (`indra.sources.bel`)
- Imrpoved modification and mutation site extraction, and handling of complex controllers in REACH input processor (`indra.sources.reach`)
- Bug fixes and improvements in TEES (`indra.sources.tees`) and Sparser (`indra.sources.sparser`) inputs
- Broader set of Increase/DecreaseAmount Statements extracted from TRIPS (`indra.sources.trips`)
- Added robust DRUM reader to use TRIPS/DRUM locally (`indra.sources.trips.drum_reader`)

Core assembly modules
- Contradiction finding in Preassembler (`indra.preassembler`)
- Handling of new `is_opposite` relationship in Statements, HierarchyManager and Preassembler
- Generalized reasoning across `isa/partof` paths in HierarchyManager (`indra.preassembler.hierarchy_manager`)
- New Ontology Mapper added (`indra.preassembler.ontology_mapper`)
- SiteMapper uses cache to speed up sequence access (`indra.preassembler.site_mapper`)
- Assembly filters now extend across BoundConditions (`indra.tools.assemble_corpus`)

Output assemblers
- BMI wrapper implemented for PySB models for integrated simulation in workflows (`indra.assemblers.bmi_wrapper`)
- CAG Assembler can produce Jupyter-embeddable graphs and TSVs (`indra.assemblers.cag_assembler`)
- Many generalizations and improvements in PyBEL assembler (`indra.assemblers.pybel_assembler`)
- Bug fixes and improvements in PySB, SIF, Graph, English, IndexCard and CX assemblers

Literature and database clients:
- NewsAPI client added (`indra.literature.news_api`)
- Many new functionalities to access content in Elsevier client (`indra.literature.elsevier`)
- UniProt client (`indra.databases.uniprot_client`) extended with protein synonyms; optimizations in resource loading result in faster import times

Resources
- Many resource files updated, including FamPlex (`indra.resources.famplex`)
- New INDRA JSON schema and documentation added (`indra/resources/statements_schema.json`)

Analysis and explanation
- Paths graph is now a separate package, and a dependency of INDRA (https://github.com/johnbachman/paths_graph) and has been removed from `indra.explanation`

High-throughput reading and database
- REST API updates and extensions (`db_rest_api` and `rest_api`)
- Separate client and config modules for DB (`indra.db.client` and `indra.db.config`)
- New reading and preassembly managers for DB (`indra.db.reading_manager`, `indra.db.preassembly_manager`)
- New statistics module for DB (`indra.db.stats`)
- New reporting and log analysis functionalities for AWS-based reading (`indra.tools.reading` and `indra.tools.reading.db_reading`)
- Extensive improvements to running and managing AWS-based reading jobs (`indra.tools.reading`)

Configuration:
- INDRA now comes with a dedicated configuration file (created when
first importing indra) at ~/.config/indra/config.ini in which paths to various
resources and access keys can be configured

Representation:
- New Concept parent class for Agent. Concepts allow representing entities
of interest without the assumption of molecular state that Agents have.

Input processors
- BioGRID: new dedicated input processor for interactions from BioGRID
in `indra.sources.biogrid`. This is preferable to the BioGRID content
coming from Pathway Commons (via `indra.sources.biopax`), in some cases.
- CWMS: new API and processor for the TRIPS/CWMS reading system
(http://trips.ihmc.us/parser/cgi/cwmsreader) in `indra.sources.cwms`
- SOFIA: initial implementation for input processor for reading output from
the SOFIA reading system added in `indra.sources.sofia`
- Grounding information from Eidos, BBN and CWMS systems now represented and
used in assembly

Core assembly modules
- All core assembly procedures now extend to BoundConditions of Agents
- HierarchyManager generalized to allow merging multiple hierarchies
into a single manager
- New `isa` hierarchies added for non-biology use cases that now work with
HierarchyManager / Preassembler
- Custom grounding map can now be propagated into assembly from
`indra.tools.machine` and also via `indra.tools.assemble_corpus`

Output assemblers
- TSV Assembler: new assembler in `indra.assemblers.tsv_assembler` to
output knowledge in a tabular format, suitable for human curation

High-throughput reading and database
- REST API for database module to query INDRA Statement content from local or
remote database
- Various improvements and extensions of high-throughput reading framework
and database utilities

Representation:
- New Influence Statement to represent causal influence between a subject
and an object, and optionally the polarity and magnitude of change in
subject and object

Input processors
- TEES: new API in `indra.sources.tees` to read with the
Turku Event Extraction System
(https://github.com/jbjorne/TEES) and instantiate INDRA Statements
- PyBEL: new API in `indra.sources.bel` that allows processing PyBEL graphs
(https://github.com/pybel/pybel) as an input to INDRA
- NDEx CX: new API in `indra.sources.ndex_cx` to process networks on NDEx
(http://ndexbio.org) and extract INDRA Statements
- Signor: new API in `indra.sources.signor` to read the content of the
Signor signaling network database (https://signor.uniroma2.it/) as
INDRA Statements
- Geneways: new API in `indra.sources.geneways` to read the output generated
by the Genewayes reading system
(https://www.ncbi.nlm.nih.gov/pubmed/15016385)
- Eidos: new API in `indra.sources.eidos` to read text using the Eidos reading
system (https://github.com/clulab/eidos/) and process its output of
causal interactions into INDRA Statements
- BBN: new API in `indra.sources.bbn` to process causal interactions extracted
by BBN's world modelers reading system

Core assembly modules
- Flexible parameterization of the BeliefEngine in `indra.belief`, and support for prior
belief assignment based on more granular attributes of a Statement's evidence

Output assemblers
- Causal Analysis Graphs: new assembler in `indra.assemblers.cag_assembler`
to assemble into "Causal Analysis Graphs", and display with the Delphi package
(https://github.com/ml4ai/delphi)
- PyBEL: new assembler in `indra.assemblers.pybel_assembler` to assemble into
a PyBEL graph and allow analysis on PyBEL web
(https://pybel.scai.fraunhofer.de/)

Model analysis
- Integration with `kappy` to facilitate Kappa model analysis and simulation
- Several new, optimized graph data structures in `indra.explanation`

Tools
- The "machine" (basis of The RAS Machine) has been moved
to `indra.tools.machine`.

Database clients and resources
- FamPlex (renamed from Bioentities) is no longer a submodule, it is available
directly under `indra.resources.famplex`
- New ChEBML client in `indra.databases.chembl_client`
- New functionalities in cBio portal and ChEBI clients

High-throughput reading and database
- Various high-througput reading modules refactored into four submodules based
on use case in `indra.tools.reading`
- New database module in `indra.db` to cache INDRA Statements from
reading at scale