Deep-lynx

This release of Deep Lynx is considered a major, non-breaking release.

Deep Lynx is unique in its ability to store data in graph-like format and under a user-defined ontology. Data has been versioned in Deep Lynx since roughly version 0.0.5. Data ingested from sources have their changes tracked and theoretically a user could see what the data looked like at any given point in time.

However, while data was versioned the ontology it was stored under was not. This led to problems when users might have edited the ontology - removing or requiring new fields for example. These changes would not be tracked and if a user accessed data stored under an old version of a class, they might see deprecated properties, or be lacking required data. There was no way to reprocess data that had come in to fit the new ontology and the type mapping process had to be manually updated to handle changes.

To solve these problems and give users confidence in the accuracy of stored data, versioning was introduced to ontologies stored in Deep Lynx. Now when users query data they will always see the class and properties the way they existed when that data was stored. Changes to the ontology are now tracked and managed- and final approval of changes prior to application now falls to the container administrators, not all users.

https://gitlab.software.inl.gov/b650/Deep-Lynx/-/wikis/Ontology-Versioning

This is a minor release for Deep Lynx and contains various bugfixes and stability corrections. Please run `npm run migrate` as soon as you pull this release as database structure has changed.

Added the ability to back artifact storage with Postgres's Large Object storage capability. This is now the default for artifact storage over local filesystem, as it's a better respecter of keys and not overwriting data.

Also includes routes for managing changelists, ontology versions, and container alerts - though these have not been fully implemented on the GUI as of yet.

Deep Lynx has been modified to no longer process data and data sources in batches, but in an event-driven queue based manner. Using this methodology we hope to scale more efficiently to larger data sets and allow users to more easily process data. You can find a diagram attached detailing the changes.


- Query operators < and > added
- Event system completely rewritten
- Queue systems built, Database and Rabbitmq systems implemented
- Dynamic graphql and resolver generation
- Test data generator UI portion

Deep Lynx was using a sophisticated database trigger for managing node and edge insertion. These triggers allowed the automatic setting of a `deleted_at` field whenever someone entered a record with an existing id. Unfortunately, this method proved to be extremely time consuming on the database side and caused numerous slowdowns on even the small tables.

This release removed those triggers and replaced them with different functions and functionality. We still have point in time versioning, and we will still be able to support point in time querying - but the `deleted_at` field will no longer be as important when viewing the data. That field now indicates manual or automatic deletion of records not relating to initial data ingestion.