Python-catmaid

Contributors: Albert Cardona, Andrew Champion, Daniel Witvliet, Stephan Gerhard, Tom Kazimiers

Notes

Starting with this release CATMAID uses a new database migration system. To
update an existing CATMAID instance safely, please follow these steps:

Contributors: Albert Cardona, Andrew Champion, Eric Trautman, Tom Kazimiers

Features and enhancements

Selection table:

- A new option ("Append with batch color") allows to override the color and
opacity of appended skeletons with the current batch color and batch opacity.
It is deselected by default.

- Clearing the table doesn't ask for confirmation anymore. This has been removed
for consistency with other widgets and because of the now available option to
save/open skeleton lists.


New widget "Synapse Distribution Plot":

- For one or more neurons, plot distances of postsynaptic sites relative
to an axon initial segment, represented by a skeleton node that is either
computed or given via text tags.
Each row represents the inputs contributed by a presynaptic arbor.
Presynaptic arbors (rows) are sorted from more to less synapses.
Presynaptic arbors can be filtered (i.e. hidden away) by a threshold
on the number of inputs each provides, or by being listed in another
widget that has selected skeletons.
Individual synapses take by default the color of the postsynaptic arbor
(that is, the arbors added via "Append"), but can be colored as well
according to neuron colors in another widget.
Click on an individual postsynaptic site to go to the corresponding
skeleton node.
Click on the legend to jump to the skeleton node representing the
axon initial segment, relative to which all distance measurements
where made. All presynaptic neurons are available in a separate
skeleton source for each widget to pull neurons from.


New widget "Synapse Fraction":

- For one or more neurons, render a normalized stacked bar chart with
the number of synapses for/from each partner skeletons (directly either
upstream or downstream). Can group partner skeletons: add groups by
selecting a skeleton source (another widget listing skeletons).
Click on the legend to edit (title, color) the group, or remove it.
Click on the legend to go to the nearest node in the partner skeleton.
To open this new widget, open a connectivity widget and push the
button named "Open partner chart".


Settings widget:

- Persistent settings are now scoped, so that default settings may be
configured for an entire CATMAID instance ("global"), for each project
("project"), for each user ("user"), and for each user for each project
("session"). Only administrators have access to change project settings,
and only superusers have access to change global settings. This allows,
for example, administrators to set up recommended defaults for projects so
that users only need to adjust their settings where their preferences differ
from the project defaults. A selection box for which scope to adjust is
at the top of the settings widget. Persistent settings will display
information about scopes and defaults when hovering over them with the cursor.

- Administrators may also lock persistent settings so that global or project
defaults can not be changed by users.


Graph widget:

- In the main tab, a new remove button removes skeletons in the selected
skeleton source from the graph.


3D Viewer:

- Added new buttons under the "Export" tab to export connectors and synapses
as CSV. And Skeletons are now exported to CSV with a new column, the radius
at each skeleton node, and another new column for the neuron name as
rendered by the NeuronNameService (controlled by the Settings).
The connectors CSV contains, for each row, the connector ID, the treenode ID,
the skeleton ID and the relation ID, mimicking the treenode_connector table.
The synapses CSV exports two files:
1. The skeleton ID vs the neuron name
2. The list of synaptic relations of any arbor visible in the 3D Viewer,
with columns for the presynaptic skeleton ID, its treenode ID emitting
the synapse, the postsynaptic skeleton ID, and its treenode ID that
receives the synapse.


Review system:

- Creation, deletion, and edition of synapses and relations now causes related
nodes to become unreviewed. Changes to presynaptic relations or the connector
itself cause all related treenodes to become unreviewed, while changes to
postsynaptic relations affect only that specific related treenode. Changes
to other connection relations (abutment, etc.) behave like presynaptic
relations, propagating to all related treenodes.


Skeleton source subscriptions:

- So far some widgets allowed to synchronize their skeleton list along with
individual property changes. This was done through a "Sync to" selection which
pushed this information to other widgets. This has now been replaced with a
subscription option. Many widgets allow now to react to changes in skeleton
lists in other widgets. Widgets supporting this got a new small chain icon in
their title bar with which a subscription management user interface can be
shown and hidden. Widgets that contain multiple sources, like the connectivity
matrix, have one icon per source. A hover title will show which one to use for
each source.

- The UI allows to add subscriptions to multiple sources which can then be
combined through set operations. Currently sources are combined in a strict
left-associative fashion from top to bottom of the list. When "Override
existing" is checked, widget local skeletons are not used when subscriptions
are refreshed and will subsequently be removed. Otherwise, the local set is
united with the first subscription before all other subscription sources are
applied.

- The scope of individual subscriptions can be adjusted: By default each
subscription reacts to skeletons added, removed and updated in a source. The
"Filter" selection allows to listen to only one of these events. For instance,
subscribing to the active skeleton with the "Only additions" filter, allows to
collect skeletons selected active skeletons without removing them again from
a widget.

- By default, only selected skeletons are subscribed to. This means if a
skeleton becomes unselected in a source it is removed from the target widget.
If the "Only selected" checkbox is unchecked, also unselected skeletons are
added to a target widget. They are removed when skeletons are removed from the
source and their selection state is synced.

- All widget still feature the "From [Source] Append/Clear/Refresh" work-flow.
The subscription UI's "Pull" button does the same as the regular "Append"
button: a one-time sync from a source.


Miscellaneous:

- Many tracing widgets now allow a user to hide their controls. A little gear
icon in their title bar toggles their visibility.

- Rather than only specifying annotations that are used as successive
fallbacks for labeling neurons, neuron labels can now be specified as
arbitrary combinations of annotation-based components using a format string.
This is still configured in the annotations section of the settings widget.

- Volumes can now be edited when clicked on in the volume widget. This will also
display the edited volume as layer in the active stack viewer.

- Moving a node in the tracing overlay now updates its position in the database
as soon as the mouse is released, rather than waiting until the section
changes.

- Changes to the CATMAID API are now documented in `API_CHANGELOG.md`.

- A docker image of a running CATMAID instance is now available for
evaluating or developing CATMAID without needing to perform a complete
install. The latest release is available via the "stable" tag, and the
current development version is available via the "latest" tag. To try it:

docker run -p 8080:80 aschampion/catmaid

Then point your browser to http://localhost:8080. The default superuser has
username "admin" and password "admin".


Bug fixes

- API testing URL generated by Swagger (used for the API documentation at /apis)
now respect a sub-directory that CATMAID might run from.

- Fixed near active node shading in the 3D viewer.

- Pre- and post-synaptic edges in the tracing overlay now update when dragging
a related treenode.

Contributors: Albert Cardona, Andrew Champion, Tom Kazimiers


Features and enhancements

Key shortcuts / mouse operations:

- Pressing \ now brings up a dialog to go to the nearest node with a label
matching a query regex. If a node is active, this search is limited to the
active skeleton. Shift + \ cycles through matching nodes in ascending
distance order. Ctrl + \ repeats the last search regex without prompting.

- Deleting a virtual node with DELETE or Ctrl+Shift+click instead suppresses
the virtual node. Suppressed virtual nodes are skipped during review. A
setting is available to also skip suppressed virtual nodes during normal
navigation with [ and ].


Selection table:

- Batch coloring is now much quicker.

- If the batch color button is pressed a second time the color picker will not
only close but also will the batch coloring be re-applied. This won't happen
if the color picker is closed by clicking somewhere else.

- The status text line at the bottom of the table includes now the number of
selected neurons. This is helpful when a filter is active and more neurons are
selected than visible.

- Sorting for visibility columns has been removed.

- Neurons part of a Selection Table can now also be filtered based on
annotations. This can be done with the help of the input field next to the
name filter. Like with the name filter input, pressing the Enter key will
activate both filters and starting with a slash character ("/") will make the
input be treated as a regular expression (to e.g. only show neurons that are
annotated with a1 and b2, use "/a1|b2"). For now no meta-annotations are taken
into account.

- With the help of the new "Open" and "Close" buttons, skeleton lists can be
stored into JSON files as well as loaded from them. Along with each skeleton
ID, the current color and opacity is stored.


Skeleton projection layer

- With the new "Skeleton color gradient" coloring mode, the skeleton's tracing
color (currently only yellow fo the active skeleton) is used for coloring. It
fades into downstream and upstream colors, respectively (which are black and
white by default).

- Nodes can be selected by placing the mouse close to them (regardless if
displayed or not) and pressing 'g'. If no node is found in close proximity
(<50px screen space), the tracing layer's node selection is used.


Graph widget:

- Synapses can be filtered from edges based on their confidence. The confidence
threshold is applied to the minimum of the pre- and post-synaptic relation
confidences. Confidence filtering is applied prior to synapse count filtering.

- Synapse count coloring on edges can now be configured independently from edge
colors.


Volumes:

- A new widget, the Volume Manager, allows to create and list volumetric
geometries. These geometries are not yet displayable and for now only
box volumes can be created. The widget is available through a new 3D box
icon, last in the list of tracing tool widgets.

- New nodes can now be tested for intersection with a certain volume. The
Tracing section of the settings widget allows to choose a volume
against which new nodes will be tested. If they are outside of it, a
warning will be shown.


Neuron Search:

- Partial annotations as well as regular expressions are now supported for
searching. If the text entered in an 'annotated' search field matches a single
existing annotation (i.e. one that would also show up in the auto-completion),
it is used as search constraint, just like before. However, if no matching
annotation was found, the input text is treated as a regular expression on
annotation names if it starts with a slash character ('/'), otherwise it is
treated as a regular search pattern over all annotations. For instance,
finding all things that are are annotated by either A1 or B2 would look
like '/A1|B2' or requiring annotations that end on 'xyz' could be searched for
by '/xyz$'. This also works with sub-annotation queries.


3D viewer:

- Different neuron visibility modes are now available for animations. A
drop down list replaces the check-box and an option dialog is shown if
a particular animation mode requires user input. Besides the 'show one neuron
per rotation' mode, there is now also the 'Show n neurons per rotation' mode
and a mode which uses a pattern to explicitly define the visibility of
particular neurons after a particular rotation. The animation export now uses
the visibility mode selected in the 3D viewer.


Administration:

- CATMAID has been able to use DVID as a project/stack back-end and as a
image source for quite a wile now. To make the latter option easier to setup,
a new admin tool is available to create CATMAID stacks based on a DVID server.
It can be found in the "Custom views" section of CATMAID's admin interface,
labeled as "DVID stack importer". With the help of this tool on can inspect
all available repositories and data instances on a DVID server and create a
stack based on one data instance. CATMAID will make sure that all
pre-conditions are met by a stack created this way.


Miscellaneous:

- By default new widgets will now select the last widget created as skeleton
source. If wanted, this can be adjusted to the previous behavior (always
select 'Active skeleton') through the 'Auto-select widget created last as
source for new widgets' option in the settings widget.

- Multiple stacks opened through a URL can now optionally be opened in the same
stack viewer window by adding "&composite=1" to the URL.

- If an already reviewed node is moved it will now become unreviewed again.

- Links clicked in the message menu will now open in a new page.


Bug fixes

- The skeleton projection layer will now update automatically on skeleton
changes like new or removed nodes as well as splits and merges. It will also
not complain anymore if a connector was selected.

- Text rendered in the 3D viewer is now upright again (instead of upside-down).

Contributors: Albert Cardona, Andrew Champion, Tom Kazimiers


Features and enhancements

Scripting:

- The widget instance associated with the focused window can be retrieved with
the convenience function `CATMAID.front()`.


Orthogonal views and multi-channel data:

- Stack groups can be used to relate different stacks to each other within one
project, e.g. to make clear that some stacks are different orthogonal views or
different channels of the same dataset. If there are stack groups defined in a
project, they are for now available through the "Projects" menu, which
provides sub-menus for stacks and stack groups for each project. When opened,
the stacks of a channel based stack groups are added as layers to the first
stack. Ortho-view stacks are all opened in a separate stack viewer.

- If a stack group consists of the three different orthogonal views for a
dataset, the window layout is adapted automatically as soon as the stack group
is opened. The layout will be a four-pane layout in which the left half of the
screen is the XY view on top of the XZ view and the right half of the screen
is the ZY view on top of a selection table.

- Since stack group are instances of the 'stackgroup' class, they can be
referenced from within ontologies. All projects now have a 'stackgroup' class
and the relations 'has_view' and 'has_channel' created by default. They are
also created for projects that don't have them, yet.

- Stack groups can be created and managed from with the admin interface through
either the new Stack group page or while editing/creating a stack.


3D viewer:

- Skeletons can be shaded by distance from a plane through the active node. The
plane can either be a Z-plane in project space or a plane normal to the ray
from the camera to the active node.

- New "Count" button to count the number of pre- or postsynaptic sites, or the
number of treenodes tagged with a given text tag, within a distance of the
active node in the selected arbor along the cable of the arbor, or within a
given Euclidean distance for any arbor present in the 3D viewer.


Tile layer:

- WebGL rendering is now compatible with orthogonal views.

- Tiles can now be rendered either with linear pixel interpolation (previous
default behavior) or nearest neighbor interpolation. This is controlled by
the "Image tile interpolation" setting.


Graph widget:

- When growing by circles, the set of neurons added can be filtered to include
only those with annotations matching a regex.


Miscellaneous:

- A new color picker replaces the color wheel. The new control hovers over other
elements so it can be moved, has color memory slots, defaults to a smaller
size and can be resized to show input elements for different color spaces. To
save a color in a memory slot, click on the rectangle containing the small
circle next to the memory slots.

- Documentation for some HTTP API endpoints is now available from your CATMAID
server at the `/apis/` URL.


Bug fixes

Tile layer:

- Fixed a major WebGL tile layer GPU memory leak.


3D viewer:

- In orthographic mode, the correct depth ordering is now used again.


Selection table:

- Color sorting works again.


Miscellaneous:

- An error no longer appears when selecting an un-annotated skeleton while
neuron labels are configured to use a meta-annotation.

Contributors: Albert Cardona, Andrew Champion, Tom Kazimiers


Features and enhancements

Neuron Navigator:

- It is now possible to remove multiple annotations at once from a neuron. A new
column is added to annotation tables, each annotation row has now a checkbox
in its first column. A click on the de-annotate link in this column's header
or footer will remove all selected annotations from the current neuron.


Tracing:

- If a single-node skeleton is merged into another skeleton, no merge dialog is
now shown by default. All annotations of this single-node skeleton are merged
into the target skeleton without asking. This behavior can be changed to again
show a merge UI if the single-node skeleton has annotations (behavior so far)
through a new entry in the Tracing section of the settings widget.


Graph widget:

- New layout modes "dagre", "cola", "spread" and "springy". The first is based
on DAGs (directed acyclic graphs) and the last three are force-directed. To
note that "spread" will evenly layout neurons in trying to occupy as much
space as possible, and also leads to symmetric-looking graphs when rendering
multiple disconnected graphs of e.g. left and right homologous neurons. Try
it.


Connectivity matrix:

- Descending and ascending sorting is now available for all sorting modes.

- The new 'order of other' sorting mode will try to follow the column order for
rows and vice versa. If skeletons are not found in the reference dimension,
they are pushed to the end.


Selection table:

- When changing the color of a neuron, all other selected neurons can be colored
at the same time, when the new 'all selected' checkbox (right above the color
wheel) is checked.


Neuron sarch:

- The "select all" checkbox now does what it says and selects all neurons in the
results set.

- Pagination now works like in other widgets and the number of elements per page
can be adjusted.

- Annotations are not loaded by default anymore, but can be shown with the help
of the new "Show annotations" checkbox.


Miscellaneous:

- When a connector is selected, basic information about it is displayed at the
top of the window (where otherwise the neuron name is displayed).

- A neuron search result's annotation list is now kept in sync with the rest of
the widgets. If annotations change on a listed neuron or annotation, the
search query is re-done.

- If only transparency is changed in the batch color picker of the Selection
Table, the transparency alone (and not the color) of the target skeletons will
be updated. To also update the skeleton color, the color has to be changed in
the color picker, too.


Administration:

- Adding custom code to CATMAID's front end is now easier: Add file names to the
STATIC_EXTENSION_FILES array variable and have your web-server point the URL
defined in STATIC_EXTENSION_URL (defaults to /staticext/) to the folder were
those files live. CATMAID will then load those files after its own files.


Bug fixes

- Nodes are now correctly scaled in skeleton projection layers.

- Neuron navigator now updates if a skeleton is changed (e.g. due to a merge).

- 'Sync to' selections to push changes from one widget to another (e.g. 3D
viewer controlled by selection table) are now updated correctly, if a selected
target is closed.

- Changing the order of rows and columns of the connectivity matrix manually
does now work like expected.

- From within the neuron search removed annotations will now disappear again
from the search widget after they are unlinked.

- Using the CATMAID coloring scheme in the Selection Table is not random
anymore.

- CSV files exported from the Connectivity Widget now respect the table ordering
and include the target neuron names.

- Spheres that intersect in the 3D viewer (e.g. somas) don't appear broken up
anymore.

- The 3D viewer's SVG export will now correctly calculate the size of exported
spheres (e.g. soma tags).

Contributors: Albert Cardona, Andrew Champion, Tom Kazimiers


Features and enhancements

Key shortcuts / mouse operations:

- Ctrl + [ or ] now navigates to the next real (non-virtual) parent or child of
the active node, respectively.


Connectivity widget:

- Upstream and downstream partners can now be filtered by synaptic confidence
in addition to synaptic count. Synaptic confidence filtering is applied before
count filtering. Confidence is taken to be the minimum of the presynaptic
and postsynaptic connector confidence.


3D Viewer:

- Tags matching a custom regex can be shown as handle spheres in addition to
tags containing "uncertain" or "end". Note that after entering the regex in
the "View Settings" tab the viewer must be refreshed from the "Main" tab
before tag spheres are updated.

- The active node marker now resizes based on the radius of the active node.

- The active node marker can now optionally always be drawn on top of other
objects, even if it is occluded.


Miscellaneous:

- A default neuron name can now be specified in the settings widget. Similar
to annotations, the pattern "{nX}" can be used to add an automatically
incrementing number to each new neuron created, starting at X. Omitting X
will be interpreted to start from 1. This default name does not persist
between sessions.

- Neuron navigator: neuron name and annotation search are now case-insensitive.

- The client now checks every 15 minutes whether it is the same version as the
server. If not, an error dialog is shown prompting the user to refresh their
browser.

- It is now possible to show a projection of the active skeleton in the tracing
overlay. All nodes will be displayed in the current slice, but no interaction
is allowed with them. This feature can be useful to get more context on the
current location in the active skeleton. This mode can be toggled with F10 or
through a new entry in the settings (Tracing Overlay > Active skeleton
projection), where different parameters can be adjusted. The color for the
upstream and downstream part can be independently changed and various shading
modes can be selected (plain color, Z distance transparency, Strahler based
transparency or cut off).


Bug fixes

- 3D viewer: the correct synapse colors are now used when connectors are
restricted.

- If annotations are added or removed, annotation search widgets are updated
correctly. You can now search for newly created annotations without having to
open a new search widget.

- The order of the selection table is now remembered before it is refreshed.

- Connectivity widget: common partners filtering now correctly enforces that
partners are partnered with all target neurons, not just any two.

- Review widget: the skipped node warning will now only show up when in fact
more nodes have been skipped than allowed.

- Fixed a vulnerability that allowed any user with "browse" access permissions
to any project to execute arbitrary SQL statements on the CATMAID database.