Bluecellulab

-----------------

This release adds new functionality and introduces simplifications over the API and codebase.

List of changes
~~~~~~~~~~~~~~~~

* 988cb64 Remove duplicated fields in Synapse class [Anil Tuncel]
* d0ccd93 Resolve the conflict between x86_64 and BGLIBPY_MOD_LIBRARY_PATH [Pranav Rai]
* f3f5ad9 remove 3 unused hoc dependencies [Anil Tuncel]
* 3d51aa5 make add_synapses/add_minis bool instead of Optional[bool] [Anil Tuncel]
* 056a681 simplify instantiate_gids [Anil Tuncel]
* aafe0b1 add NEURON nogui flag [Anil Tuncel]
* 13315e3 add unit test for NumpyEncoder [Anil Tuncel]
* e12cd39 remove TestSSimBaseClass_v6_full_run test [Anil Tuncel]
* ba7aa46 bugfix: possibly unassigned variable in template::get_cell [Anil Tuncel]
* 8f2f7bb remove random123 simulation test, now all sims are random123 [Anil Tuncel]
* 3e79d7b remove deprecated SynapseID field of BlueConfig [Anil Tuncel]
* 43f0fb1 Add functionality to start recording spikes and retrieving them [Anil Tuncel]
* 85121ec remove duplication in Cell add/retrieve recording/s [Anil Tuncel]
* 5705115 Removed hip20180219_1 from test_gpfs. The circuit no longer exists [Anil Tuncel]
* 6e9dad8 implement conductance clamp for shotnoise and OU stimuli [Anil Tuncel]
* c6ac57b replace deprecated usage of np.float -> float [Anil Tuncel]
* 6837d64 simplify handling of morphologies from blueconfig to hoc [Anil Tuncel]
* 4d99a15 address type ambiguity in pandas method usage [Anil Tuncel]
* d5f2cf7 switch to tox>4.0.0 INFRA-8113 [Anil Tuncel]
* 1c80557 Resolve NSETM-2070 "Update simulation config" [Eduard Subert]
* 644e5f5 add v6 mark to test sonata proxy [Anil Tuncel]
* 0ecea78 fix return type annotation of get_cell_properties [Anil Tuncel]
* 6a77ac0 update changelog to be sphinx compatible rst [Anil Tuncel]
* Add add_connections in place of removed add_section to simulation config [Eduard Subert]

----------------

The business logic of BGLibPy no longer makes any assumptions on bluepy or bluepy-configfile or snap (in the future) API and data structures. BGLibPy has an interface to abstract circuit access logic. The implementation code (written using bluepy (or snap in the future)) adheres to that interface (dependency inversion).

Rationale
~~~~~~~~~~~~

1. No module except for circuit_access and tests should know about the bluepy API or data structures.
2. Separate the information coming from config file from the information coming from circuit. All config default values shall therefore be defined in one place and in nowhere else in the code.
3. Add type annotations to the circuit access functions to be sure bluepy or snap implementations will follow the same signatures.

E.g. if one is returning numpy.ndarray, the other should not return the same values in a list. Assure this through static type checking.

API Changes
~~~~~~~~~~~~

* ssim constructor: rename `blueconfig_filename` argument into `simulation_config`
* instantiate_gids: Don't take 3 different arguments named `projection`, `projections`, `add_projections`.

Use only `add_projections`. This avoids confusion and simplifies the downstream logic that deals with projections and synapses.

Signature:

`add_projections: bool | List[str] = False`

Docstring:


add_projections:
If True, adds all of the projection blocks of the
BlueConfig. If False, no projections are added.
If list, adds only the projections in the list.



Performance
~~~~~~~~~~~~

* Loop interchange in add_stimuli. Before it was iterating for each cell, for each stimuli.

The problem here was that stimuli entries are defined in the blueconfig and they were loaded from the same blueconfig file **n_cell** times.
With the loop interchange, stimuli entries are loaded **only one time** and added to the cells.

* return Set[int] int gell_cell_ids to avoid searching. To figure out if a cell belongs to a target, the cell's id was searched in all target ids which can be a very long list of files. With this change the search becomes hash and takes constant time.


Removed
~~~~~~~~~~~~

* Out of date blueconfig parser functions. The up-to-date parser is available at bluepy-configfile
* Remove unused methods returning unavailable bb5 paths

Simplification
~~~~~~~~~~~~~~

* remove condition nestedness by 1 level in _add_stimuli
* avoid iterating all config for StimulusInject, iterate only StimulusInject blocks using bluepyconfigfile's api

Tests
~~~~~~~~~~~~

* modify coveragerc to support concurrency

----------------

Implement a layer responsible of circuit and simulation access.

Why now?

- Required before fully supporting the sonata specification (including reports and config files).
- Neurodamus now supports full-sonata simulations, soon there will be simulations without blueconfig and out.dat
- Various custom stimuli in cell/injector module require access to a subset of Sonata Node properties (BGLPY-103).

Passing each Node property required by a single stimuli to every cell object can no longer be the solution.
- Abstracting away the bluepy layer enables supporting bluepy alternatives e.g. snap in the future.

snap: https://github.com/BlueBrain/snap

Changes

A summary of all changes introduced are listed below.

Other design changes
- add SonataProxy to interface Cell with Sonata
- create a validate module for circuit/simulation config validation.
- Separate BlueConfig validation and object creation (e.g. synapse or connection object creation)
- create neuron_globals in simulation module to set global NEURON simulator variables

Documentation changes

- use NEURON when referring to the simulator.

Deprecates

- storing and parsing of neuronconfigure_entries that are no longer in the standard.
- remove addCell method deprecated 9 years ago
- remove ssim wrappers of cell/injector functions

Performance

- separate parser and validator in minis single vesicle to avoid validating the same BlueConfig multiple times
- call validator only once in SSim's constructor
- in `get_gids_of_mtypes`, use `bluepy_circuit.cells.ids` instead of `bluepy_circuit.cells.get` to avoid creating and retrieving `pd.DataFrame` objects.

Debugging

- add __repr__ for RNGSettings useful in debugging & logging

Testing

- Testing of circuit/simulation properties are simpler, with no SSim and gpfs dependency.
- More fine grained code coverage through unit tests, (catches some edge cases that were missed in the large functional tests)

Bug fix

- get_gids_of_mtypes and get_gids_of_targets to return set to avoid duplicated gids coming from multiple targets or mtypes

Typing

- annotate config filepath types accepted by bluepy-configfile
- use type annotations to avoid docstrings becoming out-of-date
- add pandas stubs

Future work

- some bluepy objects are still accessed from other modules e.g. Synapses.

Decouple the bluepy dependency from other modules.
E.g. If pop_ids is needed, don't pass the bc object, pass pop_ids only.

----------------
- PERFORMANCE: ~22.4x speedup in add_synapses on large simulations
Source: https://bbpgitlab.epfl.ch/cells/bglibpy/-/merge_requests/48
- PERFORMANCE: ~160x speedup in _intersect_pre_gids on large simulations
Source: https://bbpgitlab.epfl.ch/cells/bglibpy/-/merge_requests/48
- Include tests in coverage. [Anil Tuncel]
- Support Simplify AST for subscription bpo-34822. [Anil Tuncel]
- Add mypy to tox:lint environment. [Anil Tuncel]
- Add self.hsynapse.gmax_NMDA in glusynapse helper BBPBGLIB-809. [Anil
Tuncel]
- Representing synapse description as a multiindex dataframe. [Anil
Tuncel]
- Support debugging gids with prcellstate. [Sirio Bolaños Puchet]
- Performance: lazily evaluate string expressions used in logging. [Anil
Tuncel]
- Addressing some of pylint warnings. [Anil Tuncel]
- Singleton RNG: __init__ can be run multiple times on the same
instance. [Anil Tuncel]

this is more intuitive for the end users
- Add globalseed,synapseseed,ionchannelseed,stimulusseed,minisseed to
bglibpy.neuron.h. [Anil Tuncel]
- Have a limited neuron interpreter to control neuron code evaluation.
[Anil Tuncel]
- FIX glusynapse parser to read tau_d_AMPA parameter. [Anil Tuncel]
- Remove unused cell attributes: netstims and pre_spiketrains. [Anil
Tuncel]
- Refactor: use set comprehension in get_pre_gids. [Anil Tuncel]
- Extract cell's plotting functionality to cell.plotting.PlottableMixin.
[Anil Tuncel]
- Performance: avoid materialising mecombo parameters in ssim class.
[Anil Tuncel]
- Fixing Ca++ dep. U_SE scaling of GluSynapse. [Ecker András]
- Merge_pre_spike_train to always return a dict. [Anil Tuncel]
- Use to_numpy for pandas to numpy conversion. [Anil Tuncel]

https://pandas-docs.github.io/pandas-docs-travis/whatsnew/v0.24.0.html\#accessing-the-values-in-a-series-or-index
- Add support for SYNAPSES__minis_single_vesicle in conditions block.
[Anil Tuncel]
- Parse SYNAPSES__init_depleted global condition parameter. [Anil
Tuncel]
- Speeding up _parse_outdat() [András Ecker]
- Tox run the v6 tests isolated. [Anil Tuncel]
- Revert "DEBUG: run only test_relative_shotnoise in v6 and check rms in
assert" [Anil Tuncel]

This reverts commit f0259f6397dd55ea3a515a2c163f435d4bb9ffdc.
- DEBUG: run only test_relative_shotnoise in v6 and check rms in assert.
[Anil Tuncel]
- Add test comparing relative shot noise to Neurodamus. [SBP]
- Update test_add_replay_relative_shotnoise. [SBP]
- Fix ssim to use new parameters in add_replay*shotnoise. [SBP]
- Take section and segx as parameters in add_replay_shotnoise. [Anil
Tuncel]
- Unit tests to check edge cases in shotnoise. [Anil Tuncel]
- Add unit test for test_add_replay_relative_shotnoise. [Anil Tuncel]
- Pass section and segx as argument to add_replay_relative_shotnoise.
[Anil Tuncel]
- Use 0 as default value for stim count in
add_replay_relative_shotnoise. [Anil Tuncel]
- Add unit test test_get_relative_shotnoise_params. [Anil Tuncel]
- Add unit test for add_shotnoise_step. [Anil Tuncel]
- Use 0 as default value for shotnoise_stim_count to prevent type error.
[Anil Tuncel]
- Implement shot noise stimuli. [SBP]
- Refactor Cell class: add injector, template, section_distance modules.
[Anil Tuncel]

----------------
- Docs: update CHANGELOG.rst. [Anil Tuncel]
- Docs: update bglibpy and neurodamus repository urls. [Anil Tuncel]
- Docs: updated documentation link on package json. [Anil Tuncel]
- Glusynapse support bglpy-86. [Anil Tuncel]
- TOX: use v5 v6 thal tests in coverage. [Anil Tuncel]
- Simplify the syn_description_dict. [Anil Tuncel]
- Install neurodamus and mods from gitlab. [Anil Tuncel]
- Refactor: separate gabaab and ampanmda functions as methods for
Synapse. [Anil Tuncel]
- Require bluepy[bbp], let it handle the bbp dependencies e.g. brion.
[Anil Tuncel]

drop brion>=3.3.0 dependency

require bluepy[bbp], let it handle the bbp dependencies e.g. brion

previously brion>3.3.0 is handled here to have support of a wide range of bluepys
- Bug fix: ssim.get_voltage_trace contains ForwardSkip voltages
BGLPY-94. [Anil Tuncel]
- Addressing setup.py Warning: 'keywords' should be a list, got type
'tuple' [Anil Tuncel]
- Require python>=3.7 since morphio drops py36. [Anil Tuncel]
- Refactor synapse parameters: turn synapses into a module. [Anil
Tuncel]
- Solve pre_spike_train concatenation problem. [Anil Tuncel]
- Cleanup: remove unnecessary code. [Anil Tuncel]
- Add gitlab tokens required to upload documentation. [Anil Tuncel]
- Use a simpler implementation (using metaclasses) for singleton. [Anil
Tuncel]
- Update bluepy dependencies after BLPY-267. [Anil Tuncel]

----------------
- Isolate the unit v5,v6, thal tests. [Anil Tuncel]
- Upload-docs not to depend on bglibpy's importer. [Anil Tuncel]

i.e. no neurodamus is needed therefore no gitlab access permission is needed by ci
- Use upload_docs tox environment instead of bbp-nse-ci. [Anil Tuncel]
- Update readme & documentation. [Anil Tuncel]
- Add version.py that reads from versioneer. [Anil Tuncel]
- Update bglibpy to be compatible with the gitlab ci pipelines. [Anil
Tuncel]
- Update README.rst. [Werner Van Geit]
- Added initial .gitlab-ci.yml. [Anil Tuncel]
- Deleted .gitreview. [Anil Tuncel]
- Revert "Initial commit" [Anil Tuncel]

This reverts commit e5fa4a7bf820b82a948ac009e49ec78e90b7ca74.
- Initial commit. [Tharayil Joseph]
- Merge "add edge_id attribute to Synapse" [Werner Van Geit]
- Add edge_id attribute to Synapse. [Anil Tuncel]
- Tox.ini added cmake as dep. [Anil Tuncel]
- .install_neuron: print cmake output to console. [Anil Tuncel]
- .install_neuron.sh removed. [Anil Tuncel]

it was used for debug purposes
- Using cmake for neuron installation. [Anil Tuncel]
- Check forward_skip value to be positive BGLPY-85. [Anil Tuncel]

patch1: applying Werner's feedback
- Merge changes from topic 'ais' [Anil Tuncel]

* changes:
enable spike detection at AIS BGLPY-83
replace connect2target hoc function with python
- Enable spike detection at AIS BGLPY-83. [Anil Tuncel]

Patch 2
* added helper functions to record and read from ais
* added test to compare voltages at AIS (using sscx sim)

Patch 3
* check for bluepy's soma_report.get_gid response BLPY-259
there are sometimes duplicated columns

Patch 4
* explicitly delete the cell object for python's reference counter
* set ecord_dt 0.1
* check for length of voltages
- Replace connect2target hoc function with python. [Anil Tuncel]

Patch 2: removed M. Hines magic comment
Patch 3: spikedetection threshold as an argument with a default val
Patch 4: documentation fix
Patch 5: pass Connection.spike_threshold to create_netcon_spike
Patch 6: fix function call in create_netcon_spikedetector
- Improve loading time of synapses. [Sirio Bolaños Puchet]

Two efficiency improvements:
+ check sanity of connection entries only once at the beggining,
instead of for every synapse
+ cache what GIDs belong to a target, so that matching GIDs to
connection entries proceeds much faster
- Updated tutorial with an example single cell sim (without network)
[Anil Tuncel]

* changelog update
- Removed unreachable bluepy<=0.16.0 branch. [Anil Tuncel]

Patch 2: setup.py bluepy remove bbp and add brion

it's unreachable since setup.py assumes bluepy>2.1.0
- Added numpy and matplotlib dependencies. [Anil Tuncel]
- Replace methodtools with cachetools to reduce dependencies. [Anil
Tuncel]

as methodtools depend on wirerope and inspect2 BGLPY-80
Patch 2: remove the cache of is_cell_target method
since it's cheap
- Remove extra[bbp] since brion is in install_requires. [Anil Tuncel]

*PATCH1*: bluepy>=2.1.0.dev6 -> bluepy>=2.1.0
- Drop deprecated bluepy.v2 subpackage. [Anil Tuncel]
- Merge "Merge branch 'warnings'" [Anil Tuncel]
- Merge branch 'warnings' [Anil Tuncel]
- Merge "error message made more informative" [Anil Tuncel]
- Error message made more informative. [Anil Tuncel]

ignore_populationid_error=True is mentioned
- Changelog update upon updating the tag. [Anil Tuncel]
- Use methodtools lru cache to prevent memory leak. [Werner Van Geit]
- Apply the sonata spike report update [BLPY-244] [Anil Tuncel]

* apply renaming of brian->brion
- Added FAQ page with MPT ERROR: PMI2_Init. [Anil Tuncel]
- Documentation update. [Anil Tuncel]

* mention missing parameters in docstring
* refer to the jupyter notebook in insilico-cookbook
in the tutorial section.
* add changelog to sphinx.
* PATCH 2: added docs/source/changelog.rst
- Added option to use hoc with AIS_scaler, aligned with
https://bbpcode.epfl.ch/code/#/c/52044/ [arnaudon]

* rebase master
* added changelog entry
* rebase master at ffc293a bluepy v1.0.0 integration
- Bluepy v1.0.0 integration. [Anil Tuncel]

* PATCH 2: BLPSynapse.POST_SEGMENT_ID for newer bluepy
* PATCH 3: Added changelog entry
- Updated docstring for Synapse.synid to contain tuple idx info. [Anil
Tuncel]
- Apply pep8 code style with E501,W504,W503,E741 ignored. [Anil Tuncel]

Tox & Jenkins plans are updated accordingly
- Read synapse locations from SONATA field and round synapse delays to
timestep. [Sirio Bolaños Puchet]

* style: line lengths decreased to 80
- Remove bluepy 'sonata' extra in version >=0.16.0 [BGLPY-78] [Anil
Tuncel]

* also remove the explicit h5py<3.0.0 dependency since bluepy handles it
- Merge "Add support for MinisSingleVesicle, SpikeThreshold, V_Init,
Celsius" [Anil Tuncel]
- Add support for MinisSingleVesicle, SpikeThreshold, V_Init, Celsius.
[Sirio Bolaños Puchet]

* Added a gpfs test
* added a custom exception
* rebased master branch
* used get_mainsim_voltage with t_start, t_stop, t_step parameters in the test
* CHANGELOG updated
- Tests remove unnecessary ssim object creations. [Anil Tuncel]
- Use absolute paths in blueconfigs [BLPY-178] [Anil Tuncel]

* adapted the tests accordingly
- Remove python27 from jenkins plan. [Anil Tuncel]
- Introducing t_start, t_stop, t_step parameters for
get_mainsim_voltage_trace. [Anil Tuncel]

The motivation is due to the performance.
Retrieving the mainsim voltage using bluepy on large simulations takes very long.
With the use of bluepy api v2 this change enables retrieving only a section of voltage rather than the entire simulation voltage.
- Use h5py<3.0.0. [Anil Tuncel]

h5py 3.0.0 is parsing the dtype (previously parsed as str) as bytes.
There may be other changed datatypes as well.
Until a long-term solution can be found, it's best to pin the version down.
- Pin version of pyrsistent in tox. [Werner Van Geit]
- Moved download = true in tox.ini. [Werner Van Geit]
- Trying to avoid pinning virtualenv. [Werner Van Geit]
- Removed pyrsistent dependency since it became a dependency of bluepy.
[Anil Tuncel]

* in tox use download=true to get the recent pip that comes with a new dependency resolver
* remove unused pandas dependency
* removed the old bluepy-configfile-0.1.2.dev1 version dependency (bluepy already has bluepy-configfile>=0.1.11)
- Setting RNGSettings.mode to automatically set neuron.h.rngMode. [Anil
Tuncel]

This implementation is based on bglpy-68 issue.
The purpose is to behave the same as neurodamus does.
* Made RNGSettings a singleton class since it's dealing with a global variable.
- Make sure targets used by _evaluate_connection_parameters exist. [Anil
Tuncel]
- Noisestim_count to be incremented whether or not it's applied to the
gid. [Anil Tuncel]

see bglpy69 for further info
- Merge branch 'master' of ssh://bbpcode.epfl.ch/sim/BGLibPy. [Werner
Van Geit]
- Fix synapse_detail error when add_minis is False and synapse_detail>0.
[Anil Tuncel]
- Fix idiotic warning thrown by python lately. [Werner Van Geit]