Nengo-dl

-------------------------

**Added**

- NengoDL will now automatically use a rate-based approximation to compute the
gradient for spiking neuron types, if one is known (no more need to manually
swap neuron types for training and inference).
- Added ``nengo_dl.configure_settings(inference_only=True)`` option, which will
build the network in inference-only mode. This will slightly improve the
inference speed of the simulation, but the network will not be trainable.
- Added ``nengo_dl.configure_settings(lif_smoothing=x)`` option, which will
control how much smoothing is applied to the LIF function during gradient
calculations (if any).
- Added `documentation <https://www.nengo.ai/nengo-dl/config.html>`__ on the
various NengoDL config options.
- Added better validation for TensorNode output when ``size_out != None``
(`51 <https://github.com/nengo/nengo-dl/issues/51>`_)

**Changed**

- More informative error message if the user tries to pass target values for
a probe that isn't used in the objective function.
- Switched to ADD_N gradient accumulation (from TREE); this will increase
the memory usage during training, but improve performance.
- Revert to ``Timeline`` profiling method. ``tf.profiler`` can produce
incorrect output, and isn't maintained any more
(https://github.com/tensorflow/tensorflow/issues/15214#issuecomment-382442357)
- Reduce memory usage during training by caching temporary variables used
when computing ``ScatterUpdate`` gradient.
- Increase minimum TensorFlow version to 1.4.0.
- Increased minimum NumPy version to 1.12.1 (required by TensorFlow)
- Sort write signals as well as reads during graph optimization (encourages
tighter partitioning, which can improve training/inference speed).
- Moved ``configure_settings`` from ``utils.py`` to ``config.py``.

**Fixed**

- Fixed a bug where
``nengo_dl.dists.VarianceScaling(..., distribution="normal")`` did not
respect the seed if one was given.

**Deprecated**

- The ``Simulator(dtype=...)`` argument has been deprecated; use
``nengo_dl.configure_settings(dtype=...)`` instead. Will be removed in
1.3.0.

---------------------

**Added**

- The default TensorFlow Session is now set to the underlying Simulator session
within the Simulator context.
- Added CLI for benchmarks.py
- Added ``sim.freeze_params`` tool, to more easily extract model parameters for
reuse in different Simulators.
- Added `documentation on saving and loading model parameters
<https://www.nengo.ai/nengo-dl/simulator.html#saving-and-loading-parameters>`_.
- Added `Spaun <https://science.sciencemag.org/content/338/6111/1202.full>`_
example in ``benchmarks.py``

**Changed**

- Move ``tensorflow-gpu`` installation check to Simulator init, and only apply
if ``device=None``.
- Switched to ``pylint`` for style checks.
- TensorFlow INFO-level log messages are now disabled by default on import
- All previous releases now tracked in documentation
- Updated spiking MNIST example to simplify and improve performance.
- Passing unknown configuration options to ``nengo_dl.configure_settings``
will now give a more explicit error message.
- Improved speed of parameter fetching though ``get_nengo_params``
- Raise a warning if user tries to train a network with non-differentiable
elements (requires ``tensorflow>=1.9.0``)
- Improved accuracy of ``SoftLIFRate`` implementation for small values (`45
<https://github.com/nengo/nengo-dl/pull/45>`_)
- Simplified how ``TensorSignals`` are loaded into the TensorFlow graph

**Fixed**

- Better handling of Simulator errors not associated with a specific op (fixes
`41 <https://github.com/nengo/nengo-dl/issues/41>`_)
- Fixed node outputs changing after simulator is built (fixes `4
<https://github.com/nengo/nengo-dl/issues/4>`__)
- Fixed some broken cross references in the documentation
- Fixed several edge cases for ``get_nengo_params``; don't use trained gains
for direct neuron connections, error raised if ``get_nengo_params`` applied
to an Ensemble with Direct neurons
- Compatible with ``tensorflow==1.9.0`` release
- Fixed bug in ``nengo_dl.configure_settings(session_config=...)`` when passing
a pre-build model to the Simulator instead of a Network
- Fixed TensorFlow version comparisons for 1.10.0

**Deprecated**

- ``Simulator.trange`` argument ``dt`` has been deprecated (replaced with
``sample_every``, see https://github.com/nengo/nengo/pull/1384)

**Removed**

- Removed ``nengo_dl.DATA_DIR`` constant
- Removed ``benchmarks.compare_backends`` (use
``whitepaper2018_plots.py:compare_backends`` instead)
- Removed ``ghp-import`` dependency

--------------------

**Added**

- User can now directly specify the output error gradient, rather than using
targets/objective (useful for when you have some external process for
computing error that is not easy to implement as an objective function).
See `the documentation
<https://www.nengo.ai/nengo-dl/v1.0.0/training.html#objective>`__ for details.
- Added `NengoDL white paper <https://arxiv.org/abs/1805.11144>`_

**Changed**

- Extra requirements for documentation/testing are now stored in ``setup.py``'s
``extra_requires`` instead of ``requirements-*.txt``. For example, instead
of doing ``pip install -r requirements-test.txt``, instead use
``pip install nengo-dl[tests]`` (or ``pip install -e .[tests]`` for a
developer installation).
- Improved efficiency of PES implementation

**Removed**

- Removed ``sphinxcontrib-versioning`` dependency for building documentation

-------------------

**Added**

- Added ``sim.get_nengo_params`` function to more easily extract
model parameters for reuse when building different models.
- Added ``Simulator(..., progress_bar=False)`` option to disable the progress
information printed to console when the network is building.
- TensorFlow session config options can now be set using
``nengo_dl.configure_settings`` (e.g.,
``nengo_dl.configure_settings(session_config={"gpu_options.allow_growth": True})``)
- The signal sorting/graph simplificaton functions can now be configured
through ``nengo_dl.configure_settings``
- Added ``extra_feeds`` parameter to ``sim.run/train/loss``, which can be
used to feed Tensor values directly into the TensorFlow session

**Changed**

- Improved speed of PES implementation by adding a custom operator.
- Renamed project from ``nengo_dl`` to ``nengo-dl`` (to be more consistent with
standard conventions). This only affects the display name of the project
on PyPI/GitHub, and the documentation now resides at
https://www.nengo.ai/nengo-dl/; there are no functional changes to user code.
- Minor efficiency improvements to graph planner
- Avoid using ``tf.constant``, to get around TensorFlow's 2GB limit on graph
size when building large models

**Fixed**

- Checking ``nengo_dl`` version without ``nengo`` installed will no longer
result in an error.
- Updated progress bar to work with ``progressbar2>=3.37.0``
- Updated PES implementation to work with generic synapse types
(see https://github.com/nengo/nengo/pull/1095)
- Fixed installation to work with ``pip>=10.0``
- Fixed bug when using a TensorNode with a ``pre_build`` function and
``size_in==0``

---------------------

**Added**

- Added TensorFlow implementation for ``nengo.SpikingRectifiedLinear`` neuron
type.

**Changed**

- Optimizer variables (e.g., momentum values) will only be initialized the
first time that optimizer is passed to ``sim.train``. Subsequent calls to
``sim.train`` will resume with the values from the previous call.
- Low-level simulation input/output formats have been reworked to make them
slightly easier to use (for users who want to bypass ``sim.run`` or
``sim.train`` and access the TensorFlow session directly).
- Batch dimension will always be first (if present) when checking model
parameters via ``sim.data``.
- TensorFlow ops created within the Simulator context will now default to
the same device as the Simulator.
- Update minimum Nengo version to 2.7.0

**Fixed**

- Better error message if training data has incorrect rank
- Avoid reinstalling TensorFlow if one of the nightly build packages is already
installed
- Lowpass synapse can now be applied to multidimensional inputs
- TensorNodes will no longer be built into the default graph when checking
their output dimensionality.

**Removed**

- Removed ``utils.cast_dtype`` function

-------------------------

**Added**

- The ``SoftLIFRate`` neuron type now has an ``amplitude`` parameter, which
scales the output in the same way as the new ``amplitude`` parameter in
``LIF``/``LIFRate`` (see `Nengo PR 1325
<https://github.com/nengo/nengo/pull/1325>`_).
- Added ``progress_bar=False`` option to ``sim.run``, which will disable the
information about the simulation status printed to standard output (`17
<https://github.com/nengo/nengo-dl/issues/17>`_).
- Added progress bars for the build/simulation process.
- Added truncated backpropagation option to ``sim.train`` (useful for reducing
memory usage during training). See `the documentation for details
<https://www.nengo.ai/nengo-dl/v0.6.0/training.html#truncation>`__.

**Changed**

- Changed the default ``tensorboard`` argument in ``Simulator`` from ``False``
to ``None``
- Use the new `tf.profiler
<https://github.com/tensorflow/docs/blob/r1.14/site/en/api_docs/python/tf/profiler/profile.md>`_
tool to collect profiling data in ``sim.run_steps`` and ``sim.train`` when
``profile=True``.
- Minor improvements to efficiency of build process.
- Minor improvements to simulation efficiency targeting small ops
(``tf.reshape/identity/constant``).
- Process inputs are now reseeded for each input when batch processing (if seed
is not manually set).
- Users can pass a dict of config options for the ``profile`` argument in
``run_steps``/``train``, which will be passed on to the TensorFlow
profiler; see the ``tf.profiler`` documentation for the `available options
<https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/profiler/g3doc/options.md>`_.

**Removed**

- Removed ``backports.print_function`` dependency

**Fixed**

- Fixed a bug where input nodes that were only read as a view were not
feedable
- Updated ``tensorflow-gpu`` installation check
- Improved numerical stability of ``LIFRate`` gradients (`26
<https://github.com/nengo/nengo-dl/issues/26>`_)
- Added more informative error message when data is provided with fewer items
than ``sim.minibatch_size`` (`30 <https://github.com/nengo/nengo-dl/issues/30>`_)