===========================
This release marks a major change in the structure of SimPEG, and it all started
with this,
* 562: What's the problem with ``Problem``?
We hope to answer that question with this release!
This release will break backward compatibility as many modules and classes have
been renamed. Check out the Examples and Tutorials (which have also been improved) to see how it's done.
We are also only supporting Python versions >=3.6 at this time. We have dropped all
testing and development on Python versions previous to this (especially
2.7).
Highlights
==========
* No more ``Problem``-s, only ``Simulation``-s
* ``Data`` is important
* PEP 8 renaming
* Dask parallelism
* Resistivity Simulation improvements
Simulation
==========
We will refer to this update as the simulation update, and there are a few large
changes that we will attempt to describe here.
We (the developers) believed that there were some significant challenges with the
overall structure of the SimPEG framework, which all revolved around constructing
a forward simulation by ``pair``-ing a ``Survey`` and a ``Problem``. The ``Survey``
handled things like sources, receivers, and the ``Problem`` handled the physics
engine of the forward simulation. These two items had to be created separately,
then ``pair``-ed afterwards for both the ``Problem`` to be able to use sources, and for
the ``Survey`` to predict data. We found that this structure made it difficult to
interface with other packages and was also generally difficult to explain. Also,
field data was then attached to the ``Survey`` class.
These changes within this section are also the ones which will require the most
changes to code (which are still hopefully small).
The ``Simulation`` class
------------------------
``Problem`` has been renamed to ``Simulation``
We decided to refactor this code into something a little more understandable.
The ``Simulation`` class is now the workhorse of the SimPEG forward simulations.
It handles things like modeling fields and projecting those fields to the data
locations defined by its ``survey``. The ``Survey`` class is much lighter weight,
now only handling the sources and receivers. Also a single ``survey`` can now be
attached to many ``Simulation``-s.
Previously we had something like,
survey = DC.Survey(srcList)
prob = DC.Problem3D_CC(mesh, rhoMap=mapping)
prob.pair(survey)
Compute the fields from `prob`
fields = prob.fields(model)
And predict data using the `survey`
dpred = survey.dpred(model, f=fields)
Now,
survey = resistivity.Survey([source_list])
sim = resistivity.Simulation3DCellCentered(
mesh,
survey=survey,
rhoMap=mapping
)
Compute the fields from `sim`
fields = sim.fields(model)
Predict data also using `sim`
dpred = sim.dpred(model, f=fields)
See? The ``Simulation`` handles predicting data. This change will also make it
easier to interface with external codes for inversion purposes, as all that is
needed to be defined to use a ``Simulation`` for an ``InvProblem``, is
``sim.dpred``, ``sim.Jvec`` and ``sim.Jtvec``.
Please see the documentation for the `SimPEG.simulation.BaseSimulation` class
as well as the individual methods' ``Simulation``-s, for a detailed description of
arguments, but largely it accepts the same arguments as the ``Problem`` class,
but now also requires a ``Survey`` to be set.
The ``Data`` class
------------------
Previously, field data would also live within the ``Survey`` class. Which was not
only confusing, but placed the importance on the wrong component. When inverting geophysical
data, we are concerned with the data. Thus we would like to enforce this importance
by making data live in a dedicated ``Data`` class. This ``Data`` class can act like a smart
dictionary to grab data associated with a specific source, receiver combination.
More importantly, this ``Data`` class is where we store information related to observed
data and its errors. This class started in the ``SimPEG.Survey`` module, but has
now been moved into its own new module ``SimPEG.data``. See the documentation for
the `SimPEG.data.Data` for all of the details.
Previously,
Add data to the survey
survey.dobs = dobs
survey.std = 0.05 a 5% relative error
survey.eps = 1.0E-6 a noise floor
Now,
Create a data object
data = data.Data(dobs=dobs, relative_error=0.05, noise_floor=1e-6)
You might also notice that we changed the name of the terms used to construct
the standard deviation. See issue 846.
Previously ``survey.std`` represented an error that was relative to the absolute
value of the data. The name of this term is misleading, as it is not actually
the classic statistical standard deviation.
Previously the uncertainty was constructed as:
uncertainty = survey.std * np.abs(survey.dobs) + survey.eps
We now have updated the names to be clearer and more in line with what we would
naturally expect, which is accessed from `data.standard_deviation`. The value
that is returned from this is now defined as:
data.standard_deviation = (
data.relative_error * np.abs(data.dobs) +
data.noise_floor
)
You can also directly set the value of ``data.standard_deviation`` if you prefer
to work with that quantity.
data.standard_deviation = 0.01
This ``Data`` class is now also the object that is returned from:
data = sim.make_synthetic_data(
m, relative_error=0.05, noise_floor=0.0, f=None, add_noise=True
)
The ``DataMisfit`` class
------------------------
Previously, because the ``Survey`` class handled predicting data at the receivers,
and it also had knowledge of the observed data and its noise, we constructed the
data misfit measure using only the survey. Now we have specifically broken this
piece up into a forward ``Simulation`` object, and a ``Data`` object. This mimics
the definition of the classic data misfit measure.
<img width="198" alt="image" src="https://user-images.githubusercontent.com/16258927/83070396-a8d6b200-a028-11ea-9a7c-7600526ec378.png">
The ``Simulation`` class handles the forward operation, <img width="14" alt="image" src="https://user-images.githubusercontent.com/16258927/83070609-f81ce280-a028-11ea-87f6-2a2a938860e0.png">, and
the ``Data`` class handles the noise, <img width="108" alt="image" src="https://user-images.githubusercontent.com/16258927/83070437-b5f3a100-a028-11ea-8957-61fc6801a9fe.png">, and the observed data, <img width="26" alt="image" src="https://user-images.githubusercontent.com/16258927/83070484-c7d54400-a028-11ea-8c68-801029afcaf6.png">.
See the documentation for the `SimPEG.data_misfit.L2DataMisfit` for all of the details.
Previously,
Survey knows how to predict data, knows the observed data,
and its standard deviation
dmis = DataMisfit.l2_DataMisfit(survey)
Now,
Create a data misfit
The data class now knows the observed data and its standard deviation.
The simulation knows how to create data from a model.
dmis = data_misfit.L2DataMisfit(simulation=sim, data=data)
Dask
====
We have begun a concerted effort to incorporate ``dask`` as a means to allow SimPEG
to scale to larger computers (and take advantage of parallelism). Checkout the
``dask`` docs at https://docs.dask.org/en/latest/.
This feature is experimental at the moment and can be toggled on like so,
import SimPEG.dask
which will then enable parallel operations for a few modules. It will specifically
replace these functions with ``dask`` versions,
* ``SimPEG.potential_fields.BasePFSimulation.linear_operator``
* ``SimPEG.potential_fields.magnetics.Simulation3DIntegral.getJtJdiag``
* ``SimPEG.potential_fields.gravity.Simulation3DIntegral.getJtJdiag``
* ``SimPEG.electromagnetics.static.resistivity.simulation.BaseDCSimulation.getJ``
* ``SimPEG.electromagnetics.static.resistivity.simulation.BaseDCSimulation.getJtJdiag``
* ``SimPEG.electromagnetics.static.induced_polarization.simulation.BaseDCSimulation.getJ``
* ``SimPEG.electromagnetics.static.induced_polarization.simulation.BaseDCSimulation.getJtJdiag``
Changelog
=========
As can be expected, there are many changes in this release, and we hope to identify
most of them here (or at least point you in the right direction).
Renamed Modules
---------------
We have taken steps to rename the modules of SimPEG to a more PEP 8 friendly
system. The previous locations do not exist.
* ``EM`` → ``electromagnetics``
* ``EM.FDEM`` → ``electromagnetics.frequency_domain``
* ``EM.TDEM`` → ``electromagnetics.time_domain``
* ``EM.NSEM`` → ``electromagnetics.natural_source``
* ``EM.Static`` → ``electromagnetics.static``
* ``EM.Static.DC`` → ``electromagnetics.static.resistivity``
* ``EM.Static.DC.Utils`` → ``electromagnetics.static.resistivity.utils``
* ``EM.Static.IP`` → ``electromagnetics.static.induced_polarization``
* ``EM.Static.SIP`` → ``electromagnetics.static.spectral_induced_polarization``
* ``EM.Static.Utils`` → ``electromagnetics.static.utils``
* ``EM.Utils`` → ``electromagnetics.utils``
* ``VRM`` → ``electromagnetics.viscous_remanent_magnetization``
* ``FLOW`` → ``flow``
* ``SEIS`` → ``seismic``
* ``PF`` → ``potential_fields``
* ``PF.Gravity`` → ``potential_fields.gravity``
* ``PF.GravAnalytics`` → ``potential_fields.gravity.analytics``
* ``PF.Magnetics`` → ``potential_fields.magnetics``
* ``PF.MagAnalytics`` → ``potential_fields.magnetics.analytics``
* ``Utils`` → ``utils``
* ``DataMisfit`` → ``data_misfit``
* ``Directives`` → ``directives``
* ``Fields`` → ``fields``
* ``InvProblem`` → ``inverse_problem``
* ``Inversion`` → ``inversion``
* ``Maps`` → ``maps``
* ``Models`` → ``models``
* ``ObjectiveFunction`` → ``objective_function``
* ``Optimization`` → ``optimization``
* ``Props`` → ``props``
* ``Survey`` → ``survey``
* ``Problem`` → ``simulation``
Also generally modules with the following names have changed:
* ``METHOD.SurveyMETHOD`` → ``method_name.survey``
* ``METHOD.SrcMETHOD`` → ``method_name.sources``
* ``METHOD.RxMETHOD`` → ``method_name.receivers``
* ``METHOD.ProblemMETHOD_xD`` → ``method_name.simulation_xd``
* ``METHOD.FieldsMETHOD`` → ``method_name.fields``
where ``METHOD`` was the old module name for the method and ``method_name`` is the new name.
Also many of the utilities modules within these are also being deprecated. Their
old names are still around and should throw a deprecation warning when loaded pointing
to the correct name.
For example,
* ``Utils.codeutils`` → ``utils.code_utils``
* ``Utils.ModelBuilder`` → ``utils.model_builder``
* ``EM.Utils.EMUtils`` → ``electromagnetics.utils.waveform_utils`` (this one is a little less obvious)
``Problem`` to ``Simulation``
-----------------------------
Many of the previous ``Problem`` classes are still within the modules, but
will now throw a deprecation warning which points to the updated ``Simulation``.
The renaming scheme for the ``Simulation``-s was decided in
857.
For example,
* ``Problem3D_CC`` → ``Simulation3DCellCentered``
* ``Problem2D_CC`` → ``Simulation2DCellCentered``
* ``Problem3D_e`` → ``Simulation3DElectricField``
* ``Problem3D_b`` → ``Simulation3DMagneticFluxDensity``
* ``Problem3D_h`` → ``Simulation3DMagneticField``
* ``Problem3D_j`` → ``Simulation3DCurrentDensity``
* etc.
Our current plan is to remove these deprecated versions in the 0.15.0 SimPEG
release.
Potential field reorganization
------------------------------
The ``potential_fields`` module has likely received the largest amount of reorganization
to make it more in line with the structure of the other modules, and some things have
moved around and been renamed. There are now two separate modules within ``potential_fields``:
``gravity`` and ``magnetics``. All of the classes in ``PF.BaseGrav`` have been
moved to ``potential_fields.gravity``, and the classes in ``PF.BaseMag`` have been
moved to ``potential_fields.magnetics``. The ``Map``-s that were within them have
been deprecated and can instead be found in ``SimPEG.maps``.
The option of a ``coordinate_system`` for the magnetics simulation is no longer
valid and will throw an ``AttributeError``. Instead use the `SimPEG.maps.SphericalSystem`.
Improvements and Additions to ``resistivity``
---------------------------------------------
We have made a few improvements to the ``SimPEG.electromagnetics.static.resistivity``
that were motivated by our work under the Geoscientists Without Borders project.
One is that we now have a 1D layered Earth simulation class,
`SimPEG.electromagnetics.static.resistivity.simulation_1d.Simulation1DLayers`,
that can be used to invert resistivity sounding data for conductivity and/or
thicknesses of a set number of layers.
The second, is a new ``miniaturize`` option for the 2D and 3D resistivity simulations.
This option causes the class to internally replace ``Dipole`` sources and receivers
with only unique ``Pole`` sources and receivers. This can result in a dramatic speedup
and reduced memory requirements when the input ``survey`` contains many more
``Dipole``-s than electrode locations. This is especially common in the wenner
type survey acquisitions that use a unique source and receiver for each measured
data point. This option is disabled by default, and can be enabled by passing the
``minaturize=True`` keyword to the resistivity ``Simulation`` upon initialization.
The third is automated ``TreeMesh`` construction within the ``resistivity.IO`` class for
a 2D survey.
Deprecations
------------
Some functions and properties have been renamed to more PEP 8 friendly names,
for example:
* ``Survey.makeSyntheticData`` → ``Simulation.make_synthetic_data``
* ``Survey.srcList`` → ``Survey.source_list``
* ``Source.rxList`` → ``Source.receiver_list``
* ``Source.loc`` → ``Source.location``
* ``Receiver.locs`` → ``Receiver.locations``
* etc...
As mentioned before, the old names of these items will still be around, but
will throw ``DeprecationWarnings`` pointing the user to the new names.
We have done work to make this release as backwards compatible as possible, but can
not guarantee that all code will work. At the minimum, module imports must be changed.
The old style of ``pair``-ing a survey will still work, in which case all of the older
functions will still work. This is only intended for old code to continue working,
and should not be relied upon for developing new code.
As noted all of the ``Problem``-s are being deprecated and will be removed as of
version 0.15.0 of SimPEG. The deprecated utility models will also be removed then.
Pull Requests
=============
There were 25 pull requests contributing to this release.
* 786: Simulation class refactor.
* 792: Use scooby for Versions.
* 802: Simulation jk.
* 819: Simulation jk df.
* 835: Add simulation PF tutorials to simulation PF branch
* 843: Fix drapeTopoLoc.
* 844: Static receiver midpoint 3D.
* 845: Tile map.
* 848: Fix Ward and Hohmann example.
* 849: Update and fix tutorials.
* 851: Simulation method names .
* 853: CSEM TDEM 3D example.
* 854: Simulation dc2.5 d speedup.
* 861: Fix typo.
* 863: light formatting to be closer to black, update filepaths to be os independent.
* 864: Sim dask split.
* 865: simulation fdem testing bug.
* 866: waveform logic bug.
* 868: This change was necessary for fields to be called correctly for dcip2...
* 869: Simulation tutorials 2.
* 872: Uncertainty rename in simulation.
* 874: Release notes for simulation as part of the Documentation.
* 876: Simulation tutorials cleanup.
* 879: Run black on SimPEG.
* 882: Replace html links to documentation with local links.
Closed Issues
=============
* 865: FDEM: derivative test seems to ignore imag-real-flag
* 857: DOC: Broken link
* 850: Can't set a waveform on initialized BaseTDEMSrc object
* 847: Renaming of methods' Problems
* 846: Standard deviation and it's place in the code base
* 842: Static drape topo not on surface.
* 787: Plotting integrated sensitivity for DC resistivity problem.
* 765: DOC: Source code not linked, 404
* 751: run black on SimPEG
* 562: What's the problem with ``Problem``?
* 527: Remove MapPair, RxPair, Pair
* 524: Remove rxType from BaseRx
* 202: Using the DataClass
Contributors
============
Combining to produce over 500 commits (listed in no particular order):
* lheagy
* jcapriot
* fourndo /domfournier
* sgkang
* dccowan
* JKutt
* prisae
* craigmillernz