Kipet

The first major announced/labelled release since May. This new version includes a host of new features and tutorial examples, as well as being far more stable, user-friendly, and customizable. I will list some of the additions since the previous release here. There may be others that have been forgotten, however these are all included in the documentation.

Major Features

Inclusion of dosing

It is now possible to do both variance and parameter estimation in systems with dosing using the fe_factory to do the initialization and problem setup. Examples showing how this can be implemented are included:

'Ex_2_estimationfefactoryTempV.py'
'Ex_5_sim_fe_by_fe_multjumpsandinputs.py'

Pre-processing

We can now use Kipet to perform data pre-processing, including the Savitzky-Golay filter, Standard Normal Variate, and Multiplicative Scatter Correction. Examples included:

'Ex_2_estimation_filter.py'
'Ex_2_estimation_filter_deriv.py'
'Ex_2_estimation_filter_msc.py'
'Ex_2_estimation_filter_snv.py'

Inclusion of parameter estimation for problems with concentration data

Kipet can now do estimation of kinetic parameters for problems where only concentration data is provided. New examples are found in:
'Ex_7_concentration_input.py'
'Ex_7_conc_input_conf.py"
'Ex_7_conc_input_conf_k_aug.py'

Inclusion of estimability analysis for problems with concentration data.

Adding of a new class for estimability analysis, EstimabilityAnalyzer. EstimabilityAnalyzer includes new tools to perform the estimability using the method of Wu et al. (2011) for problems that have concentration data. The method first does parameter ranking using the method of Yao (2003) using scaled matrix sensitivities obtained at the initial parameter values using k_aug. After this, the list of ranked parameters is used to calculate the number of parameters to estimate using the method of Wu et al, where a sequence of parameter estimation problems is solved using the ParameterEstimator with different numbers of parameters fixed and variable. These simplified models and the extended model are compared using the mean squared error. Finally a list of which parameters should be made variable and which should be fixed is returned to the user.
Examples that show how this feature works and is used:
'Ex_8_estimability.py'
'Ex_9_estimability_with_problem_gen.py'

Minor Features

Non-reacting components / interfering components

This is done for two situations. First where the interfering species' absorbance is not known ahead of time. This example is found in:
'Ex_2_abs_not_react.py'

The second example is when the interfering component is known ahead of time. This can be shown in:
'Ex_2_abs_known_non_react.py'

setting species' absorbances easily

With this, it is now also possible to fix any species' absorbance easily using set_known_absorbing_species() function. It is possible for cases where species are reacting or not, however care should be taken in these cases, especially when using the confidence intervals as fixing these absorbances results in fewer degrees of freedom.

Adding noise to simulated data

New function add_noise_to_signal() adds random normally distributed noise to any dataframe input.

k_aug can be used for confidences

k_aug has now been integrated for providing the reduced hessian information from Ipopt output. The utility of k_aug and its preference over sIpopt is shown in a number of examples and the additional advantage that k_aug provides, that of easily providing sensitivity information is utilized in the estimability analysis examples.

Bug fixes for aspirin and Michael's examples

New stable release, which includes the features and code of the latest KIPET publication.

Major Features:
Multiple Experimental Datasets
The major addition to the KIPET package since the last release is the inclusion of the MultipleExperimentsEstimator class that allows for multiple experimental datasets to be analyzed simultaneously. The class allows for different models or the same model to be inputted, with parameters that are local or global to the specific datasets automatically detected. The class contains functions to run separate variance estimation upon each dataset that can also be used to initialize the problem. In addition the parameter estimation can be done simultaneously across multiple datasets and models for both spectra and concentration problems with or without shared spectra between species.

Among some of the additional useful features are:
- Added initialization techniques for parameter estimation from multiple experiments: Initialize from simulation (FESimulator and PyomoSimulator) and previous results files as is possible in single experiment cases (mostly for concentration data).
- Initial values of complementary states, as well as regular states can now be estimated, and their confidence intervals determined.

New Variance Estimation Method
The new variance estimation method described in the second KIPET paper is also included, providing a more rigorous approach to the variance estimation.

Minor Features:
- Made changes such that Casadi can still be used with new syntax from last merge.
- Added option to rerun failed parameter estimation in estimability analysis (running according to ranking) with additional IPOPT options.
- Parameter estimation from multiple experiments is working now with new version of k_aug using ma57.
- New ability to read data directly from Mettler Toledo csv files, removing negative values through either setting to zero or by performing a baseline shift (See data_tools).
- A host of new examples to showcase the multiple experiments estimation, new documentation, as well as a few minor bug fixes.

- Includes new feature to exclude the C-Z term and only perform parameter estimation using the measurement error. 2 new examples show how to do this and including confidences with k_aug
- Also a new requirements.txt file that should be good across Python 2.7 and Python 3.5 and up
- A new KIPET documentation via readthedocs to make it easier to collaborate and host and also to adhere to Python package standards
- Contains the new variance estimation techniques with new examples.
- Includes some new tools to analyze the pseudo-equivalency matrix, although these tools will still be improved to be more general.
- Addition of option to take H/UPLC data with just liquids or liquids and solids into account
- Addition of option for quadratic penalty term for penalizing values deviating from the defined optimal parameter values in the objective of the parameter estimation problem
- Addition of option to read values for mutable parameter from file, interpolate and use in the modeling context
- Addition of option to write eigenvalues of reduced Hessian automatically to a file
- Fixed bugs and other small enhancements

New stable release, which coincides with the first paper submission, includes a few useful new features as well as a slew of smaller features for convenience, slight syntax changes, as well as new documentation.

Major Features

Wavelength selection based on lack of fit

Useful tool to firstly find which wavelength correlate to the concentration profiels the most and then tools to remove wavelengths that are the least correlated to the concentration profile. Helps reduce problem sizes as well as increase the lack of fit. Examples and documentation included.

Diagnostic tools

Added in a basic PCA feature and SVD as well as the ability to easily identify the lack of fit of the solution.

Upgrade to Python 3.7 and Pyomo 5.6

Number of fixes to include new versions of dependencies.

Minor Features

Can now fix the device variance in VarianceEstimator

Useful for problems where the variance estimation algorithm struggles to converge to useful variance values, putting too much variance into the instrument. Also useful if manufacturer provides variance of instrument.

Can now read spectra directly from device CSV

While it doesn't allow data to be read from other file types, it automatically transforms timestamped data into seconds and also can be used to remove negative values

New data_tools

New data tools for baseline shift, removing negative values and removing certain wavelengths to decrease problem size.

Added ability to set bounds on certain profiles

This is a really useful new feature for when we know of certain peak behaviour or know that a certain species' concentration does not go below a certain value.

New structure for non-absorbing species

A new structure was implemented for the case with non-absorbing species. They are not included in the S matrix any more and are taken out of the first term in the objective of the parameter estimation problem. The user interface is still the same.

Warmstart options now included

When solving the optimization problem the user can now use the ipopt warmstart option as well, such that we can use the dual variables from a previous problem for the solution of the current problem.

New installation validation script and setup.py

Can now ask to report time spent on variance estimator and parameter estimator

New examples and documentation

Bug fixes

- Added using ipopt for least squares into a standard example as it is actually faster and users did not have an example of its implementation.

- fixed issue in variance estimator where solver options were not passed onto suboptimizations correctly

- spotted the use of the wrong keyword in example for variance estimation ("solver_options" instead of "solver_opts"

- spotted and fixed issue where negative D values were not being sensed by the VarianceEstimator and therefore the S matrix was still bounded to 0.

- Estimability analysis for concentrations is now fully functional.

- Lack of fit tool works for non-absorbing species as well.

This release includes:
- Fixed several bugs
- Comprehensible unit tests.
- Lambda selection
- fe_factory.

We are moving back the release numbers to coincide with the PyPi packages that are now available. This is why we now use 0.1.3 instead of continuing from the 1.1.0 from almost two years ago.

The code is ready, but the documentation will be coming soon. Once the documentation is complete and all of the older methods that have not yet been updated to the new format have been renewed, I will be using version 1.0.0 again.