Pysipfenn

**This version introduces 3 exciting changes!** (1) The all new [`ModelAdjusters`](https://github.com/PhasesResearchLab/pySIPFENN/blob/main/pysipfenn/core/modelAdjusters.py) submodule automates tuning and can fetch data directly from [`OPTIMADE API`](https://www.optimade.org); (2) A new manuscript detailing advantages of our featurization tools has been put on [arXiv:2404.02849](https://arxiv.org/abs/2404.02849); and (3) the name of the software was updated to **py**thon toolset for **S**tructure-**I**nformed **P**roperty and **F**eature **E**ngineering with **N**eural **N**etworks to retain the `pySIPFENN` acronym but better reflect our strengths and development direction.

Major Changes:
- Submodule of [`ModelAdjusters`](https://github.com/PhasesResearchLab/pySIPFENN/blob/main/pysipfenn/core/modelAdjusters.py) has been set up for all kinds of model adjusting efforts.
- The [`LocalAdjuster`](https://github.com/PhasesResearchLab/pySIPFENN/blob/5f20178937f1a5f2b1af309a1e87a843e089c3f5/pysipfenn/core/modelAdjusters.py#L22) implements local model tuning, plotting of results, and hyperparameter matrix search.
- The [`OPTIMADEAdjuster`](https://github.com/PhasesResearchLab/pySIPFENN/blob/5f20178937f1a5f2b1af309a1e87a843e089c3f5/pysipfenn/core/modelAdjusters.py#L637) class combines it with the powerful [`OPTIMADE API`](https://www.optimade.org) to automate data fetching. Now, you can quickly
python
from pysipfenn import Calculator, OPTIMADEAdjuster
c = Calculator(autoLoad=False)
c.loadModels("SIPFENN_Krajewski2022_NN30")
ma = OPTIMADEAdjuster(c,
model="SIPFENN_Krajewski2022_NN30",
provider="mp",
targetPath=("attributes", "_mp_stability", "gga_gga+u", "formation_energy_per_atom"),
device="mps" MPS is for Apple M-series GPU
)

ma.fetchAndFeturize(
'elements HAS "Hf" AND elements HAS "Mo" AND NOT elements HAS ANY "O","C","F","Cl","S"',
parallelWorkers=4)
ma.adjust()

ma.plotStarting() See the starting performance
ma.plotAdjusted() See the adjusted performance

or to perform a hyperparameter search, replace the `ma.adjust()` with:
python
ma.matrixHyperParameterSearch()
ma.adjust(learningRate=0.0001, optimizer='AdamW', weightDecay=1e-05, epochs=37)

- The new manuscript on _Efficient Structure-Informed Featurization and Property Prediction of Ordered, Dilute, and Random Atomic Structures_ has been uploaded to [arXiv:2404.02849](https://arxiv.org/abs/2404.02849) and will be submitted to journal in a couple days after comments from collaborators.


Minor Changes:
- Added `writeDescriptorsToNPY` function to streamline persisting feature data into NumPy for our end-users. Appropriate tests were added. Thanks rdamaral for making this contribution!
- Improved numerous docstrings.
- Random solution featurizer now exits gently on `KeyboardInterrupt`
- The `LocalAdjuster` and `OPTIMADEAdjuster` were (optionally) connected to [ClearML](https://clear.ml) for neat tracking of ML training histories.
- Minor bugfixes in several spots.


New Contributors
* rdamaral made their first contribution in https://github.com/PhasesResearchLab/pySIPFENN/pull/15

**Full Changelog**: https://github.com/PhasesResearchLab/pySIPFENN/compare/v0.15.1...v0.16.0

Major Changes:
- The printout of `Calculator` initialization, model parsing, prototype parsing, etc., is now colorized (granted `verbose=True`) for quick and effortless reading of critical settings and information.
- Model download is now handled by our in-house-maintained `pysmartdl2`, instead of `pySmartDL`. It allows us to better test CI with modern Python versions and ensure long term stability.

Minor Changes:
- Requires `pymatgen` version `2024.2.20`, which fixed a serious security issue in `pymatgen.io`.
- Small bug fixes for two rare cases.
- Workshop materials in `examples` updated to be compatible with the current `pymatgen` version.
- Minor style improvements.


**Full Changelog**: https://github.com/PhasesResearchLab/pySIPFENN/compare/v0.15.0...v0.15.1

Welcome to pySIPFENN v0.15, which is ***our biggest singular jump since 2020 spanning 196 commits since v0.13.0*** 🎉 It covers improvements in nearly every place of the codebase, including ***new features, better performance and more convenient high-level API***. Two key new features are (1) a ***neat prototype library*** (v0.14) simplifying the handling of atomic structures, and (2) ***random solid solution (RSS) featurizer*** / descriptor calculator (v0.15), which allows you to featurize chemical composition occupying a prototype structure.

Now, you can efficiently utilize pySIPFENN to, ***for instance, featurize high entropy alloys (HEAs) occupying BCC, FCC, C14, and C15 structures.*** Have a look below for details!

Major Changes:

Random Solid Solution Featurization
- KS2022 Random Solution Featurization ([`KS2022_randomSolutions`](pysipfenn/descriptorDefinitions/KS2022_randomSolutions.py)) added by amkrajewski (mostly in https://github.com/PhasesResearchLab/pySIPFENN/pull/10), which **allows users to quickly obtain KS2022 feature vector corresponding to a random solid solution with composition `comp` occupying arbitrary structure `struct` (including library `BCC`, `FCC`, `HCP`) through iterative expansion of ensemble of local chemical environments (LCEs) until all features, i.e., statistics over LCEs, and composition converge onto stable values.**
- This method is philosophically similar to the SQS approach [(10.1103/PhysRevLett.65.353)](https://doi.org/10.1103/PhysRevLett.65.353), which one can generate with [Alloy Theoretic Automated Toolkit](https://www.brown.edu/Departments/Engineering/Labs/avdw/atat/) (ATAT) [(10.1016/j.calphad.2013.06.006)](https://doi.org/10.1016/j.calphad.2013.06.006) or approximate with [sqsgenerator](https://github.com/dgehringer/sqsgenerator); however, here, we are not only encoding the correlations between anonymous species types but also (1) consider differences between them, thus, e.g., solutions of elements with similar electronic configurations will converge faster) and (2) allow arbitrary complexity of the chemical composition under a single methodology.
- It should also be better at capturing randomly occurring uncommon extreme-case events, which may not occur in low-correlation cases.
- We have tuned the default values of the method for complex cases of (a) high entropy alloys (HEAs) with 6+ components and (b) multicomponent steels with several minor alloying elements.
- Added matching tests for all methods of [`KS2022_randomSolutions`](pysipfenn/descriptorDefinitions/KS2022_randomSolutions.py) featurizer. Added documentation for all tests, so that they can serve as runnable examples.
- Added complete API documentation of all methods and all parameters, with both meaning and motivation, for [`KS2022_randomSolutions`](pysipfenn/descriptorDefinitions/KS2022_randomSolutions.py).
- Added top-level [`Calculator.calculate_KS2022_randomSolutions()`](pysipfenn/core/pysipfenn.pyL480) and [`Calculator.runModels_randomSolutions()`](pysipfenn/core/pysipfenn.pyL800) for conveniently calling everything on complex problems, even with mixed-type inputs like:
python
c.calculate_KS2022_randomSolutions(
['FCC', myDistortedBCC, 'BCC', 'HCP'],
['WMo', Composition.from_weight_dict({'Mo': 20, 'W': 70, 'Zr': 10}), 'FeNi', 'CrNi'],
mode='parallel',
max_workers=4)




Prototype Library:
- The new prototype library functionality allows you to store commonly used structures in a consistent fashion and pleasant schema under [`misc/prototypeLibrary.yaml`](pysipfenn/misc/prototypeLibrary.yaml) file in your pySIPFENN installation. It is initialized alongside every `Calculator` instance.
- Parsing custom libraries is very easy and can be done from a file in your directory or from a remote repository like Zenodo! Simply use the [`parsePrototypeLibrary`](pysipfenn/core/pysipfenn.pyL130) to fetch it. Or, to append your library with an external one and persist it your installation, use [`appendPrototypeLibrary`](pysipfenn/core/pysipfenn.pyL180).
yaml
- name: BCC
origin: https://www.oqmd.org/materials/prototype/A2_W
POSCAR: |
W
1.0
-1.58250 1.58250 1.58250
1.58250 -1.58250 1.58250
1.58250 1.58250 -1.58250
W
1
Direct
0.00000 0.00000 0.00000

- Added complete API documentation of all methods and all parameters, with both meaning and motivation, for everything prototypeLibrary-related.
- Added matching tests for all new prototypeLibrary-related methods. Added documentation for all tests, so that they can serve as runnable examples.

General:
- Complete overhaul of the documentation in terms of consistent styling, type-hinting, and extent of the content. Most functions can be well-understood just by looking at information displayed by IDE like PyCharm or VSCode.
- The `Calculator` class now can self-destruct and deallocate by call to its `destroy()` function, allowing users to better manage memory used in calculations on low-power systems.
- Performance improvements in chemical attributes calculation of all `KS2022`-calculating featurizers.
- Performance improvements and minor fixes in `KS2022_dilute` featurizer.
- Performance and feature improvements in the [`modelExporters`](pysipfenn/core/modelExporters.py), including imports optimization so that are now they only run when needed by given exporter class.
- Optimized testing workflows in terms of events triggering them and general performance.
- Improved testing clarity and platform compatibility, by splitting them into individual workflows.
[](https://github.com/PhasesResearchLab/pySIPFENN/actions/workflows/coreTests_LinuxUbuntu.yaml) [](https://github.com/PhasesResearchLab/pySIPFENN/actions/workflows/coreTests_MacM1.yaml) [](https://github.com/PhasesResearchLab/pySIPFENN/actions/workflows/coreTests_MacIntel.yaml) [](https://github.com/PhasesResearchLab/pySIPFENN/actions/workflows/coreTests_Windows.yaml)
- Dependencies were analyzed for compatibility and set in a way to ensure long-term maintainability and near-future compatibility with Python 3.12.
- Added [instructions for contributing to pySIPFENN](https://pysipfenn.readthedocs.io/en/latest/contributing.html), explaining philosophy of the software and several design choices.

Minor Changes:
- Improved printouts from `Calculator` initialization and current state reporting.
- Improved verbosity handling in `Calculator` instances.
- `CITATION.cff` added for clear citation guidance in the future.
- Documentation now includes [miscellaneous notes](https://pysipfenn.readthedocs.io/en/latest/miscellaneousnotes.html) users may find useful.
- `.gitignore` improvements as requested by users.
- Added testing for several smaller functions like `ward2ks2022`, to increase coverage by 1%. Now it is over 95%.
- Code style improvements.
- Commenting improvements scattered throughout the codebase to make modifying our code easier.
- And many more added on the way :)

Notices:
- In the near future (likely next release), pySIPFENN will drop **official support for Python 3.9**, due to (1) noticeably lower performance, (2) deprecation of 3.9 on some platforms (MacOS14), and (3) next release using several pure-Python features added in Python 3.9.
- In the near future (likely next release), pySIPFENN will **add support for Python 3.12** to keep it up-to-date and take advantage of performance boost. Our codebase is ready for the switch. We are waiting for PyTorch support for it in its stable release.

**Full Changelog**: https://github.com/PhasesResearchLab/pySIPFENN/compare/v0.13.1...v0.15.0

Major Changes
- All tests are now documented under API [pySIPFENN Tests](https://pysipfenn.readthedocs.io/en/latest/source/pysipfenn.tests.html#pysipfenn-tests) page with hyperlinks to their sources, acting as minimalistic examples of using the software.

Minor Changes
- Improvements to FAQ, docs index, and README.
- Polishing of the documentation page and general maintenance.

**Full Changelog**: https://github.com/PhasesResearchLab/pySIPFENN/compare/v0.13.0...v0.13.1

Major Changes:
- Per [[add] modelExporters](https://github.com/PhasesResearchLab/pySIPFENN/pull/8) by amkrajewski, 3 model exporter classes have been added to pySIPFENN in the `modelExporters` module:
- `ONNXExporter` allowing (1) exporting back to ONNX after models were adjusted to new datasets or new properties (transfer learning), (2) automated simplification of model architecture for improved performance, and (3) adjusting model internal precision to FP16 to reduce its size by half with only minor performance impact.
- `TorchExporter` to export models to `PyTorch` which are used internally by pySIPFENN.
- `CoreMLExporter` to export models to Apple's CoreML format developed for use in their devices, where it provides the most seamless integration with existing apps and can harvest very efficient Neural Engine hardware acceleration. At the same time, it can be used on other platforms as well, such as Linux or Windows. Note that by default, models will be converted to FP16 precision, similar to one of the `ONNXExporter` options.

The above changes are a step in our effort to make advanced use of pySIPFENN easier for end-users. Next items on the roadmap include (1) automation of transfer learning on small datasets and (2) OPTIMADE integrations, which will make pySIPFENN API a one-stop solution for model fine-tuning.

Minor Changes:
- Small improvements in the `Calculator` object printout with the `verbose` option (true by default) added to its initialization.
- Matching tests were added to the CI pipelines.
- `dev` extra dependencies have been established, alongside appropriate documentation, for all future dependencies that are not required for core pySIPFENN functionalities to keep it as light as possible while not limiting advanced users. One of the future directions will be to make core dependencies lighter.
- Model exports and motivations were added to the documentation under [Exporting pySIPFENN Models](https://pysipfenn.org/exportingmodels.html) page.

**Full Changelog**: https://github.com/PhasesResearchLab/pySIPFENN/compare/v0.12.2...v0.13.0

Major Changes:
- This is a minor release that changes the license to [LGPLv3](https://www.gnu.org/licenses/lgpl-3.0.en.html) in order to allow for integration with proprietary software developed by the CALPHAD community (shipping pySIPFENN and then calling it from closed-source tools will be allowed) while supporting the development of new pySIPFENN features for all users, as modifications to pySIPFENN (except for models, see below) itself need to be open.

- This change does not affect the [default pySIPFENN models](https://doi.org/10.5281/zenodo.4006802), which are distributed under [Creative Commons 4.0](https://creativecommons.org/licenses/by/4.0/legalcode) that already comes with no restrictions.

- Many thanks to our colleagues from [GTT-Technologies](https://gtt-technologies.de) and other participants of [50th CALPHAD 2023 conference in Boston](https://calphad.org/calphad-2023) for fruitful discussions.

**Full Changelog**: https://github.com/PhasesResearchLab/pySIPFENN/compare/v0.12.1...v0.12.2