Carefree-learn

Release Notes

`carefree-learn 0.1.4` fixed some critical bugs in `0.1.3`, as well as introduced some new features (supported evaluating multiple models, customizing losses, etc.).


Backward Compatible Breaking

`carefree-learn` now deals with list of pipelines instead of a single pipeline in most APIs (27)

<table align=center>
<tr>
<td align=center><b>v0.1.3</b></td>
<td align=center><b>v0.1.4</b></td>
</tr>
<tr>
<td>
<pre lang="python">
cflearn.save(m) -> cflearn^_^fcnn.zip
print(cflearn.load()) -> {'fcnn': FCNN()}
</pre>
</td>
<td>
<pre lang="python">
cflearn.save(m) -> cflearn^_^fcnn^_^0000.zip
print(cflearn.load()) -> {'fcnn': [FCNN()]}
</pre>
</td>
</tr>
</table>


Misc

+ Supported [customizing new losses](https://carefree0910.me/carefree-learn-doc/docs/developer-guides/customization#customizing-new-losses).
+ Enhanced `cflearn.evaluate`, it can now evaluate on multiple pipelines.
+ Changed default parallel settings to `non-parallel`.
+ Supported specify `loss` and `loss_config` in `Elements`.
+ Optimized `auto` metric settings. It will now depend itself on `loss`.
+ Implemented `QuantileFCNN` for quantile regression (experimental).

---

+ Fixed `Pipeline.load`.
+ Fixed the configuration stuffs.
+ Fixed other bugs.
+ Optimized some default settings.

Release Notes

`carefree-learn 0.1.3` focuses on performances and accessability.


[`pipe`](https://carefree0910.me/carefree-learn-doc/docs/design-principles#pipe)

A new abstraction, `pipe`, was introduced to `carefre-learn` which significantly improved accessability for developers. Now we can introduce a new model to `carefree-learn` with only one line of code:

python
cflearn.register_model("wnd_full", pipes=[cflearn.PipeInfo("fcnn"), cflearn.PipeInfo("linear")])
m = cflearn.make("wnd_full")


> Please refer to [Build Your Own Models](https://carefree0910.me/carefree-learn-doc/docs/developer-guides/customization) for detailed information.


[`Auto`](https://carefree0910.me/carefree-learn-doc/docs/user-guides/auto-ml/)

`carefree-learn` now provides a high level AutoML API:

python
import cflearn
from cfdata.tabular import TabularDataset

x, y = TabularDataset.iris().xy
auto = cflearn.Auto("clf").fit(x, y)
predictions = auto.predict(x)



[Production](https://carefree0910.me/carefree-learn-doc/docs/user-guides/production)

`carefree-learn` now supports `onnx` export, and provides a high level API `Pack` to pack everything up for production:

python
import cflearn
from cfdata.tabular import TabularDataset

x, y = TabularDataset.iris().xy
m = cflearn.make().fit(x, y)
cflearn.Pack.pack(m, "onnx")


This piece of code will generate an `onnx.zip` in the working directory with following file structure:

text
|--- preprocessor
|-- ...
|--- binary_config.json
|--- m.onnx
|--- output_names.json
|--- output_probabilities.txt


With `onnx.zip` we can make predictions (inference) on the fly:

python
predictor = cflearn.Pack.get_predictor("onnx")
predictions = predictor.predict(x)



Misc

+ `carefree-learn` should be ~3x faster than before on small datasets thanks to optimizations on [categorical encodings](https://carefree0910.me/carefree-learn-doc/docs/optimizations#categorical-encodings).
+ [`DNDF`](https://github.com/carefree0910/carefree-learn/blob/94ca9770560896fa424dc23c39931b8c8f4e0b2b/cflearn/modules/blocks.py#L406) in `carefre-learn` is highly optimized and should be ~3x faster than before.
+ APIs have been re-designed and are much easier to use now.
+ Much better documented than before ([documentations](https://carefree0910.me/carefree-learn-doc/docs/)).
+ Implemented more models (`Linear`, `TreeLinear`, `Wide and Deep`, `RNN`, `Transformer`, etc.).
+ Implemented more modules (`CrossBlock`, `ConditionalBlocks`, `MonotonousMapping`, `MLP.funnel`, etc.).

---

+ Fixed some bugs.
+ Optimized some default settings.

Release Notes

`Experiments`

`Experiments` is much more powerful and much easier to use now:

> **Updated 2020.12.13**: `Experiment` is more useful now in `v0.1.7`! Please refer to the [documentation](https://carefree0910.me/carefree-learn-doc/docs/user-guides/distributed/#experiment) for more details.

python
import cflearn
import numpy as np

from cfdata.tabular import *

def main():
x, y = TabularDataset.iris().xy
experiments = cflearn.Experiments()
experiments.add_task(x, y, model="fcnn")
experiments.add_task(x, y, model="fcnn")
experiments.add_task(x, y, model="tree_dnn")
experiments.add_task(x, y, model="tree_dnn")
results = experiments.run_tasks(num_jobs=2)
{'fcnn': [Task(fcnn_0), Task(fcnn_1)], 'tree_dnn': [Task(tree_dnn_0), Task(tree_dnn_1)]}
print(results)
ms = {k: list(map(cflearn.load_task, v)) for k, v in results.items()}
{'fcnn': [FCNN(), FCNN()], 'tree_dnn': [TreeDNN(), TreeDNN()]}
print(ms)
experiments could be saved & loaded easily
saving_folder = "__temp__"
experiments.save(saving_folder)
loaded = cflearn.Experiments.load(saving_folder)
ms_loaded = {k: list(map(cflearn.load_task, v)) for k, v in loaded.tasks.items()}
{'fcnn': [FCNN(), FCNN()], 'tree_dnn': [TreeDNN(), TreeDNN()]}
print(ms_loaded)
assert np.allclose(ms["fcnn"][1].predict(x), ms_loaded["fcnn"][1].predict(x))

if __name__ == '__main__':
main()


We can see that `experiments.run_tasks` returns a bunch of `Task`s, which can be easily transfered to models through `cflearn.load_task`.

> It is important to wrap the codes with `main()` on some platforms (e.g. Windows), because running codes in parallel will cause some issues if we don't do so. [Here](https://stackoverflow.com/questions/20222534/python-multiprocessing-on-windows-if-name-main)'s an explaination.

`Benchmark`

`Benchmark` class is implemented for easier benchmark testing:

> **Updated 2020.12.13**: `Benchmark` was moved to a separated repo ([carefree-learn-benchmark](https://github.com/carefree0910/carefree-learn-benchmark)).

python
import cflearn
import numpy as np

from cfdata.tabular import *

def main():
x, y = TabularDataset.iris().xy
benchmark = cflearn.Benchmark(
"foo",
TaskTypes.CLASSIFICATION,
models=["fcnn", "tree_dnn"]
)
benchmarks = {
"fcnn": {"default": {}, "sgd": {"optimizer": "sgd"}},
"tree_dnn": {"default": {}, "adamw": {"optimizer": "adamw"}}
}
msg1 = benchmark.k_fold(3, x, y, num_jobs=2, benchmarks=benchmarks).comparer.log_statistics()
"""
~~~ [ info ] Results
===============================================================================================================================
| metrics | acc | auc |
--------------------------------------------------------------------------------------------------------------------------------
| | mean | std | score | mean | std | score |
--------------------------------------------------------------------------------------------------------------------------------
| fcnn_foo_default | 0.780000 | -- 0.032660 -- | 0.747340 | 0.914408 | 0.040008 | 0.874400 |
--------------------------------------------------------------------------------------------------------------------------------
| fcnn_foo_sgd | 0.113333 | 0.080554 | 0.032780 | 0.460903 | 0.061548 | 0.399355 |
--------------------------------------------------------------------------------------------------------------------------------
| tree_dnn_foo_adamw | -- 0.833333 -- | 0.077172 | -- 0.756161 -- | -- 0.944698 -- | -- 0.034248 -- | -- 0.910451 -- |
--------------------------------------------------------------------------------------------------------------------------------
| tree_dnn_foo_default | 0.706667 | 0.253684 | 0.452983 | 0.924830 | 0.060007 | 0.864824 |
================================================================================================================================
"""
save & load
saving_folder = "__temp__"
benchmark.save(saving_folder)
loaded_benchmark, loaded_results = cflearn.Benchmark.load(saving_folder)
msg2 = loaded_results.comparer.log_statistics()
assert msg1 == msg2

if __name__ == '__main__':
main()


Misc

+ Integrated `trains`.
+ Integrated `Tracker` from `carefree-toolkit`.
+ Integrated native `amp` from PyTorch.
+ Implemented `FocalLoss`.
+ Implemented `cflearn.zoo`.

---

+ Introduced CI.
+ Fixed some bugs.
+ Simplified some APIs.
+ Optimized some default settings.

First release