Carefree-learn

Release Notes

`carefree-learn 0.1.16` improved overall performances.


Optimizer

`MADGRAD` (4466c9f) & `Ranger` (acdeec4) are now introduced.

> Reference: [Best-Deep-Learning-Optimizers](https://github.com/lessw2020/Best-Deep-Learning-Optimizers).


Misc

+ Fixed `ddp` when np.ndarray is provided (969a6c8).
+ Fixed `RNN` when `bidirectional` is `True` (be974df) (6ef49f7).

---

+ Optimized `Transformer` (00bd2c4) (dc6abc4) (aec1846).
+ Re-designed the `reduce` part (b99d4a2).
+ Summary will now be written to disk (d5435e9).
+ `batch_indices` will be injected to `forward_results` (7a40dcc).

Release Notes

`carefree-learn 0.1.15` improved overall performances.


`DDP`

Since `PyTorch` is introducing `ZeRO` optimizer, we decided to remove `deepspeed` dependency and use native `DDP` from `PyTorch`.

python
results = cflearn.ddp(tr_file, world_size=2)
predictions = results.m.predict(te_file)



`JitLSTM`

Since native `RNN`s of `PyTorch` do not support dropouts on `w_ih` and `w_hh`, we followed the official implementation of `jit` version `LSTM` and implemented these dropouts.

python
m = cflearn.make(
"rnn",
model_config={
"pipe_configs": {
"rnn": {
"extractor": {
"cell": "JitLSTM"
}
}
}
}
)



Misc

+ Fixed `NNB` when `std` is 0 (177363e).
+ Fixed `summary` in some edge cases (945ca15, f95f667, 2768153).
+ Introduced `ONNXWrapper` for more general ONNX exports (226de5b).

---

+ Optimized `Transformer` (b09916b).
+ Upgraded `PyTorch` dependency (a596031).
+ Supported `reuse_extractor` in `PipeInfo` (149aa49).
+ Implemented `HighwayBlock` (3dad99e, 436ebab) and Introduced `*FCNNConfig` (e0670f7).
+ Implemented Initializer.orthogonal (1019114) and Optimized initializations of `RNN` (2193706).

Release Notes

`carefree-learn 0.1.14` improved overall performances.


Summary

For non-distributed trainings, `carefree-learn` will print out model summaries now by default (inspired by [`torchsummary`](https://github.com/sksq96/pytorch-summary)):

text
========================================================================================================================
Layer (type) Input Shape Output Shape Trainable Param
------------------------------------------------------------------------------------------------------------------------
RNN [-1, 5, 1] [-1, 256] 198,912
GRU [-1, 5, 1] [[-1, 5, 256], [-1, 128, 256]] 198,912
FCNNHead [-1, 256] [-1, 1] 395,777
MLP [-1, 256] [-1, 1] 395,777
Mapping-0 [-1, 256] [-1, 512] 132,096
Linear [-1, 256] [-1, 512] 131,072
BN [-1, 512] [-1, 512] 1,024
ReLU [-1, 512] [-1, 512] 0
Dropout [-1, 512] [-1, 512] 0
Mapping-1 [-1, 512] [-1, 512] 263,168
Linear [-1, 512] [-1, 512] 262,144
BN [-1, 512] [-1, 512] 1,024
ReLU [-1, 512] [-1, 512] 0
Dropout [-1, 512] [-1, 512] 0
Linear [-1, 512] [-1, 1] 513
========================================================================================================================
Total params: 594,689
Trainable params: 594,689
Non-trainable params: 0
------------------------------------------------------------------------------------------------------------------------

Weekly patch with miscellaneous fixes and updates.

Weekly patch with miscellaneous fixes and updates.

carefree-learn 0.1.11` is mainly a patch release which supported more customizations. However, these features are still at early stage and are likely to be changed in the future. If you want to customize `carefree-learn`, it is still highly recommended to clone this repo and install it with edit mode (`pip install -e .`).