Excore

Latest version: v0.1.1

Safety actively analyzes 623739 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 1 of 2

2.0

0.25

or explicitly specify ``PrimaryFields
[Transforms.RandomPaddingCrop]
crop_size = [1024, 512]

It can even be undefined when there are no arguments
[Normalize]



</details>

<details>
<summary> :sparkles:Auto-complement, type-hinting, docstring and code navigation for config files </summary>

The old-style design of plain text configs has been criticized for being difficult to write (without auto-completion) and not allowing navigation to the corresponding class. However, Language Server Protocol can be leveraged to support various code editing features, such as auto-completion, type-hinting, and code navigation. By utilizing lsp and json schema, it's able to provide the ability of auto-completion, some weak type-hinting (If code is well annotated, such as standard type hint in python, it will acheive more) and docstring of corresponding class.

![](https://user-images.githubusercontent.com/72954905/267884541-56e75031-48a2-4768-8a6c-fc7b83ed977e.gif)

![config](https://github.com/Asthestarsfalll/ExCore/assets/72954905/2b0e151c-5c2b-4082-9796-d171e211c7c8)

`ExCore` dump the mappings of class name and it file location to support code navigation. Currently only support for neovim, see [excore.nvim](https://github.com/Asthestarsfalll/excore.nvim).

![to_class](https://github.com/Asthestarsfalll/ExCore/assets/72954905/9677c204-eb46-4cf3-a8bf-03f9bee8d6fb)

</details>

<details>
<summary>Config inheritance</summary>
Use `__base__` to inherit from a toml file. Only dict can be updated locally, other types are overwritten directly.

toml
__base__ = ["xxx.toml", "xxxx.toml"]


</details>

<details>
<summary>``Reused module</summary>

`ExCore` use `` to mark the reused module, which is shared between different modules.

toml
FCN and SegNet will use the same ResNet object
[Model.FCN]
backbone = "ResNet"

[Model.SegNet]
backbone = "ResNet"

[ResNet]
layers = 50
in_channel = 3


equls to

python
resnet = ResNet(layers=50, in_channel=3)

FCN(backbone=resnet)
SegNet(backbone=resnet)

If use `!`, it equls to

FCN(backbone=ResNet(layers=50, in_channel=3))
SegNet(backbone=ResNet(layers=50, in_channel=3))


</details>

<details>
<summary>`$`Refer Class and cross file</summary>

`ExCore` use `$` to represents class itself, which will not be instantiated.

toml
[Model.ResNet]
$block = "BasicBlock"
layers = 50
in_channel = 3


equls to

python
from xxx import ResNet, BasicBlock
ResNet(block=BasicBlock, layers=50, in_channel=3)


In order to refer module accross files, `$` can be used before `PrimaryFields`. For example:

File A:

toml
[Block.BasicBlock]


File B:

toml
[Block.BottleneckBlock]


File C:

toml
[Model.ResNet]
!block="$Block"


So we can combine file A and C or file B and C with a toml file

toml
__base__ = ["A.toml", "C.toml"]
or
__base__ = ["B.toml", "C.toml"]


</details>

<details>
<summary>`&`Variable reference</summary>

`ExCore` use `&` to refer a variable from the top-level of config.

**Note: The value may be overwritten when inheriting, so the call it variable.**

toml
size = 224

[TrainData.ImageNet]
&train_size = "size"
!transforms = ['RandomResize', 'Pad']
data_path = 'xxx'

[Transform.Pad]
&pad_size = "size"

[TestData.ImageNet]
!transforms = ['Normalize']
&test_size = "size"
data_path = 'xxx'


</details>

<details>
<summary>:sparkles:Using python module in config file</summary>

The `Registry` in `ExCore` is able to register a module:

python
from excore import Registry
import torch

MODULE = Registry("module")
MODULE.register_module(torch)


Then you can use torch in config file:

toml
[Model.ResNet]
$activation = "torch.nn.ReLU"
or
!activation = "torch.nn.ReLU"


python
import torch
from xxx import ResNet

ResNet(torch.nn.ReLU)
or

ResNet(torch.nn.ReLU())


**Note: You shouldn't define arguments of a module.**

</details>

<details>
<summary>:sparkles:Argument-level hook</summary>

`ExCore` provide a simple way to call argument-level hooks without arguments.

toml
[Optimizer.AdamW]
params = "$Model.parameters()"

0.9

0.5

├──────────────────────────┼──────────────────────────────────────────────────────────────────────┤
│ TestData.CityScapes │ ╒═════════════╤═══════════════╕ │
│ │ │ !transforms │ ['Normalize'] │ │
│ │ ├─────────────┼───────────────┤ │
│ │ │ &test_size │ size │ │
│ │ ├─────────────┼───────────────┤ │
│ │ │ data_path │ xxx │ │
│ │ ╘═════════════╧═══════════════╛ │
├──────────────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Model.FCN │ ╒═══════════╤════════════╕ │
│ │ │ backbone │ ResNet │ │
│ │ ├───────────┼────────────┤ │
│ │ │ head │ SimpleHead │ │
│ │ ╘═══════════╧════════════╛ │
...


</details>

Registry

<details>
<summary>:sparkles:LazyRegistry</summary>
To reduce the unnecessary imports, `ExCore` provides `LazyRegistry`, which store the mappings of class/function name to its `qualname` and dump the mappings to local. When config parsing, the necessary modules will be imported.

</details>

<details>
<summary>Extra information</summary>

python
from excore import Registry

Models = Registry("Model", extra_field="is_backbone")


Models.register(is_backbone=True)
class ResNet:
pass



</details>

<details>
<summary>Modules classification and fuzzy search</summary>

python
from excore import Registry

Models = Registry("Model", extra_field="is_backbone")


Models.register(is_backbone=True)
class ResNet:
pass

Models.register(is_backbone=True)
class ResNet50:
pass

Models.register(is_backbone=True)
class ResNet101:
pass

Models.register(is_backbone=False)
class head:
pass


print(Models.module_table(select_info='is_backbone'))

print(Models.module_table(filter='**Res**'))


results:


╒═══════════╤═══════════════╕
│ Model │ is_backbone │
╞═══════════╪═══════════════╡
│ ResNet │ True │
├───────────┼───────────────┤
│ ResNet101 │ True │
├───────────┼───────────────┤
│ ResNet50 │ True │
├───────────┼───────────────┤
│ head │ False │
╘═══════════╧═══════════════╛

╒═══════════╕
│ Model │
╞═══════════╡
│ ResNet │
├───────────┤
│ ResNet101 │
├───────────┤
│ ResNet50 │
╘═══════════╛


</details>

<details>
<summary>Register all</summary>

python
from torch import optim
from excore import Registry

OPTIM = Registry("Optimizer")


def _get_modules(name: str, module) -> bool:
if name[0].isupper():
return True
return False


OPTIM.match(optim, _get_modules)
print(OPTIM)


results:


╒════════════╤════════════════════════════════════╕
│ NAEM │ DIR │
╞════════════╪════════════════════════════════════╡
│ Adadelta │ torch.optim.adadelta.Adadelta │
├────────────┼────────────────────────────────────┤
│ Adagrad │ torch.optim.adagrad.Adagrad │
├────────────┼────────────────────────────────────┤
│ Adam │ torch.optim.adam.Adam │
├────────────┼────────────────────────────────────┤
│ AdamW │ torch.optim.adamw.AdamW │
├────────────┼────────────────────────────────────┤
│ SparseAdam │ torch.optim.sparse_adam.SparseAdam │
├────────────┼────────────────────────────────────┤
│ Adamax │ torch.optim.adamax.Adamax │
├────────────┼────────────────────────────────────┤
│ ASGD │ torch.optim.asgd.ASGD │
├────────────┼────────────────────────────────────┤
│ SGD │ torch.optim.sgd.SGD │
├────────────┼────────────────────────────────────┤
│ RAdam │ torch.optim.radam.RAdam │
├────────────┼────────────────────────────────────┤
│ Rprop │ torch.optim.rprop.Rprop │
├────────────┼────────────────────────────────────┤
│ RMSprop │ torch.optim.rmsprop.RMSprop │
├────────────┼────────────────────────────────────┤
│ Optimizer │ torch.optim.optimizer.Optimizer │
├────────────┼────────────────────────────────────┤
│ NAdam │ torch.optim.nadam.NAdam │
├────────────┼────────────────────────────────────┤
│ LBFGS │ torch.optim.lbfgs.LBFGS │
╘════════════╧════════════════════════════════════╛


</details>

<details>
<summary>All in one</summary>

Through Registry to find all registries. Make registries into a global one.

python
from excore import Registry

MODEL = Registry.get_registry("Model")

G = Registry.make_global()



</details>

<details>
<summary>:sparkles:Register module</summary>

`Registry` is able to not only register class or function, but also a python module, for example:

python
from excore import Registry
import torch

MODULE = Registry("module")
MODULE.register_module(torch)


Then you can use torch in config file:

toml
[Model.ResNet]
$activation = "torch.nn.ReLU"
or
!activation = "torch.nn.ReLU"


equls to

python
import torch
from xxx import ResNet

ResNet(torch.nn.ReLU)
or
ResNet(torch.nn.ReLU())


</details>

RoadMap

For more features you may refer to [Roadmap of ExCore](https://github.com/users/Asthestarsfalll/projects/4)

What's Changed
* :sparkles: Support LazyRegistry by Asthestarsfalll in https://github.com/Asthestarsfalll/ExCore/pull/9
* :sparkles: Support completion for config file by Asthestarsfalll in https://github.com/Asthestarsfalll/ExCore/pull/11
* :construction_worker: Add CI by Asthestarsfalll in https://github.com/Asthestarsfalll/ExCore/pull/15
* :sparkles: add workspace config and cli by Asthestarsfalll in https://github.com/Asthestarsfalll/ExCore/pull/16
* :sparkles: Support non-instance value: class and post-hook by Asthestarsfalll in https://github.com/Asthestarsfalll/ExCore/pull/18
* :bug: fix some config/json_schema bugs by Asthestarsfalll in https://github.com/Asthestarsfalll/ExCore/pull/19
* :sparkles: Support Variable Reference by Asthestarsfalll in https://github.com/Asthestarsfalll/ExCore/pull/22
* :art: Turn to ruff by Asthestarsfalll in https://github.com/Asthestarsfalll/ExCore/pull/23
* :art: Use black as formatter by Asthestarsfalll in https://github.com/Asthestarsfalll/ExCore/pull/24
* :recycle::white_check_mark: Refactor config parsing, add tests for co… by Asthestarsfalll in https://github.com/Asthestarsfalll/ExCore/pull/27
* :white_check_mark: Add more tests by Asthestarsfalll in https://github.com/Asthestarsfalll/ExCore/pull/28
* :sparkles: Add simple code navigation by Asthestarsfalll in https://github.com/Asthestarsfalll/ExCore/pull/29
* :white_check_mark: Add more tests of config by Asthestarsfalll in https://github.com/Asthestarsfalll/ExCore/pull/30
* :sparkles: Support type-hints for modules by Asthestarsfalll in https://github.com/Asthestarsfalll/ExCore/pull/32


**Full Changelog**: https://github.com/Asthestarsfalll/ExCore/commits/0.1.1beta

0.05

0.1.1beta

ExCore

![](./docs/static/img/lo.png)

`ExCore` is a Configuration/Registry System designed for deeplearning, with some utils.

:sparkles: `ExCore` supports auto-completion, type-hinting, docstring and code navigation for config files

`ExCore` is still in an early development stage.

English | [中文](./README_cn.md)

Features

Config System

Config system is the **core** of deeplearning projects which enable us to manage and adjust hyperparameters and expriments. There are some attempts of config system because the whole community has been suffering from the plain text config files for a long while.

Config System in `ExCore` is specifically designed for deeplearning training (generally refers to all similar part, e.g. testing, evaluating) procedure. _The core premise is to categorize the objects to be created in the config into three classes - `Primary`, `Intermediate`, and `Isolated` objects_

1. `Primary` objects are those which are **directly** used in training, e.g. model, optimizer. `ExCore` will instantiate and return them.
2. `Intermediate` objects are those which are **indirectly** used in training, e.g. backbone of the model, parameters of model that will pass to optimizer. `ExCore` will instantiate them, and pass them to target `Primary` objects as arguments according some rules.
3. `Isolated` objects refer to python built-in objects which will be parsed when loading toml, e.g. int, string, list and dict.

`ExCore` extends the syntax of toml file, introducing some special prefix characters -- `!`, ``, `$` and '&' to simplify the config defination.

The config system has following features.

<details>
<summary>Get rid of `type`</summary>

yaml
Model:
type: ResNet <----- ugly type
layers: 50
num_classes: 1


In order to get rid of `type`, `ExCore` regards all registered names as `reserved words`. The `Primary` module need to be defined like `[PrimaryFields.ModuleName]`. `PrimaryFields` are some pre-defined fields, e.g. `Model`, `Optimizer`. `ModuleName` are registered names.

toml
[Model.FCN]
layers = 50
num_classes = 1


</details>

<details>
<summary>Eliminate modules nesting</summary>

yaml
TrainData:
type: Cityscapes
dataset_root: data/cityscapes
transforms:
- type: ResizeStepScale
min_scale_factor: 0.5
max_scale_factor: 2.0
scale_step_size: 0.25
- type: RandomPaddingCrop
crop_size: [1024, 512]
- type: Normalize
mode: train



`ExCore` use some special prefix characters to specify certain arguments are modules as well. More prefixes will be introduced later.

toml
[TrainData.Cityscapes]
dataset_root = "data/cityscapes"
mode = 'train'
use `!` to show this is a module, It's formal to use a quoted key "!transforms", but whatever
!transforms = ["ResizeStepScale", "RandomPaddingCrop", "Normalize"]

`PrimaryFields` can be omitted in defination of `Intermediate` module
[ResizeStepScale]

Page 1 of 2

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.