Added
- `mains` module explanation article https://medium.com/alex_ber/making-relative-path-to-file-to-work-d5d0f1da67bf is published.
- `fabs` module. It adds cp method to fabric.Connection.
This method is Linux-like cp command. It copies single file to remote (Posix) machine.
- Spited dependency list for setup.py req.txt (inexact versions, direct dependency only) and for
reproducible installation requirements.txt (exact versions, all, including transitive dependencies).
- Added req-fabric.txt, requirements-fabric.txt - Fabric, used in `fabs` module.
- Added req-yml.txt, requirements-yml.txt - Yml-related dependencies, used in ymlparsers.py
and in init_app_conf.py, deploys.py; optionally used in ymlparsers_extra.py, emails.py.
Main dependency is HiYaPyCo. I'm using feature that is availlable in the minimal version.
HiYaPyCo depends upon PyYAML and Jinja2. Limitations for Jinja2 is from HiYaPyCo project.
- Added req-env.txt, requirements-env.txt - pydotenv, optionally used in deploys.py.
- Added `inspects.has_method`(cls, methodName). Check if class cls has method with name methodName directly,
or in one of it's super-classes.
- Added `pareser.parse_sys_args` function parses command line arguments.
- Added `ymlparsers` module - `load`/`safe_dump` a Hierarchical Yml files. This is essentially wrapper arround HiYaPyCo project with streamlined
and extended API and couple of work-arrounds.
Note: this module doesn't use any package-level variables in hiYaPyCo module, including hiYaPyCo.jinja2env.
This module do use Jinja2's `Environment`.
It also has another defaults for `load`/`safe_dump` methods.
They can be overridden in `initConfig()` function.
`safe_dump()` method supports simple Python objects like primitive types (str, integer, etc), list, dict, **OrderedDict**.
`as_str()`- convenient method for getting str representation of the data,
for example of dict.
`DisableVarSubst` - use of this context manager disables variable substation in the `load()` function.
`initConfig` - this method reset some defaults. If running from the MainThread, this method is idempotent.
- Added `init_app_conf` **major** module.
The main function is `parse_config`. This function parses command line arguments first.
Than it parse yml files. Command line arguments overrides yml files arguments.
Parameters of yml files we always try to convert on best-effort basses.
Parameters of system args we try convert according to `implicit_convert` param.
If you supply `implicit_convert=True`, than `mask_value()` will be applied to the flat map (first parameter).
Otherwise, `implicit_convert` wiil have the value that was set in `intiConfig()`. By default it is `True`.
Command line key --general.profiles or appropriate key default yml file is used to find 'profiles'.
Let suppose, that --config_file is resolved to config.yml.
If 'profiles' is not empty, than it will be used to calculate filenames
that will be used to override default yml file.
Let suppose, 'profiles' resolved to ['dev', 'local']. Than first config.yml
will be loaded, than it will be overridden with config-dev.yml, than
it will be overridden with config-local.yml.
At last, it will be overridden with system args.
This entry can be always be overridden with system args.
`ymlparsers` and `parser` modules serves as Low-Level API for this module.
`mask_value()` implemented as a wrapper to `parsers.safe_eval()` method with support for boolean
variables. This implementation is used to get type for arguments that we get from system args.
This mechanism can be easily replaced with your own one.
`to_convex_map()` This method receives dictionary with 'flat keys', it has simple key:value structure
where value can't be another dictionary.
It will return dictionary of dictionaries with natural key mapping,
optionally, entries will be filtered out according to white_list_flat_keys and,
optionally, value will be implicitly converted to appropriate type.
In order to simulate dictionary of dictionaries 'flat keys' compose key from outer dict with key from inner dict
separated with dot.
For example, 'general.profiles' 'flat key' corresponds to convex map with 'general' key with dictionary as value
that have one of the keys 'profiles' with corresponding value.
If you supply `implicit_convert=True`, than `mask_value()` will be applied to the values of the received flat dictionary.
Otherwise, `implicit_convert` wiil have the value that was set in `intiConfig()`. By default it is True.
`merge_list_value_in_dicts` - merges value of 2 dicts. This value represents list of values.
Value from flat_d is roughly obtained by flat_d[main_key+'.'+sub_key].
Value from d is roughly obtained by d[main_key][sub_key].
If you supply `implicit_convert=True`, than `mask_value()` will be applied to the flat map (first parameter).
Otherwise, `implicit_convert` wiil have the value that was set in `intiConfig()`. By default it is `True`.
`initConfig` - you can set default value of `implicit_convert`. By default it is `True`.
This parameters is used if `implicit_convert` wasn't explicitly supplied. This method is idempotent.
- Added `deploys` module.
This module is usable in your deployment script. See also `fabs` module.
This method use `parsers`, ymlparsers`, `init_app_conf` as it's low-level API. `init_app_conf` usage is limited.
The main function is `load_config()`. It is simplified method for parsing yml configuration file with optionally
overrided profiles only. See `init_app_conf.parse_config()` for another variant.
`split_path` - Split filename in 2 part parts by split_dirname. first_part will ends with split_dirname.
second_part will start immediately after split_dirname.
`add_to_zip_copy_function` - Factory method that returns closure that can be used as copy_function param in
`shutil.copytree()`.
- Added `emails` module.
This module contains extensions of the logging handlers.
This module optionally depends on `ymlparseser` module.
It is better to use `EmailStatus` context manager with configured `emailLogger`.
It is intended to configure first your `emailLogger` with `OneMemoryHandler` (together with `SMTPHandler`).
Than the code block that you want to aggregate log messages from is better to be enclosed with `EmailStatus`
context manager.
`alexber.utils.emails.SMTPHandler` is customization of `logging.handlers.SMTPHandler`. It's purpose is to connect to
SMTP server and actually send the e-mail. Unlike `logging.handlers.SMTPHandler` this class expects for record.msg to be built EmailMessage.
You can also change use of underline SMTP class to SMTP_SSL, LMTP, etc.
This implementation is *thread-safe*.
`alexber.utils.emails.OneMemoryHandler` is variant of `logging.handlers.MemoryHandler`. This handler aggregates
log messages until `FINISHED` log-level is received or application is going to terminate abruptly (see docstring
of `calc_abrupt_vars()` method for the details) and we have some log messages in the buffer. On such event
all messages (in the current Thread) are aggregated to the single `EmailMessage`. The subject of the `EmailMessage`
is determined by `get_subject()` method.
If you want to change delimeters used to indicate variable declaration inside template, see docstring of the
`get_subject()` method.
It is better to use `EmailStatus` context manager with configured emailLogger. See docstring of `EmailStatus`.
This implementation is *thread-safe*.
`alexber.utils.emails.EmailStatus` - if contextmanager exits with exception (it fails), than e-mail with
subject formatted with faildargs and faildkwargs will be send.
Otherwise, e-mail with subject formatted with successargs and successkwargs will be send.
All messages (in the current Thread) will be aggregated to one long e-mail with the subject described in
`OneMemoryHandler.get_subject()` method.
`alexber.utils.emails.initConfig` - this method reset some defaults. This method is idempotent.
By default, `SMTP` class from `smtplib` is used to send actual e-mail. You can change it to `SMTP_SSL`, `LMTP`,
or another class by specifying default_smpt_cls_name.
You can also specified default port for sending e-mails.
`processInvokes` module has one primary function - `run_sub_process()` This method run subprocess and logs it's out
to the logger. This method is sophisticated decorator to `subprocess.run()`. It is useful, when your subprocess
run's a lot of time and you're interesting to receive it's `stdout` and `stderr`. By default, it's streamed to log.
You can easily customize this behavior, see `initConig()` method.
`initConig()` This method can be optionally called prior any call to another function in this module. You can use your
custom class for the logging. For example, FilePipe.
Changed
- Spited dependency list for setup.py req.txt (inexact versions, direct dependency only) and for
reproducible installation requirements.txt (exact versions, all, including transitive dependencies).
- README.md changed, added section 'Alternatively you install install from requirements file:'.
Some other misc changed done.
- CHANGELOG.md version 0.4.1 misc changes.
- Misc improvement in unit tests.
- Fixed `parser.safe_eval` - safe_eval('%(message)s') was blow up, now it returns value as is.
See https://github.com/alex-ber/AlexBerUtils/issues/2
- Enhanced `importer.importer` - added support for PEP 420 (implicit Namespace Packages).
Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk.
When interpreted encounter with non-empty __path__ attribute it adds modules found in those locations
to the current package.
See https://github.com/alex-ber/AlexBerUtils/issues/3
- In all documentation refference to `pip3` was changed to `python3 -m pip`