Omegaml

*Key features*
- data revisions enable automated revision tracking for pandas Dataframes
- task callbacks provide easy to use follow up task handling for the remote runtime
- csv, image and binary files handling is now supported by the cli

*changes*
faf6f31 release support for tenant deployment in commercial edition (173)
fcb7102 Wip (166)
062593f enable apphub remote cluster at different url from restapi (170)
f0bfd55 replace flaskrestplus, enable better cli support for csv, image and binary files (172)
5440ad2 enable automated data revisions for dataframes (171)
94902a4 enable task callbacks (157)
98b204a add improvements for job.map() (162)
59eb185 document release process (161)
02cd1ba Python39 (167)
d7a6301 restrict license to No Sell, Consulting Yes clause (165)
d824359 Rename NOSELLCLAUSE to LICENSE-NOSELLCLAUSE (164)
03962ea Create NOSELLCLAUSE (163)
567f15f Improve docs (158)

new features and enhancements

* 136 cli `om cloud` enables live status on k8s-provided cloud resources such as nodes, pods, storage & to query prometheus for metrics
* 142 cli `om runtime env` allows remote update of worker's pip venv
* 143 enables easy task chaining, obsoleting https://gist.github.com/omegaml/c5a8cb3b08fcc38d7dca34ed272f7528
* 146 integrates parallel notebook tasks as a runtime plugin, obsoleting https://gist.github.com/omegaml/08dc548d25f0ab6e8027c79af18bc07d
* 152 simplifies loading the omegaml cloud client based on environment variables and/or configuration files
* 155 cli `om runtime` provides easier access to the celery cluster status, allows calling celery commands directly

backwards-incompatible changes

* 140 fixes MDataFrame filter operators (lt, lte, gt, gte), potentially breaking existing code
* 152 config.yml now only exposes OMEGA_USERID, OMEGA_APIKEY and OMEGA_RESTAPI_URL
* 154 introduces a new naming scheme for mongodb collections using md5-hashed names to support arbitrary names. Old datasets continue to work unchanged. To migrate the naming scheme run ` omegaml.util.migrate_unhashed_datasets(om.datasets)`

changes

f18d841 Release 0.13.7 (160)
845d99e enable dataset names of arbitrary length (154)
6eb811f simplify access to celery commands via om runtime (155)
50e2966 Enhance base loading (152)
8c3a39c ensure compatibility with omegaee (151)
ed12d19 fix command line parsing, cloud config (150)
d02b618 Revert "Upgrade dependencies (148)" (149)
1788afc Upgrade dependencies (148)
f6e967b enable parallel notebook tasks (146)
72495e9 Refactor cli (144)
5149735 update docs (118)
9b9dca5 add task chaining (143)
b4c780a enable worker utilities (142)
f7dd35a fix filterops lt, lte, gt, gte (140)
ddb7974 enable cli cloud status report and pod log access (136)
7e3cc87 return the users value instead if valid (138)
2e81c32 various improvements (132)
0fec919 various improvements (131)
a37eebe several stability updates and bug fixes (126)
fea89c2 Release build (124)

simplify/enable minibatch streams, pip+git sourced scripts, tensorflow 2.3, versioned model promotion, sql-dbms namespaces, bug fixes

Examples:


$ om shell
stream producer, benchmarked to 60K messages/s
[] stream = om.streams.get('astream')
stream.append({'foo': 'bar'])

stream consumer with window size = 2, windows are processed in parallel
[] streaming = om.streams.getl('astream', size=2)
streaming(lambda window: print(window)) => print every 2 messages

pip+git sources scripts
[] om.scripts.put('pypi://package', 'mymodule')
om.scripts.put('git+https://url/repo/package', 'mymodule')
om.runtime.script('mymodule').run()

sql namespacing using buckets, like with MongoDB
[] -- the default table when storing data is {bucket}_{name}, override using table='myname'
om.datasets.put(sqlalchemy_constr, 'mysqlalchemy', table='mytable', sql='select ...', copy=True)
-- to use a specific table, without bucket information use table=':myname'
om.datasets.put(sqlalchemy_constr, 'mysqlalchemy', table=':mytable', sql='select ...', copy=True)
om.datasets.get('mysqlalchemy') read from myname

Changes:
684f83d fixing several bugs (123)
841e19c enable combined stores (120)
b7b732e add streams as a first-class object type (121)
da5675f various improvements (119)
0a2db44 Simplify releases (122)
b322a11 fix mdataframe.merge duplicate keys error on 1:n merges (117)
9da78b5 enable bucket namespacing for sql datasources (116)
7a03043 support omegamlee-1.4.1 (115)
5c6d56d enable remote source packages (113)
22ffcb2 updates for stability and increased performance (111)
c3e9e5e enable tensorflow 2.3.0 (unit testing) (110)
b20e8c6 enable promotion of versioned models to work as expected (109)
89bdb96 various updates, stability improvements (105)

This release introduces many productivity-increasing features, as well as fixing some bugs

Major extensions

* arbitrary large datasets to scikit-learn online models by using a simple `` in the dataset name, e.g. `om.runtime.model('mymodel').partial_fit('data[x]', 'datay[y]`), use `iterchunks:chunksize=N` to specify the batch size
* extend REST API for all model tasks, not just predict (now supports all methods like fit, predict, partial_fit, score, transform etc)
* extend REST API for notebooks, tasks and scripts
* async REST APIs for all long-running tasks like notebooks, tasks, scripts, model fitting and prediction
* enable referencing arbitrary datasets from external URIs, return a Pandas Dataframe on access
* direct export of raw files to local files

Changes

e4c3ca8 apply several fixes, bump version
cbd2ab6 fix wrong argument passing in maybe_chunked (103)
4942a9f enable implicit chunking by yielding an iterable from dataset name (102)
c708114 enable kwargs for smartopen, supporting az blob storage (99)
6566911 enable async rest api (98)
c646673 add externaldata backend, make jobs run return metadata repr, rawfiles enhancement (97)
c6d8d5d improve tensorflow 2.x compatibility (100)

this release enables various managed services features that are not yet part of the open source core

* apphub: deploy flask, dash and minibatch streaming apps
* gpu enabled tensorflow and pytorch support
* upgrade to jupyterlab

changes

2a0aa21 version bump (96)
aa243ce ensure rname works the same everywhere (91)
fad7054 various stability improvements, tensorflow 2.0 support (76)
808b53b enable storing dash apps (89)

release summary

2d159da refactor file storage (87)
d55638e enable automated model versioning (65)
85fa1ef Integrate omxiotools (86)
8e0693f add mdataframe native pandas parallel transform (85)
7a4fd2e bump version, fix formatted log messages (83)
8a67bee fix windows ctrl-c sighup handling (84)