Dcicutils

Latest version: v8.11.0

Safety actively analyzes 638346 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 19 of 25

2.2.0

=====

* In ``cloudformation_utils``:

* Add ``hyphenify`` to change underscores to hyphens.

* In ``command_utils``:

* Add ``shell_script`` context manager and its implementation class ``ShellScript``.

* Add ``module_warnings_as_ordinary_output`` to help work around the problem that S3Utils outputs
text we'd sometimes rather see as ordinary output, not log output.

* In ``lang_utils``:

* Add support for ``string_pluralize`` to pluralize 'nouns' that have attached prepositional phrases, as in::

string_pluralize('file to load')
'files to load`

string_pluralize('brother-in-law of a proband')
'brothers-in-law of probands'

string_pluralize('brother-in-law of the proband')
'brothers-in-law of the proband'

But, importantly, this also means one can give have arguments to functions that use these do something
sophisticated in terms of wording with almost no effort at the point of need, such as::

[there_are(['foo.json', 'bar.json'][:n], kind='file to load') for n in range(3)]
[
'There are no files to load.',
'There is 1 file to load: foo.json',
'There are 2 files to load: foo.json, bar.json'
]

[n_of(n, 'bucket to delete') for n in range(3)]
[
'0 buckets to delete',
'1 bucket to delete',
'2 buckets to delete'
]

* Miscellaneous other changes:

* In ``docs/source/dcicutils.rst``, add autodoc for various modules that are not getting documented.

* In ``test/test_misc.py``, add unit test to make sure things don't get omitted from autodoc.

Specifically, a test will now fail if you make a new file in ``dcicutils`` and do not add a
corresponding autodoc entry in ``docs/source/dcicutils.rst``.

2.1.0

=====

* In ``s3_utils``, add various variables that can be used to assure values are synchronized across 4DN/CGAP products:

* Add new slots on ``s3Utils`` to hold the token at the end of each kind of bucket:

* ``s3Utils.SYS_BUCKET_SUFFIX == "system"``
* ``s3Utils.OUTFILE_BUCKET_SUFFIX == "wfoutput"``
* ``s3Utils.RAW_BUCKET_SUFFIX == "files"``
* ``s3Utils.BLOB_BUCKET_SUFFIX == "blobs"``
* ``s3Utils.METADATA_BUCKET_SUFFIX == "metadata-bundles"``
* ``s3Utils.TIBANNA_OUTPUT_BUCKET_SUFFIX == 'tibanna-output'``

* Add new slots on ``s3Utils`` for various bits of connective glue in setting up the template slots:

* ``s3Utils.EB_PREFIX == "elasticbeanstalk"``
* ``s3Utils.EB_AND_ENV_PREFIX == "elasticbeanstalk-%s-"``

* Add new slots on ``s3Utils`` for expected keys on a health page corresponding to each kind of bucket:

* ``s3Utils.SYS_BUCKET_HEALTH_PAGE_KEY == 'system_bucket'``
* ``s3Utils.OUTFILE_BUCKET_HEALTH_PAGE_KEY == 'processed_file_bucket'``
* ``s3Utils.RAW_BUCKET_HEALTH_PAGE_KEY == 'file_upload_bucket'``
* ``s3Utils.BLOB_BUCKET_HEALTH_PAGE_KEY == 'blob_bucket'``
* ``s3Utils.METADATA_BUCKET_HEALTH_PAGE_KEY == 'metadata_bundles_bucket'``
* ``s3Utils.TIBANNA_OUTPUT_BUCKET_HEALTH_PAGE_KEY == 'tibanna_output_bucket'``

* In ``deployment_utils``, use new variables from ``s3_utils``.

2.0.0

=====

`PR 150: Add json_leaf_subst, conjoined_list and disjoined_list <https://github.com/4dn-dcic/utils/pull/150>`_

We do not believe this is an incompatible major version, but there is a lot here, an hence some opportunity for
difference in behavior to have crept in. As such, we opted to call this a new major version to highlight where
that big change happened.

* In ``beanstalk_utils``:

* Add ``'elasticbeanstalk-%s-metadata-bundles'`` to the list of buckets that ``beanstalk_utils.delete_s3_buckets``
is willing to delete.

* In ``cloudformation_utils``:

* New functions ``camelize`` and ``dehyphenate`` because they're needed a lot in our ``4dn-cloud-infra`` repo.

* New implementation of functions ``get_ecs_real_url`` and ``get_ecr_repo_url`` that are not Alpha-specific.

* New classes ``AbstractOrchestrationManager``, ``C4OrchestrationManager``, and ``AwsemOrchestrationManager``
with various utilities ported from ``4dn-cloud-infra`` (so they could be used to re-implement
``get_ecs_real_url``and ``get_ecr_repo_url``).

* New ``test_cloudformation_utils.py`` testing each of the bits of functionality in ``cloudformation_utils``
along normal paths, including sometimes mocking both the Alpha and KMP environments, hoping transitions
will be smooth.

* In ``deployment_utils``:

* Support environment variable ``ENCODED_IDENTITY`` and ``--identity`` to control
environment variable ``$IDENTITY`` in construction of ``production.ini``.

* Support environment variable ``ENCODED_TIBANNA_OUTPUT_BUCKET`` and ``--tibanna_output_bucket`` to control
environment variable ``$TIBANNA_OUTPUT_BUCKET`` in construction of ``production.ini``.

* Support environment variable ``ENCODED_APPLICATION_BUCKET_PREFIX`` and ``--application_bucket_prefix`` to control
environment variable ``$APPLICATION_BUCKET_PREFIX`` in construction of ``production.ini``.

* Support environment variable ``ENCODED_FOURSIGHT_BUCKET_PREFIX`` and ``--foursight_bucket_prefix`` to control
environment variable ``$FOURSIGHT_BUCKET_PREFIX`` in construction of ``production.ini``.

* New class variable ``APP_KIND`` in ``IniFileManager``.
Default is ``None``, but new subclasses adjust the default to ``cgap`` or ``fourfront``.

* New class variable ``APP_ORCHESTRATED`` in ``IniFileManager``.
Default is ``None``, but new subclasses adjust the default to ``True`` or ``False``.

* New classes

* ``BasicCGAPIniFileManager``
* ``BasicLegacyCGAPIniFileManager``
* ``BasicOrchestratedCGAPIniFileManager``
* ``BasicFourfrontIniFileManager``
* ``BasicLegacyFourfrontIniFileManager``
* ``BasicOrchestratedFourfrontIniFileManager``

In principle, this should allow some better defaulting.

* In ``exceptions``:

* Add ``InvalidParameterError``.

* In ``lang_utils``:

* Add ``conjoined_list`` and ``disjoined_list`` to get a comma-separated
list in ordinary English form with an "and" or an "or" before the
last element. (Note that these also support new functions
``there_are`` and ``must_be_one_of``).

* Add ``there are`` and ``must_be_one_of`` to handle construction of
messages that are commonly needed but require nuanced adjustment of
wording to sound right in English. (Note that ``must_be_one_of`` also
supports ``InvalidParameterError``.)

* In ``misc_utils``:

* Add ``json_leaf_subst`` to do substitutions at the leaves
(atomic parts) of a JSON object.

* Add ``NamedObject`` for creating named tokens.

* Add a ``separator=`` argument to ``camel_case_to_snake_case`` and ``snake_case_to_camel_case``.

* In ``qa_utils``, support for mocking enough of ``boto3.client('cloudformation')`` that we can test
``cloudformation_utils``. The ``MockBoto3Client`` was extended, and several mock classes were added,
but most importantly:

* ``MockBotoCloudFormationClient``
* ``MockBotoCloudFormationStack``
* ``MockBotoCloudFormationResourceSummary``

* In ``s3_utils``:

* Make initialize attribute ``.metadata_bucket`` better.

* Add an attribute ``.tibanna_output_bucket``

1.20.0

======

`PR 148: Support auth0 client and secret in deployment_utils <https://github.com/4dn-dcic/utils/pull/148>`_

* In ``deployment_utils``, add support for managing auth0 client and secret:

* To pass client and secret into the ini file generator:

* ``--auth0_client`` and ``--auth0_secret`` command line arguments.
* ``$ENCODED_AUTH0_CLIENT`` and ``ENCODED_AUTH0_SECRET`` as environment variables.

* Ini file templates can just use ``AUTH0_CLIENT`` and ``AUTH0_SECRET`` to obtain a properly defaulted value.
It is recommended to put something like this in the ini file template::

auth0.client = ${AUTH0_CLIENT}
auth0.secret = ${AUTH0_SECRET}

1.19.0

======

`PR 147: Init s3Utils via GLOBAL_ENV_BUCKET and misc S3_BUCKET_ORG support (C4-554) <https://github.com/4dn-dcic/utils/pull/147>`_
`PR 146: Better S3 bucket management in deployment_utils <https://github.com/4dn-dcic/utils/pull/146>`_

* In ``cloudformation_utils``:

* Small bug fix to ``get_ecs_real_url``.

* Add ``get_ecr_repo_url``.

* In ``deployment_utils``:

* Add environment variables that can be set per stack/instance:

* ``ENCODED_S3_BUCKET_ORG`` - a unique token for your organization to be used in auto-generating S3 bucket orgs.
The defaulted value (which includes possible override by a ``--s3_bucket_org`` argument in the generator command)
will be usable as ``${S3_BUCKET_ORG}`` in ``.ini`` file templates.

* ``ENCODED_S3_BUCKET_ENV`` - a unique token for your organization to be used in auto-generating S3 bucket names.
The defaulted value (which includes possible override by a ``--s3_bucket_env`` argument in the generator command)
will be usable as ``${S3_BUCKET_ENV}`` in ``.ini`` file templates.

* ``ENCODED_FILE_UPLOAD_BUCKET`` - the name of the file upload bucket to use if a ``--file_upload_bucket`` argument
is not given in the generator command, and the default of ``${S3_BUCKET_ORG}-${S3_BUCKET_ENV}-files``
is not desired. This fully defaulted value will be available as ``${FILE_UPLOAD_BUCKET}`` in ``.ini`` file
templates, and is the recommended way to compute the proper value for the ``file_upload_bucket`` configuration
parameter.

* ``ENCODED_FILE_WFOUT_BUCKET`` - the name of the file wfout bucket to use if a ``--file_wfout_bucket`` argument
is not given in the generator command, and the default of ``${S3_BUCKET_ORG}-${S3_BUCKET_ENV}-wfoutput``
is not desired. This fully defaulted value will be available as ``${FILE_WFOUT_BUCKET}`` in ``.ini`` file
templates, and is the recommended way to compute the proper value for the ``file_wfout_bucket`` configuration
parameter.

* ``ENCODED_BLOB_BUCKET`` - the name of the blob bucket to use if a ``--blob_bucket`` argument
is not given in the generator command, and the default of ``${S3_BUCKET_ORG}-${S3_BUCKET_ENV}-blobs``
is not desired. This fully defaulted value will be available as ``${BLOB_BUCKET}`` in ``.ini`` file
templates, and is the recommended way to compute the proper value for the ``blob_bucket`` configuration
parameter.

* ``ENCODED_SYSTEM_BUCKET`` - the name of the system bucket to use if a ``--system_bucket`` argument
is not given in the generator command, and the default of ``${S3_BUCKET_ORG}-${S3_BUCKET_ENV}-system``
is not desired. This fully defaulted value will be available as ``${SYSTEM_BUCKET}`` in ``.ini`` file
templates, and is the recommended way to compute the proper value for the ``system_bucket`` configuration
parameter.

* ``ENCODED_METADATA_BUNDLES_BUCKET`` - the name of the metadata bundles bucket to use if a
``--metadata_bundles_bucket`` argument is not given in the generator command, and the default of
``${S3_BUCKET_ORG}-${S3_BUCKET_ENV}-metadata-bundles`` is not desired. This fully defaulted value will be
available as ``${METADATA_BUNDLES_BUCKET}`` in ``.ini`` file
templates, and is the recommended way to compute the proper value for the ``metadata_bundles_bucket`` configuration
parameter.

* Fixed a bug that the index_server argument was not being correctly passed into lower level functions when
``--index_server`` was specified on the command line.

* Fixed a bug where passing no ``--encoded_data_set`` but an explicit null-string value of the environment variable
``ENCODED_DATA_SET`` did not lead to further defaulting in some circumstances.

* In ``ff_utils``:

* Add ``fetch_network_ids``.

* In ``misc_utils``:

* Add ``dict_zip``.

* In ``s3_utils``:

* Add new methods ``fetch_health_page_json`` and ``verify_and_Get_env_config`` in support of new initialization
protocol for ``s3Utils``.

* Extend ``s3Utils`` initialization protocol so that under certain conditions,
environment variable if ``GLOBAL_ENV_BUCKET`` is set,
the init protocol will be discovered from that bucket.

NOTE WELL: The name ``GLOBAL_BUCKET_ENV`` is also supported as a synonm for ``GLOBAL_ENV_BUCKET``
because it was used in testing before we settled on a final name, and we're allowing a
grace period. But this name should not be considered properly supported. That it works now
is a courtesy and anyone concerned about incompatible changes should use the newer name,
``GLOBAL_ENV_BUCEKT``.

1.18.1

======

`PR 145: Fix internal import problems <https://github.com/4dn-dcic/utils/pull/145>`_

* Make ``lang_utils`` import ``ignored`` from ``misc_utils``, not ``qa_utils``.
* Make ``deployment_utils`` import ``override_environ`` from ``misc_utils``, not ``qa_utils``.
* Move ``local_attrs`` from ``qa_utils`` to ``misc_utils``
so that similar errors can be avoided in other libraries that import it.

Page 19 of 25

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.