Aiootp

--------------------------


Minor Changes
^^^^^^^^^^^^^

- Made the output lengths of the ``Padding`` class' generator functions
uniform. When the footer padding on a stream of plaintext needs to
exceed the 256-byte blocksize (i.e. when the last unpadded plaintext
block's length ``L`` is ``232 < L < 256``), then another full block of
padding is produced. The generators now yield 256-byte blocks
consistently (except during depadding when the last block of plaintext
may be smaller than the blocksize), instead of sometimes producing a
final padded block which is 512 bytes.

--------------------------


Minor Changes
^^^^^^^^^^^^^

- Fixed a bug where database classes were evaluating as falsey when they
didn't have any tags saved in them. They should be considered truthy
if they're instantiated & ready to store data, even if they're
currently empty & not saved to disk. This was reflected in their
``__bool__`` methods. The bug caused empty metatags not to be loaded
when an instance loads, even when ``preload`` is toggled ``True``.
- Removed the coroutine-receiving logic from the ``Padding`` class'
``Comprende`` generators. Since they buffer data, the received values
aren't ever going to coincide with the correct iteration & will be
susceptible to bugs
- Fixed a bug in the ``Padding`` class' ``Comprende`` generators which
cut iteration short because not enough data was available from the
underlying generators upfront. Now, if used correctly to pad/depad
chunks of plaintext 256 bytes at a time, then they work as expected.
- The ``update``, ``aupdate``, ``update_key`` & ``aupdate_key`` methods
in both the ``StreamHMAC`` & ``DomainKDF`` classes now return ``self``
to allow inline updates.
- Added ``acsprng`` & ``csprng`` function pointers to the ``Chunky2048``
class.
- Updates to docstrings which didn't get updated with info on the new
*synthetic IV* feature.
- Some other docstring fixes.
- Some small code cleanups & refactorings.

--------------------------


Major Changes
^^^^^^^^^^^^^

- Security Upgrade: The package's cipher was changed to an online,
authenticated scheme with salt misuse-reuse resistance. This was
acheived through a few backwards incompatible techniques:

1. A synthetic IV (SIV) is calculated from the keyed-hash of the first
256-byte block of plaintext. The SIV is then used to seed the
keystream generator, & is used to update the validator object. This
ensures that if the first block is unique, then the whole ciphertext
will be unique.
2. A 16-byte ephemeral & random SIV-key is also prepended to the
first block of plaintext during message padding. Since this value
is also hashed to derive the SIV, this key gives a strong
guarantee that a given message will produce a globally unique
ciphertext.
3. An 8-byte timestamp is prepended to the first block of plaintext
during padding. Timestamps are inherently sequential, they can be
verified by a user within some bounds, & can also be used to
mitigate replay attacks. Since it's hashed to make the SIV, then
it helps make the entire ciphertext unique.
4. After being updated with each block of ciphertext, the validator's
current state is again fed into the keystream generator as a new
rotating seed. This mitigation is limited to ensuring only that
every following block of ciphertext to a block which is unique
will also be unique. More specifically this means that: **if**
*all* **other mitigations fail to be unique**, or are missing, then
the first block which is unique **will appear the same**, except
for the bits which have changed, **but, all following blocks will
be randomized.** This limitation could be avoided with a linear
expansion in the ciphertext size by generating an SIV for each
block of plaintext. This linear expansion is prohibitive as a
default setting, but the block level secrecy, even when all other
mitigations fail, is enticing. This option may be added in the
future as a type of padding mode on the plaintext.

The SIV-key is by far the most important mitigation, as it isn't
feasibly forgeable by an adversary, & therefore also protects against
attacks using encryption oracles. These changes can be found in the
``SyntheticIV`` class, the (en/de)cipher & xor generators, & the
``StreamHMAC`` class in the ``ciphers.py`` module. The padding
changes can also be found in the new ``Padding`` class in the ``generics.py``
module. The SIV is attached in the clear with ciphertexts & was
designed to function with minimal user interaction. It needs only to
be passed into the ``StreamHMAC`` class during decryption -- during
encryption it's automatically generated & stored in the ``StreamHMAC``
validator object's ``siv`` property attribute.
- Security Patch: The internal ``sha3_512`` kdf's to the ``akeys``, ``keys``,
``abytes_keys`` & ``bytes_keys`` keystream generators are now updated
with 72 bytes of (64 key material + 8 padding), instead of just 64
bytes of key material. 72 bytes is the *bitrate* of the ``sha3_512``
object. This change causes the internal state of the object to be permuted
for each iteration update & before releasing a chunk of key material.
Frequency analysis of ciphertext bytes didn't smooth out to the
cumulative distribution expected for all large ciphertexts prior to
this change. But after the change the distribution does normalize as
expected. This indicates that the key material streams were biased
away from random in a small but measurable way. Although, no
particular byte values seem to have been preferred by this bias, this
is a huge shortcoming with unknown potential impact on the strength
of the package's cipher. This update is strongly recommended & is
backwards incompatible.
- This update gives a name to the package's pseudo-one-time-pad cipher
implementation. It's now called ``Chunky2048``! The ``OneTimePad``
class' name was updated to ``Chunky2048`` to match the change.
- The ``PreemptiveHMACValidation`` class & its related logic in the
``StreamHMAC`` class was removed. The chaining of validator output
into the keystream makes running the validator over the ciphertext
separately or prior to the decryption process very costly. It would
either mean recalculating the full hash of the ciphertext a second
time to reproduce the correct outputs during each block, or a large
linear memory increase to hold all of its digests to be fed in some
time after preemtive validation. It's much simpler to remove that
functionality & potentially replace it with something else that fits
the user's applications better. For instance, the ``current_digest``
& ``acurrent_digest`` methods can produce secure, 32-byte authentication
tags at any arbitrary blocks throughout the cipher's runtime, which
validate the cipehrtext up to that point. Or, the ``next_block_id``
& ``anext_block_id`` methods, which are a more robust option because
each id they produce validates the next ciphertext block before
updating the internal state of the validator. This acts as an
automatic message ordering algorithm, & leaves the deciphering
party's state unharmed by dropped packets or manipulated ciphertext.
- The ``update_key`` & ``aupdate_key`` methods were also added to the
``StreamHMAC`` class. They allow the user to update the validators'
internal key with new entropy or context information during its
runtime.
- The ``Comprende`` class now takes a ``chained`` keyword-only argument
which flags an instance as a chained generator. This flag allows
instances to communicate up & down their generator chain using the
shared ``Namespace`` object accessible by their ``messages`` attribute.
- The chainable ``Comprende`` generator functions had their internals
altered to allow them to receive, & pass down their chain, values
sent from a user using the standard coroutine ``send`` & ``asend``
method syntax.
- ``Comprende`` instances no longer automatically reset themselves every
time they enter their context managers or when they are iterated over.
This makes their interface more closely immitate the behavior of
async/sync generator objects. To get them to reset, the ``areset`` or
``reset`` methods must be used. The message chaining introduced in
this update allows chains of ``Comprende`` async/sync generators to
inform each other when the user instructs one of them to reset.
- The standard library's ``hmac`` module is now used internally to the
``generics.py`` module's ``sha_512_hmac``, ``sha_256_hmac``, ``asha_512_hmac``
& ``asha_256_hmac`` functions. They still allow any type of data to be
hashed, but also now default to hashing ``bytes`` type objects as
they are given.
- The new ``Domains`` class, found in ``generics.py``, is now used to
encode constants into deterministic pseudo-random 8-byte values for
helping turn hash function outputs into domain-specific hashes. Its
use was included throughout the library. This method has an added
benefit with respect to this package's usage of SHA-3. That being, the
*bitrate* for both ``sha3_512`` & ``sha3_256`` are ``(2 * 32 * k) + 8``
bytes, where ``k = 1`` for ``sha3_512`` & ``k = 2`` for ``sha3_256``.
This means that prepending an 8-byte domain string to their inputs
also makes it more efficient to add some multiple of key material
to make the input data precisely equal the *bitrate*. More info on
domain-specific hashing can be found here_.

.. _here: https://eprint.iacr.org/2020/241.pdf

- A new ``DomainsKDF`` class in ``cipehrs.py`` was added to create a
more standard & secure method of key derivation to the library which
also incorporates domain separation. Its use was integrated thoughout
the ``AsyncDatabase`` & ``Database`` classes to mitigate any further
vulnerabilities of their internal key-derivation functions. The
database classes now also use bytes-type keys internally, instead
of hex strings.
- The ``Passcrypt`` class now contains methods which create & validate
passcrypt hashes which have their settings & salt attached to them.
Instances can now also be created with persistent settings that are
automatically sent into instance methods.


Minor Changes
^^^^^^^^^^^^^

- Many fixes of docstrings, typos & tutorials.
- Many refactorings: name changes, extracted classes / functions,
reorderings & moves.
- Various code clean-ups, efficiency & usability improvements.
- Many constants used throughout the library were given names defined
in the ``commons.py`` module.
- Removed extraneous functions throughout the library.
- The asymmetric key generation & exchange functions/protocols were
moved from the ``ciphers.py`` module to ``keygens.py``.
- Add missing modules to the MANIFEST.rst file.
- Added a ``UniformPrimes`` class to the ``__datasets`` module for efficient
access to primes that aren't either mostly 1 or 0 bits, as is the case for
the ``primes`` helper table. These primes are now used in the ``Hasher``
class' ``amask_byte_order`` & ``mask_byte_order`` methods.
- The ``time_safe_equality`` & ``atime_safe_equality`` methods are now
standalone functions available from the ``generics.py`` module.
- Added ``reset_pool`` to the ``Processes`` & ``Threads`` classes. Also
fixed a missing piece of logic in their ``submit`` method.
- Added various conversion values & timing functions to the ``asynchs.py``
module.
- The ``make_uuid`` & ``amake_uuid`` coroutines had their names changed to
``make_uuids`` & ``amake_uuids``.
- Created a new ``Datastream`` class in ``generics.py`` to handle buffering
& resizing iterable streams of data. It enables simplifying logic that
must happen some number of iterations before the end of a stream. It's
utilized in the ``Padding`` class' generator functions available as
chainable ``Comprende`` methods.
- The ``data`` & ``adata`` generators can now produce a precise number of
``size``-length ``blocks`` as specified by a user. This gets rid of the
confusing usage of the old ``stop`` keyword-only argument, which stopped
a stream after *approximately* ``size`` number of elements.
- Improved the efficiency & safety of entropy production in the
``randoms.py`` module.

--------------------------


Major Changes
^^^^^^^^^^^^^

- Security Patch: Deprecated & replaced an internal kdf for saving
database tags due to a vulnerability. If an adversary can get a user
to reveal the value returned by the ``hmac`` method when fed the tag
file's filename & the salt used for that encrypted tag, then they
could deduce the decryption key for the tag. A version check was
added only for backwards compatibility & will be removed on the next
update. All databases should continue functioning as normal, though
all users are advised to **re-save their databases** after upgrading
so the new kdf can be used. This will not overwrite the old files,
so they'll need to be deleted manually.
- Replaced usage of the async ``switch`` coroutine with ``asyncio.sleep``
because it was not allowing tasks to switch as it was designed to.
Many improvements were made related to this change to make the
package behave better in async contexts.
- Removed the private method in the database classes which held a
reference to the root salt. It's now held in a private attribute.
This change simplifies the code a bit & allows instances to be
pickleable.
- The ``atimeout`` & ``timeout`` chainable ``Comprende`` generator
methods can now stop the generators' executions mid-iteration. They
run them in separate async tasks or thread pools, respectively, to
acheive this.
- The ``await_on`` & ``wait_on`` generators now restart their timeout
counters after every successful iteration that detected a new value
in their ``queue``. The ``delay`` keyword argument was changed to
``probe_frequency``, a keyword-only argument.
- Removed the package's dependency on the ``aioitertools`` package.
- Made the ``sympy`` package an optional import. If any of its
functionalities are used by the user, the package is only then
imported & this is done automatically.
- Various streamlining efforts were made to the imports & entropy
initialization to reduce the package's import & startup time.


Minor Changes
^^^^^^^^^^^^^

- Fixes of various typos, docstrings & tutorials.
- Various cleanups, refactorings & efficiency improvements.
- Added new tests for detecting malformed or modified ciphertexts.
- Removed extraneous functions in ``generics.py``.
- Add a ``UNIFORM_PRIME_512`` value to ``__datasets.py`` for use in the
``Hasher.mask_byte_order`` & ``Hasher.amask_byte_order`` methods.
Those methods were also altered to produce more uniform looking
results. The returned masked values are now also 64 bytes by default.
- Added an ``automate_key_use`` keyword-only boolean argument to the init
for the ``OneTimePad``, ``Keys`` & ``AsyncKeys`` classes. It can be toggled to
stop the classes from overwriting class methods so they
automatically read the instance's key attribute. This optionally
speeds up instantiation by an order of magnitude at the cost of
convenience.
- Fixed ``asynchs.Threads`` class' wrongful use of a ``multiprocessing``
``Manager.list`` object instead of a regular list.
- Changed the ``_delay`` keyword-only argument in ``Processes`` & ``Threads``
classes' methods to ``probe_freqeuncy`` so users can specify how often
results will be checked for after firing off a process, thread, or
associated pool submission.
- Now the ``asubmit`` & ``submit`` methods in ``Processes`` & ``Threads``
can accept keyword arguments.
- Added ``agather`` & ``gather`` methods to the ``Threads`` & ``Processes``
classes. They receive any number of functions, & ``args`` &/or ``kwargs`` to
pass to those functions when submitting them to their associated
pools.
- Changed the ``runsum`` instance IDs from hex strings to bytes & cleaned
up the instance caching & cleaning logic.
- Altered & made private the ``asalted_multiply`` & ``salted_multiply``
functions in the ``randoms.py`` module.
- Started a new event loop specific to the ``randoms.py`` module which
should prevent the ``RuntimeError`` when ``random_number_generator``
is called from within the user's running event loop.
- Added a ``ValueError`` check to the ``(a)cspr(b/n)g`` functions in
``randoms.py``. This will allow simultaneously running tasks to
request entropy from the function by returning a result from a
newly instantiated generator object.
- Added checks in the ``*_encipher`` & ``*_decipher`` generators to
help assure users correctly declare the mode for their StreamHMAC
validator instances.
- Fixed the ``__len__`` function in the database classes to count the
number of tags in the database & exclude their internal maintenaince
files.
- The ``TimeoutError`` raised after decrypting a ciphertext with an
expired timestamp now contains the seconds it has exceeded the ``ttl``
in a ``value`` attribute.
- The timestamp used to sign the package now displays the day of
signing instead of the second of signing.
- The ``(a)sum_sha_*`` & ``(a)sum_passcrypt`` generators were altered to
reapply the supplied ``salt`` on every iteration.
- Stabilized the usability of the ``stop`` keyword-only argument in the
``adata`` & ``data`` generators. It now directly decides the total
number of elements in a ``sequence`` allowed to be yielded.

--------------------------


Major Changes
^^^^^^^^^^^^^

- Security Patch: Rewrote the HMAC-like creation & authentication
process for all of the package's ciphers. Now, the ``*_encipher``
& ``*_decipher`` ``Comprende`` generators must be passed a validator
object to hash the ciphertext as it's being created / decrypted.
The ``StreamHMAC`` class was created for this purpose. It's initalized
with the user's long-term key, the ephemeral salt & the pid value.
The pid value can now effectively be used to validate additional data.
These changes force the package's cipher to be used as an AEAD cipher.
- Security Patch: The package's ``*_hmac`` hash functions & the ``Comprende``
class' hash generators were rewritten to prepend salts & keys to data
prior to hashing instead of appending. This is better for several
important reasons, such as: reducing the amortizability of costs in
trying to brute-force hashes, & more closely following the reasoning
behind the HMAC spec even though sha3 has a different security profile.
- Algorithm Patch: The ``akeys``, ``keys``, ``abytes_keys``, & ``bytes_keys``
algorithms have been patched to differentiate each iteration's two
sha3_512 hashes from one another in perpetuity. They contained a design
flaw which would, if both sha3_512 objects landed upon the same
1600-bit internal state, then they would produce the same keystreams
from then on. This change in backwards incompatible. This flaw is
infeasible to exploit in practice, but since the package's hashes &
ciphertext validations were already channging this release, there was
no reason to not fix this flaw so that it's self-healing if they ever
do land on the same internal states.
- The ``Passcrypt`` class & its algorithm were made more efficient to
better equalize the cost for users & adversaries & simplifies the
algorithm. Any inefficiencies in an implementation would likely cause
the adversary to be able to construct optimized implementations to
put users at an even greater disadvantage at protecting their inputs
to the passcrypt algorithm. It used the ``sum_sha_256`` hash function
internally, & since it was also changing in a non-backwards
compatible way with this update, it was the best time to clean up
the implementation.
- Updated the package's description & its docstrings that refer to
the package's cipher as an implementation of the one-time-pad. It's
not accurate since the package uses pseudo-random hash functions to
produce key material. Instead, the package's goal is to create a
pseudo-one-time-pad that's indistinguishable from a one-time-pad.
The ``OneTimePad`` class will keep its name for succinctness.
- New ``amake_token``, ``make_token``, ``aread_token`` & ``read_token``
class & instance methods added to the ``OneTimePad`` class. These
tokens are urlsafe base64 encoded, are encrypted, authenticated &
contain timestamps that can enforce a time-to-live for each token.
- Non-backwards compatible changes to the database classes' filenames,
encryption keys & HMACs. The ``*_hmac`` hash functions that the
databases rely on were changing with this update, so additionally the
filenames table used to encode the filenames was switched from the
``BASE_36_TABLE`` to the ``BASE_38_TABLE``. Both tables are safe for
uri's across all platforms, but the new table can encode information
slightly more efficiently.
- Major refactorings & signature changes across the package to make
passing keys & salts to ``*_hmac`` functions & the ``Comprende``
class' hash generators explicit.
- Removed the ``of`` keyword argument from all of the ``Comprende``
class' generators. It was overly complicating the code, & was not
entirely clear or useful for settings outside of the ``tags`` &
``atags`` generators.
- Removed ``pybase64`` from the package & its dependencies list. The
built-in python ``base64`` module works just fine.
- Sorted the ``WORDS_LIST``, ``ASCII_ALPHANUMERIC``, & ``BASE_64_TABLE``
datasets.
- The ``salt`` & ``asalt`` functions have been renamed to ``generate_salt``
& ``agenerate_salt`` for clarity's sake, & to reduce naming
collisions.
- Added another redundancy to the ``arandom_number_generator`` &
``random_number_generator`` functions. Now the async tasks it prepares
into a list are pseudo-randomly shuffled before being passed into
``asyncio.gather``.


Minor Changes
^^^^^^^^^^^^^

- Added a logo image to the package.
- Separated the FAQ section from ``PREADME.rst``.
- The ``primes`` & ``bits`` datasets are now represented in hex in the
source code.
- Added a ``BASE_38_TABLE`` dataset to the package.
- The database classes now fill an ephemeral dictionary of filenames
that couldn't be used to successfully load a tag file, available from
within the ``_corrupted_files`` attribute.
- The ``Comprende`` class' ``acache_check`` & ``cache_check`` context
manager methods are now called ``aauto_cache`` & ``auto_cache``.
- Added new ``bytes_count`` & ``abytes_count`` generators to ``generics.py``
module which increment each iteration & yield the results as bytes.
- Removed the ``akeypair`` & ``keypair`` functions from the package.
Their successors are the ``asingle_use_key`` & ``single_use_key`` methods
in the ``AsyncKeys`` & ``Keys`` classes. The attempt is to clarify &
put constraints on the interface for creating a bundle of key
material that has a single-use-only salt attached, as well as the pid
value.
- Moved ciphertext encoding functions into the ``BytesIO`` class from
the global ``generics.py`` module.
- Split ``PrimeGroups`` into two classes, one higher-level class by the
same name & a ``BasePrimeGroups`` class. The former also has some
added functionality for masking the order of bytes in a sequence
using an modular exponentiation.
- The ``Hasher`` class now has functionality added to mask the order
of a bytes sequence with a modular multiplication.
- Fixed the name of the project in the attribution lines in several
source files.
- Reconciled tests with the major changes in this release.
- The old identity key for the package that was signed by the gnupg
identity key was shredded & replaced with a new signed key.
- Several bug fixes to the ``setup.py`` automated code signing.

--------------------------


Major Changes
^^^^^^^^^^^^^

- Security Patch: The HMAC verifiers on ciphertexts did not include
the ``salt`` or ``pid`` values when deriving the HMAC. This
associated data can therefore be changed to cause a party to
decrypt a past ciphertext with a salt or pid of an attacker's
choosing. This is a critical vulnerability & it is highly recommended
all users update. The fix is to hash the ciphertext, ``salt``
& ``pid`` together & sending that hash into the validator to have
the HMAC created / tested. This change will cause all prior
ciphertexts to be marked invalid by the validator.
- Refactored the names of the Comprende cipher methods to better
communicate their intended use as lower level tools that cannot be
used on their own to obtain authenticated, CCA or CPA secure
encryption.
- Added more comprehensive tests for ``X25519`` & ``Ed25519`` classes,
as well as the protocols that utilize the ``X25519`` ecdh exchange.
Fixed some bugs in the process.
- ``X25519`` instances that contain a secret key now have access to
protocol methods which automatically pass their key in as a keyword
argument. This simplifies their usage further.
- Incorporated the new ``Hasher`` class into the package's random
number generator to improve its entropy production.


Minor Changes
^^^^^^^^^^^^^

- Various fixes to typos, docstrings & tutorials.
- New tutorials & docs added.
- Changed the default table in ``ByteIO`` 's ``json_to_ascii``, ``ajson_to_ascii``,
``ascii_to_json`` & ``aascii_to_json`` to the ``URL_SAFE_TABLE`` to
facilitate the creation of urlsafe_tokens.
- Removed all code in the ``Ropake`` class that was used to create a default
database to store a default salt for users. All of that functionality
is expected to be handled by the database classes' token & profile
creation tools.
- Fixed bug in package signing script that called hex from a string.
- Updated the package signing script to include these metadata in the
signatures of the ephemeral keys: name of the package, version, the
date in seconds.
- Added metadata to the ``setup.cfg`` file.
- Make passcrypt objects available from the ``keygens`` module.
- Add more consistent ability within ``Ropake`` class to specify a
time-to-live for protocol messages.
- Added check to make sure instances of ``X25519`` & ``Ed25519`` are
not trying to import a new secret key once they already have one.
This won't be allowed in favor of creating a new object for a new
secret key.
- Fixed bug in database classes' bytes ciphers which called themselves
recursively instead of calling the global functions of the same name.