Pulpcore

REST API

Bugfixes

- Skip allowed content checks on collectstatic
(backported from 8711)
[8712](https://pulp.plan.io/issues/8712)
- Fixed cleanup of UploadChunks when their corresponding Upload is deleted.
(backported from 7316)
[8757](https://pulp.plan.io/issues/8757)
- Fixed compatibility with Django 2.2 LTS. Pulp now requires Django~=2.2.23
(backported from 8691)
[8758](https://pulp.plan.io/issues/8758)
- Pinned click~=7.1.2 to ensure RQ is compatible with it.
[8767](https://pulp.plan.io/issues/8767)

Plugin API

No significant changes.

REST API

Bugfixes

- Fixed a race condition that sometimes surfaced during handling of reserved resources.
[8632](https://pulp.plan.io/issues/8632)
- Handled a tasking race condition where cleaning up resource reservations sometimes raised an IntegrityError.
[8648](https://pulp.plan.io/issues/8648)

Plugin API

Bugfixes

- Allow plugins to unset the `queryset_filtering_required_permission` attribute in `NamedModelViewSet`.
[8444](https://pulp.plan.io/issues/8444)

REST API

Features

- Raise error when syncing content with a checksum not included in `ALLOWED_CONTENT_CHECKSUMS`.
[7854](https://pulp.plan.io/issues/7854)
- User can evaluate how many content units are affected with checksum type change with 'pulpcore-manager handle-artifact-checksums --report'.
[7986](https://pulp.plan.io/issues/7986)
- The fields `proxy_username` and `proxy_password` have been added to remotes.
Credentials can no longer be specified as part of the `proxy_url`.
A data migration will move the proxy auth information on existing remotes to the new fields.
[8167](https://pulp.plan.io/issues/8167)
- Added the `WORKER_TTL` setting, that specifies the interval a worker is considered missing after its last heartbeat.
[8291](https://pulp.plan.io/issues/8291)
- Due to the removal of `md5` and `sha1` from the `ALLOWED_CONTENT_CHECKSUMS` setting, every
system that had any Artifacts synced in in prior to 3.11 will have to run the `pulpcore-manager handle-content-checksums` command. A data migration is provided with 3.11 that will run this
automatically as part of the `pulpcore-manager migrate` command all upgrades must run anyway.
[8322](https://pulp.plan.io/issues/8322)

Bugfixes

- Fixed a bug experienced by the migration plugin where all content objects are assumed to have a
remote associated with them.
[7876](https://pulp.plan.io/issues/7876)

- Restored inadvertently-changed content-guards API to its correct endpoint.

In the process of adding generic list-endpoints, the /pulp/api/v3/contentguards
API was inadvertently rehomed to /pulp/api/v3/content_guards. This change restores
it to its published value.
[8283](https://pulp.plan.io/issues/8283)

- Added headers field to the list of fields in the `RemoteSerializer` base class and marked it optional to make it accessible via the REST api.
[8330](https://pulp.plan.io/issues/8330)

- Fixed AccessPolicy AttributeError.
[8395](https://pulp.plan.io/issues/8395)

Improved Documentation

- Removed correlation id feature from tech preview.
[7927](https://pulp.plan.io/issues/7927)

- Removed 'tech preview' label from `handle-artifact-checksums` command.

`handle-artifact-checksums` is now a fully-supported part of Pulp3.
[7928](https://pulp.plan.io/issues/7928)

- Added a warning banner to the `ALLOWED_CONTENT_CHECKSUMS` setting section indicating the setting
is not fully enforcing in `pulpcore` code and various plugins.
[8342](https://pulp.plan.io/issues/8342)

Removals

- The `component` field of the `versions` section of the status API `` `/pulp/api/v3/status/ `` now
lists the Django app name, not the Python package name. Similarly the OpenAPI schema at
`/pulp/api/v3` does also.
[8198](https://pulp.plan.io/issues/8198)
- Removed sensitive fields `username`, `password`, and `client_key` from Remote responses. These
fields can still be set and updated but will no longer be readable.
[8202](https://pulp.plan.io/issues/8202)
- Adjusted the `ALLOWED_CONTENT_CHECKSUMS` setting to remove `md5` and `sha1` since they are
insecure. Now, by default, the `ALLOWED_CONTENT_CHECKSUMS` contain `sha224`, `sha256`,
`sha384`, and `sha512`.
[8246](https://pulp.plan.io/issues/8246)

Misc

- [7797](https://pulp.plan.io/issues/7797), [#7984](https://pulp.plan.io/issues/7984), [#8315](https://pulp.plan.io/issues/8315)

Plugin API

Features

- Allow developers to use more than one WorkingDirectory() within a task, including nested calls. Tasks will also now use a temporary working directory by default.
[7815](https://pulp.plan.io/issues/7815)
- Added the `pulpcore.app.pulp_hashlib` module which provides the `new` function and ensures only
allowed hashers listed in `ALLOWED_CONTENT_CHECKSUMS` can be instantiated. Plugin writers should
use this instead of `hashlib.new` to generate checksum hashers.
[7984](https://pulp.plan.io/issues/7984)
- Add a `get_content` method to `pulpcore.plugin.models.RepositoryVersion` that accepts a
queryset and returns a list of content in that repository using the given queryset.
This allows for specific content type to be returned by executing
`repo_version.get_content(content_qs=MyContentType.objects)`.
[8375](https://pulp.plan.io/issues/8375)

Improved Documentation

- Added docs identifying plugin writers to use the `pulpcore.app.pulp_hashlib` module which provides
the `new` function and ensures only allowed hashers can be instantiated. This should be used in
place of `hashlib.new`.
[7984](https://pulp.plan.io/issues/7984)
- The use of `tempdir.TemporaryDirectory` in tasks has been documented.
[8231](https://pulp.plan.io/issues/8231)

Removals

- Adjusted the `ALLOWED_CONTENT_CHECKSUMS` setting to remove `md5` and `sha1` since they are
insecure. Now, by default, the `ALLOWED_CONTENT_CHECKSUMS` contain `sha224`, `sha256`,
`sha384`, and `sha512`.
[8246](https://pulp.plan.io/issues/8246)
- Removed unused get_plugin_storage_path method.
[8343](https://pulp.plan.io/issues/8343)
- It is not longer possible to address AccessPolicy via the viewset's classname. Viewset's urlpattern should be used instead.
[8397](https://pulp.plan.io/issues/8397)
- Removed deprecated key field returned by the signing service.
Plugin writers must now refer directly to the public_key field on the signing service object.
[8398](https://pulp.plan.io/issues/8398)

Deprecations

- `pulpcore.plugin.tasking.WorkingDirectory` has been deprecated.
[8231](https://pulp.plan.io/issues/8231)

REST API

Features

- Change the default deployment layout

This changes the default deployment layout. The main change is that MEDIA_ROOT gets its own
directory. This allows limiting the file permissions in a shared Pulp 2 + Pulp 3 deployment and the
SELinux file contexts. Another benefit is compatibility with django_extensions' unreferenced_files
command which lists all files in MEDIA_ROOT that are not in the database.

Other paths are kept on the same absolute paths. The documentation is updated to show the latest
best practices.
[7178](https://pulp.plan.io/issues/7178)

- Added general endpoints to list `Content`, `ContentGuards`, and `Repositories`.
[7204](https://pulp.plan.io/issues/7204)

- Added /importers/core/pulp/import-check/ to validate import-parameters.
[7549](https://pulp.plan.io/issues/7549)

- Added a new field called public_key to SigningService. This field preserves the value of the public
key. In addition to that, the field fingerprint was introduced as well. This field identifies the
public key.
[7700](https://pulp.plan.io/issues/7700)

- Added possibility to filter users and groups by various fields.
[7975](https://pulp.plan.io/issues/7975)

- Added pulp_labels to allow users to add key/value data to objects.
[8065](https://pulp.plan.io/issues/8065)

- Added `pulp_label_select` filter to allow users to filter by labels.
[8067](https://pulp.plan.io/issues/8067)

- Added optional headers field to the aiohttp ClientSession.
[8083](https://pulp.plan.io/issues/8083)

- Allow querying names on the api using name__icontains, name__contains and name__startswith query parameters.
[8094](https://pulp.plan.io/issues/8094)

- Added RBAC to the endpoint for managing groups.
[8159](https://pulp.plan.io/issues/8159)

- Added RBAC to the endpoint for managing group users.
[8160](https://pulp.plan.io/issues/8160)

- Added the `AccessPolicy.customized` field which if `True` indicates a user has modified the
default AccessPolicy.
[8182](https://pulp.plan.io/issues/8182)

- Added filtering for access policies.
[8189](https://pulp.plan.io/issues/8189)

- As an authenticated user I can create and view artifacts.
[8193](https://pulp.plan.io/issues/8193)

Bugfixes

- Fixed bug where duplicate artifact error message was nondeterministic in displaying different error
messages with different checksum types. Also, updated duplicate artifact error message to be more
descriptive.
[3387](https://pulp.plan.io/issues/3387)
- Fixed Pulp import/export bug that occurs when sha384 or sha512 is not in `ALLOWED_CONTENT_CHECKSUMS`.
[7836](https://pulp.plan.io/issues/7836)
- X-CSRFToken is not sent through ajax requests (PUT) in api.html. Fixed by setting the right value in
the JS code.
[7888](https://pulp.plan.io/issues/7888)
- Provide a mechanism to automatically resolve issues and prevent deadlocks when Redis experiences data loss (such as a restart).
[7912](https://pulp.plan.io/issues/7912)
- Silence unnecessary log messages from django_guid which were spamming up the logs.
[7982](https://pulp.plan.io/issues/7982)
- Changed the default permission class to `IsAdminUser` to protect endpoints not yet guarded by an access policy from users without permission.
[8018](https://pulp.plan.io/issues/8018)
- Fixed apidoc bug, where model and object permissions on groups overlapped.
[8033](https://pulp.plan.io/issues/8033)
- Fixed the viewset_name used by access policy for the cases when parent_viewset is involved.
[8152](https://pulp.plan.io/issues/8152)
- Made the viewset_name property of access policies read only.
[8185](https://pulp.plan.io/issues/8185)

Improved Documentation

- Added a description of the common filesystem layout in the deployment section.
[7750](https://pulp.plan.io/issues/7750)
- Updated the reference to the new location of pulplift at the installer repository in the development section.
[7878](https://pulp.plan.io/issues/7878)
- Add links to plugin docs into docs.pulpproject.org.
[8131](https://pulp.plan.io/issues/8131)
- Added documentation for labels.
[8157](https://pulp.plan.io/issues/8157)

Misc

- [8203](https://pulp.plan.io/issues/8203)

Plugin API

Features

- Add `rate_limit` option to `Remote`
[7965](https://pulp.plan.io/issues/7965)
- Made DistributionFilter accessible to plugin writers.
[8059](https://pulp.plan.io/issues/8059)
- Adding `Label` and `LabelSerializer` to the plugin api.
[8065](https://pulp.plan.io/issues/8065)
- Added `LabelSelectFilter` to filter resources by labels.
[8067](https://pulp.plan.io/issues/8067)
- Added ReadOnlyRepositoryViewset to the plugin API.
[8103](https://pulp.plan.io/issues/8103)
- Added NAME_FILTER_OPTIONS to the plugin API to gain more consistency across plugins when filter by name or similar CharFields.
[8117](https://pulp.plan.io/issues/8117)
- Added has_repo_attr_obj_perms and has_repo_attr_model_or_obj_perms to the global access checks available to all plugins to use.
[8161](https://pulp.plan.io/issues/8161)

Removals

- Plugins are required to define a `version` attribute on their subclass of
`PulpPluginAppConfig`. Starting with pulpcore==3.10, if undefined while Pulp loads, Pulp will
refuse to start.
[7930](https://pulp.plan.io/issues/7930)
- Changed the default permission class to from `IsAuthenticated` to `IsAdminUser`.
Any endpoints that should be accessible by all known to the system users need to specify the permission_classes accordingly.
[8018](https://pulp.plan.io/issues/8018)
- `pulpcore.plugin.models.UnsupportedDigestValidationError` has been removed. Plugins should
look for this at `pulpcore.plugin.exceptions.UnsupportedDigestValidationError` instead.
[8169](https://pulp.plan.io/issues/8169)

Deprecations

- Access to the path of the public key of a signing service was deprecated. The value of the public
key is now expected to be saved in the model instance as `SigningService.public_key`.
[7700](https://pulp.plan.io/issues/7700)
- The `pulpcore.plugin.storage.get_plugin_storage_path()` method has been deprecated.
[7935](https://pulp.plan.io/issues/7935)

REST API

Removals

- CHUNKED_UPLOAD_DIR was converted to a relative path inside MEDIA_ROOT.
[8099](https://pulp.plan.io/issues/8099)

Plugin API

No significant changes.

REST API

Features

- Made uploaded chunks to be stored as separate files in the default storage. This feature removes
the need for a share storage of pulp api nodes, as the chunks are now stored individually in the
shared storage and are therefore accessible by all nodes.
[4498](https://pulp.plan.io/issues/4498)

- Add support for logging messages with a correlation id that can either be autogenerated or passed in
with a `Correlation-ID` header. This feature is provided as a tech preview in pulpcore 3.9.
[4689](https://pulp.plan.io/issues/4689)

- Added progress reporting for pulp imports.
[6559](https://pulp.plan.io/issues/6559)

- Exposed `aiohttp.ClientTimeout` fields in `Remote` as `connect_timeout`,
`sock_connect_timeout`, `sock_read_timeout`, and `total_timeout`.

This replaces the previous hard-coded 600 second timeout for sock_connect and sock_read,
giving per-`Remote` control of all four `ClientTimeout` fields to the user.
[7201](https://pulp.plan.io/issues/7201)

- Enabled users to add checksums to ALLOWED_CONTENT_CHECKSUMS by allowing them to populate checksums
with handle-artifact-checksums command.
[7561](https://pulp.plan.io/issues/7561)

- Added version information to api docs.
[7569](https://pulp.plan.io/issues/7569)

- Made signing services to be immutable. This requires content signers to create a new signing
service explicitly when a change occurs.
[7701](https://pulp.plan.io/issues/7701)

- Added support for repairing Pulp by detecting and redownloading missing or corrupted artifact files. Sending a POST request to `/pulp/api/v3/repair/` will trigger a task that scans all artifacts for missing and corrupted files in Pulp storage, and will attempt to redownload them from the original remote. Specifying `verify_checksums=False` when POSTing to the same endpoint will skip checking the hashes of the files (corruption detection) and will instead just look for missing files.

The `verify_checksums` POST parameter was added to the existing "repository version repair" endpoint as well.
[7755](https://pulp.plan.io/issues/7755)

- Added check to prevent Pulp to start if there are Artifacts with forbidden checksums.
[7914](https://pulp.plan.io/issues/7914)

Bugfixes

- Fixed a serious bug data integrity bug where some Artifact files could be silently deleted from storage in specific circumstances.
[7676](https://pulp.plan.io/issues/7676)
- Moved the initial creation of access_policies to post_migrate signal.
This enforces their existance both with migrate and flush.
[7710](https://pulp.plan.io/issues/7710)
- Fixed incremental export to happen if start_version provided, even if last_export is null.
[7716](https://pulp.plan.io/issues/7716)
- Fixed a file descriptor leak during repository version repair operations.
[7735](https://pulp.plan.io/issues/7735)
- Fixed bug where exporter directory existed and was writable but not owned by worker process and thus
not chmod-able.
[7829](https://pulp.plan.io/issues/7829)
- Properly namespaced the viewset_name in AccessPolicy to avoid naming conflicts in plugins.
[7845](https://pulp.plan.io/issues/7845)
- Update jquery version from 3.3.1 to 3.5.1 in API.html template. It is the version provided by djangorestframework~=3.12.2
[7850](https://pulp.plan.io/issues/7850)
- Prevented a Redis failure scenario from causing the tasking system to back up due to "tasking system
locks" not being released, even on worker restart.
[7907](https://pulp.plan.io/issues/7907)
- Use subclassed plugin downloaders during the pulp repair.
[7909](https://pulp.plan.io/issues/7909)

Improved Documentation

- Added requirement to record a demo with PRs of substantial change.
[7703](https://pulp.plan.io/issues/7703)
- Removed outdated reference stating Pulp did not have an SELinux policy.
[7793](https://pulp.plan.io/issues/7793)

Removals

- The local file system directory used for uploaded chunks is specified by the setting
CHUNKED_UPLOAD_DIR. Users are encouraged to remove all uncommitted uploaded files before
applying this change.
[4498](https://pulp.plan.io/issues/4498)

Misc

- [7690](https://pulp.plan.io/issues/7690), [#7753](https://pulp.plan.io/issues/7753), [#7902](https://pulp.plan.io/issues/7902), [#7890](https://pulp.plan.io/issues/7890)

Plugin API

Features

- Added pre_save hook to Artifact to enforce checksum rules implied by ALLOWED_CONTENT_CHECKSUMS.
[7696](https://pulp.plan.io/issues/7696)
- Enabled plugin writers to retrieve a request object from a serializer when look ups are
performed from within the task serializer.
[7718](https://pulp.plan.io/issues/7718)
- Expose ProgressReportSerializer through pulpcore.plugin
[7759](https://pulp.plan.io/issues/7759)
- Allowed plugin writers to access the models Upload and UploadChunk
[7833](https://pulp.plan.io/issues/7833)
- Exposed `pulpcore.plugin.constants.ALL_KNOWN_CONTENT_CHECKSUMS`.
[7897](https://pulp.plan.io/issues/7897)
- Added `UnsupportedDigestValidationError` to `pulpcore.plugins.exceptions`. Going
forward, plugin authors can expect to find all unique exceptions under
`pulpcore.plugin.exceptions`.
[7908](https://pulp.plan.io/issues/7908)

Deprecations

- Plugins are encouraged to define a `version` attribute on their subclass of
`PulpPluginAppConfig`. If undefined while Pulp loads a warning is now shown to encourage plugin
writers to implement this attribute, which will be required starting in pulpcore==3.10.
[6671](https://pulp.plan.io/issues/6671)

- Using the ViewSet's classname to identify its AccessPolicy has been deprecated and is slated for removal in 3.10.
Instead the urlpattern is supposed to be used.

Plugins with existing AccessPolicies should add a data migration to rename their AccessPolicies:

::

: access_policy = AccessPolicy.get(viewset_name="MyViewSet")
access_policy.viewset_name = "objectclass/myplugin/myclass"
access_policy.save()

[7845](https://pulp.plan.io/issues/7845)

- The `pulpcore.plugin.models.UnsupportedDigestValidationError` is being deprecated and
will be removed in 3.10.

It can now be found at `pulpcore.plugin.exceptions.UnsupportedDigestValidationError`
instead; please change any code that imports it to access it from its new location.
[7908](https://pulp.plan.io/issues/7908)