Dirac

[integration]

*DataManagementSystem

FIX: (5858) FileCatalogCLI - removed faulty evaluation of undefined option to the rebuild command. Fixes 5759

*Core

FIX: (5857) TorandoREST ignore arguments that not defined in the target method
FIX: (5857) dirac-info show "None found" instead of exception
CHANGE: (5854) Treat X509 data as str instead of bytes

*FrameworkSystem

FIX: (5857) use payload keyword argument for handle_response method
CHANGE: (5857) use DIRAC CAs location to request DIRAC AS
FIX: (5857) fix message in dirac-login after authorization
FIX: (5854) UserProfileManager usage with JEncode

*Resources

CHANGE: (5857) verify CAs when request IdP by default

FIX: (5856) Changed the nomenclature for PilotSubmissionMonitoring

*WorkloadManagement

FIX: (5854) JobDB usage with JEncode

*Production

FIX: (5854) ProductionManager use with JEncode

*WorkloadManagementSystem

FIX: (5848) WMS DBs: do not try to decode when it's not bytes
CHANGE: (5846) added Modules and PipInstallOptions to SiteDirector/Pilot option

*ConfigurationSystem

FIX: (5844) VOMS2CSAgent correctly takes into account multiple CAs for users with multiple DNs

*MonitoringSystem

FIX: (5838) Fixing nomenclature and other issues with PilotMonitoringSubmission

*Tornado

FIX: (5837) better logging formatting

*TransformationSystem

NEW: (5829) Added TornadoTransformationManager service

[integration]

*FrameworkSystem

CHANGE: (5836) added a flag for disabling the use of SecurityLogging service

*Core

CHANGE: (5833) JEncode bytes, tests
FIX: (5810) do not allow arbitrary code execution in JEncode

*ConfigurationSystem

CHANGE: (5833) align with JEncode changes

*ResourceStatusSystem

NEW: (5830) Added tornado version for the three RSS services

*TransformationSystem

CHANGE: (5828) TransformationDB: removing BLOB in favor of TEXT for non-binary data
FIX: (5822) TransformationDB: removed old compatibility layer with mySAM tables
FIX: (5796) Convert fileID / taskID to fix issue when task status is updated in transformation treatment

*docs

FIX: (5825) fix indentation in the doc related to the server installation
CHANGE: (5823) add further details about the server installation process with python3

*WorkloadManagement

CHANGE: (5824) enable logs from db12 in dirac-wms-cpu-normalization

*Interface

FIX: (5818) send only single jobID to checkJobStateTransition

Thank you for writing the text to appear in the release notes. It will show up
exactly as it appears between the two bold lines
Please follow the template:
Thank you for writing the text to appear in the release notes. It will show up
exactly as it appears between the two bold lines
Please follow the template:

*WorkloadManagementSystem

FIX: (5816) Allow transition Rescheduled->Failed

*Subsystem

NEW/CHANGE/FIX: (5804) explanation
For examples look into release.notes

*MonitoringSystem

CHANGE: (5788) moving pilot submission to monitoring

*Docs

FIX: (5711) updated/simplified the documentation for adding new components

[integration]

*WorkloadManagementSystem

FIX: (5793) better use the job state machine when setting the job status, in particular when it comes from failover requests that may be inserted between 2 successful updates.
FIX: (5751) allow Stalled, Running and Matched jobs to go to Reschedule
FIX: (5751) JobAgent forces the reschedule of jobs in case of exceptions
FIX: (5746) convert inputs from tuple to list in JobAgent before submitting to a CE
NEW: (5714) added WMSDashboard and diracLogsDashboard in NDJSON format.
FIX: (5714) removed old index patterns in WMSDashboard.json, updated documentation.

*DataManagementSystem

FIX: (5791) Fix pickle error in dirac-dms-directory-sync

*WorkloadManagement

FIX: (5789) JobSanity doesn't handle input sanboxes named "LFN:"

*TransformationSystem

Fix: Restore task state "Scheduled"

*Resources

CHANGE: (5774) SingularityComputingElement: install python3 version
NEW: (5766) allow to overwrite input/output storage plugin parameter with the CS
CHANGE: (5766) add "xroot" to the list of input protocol for the XROOT plugin

*Interfaces

CHANGE: (5772) DIRAC APIs check if a JobState transition is allowed

*Subsystem

FIX: (5768) dirac_production_runjoblocal.py: moved to py3 Pilot install

*All

CHANGE: (5764) use Client instead of RPCClient

*HTTPS

FIX: (5762) Replace DIRAC.Core.DISET.TransferClient with DIRAC.Core.Tornado.Client.ClientSelector

*FrameworkSystem

FIX: (5757) rename downloadablePersonalProxy value to allowProxyDownload
FIX: (5748) Get Tornado's port from the CS when adding URLs

*docs

CHANGE: (5756) add document image source files

*Core

FIX: (5743) removed isPy3VersionNumber function

[integration]

*WorkloadManagement

NEW: (5736) PushJobAgent to prepare jobs for sites with no external connectivity
NEW: (5736) RemoteRunner to execute applications remotely (submitting them to remote CE)

*Resources

CHANGE: (5736) add inputs and outputs parameters in ARC.submitJob()
CHANGE: (5702) ParallelLibrary: Embed the executable in the srun wrapper

*Workflow

CHANGE: (5736) integrate RemoteRunner

*WorkloadManagementSystem

FIX: (5735) securityLogging: zip after 1 day
CHANGE: (5728) Replace DB12.py with the db12 package
FIX: (5725) ElasticJobParametersDB: get in output all entries
CHANGE: (5724) JobAgent and JobWrapper: only reporting changes on major status when needed

*ResourceStatusSystem

FIX: (5734) CSHelpers: some sites have no resources

*HTTPS

FIX: (5730) Add URLs to CS when installing HTTPS services

*Core

FIX: (5726) TornadoBaseClientcorrectly checks CA
FIX: (5717) encodeDict when the dict keys have mixed types
CHANGE: (5698) Replacing Elasticsearch python client with OpenSearch.
FIX: (5694) do no look anymore for __RCSID__
FIX: (5683) ThreadSafe.Synchronizer: Add target method docstring to preserve docstring of decorated function

*ConfigurationSystem

FIX: (5701) fix an exception in the VOMS2CSAgent when a user joins a second VO
FIX: (5684) fix CSGlobals.Extensions class

*TransformationSystem

FIX: (5690) fix more flaky tests for RequestTasks

[integration]

*ConfigurationSystem

CHANGE: (5682) Bdii2CSAgent: add InjectSingleCoreQueue option to automatically create single core equivalents for MutliCore Queues , kind of fixes 5582
CHANGE: (5682) Bdii2CSAgent: CEs in the BannedCEs list are no longer updated by the agent, previously this list only concerned CEs that are not already in the Configuration, fixes 5224
CHANGE: (5682) dirac-admin-add-resources: add --onecore option to automatically create single core equivalents for MutliCore Queues
FIX: (5682) dirac-admin-add-resources: fix the query about adding new CEs, this can now be answered in the negative

*Resources

FIX: (5674) decode the http response in WLCGAccountingHTTPJson
CHANGE: (5643) HTCondorCE submits jobs with -spool option when a local schedd is used to spool the pilot wrappers that can be deleted afterwards

*Core

FIX: (5671) DIRAC.isPy3VersionNumber always returns True
FIX: (5656) Tasks never get removed from the ExecutorState if an UnrecoverableTaskException is raised
CHANGE: (5629) use DIRAC_DEBUG_M2CRYPTO to enable SSL debugging in tornado

*RMA

FIX: (5670) If a job was Killed, set it Killed at request completion

*TransformationSystem

FIX: (5666) fix RequestTasks flaky tests
NEW: (5642) Allow MultiOperation body to fetch values from the task dict
NEW: (5642) Introduce Body Plugins for DMS transformations
FIX: (5630) TransformationCleaningAgent 'deletes', do not 'remove' jobs

*WorkloadManagement

FIX: (5665) Python 3 support in SSHComputingElement
FIX: (5657) Bad error handling in OptimizationMindHandler.exec_taskError

*Docs

CHANGE: (5663) removed a bunch of py2 related documentation

*MonitoringSystem

FIX: (5658) Wrong link on the docs page for the WMSHistory dashboard. Fixed the link and edited some of the text.
NEW: (5634) JSON file for WMS monitoring dashboard

*docs

FIX: (5654) fix trailing-whitespace

*FrameworkSystem

FIX: (5652) SystemAdministratorCLI: get DIRAC version from Extensions
CHANGE: (5641) proxy expiration emails are sent from an address taken from the Service/Agent config instead of the DB
NEW/CHANGE/FIX: (5627) avoid duplication of code in TornadoBundleDeliveryHandler
FIX: (5619) continue on empty service list in getComponentsStatus

*DataManagementSystem

FIX: (5651) dirac-dms-directory-sync: The script will now try all files and not end a thread if one of the files assigned to be uploaded or downloaded to a thread fails. The script will also print out all failed files, and their error at the end of a thread. A few log messages have been corrected as well.

*ResourceStatusSystem

FIX: (5647) specifying the gocDB type for known types

*WorkloadManagementSystem

CHANGE: (5643) Add functions in WMSUtilities, move content of getPilotLoggingInfo() and getGridJobOutput() in PilotManagerHandler

*RequestManagementSystem

NEW: (5615) add SweepSize option to the ReqProxy

[integration]

*TransformationSystem

FIX: (5612) fix bug in TransformationCleaningAgent, that was NOT removing jobs but only deleting them

*WorkloadManagementSystem

FIX: (5612) removing jobs instead of deleting them when bulk submission fails
CHANGE: (5570) better logging output in the JobStateUpdate

*WorkloadManagement

FIX: (5610) Return error if JobPolicy.getControlledUsers returns an empty list
FIX: (5603) Don't run JobsStateMachine if newStat is ''
FIX: (5595) UnicodeDecodeErrors in JobDB.getJobParameters
FIX: (5585) Error logging when optimisation fails

*FrameworkSystem

FIX: (5609) Cleaning old installations from a Python 3 based server
FIX: (5604) Issue installing DIRAC and extensions on servers running Python 3
FIX: (5583) display correctly the help in sysadmin-cli
CHANGE: (5571) allow DIRAC AS return proxy by default

*ConfigurationSystem

FIX: (5608) ServiceInterface.py - use ThreadPoolExecutor instead of ThreadPool to avoid thread leaks in the ConfigurationServer
CHANGE: (5596) do not add the master CS first in the list of CS when new slaves are added

*Test

FIX: (5605) fix typo in the HTTPs Resources tests

*Core

FIX: (5594) Extension ordering in DIRACScript

*Resources

FIX: (5586) cloudinit-template - allow time in the monitor for the pilot to start
FIX: (5586) cloudinit-template - set DIRACSYSCONFIG to point to the pilot.cfg in order to allow its use in the user applications

*docs

CHANGE: (5573) move the ConfReference DIRAC section part to dirac.cfg and add there OAuth 2 settings.
FIX: (5572) update nginx installation instruction
NEW: (5540) add documentation with information about new system APIs component

Align `WebApp` installation manual with last changes (using BaseRequestHandler in WebAppDIRAC 5.0).
The request Accounting -> Network -> Packet lost rate (magnified) causes an error:

ERROR: (5569) Error while generating the plots Value keys 100 - IF(SUM(PacketLossRate)/SUM(entriesInBucket)*10 > 100, 100, SUM(PacketLossRate)/SUM(entriesInBucket)*10), 100 are not defined

This RP should provide correct pattern.

*Accounting

FIX: (5569) provide correct pattern for reportMagnifiedPacketLossRate