Dirac

[integration]

*DataManagementSystem

NEW: (7817) experimental token support for FTS transfers
FIX: (7781) dirac_dms_find_lfns: Check if requested path exists and returns error if not. Prevents users hitting 7487
FIX: (7780) Return correct error for non-existing directory. Partial fix for 7487
NEW: (7764) added checksum comparison in the fc/se consistency tool
NEW: (7756) FileCatalogHandler: add function export_getFileDetails to get the (user) metadata for a list of LFNs
FIX: (7719) Remove `def findDirIDsByMetadata(self, metaDict, dPath, credDict):` method from `MultiVODirectoryMetadata` (derived) class which caused an extra VO suffix added when searching. The method is meant to be used _internally_ only on keys which are already expanded in a MultiVO case. Add a user-level def `findDirectoriesByMetadata(self, queryDict, path, credDict)` to the derived class thus adding a VO suffix for a directory search. Fixes 7687.
CHANGE: (7694) dirac-dms-replicate-and-register-request: Make request chunk size configurable; default behaviour unchanged.

*FrameworkSystem

NEW: (7817) allow to bypass cache when retrieving tokens

FIX: (7813) SSHCE, Try python3 before unversioned python
Fix a typo in the key Addler -> Adler of the return dict of putAndRegister

*TransformationSystem

FIX: (7806) RequestTaskAgent only considers requests in final states, and consider files in intermediate state as problematic (https://github.com/DIRACGrid/DIRAC/issues/7116)
NEW: (7806) RequestTaskAgent uses getBulkRequestStatus instead of getRequestStatus
RMS: (7806)
NEW: (7806) implement getRequestStatus
NEW: (7697) InputDataAgent: new Option MultiVO, which makes the FileCatalog Query use the author of the DN, rather than the Host, to resolve MultiVO metadata correctly. Fixes 7681

*Resources

FIX: (7803) added a 30s gfal2 timeout for downloading the SRR
FIX: (7790) SSHComputingElement fix: added check of result
FIX: (7726) Update CloudCE cloudinit.template for EL8+
CHANGE: (7715) HTCondorCE: UseSSLSubmission: use the generated proxy file for everything, no longer need to have certificate of user present on the server.
FIX: (7713) AREX submission issue not properly handled
NEW: (7695) better error message when no matching protocol between 2 SE
CHANGE: (7689) remove BOINCCE

*ResourceStatusSystem

FIX: (7801) use always a from address (from Operations ResourceStatus/Config/FromAddress ) when sending email notifications, to avoid "spoofing" domains restrictions
NEW: (7783) add a DIRAC to GOCDB service type conversion
CHANGE: (7774) do not consider Endpoint for StorageOccupancy
FIX: (7765) Do not use tinezone aware datetime
FIX: (7755) delete the Occupancy cache only for older entries

*WorkloadManagement

FIX: (7797) Allow jobs to be KILLED from more states
FIX: (7797) KILLED is a final job state
FIX: (7716) PilotWrapper - check for the presence of the -l pilot option

*ConfigurationSystem

CHANGE: (7796) VOMS2CSAgent: if a nickname is set, this nickname will always be used and no new accounts are going to be created if a DN changes or a user is in multiple VOs
NEW: (7796) VOMS2CSAgent: New option "ForceNickname", if this option is enabled no dirac user is created if no nickname attribute is set for a user
CHANGE: (7796) IAMService: use logger and return errors for users so that the VOMS2CSAgent can notify admins about issues
NEW: (7742) Configuration-system-shell: added reload and sort commands

*WorkloadManagementSystem

CHANGE: (7792) SiteDirector will always bundle the proxy
CHANGE: (7762) removed JobDB's SiteMask and Logging
FIX: (7751) Proper killing of jobs when not matched, running or stalled
FIX: (7707) JobDB: update LastUpdateTime when the job is matched
NEW: (7699) added possibility to specify UserEnvVariable (pilot option) at CE level

*Workflow

FIX: (7786) Avoid incorrect error strings in Workflow execute

*Core

NEW: (7775) Support aggregating by date in MySQL.getCounters
FIX: (7750) add more safeguard when processing result queue in ProcessPool
CHANGE: (7720) register the DictCache destructor as an atexit handler
NEW: (7700) fetch nickname attribute from IAM
CHANGE: (7696) drop DIRAC_MYSQL_CONNECTION_GRACE_TIME and stalled connections reuse

*Subsystem

CHANGE: (7733) For PilotWrapper tests, use the artifacts (created in Pilot repo)

*Diracx

NEW: (7711) populate diracx section from Iam

*Doc

FIX: (7688) Move the explanation how to enable tokens to a place that is shown in ReadTheDocs in the end.

*Interfaces

FIX: (7684) Document BadJobParameterError

[integration]

*WorkloadManagementSystem

FIX: (7649) added log headers to InputDataResolution modules
CHANGE: (7629) add jobGroup to job parameters
FIX: (7584) ServerUtils: prevent getPilotAgentsDB from returning None
FIX: (7576) Fix potential circular import in WorkflowReader. Mostly seen in the creation of the documentation.

*WorkloadManagement

NEW: (7643) Support Pydantic 2
FIX: (7621) remove random shuffle in PilotWrapper
CHANGE: (7609) Perform bulk lookup of job parameters from elasticsearch
CHANGE: (7608) Make RemoteRunner more resilient to CE issues
FIX: (7594) JobMonitoring.getJobParameters should pass jobID as an int to ElasticJobParametersDB
FIX: (7590) AREX "out" and "err" need to exist before file integrity check

*Accounting

FIX: (7640) AccountingDB only generate condition if needed

*Resources

NEW: (7638) HTCondorCE: Added UseSSLSubmission option. Allows one to use a configured DN at given Sites for job submission, instead of proxies or tokens. Only at participating CEs and conditions apply.

*Core

FIX: (7634) Avoid printing out clear text password in SQLAlchemy
FIX: (7591) File.secureOpenForWrite: fix exception when opening in binary mode, fixes 7581

*DataManagementSystem

NEW: (7633) Add a protocol parameter to the getReplicas method family
NEW: (7619) prepare for FTS 3.13 release with breaking API
NEW: (7617) DataManager.putAndRegister rejects too long filename

*Deployment

FIX: (7628) fix the path of the CVMFS sync_packages.sh script

*All

FIX: (7616) fix pylint 3.2.0 warnings

*MonitoringSystem

FIX: (7584) ServerUtils: prevent getMonitoringDB from returning None

[integration]

*WorkloadManagementSystem

FIX: (7574) serverUtils: jit imports
FIX: (7534) Pilots submitted by SiteDirector won't add the pythonVersion flag
FIX: (7521) Fix memory reporting
FIX: (7510) SandboxStore: add VO if needed

*WorkloadManagement

FIX: (7571) support file:/... as a location for the pilot files
FIX: (7564) make sure CVMFS_locations is a list
CHANGE: (7553) Remove files from the RemoteRunner execution
FIX: (7552) JobCleaningAgent: fix exception in deleteJobsByStatus caused by mismatching job ID types
NEW: (7529) introduce JobWrapperOfflineTemplate for uses in systems without external connectivity
CHANGE: (7460) introduce JobWrapper.preprocess, process and postprocess

*test

NEW: (7570) added pilot workflow tests to integration_tests

*Core

[integration]

*Core

FIX: (7505) plotting TypeLoader works with editable installation
NEW: (7453) Introduce an RPC stub equivalent for DiracX

*Test

CHANGE: (7502) use pytest-rerun instead of flaky

*WorkloadManagementSystem

CHANGE: (7498) removed GridEnv
FIX: (7497) If the SoftwareDistModule is set in the Operations Section, add it to the Job JDL to restore previous behaviour
NEW: (7453) FutureJobStateUpdate.setJobStatusBulk return a DiracX RPC stub

*WorkloadManagement

FIX: (7493) check the VO from the task queues before submitting pilots
FIX: (7488) JobAgent.setupProxy takes owner instead of ownerDN

*RequestManagementSystem

CHANGE: (7453) adapt ForwardDISET to DiracX stub

*FrameworkSystem

CHANGE: (7442) removed NotificationDN

[integration]

*FrameworkSystem

FIX: (7491) dirac-proxy-info without the dirac group

*WorkloadManagement

FIX: (7490) missing result in return statement

*tests

NEW: (7484) add the state key in the test environment to fix diracx execution

[integration]

*Accounting

FIX: (7486) Fix errors during insert into ac_in_* tables.

*Core

FIX: (7483) Fix DISET calls with proxy to be used passed as an argument

*RequestManagementSystem

FIX: (7482) RequestTask - download no-VOMS proxy if the owner group does not define VOMSRole