Dirac

[rel-v8r0]

*WorkloadManagementSystem

FIX: (6435) WMSUtilities.killPilotsInQueues have a consistent return value
FIX: (6417) removed unused SubmissionMode CE parameter
CHANGE: (6399) ElasticJobParametersDB: create a new index every 1M jobs
FIX: (6394) JobScheduling: select the correct RescheduleDelay instead of 1 higher
FIX: (6387) replace opPath by opChain in JobPath
CHANGE: (6381) JobWrapper uses getDestinationSEList rather that resolveSEGroup

NEW: (6427) Information about VirtualOrganization parameter added.

*MonitoringSystem

CHANGE: (6418) Remove PilotSubmissionPlotter
CHANGE: (6386) Remove DataOperationPlotter

*DataManagementSystem

FIX: (6412) Proper casting for data sent to the monitoring
FIX: (6391) DirectoryLevelTree: fix missing keyword argument name causing failures to upload files for example
NEW: (6381) backport getDestinationSEList from LHCbDIRAC

*Test

FIX: (6411) server installation do not install JobAgent

*Core

FIX: (6411) MySQL tracer bugs
FIX: (6396) Use safer mode for grid-security directories
FIX: (6394) TimeUtilities.fromString: when given a datetime.datetime, return the same object, instead of None. Also Fixes JobScheduling issue

*Production

NEW: (6401) add dirac-prod-complete command

*Tests

NEW: (6381) introduce generateDIRACConfig utility to generate DIRAC config during tests

*Docs

CHANGE: (6381) move site documentation from ConfigRef to Resources and dirac.cfg
NEW: (6381) document Countries

[rel-v8r0]

*DataManagementSystem

NEW: (6382) cancel the FTS transfers when an RMS request is found to be canceled
FIX: (6364) Monitoring does not update the dict using by the accounting, avoiding a crash
NEW: (6364) FailedDataOperation monitoring now keeps track of interactive/job failures

*Resources

CHANGE: (6376) negative free space value is transformed to 0

*Core

FIX: (6375) Explicitly depend on db12

*Core

NEW: Basic support for the OAuth/OIDC tokens
NEW: (6299) Experimental support for Python 3.10
NEW: (6017) TornadoBaseRequestHandler define a class logger with an extra attribute "tornadoComponent"
NEW: (6017) DIRACDB can now use a parentLogger instead of direct gLogger
NEW: (6335) DIRAC.initialize now provides keyword arguments to customise initialisation
NEW: (6335) SecurityProperty enum for expressing security properties which can be assigned to groups/hosts
NEW: (6335) LogLevel enum for choosing the logging level
NEW: (6309) Introduce DReturnType/DOKReturnType/DErrorReturnType for DIRAC return structures
NEW: (6347) Introduce ``DIRAC_MYSQL_OPTIMIZER_TRACES_PATH`` environment variable for advance optimization of MySQL calls
NEW: (6094) option op_type in index API for Elasticsearch
NEW: (6299) Suppress InsecureRequestWarning when using dirac-configure with HTTPS servers
NEW: (6299) Support overriding ciphers and methods with environment variables when using HTTPS
NEW: (6291) adding general flag for disabling Monitoring for Agents (used by JobAgent)
CHANGE: (5872) Minimum Python version is now 3.9
CHANGE: (5883) Show callstack in service logs when returning S_ERROR
CHANGE: (5764) use Client instead of RPCClient
CHANGE: (5698) Replacing Elasticsearch python client with OpenSearch.
CHANGE: (5571) allow DIRAC AS return proxy by default
CHANGE: (6188) Propagate original proxy strength when generating new proxies
CHANGE: (6281) Removed Core.Utilities.MemStat module (use psutil module instead)
CHANGE: (6281) Removed possibility to Register a DISET service with a secondary name
CHANGE: (6347) calls to the MySQL class with keyword parameters MUST now be named
CHANGE: (6297) Use psutil.net_if_addrs for Network.discoverInterfaces
CHANGE: (6286) Default proxy length is 2048 bits
CHANGE: (6286) remove RFC argument for proxy generation
CHANGE: (6091) changed name and simplified repetitive methods of Utilities.TIme
CHANGE: (6017) services propagate their local loggers to the DB
CHANGE: (6017) tornado services use local logger
CHANGE: (5854) Treat X509 data as str instead of bytes
FIX: (5743) removed isPy3VersionNumber function
FIX: (5762) Replace DIRAC.Core.DISET.TransferClient with DIRAC.Core.Tornado.Client.ClientSelector
FIX: (6151) Replace deprecated threading.activeCount with active_count
FIX: (6149) use AnyThreadEventLoopPolicy asyncio policy
FIX: (6144) Removing time dependence of TimeUtilities Module
FIX: (5870) allow to use `deprecated` decorator with classes without predefined `__init__` function.


*DataManagementSystem

NEW: (5927) TornadoDataIntegrityHandler
CHANGE: (5391) The LcgFileCatalogClient has been removed

*FrameworkSystem

NEW: (6248) Add StdoutJsonBackend
NEW: (5901) move ProxyManager to HTTPs
NEW: (5899) dirac-login (AS) added VOMS extension
NEW: (5896) dirac-login try to connect to DIRAC AS if no local certificate
NEW: (5862) add password mask to dirac-login
CHANGE: (5965) removed gMonitor
CHANGE: (5865) remove unused 'rfc' argument
CHANGE: (5857) use DIRAC CAs location to request DIRAC AS

*MonitoringSystem

NEW: (5974) TornadoMonitoring handler
NEW: (5999) Added a PilotsHistory Monitoring
NEW: (5807) Move DataOperation to Monitoring
NEW: (5634) JSON file for WMS monitoring dashboard
CHANGE: (5788) moving pilot submission to monitoring
CHANGE: (6002) Replaced the flags for monitoring with a new unified flag system
CHANGE: (5944) Divided ComponentMonitoring into AgentMonitoring and ServiceMonitoring
FIX: (6286) ElasticSearchDB generates full index names based on UTC
FIX: (6286) DataOperationSender works on a copy of the documents
FIX: (6076) Removing multiplication in timestamp when committing to monitoring

*RequestManagementSystem

FIX: (5960) moved REA initialization inside the initialize() method

*Resources

NEW: (5996) Simple stomp interface
NEW: (6061) add SRM+HTTPs as a default TPC protocol in SRM
CHANGE: (6283) GFA2_SRM2 does not use anymore StorageElements/DefaultProtocols option in the CS (relies on specific OutputProtocols config)
CHANGE: (6283) explicit distinction between PluginName and ProtocolSection name in all the StorageElement components
CHANGE: (6283) API change for the StorageElement. See https://github.com/DIRACGrid/DIRAC/pull/6283 for details
CHANGE: (6283) StorageElement favors local plugins

*ResourceStatusSystem

NEW: (5830) Added tornado version for the three RSS services

*TransformationSystem

NEW: (5829) Added TornadoTransformationManager service
FIX: (5822) TransformationDB: removed old compatibility layer with mySAM tables

*WorkloadManagementSystem

NEW: (5925) added TornadoWMSAdministratorHandler
NEW: (6134) adding a test to show the modifications made to the JDL before being inserted in the JobDB
NEW: (5714) added WMSDashboard and diracLogsDashboard in NDJSON format.
CHANGE: (6336) Remove CPU(MHz) from CacheSize(kB) job parameters
FIX: (5714) removed old index patterns in WMSDashboard.json, updated documentation.

*docs

NEW: (5867) Added general service configuration options description including Authorization options
NEW: (5540) add documentation with information about new system APIs component
CHANGE: (5573) move the ConfReference DIRAC section part to dirac.cfg and add there OAuth 2 settings.
FIX: (5572) update nginx installation instruction

[integration]

*Core

NEW: (6347) Introduce ``DIRAC_MYSQL_OPTIMIZER_TRACES_PATH`` environment variable for advance optimization of MySQL calls
CHANGE: (6347) calls to the MySQL class with keyword parameters MUST now be named
FIX: (6345) Setting codeProperties["version"] in AgentModule

[integration]

*Core

FIX: (6341) createClient now creates subclasses of the original class
NEW: (6335) DIRAC.initialize now provides keyword arguments to customise initialisation
NEW: (6335) SecurityProperty enum for expressing security properties which can be assigned to groups/hosts
NEW: (6335) LogLevel enum for choosing the logging level
NEW: (6309) Introduce DReturnType/DOKReturnType/DErrorReturnType for DIRAC return structures
CHANGE: (6299) Set OpenSSL ciphers to "DEFAULT:SECLEVEL=1"
NEW: (6299) Experimental support for Python 3.10
NEW: (6299) Suppress InsecureRequestWarning when using dirac-configure with HTTPS servers
NEW: (6299) Support overriding ciphers and methods with environment variables when using HTTPS
CHANGE: (6298) Replace suds-jurko with suds
CHANGE: (6297) Use psutil.net_if_addrs for Network.discoverInterfaces
NEW: (6291) adding general flag for disabling Monitoring for Agents (used by JobAgent)
CHANGE: (6286) Default proxy length is 2048 bits
CHANGE: (6286) remove RFC argument for proxy generation

*WorkloadManagement

FIX: (6338) Log error if CE Platform is missing.
CHANGE: (6336) Remove CPU(MHz) from CacheSize(kB) job parameters

*Resources

FIX: (6320) Forward jobDesc to innerCE in PoolCE
FIX: (6318) Set HOME in SingularityCE
FIX: (6317) Return proper error if provider value missing in CloudCE
CHANGE: (6283) GFA2_SRM2 does not use anymore StorageElements/DefaultProtocols option in the CS (relies on specific OutputProtocols config)
CHANGE: (6283) explicit distinction between PluginName and ProtocolSection name in all the StorageElement components
CHANGE: (6283) API change for the StorageElement. See https://github.com/DIRACGrid/DIRAC/pull/6283 for details
CHANGE: (6283) StorageElement favors local plugins

*WorkloadManagementSystem

FIX: (6288) ElasticJobParametersDB: do not return parameters that are not needed

*tests

FIX: (6287) replication transformation test: avoid target==source
FEAT: (6287) transformation tests: allow -disk SEs, avoid RAL, RALPP, CESNET (don't work for me, YMMV)

*MonitoringSystem

FIX: (6286) ElasticSearchDB generates full index names based on UTC
FIX: (6286) DataOperationSender works on a copy of the documents
CHANGE: (6286) DataOperationSender fails if the master accounting system fails

[integration]

*ConfigurationSystem

CHANGE: (6285) Make GOCDB2CSAgent functionMap less private

*Core

CHANGE: (6281) Removed Core.Utilities.MemStat module (use psutil module instead)
CHANGE: (6281) Removed possibility to Register a DISET service with a secondary name

*Resources

FIX: (6275) Conditional FC ProxyPlugin checks the ThreadConfig

*DataManagementSystem

FIX: (6268) Fix FC change{group,owner,mode} for multiple LFNs

*WorkloadManagement

NEW: (6267) ContainerBind option can be set on the per-site and per-CE level

*WorkloadManagementSystem

FIX: (6256) Making getJobParameters handle specific parameters
CHANGE: (6178) removing filtering user sites by platform during the CHECKING state

*FrameworkSystem

NEW: (6248) Add StdoutJsonBackend
FIX: (6246) dirac_login HTML error hiding the actual error