Dirac

[integration]

*WorkloadManagementSystem

CHANGE: (5566) JobParameters - generalize getting CS parameters for Cloud VMTypes also
FIX: (5563) PilotCStoJSONSynchronizer: do not look anymore in releases.cfg
CHANGE: (5555) SiteDirector submits only py3 pilots
FIX: (5541) old OptimizerModule can commit to ElasticJobParametersDB
FIX: (5529) QueueUtilities - do not stop queue instantiation in case of failures
FIX: (5518) CloudDirector - create only one pilot reference

*Core

FIX: (5561) Fix overwriting files from an X509Chain object
NEW: (5543) Utility for parsing DIRAC version number to Py3
CHANGE: (5537) Do not return failures when sending tasks to executors
FIX: (5530) wait that the CS is loaded before checking for master CS in tornado-start-all
FIX: (5516) Fix finding Dwatermark.png for the accounting watermark

*Docs

CHANGE: (5559) Forces docutils to 0.17 because of bug with sphynx

CHANGE: (5546) do not calculate FQDN unnecessarily

*FrameworkSystem

NEW: (5543) dirac-admin-update-instance converts version number to PEP-440 style if needed
CHANGE: (5537) ComponentInstaller: look also for HTTPs services
FIX: (5482) Allow Python 3 style pre-releases to be installed if the primary extension is a pre-release

*ConfigurationSystem

FIX: (5535) VOMS2CSAgent: VOMS users that have multiple DNs and are suspended should no longer lead to "User not registered" errors and have their status correctly set

*Resources

FIX: (5529) ComputingElement - remove unnecessary warning
CHANGE: (5528) Use PilotManagerClient to get CE status instead of interrogating PilotAgentsDB in HTCondorCE
FIX: (5518) OpenStackCE - removed buggy line
NEW: (5518) Utilities - added possibility to specify extra yum installable packages in the configuration
NEW: (5518) Utilities - added possibility to specify ssh connection to the VM in the configuration
FIX: (5518) cloudinit.template - use VMType as the Queue analog
FIX: (5518) dirac_resource_get_parameters - fixed bugs in the VMType getting logic

*Interfaces

FIX: (5522) Dirac.py - JobMonitoring.getJobsSummary now returns a structure and not a string

*RequestManagementSystem

CHANGE: (5521) getRequestFileStatus takes better into account cases of multiple operations on the same file
CHANGE: (5521) physicalRemoval checker expects an LFN and not a PFN

*WorkloadManagement

FIX: (5516) Ensure valenc is bytes before decoding in retrieveOptimizerParam

*TransformationSystem

NEW: (5514) Allow GroupSize parameter to be mutable
NEW: (5514) Add setGroupSize to command transformation CLI

*SecurityLog

CHANGE: (5513) fed up with unreadable job logs...

[integration]

CHANGE: (5546) do not calculate FQDN unnecessarily

*Core

NEW: (5543) Utility for parsing DIRAC version number to Py3
CHANGE: (5537) Do not return failures when sending tasks to executors
FIX: (5530) wait that the CS is loaded before checking for master CS in tornado-start-all
FIX: (5516) Fix finding Dwatermark.png for the accounting watermark

*FrameworkSystem

NEW: (5543) dirac-admin-update-instance converts version number to PEP-440 style if needed
CHANGE: (5537) ComponentInstaller: look also for HTTPs services
FIX: (5482) Allow Python 3 style pre-releases to be installed if the primary extension is a pre-release

*WorkloadManagementSystem

FIX: (5541) old OptimizerModule can commit to ElasticJobParametersDB
FIX: (5529) QueueUtilities - do not stop queue instantiation in case of failures
FIX: (5518) CloudDirector - create only one pilot reference

*ConfigurationSystem

FIX: (5535) VOMS2CSAgent: VOMS users that have multiple DNs and are suspended should no longer lead to "User not registered" errors and have their status correctly set

*Resources

FIX: (5529) ComputingElement - remove unnecessary warning
CHANGE: (5528) Use PilotManagerClient to get CE status instead of interrogating PilotAgentsDB in HTCondorCE
FIX: (5518) OpenStackCE - removed buggy line
NEW: (5518) Utilities - added possibility to specify extra yum installable packages in the configuration
NEW: (5518) Utilities - added possibility to specify ssh connection to the VM in the configuration
FIX: (5518) cloudinit.template - use VMType as the Queue analog
FIX: (5518) dirac_resource_get_parameters - fixed bugs in the VMType getting logic

*Interfaces

FIX: (5522) Dirac.py - JobMonitoring.getJobsSummary now returns a structure and not a string

*RequestManagementSystem

CHANGE: (5521) getRequestFileStatus takes better into account cases of multiple operations on the same file
CHANGE: (5521) physicalRemoval checker expects an LFN and not a PFN

*WorkloadManagement

FIX: (5516) Ensure valenc is bytes before decoding in retrieveOptimizerParam

*TransformationSystem

NEW: (5514) Allow GroupSize parameter to be mutable
NEW: (5514) Add setGroupSize to command transformation CLI

*SecurityLog

CHANGE: (5513) fed up with unreadable job logs...

[integration]

*Interfaces

FIX: (5522) Dirac.py - JobMonitoring.getJobsSummary now returns a structure and not a string

*RequestManagementSystem

CHANGE: (5521) getRequestFileStatus takes better into account cases of multiple operations on the same file
CHANGE: (5521) physicalRemoval checker expects an LFN and not a PFN

*Resources

FIX: (5518) OpenStackCE - removed buggy line
NEW: (5518) Utilities - added possibility to specify extra yum installable packages in the configuration
NEW: (5518) Utilities - added possibility to specify ssh connection to the VM in the configuration
FIX: (5518) cloudinit.template - use VMType as the Queue analog
FIX: (5518) dirac_resource_get_parameters - fixed bugs in the VMType getting logic

*WorkloadManagementSystem

FIX: (5518) CloudDirector - create only one pilot reference

*Core

FIX: (5516) Fix finding Dwatermark.png for the accounting watermark

*WorkloadManagement

FIX: (5516) Ensure valenc is bytes before decoding in retrieveOptimizerParam

*TransformationSystem

NEW: (5514) Allow GroupSize parameter to be mutable
NEW: (5514) Add setGroupSize to command transformation CLI

*SecurityLog

CHANGE: (5513) fed up with unreadable job logs...

*FrameworkSystem

FIX: (5482) Allow Python 3 style pre-releases to be installed if the primary extension is a pre-release

[integration]

*FrameworkSystem

FIX: (5410) ProxyManagerClient - upload proxy without embedded DIRAC group

*DataManagementSystem

CHANGE: (5391) The LcgFileCatalogClient has been removed

Discard https://github.com/DIRACGrid/DIRAC/commit/dfbb53232dba40aff4bf0f04802877c51d078ab1 that resolve error described in previous hackathon https://trello.com/c/y7TgEuF7/40-multi-vo-tests. Now after the last lot of changes, this key returns correctly, so I canceled the last adjustment.

*Interfaces

FIX: (5388) Discard https://github.com/DIRACGrid/DIRAC/commit/dfbb53232dba40aff4bf0f04802877c51d078ab1.

CHANGE: (47) Release constraint on SQLALchemy versions

Package list

<details>
<summary>Click to expand!</summary>

yaml
name: base
channels:
- diracgrid
- conda-forge
- defaults
dependencies:
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=1_gnu
- aiohttp=3.7.4.post0=py39h3811e60_0
- alabaster=0.7.12=py_0
- apache-libcloud=3.3.1=pyh44b312d_0
- async-timeout=3.0.1=py_1000
- attrs=21.2.0=pyhd8ed1ab_0
- babel=2.9.1=pyh44b312d_0
- backcall=0.2.0=pyh9f0ad1d_0
- backports=1.0=py_2
- backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
- boost=1.74.0=py39h5472131_3
- boost-cpp=1.74.0=h312852a_4
- boto3=1.18.21=pyhd8ed1ab_0
- botocore=1.21.21=pyhd8ed1ab_0
- brotlipy=0.7.0=py39h3811e60_1001
- bzip2=1.0.8=h7f98852_4
- c-ares=1.17.2=h7f98852_0
- ca-certificates=2021.5.30=ha878542_0
- cachetools=4.2.2=pyhd8ed1ab_0
- cairo=1.16.0=h6cf1ce9_1008
- certifi=2021.5.30=py39hf3d152e_0
- cffi=1.14.6=py39he32792d_0
- cgsi-gsoap=1.3.11=h876a3cc_2
- chardet=4.0.0=py39hf3d152e_1
- charset-normalizer=2.0.0=pyhd8ed1ab_0
- click=8.0.1=py39hf3d152e_0
- cmreshandler=1.0.0=py_2
- cni=0.8.0=hc0beb16_0
- cni-plugins=0.9.1=ha8f183a_0
- colorama=0.4.4=pyh9f0ad1d_0
- commonmark=0.9.1=py_0
- conda=4.10.3=py39hf3d152e_0
- conda-package-handling=1.7.3=py39h3811e60_0
- cryptography=3.4.7=py39hbca0aa6_0
- curl=7.78.0=hea6ffbf_0
- cycler=0.10.0=py_2
- cyrus-sasl=2.1.27=h230043b_2
- davix=0.7.6=hb44b51d_0
- dcap=2.47.12=hd35416c_3
- decorator=5.0.9=pyhd8ed1ab_0
- diraccfg=0.2.2=pyhd8ed1ab_0
- docopt=0.6.2=py_1
- docutils=0.17.1=py39hf3d152e_0
- elasticsearch=7.13.4=pyhd8ed1ab_0
- elasticsearch-dsl=7.4.0=py39hf3d152e_0
- enum34=1.1.10=py39hde42818_2
- expat=2.4.1=h9c3ff4c_0
- font-ttf-dejavu-sans-mono=2.37=hab24e00_0
- font-ttf-inconsolata=3.000=h77eed37_0
- font-ttf-source-code-pro=2.038=h77eed37_0
- font-ttf-ubuntu=0.83=hab24e00_0
- fontconfig=2.13.1=hba837de_1005
- fonts-conda-ecosystem=1=0
- fonts-conda-forge=1=0
- freetype=2.10.4=h0708190_1
- fribidi=1.0.10=h36c2ea0_0
- future=0.18.2=py39hf3d152e_3
- gct=6.2.1550507116=h8cfc5f6_2
- gettext=0.19.8.1=h0b5b191_1005
- gfal2=2.19.2=h13c04c2_0
- gfal2-util=1.6.0=pyhd8ed1ab_0
- git=2.32.0=pl5321hc30692c_0
- gitdb=4.0.7=pyhd8ed1ab_0
- gitpython=3.1.18=pyhd8ed1ab_0
- glib=2.68.3=h9c3ff4c_0
- glib-tools=2.68.3=h9c3ff4c_0
- glibmm=2.58.1=h778ead0_1
- graphite2=1.3.13=h58526e2_1001
- greenlet=1.1.1=py39he80948d_0
- gsoap_abi=2.8.106=1
- harfbuzz=2.8.2=h83ec7ef_0
- htcondor-classads=9.1.2=hd3c618e_0
- htcondor-utils=9.1.2=h1bc4757_0
- hypothesis=6.14.7=pyhd8ed1ab_0
- icu=68.1=h58526e2_0
- idna=3.1=pyhd3deb0d_0
- imagesize=1.2.0=py_0
- importlib_resources=5.2.2=pyhd8ed1ab_0
- iniconfig=1.1.1=pyh9f0ad1d_0
- ipython=7.26.0=py39hef51801_0
- ipython_genutils=0.2.0=py_1
- jbig=2.1=h7f98852_2003
- jedi=0.18.0=py39hf3d152e_2
- jinja2=3.0.1=pyhd8ed1ab_0
- jmespath=0.10.0=pyh9f0ad1d_0
- jpeg=9d=h36c2ea0_0
- jq=1.6=h36c2ea0_1000
- json-c=0.15=h98cffda_0
- kiwisolver=1.3.1=py39h1a9c180_1
- krb5=1.19.2=hcc1bbae_0
- lcms2=2.12=hddcbb42_0
- ld_impl_linux-64=2.36.1=hea4e1c9_2
- lerc=2.2.1=h9c3ff4c_0
- libarchive=3.5.1=hccf745f_2
- libblas=3.9.0=11_linux64_openblas
- libcblas=3.9.0=11_linux64_openblas
- libcondor_utils=9.1.2=h6d110fb_0
- libcurl=7.78.0=h2574ce0_0
- libdeflate=1.7=h7f98852_5
- libedit=3.1.20191231=he28a2e2_2
- libev=4.33=h516909a_1
- libffi=3.3=h58526e2_2
- libgcc-ng=11.1.0=hc902ee8_8
- libgfortran-ng=11.1.0=h69a702a_8
- libgfortran5=11.1.0=h6c583b3_8
- libglib=2.68.3=h3e27bee_0
- libgomp=11.1.0=hc902ee8_8
- libiconv=1.16=h516909a_0
- liblapack=3.9.0=11_linux64_openblas
- libnghttp2=1.43.0=h812cca2_0
- libntlm=1.4=h7f98852_1002
- libopenblas=0.3.17=pthreads_h8fe5266_1
- libpng=1.6.37=h21135ba_2
- libseccomp=2.4.4=h36c2ea0_0
- libsigcpp=2.10.7=ha770c72_0
- libssh2=1.9.0=ha56f1ee_6
- libstdcxx-ng=11.1.0=h56837e0_8
- libtiff=4.3.0=hf544144_1
- libtool=2.4.6=h58526e2_1007
- libuuid=2.32.1=h7f98852_1000
- libwebp-base=1.2.0=h7f98852_2
- libxcb=1.13=h7f98852_1003
- libxml2=2.9.12=h72842e0_0
- lz4=3.1.3=py39h03635ed_0
- lz4-c=1.9.3=h9c3ff4c_1
- lzo=2.10=h516909a_1000
- m2crypto=0.37.1=py39h3da14fd_1
- markupsafe=2.0.1=py39h3811e60_0
- matplotlib-base=3.4.3=py39h2fa2bec_0
- matplotlib-inline=0.1.2=pyhd8ed1ab_2
- mock=4.0.3=py39hf3d152e_1
- more-itertools=8.8.0=pyhd8ed1ab_0
- multidict=5.1.0=py39h3811e60_1
- munge=0.5.13=h1a52d62_0
- myproxy=6.2.6=hae0e180_2
- mysql-client=8.0.25=hf09c6a7_2
- mysql-common=8.0.25=ha770c72_2
- mysql-libs=8.0.25=hfa10184_2
- mysqlclient=2.0.3=py39he80948d_0
- ncurses=6.2=h58526e2_4
- nordugrid-arc=6.12.0=py39h5ba6e9b_1
- numpy=1.21.2=py39hdbf815f_0
- olefile=0.46=pyh9f0ad1d_1
- oniguruma=6.9.7.1=h7f98852_0
- openjpeg=2.4.0=hb52868f_1
- openldap=2.4.52=h43732ee_1
- openssh=8.6p1=h1fa914a_1
- openssl=1.1.1k=h7f98852_0
- packaging=21.0=pyhd8ed1ab_0
- pango=1.48.7=hb8ff022_0
- parso=0.8.2=pyhd8ed1ab_0
- pcre=8.45=h9c3ff4c_0
- pcre2=10.37=h032f7d1_0
- perl=5.32.1=0_h7f98852_perl5
- pexpect=4.8.0=pyh9f0ad1d_2
- pickleshare=0.7.5=py_1003
- pillow=8.3.1=py39ha612740_0
- pip=21.2.4=pyhd8ed1ab_0
- pixman=0.40.0=h36c2ea0_0
- pluggy=0.13.1=py39hf3d152e_4
- prompt-toolkit=3.0.19=pyha770c72_0
- psutil=5.8.0=py39h3811e60_1
- pthread-stubs=0.4=h36c2ea0_1001
- ptyprocess=0.7.0=pyhd3deb0d_0
- pugixml=1.11.4=h9c3ff4c_0
- py=1.10.0=pyhd3deb0d_0
- pyasn1=0.4.8=py_0
- pyasn1-modules=0.2.7=py_0
- pycosat=0.6.3=py39h3811e60_1006
- pycparser=2.20=pyh9f0ad1d_2
- pycurl=7.44.1=py39h72e3413_0
- pygments=2.9.0=pyhd8ed1ab_0
- pyopenssl=20.0.1=pyhd8ed1ab_0
- pyparsing=2.4.7=pyh9f0ad1d_0
- pysocks=1.7.1=py39hf3d152e_3
- pytest=6.2.4=py39hf3d152e_0
- python=3.9.6=h49503c6_1_cpython
- python-dateutil=2.8.2=pyhd8ed1ab_0
- python-gfal2=1.10.1=py39h4309f8f_0
- python-htcondor=9.1.2=py39h471719e_0
- python-json-logger=2.0.1=pyh9f0ad1d_0
- python-xxhash=2.0.2=py39h3811e60_0
- python_abi=3.9=2_cp39
- pytz=2021.1=pyhd8ed1ab_0
- readline=8.1=h46c0cb4_0
- recommonmark=0.7.1=pyhd8ed1ab_0
- requests=2.26.0=pyhd8ed1ab_0
- rrdtool=1.7.2=h80f2daa_2
- ruamel_yaml=0.15.80=py39h3811e60_1004
- s3transfer=0.5.0=pyhd8ed1ab_0
- scitokens-cpp=0.6.2=h8cfc780_1
- setuptools=49.6.0=py39hf3d152e_3
- sigcpp-2.0=2.10.7=h9c3ff4c_0
- singularity=3.7.1=hca90b9e_0
- six=1.16.0=pyh6c4a22f_0
- smmap=3.0.5=pyh44b312d_0
- snowballstemmer=2.1.0=pyhd8ed1ab_0
- sortedcontainers=2.4.0=pyhd8ed1ab_0
- sphinx=4.1.2=pyh6c4a22f_1
- sphinxcontrib-applehelp=1.0.2=py_0
- sphinxcontrib-devhelp=1.0.2=py_0
- sphinxcontrib-htmlhelp=2.0.0=pyhd8ed1ab_0
- sphinxcontrib-jsmath=1.0.1=py_0
- sphinxcontrib-qthelp=1.0.3=py_0
- sphinxcontrib-serializinghtml=1.1.5=pyhd8ed1ab_0
- sqlalchemy=1.4.22=py39h3811e60_0
- sqlite=3.36.0=h9cd32fc_0
- squashfs-tools=4.4=h6b73730_2
- srm-ifce=1.24.5=h9da80fa_0
- stomp.py=4.1.23=pyh9f0ad1d_0
- subprocess32=3.5.4=py_1
- suds-jurko=0.6=py39hf3d152e_1005
- tk=8.6.10=h21135ba_1
- toml=0.10.2=pyhd8ed1ab_0
- tornado=5.1.1+dirac.1=py39hdb842f6_0
- tornado_m2crypto=0.1.2=pyhde42818_0
- tqdm=4.62.1=pyhd8ed1ab_0
- traitlets=5.0.5=py_0
- typing-extensions=3.10.0.0=hd8ed1ab_0
- typing_extensions=3.10.0.0=pyha770c72_0
- tzdata=2021a=he74cb21_1
- urllib3=1.26.6=pyhd8ed1ab_0
- voms=2.1.0rc0=h2c113ba_1
- wcwidth=0.2.5=pyh9f0ad1d_2
- wheel=0.37.0=pyhd8ed1ab_1
- xmltodict=0.12.0=py_0
- xorg-kbproto=1.0.7=h7f98852_1002
- xorg-libice=1.0.10=h7f98852_0
- xorg-libsm=1.2.3=hd9c2040_1000
- xorg-libx11=1.7.2=h7f98852_0
- xorg-libxau=1.0.9=h7f98852_0
- xorg-libxdmcp=1.1.3=h7f98852_0
- xorg-libxext=1.3.4=h7f98852_1
- xorg-libxrender=0.9.10=h7f98852_1003
- xorg-renderproto=0.11.1=h7f98852_1002
- xorg-xextproto=7.3.0=h7f98852_1002
- xorg-xproto=7.0.31=h7f98852_1007
- xrootd=5.3.0=py39h8ef998a_0
- xxhash=0.8.0=h7f98852_3
- xz=5.2.5=h516909a_1
- yaml=0.2.5=h516909a_0
- yarl=1.6.3=py39h3811e60_2
- zipp=3.5.0=pyhd8ed1ab_0
- zlib=1.2.11=h516909a_1010
- zstd=1.5.0=ha95c52a_0
prefix: /home/runner/work/DIRACOS2/DIRACOS2/diracos


</details>

Changes with respect to 2.9 ignoring build strings

<details>
<summary>Click to expand!</summary>

diff
---
+++
-20,2 +20,2
- - boto3=1.18.14
- - botocore=1.21.14
+ - boto3=1.18.21
+ - botocore=1.21.21
-24 +24
- - c-ares=1.17.1
+ - c-ares=1.17.2
-51 +51
- - elasticsearch=7.14.0
+ - elasticsearch=7.13.4
-76 +76
- - greenlet=1.1.0
+ - greenlet=1.1.1
-81 +81
- - hypothesis=6.14.5
+ - hypothesis=6.14.7
-136 +136
- - matplotlib-base=3.4.2
+ - matplotlib-base=3.4.3
-149 +149
- - numpy=1.21.1
+ - numpy=1.21.2
-165 +165
- - pip=21.2.2
+ - pip=21.2.4
-178 +178
- - pycurl=7.43.0.6
+ - pycurl=7.44.1
-224 +224
- - tqdm=4.62.0
+ - tqdm=4.62.1
-232 +232
- - wheel=0.36.2
+ - wheel=0.37.0

</details>