Roundup

Fixed:

- CVE-2024-39124 - The classhelpers (_generic.help.html) are
vulnerable to an XSS attack. A specially crafted URL that used
that endpoint would result in running a script embedded in the
URL. (Found/reported by Alec Romano (4rdr), fix/tests John
Rouillard)
- CVE-2024-39125 - If the Referer header is set to a script tag,
it will be executed when the error in the Referer header is
reported. (Found/reported by Alec Romano (4rdr), fix/tests John
Rouillard)
- CVE-2024-39126 - PDF, XML and SVG files attached to an issue can contain
embedded JavaScript. This JavaScript was executed when the file was
accessed. PDF files are now downloaded and not displayed in the
browser. A content security policy is added for all download files
which prevents code execution in SVG files. (Found/reported by Alec
Romano (4rdr), fix/tests John Rouillard)
- issue2551282 - MySQL utf8mb4 issues and
issue2551115 - Use utf8mb4 as a default for MySQL instead of utf8
The default database type and collations have been set to:
utf8mb4, utf8mb4_unicode_ci and utf8mb4_0900_bin. They are (sadly)
configurable from config.ini. Require directions on upgrading the
MySQL db have been documented in upgrading.txt.
- issue2551063 - Rest/Xmlrpc interfaces needs failed login protection.
Failed API login rate limiting with expiring lockout added. (John
Rouillard)
- issue2551184 - improve i18n handling. Patch to test to make sure it
uses the test tracker's locale files and not other locale
files. (Marcus Priesch)
- issue2551283 - fail if version 2.4.9 of markdown2 is used, it broke
[issue1](issue1) style links. Support markdown2 2.4.8 and earlier
and 2.4.10 with its new schema filtering method. (John Rouillard)
- multiple flake8 fixes (John Rouillard)
- rename loop variable in 'for sendto in sendto:' (John Rouillard)
- issue2551193 - Fix roundup for removal of cgi and cgitb standard
python modules (and FieldStorage/MiniFieldStorage). Replaced imports
from cgi to use roundup.anypy.cgi_ which will load the system cgi
unless it is missing. Then it will load roundup.anypy.vendored.cgi
and make *FieldStroage symbols available. Roundp uses its own
cgitb.py and not the system cgitb.py. It looks like it's the
precursor to the system cgitb.py. (John Rouillard)
- issue2551278 - datetime.datetime.utcnow deprecation. Replace
calls with equivalent that produces timezone aware dates rather than
naive dates. (John Rouillard)
- when using "roundup-admin display" indent the listing only if
headers or protected fields are requested. This makes the output
look like it did previously to 2.3.0 if the new features aren't
used. Roundup-admin output was never meant to be machine parsed, but
don't break it unless required. (John Rouillard)
- issue2551290 - pip install roundup Hangs on Windows 10
The install under windows goes into an infinite loop using pip or
source install. (John Rouillard)
- Document use of pyreadline3 to allow roundup-admin to have CLI editing
on windows. (John Rouillard)
- issue2551293 - remove schema_hook from Tracker instance. Looks like
it was an obsolete hook used for testing. Never documented and not
accessible from schema.py.
- Fix roundup-admin security command. Lowercase its optional
argument. Roles are indexed by lower case role name. So 'security
User' and 'security user' should generate the same output. (John
Rouillard from issue on mailing list by Chuck Cunningham)
- make roundup-server exit more quickly on ^C. This seems to be
limited to windows. (John Rouillard)
- Fix error handling so failure during import of a non-user item
doesn't cause a second traceback. (Found by Norbert Schlemmer, fix
John Rouillard)
- Handle out of memory error when importing large trackers in
PostgreSQL. (Found by Norbert Schlemmer, extensive testing by
Norbert, fix John Rouillard)
- use unittest.mock rather than mock for
test/test_hyperdbvals.py. (found by Ralf Schlatterbeck. Fix John
Rouillard)
- disable proxy with wget in roundup_healthcheck. (Norbert SCHLEMMER
Noschvie on github.com)
- support dicttoxml2.py for Roundup running on 3.7 and
newer. dicttoxml uses a type alias: collection.Iterator that is
dropped in Python 3.10. (found by Norbert Schlemmer, fix John
Rouillard)
- fix duplicate html id 'password' in user.item.html in all templates except
jinja2. (John Rouillard)
- fix unclosed file when saving index in indexer_dbm.py. (John Rouillard)
- fix task index in devel tracker so it doesn't cause a crash if all
fields are selected. (John Rouillard)
- fix windows install. When using pip share directory is installed in
a directory tree under the lib directory. Fix it so that Lib/share
is used to install the share tree. The lets Roundup find tracker
templates and translation files. (Found by Simon Eigeldinger, fix
John Rouillard)
- fix roundup-demo, interactive mode would nuke an existing tracker.
(Found Tonu Mikk, fix John Rouillard)
- fix detection/reporting when using a SQLite3 library without FTS5
support. Install docs updated to state that FTS5 support is required
when using SQLite for back end. (Found Tonu Mikk, fix John
Rouillard)
- issue2551320: user.help-search.html doesn't respect
properties. Setting url parameter properties when using the
classhelp for users now shows the requested properties. (Found by
Patel Malav and Nikunj Thakkar of the UMass-Boston CS682 Spring
2024 class; fix John Rouillard)
- use ast.eval_literal() rather than eval() to turn CSV exported
string values into Python object/values.
- use template's guess at Content-Type in headers only if Content-Type
is not already set. This allows a template to set its own content
type. For example: _generic.translate can set content type (via
request.client.additional_headers) to application/json and return
json from the template. This json could access the 1i18n functions
for a javascript helper. (John Rouillard)
- when template processing raises an exception the line number is
sometimes missing. This causes cgitb to raise a second exception
which clobbers the info about the template issue. As a stop-gap set
the line number to -1 so the original traceback can be seen. This
could be a bug in ZopeTAL. (John Rouillard)
- issue2551328 - REST results show next link if number of results is a
multiple of page size. There should be no next link. (Found by Patel
Malav and Bharath Kanama of the UMass-Boston CS682 Spring 2024
class; fix John Rouillard)
- issue2551264 - REST X-Total-Count header and total_size count
incorrect when paginated - correct values are now returned.
(John Rouillard)
- issue2551331 - Fix repeat first/last methods. (John Rouillard)
- Fix import/export on windows. Use unix line terminating characters.
(John Rouillard)
- Fix anydbm session/otks clear() method on windows when backed by
dumbdbm. Also make anydbm detect the initialized database when
using dumbdbm. (John Rouillard)
- Use of '-' directory in static_files config option under windows
Python fixed. (John Rouillard)
- issue2551334 - number of test bugs that prevented test suite from
running under Windows Python are fixed. WIP. (John Rouillard)
- issue2551302 - Remove support for sqlite version 1 from
back_sqlite.py. We have been using sqlite3 for over a decade. (John
Rouillard)
- issue2551285 - Remove StructuredText support. reStructuredText is
still supported. (John Rouillard)
- Use roundup-demo -p option to set listening port. Was ignored
before. (John Rouillard)
- issue2551346 - Classic tracker's statusauditor raises error if
detectors/config.ini missing
STATUSAUDITOR_CHATTING_REQUIRES_TWO_USERS. The statusauditor.py for
jinja2 and classic templates has been changed to assume that this
option is off when the setting is missing from
detectors/config.ini. Other templates do not implement this option.
(John Rouillard)
- issue2551350 - Python changes for 3.12 with roundup 2.3.0. Fixes for
cgitb.py crash due to pydoc.html.header() signature change. (Patch
by Andrew (kragacles), applied John Rouillard)
- issue2551350 - Python changes for 3.12 with roundup 2.3.0. Fixes for
mailer.py crash due to change in starttls signature change. (Patch
by Andrew (kragacles), modified and applied John Rouillard)
- make classhelper link open in a new window by setting
target="_blank". This prevents overwriting of current page with the
classhelper if javascript is disabled. (John Rouillard)
- issue2551341 - if columns missing from an index url, the
group headers colspan property = 0. Add "or 100" in
stanza's so headers span all rows (up to 100).
- fix roundup-server response requiring a 301 redirect. Did
not set content length leading to hang/error. (John
Rouillard)
- report basename of filename when template file is invalid
rather than reporting a TypeError. (John Rouillard)
- Make Last-Modified header use GMT not -0000 timezone. Fix error
reported by redbot testing. (John Rouillard)
- Send Vary: Accept-Encoding on any file that could be compressed
even if the file is not encoded/compressed. Found by Redbot
testing. (John Rouillard)
- make If-None-Match work for static file (file) case. Found by
Redbot testing (John Rouillard)
- Send vary: accept-encoding for if-modified-since conditional
requests where the file is not modified. (John Rouillard)
- Update JWT example in rest.py to use replacement for
datetime.datetime.utcnow(). (John Rouillard)
- issue2551219 - document requirements of PEM file when using
roundup-server in SSL/TLS mode. Report better error messages
when PEM file is missing certificate or private key. (John
Rouillard)
- Cleanup tracker index generation by roundup-server. Send
correct Content-Length headers so HTTP/1.1 connections don't
hang. (John Rouillard)
- Fix delay when using csv export actions. The CSV file is written
incrementally, so we can't determine the Content-Length. When using
HTTP/1.1, this causes a delay while the browser waits for a timeout.
Forcing the connection to close after the CSV file is written
removes the delay. (John Rouillard)

Features:

- issue2551323 - Remove XHTML support. Disabled option to set
html_version to xhtml. Running roundup commands with html_version
set to xhtml will result in an "Invalid value for HTML_VERSION:
'xhtml'" error. (John Rouillard)
- issue2551103 - add pragma 'display_protected' to roundup-admin. If
true, print protected attributes like id, activity, actor...
when using display or specification subcommands. (John Rouillard)
- add -P pragma=value command line option to roundup-admin. Allows
setting pragmas when using non-interactive mode. (John Rouillard)
- issue685275 - add pragma show_retired to control display of retired
items when using list/table. Add pragma display_header to print
headers for display command. Header displays designator and
retired/active status. (John Rouillard)
- issue2551299 - support config.ini rdbms option 'service'. Allow use
of a PostgreSQL connection service file (pg_service.conf) for
configuring database on a per-tracker basis. Also replaces use of
PGSERVICE env variable for single instance trackers. (From ML
question by ivanov. John Rouillard)
- issue2550852 - support for specifying a PostgreSQL schema to use for
the Roundup database. (Patch by Stuart McGraw; slight modifications,
tests, docs: John Rouillard).
- issue2551274: add configurable logging for REST API when something
fails, we now log status code and error message.
(Ralf Schlatterbeck)
- issue2551317 - add some Jinja2 examples to customizing.txt
document. (John Rouillard)
- multiple scripts/... updates - Python3, linting, enhancements:
weekly-report,schema-dump.py, roundup-reminder, copy-user.py,
dump_dbm_sessions_db.py, contributors.py (John Rouillard)
- roundup/msgfile.py can now be called as 'python msgfmt.py de.po de.mo'
or 'python msgfmt.py -o de.mo de.po' to compile a translation file if
GNU msgfmt is missing. (John Rouillard)
- save roundup-admin history between sessions. Load
~/.roundup_admin_rlrc file to set history-size persistently. Add
pragma history_length to override for a session. (John Rouillard)
- the roundup-admin history command now dumps the journal entries
in a more human readable format. Use the raw option to get the older
machine parsible output. (John Rouillard)
- Multiple JWT secrets are supported to allow key rotation. See
an updated config.ini for details. (John Rouillard)
- issue2551212 - wsgi performance improvement feature added in 2.2.0
is active by default. Can be turned off if needed. See upgrading.txt
for info. (John Rouillard)
- issue2551270 - Better templating support for JavaScript. Add
utils.readfile(file, optional=False) and utils.expandfile(file,
token_dict=None, optional=False). Allows reading an external file
(e.g. JavaScript) and inserting it using tal:contents or equivalent
jinja function. expandfile allows setting a dictionary and tokens in
the file of the form "%(token_name)s" will be replaced in the file
with the values from the dict. (John Rouillard)
- add group to rest interface collection queries. Useful when using
optgroup in select elements. (John Rouillard)
- roundup-demo can set the hostname in the URL using the -H
parameter. So you can start a demo tracker that is available from
your network using 'roundup-demo ... -B hostname -H hostname'. (John
Rouillard)
- issue2551347 - make _generic.help.html work without property
settings. THis applies to classic or minimal trackers. It allows use
of classhelp without the property seting for informtion only
(e.g. description of what a priority or status means) without being
able to select the property in the classhelper. Good for adding help
for Link properties. (John Rouilllard)
- issue1525113 - notation to filter by logged-in user. Use
current_user with properties that are a Link to the 'user' class to
match the currently logged in user. Allows sharing of queries like
"Issues I created" or "Issues I am assigned to" by removing the
hard coded user id number and replacing it with the current user's
id. Tracker templates updated to use it. (John Rouillard from a
patch by Jon C. Thomason)
- Add a /rest/data/user/roles REST endpoint. (John Rouillard)
- issue2551353 - Add roundup-classhelper for 2.4.0
release. Integrate new classhelper web component to wrap
existing classhelper link. This fixes a number of
outstanding bugs against the current classhelper using
current web features. (Patel Malav, Nikunj Thakkar,
Bharath Kanama with integration by John Rouillard)
- disable spellcheck on all password fields to try to prevent
browser from exposing passwords to external servers. (John
Rouillard)

Fixed:

- Updated directions for verifying Roundup distribution using pgp.
- Dockerfile healthcheck fixed so it works when trackers are
specified on command line. Also cleanup of unneeded
packages. (John Rouillard)
- issue2551224 - Replace dbm db for sessions and otks when using
sqlite. New databases are created for session data (db-session)
and one time key data (db-otk). The data is ephemeral so no
need to migrate. (John Rouillard)
- issue2551223 - Timestamps are truncated in mysql and postgresql
for session and otk database tables. Modify db schema to use a
numeric type that preserves more significant figures. See
upgrading.txt for required steps. (John Rouillard)
- added more testing of BasicDatabase to support use of SQLite
for that purpose. Had to fix memory, rdbms and dbm edge cases
due to new tests. (John Rouillard)
- issue2551138 - roundup-server with ssl under python2 throws
traceback on socket close. Not sure how this got fixed,
but after fixing issue2551137 it was not an issue anymore.
- issue2551137 - roundup-server won't run with ssl under python3
Fixed by using SocketIO and manually adding buffering io and
catching SSL.ZeroReturnError indicating SSL has been shut down.
- add caching header for text/javascript in addition to depricated
application/javascript. (John Rouillard)
- Enable postgres-fts: fix indexer-common::get_indexer so it returns a
postgresql-fts Test code paths in get_indexer. (John Rouillard)
- Fix Postgres native-fts, implement a two phase initialization of the
indexer. The native-fts one gets assigned after the database
connection is open. (John Rouillard)
- fix crash if postgresql native-fts backend is asked to index content
with null bytes. (John Rouillard)
- issue2551232 - modify in-reply-to threading when multiple matches
Change how in-reply-to threading works in the mailgw. If there is
more than one issue with a matching parent message, fall back to
subject matching. See upgrading.txt for details. (John Rouillard)
- issue2551195 - port scripts from optparse to argparse (Ralf Schlatterbeck)
- issue2551246 - mitigation, document how -u doesn't work for
roundup-admin. (John Rouillard)
- Document better that files in the template or static_files
directories accessed via file are available to any user with the
url. (John Rouillard)
- Fix final exception handler in roundup-server to send proper
Content-Length header to the client. (John Rouillard)
- Fix traceback if Origin header is missing. (John Rouillard)
- issue2551250: Fix sorting of detectors even if there are two with the
same name and priority (can happen if they are created in two
different files). (Ralf Schlatterbeck)
- Fix Traceback when a numeric order attribute is empty (Ralf
Schlatterbeck)
- Update some template schema files to assign Register permissions for the
Anonymous user. Replaces the old Create permission. (John Rouillard)
- Allow '*' and explicit origins in allowed_api_origins. Only return
'Access-Control-Allow-Credentials' when not matching '*'. Fixes
security issue with rest when using '*'. (John Rouillard)
- issue2551263: In REST response expose rate limiting, sunset, allow
HTTP headers to calling JavaScript. (John Rouillard)
- issue2551257: When downloading an attached (user supplied file),
make sure that an 'X-Content-Type-Options: nosniff' header is sent.
(John Rouillard)
- issue2551252 - default number of rounds for PKDF2 password increased
to 2,000,000. (John Rouillard)
- issue2551251 - migrate/re-encrypt PBKDF2 password if stored
password used a smaller number of rounds than set in
password_pbkdf2_default_rounds. (John Rouillard)
- upgrade from jquery-3.5.1 to jquery-3.6.3. Update user.help.html
to new version. (John Rouillard)
- Dockerfile scanned with hadolint. Fixed multiple issues. (John Rouillard)
- fix crash due to invalid initialization/reset of configuration.py
option_validators. Crashed roundup-admin on second command if an
option_validator was added by a detector or extension. (John Rouillard)
- Dockerfile uses dumb-init to properly wait for child/zombie
processes. Defense against child process starting from detector
and becoming a zombie when its roundup-server instance exits.
(John Rouillard)
- Move installed frontend/Zope back to frontend/ZRoundup
directory. This better identifies the directory when copied into
the Zope framework. It also matches existing
documentation. (John Rouilard)
- Multiple fixes/updates for installation documentation.
Including docker shell/admin/demo mdoes. (John Rouillard)
- Invalid item identifiers passed to REST endpoint return a 404
rather than a 400 error. E.G. /rest/data/issue/issue4 (rather
than .../issue/4). (John Rouillard)
- issue2551280 - sorted() method of MultilinkHTMLProperty is broken?
(Gabor Nagy report and fix; commit John Rouillard)
- issue2551352 - classic classhelper overwrites current
window if javascript is disabled. It now opens in a new
window (target=_blank). Without javascript it is in read
only mode but... (John Rouillard)

Features:

- Add warning about limited Python 2 support lifetime to install and
upgrading docs. (John Rouillard)
- Dockerfile supports demo mode for instant gratification
8-). Also supports shell and admin mode (John Rouillard)
- Dockerfile build allows adding additional python packages via
pip, setting UID tracker is run under. (John Rouillard)
- issue2551140 - Added redis as a session and otk database for use
with anydbm and sqlite primary databases. (John Rouillard)
- issue2550559 - Pretty printing / formatting for Number types.
Added pretty(format='%0.3f') method to NumberHTMLProperty to
print numeric values. If value is None, return empty string
otherwise str() of value. (John Rouillard)
- sqlite native-fts backend now uses the stopwords list in config.ini
to filter words from queries. (Stopwords are still indexed so that
phrase/proximity searches still work.) (John Rouillard)
- sqlite databases use WAL mode when *created* to improve read
concurrency. Existing sqlite database still use rollback journal
mode. See upgrading.txt for details. (John Rouillard)
- issue2551233 - create new roundup-admin command "templates" list all
template names, location and descriptions. Should help find where
/usr/share/roundup/templates is buried during some install
mechanisms. Does not need a tracker home to run. (John Rouillard)
- Add OAuth authentication to the mailgw script. Now IMAPS can be used
with OAuth as required by several large cloud providers. Move command
line processing of the mailgw script to ``argparse``. Note that the
command line options of the mailgw have changed, see upgrading.txt for
details. (Ralf Schlatterbeck)
- issue2551243: schema-dump.py enhanced with anti-CSRF headers. Flake8
cleanup and python2 support. (John Rouillard)
- issue2551253 - new password hash PBDKF2-SHA512 added. Not
available by default. Follow directions in upgrading document
to use. (John Rouillard)
- roundup-admin migrate command reports the schema version.
- issue2551262 - the mail gateway subject prefix now allows spaces
before/after prefix. Also allow spaces between classname and id
number in prefix designator. So "[ issue 23 ] subject" is parsed
like "[issue23] subject". (John Rouillard)
- [doc]: add section on implementing CSP for Roundup to admin
doc. (John Rouillard)
- issue2551265 - deprecate SSHA password hash method. Users using SSHA
passwords will have their passwords transprently upgraded to PBKDF2
derived hash on next login. (John Rouillard)
- issue2551253 - Modify password PBKDF2 method to use SHA512. New
hash function using PBKDF2-SHA512 available. Will be default in
future. Directions for upgrading security by using it now is
included in upgrading.txt. (John Rouillard)
- issue2551275 - Allow configuring max_children in roundup-server.
When using roundup-server in fork mode, allow raising number of
forked children above the default of 40. (Original patch by Joseph
Myers, config settings and docs by John Rouillard.)
- roundup-admin genconfig does not need a tracker home to run. (John
Rouillard)
- issue2551190 - Allow roundup-admin reindex to work in
batches. Running roundup-admin -i ... reindex issue:1-1000 will
reindex the first 1000 issues while reporting any missing issues
in the range. Also completion progress is reported when indexing a
specific class.
- doc updates: add explanation for SQL code in 1.3.3->1.4.0 upgrade.
document schema table in rdbms backends and how to dump/extract
version from them. (John Rouillard)

Fixed:

- issue2551161 - Fix ResourceWarnings when running with -W default.
Cleaned up leaking file descriptors from zopetal pre-compile, python
module compile and loading localization file. (John Rouillard)
- When using roundup-server with native SSL, only accept TLS v1.2.
Previously it used to accept only TLS v1.1. 1.1 is deprecated by
chrome. I don't expect this to be a major problem since a front
end server (apache, Nginx...) is usually customer facing and
terminates SSL. (John Rouillard)
- Fix hang when valid user without authorization for REST tries to use
the rest interface. (John Rouillard)
- Remove Content-Type and make sure no content is returned by OPTIONS
request in REST interface. (John Rouillard)
- In write_html set the Content-Length when response is not
encoded/compressed. (John Rouillard)
- In REST interface do not raise UsageError for invalid api version.
Return json error with proper message. Fixes crash. (John Rouillard)
- In REST interface, allow extensions on URI less than 6 characters in
length. All other paths with a . in then will be passed through
without change. This allows items like a JWT to be passed as a path
element. (John Rouillard)
- issue2550995 - KeyError classic during roundup-admin install. Add
paths to search for locale and template files.
- issue2551167 - pip install in containerized environments puts
template and locale files under site-packages where roundup can't find
them. Change code to find them under site-packages.
- REST replace hard coded list of child endpoints for /rest/ with list
pulled from registered endpoints. So newly added endpoints are
shown. (John Rouillard)
- issue2551107 - Handle representation of long int in history params
for python3. Causes SyntaxError crash when showing history due to
long int e.g. 2345L. This is not a problem for roundup trackers
created using 1.2.0 or newer. The fix may have predated the 1.2.0
release but where the fix actually landed (representing id as a
string and not as an int) is unknown.
- issue2551175 - Make ETag content-encoding aware. HTTP ETag headers
now include a suffix indicating the content-encoding used to send
the data per rfc7232. Properly validate any form of ETag suffixed or
non-suffixed for If-Match.
- issue2551178 - fix Traceback in Apache WSGI - during file upload
- issue2551179 - make roundup-demo initialize templates using
config_ini.ini overrides. Needed for jinja to set template lang etc.
Recognize minimal template when presented with a full
path. (John Kristensen (jerrykan) and John Rouillard)
- handle configparser.InterpolationSyntaxError raised if value
has a single %. Seems to afect python 3 only. Reported by
nomicon on IRC. (John Rouillard)
- add random delay to session database retry code between 0 and .125
seconds. This seems to help reduce stalled connections when a
number of connections are made at the same time. Log remaining
retries once 5 of them have been used. (John Rouillard)
- issue2551169 - setup.py enters endless loop on gentoo linux python2
installation. Fixed.
- issue2551185 - must set PYTHONPATH=... python2 setup.py install
--prefix=/tmp/r2. Force insert --old-and-unmangable to get it
to use a classic installer and not an easy install. This only
affects python2.
- issue2551186 - Python versions >= 3.3 no longer use socket.sslerror.
Andrew (kragacles) patched uses of socket.sslerror in mailgy.py.
Patch adapted to allow trapping sslerror under both python2 and 3.
(John Rouillard)
- issue2551142 - postgresql reworked to use savepoint/"rollback to"
rather than commit()/rollback(). Using savepoint should be faster.
- issue2551196 - Unset labelprop of a Multilink can lead to Python
error when using context/history. (reported and initial patch: Nagy
Gabor, John Rouillard)
- Fix roundup-server to pass If-Range http header so Ranges work
better. (John Rouillard)
- issue2551183 - Replace references to distutils in
roundup/dist/command (John Rouillard)
- Fix hang if Range request was not able to be satified or a HEAD
request was done.
- Mark strings involved with password reset and registration for
translation. (reported: Thomas Arendsen Hein, John Rouillard)
- issue2551159 - cl.filter fails if filterspec is None (also
group and sort). Passing a sort, group or filterprop param
set to None to any filter() call should not cause a
traceback. It will pretend as though no filter, sort or
group was specified. (John Rouillard)
- issue2551205 - Add support for specifying valid origins
for api: xmlrpc/rest. Allows CORS to work with roundup
backend. (John Rouillard)
- new option added to config.ini: login_empty_passwords set to
no by default. Setting this to yes allows a user with an
empty password to login.
- issue2551207 - Fix sorting by order attribute if order attributes can
be None. Add a test.
- issue2551203 fix CORS requests by providing proper headers and allowing
unauthenticted CORS preflight requests. (Marcus Priesch and John
Rouillard)
- issue2551206 - removed some windows installer references that were missed.
- document use of jinja2 templating as optional in config.ini
file. Report if available or not. (John Rouillard)
- make setup.py install the Zope and wsgi.py frontends under
share/frontends. This matches the install of the cgi-bin/roundup.cgi
frontend. (John Rouillard)
- prevent submit button from showing up when using _generic.item.html
if the user doesn't have edit permissions. (John Rouillard)
- issue2551216 - create new mysql databases using COLLATE
utf8_general_ci to prevent crashes in test suite. (John Rouillard)
- issue2551146 - fix issues with strings that have multiple %s
substutions that were not labeled making i18n difficult/impossible.
(John Rouillard)

Features:

- issue2551147 - Enable compression of http responses in roundup.
Allow roundup to return gzip, (br or zstd with added modules)
Content-Encoded replies. Compression could be done in upstream
proxies/wsgi server but this allows it to occur natively. (John
Rouillard)
- Change tracker templates adding required to login forms. Invokes
browser error reporting if user forgets to fill in a field.
(John Rouillard)
- issue1596345 - filtering user list (need
user.search.hml). Incorporate user search features from
issues.roundup-tracker.org into classic template. Devel and
responsive templates already have this feature.
- issue2550917 - Add a: "Welcome user, you have logged in" ok_message
on login. (Ashley Burke)
- enable HTTP/1.1 for roundup-server. This enables keep-alive for
faster response/loading. Also eliminates stalls when the front end web
server uses http 1.1 but the roundup-server uses 1.0. New option
"-V HTTP/1.0" can turn it off. (John Rouillard)
- issue2551163 - add scripts/Docker/Dockerfile to provide basic support for
containerization. See installation.txt for details. (John Rouillard)
- issue2551163 - add scripts/Docker/docker-compose.yml to get a
mysql/roundup deployment. (Norbert Schlemmer, modified by John
Rouilard)
- REST add openapi_doc decorator to add openapi_doc to
endpoints. Decorate a couple of examples. (John Rouillard)
- REST when incorrect method is used, report allowed methods in error
message as well as in an Allow header. (John Rouillard)
- REST change response to invalid attribute specified in path. Return
400 code not 405 code for this case and improve error. (John
Rouillard)
- REST correct values for some Access-Control-Allow-Methods and
Access-Control-Allow-Headers headers. (John Rouillard)
- issue2550991 - define default cache control settings for javascript
and css assets. (John Rouillard)
- issue2551181 - fragments can be appended to designators. So
issue23msg24 could jump to the element with id msg24 in issue 23.
Before this patch you would have two links issue23 and msg24
separated by (John Rouillard).
- added small utility script to dump dbm based tracker databases
(e.g. db/sessions). (John Rouillard)
- issue2551182 - Enhance configuration module to allow loading values
from an external file. Secrets (passwords, secrets) can specify
file using file:// or file:///. The first line of the file is used
as the secret. This allows committing config.ini to a VCS. (John
Rouillard)
- Added xapian indexer to Docker container. (John Rouillard)
- Add support for indexer type native-fts to use FTS5 for sqlite
databases. (John Rouillard)
- Add support for indexer type native-fts to use PostreSQL's full text
search. (John Rouillard)
- Add better error display to the user. Needed to expose errors in fts5
search syntax to the user while also displaying the template page
structure. (John Rouillard)
- issue2551189 - increase size of words in full text index.
Many terms (like exception names or symbolic constants) are larger
than 25. Also German words are long. Since there is little chance of
fixing German to shorten their words, change indexer maxlength to 50.
(Thomas Arendsen Hein provided patch; patch reworked John Rouillard)
- issue2551184 - add an i18n object to the roundupdb. This makes it
possible to translate error messages in detectors (or actions). The
i18n object is now also correctly set for the mail interface:
previously the 'language' setting in the [mailgw] section seems to
have been ignored. Thanks to Marcus Priesch for the patch.
- issue2551212 - speed up wsgi interface by caching the tracker
instance. Hidden behind a feature flag. See upgrading.txt for
details. (Marcus Priesch with feature flag by John Rouillard)

Fixed:

- issue2551122 - fixing order by a link/multilink broke other props
should be final change for that ticket. (John Rouillard)
- when isset() is used in templates on a StringHTMLProperty, it
returns True. 2.1.0 made default_value work properly. Hyperdb's
String(_Type) class sets the default value to the empty string and
not None. Change __init__ so default_value is None and not "".
roundup-user mailing list thread:
https://sourceforge.net/p/roundup/mailman/roundup-users/thread/20210801020640.73ac1729%40Dell/#msg37328813
(reported by Nagy Gabor. fix: John Rouillard)

Features:

- add image/svg-xml as valid mime type to serve. Was being served as
octet-stream. (John Rouillard)
- improve customizing.txt documentation on use of Special Form
Variables. Added example html inputs to illustrate the doc.
Fix position of designator in doc example. It occurs before
link or other edit command. (John Rouillard)

Fixed:

- Reverse multilink to *the same class* would trigger a traceback about
a modified dictionary on iteration (Ralf Schlatterbeck)
- issue2551086 - Valid class names not documented. Should follow
``[A-z][A-z0-9_]+[A-z_]``. This was never documented or enforced, but
we get obscure errors if the rules are not followed. (Tom
Ekberg tests by John Rouilard)
- issue2550564 - Roundup sets "Precedence: bulk" on all outgoing mail,
which seems wrong. Handle Auto-Submitted header on *inbound* email
like we do precedence bulk. This is part of this issue.
- roundup-admin filter calls find() not filter when using -s -c -S
(John Rouillard)
- When requesting transitive properties via ``fields`` in the REST-API,
an empty link in the transitive property (e.g. author.username when
requesting message properties) would result in a 404 error. Now we're
returning a JSON 'null' value. for an empty link (e.g. empty author in
the example). (John Rouillard)
- sphinxcontrib.cheeseshop is unmaintained and using old http
url. Attempts to override cheeseshop_url failed. Replace call to
cheeseshop in docs with raw html and remove references to
cheeseshop. (John Rouillard)
- issue2551093 - return plain text if markdown formatter throws exception
(reported by Cedric Krier, fix by John Rouillard)
- issue2551094 - make simplemde handle line breaks the same as the
backend markdown formatters. (report: Cedric Krier, patch: Christof
Meerwald)
- issue2551092 - fix crash bug by aligning
``roundup.anypy.email_.decode_header`` with stdlib ``email.header`` and
convert string to bytes for python 3. (Cedric Krier)
- issue2551097 - fix underlying bug in use of fenced codeblocks with
markdown2. Fix for issue2551093 to prevent exception trigger.
(patch: Cedric Krier)
- issue2551099 - disable processing of data url's in markdown. Display
as plain text. (John Rouillard)
- issue2551100 - old jquery has security issues, upgrade it and fix
user.help.html (John Rouillard)
- replace deprecated base64.decodestring with base64.b64decode in
roundup_server.py and roundup_xlmrpc_server.py (reported by
lmsteffan in irc)
- removed run_tests.py. Newer pytest doesn't support generating
stand alone testing bundles. Python 3.9 generates errors running
the current run_tests.py. (reported by lmsteffan in irc)
- issue2551104 - fix issue with markdown autolink next to punctuation (ced)
- removed support for old style trackers that use dbinit.py and
config.py. Also remove all uses of deprecated imp module. (John Rouillard)
- removed support for setting database type using
<database>/backend_name. (John Rouillard)
- fixed some issues when generating translations. Use mappings and
named format parameters so translators can move substituted tokens
in translations. (John Rouillard)
- in rest interface, fix uncaught exceptions when parsing invalid
Content-Type and Accept headers. Document response formats more
fully in doc/rest.txt. (John Rouillard)
- in filter, filter_iter and _materialize_multilinks, use named cursor
with postgresql. This turns of client-side cursor handling and avoids
*large* roundup process (or wsgi process) in case of large results.
Fixes issue2551114. (Ralf Schlatterbeck)
- issue2551108 - fix handling of designator links when formatted
as markdown links. (Reported by Cedric Krier; John Rouillard)
- Fix filename created from mail attachments, fixes issue2551118
- Call verifyPassword even if user does not exist. Address timing
attack to discover valid account names. Useful where anonymous user
is not allowed access. (John Rouillard)
- issue2551126 - AttributeError: 'str' object has no attribute
'local'. Fix traceback caused by DateHTMLProperty.pretty() called
on a string value due to error in some other field. (Reported by
reda, fix: John Rouillard)
- issue2550899 - Migrate setup.py to setuptools; fixes:
issue2550866 'pip install --editable .' fails; et al.
this now requires that setuptools be installed. (Patch by John
Kristensen (jerrykan); additional doc changes (upgrade.txt,
RELEASE.txt) John Rouillard)
- issue2551128 - Impossible to validate a user with unknown timezone
Raise KeyError when an unrecognized timezones is passed to
pytz. (patch Cedric Krier, test John Rouillard)
- issue2551129 - Template not found return 500
Handle traceback caused when requested template is not found.
Return 400 error in this condition. (patch Cedric Krier,
additional change and test John Rouillard)
- issue2551062: roundup-admin security now exits status 1 when
it finds an invalid property. It no longer tries to print the rest
of the security properties. (John Rouillard)
- issue2551078 - Fix traceback caused when putting two id's into a
Link html field. A ValueError is raised. Handle exception and return
value. hyperdb.py now reports 'you may only enter ID values for
property ...' to the user. (John Rouillard)
- issue2551120 - The sorted method of MultilinkHTMLProperty crashes,
if the given property is unset for an element of the list. Crash
fixed. New feature NoneFirst added to method to make unset values
sort at start or end of sorted list. (John Rouillard)
- issue2550648 - keyword boolean search. Issue has multiple problems.
Fix issue where saving the keyword boolean search would remove the
link to open the editor. (John Rouillard)
- issue2551136 - timezone extention crash on Python 3.8. cgi.escape
is used in some template to provide a select box of timezones. It
uses cgi.escape that is deprecated and removed from 3.8 and newer.
Use html.escape with fallback to cgi.escape. (Cedric Krier)
- roundup-server can act as an SSL server. Usually SSL is provided by
a front-end server like nginx, hiawatha, apache. The SSL parameters
have been upgraded to TLS 1.1. Cert is RSA 2048 bytes with SHA512
signature. Without these upgrades, ssl mode won't start. Note this
exposes other issue with roundup-server operating as an SSL
endpoint. See issue2551138 and issue2551137. (John Rouillard)
- issue2551122 - sorted method of MultilinkHTMLProperty does a string
sort even if the property is an integer. Fixed so that the orderprop
for the linked class is used. (John Rouillard, reported by Nagy Gabor)
- issue2550964 - History can (temporarily) show incorrect value when a
change is rejected. Fix history function to always use the database
values and ignore the current setting in the form. (John Rouillard)
- Fix find() with anydbm. Using protected properties raised KeyError.
Add shortcut fast return. Both changes come from rdbms_common.py's
find(). (John Rouillard)
- Fix traceback caused by calling history() with arguments in a
non-item context. (John Rouillard)
- issue2551141 - roundup-admin returns no such class when restoring
item with duplicate key. Fix incorrect error message when using
roundup-admin to restore a user when the username is already in use.
(John Rouillard)
- issue2551142 - Import of retired node with username after active
node is imported raises unique constraint failure. (Reported by Ganesh
Sittampalam/Heffalump on irc. John Rouillard)
- *** Must run roundup-admin migrate ***
Increment rdbms version from 5 to 6. Mysql rdbms classes were
missing unique key constraint. Found during fix for issue2551142.
See upgrading.txt. (John Rouillard)
- ignore blank lines in CSV class editing. (John Rouillard)

Features:

- issue2550522 - Add 'filter' command to command-line
interface. Filter command was actually added in 2.0.0, but this
issue requested transitive searching. So that::

roundup-admin -i . filter issue assignedto.username=Admin

will work. This also fixes a bug. If assignedto.username had no
matches, all issues would be returned. This is also fixed.
(John Rouillard)
- issue2550716 - Email address displayed after password reset request.
This fix actually made it into 1.6 release. However this release
documents how password reset works in user_guide.txt. (John Rouillard)
- issue2551094 - add new markdown config.ini setting to allow embedded
newlines to cause a linebreak same as GitHub Flavored Markdown.
(Patch: Cedric Krier; Doc change/checkin John Rouillard)
- issue2551096 - enable markdown autolink for email and bare url's.
Modify raw markdown adding appropriate link markers on the fly.
(Cedric Krier)
- issue2551098 - add rel="nofollow" for links generated by markdown2
backend and rel="nofollow noopener" for mistune and markdown
backends. Prevents link spam. noopener prevents security issue when
available. (John Rouillard)
- Added explanation for modifying Fileclass content files to
customizing.txt. Result of mailing list question. (John Rouillard)
- issue2551109 - Improve keyword editing in jinja2 template. (Cedric Krier)
- issue2551117 - Add example systemd config
- Allow admin to configure language used for stemming in xapian
indexer. (John Rouillard request by Nagy Gabor)
- Move memorydb from test to roundup/test to allow regression-testing in
tracker instances without copying code. Also move the test-detectors in
tx_Source_detector.py to roundup/test for two reasons: It's used in the
memorydb convenience functions and it may be useful in other tests. Make
the prefix a parameter of the convenience functions to be usable in other
tests. (Ralf Schlatterbeck)
- pytest suite now starts the server under wsgi and loads the home
page. This test is skipped if the requests module is not installed.
- extract translatable strings from devel and responsive templates. Merge
translations from https://sourceforge.net/p/roundup/code/merge-requests/3/
(John Rouillard. DE translations by Tobias Herp.)
- send_message now allows setting authid to set source of email.
(John Rouillard)
- issue2550837 - New option for web auth (also http header passing).
Allow admin to configure authentication header replacing the default
REMOTE_USER. Also allow arbitrary headers to be passed to the
tracker when using roundup-server behind a proxy. This code is
experimental see upgrading.txt admin_guide.txt. (John Rouillard)