Roundup

Features:

- issue2550782: Added a new irker detector to send notifications on IRC
when an issue is created or messages are added. (Ezio Melotti)
- Beta version of responsive templates using devel schema
and Twitter Bootstrap for styling (Pradip Caulagi)
- pywin32 is no longer required to run on Windows (anatoly techtonik)
- Rewritten portalocker.py logic in ctypes for Windows (anatoly techtonik)
- Add an interface to register clearCache callbacks in roundupdb.
Sometimes complicated computations may require an application cache.
This application can now register a callback to clear the application
cache, because roundup knows better when to clear it (usually when a
transaction ends, either with rollback or with commit). The interface
for this is currently considered experimental. The current interface
is registerClearCacheCallback(self, method, param) where method is
called with param as the only parameter. (Ralf Schlatterbeck)
- Add a script to remove file-spam from a tracker, see
scripts/spam-remover. (Ralf Schlatterbeck)

Fixed:

- issue2550765: Don't show links in calendar that will fail.
Found and fixed by C\E9dric Krier. (Bernhard)
- issue2550765: use ``<meta name="robots" content="noindex,
nofollow">`` in the _generic.calendar.html to prevent robots to
follow all the links in the calendar. (Ezio Melotti)
- "BaseException.with_traceback" is not available on Python 2, so use
"raise E, V, T" instead of "raise E(V).with_traceback(T)". This change was
originally introduced in 74476eaac38a. (Ezio Melotti)
- issue2550759: Trailing punctuation is no longer included when URLs are
converted to links. (Ezio Melotti)
- issue2550574: Restore sample detectors removed in roundup 1.4.9
(Thomas Arendsen Hein)
- Prevent AttributeError when removing all roles of a user
(Thomas Arendsen Hein)
- issue2550762 Minor Documentation fix in doc/developers.txt, thanks
to W. Trevor King. (Bernhard Reiter)
- issue2550766: Minor formatting issues in the docs for date properties,
thanks John Kristensen. (Bernhard Reiter)
- issue2550738: Fixes for various documentation typoes,
thanks Nathan Russell. (John Kristensen)
- issue2550756: Fix 'oder' typo in mailer.Mailer.bounce_message docstring,
thanks W. Trevor King (John Kristensen)
- Fix basic authentication: instatiating the login action would fail if
the user is not set. We now first set the user to anonymous and then
try basic authentication if enabled. (Ralf Schlatterbeck)
- Fix xmlrpc permissions for lookup method: Allow if the key attribute
is either searchable or viewable, don't check id attribute (Ralf
Schlatterbeck)
- Fix installation documentation (section Prerequisites) to require at
least python 2.5, thanks to John P. Rouillard for discovering this.
(committed by Ralf Schlatterbeck)
- Fix version_check.py to require at least python 2.5 (anatoly techtonik)
- Fixing the download button re-activating the cheeseshop plugin in the
sphinx config. Thanks to Richard for the hint. (Bernhard Reiter)
- issue2550783 devel template's schema.py permissions referenced the
organization property for the user, but the property is called
organisation. Thanks to Pradip Caulagi. (committed by John Rouillard)
- issue2550749 - the xmlrpc interface is invoked on content type
and not url path. Sending any text/xml data to roundup results in
invoking the xml-rpc interface, but a REST or other interface could
also consume xml data and do something different. So require the use
of 'http(s)://.../xmlrpc' uri to trigger the xmlrpc interface.
(John Rouillard)
- issue2550774: Remove generating documentation with rst2html, and update the
README.txt with how to create the html docs using sphinx, thanks Kai Storbeck
(John Kristensen)
- issue2550774: Include doc/conf.py in the release tarball, so people can build
their own documentation in html, thanks Kai Storbeck (John Kristensen)
- issue2550774: Update website/www/Makefile to symlink COPYING.txt so "make"
works again, thanks Kai Storbeck (John Kristensen)
- issue2550760: Several improvements to the manpages
thanks Kai Storbeck & Bastian Kleineidam (John Kristensen)

Features:

- Experimental support for the new Chameleon templating engine.
We now have two configurable templating engines, the old Zope TAL
templates (called zopetal in the config) and the new Chameleon (called
chameleon in the config). A new config-option "template_engine" under
[main] can take these config-options, the default is zopetal.
Thanks to Cheer Xiao for the idea of making this configurable *and*
for the actual implementation! (Ralf)
WARNING: Chameleon support is highly experimental and *not* recommended for
production use. It has known performance issues and i18n is not yet
functioning. It's still under active development. Only use this feature if
you want to experiment with Chameleon and/or help with Roundup
developement. If you found a bug in Chameleon support, please report after
testing against latest Roundup source from the Mercurial repository.
- issue2550678: Allow pagesize=-1 which returns all results.
Suggested and implemented by John Kristensen.
Tested by Satchidanand Haridas. (Bernhard)
- Allow to turn off translation of generated html options in menu method
of LinkHTMLProperty and MultilinkHTMLProperty -- default is
translation as it used to be (Ralf)
- Sending of OpenPGP encrypted mail to all users or selected users (via
roles) is now working. (Ralf)
- Add config-option "nosy" to messages_to_author setting in [nosy]
section of config: This will send a message to the author only
in the case where the author is on the nosy-list (either added
earlier or via the add_author setting). Current config-options
for this setting will send / not send to author without considering
the nosy list. (Ralf)

Fixed:

- issue2550730: FAQ has broken link to Zope book. Reported and fixed by
John Rouillard.(Bernhard)
- issue2550728: remove buggy parentheses in TAL/DummyEngine.py.
Reported and fixed by Ralf Hemmecke. (Bernhard)
- issue2550715: IndexError when requesting non-existing file via http.
Reported and fixed by C\E9dric Krier. (Bernhard)
- issue2550712: exportcsvaction errors poorly when given invalid columns.
Reported by Will Kahn-Greene, fixed by C\E9dric Krier. (Bernhard)
- issue2550695: 'No sort or group' settings not retained when editing queries.
Reported and fixed by John Kristensen. Tested by Satchidanand Haridas.
(Bernhard)
- Fix matching of incoming email addresses to the alternate_addresses
field of a user -- this would match substrings, e.g. if the user has
discuss-supportexample.com as an alternate email and an incoming mail
is addressed to supportexample.com this would (wrongly) match. (Ralf)
- issue2550729: Fix password history display for anydbm backend, thanks
to Ralf Hemmecke for reporting. (Ralf)
- OpenPGP support is again working (pyme API has changed significantly) and
we now have a regression test. We now take care that bounce-messages
for incoming encrypted mails or mails where the policy dictates that
outgoing traffic should be encrypted is actually OpenPGP encrypted. (Ralf)
- Ignore confirm set() fields by themselves in the absence of non-"confirm"
values; otherwise a bare confirm field can be used to change the a
password. Reported by Cam Blackwood. (Ralf)
- Updated version of simplified Chinese message file by Cheer Xiao:
Corrected some mistakes, added a few more items and did some
formating. (Ralf)
- Fix xmlrpc URL parsing so that passwords may contain a ':' character
(Ralf)
- Be more tolerant when parsing RFC2047 encoded mail headers. Use
backported version of my proposed changes to
email.header.decode_header in http://bugs.python.org/issue1079
(Ralf)
- issue2550684 Fix XSS vulnerability when username contains HTML code,
thanks to Thomas Arendsen Hein for reporting and patch. (Ralf)
- issue2550711 Fix XSS vulnerability in action parameter,
thanks to "om" for reporting. (Ralf)
- issue2550535 In some cases even when keep_quoted_text=yes is
configured we would strip quoted sections. This hit the python
bug-tracker especially for python interpreter examples with leading
'>>>' strings. The fix is slightly different compared to the proposal
as this broke keep_quoted_text=no in certain cases. We also fix a bug
where keep_quoted_text=no would drop the last line of a non-quoted
section if there wasn't an empty line between the next quotes. (Ralf)
- issue2431638 wrong registration link in bounce mail for non-registered
users reported *years* ago by anonymous (Ralf)
- Fix doc/upgrading.txt which produces errors with latest docutils about
wrong block structure. Fix .gitignore in doc directory. Thanks to
Cheer Xiao for the patches. (Ralf)
- Fix wrong execute permissions on some files, thanks to Cheer Xiao for
the patch. (Ralf)
- Fix override of TemplatingUtils in instance.py, thanks to Cheer Xiao
for the patch. (Ralf)
- Fix another XSS with the "otk" parameter, thanks to Jesse Ruderman for
reporting. (Ralf)
- Mark cookies HttpOnly and -- if https is used -- secure. Fixes
issue2550689, but is untested if this really works in browsers.
Thanks to Joseph Myers for reporting. (Ralf)
- Fix another XSS with the ok- and error message, see issue2550724. We
now escape messages when added to the list so we can decide whether to
escape a message individually for each message. The default is to
escape. Thanks to David Benjamin for the bug-report and to Ezio
Melotti for several proposed fixes. (Ralf)

Features:

- Xapian indexing improved: Slightly faster and slightly smaller database.
Closes issue2550687. Thanks to Olly Betts for the patch. (Bernhard Reiter)
- PostgreSQL backend minor improvement: database creation less likely to fail
for PostgreSQL versions >= 8.1 as the table "postgres" is used by default.
Closes issue2550543. Thanks to Kai Storbeck for the patch. (Bernhard Reiter)
- Allow HTMLRequest.batch to filter on other permissions than "View"
(e.g. on the new "Search" permission") by adding a "permission"
parameter. Thanks to Eli Collins for the patch. Closes issue2550699. (Ralf)

Fixed:

- Installation: Fixed an issue that prevented to use EasyInstall
and a Python egg. Thanks to Satchidanand Haridas for the patch and
John Kristensen for testing it. (Bernhard Reiter)
- The PostgreSQL backend quotes database names now for CREATE and DROP,
enabling more exotic tracker names. Closes issue2550497.
Thanks to Sebastian Harl for providing the patch. (Bernhard Reiter)
- Updated the url to point to www.roundup-tracker.org in two places in the
docs. (Bernhard Reiter)
- Do not depend on a CPython implementation detail anymore to make Roundup
more compatible with other Python implementations like PyPy.
Closes issue2550707. Thanks to Christof Meerwald. (Bernhard Reiter, Richard)
- Yet another fix to the mail gateway, messages got *all* files of
an issue, not just the new ones. Thanks to Rafal Bisingier for
reporting and proposing a fix. The regression test was updated.
(Ralf)
- Fix version numbers in upgrade documentation, the file-unlink defect
was in 1.4.17 not 1.4.16. Thanks to Rafal Bisingier. (Ralf)
- Fix encoded email header parsing if multiple encoded and non-encoded
parts are present. RFC2047 specifies that spacing is removed only
between encoded parts, we always removed the space. Note that this bug
was present before mail gateway refactoring :-) Thanks for thorough
testing of mail gateway code by Rafal Bisingier. (Ralf)
- The "Retire" permission was not being registered. (Richard)
- Fix StringIO issue2550713: io.StringIO in newer versions of python
returns unicode strings and expects a unicode string in the
constructor. Unfortunately csv doesn't handle unicode (yet). So we
need to use a BytesIO which gets the utf-8 string from the
web-interface. Compatibility for old versions by using
StringIO.StringIO for emulating a io.BytesIO also works.
Thanks to C\E9dric Krier for reporting. Closes issue2550713.
Added a regression test for EditCSVAction (Ralf)
- Fix issue2550691 where a Unix From-Header was sometimes inserted in
outgoing emails, thanks to Joseph Myers for the patch. (Ralf)

Features:

- Norwegian Bokmal translation by Christian Aastorp (Ralf)
- Allow to specify additional cc and bcc emails (not roundup users) for
nosymessage used by the nosyreaction reactor. (Ralf)

Fixed:

- File-unlink defect in mailgw fixed! If an email was received
that contained no attachments, all previous files of the issue were unlinked.
This defect was introduced with the 1.4.17 release as an unwanted result
of the mail gate code refactoring. Thanks to Rafal Bisingier for reporting
and proposing a fix. There is now a regression test in place. (Ralf)

Features:

- Allow declaration of default_values for properties in schema.
- Add explicit "Search" permissions, see Security Fix below.
- Add "lookup" method to xmlrpc interface (Ralf Schlatterbeck)
- Multilinks can be filtered by combining elements with AND, OR and NOT
operators now. A javascript gui was added for "keywords", see issue2550648.
Developed by Sascha Teichmann; funded by Intevation. (Bernhard Reiter)
- Factor MailGW message parsing into a separate class, thanks to John
Kristensen who did the major work in issue2550576 -- I wouldn't
have attempted it without this. Fixes issue2550576. (Ralf)
- Now if the -C option to roundup-mailgw specifies "issue" this refers
to an issue-like class. The real class is determined from the
configured default class, or the -c option to the mailgw, or the class
resulting from mail subject parsing. We also accept multiple -S
options for the same class now. (Ralf)
- Optimisation: Late evaluation of Multilinks (only in rdbms backends):
previously we materialized each multilink in a Node -- this creates an
SQL query for each multilink (e.g. 'files' and 'messages' for each
line in the issue index display) -- even if the multilinks aren't
displayed. Now we compute multilinks only if they're accessed (and
keep them cached).
- Add a filter_iter similar to the existing filter call. This feature is
considered experimental. This is currently not used in the
web-interface but passes all tests for the filter call except sorting
by Multilinks (which isn't supported by SQL and isn't a sane concept
anyway). When using filter_iter instead of filter this saves a *lot*
of SQL queries: Filter returns only the IDs of Nodes in the database,
the additional content of a Node has to be fetched in a separate SQL
call. The new filter_iter also returns the IDs of Nodes (one by one,
it's an iterator) but pre-seeds the cache with the content of the
Node. The information needed for seeding the cache is retrieved in the
same SQL query as the ids.

Fixed:

- Security Fix: Add a check for search-permissions: now we allow
searching for properties only if the property is readable without a
check method or if an explicit search permission (see above unter
"Features) is given for the property. This fixes cases where a user
doesn't have access to a property but can deduce the content by
crafting a clever search, group or sort query.
see doc/upgrading.txt for how to fix your trackers! (Ralf Schlatterbeck).
- Range support in roundup-server so large files can be served,
e.g. media files on iOS/iPads; issue2550694. (Bernhard Reiter;
Thanks to Jon C. Thomason for the patch.)
- Fix search for xapian 1.2 issue2550676
(Bernhard Reiter; Thanks to Olly Betts for providing the patch.)
- Some minor typos fixed in doc/customizing.txt (Thanks Ralf Hemmecke).
- XML-RPC documentation now linked from the docs/index (Bernhard Reiter).
- Fix setting of sys.path when importing schema.py, fixes issue2550675,
thanks to Bryce L Nordgren for reporting. (Ralf Schlatterbeck)
- clear the cache on commit for rdbms backends: Don't carry over cached
values from one transaction to the next (there may be other changes
from other transactions) see new ConcurrentDBTest for a
read-modify-update cycle that fails with the old caching behavior.
(Ralf Schlatterbeck)
- Fix incorrect setting of template in customizing.txt example action,
patch via issue2550682 (thanks John Kristensen)
- Configuration issue: On some postgresql 8.4 installations (notably on
debian squeeze) the default template database used for database
creation doesn't match the needed character encoding UTF8 -- a new
config option 'template' in the rdbms section now allows specification
of the template. You know you need this option if you get the error
message:
psycopg2.DataError: new encoding (UTF8) is incompatible with the
encoding of the template database (SQL_ASCII)
HINT: Use the same encoding as in the template database, or use
template0 as template.
(Ralf Schlatterbeck)
- Fixed bug in mailgw refactoring, patch issue2550697 (thanks Hubert
Touvet)
- Fix Password handling security issue2550688 (thanks Joseph Myers for
reporting and Eli Collins for fixing) -- this fixes all observations
by Joseph Myers except for auto-migration of existing passwords.
- Add new config-option 'migrate_passwords' in section 'web' to
auto-migrate passwords at web-login time. Default for the new option
is "yes" so if you don't want that passwords are auto-migrated to a
more secure password scheme on user login, set this to "no" before
running your tracker(s) after the upgrade.
- Add new config-option 'password_pbkdf2_default_rounds' in 'main'
section to configure the default parameter for new password
generation. Set this to a higher value on faster systems which want
more security. Thanks to Eli Collins for implementing this (see
issue2550688).
- Fix documentation for roundup-server about the 'host' parameter as
suggested in issue2550693, fixes the first part of this issue. Make
'localhost' the new default for this parameter, note the upgrading
documentation of changed behaviour. We also deprecate the empty host
parameter for binding to all interfaces now (still left in for
compatibility). Thanks to Toni Mueller for providing the first version
of this patch and discussing implementations.
- Fixed bug in filter_iter refactoring (lazy multilinks), in rare cases
this would result in duplicate multilinks to the same node. We're now
going the safe route and doing lazy evaluation only for read-only
access, whenever updates are done we fetch everything.

Features:

- allow trackers to override the classes used to render properties in
templating per issue2550659 (thanks Ezio Melotti)
- new mailgw configuration item "subject_updates_title": If set to "no"
a changed subject in a reply to an issue will not update the issue
title with the changed subject. Thanks to Arkadiusz Kita and Peter
Funk for requesting the feature and discussing the implementation.
http://thread.gmane.org/gmane.comp.bug-tracking.roundup.user/10169
- new rdbms config item sqlite_timeout makes the previously hard-coded
timeout of 30 seconds configurable. This is the time a client waits
for the locked database to become free before giving up. Used only for
SQLite backend.
- new mailgw config item unpack_rfc822 that unpacks message attachments
of type message/rfc822 and attaches the individual parts instead of
attaching the whole message/rfc822 attachment to the roundup issue.

Fixed:

- fixed reporting of source missing warnings
- relevant tests made locale independent, issue2550660 (thanks
Benni B\E4rmann for reporting).
- fix for incorrect except: syntax, issue2550661 (thanks Jakub Wilk)
- No longer use the root logger, use a logger with prefix "roundup",
see http://thread.gmane.org/gmane.comp.bug-tracking.roundup.devel/5356
- improve handling of '&gt;' when URLs are converted to links, issue2550664
(thanks Ezio Melotti)
- fixed registration, issue2550665 (thanks Timo Paulssen)
- make sorting of multilinks in the web interface more robust, issue2550663
- Fix charset of first text-part of outgoing multipart messages, thanks Dirk
Geschke for reporting, see
http://thread.gmane.org/gmane.comp.bug-tracking.roundup.user/10223
- Fix handling of incoming message/rfc822 attachments. These resulted in
a weird mail usage error because the email module threw a TypeError
which roundup interprets as a Reject exception. Fixes issue2550667.
Added regression tests for message/rfc822 attachments with and without
configured unpacking (mailgw unpack_rfc822, see Features above)
Thanks to Benni B\E4rmann for reporting.
- Allow search_popup macro to work with all db classes, issue2550567
(thanks John Kristensen)
- lower memory footprint for (journal-) import