Tahoe-lafs

''''''''''''''''''''''''''

Security-related Bugfix
-----------------------

- Fix flaw that would allow servers to cause undetected corruption when
retrieving the contents of mutable files (both SDMF and MDMF). (`1654`_)

.. _`1654`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1654

''''''''''''''''''''''''''

New Features
------------

- The most significant new feature in this release is MDMF: "Medium-size
Distributed Mutable Files". Unlike standard SDMF files, these provide
efficient partial-access (reading and modifying small portions of the file
instead of the whole thing). MDMF is opt-in (it is not yet the default
format for mutable files), both to ensure compatibility with previous
versions, and because the algorithm does not yet meet memory-usage goals.
Enable it with ``--format=MDMF`` in the CLI (``tahoe put`` and ``tahoe
mkdir``), or the "format" radioboxes in the web interface. See
`<docs/specifications/mutable.rst>`__ for more details (`393`_, `1507`_)
- A "blacklist" feature allows blocking access to specific files through
a particular gateway. See the "Access Blacklist" section of
`<docs/configuration.rst>`__ for more details. (`1425`_)
- A "drop-upload" feature has been added, which allows you to upload
files to a Tahoe-LAFS directory just by writing them to a local
directory. This feature is experimental and should not be relied on
to store the only copy of valuable data. It is currently available
only on Linux. See `<docs/frontends/drop-upload.rst>`__ for documentation.
(`1429`_)
- The timeline of immutable downloads can be viewed using a zoomable and
pannable JavaScript-based visualization. This is accessed using the
'timeline' link on the File Download Status page for the download, which
can be reached from the Recent Uploads and Downloads page.

Configuration/Behavior Changes
------------------------------

- Prior to Tahoe-LAFS v1.3, the configuration of some node options could
be specified using individual config files rather than via ``tahoe.cfg``.
These files now cause an error if present. (`1385`_)
- Storage servers now calculate their remaining space based on the filesystem
containing the ``storage/shares/`` directory. Previously they looked at the
filesystem containing the ``storage/`` directory. This allows
``storage/shares/``, rather than ``storage/``, to be a mount point or a
symlink pointing to another filesystem. (`1384`_)
- ``tahoe cp xyz MUTABLE`` will modify the existing mutable file instead of
creating a new one. (`1304`_)
- The button for unlinking a file from its directory on a WUI directory
listing is now labelled "unlink" rather than "del". (`1104`_)

Notable Bugfixes
----------------

- The security bugfix for the vulnerability allowing deletion of shares,
detailed in the news for v1.8.3 below, is also included in this
release. (`1528`_)
- Some cases of immutable upload, for example using the ``tahoe put`` and
``tahoe cp`` commands or SFTP, did not appear in the history of Recent
Uploads and Downloads. (`1079`_)
- The memory footprint of the verifier has been reduced by serializing
block fetches. (`1395`_)
- Large immutable downloads are now a little faster than in v1.8.3 (about
5% on a fast network). (`1268`_)

Packaging Changes
-----------------

- The files related to Debian packaging have been removed from the Tahoe
source tree, since they are now maintained as part of the official
Debian packages. (`1454`_)
- The unmaintained FUSE plugins were removed from the source tree. See
``docs/frontends/FTP-and-SFTP.rst`` for how to mount a Tahoe filesystem on
Unix via sshfs. (`1409`_)
- The Tahoe licenses now give explicit permission to combine Tahoe-LAFS
with code distributed under the following additional open-source licenses
(any version of each):

* Academic Free License
* Apple Public Source License
* BitTorrent Open Source License
* Lucent Public License
* Jabber Open Source License
* Common Development and Distribution License
* Microsoft Public License
* Microsoft Reciprocal License
* Sun Industry Standards Source License
* Open Software License

Compatibility and Dependencies
------------------------------

- To resolve an incompatibility between Nevow and zope.interface (versions
3.6.3 and 3.6.4), Tahoe-LAFS now requires an earlier or later
version of zope.interface. (`1435`_)
- The Twisted dependency has been raised to version 10.1 to ensure we no
longer require pywin32 on Windows, the new drop-upload feature has the
required support from Twisted on Linux, and that it is never necessary to
patch Twisted in order to use the FTP frontend. (`1274`_, `1429`_,
`1438`_)
- An explicit dependency on pyOpenSSL has been added, replacing the indirect
dependency via the "secure_connections" option of foolscap. (`1383`_)

Minor Changes
-------------

- A ``man`` page has been added (`1420`_). All other docs are in ReST
format.
- The ``tahoe_files`` munin plugin reported an incorrect count of the number
of share files. (`1391`_)
- Minor documentation updates: 627, 1104, 1225, 1297, 1342, 1404
- Other minor changes: 636, 1355, 1363, 1366, 1388, 1392, 1412, 1344,
1347, 1359, 1389, 1441, 1442, 1446, 1474, 1503

.. _`393`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/393
.. _`1079`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1079
.. _`1104`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1104
.. _`1268`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1268
.. _`1274`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1274
.. _`1304`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1304
.. _`1383`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1383
.. _`1384`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1384
.. _`1385`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1385
.. _`1391`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1391
.. _`1395`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1395
.. _`1409`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1409
.. _`1420`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1420
.. _`1425`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1425
.. _`1429`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1429
.. _`1435`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1435
.. _`1438`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1438
.. _`1454`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1454
.. _`1507`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1507

''''''''''''''''''''''''''

Security-related Bugfix
-----------------------

- Fix flaw that would allow a person who knows a storage index of a file to
delete shares of that file. (`1528`_)
- Remove corner cases in mutable file bounds management which could expose
extra lease info or old share data (from prior versions of the mutable
file) if someone with write authority to that mutable file exercised these
corner cases in a way that no actual Tahoe-LAFS client does. (Probably not
exploitable.) (`1528`_)

.. _`1528`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1528

''''''''''''''''''''''''''

Compatibility and Dependencies
------------------------------

- Tahoe is now compatible with Twisted-10.2 (released last month), as
well as with earlier versions. The previous Tahoe-1.8.1 release
failed to run against Twisted-10.2, raising an AttributeError on
StreamServerEndpointService (`1286`_)
- Tahoe now depends upon the "mock" testing library, and the foolscap
dependency was raised to 0.6.1 . It no longer requires pywin32
(which was used only on windows). Future developers should note that
reactor.spawnProcess and derivatives may no longer be used inside
Tahoe code.

Other Changes
-------------

- the default reserved_space value for new storage nodes is 1 GB
(`1208`_)
- documentation is now in reStructuredText (.rst) format
- "tahoe cp" should now handle non-ASCII filenames
- the unmaintained Mac/Windows GUI applications have been removed
(`1282`_)
- tahoe processes should appear in top and ps as "tahoe", not
"python", on some unix platforms. (`174`_)
- "tahoe debug trial" can be used to run the test suite (`1296`_)
- the SFTP frontend now reports unknown sizes as "0" instead of "?",
to improve compatibility with clients like FileZilla (`1337`_)
- "tahoe --version" should now report correct values in situations
where 1.8.1 might have been wrong (`1287`_)

.. _`1208`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1208
.. _`1282`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1282
.. _`1286`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1286
.. _`1287`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1287
.. _`1296`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1296
.. _`1337`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1337

''''''''''''''''''''''''''

Bugfixes and Improvements
-------------------------

- Allow the repairer to improve the health of a file by uploading some
shares, even if it cannot achieve the configured happiness
threshold. This fixes a regression introduced between v1.7.1 and
v1.8.0. (`1212`_)
- Fix a memory leak in the ResponseCache which is used during mutable
file/directory operations. (`1045`_)
- Fix a regression and add a performance improvement in the
downloader. This issue caused repair to fail in some special
cases. (`1223`_)
- Fix a bug that caused 'tahoe cp' to fail for a grid-to-grid copy
involving a non-ASCII filename. (`1224`_)
- Fix a rarely-encountered bug involving printing large strings to the
console on Windows. (`1232`_)
- Perform ~ expansion in the --exclude-from filename argument to
'tahoe backup'. (`1241`_)
- The CLI's 'tahoe mv' and 'tahoe ln' commands previously would try to
use an HTTP proxy if the HTTP_PROXY environment variable was set.
These now always connect directly to the WAPI, thus avoiding giving
caps to the HTTP proxy (and also avoiding failures in the case that
the proxy is failing or requires authentication). (`1253`_)
- The CLI now correctly reports failure in the case that 'tahoe mv'
fails to unlink the file from its old location. (`1255`_)
- 'tahoe start' now gives a more positive indication that the node has
started. (`71`_)
- The arguments seen by 'ps' or other tools for node processes are now
more useful (in particular, they include the path of the 'tahoe'
script, rather than an obscure tool named 'twistd'). (`174`_)

Removed Features
----------------

- The tahoe start/stop/restart and node creation commands no longer
accept the -m or --multiple option, for consistency between
platforms. (`1262`_)

Packaging
---------

- We now host binary packages so that users on certain operating
systems can install without having a compiler.
<https://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/README.html>
- Use a newer version of a dependency if needed, even if an older
version is installed. This would previously cause a VersionConflict
error. (`1190`_)
- Use a precompiled binary of a dependency if one with a sufficiently
high version number is available, instead of attempting to compile
the dependency from source, even if the source version has a higher
version number. (`1233`_)

Documentation
-------------

- All current documentation in .txt format has been converted to .rst
format. (`1225`_)
- Added docs/backdoors.rst declaring that we won't add backdoors to
Tahoe-LAFS, or add anything to facilitate government access to data.
(`1216`_)

.. _`71`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/71
.. _`174`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/174
.. _`1212`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1212
.. _`1045`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1045
.. _`1190`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1190
.. _`1216`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1216
.. _`1223`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1223
.. _`1224`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1224
.. _`1225`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1225
.. _`1232`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1232
.. _`1233`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1233
.. _`1241`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1241
.. _`1253`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1253
.. _`1255`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1255
.. _`1262`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1262

''''''''''''''''''''''''''

New Features
------------

- A completely new downloader which improves performance and
robustness of immutable-file downloads. It uses the fastest K
servers to download the data in K-way parallel. It automatically
fails over to alternate servers if servers fail in mid-download. It
allows seeking to arbitrary locations in the file (the previous
downloader which would only read the entire file sequentially from
beginning to end). It minimizes unnecessary round trips and
unnecessary bytes transferred to improve performance. It sends
requests to fewer servers to reduce the load on servers (the
previous one would send a small request to every server for every
download) (`287`_, `288`_, `448`_, `798`_, `800`_, `990`_,
`1170`_, `1191`_)
- Non-ASCII command-line arguments and non-ASCII outputs now work on
Windows. In addition, the command-line tool now works on 64-bit
Windows. (`1074`_)

Bugfixes and Improvements
-------------------------

- Document and clean up the command-line options for specifying the
node's base directory. (`188`_, `706`_, `715`_, `772`_,
`1108`_)
- The default node directory for Windows is ".tahoe" in the user's
home directory, the same as on other platforms. (`890`_)
- Fix a case in which full cap URIs could be logged. (`685`_,
`1155`_)
- Fix bug in WUI in Python 2.5 when the system clock is set back to
1969. Now you can use Tahoe-LAFS with Python 2.5 and set your system
clock to 1969 and still use the WUI. (`1055`_)
- Many improvements in code organization, tests, logging,
documentation, and packaging. (`983`_, `1074`_, `1108`_,
`1127`_, `1129`_, `1131`_, `1166`_, `1175`_)

Dependency Updates
------------------

- on x86 and x86-64 platforms, pycryptopp >= 0.5.20
- pycrypto 2.2 is excluded due to a bug

.. _`188`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/188
.. _`288`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/288
.. _`448`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/448
.. _`685`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/685
.. _`706`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/706
.. _`715`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/715
.. _`772`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/772
.. _`798`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/798
.. _`800`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/800
.. _`890`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/890
.. _`983`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/983
.. _`990`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/990
.. _`1055`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1055
.. _`1074`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1074
.. _`1108`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1108
.. _`1155`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1155
.. _`1170`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1170
.. _`1191`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1191
.. _`1127`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1127
.. _`1129`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1129
.. _`1131`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1131
.. _`1166`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1166
.. _`1175`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1175