Tahoe-lafs

''''''''''''''''''''''''''

Bugfixes and Improvements
-------------------------

- Fix bug in which uploader could fail with AssertionFailure or report
that it had achieved servers-of-happiness when it hadn't. (`1118`_)
- Fix bug in which servers could get into a state where they would
refuse to accept shares of a certain file (`1117`_)
- Add init scripts for managing the gateway server on Debian/Ubuntu
(`961`_)
- Fix bug where server version number was always 0 on the welcome page
(`1067`_)
- Add new command-line command "tahoe unlink" as a synonym for "tahoe
rm" (`776`_)
- The FTP frontend now encrypts its temporary files, protecting their
contents from an attacker who is able to read the disk. (`1083`_)
- Fix IP address detection on FreeBSD 7, 8, and 9 (`1098`_)
- Fix minor layout issue in the Web User Interface with Internet
Explorer (`1097`_)
- Fix rarely-encountered incompatibility between Twisted logging
utility and the new unicode support added in v1.7.0 (`1099`_)
- Forward-compatibility improvements for non-ASCII caps (`1051`_)

Code improvements
-----------------

- Simplify and tidy-up directories, unicode support, test code
(`923`_, `967`_, `1072`_)

.. _`776`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/776
.. _`923`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/923
.. _`961`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/961
.. _`967`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/967
.. _`1051`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1051
.. _`1067`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1067
.. _`1072`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1072
.. _`1083`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1083
.. _`1097`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1097
.. _`1098`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1098
.. _`1099`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1099
.. _`1117`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1117
.. _`1118`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1118

''''''''''''''''''''''''''

New Features
------------

- SFTP support (`1037`_)
Your Tahoe-LAFS gateway now acts like a full-fledged SFTP server. It
has been tested with sshfs to provide a virtual filesystem in Linux.
Many users have asked for this feature. We hope that it serves them
well! See the `FTP-and-SFTP.rst`_ document to get
started.
- support for non-ASCII character encodings (`534`_)
Tahoe-LAFS now correctly handles filenames containing non-ASCII
characters on all supported platforms:

- when reading files in from the local filesystem (such as when you
run "tahoe backup" to back up your local files to a Tahoe-LAFS
grid);
- when writing files out to the local filesystem (such as when you
run "tahoe cp -r" to recursively copy files out of a Tahoe-LAFS
grid);
- when displaying filenames to the terminal (such as when you run
"tahoe ls"), subject to limitations of the terminal and locale;
- when parsing command-line arguments, except on Windows.

- Servers of Happiness (`778`_)
Tahoe-LAFS now measures during immutable file upload to see how well
distributed it is across multiple servers. It aborts the upload if
the pieces of the file are not sufficiently well-distributed.
This behavior is controlled by a configuration parameter called
"servers of happiness". With the default settings for its erasure
coding, Tahoe-LAFS generates 10 shares for each file, such that any
3 of those shares are sufficient to recover the file. The default
value of "servers of happiness" is 7, which means that Tahoe-LAFS
will guarantee that there are at least 7 servers holding some of the
shares, such that any 3 of those servers can completely recover your
file. The new upload code also distributes the shares better than the
previous version in some cases and takes better advantage of
pre-existing shares (when a file has already been previously
uploaded). See the `architecture.rst`_ document [3] for details.

Bugfixes and Improvements
-------------------------

- Premature abort of upload if some shares were already present and
some servers fail. (`608`_)
- python ./setup.py install -- can't create or remove files in install
directory. (`803`_)
- Network failure => internal TypeError. (`902`_)
- Install of Tahoe on CentOS 5.4. (`933`_)
- CLI option --node-url now supports https url. (`1028`_)
- HTML/CSS template files were not correctly installed under
Windows. (`1033`_)
- MetadataSetter does not enforce restriction on setting "tahoe"
subkeys. (`1034`_)
- ImportError: No module named
setuptools_darcs.setuptools_darcs. (`1054`_)
- Renamed Title in xhtml files. (`1062`_)
- Increase Python version dependency to 2.4.4, to avoid a critical
CPython security bug. (`1066`_)
- Typo correction for the munin plugin tahoe_storagespace. (`968`_)
- Fix warnings found by pylint. (`973`_)
- Changing format of some documentation files. (`1027`_)
- the misc/ directory was tied up. (`1068`_)
- The 'ctime' and 'mtime' metadata fields are no longer written except
by "tahoe backup". (`924`_)
- Unicode filenames in Tahoe-LAFS directories are normalized so that
names that differ only in how accents are encoded are treated as the
same. (`1076`_)
- Various small improvements to documentation. (`937`_, `911`_,
`1024`_, `1082`_)

Removals
--------

- The 'tahoe debug consolidate' subcommand (for converting old
allmydata Windows client backups to a newer format) has been
removed.

Dependency Updates
------------------

- the Python version dependency is raised to 2.4.4 in some cases
(2.4.3 for Redhat-based Linux distributions, 2.4.2 for UCS-2 builds)
(`1066`_)
- pycrypto >= 2.0.1
- pyasn1 >= 0.0.8a
- mock (only required by unit tests)

.. _`534`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/534
.. _`608`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/608
.. _`778`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/778
.. _`803`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/803
.. _`902`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/902
.. _`911`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/911
.. _`924`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/924
.. _`937`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/937
.. _`933`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/933
.. _`968`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/968
.. _`973`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/973
.. _`1024`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1024
.. _`1027`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1027
.. _`1028`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1028
.. _`1033`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1033
.. _`1034`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1034
.. _`1037`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1037
.. _`1054`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1054
.. _`1062`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1062
.. _`1066`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1066
.. _`1068`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1068
.. _`1076`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1076
.. _`1082`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1082
.. _architecture.rst: docs/architecture.rst
.. _FTP-and-SFTP.rst: docs/frontends/FTP-and-SFTP.rst

''''''''''''''''''''''''''

Bugfixes
--------

- Correct handling of Small Immutable Directories

Immutable directories can now be deep-checked and listed in the web
UI in all cases. (In v1.6.0, some operations, such as deep-check, on
a directory graph that included very small immutable directories,
would result in an exception causing the whole operation to abort.)
(`948`_)

Usability Improvements
----------------------

- Improved user interface messages and error reporting. (`681`_,
`837`_, `939`_)
- The timeouts for operation handles have been greatly increased, so
that you can view the results of an operation up to 4 days after it
has completed. After viewing them for the first time, the results
are retained for a further day. (`577`_)

''''''''''''''''''''''''''

New Features
------------

- Immutable Directories

Tahoe-LAFS can now create and handle immutable
directories. (`607`_, `833`_, `931`_) These are read just like
normal directories, but are "deep-immutable", meaning that all their
children (and everything reachable from those children) must be
immutable objects (i.e. immutable or literal files, and other
immutable directories).

These directories must be created in a single webapi call that
provides all of the children at once. (Since they cannot be changed
after creation, the usual create/add/add sequence cannot be used.)
They have URIs that start with "URI:DIR2-CHK:" or "URI:DIR2-LIT:",
and are described on the human-facing web interface (aka the "WUI")
with a "DIR-IMM" abbreviation (as opposed to "DIR" for the usual
read-write directories and "DIR-RO" for read-only directories).

Tahoe-LAFS releases before 1.6.0 cannot read the contents of an
immutable directory. 1.5.0 will tolerate their presence in a
directory listing (and display it as "unknown"). 1.4.1 and earlier
cannot tolerate them: a DIR-IMM child in any directory will prevent
the listing of that directory.

Immutable directories are repairable, just like normal immutable
files.

The webapi "POST t=mkdir-immutable" call is used to create immutable
directories. See `webapi.rst`_ for details.

- "tahoe backup" now creates immutable directories, backupdb has
dircache

The "tahoe backup" command has been enhanced to create immutable
directories (in previous releases, it created read-only mutable
directories) (`828`_). This is significantly faster, since it does
not need to create an RSA keypair for each new directory. Also
"DIR-IMM" immutable directories are repairable, unlike "DIR-RO"
read-only mutable directories at present. (A future Tahoe-LAFS
release should also be able to repair DIR-RO.)

In addition, the backupdb (used by "tahoe backup" to remember what
it has already copied) has been enhanced to store information about
existing immutable directories. This allows it to re-use directories
that have moved but still contain identical contents, or that have
been deleted and later replaced. (The 1.5.0 "tahoe backup" command
could only re-use directories that were in the same place as they
were in the immediately previous backup.) With this change, the
backup process no longer needs to read the previous snapshot out of
the Tahoe-LAFS grid, reducing the network load
considerably. (`606`_)

A "null backup" (in which nothing has changed since the previous
backup) will require only two Tahoe-side operations: one to add an
Archives/$TIMESTAMP entry, and a second to update the Latest/
link. On the local disk side, it will readdir() all your local
directories and stat() all your local files.

If you've been using "tahoe backup" for a while, you will notice
that your first use of it after upgrading to 1.6.0 may take a long
time: it must create proper immutable versions of all the old
read-only mutable directories. This process won't take as long as
the initial backup (where all the file contents had to be uploaded
too): it will require time proportional to the number and size of
your directories. After this initial pass, all subsequent passes
should take a tiny fraction of the time.

As noted above, Tahoe-LAFS versions earlier than 1.5.0 cannot list a
directory containing an immutable subdirectory. Tahoe-LAFS versions
earlier than 1.6.0 cannot read the contents of an immutable
directory.

The "tahoe backup" command has been improved to skip over unreadable
objects (like device files, named pipes, and files with permissions
that prevent the command from reading their contents), instead of
throwing an exception and terminating the backup process. It also
skips over symlinks, because these cannot be represented faithfully
in the Tahoe-side filesystem. A warning message will be emitted each
time something is skipped. (`729`_, `850`_, `641`_)

- "create-node" command added, "create-client" now implies
--no-storage

The basic idea behind Tahoe-LAFS's client+server and client-only
processes is that you are creating a general-purpose Tahoe-LAFS
"node" process, which has several components that can be
activated. Storage service is one of these optional components, as
is the Helper, FTP server, and SFTP server. Web gateway
functionality is nominally on this list, but it is always active; a
future release will make it optional. There are three special
purpose servers that can't currently be run as a component in a
node: introducer, key-generator, and stats-gatherer.

So now "tahoe create-node" will create a Tahoe-LAFS node process,
and after creation you can edit its tahoe.cfg to enable or disable
the desired services. It is a more general-purpose replacement for
"tahoe create-client". The default configuration has storage
service enabled. For convenience, the "--no-storage" argument makes
a tahoe.cfg file that disables storage service. (`760`_)

"tahoe create-client" has been changed to create a Tahoe-LAFS node
without a storage service. It is equivalent to "tahoe create-node
--no-storage". This helps to reduce the confusion surrounding the
use of a command with "client" in its name to create a storage
*server*. Use "tahoe create-client" to create a purely client-side
node. If you want to offer storage to the grid, use "tahoe
create-node" instead.

In the future, other services will be added to the node, and they
will be controlled through options in tahoe.cfg . The most important
of these services may get additional --enable-XYZ or --disable-XYZ
arguments to "tahoe create-node".

- Performance Improvements

Download of immutable files begins as soon as the downloader has
located the K necessary shares (`928`_, `287`_). In both the
previous and current releases, a downloader will first issue queries
to all storage servers on the grid to locate shares before it begins
downloading the shares. In previous releases of Tahoe-LAFS, download
would not begin until all storage servers on the grid had replied to
the query, at which point K shares would be chosen for download from
among the shares that were located. In this release, download begins
as soon as any K shares are located. This means that downloads start
sooner, which is particularly important if there is a server on the
grid that is extremely slow or even hung in such a way that it will
never respond. In previous releases such a server would have a
negative impact on all downloads from that grid. In this release,
such a server will have no impact on downloads, as long as K shares
can be found on other, quicker, servers. This also means that
downloads now use the "best-alacrity" servers that they talk to, as
measured by how quickly the servers reply to the initial query. This
might cause downloads to go faster, especially on grids with
heterogeneous servers or geographical dispersion.

Minor Changes
-------------

- The webapi acquired a new "t=mkdir-with-children" command, to create
and populate a directory in a single call. This is significantly
faster than using separate "t=mkdir" and "t=set-children" operations
(it uses one gateway-to-grid roundtrip, instead of three or
four). (`533`_)

- The t=set-children (note the hyphen) operation is now documented in
webapi.rst, and is the new preferred spelling of the
old t=set_children (with an underscore). The underscore version
remains for backwards compatibility. (`381`_, `927`_)

- The tracebacks produced by errors in CLI tools should now be in
plain text, instead of HTML (which is unreadable outside of a
browser). (`646`_)

- The [storage]reserved_space configuration knob (which causes the
storage server to refuse shares when available disk space drops
below a threshold) should work on Windows now, not just
UNIX. (`637`_)

- "tahoe cp" should now exit with status "1" if it cannot figure out a
suitable target filename, such as when you copy from a bare
filecap. (`761`_)

- "tahoe get" no longer creates a zero-length file upon
error. (`121`_)

- "tahoe ls" can now list single files. (`457`_)

- "tahoe deep-check --repair" should tolerate repair failures now,
instead of halting traversal. (`874`_, `786`_)

- "tahoe create-alias" no longer corrupts the aliases file if it had
previously been edited to have no trailing newline. (`741`_)

- Many small packaging improvements were made to facilitate the
"tahoe-lafs" package being included in Ubuntu. Several mac/win32
binary libraries were removed, some figleaf code-coverage files were
removed, a bundled copy of darcsver-1.2.1 was removed, and
additional licensing text was added.

- Several DeprecationWarnings for python2.6 were silenced. (`859`_)

- The checker --add-lease option would sometimes fail for shares
stored on old (Tahoe v1.2.0) servers. (`875`_)

- The documentation for installing on Windows (docs/quickstart.rst)
has been improved. (`773`_)

For other changes not mentioned here, see
<https://tahoe-lafs.org/trac/tahoe-lafs/query?milestone=1.6.0&keywords=!~news-done>.
To include the tickets mentioned above, go to
<https://tahoe-lafs.org/trac/tahoe-lafs/query?milestone=1.6.0>.

.. _`121`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/121
.. _`287`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/287
.. _`381`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/381
.. _`457`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/457
.. _`533`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/533
.. _`577`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/577
.. _`606`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/606
.. _`607`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/607
.. _`637`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/637
.. _`641`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/641
.. _`646`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/646
.. _`681`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/681
.. _`729`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/729
.. _`741`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/741
.. _`760`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/760
.. _`761`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/761
.. _`773`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/773
.. _`786`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/786
.. _`828`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/828
.. _`833`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/833
.. _`859`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/859
.. _`874`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/874
.. _`875`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/875
.. _`931`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/931
.. _`837`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/837
.. _`850`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/850
.. _`927`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/927
.. _`928`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/928
.. _`939`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/939
.. _`948`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/948
.. _webapi.rst: docs/frontends/webapi.rst

''''''''''''''''''''''''''

Improvements
------------

- Uploads of immutable files now use pipelined writes, improving
upload speed slightly (10%) over high-latency connections. (`392`_)

- Processing large directories has been sped up, by removing a O(N^2)
algorithm from the dirnode decoding path and retaining unmodified
encrypted entries. (`750`_, `752`_)

- The human-facing web interface (aka the "WUI") received a
significant CSS makeover by Kevin Reid, making it much prettier and
easier to read. The WUI "check" and "deep-check" forms now include a
"Renew Lease" checkbox, mirroring the CLI --add-lease option, so
leases can be added or renewed from the web interface.

- The CLI "tahoe mv" command now refuses to overwrite
directories. (`705`_)

- The CLI "tahoe webopen" command, when run without arguments, will
now bring up the "Welcome Page" (node status and mkdir/upload
forms).

- The 3.5MB limit on mutable files was removed, so it should be
possible to upload arbitrarily-sized mutable files. Note, however,
that the data format and algorithm remains the same, so using
mutable files still requires bandwidth, computation, and RAM in
proportion to the size of the mutable file. (`694`_)

- This version of Tahoe-LAFS will tolerate directory entries that
contain filecap formats which it does not recognize: files and
directories from the future. This should improve the user
experience (for 1.5.0 users) when we add new cap formats in the
future. Previous versions would fail badly, preventing the user from
seeing or editing anything else in those directories. These
unrecognized objects can be renamed and deleted, but obviously not
read or written. Also they cannot generally be copied. (`683`_)

Bugfixes
--------

- deep-check-and-repair now tolerates read-only directories, such as
the ones produced by the "tahoe backup" CLI command. Read-only
directories and mutable files are checked, but not
repaired. Previous versions threw an exception when attempting the
repair and failed to process the remaining contents. We cannot yet
repair these read-only objects, but at least this version allows the
rest of the check+repair to proceed. (`625`_)

- A bug in 1.4.1 which caused a server to be listed multiple times
(and frequently broke all connections to that server) was
fixed. (`653`_)

- The plaintext-hashing code was removed from the Helper interface,
removing the Helper's ability to mount a
partial-information-guessing attack. (`722`_)

Platform/packaging changes
--------------------------

- Tahoe-LAFS now runs on NetBSD, OpenBSD, ArchLinux, and NixOS, and on
an embedded system based on an ARM CPU running at 266 MHz.

- Unit test timeouts have been raised to allow the tests to complete
on extremely slow platforms like embedded ARM-based NAS boxes, which
may take several hours to run the test suite. An ARM-specific
data-corrupting bug in an older version of Crypto++ (5.5.2) was
identified: ARM-users are encouraged to use recent
Crypto++/pycryptopp which avoids this problem.

- Tahoe-LAFS now requires a SQLite library, either the sqlite3 that
comes built-in with python2.5/2.6, or the add-on pysqlite2 if you're
using python2.4. In the previous release, this was only needed for
the "tahoe backup" command: now it is mandatory.

- Several minor documentation updates were made.

- To help get Tahoe-LAFS into Linux distributions like Fedora and
Debian, packaging improvements are being made in both Tahoe-LAFS and
related libraries like pycryptopp and zfec.

- The Crypto++ library included in the pycryptopp package has been
upgraded to version 5.6.0 of Crypto++, which includes a more
efficient implementation of SHA-256 in assembly for x86 or amd64
architectures.

dependency updates
------------------

- foolscap-0.4.1
- no python-2.4.0 or 2.4.1 (2.4.2 is good) (they contained a bug in base64.b32decode)
- avoid python-2.6 on windows with mingw: compiler issues
- python2.4 requires pysqlite2 (2.5,2.6 does not)
- no python-3.x
- pycryptopp-0.5.15

.. _392: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/392
.. _625: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/625
.. _653: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/653
.. _683: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/683
.. _694: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/694
.. _705: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/705
.. _722: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/722
.. _750: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/750
.. _752: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/752

''''''''''''''''''''''''''

Garbage Collection
------------------

- The big feature for this release is the implementation of garbage
collection, allowing Tahoe storage servers to delete shares for old
deleted files. When enabled, this uses a "mark and sweep" process:
clients are responsible for updating the leases on their shares
(generally by running "tahoe deep-check --add-lease"), and servers
are allowed to delete any share which does not have an up-to-date
lease. The process is described in detail in
`garbage-collection.rst`_.

The server must be configured to enable garbage-collection, by
adding directives to the [storage] section that define an age limit
for shares. The default configuration will not delete any shares.

Both servers and clients should be upgraded to this release to make
the garbage-collection as pleasant as possible. 1.2.0 servers have
code to perform the update-lease operation but it suffers from a
fatal bug, while 1.3.0 servers have update-lease but will return an
exception for unknown storage indices, causing clients to emit an
Incident for each exception, slowing the add-lease process down to a
crawl. 1.1.0 servers did not have the add-lease operation at all.

Security/Usability Problems Fixed
---------------------------------

- A super-linear algorithm in the Merkle Tree code was fixed, which
previously caused e.g. download of a 10GB file to take several hours
before the first byte of plaintext could be produced. The new
"alacrity" is about 2 minutes. A future release should reduce this
to a few seconds by fixing ticket `442`_.

- The previous version permitted a small timing attack (due to our use
of strcmp) against the write-enabler and lease-renewal/cancel
secrets. An attacker who could measure response-time variations of
approximatly 3ns against a very noisy background time of about 15ms
might be able to guess these secrets. We do not believe this attack
was actually feasible. This release closes the attack by first
hashing the two strings to be compared with a random secret.

webapi changes
--------------

- In most cases, HTML tracebacks will only be sent if an "Accept:
text/html" header was provided with the HTTP request. This will
generally cause browsers to get an HTMLized traceback but send
regular text/plain tracebacks to non-browsers (like the CLI
clients). More errors have been mapped to useful HTTP error codes.

- The streaming webapi operations (deep-check and manifest) now have a
way to indicate errors (an output line that starts with "ERROR"
instead of being legal JSON). See `webapi.rst`_ for
details.

- The storage server now has its own status page (at /storage), linked
from the Welcome page. This page shows progress and results of the
two new share-crawlers: one which merely counts shares (to give an
estimate of how many files/directories are being stored in the
grid), the other examines leases and reports how much space would be
freed if GC were enabled. The page also shows how much disk space is
present, used, reserved, and available for the Tahoe server, and
whether the server is currently running in "read-write" mode or
"read-only" mode.

- When a directory node cannot be read (perhaps because of insufficent
shares), a minimal webapi page is created so that the "more-info"
links (including a Check/Repair operation) will still be accessible.

- A new "reliability" page was added, with the beginnings of work on a
statistical loss model. You can tell this page how many servers you
are using and their independent failure probabilities, and it will
tell you the likelihood that an arbitrary file will survive each
repair period. The "numpy" package must be installed to access this
page. A partial paper, written by Shawn Willden, has been added to
docs/proposed/lossmodel.lyx .

CLI changes
-----------

- "tahoe check" and "tahoe deep-check" now accept an "--add-lease"
argument, to update a lease on all shares. This is the "mark" side
of garbage collection.

- In many cases, CLI error messages have been improved: the ugly
HTMLized traceback has been replaced by a normal python traceback.

- "tahoe deep-check" and "tahoe manifest" now have better error
reporting. "tahoe cp" is now non-verbose by default.

- "tahoe backup" now accepts several "--exclude" arguments, to ignore
certain files (like editor temporary files and version-control
metadata) during backup.

- On windows, the CLI now accepts local paths like "c:\dir\file.txt",
which previously was interpreted as a Tahoe path using a "c:" alias.

- The "tahoe restart" command now uses "--force" by default (meaning
it will start a node even if it didn't look like there was one
already running).

- The "tahoe debug consolidate" command was added. This takes a series
of independent timestamped snapshot directories (such as those
created by the allmydata.com windows backup program, or a series of
"tahoe cp -r" commands) and creates new snapshots that used shared
read-only directories whenever possible (like the output of "tahoe
backup"). In the most common case (when the snapshots are fairly
similar), the result will use significantly fewer directories than
the original, allowing "deep-check" and similar tools to run much
faster. In some cases, the speedup can be an order of magnitude or
more. This tool is still somewhat experimental, and only needs to
be run on large backups produced by something other than "tahoe
backup", so it was placed under the "debug" category.

- "tahoe cp -r --caps-only tahoe:dir localdir" is a diagnostic tool
which, instead of copying the full contents of files into the local
directory, merely copies their filecaps. This can be used to verify
the results of a "consolidation" operation.

other fixes
-----------

- The codebase no longer rauses RuntimeError as a kind of
assert(). Specific exception classes were created for each previous
instance of RuntimeError.

- Many unit tests were changed to use a non-network test harness,
speeding them up considerably.

- Deep-traversal operations (manifest and deep-check) now walk
individual directories in alphabetical order. Occasional turn breaks
are inserted to prevent a stack overflow when traversing directories
with hundreds of entries.

- The experimental SFTP server had its path-handling logic changed
slightly, to accomodate more SFTP clients, although there are still
issues (`645`_).

.. _442: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/442
.. _645: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/645
.. _garbage-collection.rst: docs/garbage-collection.rst