Gsutil

=======================================
New Features
------------------
- gsutil now outputs a message that estimates the total number of objects for
commands affecting more than 30,000 objects. This value can be adjusted via
task_estimation_threshold in the .boto configuration file.

Bug Fixes
------------------
- Fixed a bug in resumable downloads that could raise UnboundLocalError if
the download file was not readable.
- Updated oauth2client to version 2.2.0, fixing some IOError and OSError cases.
- Fixed bug in gsutil ls that would show update time instead of creation
time when using the JSON API and -L or -l flags.
- Fixed a bug in gsutil rsync -d that could cause erroneous removal of
an extra destination object.
- Fixed downloads to include accept-encoding:gzip logic when appropriate.

Other Changes
------------------
- gsutil rsync now stores modification time (mtime) for cloud objects.
- Changed the default change detection algorithm of gsutil rsync from file
size to file mtime, falling back to checksum and finally file
size as alternatives. This allows for increased accuracy of rsync without
the speed sacrifice that comes from checksum calculation, and makes gsutil
rsync work more similarly to how Linux rsync works. Note that the first
run of a local-to-cloud rsync using this new algorithm may be slower than
subsequent runs, as the cloud objects will not initially have an mtime, and
the algorithm will fall back to the slower checksum-based alternative in
addition to adding mtime to the cloud objects.
- When using the JSON API, gsutil will output a progress message before
retrying a request if enough time has passed since the first attempt.
- Improved error detection in dry runs of gsutil rsync by attempting to open
files that would be copied.
- Added time created and time updated properties to output of gsutil ls -Lb.
- Added archived time property to output of gsutil ls -La and gsutil stat.
- Changed minimum number of source objects for gsutil compose from 2 to 1.
- Removed an HTTP metadata get call from cp, acl, and setmeta commands. This
improves the speed of gsutil cp for small objects by over 50%.
- Several documentation updates and clarifications.

=======================================
Deprecation Notice
------------------
- gsutil support for Python 2.6 is deprecated, and gsutil will stop
supporting Python 2.6 on September 1, 2016. This change is
being made for two reasons. First, Python 2.6 stopped
receiving security patches since October 2013. Second,
removing Python 2.6 support will enable gsutil to add support
for Python 3. Versions of gsutil released prior to September 1,
2016 will continue to work with Python 2.6. However, bug fixes
will not be made retroactively to older gsutil versions, and users
reporting bugs will be asked to upgrade to the current gsutil
version (using Python 2.7, or, when it is supported, Python 3).

Other Changes
-------------
- Improved documentation around Cloud SDK (gcloud) installs.

=======================================
New Features
------------
- gsutil now supports the beta Customer-Supplied Encryption Keys
features for Google Cloud Storage objects, via the JSON API.
This feature allows you to encrypt your data with keys that
are not permanently stored on Google's servers. You can provide
encryption and decryption keys in your .boto configuration file.
As long as an encryption key is specified in the configuration file,
all gs:// objects that gsutil creates will be stored encrypted
with that key, and all encrypted objects will be decrypted when
downloaded. See "gsutil help csek" for more information.
- Added the rewrite command, which can be used to perform
key rotation for objects encrypted with customer-supplied
encryption keys.

Bug Fixes
---------
- Fixed an AttributeError that could occur when running in debug mode
for operations involving wildcards, recursion, or listing.
- Fixed an ArgumentException that could occur when using perfdiag
write throughput tests with parallelism.
- Restored debug mode output for resumable uploads using the JSON API.
- Fixed a bug where rm -r on a bucket subdirectory would exit with
code 1 even though it succeeded.
- Fixed a bug where cp -R of a single file with a destination other
than the bucket root would copy to the bucket root.
- Fixed a bug when using a single process with multiple threads
where CTRL-C would not stop the process until one thread completed
a task.

Other Changes
-------------
- Improved exception logging in debug mode.
- Added "cache-control: no-transform" to all uploads
using compression ("gsutil cp -z" or "gsutil cp -Z") to ensure that
doubly-compressed objects can be downloaded with data integrity
checking.
- Reduced the default number of threads per-process on Linux systems
from 10 to 5.
- Documented additional OS X Unicode limitations.
- The config command now includes the custom client ID and secret in the
configuration file output if the command is run with those values
configured.

=======================================
Bug Fixes
---------
- Fixed an oauth2client dependency break that caused
the PyPi distribution of gsutil to allow oauth2client 2.0.0,
causing all commands to fail with an ImportError.
- Fixed a bug where gsutil could leak multiprocessing.Manager
processes when terminating signals (such as CTRL-C) were sent.
- Fixed a bug where the -q option did not suppress output
from the stat command.
- Fixed a bug where deleting an empty bucket with rm -r
would return a non-zero exit code even when successful.
- Fixed UnicodeEncodeErrors that could occur when using the du
command with a pipe on objects containing non-ASCII characters.

Other Changes
-------------
- Many documentation improvements and clarifications.

=======================================
New Features
------------
- The ls command now supports a -d option (similar to Unix ls -d)
for suppressing recursion into subdirectories.
- The signurl command now accepts JSON-format private key files
generated by the Google Developers Console.
- The signurl command now supports generating resumable upload
start URLs.
- The cp command now supports a -Z option which will gzip-compress all
files (regardless of their extensions) as they are uploaded.

Bug Fixes
---------
- Fixed a bug where an expired OAuth2 token could include the OAuth2
token response as part of the download, causing it to fail end-to-end
integrity checks and be deleted.
- Fixed a bug where streaming downloads using the JSON API would restart
from the beginning of the stream if the connection broke. This bug
could also cause data corruption in streaming downloads, because
streaming downloads are not validated with end-to-end integrity checks.
- Fixed an internal_failure exception that could occur when an OAuth2
token refresh returned a transient error, such as an HTTP 503.
- Fixed a resource deadlock exception in oauth2client that could cause
sliced downloads to hang.
- Fixed a bug where cp/rsync -p would use the default object
ACL for the destination object if the caller did not have OWNER
permission on the source object.
- Fixed a potential race condition when using perfdiag with multiple
threads and/or processes.
- Fixed an AttributeError that could occur when using acl ch -p.
- Fixed a bug where the mv command did not properly handle the global
-m flag.
- Fixed a UnicodeEncodeError that could occur when iterating over
non-Unicode-named directories.
- Fixed a bug where an object name including a wildcard could cause
an infinite loop during a recursive listing.
- Fixed a Unicode bug when using cp or ls on Windows on an object
containing certain Unicode characters. However, even with this fix
Unicode can still be problematic on Windows; consult
"gsutil help encoding" for details.
- Fixed a Windows performance issue during rsync diff generation.
- Fixed a bug in the ordering of ls output.
- Fixed a bug where Windows help output included ANSI escape codes.
- Fixed a compatibility bug with perfdiag -i with input generated prior
to gsutil 4.14.

Other Changes
-------------
- Several documentation updates, including rsync exclusion pattern matching,
service account configuration, cp/rsync recursion flags, multi-object rm,
versioned URL removal, destination subdirectory naming, wildcard behavior,
regional buckets, and s3 connection reset.
- Improvements to Unicode documentation including LANG variable and iconv
tool.

=======================================
Bug Fixes
---------
- Fixed an OverflowError in apitools that caused download
failures for large files on 32-bit machines.
- Removed unnecessary sending of range headers for downloads when
using the XML API.
- Fixed a bug that caused perfdiag to report extremely high throughput
when the -p flag was unspecified and exactly one of the -c or -k flags
were specified.
- Fixed a ValueError that occurred on Python 2.6 with sliced object downloads.

Other Changes
-------------
- HTTP connections for downloads and uploads in the JSON API are now
re-used per-thread.
- When gsutil's automatic update feature prompts and the user
chooses to update, gsutil will now exit with status code 1 after
updating (because the original command was not executed).
- The cp -A flag is disabled when using gsutil -m to ensure that
ordering is preserved when copying between versioned buckets.