Gsutil

=======================================
New Features
------------
- gsutil now supports HTTP proxy configuration via the http_proxy,
https_proxy, or HTTPS_PROXY environment variables. This configuration
is used only if proxy configuration is not present in the .boto
configuration file.
- gsutil rsync now supports regex-based source and destination URL
exclusion via the -x flag.
- The rm command now supports arguments on stdin via the -I flag.

Bug Fixes
---------
- Fixed a bug where perfdiag would fail if netstat was not available.
- Fixed a bug where temporary ca_certs files were not being cleaned up.
- Fixed a bug in rsync to unnecessarily remove or write objects, in some
cases leaving the destination in a non-synchronized state.
caused rsync to unnecessarily remove or rewrite objects.
- Fixed a bug where rsync temporary listing files were not being
cleaned up when the rsync process was killed.
- Fixed a bug where rsync would remove destination URLs if listing the
source encountered a non-retryable failure (for example, if the source
did not exist).
- Fixed a bug where mv would fail for some Unicode filenames.
- Fixed a bug where mv would remove the source URL after skipping the
destination URL.
- Fixed a bug that caused daisy chain uploads to hang if the download thread
raised an exception.
- Fixed a bug where acl ch would return a zero exit code even if it failed.
- Fixed a bug that sometimes caused the progress display to render multiple
times at the end of an upload or download.

Other Changes
-------------
- Resumable uploads of files using the JSON API now send their data in a
single request, making separate HTTP calls only when resuming is necessary.
- The test command now runs tests in parallel by default, and test
parallelism on Windows is now supported.
- All non-streaming downloads are now resumable (and retryable) by default,
regardless of size.
- Canned ACLs and canned default object ACLs are now supported in the JSON
API (previously they would fall back to using the XML API).
- Google Compute Engine service account credential tokens are now cached,
avoiding unnecessary refreshes.
- Improved detection of the Google Compute Engine metadata server,
particularly when using the -m flag for multiprocessing.
- Added new help sections about filename encoding and security/privacy
considerations.
- Download progress is now displayed for small XML API downloads.

=======================================
New Features
------------
- Tab completion now works on gs:// URLs (for Cloud SDK installs only).
To install via Cloud SDK, see https://cloud.google.com/sdk/#Quick_Start
- Streaming uploads (with source URL "-") using the JSON API now buffer
in-memory up to 100MB, allowing large streams to be retried in the event
of network flakiness or service errors.
- Resumable uploads that receive a fatal service error (typically a 410)
are now automatically retried from the beginning.

Bug Fixes
---------
- Fixed an apitools bug that impacted upload performance and caused
"Retrying from byte __ after exception" to print after every 100MiB.
- Fixed _$folder$ placeholder object detection on versioned buckets.
- Removed an unnecessary credential check on load which increased
startup time by over one second in some cases.
- SignURL now properly retries when checking if the signed object is
readable.
- Files with both Content-Encoding and Content-Type gzip are now properly
removed when hash validation fails (only one of the two should be set).
- The x-goog-if-generation-match:0 header now works properly with the XML API.
- Fixed a bug that caused "defacl ch" on a bucket with a private ACL to fail.
- The rm command now properly supports precondition headers.
- Fixed a bug that caused large streaming uploads to fail with the message
"Failure: length too large" when using the JSON API.
- Fixed a bug that caused JSON lifecycle configurations with createdBefore
conditions to fail with a DecodeError.

Other Changes
-------------
- Byte counts now display accurate abbreviations of binary sizing. For example,
messages previously labeled MB are now properly labeled MiB to indicate
2**20 bytes. Only the labeling changed - the actual sizes were always binary.
- Improved Cloud SDK integration, including improved error messages
and instructions.
- The num_retries .boto configuration value now applies to all requests
(previously it was ignored for JSON API requests).
- rsync now works with non-existent destination subdirectories.
- Raised the default resumable upload threshold from 2MB to 8MB to
improve performance.
- Benign retry messages now print only when debug mode is enabled via the
top-level -d flag.
- The top-level -q flag now suppresses suggestions to use the -m flag.
- Command synopsis is now output when the wrong number of arguments are
provided.
- Removed dependency on google-api-python-client module, added dependencies on
oauth2client and six modules.

=======================================

Bug Fixes
---------
- Fixed a TypeError bug that occurred in perfdiag write throughput tests.
- Fixed an rsync bug that caused invalid symlinks to abort the transfer
even when -e option was specified.
- Fixed a perfdiag assumption that ipaddrlist was populated.
- Fixed an AttributeError when setting an invalid canned ACL with defacl set.
- Fixed a bug where non-resumable uploads would include payload in debug output
when for running in debug mode (-D).

Other Changes
-------------
- Added the proxy_rdns configuration variable for clients that
do DNS lookups via a proxy.
- Added the state_dir configuration variable for choosing the location of
gsutil's internal state files, including resumable transfer tracker files.
resumable_tracker_dir configuration variable is now deprecated.
- Added DNS, connection latency, and proxy use information to perfdiag
command.
- perfdiag command will not perform DNS lookups if they are disabled in
boto config.
- perfdiag command will now only attempt to delete uploaded objects when
running write tests.
- Added code coverage support to test command.
- rsync -d now succeeds on a 404 for a to-be-deleted object (for example, when
the object was already deleted by an external process).

=======================================

Bug Fixes
---------
- Fixed a bug that caused resumable uploads to restart if gsutil was
terminated with CTRL-C.
- Fixed a bug in defacl ch command that caused a failure when updating
an existing default object ACL entry.
- Fixed an invalid literal bug during rsync file listing.
- Made several improvements to JSON upload stability, including fixing a bug
which occasionally caused resumable upload hashes not to catch up properly.
- All JSON calls now have socket timeouts, eliminating hangs under
flaky network conditions.
- Fixed a bug where acl ch -g AllAuthenticatedUsers would instead add
AllUsers.
- Fixed a bug that caused object custom metadata not to be preserved when
copying in the cloud.
- Fixed a bug where du -s did not properly elide subdirectories.

Other Changes
-------------
- Parallel composite uploads are now disabled by default until crcmod is
available in major Linux distributions. To re-enable the setting from
prior versions, in the [GSUtil] section of your .boto config file, add:
parallel_composite_upload_threshold=150M
- Non-wildcarded URLs for existing objects now use Get before trying List
(as in gsutil3), and thus are not subject to eventual listing consistency.
- gsutil -D now redacts proxy configuration values in the output.

=======================================

New Features
------------
- Added the hash command, which can calculate hashes of local files.
gsutil already calculates hashes for integrity checking, but this allows
the user to separately calculate the MD5 and CRC32c hashes of a local file.

Bug Fixes
---------
- Many improvements to JSON API media transfers, including far
more robust retry logic.
- Fixed "File changed during upload: EOF..." errors on XML resumable uploads.
- Fixed rsync command to read and write index files in binary mode.
- Fix potential TypeError in _CheckAndHandleCredentialException.
- Fixed possible data corruption when using JSON API uploads for
small files with lines starting with "From:", which would cause
integrity checks to fail.
- Fixed gsutil cp to skip directory placeholders when downloading, avoiding
"directory exists where the file needs to be created" errors.
- Fixed daisy chain cp/rsync for files >= 100MB.
- Fixed a bug in JSON proxy support where the proxy info was sometimes unused.
- Fixed a bug where an acl get on a private default object ACL returned an
error instead of a blank ACL.
- Fixed a JSON API issue with large HTTP responses on 32-bit systems.

Other Changes
-------------
- Improved object listing performance when using the XML API.
- Improved various error messages.
- Improved progress display during media transfer.
- Switched to truncated exponential backoff for retries.
- Improved OS-specific ulimit checks.
- Added some information such as OS and Cloud SDK wrapping to gsutil version,
and changed the output format to be more uniform.
- Daisy chain cp/rsync now supports resumable uploads.
- Improved proxy support for proxy username and passwords.
- x-amz headers are now supported for cp, rsync, and setmeta. x-amz-meta
headers continue to be supported as well.

=======================================

Bug Fixes
---------
- Fix acl/defacl ch changing the role of an existing group.
- Fix unicode and 404 errors when using manifests.
- Fix parallelism configuration bug that limited gsutil rsync to two threads
and could lead to rsync hangs. "gsutil -m rsync" runs much faster, and rsync
uploads of large local files are now faster via parallel composite upload.
Parallel composite uploads of large files are also faster.
- Fix rsync bug with parallel composite uploads.
- Fix TypeError that could occur when running the cp command with no
credentials.

Other Changes
-------------
- Progress indicators for -m cp/rsync commands are now more readable.
- Added command being run to gsutil -d/-D output.
- Lowered default parallelism for 'gsutil -m test' and added hang detection.