Wiredtiger

The WiredTiger 2.9.1 release contains new features, new supported platforms, minor API changes and bug fixes. Refer to the [change log](http://source.wiredtiger.com/2.9.1/md_changelog.html) for more information.

New features and API changes; refer to the API documentation for full details:
- SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers. See the upgrading documentation for details
- WT-283 Add a new WT_SESSION::alter method that can be used to reconfigure table metadata
- WT-2670 Change the default file system access pattern advice for data files from random to no advice. Add access_pattern_hint configuration option for WT_SESSION::create API that can be used to advise the file system of expected access semantics. See the upgrading documentation for details.
- WT-3034 Add support for including updates when reading from named snapshots

Significant changes and bug fixes:
- WT-2960 Reduce likelihood of using the lookaside file, especially when inserting multi-megabyte values
- WT-3056 Allow projected table and join cursors to use primary keys
- WT-3070 Fix a bug in search_near on indexes

The WiredTiger 2.9.0 release contains new features, new supported platforms, minor API changes and bug fixes. Refer to the [change log](http://source.wiredtiger.com/2.9.0/md_changelog.html) for more information.

New features and API changes; refer to the API documentation for full details:
- WT-2360 Add nesting and disjunction functionality to cursor joins
- WT-2552 Add a public API to allow custom filesystem implementations
- WT-2738 Remove the ability to change the default checkpoint name. See the upgrading documentation for details.
- WT-2711 Change the statistics log configuration options. Restrict where statistics log files can be created to remove security vulnerability. See upgrading documentation for details.
- WT-2764 Enhance the checkpoint implementation, which led to a change in default eviction settings. See the upgrading documentation for details.
- WT-2880 Add Zstandard compression support
- See the upgrading documentation for information about renamed and removed statistics.

Significant changes and bug fixes:
- WT-2026 Fix a bug where in-memory pages were allowed to grow too large
- WT-2343 Ensure we cannot rename/drop when a backup cursor is open
- WT-2520 Ensure WT_SESSION::verify does not alter tables
- WT-2576 Ensure variable-length column-store cannot out-of-order return
- WT-2664 Change eviction so any eviction thread can find candidates
- WT-2702 Resolve an issue where under high thread load, WiredTiger exceeds cache size
- WT-2731 Resolve an issue where raw compression can create pages that are larger than expected
- WT-2737 Allow reconciliation to scrub dirty pages rather than evicting them
- WT-2757 Fix a bug in column stores where it was not possible to retrieve the new record number after an append
- WT-2816 Improve eviction performance

The WiredTiger 2.8.0 release contains new features, new supported platforms, minor API changes and bug fixes. Refer to the [change log](http://source.wiredtiger.com/2.8.0/md_changelog.html) for more information.

New features and API changes; refer to the API documentation for full details:
- WT-60 Port WiredTiger to run on big endian platforms
- WT-2287 Add a new WT_SESSION.rebalance API
- WT-2333 Add a lock_wait configuration setting to WT_SESSION.drop to avoid blocking
- WT-2349 Add a readonly configuration setting to wiredtiger_open
- WT-2363 Remove built in support for bzip2 compression
- WT-2404 Add streaming pack/unpack methods to the extension API

Significant changes and bug fixes:
- WT-1801 Add a directory sync after rollback of a WT_SESSION::rename operation
- WT-2130 Improve on-disk page utilization with random workloads
- WT-2275 Fix a database corruption after truncate and crash
- WT-2264 High update workloads can cause checkpoints to never complete
- WT-2290 Improve effectiveness of WT_SESSION.compact
- WT-2361 Fix a bug in column-store where verify identifies out of order data
- WT-2367 Fix a bug in WT_CURSOR.next that could cause out-of-order key returns
- WT-2374 Fix a bug where a database was corrupted when restoring a backup
- WT-2381 Fix the dump utility to include the table configuration
- WT-2451 Allow the WiredTiger metadata to be evicted
- WT-2490 Fix a bug in column-store where search_near() returns the wrong key

The WiredTiger 2.7.0 release contains new features, minor API changes and bug fixes.

New features and API changes; refer to the API documentation for full details:
- 959376c WT-147: Create indexes on non-empty tables.
- 4368d39 WT-1315: Add an implementation of cursor joins via a new WT_SESSION::join API.
- 944ccd1 WT-1350: Add a new configuration option to ::wiredtiger_open and WT_CONNECTION::reconfigure called "eviction_dirty_trigger" that causes eviction to start evicting dirty pages from cache once the given threshold has been reached.
- ab5a8fb WT-1728: Add a WT_SESSION::reset method to release resources held by a session.
- 263c5b7 WT-1930: Allow setting "file_manager=(close_idle_time=0)" to ::wiredtiger_open and WT_CONNECTION::reconfigure to disable closing idle handles.
- 6310c3f WT-1959: Change verify to distinguish between warnings and errors. Add a new strict mode to verify that causes warnings to be reported as errors. Use strict mode to match earlier behavior. See the upgrading documentation for more information.
- e0d6229 WT-1980: Add a new "metadata:create" URI to WT_SESSION::open_cursor for metadata cursors that return strings useful for passing to WT_SESSION::create.
- 292712e WT-2065: Add a new configuration option to ::wiredtiger_open and WT_CONNECTION::reconfigure called "shared_cache=(quota)" that limits the amount of shared cache a participant can be assigned.
- 4d0ebf4 WT-2104: Add a method to flush log files via a new WT_SESSION::log_flush API. Made WT_SESSION::commit_transaction configuration options match WT_SESSION::log_flush. Change the default WT_SESSION::transaction_sync timeout to 20 minutes rather than infinity.
- 21b8330 WT-2151: Enhance logging configuration to allow reconfiguration and add a new "log=(zero_fill)" configuration option that causes WiredTiger to zero-fill log files on creation.
- 368b307 WT-2200: Add a new configuration option to ::wiredtiger_open called "write_through" that causes WiredTiger to specify the FILE_FLAG_WRITE_THROUGH on Windows when writing files (default false, including when "direct_io" is configured).
- 08c0fcd WT-2217: After a successful call to WT_CURSOR::insert, the key and value will be cleared from the cursor. See the upgrading documentation for more information.
- d4fc69a SERVER-17078: Add a "statistics=(size)" mode to statistics cursors, which allows for retrieving file size only.
- b83b901 SERVER-18356: Changed the handling of the "config_base" option to ::wiredtiger_open. See upgrading documentation for more information.

See the NEWS file in the release package for information about statistics changes and other significant changes since the last release.

The WiredTiger 2.6.1 release contains new features, minor API changes and bug
fixes.

New features:
- Move the sync configuration setting from WT_SESSION::begin_transaction to WT_SESSION::commit_transaction. Change the setting from a boolean to a string. See upgrading documentation for more information.
refs WT-1908
- Add the ability to flag a transaction to be flushed asynchronously on commit via a new sync=[background] configuration option. Add a new WT_SESSION::transaction_sync API to wait for asynchronous flushes to
complete.
refs WT-1908, 1943
- Add the ability to create a named in-memory snapshot via a new WT_SESSION::snapshot API.
refs WT-1839
- Add the ability to disable write ahead logging at a per-table granularity. Accessed via log=(enabled) configuration for WT_SESSION::create API. Partial logging has serious implications for recovery, it should be used with caution.
refs 1989

Other noteworthy changes:
- Fix several bugs related to syncing files for checkpoint durability.
refs WT-1944
- Fix a segfault during checkpoint where we could attempt to access a file that was in the process of being dropped in the background.
refs SERVER-18014
- Fix a segfault during eviction where we could attempt to evict a page from a tree that was in the process of being dropped in the background.
refs SERVER-18460
- Fix a bug where WiredTiger could segfault in a workload with lots of cache pressure.
refs WT-1937
- Fix a performance issue with WT_SESSION::compact, where it would spend a long time compacting tables that had no space to reclaim.
refs WT-1953
- Fix a bug where accessing an overflow item could return WT_NOTFOUND incorrectly. The issue was related to an invalid transaction visibility check.
refs WT-1745
- Improve performance and avoid changing files on startup if no recovery is required by avoiding the creation of unnecessary checkpoints and log records for files that haven't changed.
refs WT-1936
- Improve how we handle create of a table, if a file with the same name already exists (possibly from an earlier failed create).
refs 1974
- Fix compiler warnings for LZ4 implementation on Windows.
refs 2006
- Fix a bug in the WiredTiger command line utility where it could create a base configuration file for an existing database, if there had been a crash while creating the database.
refs WT-1943
- Fix a build problem where recent versions of RedHat would fail to detect posix_memalign presence correctly.
refs WT-1951
- Fix several problems with how we create, recover and backup databases. Related to order of creation and differences between Windows and POSIX file system semantics.
refs 1993
- Fix a bug where we could flush the log file more often than required if using auto-commit transactions.
refs WT-1949
- Fix a performance problem in LSM, where trees created with an initial bulk load could choose poor merges.
refs WT-1947
- Improve how we decide whether to deepen a tree during an internal split operation. Append workloads could create trees that were excessively deep.
- Fix a bug in LSM which could lead to a hang on connection close.
refs WT-1935
- Fix a bug in the internal random number generator, where concurrent calls could lead to invalid sequences. Never seen in the wild.

The WiredTiger 2.6.0 release contains new features, minor API changes and many bug fixes.

New features:
- Add support for "at rest" encryption of WiredTiger databases via a new encryption API.
refs WT-1822
- Add support for bulk load in LSM trees (previously the bulk configuration for cursor create was ignored by LSM trees).
refs SERVER-18321, WT-1922
- Add enhanced compression support for LZ4. Change is not compatible with tables created with LZ4 compression using earlier versions of WiredTiger. See the upgrading documentation for more information.

API and behavior changes:
- Enhance performance of WT_SESSION::drop with force enabled (mark the table as dead and discard it in the background without writing any content to disk, rather than flushing content from the cache).
refs WT-1894
- Add an API to validate configuration strings.
refs WT-1739
- Disallow the cache_resident flag on LSM trees.
refs WT-1905
- Enhance the controls for how aggressively idle handles are closed, update the wiredtiger_open file_manager configuration options to expose that control.
refs SERVER-17907, WT-1856

Significant bug fixes and performance enhancements:
- Fix bugs in checkpoint: committing the checkpoint transaction before it was safe, failure to sync the metadata file before updating the turtle file.
refs SERVER-18316
- Don't attempt to validate configuration settings for extensions (collators, compressors, encryptors, extractors). The extension may be valid but not yet loaded so it is not possible to validate.
- Check the magic and version numbers in log files when first opening them.
- Fix a bug with cache leaf size accounting for statistics.
refs WT-1885, WT-1919
- Fix a bug where checkpoint could skip a page that was rewritten in memory.
refs WT-1946
- Fix a bug in WT_CURSOR::search_near with a random cursor where deleted records could be returned.
refs WT-1921, WT-1944
- Fix a bug in handle close where we could fail to clear the open handle flag.
refs WT-1915
- Fix a bug in in-memory splits, don't allow any other eviction of a page if the page is being split.
refs WT-1916, WT-1917
- Fix a bug in btree open where failure while opening a checkpoint may not clean up completely.
refs WT-1598
- Fix a performance problem where long running transactions could continually scan long update chains looking for obsolete entries, when there is no chance of finding any.
refs WT-1913
- Fix a performance problem where many threads accessing a single page could prevent WiredTiger from evicting or splitting the page.
refs WT-1912
- Fix a source of deadlock bugs by restructuring how we lock handles. Clean up the definition of when it is safe to acquire different locks.
refs WT-1598
- Fix a bug where we could evict pages from a file marked as cache resident, add optimizations for cache resident files.
refs SERVER-18192
- Fix a deadlock related to LSM (cases where closing a file with an existing checkpoint could self deadlock).
refs WT-716
- Only split large in-memory pages if there is a need to keep them in cache (otherwise, it is better to reconcile and evict them immediately).
refs WT-1890, WT-1896
- Fix a race on shutdown where the eviction server could still have been accessing files while they were being closed.
refs WT-1893
- Fix a case where we could run recovery unnecessarily if there were only non-data changing log records since the last checkpoint.
refs WT-1892
- Update API documentation to explain internal session handle usage. This allows users to do specific calculations based on the session_max setting.
- Fix a bug in LSM where updates with overwrite could be skipped incorrectly.
refs BF-829
- Fix a cursor bug where searching and traversing in different combinations could lead to data buffers being freed before they should be.
refs WT-1887
- Fix a bug when closing bulk cursors, where a file could be left open.
- Fix a bug on Windows related to truncating files (SetEndofFile does not ignore truncation requests like POSIX fallocate).
- Fix a bug in file truncate, where we could truncate to the wrong place if there was a race extending the file at the same time.
refs WT-1871
- Fix a bug in reconciliation where page boundary structures could reference freed memory.
refs WT-1852
- Change the WT_CURSOR::search and WT_CURSOR::search_near API to first check any currently pinned page before starting a regular descending search.
- Fix a bug in recovery where an error was returned if a transaction spanned an entire log file.
- Fix a bug where recovery could be unnecessarily run during startup.
- Enhance the Python cursor API to allow setting keys and values using array notation.
- Fix a bug where creating a column store with large gaps in the key range would lead to poor performance.
refs WT-1807.
- Improve the performance of the core btree search routine by using low-level x86 vectorized search instructions.