Logstash

general
- Fix path to the built-in elasticsearch index template

general
- oops: The --help flag now reports help again, instead of barfing an "I need
help" exception (LOGSTASH-1436, LOGSTASH-1392; Jordan Sissel)
- Resolved encoding errors caused by environmental configurations, such as
'InvalidByteSequenceError ... on US-ASCII' (LOGSTASH-1595, 842;
Jordan Sissel)
- Fix bug causing "no such file to load -- base64" (LOGSTASH-1310,
LOGSTASH-1519, LOGSTASH-1325, LOGSTASH-1522, 834; Jordan Sissel)
- Elasticsearch version 0.90.7
- Bug fixes galore!

inputs
- new: collectd: receive metrics from collectd's network protocol
(785, Aaron Mildenstein)
- bugfix: gelf: handle chunked gelf message properly (718, Thomas De Smedt)
- bugfix: s3: fix bug in region endpoint setting (740, Andrea Ascari)
- bugfix: pipe: restart the command when it finishes (754, Jonathan Van
Eenwyk)
- bugfix: redis: if redis fails, reconnect. (767, LOGSTASH-1475; Jordan Sissel)
- feature: imap: add 'content_type' setting for multipart messages and
choosing the part that becomes the event message. (784, Brad Fritz)
- bugfix: zeromq: don't override the 'host' field if the event already
has one. (Jordan Sissel)
- bugfix: ganglia: fix regressions; plugin should work again (LOGSTASH-1655,
818; Jordan Sissel)
- bugfix: Fix missing library in sqs input (775, LOGSTASH-1294; Toby
Collier)

filters
- new: unique: removes duplicate values from a given field in an event.
(676, Adam Tucker)
- new: elapsed: time duration between two tagged events. (713, Andrea Forni)
- new: i18n: currently supports 'transliterate' which does best-effort
conversion of text to "plain" letters. Like 'ó' to 'o'. (671,
Juarez Bochi)
- bugfix: restore filter flushing thread (LOGSTASH-1284, 689; Bernd Ahlers)
- new: elasticsearch: query elasticsearch and update your event based on the
results. (707, Jonathan Van Eenwyk)
- new: sumnumbers: finds all numbers in a message and sums them (752, Avleen
Vig)
- feature: geoip: new field 'location' is GeoJSON derived from the lon/lat
coordinates for use with elasticsearch, kibana, and anything else that
understands GeoJSON (763, Aaron Mildenstein)
- new: punct: Removes all text except punctuation and stores it in another
field. Useful for as a means for fingerprinting events. (813, Guixing Bai)
- feature: metrics: Make percentiles configurable. Also make rates (1, 5,
15-minute) optional. (817, Juarez Bochi)

codecs
- new: compressed_spooler: batches events and sends/receives them in
compressed form. Useful over high latency links or with transports
with higher-than-desired transmission costs. (Avleen Vig)
- new: fluent: receive data serialized using the Fluent::Logger for easier
migration away from fluentd or for folks who simply like the logger
library (759, Jordan Sissel)
- new: edn: encode and decode the EDN serialization format. Commonly used
in Clojure. For more details, see: https://github.com/edn-format/edn
(778, Lee Hinman)
- bugfix: oldlogstashjson: Fix encoding to work correctly. (788, 795;
Brad Fritz)
- bugfix: oldlogstashjson: Fallback to plain text on invalid JSON
(LOGSTASH-1534, 850; Jordan Sissel)

outputs
- feature: elasticsearch and elasticsearch_http now will apply a default
index mapping template (included) which has the settings recommended by
Elasticsearch for Logstash specifically.
Configuration options allow disabling this feature and providing a path
to your own template. (826, 839; Aaron Mildenstein)
- feature: elasticsearch_http: optional 'user' and 'password' settings to
make use of http authentication (LOGSTASH-902, 684; Ian Neubert)
- new: google_bigquery: upload logs to bigquery for analysis later (Rodrigo
De Castro)
- bugfix: datadog_metrics: fix validation bug (789, Ian Paredes)
- feature: elasticsearch: new 'transport' setting letting you tell logstash
to act as a cluster node (default, prior behavior) or as a 'transport
client'. With the new 'transport' mode, your firewall rules may be simpler
(unicast, one direction) and transport clients do not show up in your
cluster node list. (LOGSTASH-102, 841; Jordan Sissel)
- feature: elasticsearch: new 'bind_port setting for 'node' protocol which
lets you chose the local port to bind on (841, Jordan Sissel)
- bugfix: Fix missing library in sqs input (775, LOGSTASH-1294; Toby
Collier)

general
- new 'worker' setting for outputs. This helps improve throughput on
request-oriented outputs such as redis, rabbitmq, elasticsearch,
elasticsearch_http, etc. Workers run in separate threads each handling
events as they come in. This allows you to linearly scale up outputs across
cores or as blocking-io permits.
- grok performance is up 600%
- lots of bug fixes
- bugfixes to conditionals (682, Matt Dainty)
- rabbitmq now replaces the old deprecated amqp plugins. amqp plugins are
removed.
- inputs will now do their best to handle text which is encoded differently
than the charset you have specified (LOGSTASH-1443, Jordan Sissel)

inputs
- bugfix: udp: respects teardown requests via SIGINT, etc (LOGSTASH-1290,
Jordan Sissel)
- bugfix: rabbitmq: disable automatic connection recovery (LOGSTASH-1350,
641, 642; Michael Klishin)
- bugfix: twitter: works again (640, Bernd Ahlers)
- compatibility: Restored the old 'format' setting behavior. It is still
deprecated, but was accidentally removed in 1.2.0. It will be removed
later, but is restored as part of our backwards-compat promise (Jordan
Sissel)
- bugfix: s3: fix LOGSTASH-1321 and LOGSTASH-1319 (Richard Pijnenburg)
- bugfix: log4j: fix typo (Jordan Sissel)
- bugfix: rabbitmq: disable automatic connection recover because logstash
will handle it (LOGSTASH-1350, Michael Klishin)
- bugfix: heroku: works again (LOGSTASH-1347, 643; Bernd Ahlers)
- bugfix: tcp: improve detection of closed connections to reduce lost events
(Jordan Sissel)
- bugfix: elasticsearch: now works correctly (670, Richard Pijnenburg)
- improvement: elasticsearch: make size and scroll time configurable (670,
Richard Pijnenburg)
- improvement: elasticsearch: tunable search type (670, Richard Pijnenburg)
- compatibility: restore 'format' setting which was accidentally removed in
1.2.0. This feature is still deprecated, but it has been restored
temporarily as part of our backwards compatibility promise. (706, Jordan
Sissel)
- bugfix: syslog: fix socket leakage (704, Bernd Ahlers)
- improvement: all aws-related plugins: Add proxy_uri setting (714, Malthe
Borch)
- bugfix: unix: fix variable name crash (720, Nikolay Bryskin)

codecs
- new: graphite: parse graphite formated events (Nick Ethier)
- new: json_lines: parse streams that are lines of json objects (731, Nick
Ethier)
- bugfix: multiline: time is now correctly in UTC. (Jordan Sissel)
- bugfix: oldlogstashjson: improved conversion of old logstash json to the
new schema (654, Jordan Sissel)
- bugfix: oldlogstashjson: fix typo breaking encoding (665, Tom Howe)
- bugfix: json: now assumes json delimited by newline character
(LOGSTASH-1332, 710; Nick Ethier)
- improvements: netflow: new target and versions settings (686, Matt Dainty)

filters
- performance: grok: 6.3x performance improvement (681, Jordan Sissel)
- bugfix: geoip: empty values (nil, empty string) are not put into the event
anymore. (Jordan Sissel)
- bugfix: geoip: allow using Maxmind's ASN database (LOGSTASH-1394, 694;
Bernd Ahlers)
- improvement: kv: target will now overwrite any existing fields, including
the source (Jordan Sissel).
- improvement: Kv: 'prefix' setting now respects sprintf (LOGSTASH-913,
647; Richard Pijnenburg)
- checksum: sha128 was not a valid digest, removed from list
- feature: metrics: added clear_interval and flush_interval parameters for
setting flush rates and when to clear metrics (545)
- new: collate: group events by time and/or count into a single event. (609,
Neway Liu)
- feature: date: now supports a 'target' field for writing the timestamp into
a field other than timestamp. (625, Jonathan Van Eenwyk)
- bugfix: riemann: event tagging works again (631, Marc Fournier)
- improvement: grok: IPV6 pattern (623, Matt Dainty)
- improvement: metrics: add clear_interval and flush_interval settings (545,
Juarez Bochi)
- improvement: useragent: include operating system details (656, Philip
Kubat)
- improvement: csv: new quote_char setting (725, Alex Markham)

outputs
- feature: all outputs have a 'worker' setting now that allows you to
perform more work at the same time. This is useful for plugins like
elasticsearch_http, redis, etc, which can bottleneck on waiting for
requests to complete but would otherwise be happy processing more
simultaneous requests. (708, Jordan Sissel)
- bugfix: elasticsearch: requests are now synchronous. This avoid overloading
the client and server with unlimited in-flight requests. (688, Jordan
Sissel)
- bugfix: elasticsearch_http: fix bug when sending multibyte utf-8 events
(LOGSTASH-1328, 678, 679, 695; Steve Merrill, Christian Winther,
NickEthier, Jordan Sissel)
- performance: elasticsearch_http: http client library uses TCP_NODELAY now
which dramatically improves performance. (696, Jordan Sissel)
- feature: elasticsearch_http now supports a 'replication' setting to
allow you to choose how you wait for the response. THe default is 'sync'
which waits for all replica shards to be written. If you set it to 'async'
then all index requests will respond once only the primary shards have been
written and the replica shards will be written later. This can improve
throughput. (700, Nick Ethier, Jordan Sissel)
- bugfix: elasticsearch: the default port range is now 9300-9305; the older
range up to 9400 was unnecessary and could cause problems for the
elasticsearch cluster in some cases.
- improvement: aws-based outputs (e.g. cloudwatch) now support proxy uri.
- bugfix: rabbitmq: disable automatic connection recovery (LOGSTASH-1350)
(642)
- bugfix: riemann: fixed tagging of riemann events (631)
- bugfix: s3: fix LOGSTASH-1321 and LOGSTASH-1319 (636, 645; Richard
Pijnenburg)
- bugfix: mongodb: Fix mongodb auth (LOGSTASH-1371, 659; bitsofinfo)
- bugfix: datadog: Fix time conversion (LOGSTASH-1427, 690; Bernd Ahlers)
- bugfix: statsd: Permit plain floating point values correctly in the
config. Example: sample_rate => 0.5 (LOGSTASH-1441, 705; Jordan Sissel)
- bugfix: syslog: Fix timestamp date formation. 'timestamp' setting is now
deprecated and the format of the time depends on your rfc selection.
(LOGSTASH-1423, 692, 739; Jordan Sissel, Bernd Ahlers)

patterns
- improvement: added IPV6 support to IP pattern (623)

general
- This is primarily a bugfix/stability release based on feedback from 1.2.0
- web: kibana's default dashboard now works with the new logstash 1.2 schema.
- docs: updated the tutorials to work in logstash 1.2.x
- agent: Restored the --configtest flag (unintentionally removed from 1.2.0)
- deprecation: Using deprecated plugin settings can now advise you on a
corrective path to take. One example is the 'type' setting on filters and
outputs will now advise you to use conditionals and give an example.
- conditionals: The "not in" operator is now supported.

inputs
- bugfix: pipe: reopen the pipe and retry on any error. (619, Jonathan Van
Eenwyk)
- bugfix: syslog: 'message' field no longer appears as an array.
- bugfix: rabbitmq: can now bind the queue to the exchange (624, 628,
LOGSTASH-1300, patches by Jonathan Tron and Jonathan Van Eenwyk)

codecs
- compatibility: json: if data given is not valid as json will now be used as
the "message" of an event . This restores the older behavior when using
1.1.13's "format => json" feature on inputs. (LOGSTASH-1299)
- new: netflow: process netflow data (580, patches by Nikolay Bryskin and
Matt Dainty)

filters
- bugfix: multiline: the multiline filter returns! It was unintentionally
removed from the previous (1.2.0) release.
- bugfix: json_encode: fix a syntax error in the code. (LOGSTASH-1296)
- feature: kv: now captures duplicate field names as a list, so 'foo=bar
foo=baz' becomes the field 'foo' with value ['bar', 'baz'] (an array).
(622, patch by Matt Dainty)

outputs
- new: google_cloud_storage: archive logs to Google Cloud Storage (572,
Rodrigo De Castro)
- bugfix: fixed bug with 'tags' and 'exclude_tags' on outputs that would
crash if the event had no tags. (LOGSTASH-1286)

general
- The logstash json schema has changed. (LOGSTASH-675)
For prior logstash users, you will be impacted one of several ways:
* You should check your elasticsearch templates and update them accordingly.
* If you want to reindex old data from elasticsearch with the new schema,
you should be able to do this with the elasticsearch input. Just make
sure you set 'codec => oldlogstashjson' in your elasticsearch input.
- The old logstash web ui has been replaced by Kibana 3. Kibana is a far
superior search and analytics interface.
- New feature: conditionals! You can now make "if this, then ..." decisions
in your filters or outputs. See the docs here:
http://logstash.net/docs/latest/configurationconditionals
- A new syntax exists for referencing fields (LOGSTASH-1153). This replaces
the prior and undocumented syntax for field access (was 'foo.bar' and is
now '[foo][bar]'). Learn more about this here:
http://logstash.net/docs/latest/configurationfieldreferences
- A saner hash syntax in the logstash config is now supported. It uses the
perl/ruby hash-rocket syntax: { "key" => "value", ... } (LOGSTASH-728)
- ElasticSearch version 0.90.3 is included. (486, Gang Chen)
- The elasticsearch plugin now uses the bulk index api which should result
in lower cpu usage as well as higher performance than the previous
logstash version.
- Many deprecated features have been removed. If your config caused
deprecation warnings on startup in logstash v1.1.13, there is a good
chance that these deprecated settings are now absent.
- 'type' is no longer a required setting on inputs.
- New plugin type: codec. Used to implement decoding of events for inputs and
encoding of events for outputs. Codecs allow us to separate transport (like
tcp, redis, rabbitmq) from serialization (gzip text, json, msgpack, etc).
- Improved error messages that try to be helpful. If you see bad or confusing
error messages, it is a bug, so let us know! (Patch by Nick Ethier)
- The old 'plugin status' concept has been replaced by 'milestones'
(LOGSTASH-1137)
- SIGHUP should cause logstash to reopen it's logfile if you are using the
--log flag

inputs
- new: s3: reads files from s3 (537, patch by Mathieu Guillaume)
- feature: imap: now marks emails as read (542, Raffael Schmid)
- feature: imap: lets you delete read email (591, Jonathan Van Eenwyk)
- feature: rabbitmq: now well-supported again (patches by Michael Klishin)
- bugfix: gelf: work around gelf parser errors (476, patch by Chris McCoy)
- broken: the twitter input is disabled because the twitter stream v1 api is
no longer supported and I couldn't find a replacement library that works
under JRuby.
- new: sqlite input (484, patch by Evan Livingston)
- improvement: snmptrap: new 'yamlmibdir' setting for specifying an external
source for MIB definitions. (477, patch by Dick Davies)
- improvement: stomp: vhost support (490, patch by Matt Dainty)
- new: unix: unix socket input (496, patch by Nikolay Bryskin)
- new: wmi: for querying wmi (windows). (497, patch by Philip Seidel)
- improvement: sqs: new id_field and md5_field settings (LOGSTASH-1118, Louis
Zuckerman)

filters
- feature: grok: 'singles' now defaults to true.
- bugfix: grep: allow repeating a field in the hash config (LOGSTASH-919)
- feature: specify timezone in date filter (470, patch by Philippe Weber)
- feature: grok setting 'overwrite' now lets you overwrite fields instead
of appending to them.
- feature: the useragent filter now defaults to writing results to the top
level of the event instead of "ua"
- feature: grok now defaults 'singles' to true, meaning captured fields are
stored as single values in most cases instead of the old behavior of being
captured as an array of values.
- new: json_encoder filter (554, patch by Ralph Meijer)
- new: cipher: gives you many options for encrypting fields (493, patch by
saez0pub)
- feature: kv: new settings include_fields and exclude_fields. (patch by
Piavlo)
- feature: geoip: new 'target' setting for where to write geoip results.
(491, patch by Richard Pijnenburg)
- feature: dns: now accepts custom nameservers to query (495, patch by
Nikolay Bryskin)
- feature: dns: now accepts a timeout setting (507, patch by Jay Luker)
- bugfix: ruby: multiple ruby filter instances now work (501, patch by
Nikolay Bryskin)
- feature: uuid: new filter to add a uuid to each event (531, Tomas Doran)
- feature: useragent: added 'prefix' setting to prefix field names created
by this filter. (524, patch by Jay Luker)
- bugfix: mutate: strip works now (590, Jonathan Van Eenwyk)
- new: extractnumbers: extract all numbers from a message (579, patch by
Pablo Barrera)

outputs
- new: jira: create jira tickets from an event (536, patch by Martin Cleaver)
- feature: rabbitmq: now well-supported again (patches by Michael Klishin)
- improvement: stomp: vhost support (Patch by Matt Dainty)
- feature: elasticsearch: now uses the bulk index api and supports
a tunable bulk flushing size.
- feature: elasticsearch_http: will now flush when idle instead of always
waiting for a full buffer. This helps in slow-sender situations such
as testing by hand.
- feature: irc: add messages_per_second tunable (LOGSTASH-962)
- bugfix: email: restored initial really useful documentation
- improvement: emails: allow message, source, ... in match (LOGSTASH-826,
LOGSTASH-823)
- feature: email: can now set Reply-To (540, Tim Meighen)
- feature: mongodb: replica sets are supported (389, patch by Mathias Gug)
- new: s3: New plugin to write to amazon S3 (439, patch by Mattia Peterle)
- feature: statsd: now supports 'set' metrics (513, patch by David Warden)
- feature: sqs: now supports batching (522, patch by AaronTheApe)
- feature: ganglia: add slope and group settings (583, patch by divanikus)

general
- fixed bug in static file serving for logstash web (LOGSTASH-1067)

outputs
- feature: irc: add messages_per_second tunable (LOGSTASH-962)