Streamparse

Latest version: v5.0.1

Safety actively analyzes 688554 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 7 of 8

2.0.2

This release fixes an issue where tick tuples were not being acked in the new `BatchingBolt` implementation (e38a024). It also updates the documentation for `BatchingBolt` to indicate that you can enable tick tuples on a per-bolt basis in your topology file by adding `:conf {"topology.tick.tuple.freq.secs", 1}` to your `python-bolt-spec` arguments (d1c405a).

2.0.1

This bugfix release fixes an issue where reading non-ASCII messages on Python 2.7 would cause a `UnicodeDecodeError` (154). Thanks to daTokenizer for reporting this!

2.0.0

This release adds a bunch of new functionality (e.g., additional subcommands), but also changes some things that were not used by a lot of people in backward-incompatible ways.

:warning: API BREAKING CHANGES :warning:
- `BatchingBolt` now uses tick tuples instead of a separate timer thread. This is an API-breaking change, as `BatchingBolt.secs_between_batches` is now `BatchingBolt.ticks_between_batches`. You also will need to make sure you run your Storm topology with `topology.tick.tuple.freq.secs` set to how frequently you want the ticks to occur. [Read the docs](http://streamparse.readthedocs.org/en/master/api.html#streamparse.storm.bolt.BatchingBolt) for more details. (125, 137)
- streamparse fabric and invoke tasks have been moved to `sparse` sub-commands:
- `fab remove_logs` :arrow_right: `sparse remove_logs`
- `fab tail_logs` :arrow_right: `sparse tail`
- `fab activate_env` is no longer necessary, as all commands that need the fabric environment modified do this automatically.
- `fab create_or_update_virtualenvs` :arrow_right: `sparse update_virtualenv` (note the case change, since this only every worked on a single virtualenv at a time)
- `inv jar_for_deploy` :arrow_right: `sparse jar`
- `inv list_topologies` :arrow_right: `sparse list`
- `inv kill_topology` :arrow_right: `sparse kill`
- `inv run_local_topology` :arrow_right: `sparse run`
- `inv submit_topology` :arrow_right: `sparse submit`
- `inv tail_topology` :arrow_right: `sparse tail`
- `inv visualize_topology` :arrow_right: `sparse visualize`
- `inv prepare_topology` has been removed because the commands that relied on it (`sparse run`, `sparse submit`, and `sparse jar`) all call `streamparse.util.prepare_topology` automatically.
- The `streamparse.ext` package has been removed and so have the `streamparse.ext.fabric` and `streamparse.ext.invoke` modules.
- `streamparse.ext.util` :arrow_right: `streamparse.util`
- Users should no longer do `from streamparse.ext.fabric import *` and `from streamparse.ext.invoke import *` in their projects' `fabfile.py` and `tasks.py` files. `pre_submit` and `post_submit` hooks will be executed automatically even without this.

Major enhancements
- `sparse run` now runs indefinitely by default (122)
- Added [`Bolt.process_tick(tup)` method](http://streamparse.readthedocs.org/en/master/api.html#streamparse.storm.bolt.Bolt.process_tick) for processing tick tuples (116, 124)
- Added `sparse worker_uptime` and `sparse stats` commands for getting information about running Storm topologies and their workers. (17, 52)
- `--ackers` and `--workers` can now be specified as separate arguments to `sparse submit` and `sparse run`, instead of just using `--par`. (74, 97)
- `Bolt.emit_many()` is now deprecated and will be removed in streamparse 3.0. Please just call `Bolt.emit()` repeatedly instead. (66)
- Added lots of [documentation](https://streamparse.readthedocs.org) about how topologies work and how to get started with streamparse. (#26, 103)
- Added conda recipe template for building a streamparse conda package. (105)
- SSH tunnels are no longer required for `kill`, `list`, and `submit` commands (96, 98, 112).
- `env.use_ssh_config` is `True` by default now (54)
- Can now deploy/build simple JARs in addition to Uber-JARs. This speeds up `sparse submit` for pure Python projects. (106)
- Added `sparse jar`, `sparse remove_logs`, and `sparse update_virtualenv` commands to replace old Fabric and Invoke tasks.

Minor enhancements
- Removed dependency on `docopt` and switched to using `argparse` for command-line arguments. Now sub-commands all have their own detailed `--help` switches (e.g., `sparse run --help`) and `sparse --help` will list all of the available commands with a brief description of what they do. (115, 152)
- Added first pieces of support for a Python DSL for defining topologies (84) as part of a grander vision to move away from Clojure (136). Please note that this cannot actually be used yet, because the utility to take the Python DSL and then generate something Storm understands out of it has not been written yet.
- Overhauled unit tests to separate simplify IPC testing (41, 47).
- Added documentation on using an unofficial version of Storm (142)
- Added support for Tox (128)
- Updated spouts and bolts to allow Python tuples to be emitted. (119)
- Switched to using Travis Docker containers for building (90)
- Made update of virtualenvs optional by seeing if requirements.txt exists (60)
- Created a new `storm` subpackage, which will be split off into its own package (pystorm) for version 3.0 of streamparse. This contains all of the IPC/Multi-Lang related code. In the future streamparse will just be a collection of utilities for managing Storm topologies/clusters.
- Moved a lot of code from the `Spout` and `Bolt` classes into the `Component` parent class to cut down on code duplication.

Bugfixes
- Fixed multithreaded emitting (101, 133)
- `sparse` commands that use `lein` underneath now display output from `lein` immediately. (109)
- Fixed typo in config name to get maxbytes (110)
- We now reset `Botl.current_tups` even when receiving a heartbeat (107)
- `Spout.emit_many()` works again (144)
- `sparse tail` tails all machines now. (104)

Contributors for this release (by number of commits)
- Dan Blanchard (dan-blanchard)
- Keith Bourgoin (kbourgoin)
- Viktor Shlapakov (vshlapakov)
- Curtis Vogt (omus)
- Andrew Montalenti (amontalenti)
- Tim Hopper (tdhopper)
- Daniel Hodges (hodgesds)
- Arturo Filastò (hellais)
- Wieland Hoffmann (mineo)
- Aiyesha Ma (Aiyesha)
- Cody Wilbourn (codywilbourn)

Thanks to all our contributors!

1.1.0

This release adds support for Storm 0.9.3 in addition to a number of bug fixes.
New and updated examples available.
- Adds: [Support for Storm 0.9.3 heartbeats (82)](https://github.com/Parsely/streamparse/issues/82)
- Adds: [`StormHandler` class for logging to Storm](https://github.com/Parsely/streamparse/pull/58)
- Adds: `--wait` timeout to `sparse kill` and `spare submit`
- Adds: "kafka-jvm" example -- mixed language topology (JVM/clojure + Python) with JVM-based Kafka Spout
- Adds: "wordcount-on-redis" example
- Updates: wordcount example
- Fixes: [64: `sparse tail fails when logs are missing](https://github.com/Parsely/streamparse/issues/64)
- Fixes: ["flush" method to LogStream](https://github.com/Parsely/streamparse/pull/50)
- Fixes: [100: SSH tunnels are not always closed](https://github.com/Parsely/streamparse/issues/100)
- Fixes: -o option string issues
- Documentation updates

1.0.1

Fixes bug in 57 regarding executing local tasks (like sparse run) with
pty=True.

1.0.0

This is a major release that introduces several potentially breaking API
changes, hence the major version advancement to 1.0.0. Please read the changes
below as well as our [migration wiki](https://github.com/Parsely/streamparse/wiki/0.0.13-to-1.0.0-Migration-Guide)
guide before upgrading.

**streamparse runner**
- Added a new runner for topology components and as a result, a new way to
define topologies (extension to the existing Clojure DSL). More info can be
found in the docs and in the [1.0.0 migration guide](https://github.com/Parsely/streamparse/wiki/0.0.13-to-1.0.0-Migration-Guide).
- Quickstart projects have been updated to now have nested directories
"src/bolts" and "src/spouts".

**Component API updates**
- Added `auto_anchor`, `auto_ack` and `auto_fail` flags to base Bolt class. See
docs for detailed descriptions of these flags and migration page for info on
how to safely upgrade your bolts.
- `BasicBolt` is now deprecated.
- Class var `BatchingBolt.SECS_BETWEEN_BATCHES` renamed to
`secs_between_batches` since this isn't a constant, just a setting.
- `Spout.emit` and `Bolt.emit` now returns a list of task IDs a tuple was
emitted to unless `need_task_ids` kwarg is set to `False`.
- `Spout.emit_many` and `Bolt.emit_many` now return a two-dimensional list of
the task IDs each emit tuple was sent to unless `need_task_ids` kwarg is set
to `False`.
- `BatchingBolt` does not return task IDs due to concurrency issues that will
be addressed in a future release.
- Spouts and bolts now have the following instance variables which users are
free to use. These are initialized before the call to `initialize()`:
- `_topology_name` - name of the topology when submitted to Storm.
- `_task_id` - task ID of the current component in the topology.
- `_component_name` - the name of the current component as defined in the
Clojure definition (e.g. "my-bolt").
- `_debug` - the `topology.debug` setting (configured using the sparse
`--debug` flag).
- `_storm_conf` - the entire config dict recieved on initial handshake.
- `_context` - the entire context dict receieved on initial handshake.
- `BatchingBolt` threads (main and _batcher) now have more descriptive names.

**IPC**
- A lot of code cleanup around how we read from and write to stdin and stdout.
Much more stable here but no breaking changes for users since they should
never interface with these methods directly.

**Logging**
- Full support for Python logging has now been added. Lots of info in the docs
on this and the migration guide. Logging config settings added to new
quickstart projects.
- print statements are now properly sent to the component's log file - feel
free to add print statements for handy debugging or better yet, create a
logger for nicely formatted messages.
- Full support for Storm log levels once STORM-414 is merged in.

**Administration**
- Added pre and post submit hooks in fabric and invoke for topologies to
enable users to run arbitrary code (e.g. send IRC message) after topologies
are submitted (info in docs).\
- `sparse tail` command now requires `-n <topology>` argument as it will tail
only the logs for a specific topology and environment.
- Added `remove_logs` fab task which users can optionally hook up to a
`pre_submit` hook to clear out Python logs.

**Testing**
- Better test support added for all our components, more improvements to come
here.

Page 7 of 8

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.