This release adds a bunch of new functionality (e.g., additional subcommands), but also changes some things that were not used by a lot of people in backward-incompatible ways.
:warning: API BREAKING CHANGES :warning:
- `BatchingBolt` now uses tick tuples instead of a separate timer thread. This is an API-breaking change, as `BatchingBolt.secs_between_batches` is now `BatchingBolt.ticks_between_batches`. You also will need to make sure you run your Storm topology with `topology.tick.tuple.freq.secs` set to how frequently you want the ticks to occur. [Read the docs](http://streamparse.readthedocs.org/en/master/api.html#streamparse.storm.bolt.BatchingBolt) for more details. (125, 137)
- streamparse fabric and invoke tasks have been moved to `sparse` sub-commands:
- `fab remove_logs` :arrow_right: `sparse remove_logs`
- `fab tail_logs` :arrow_right: `sparse tail`
- `fab activate_env` is no longer necessary, as all commands that need the fabric environment modified do this automatically.
- `fab create_or_update_virtualenvs` :arrow_right: `sparse update_virtualenv` (note the case change, since this only every worked on a single virtualenv at a time)
- `inv jar_for_deploy` :arrow_right: `sparse jar`
- `inv list_topologies` :arrow_right: `sparse list`
- `inv kill_topology` :arrow_right: `sparse kill`
- `inv run_local_topology` :arrow_right: `sparse run`
- `inv submit_topology` :arrow_right: `sparse submit`
- `inv tail_topology` :arrow_right: `sparse tail`
- `inv visualize_topology` :arrow_right: `sparse visualize`
- `inv prepare_topology` has been removed because the commands that relied on it (`sparse run`, `sparse submit`, and `sparse jar`) all call `streamparse.util.prepare_topology` automatically.
- The `streamparse.ext` package has been removed and so have the `streamparse.ext.fabric` and `streamparse.ext.invoke` modules.
- `streamparse.ext.util` :arrow_right: `streamparse.util`
- Users should no longer do `from streamparse.ext.fabric import *` and `from streamparse.ext.invoke import *` in their projects' `fabfile.py` and `tasks.py` files. `pre_submit` and `post_submit` hooks will be executed automatically even without this.
Major enhancements
- `sparse run` now runs indefinitely by default (122)
- Added [`Bolt.process_tick(tup)` method](http://streamparse.readthedocs.org/en/master/api.html#streamparse.storm.bolt.Bolt.process_tick) for processing tick tuples (116, 124)
- Added `sparse worker_uptime` and `sparse stats` commands for getting information about running Storm topologies and their workers. (17, 52)
- `--ackers` and `--workers` can now be specified as separate arguments to `sparse submit` and `sparse run`, instead of just using `--par`. (74, 97)
- `Bolt.emit_many()` is now deprecated and will be removed in streamparse 3.0. Please just call `Bolt.emit()` repeatedly instead. (66)
- Added lots of [documentation](https://streamparse.readthedocs.org) about how topologies work and how to get started with streamparse. (#26, 103)
- Added conda recipe template for building a streamparse conda package. (105)
- SSH tunnels are no longer required for `kill`, `list`, and `submit` commands (96, 98, 112).
- `env.use_ssh_config` is `True` by default now (54)
- Can now deploy/build simple JARs in addition to Uber-JARs. This speeds up `sparse submit` for pure Python projects. (106)
- Added `sparse jar`, `sparse remove_logs`, and `sparse update_virtualenv` commands to replace old Fabric and Invoke tasks.
Minor enhancements
- Removed dependency on `docopt` and switched to using `argparse` for command-line arguments. Now sub-commands all have their own detailed `--help` switches (e.g., `sparse run --help`) and `sparse --help` will list all of the available commands with a brief description of what they do. (115, 152)
- Added first pieces of support for a Python DSL for defining topologies (84) as part of a grander vision to move away from Clojure (136). Please note that this cannot actually be used yet, because the utility to take the Python DSL and then generate something Storm understands out of it has not been written yet.
- Overhauled unit tests to separate simplify IPC testing (41, 47).
- Added documentation on using an unofficial version of Storm (142)
- Added support for Tox (128)
- Updated spouts and bolts to allow Python tuples to be emitted. (119)
- Switched to using Travis Docker containers for building (90)
- Made update of virtualenvs optional by seeing if requirements.txt exists (60)
- Created a new `storm` subpackage, which will be split off into its own package (pystorm) for version 3.0 of streamparse. This contains all of the IPC/Multi-Lang related code. In the future streamparse will just be a collection of utilities for managing Storm topologies/clusters.
- Moved a lot of code from the `Spout` and `Bolt` classes into the `Component` parent class to cut down on code duplication.
Bugfixes
- Fixed multithreaded emitting (101, 133)
- `sparse` commands that use `lein` underneath now display output from `lein` immediately. (109)
- Fixed typo in config name to get maxbytes (110)
- We now reset `Botl.current_tups` even when receiving a heartbeat (107)
- `Spout.emit_many()` works again (144)
- `sparse tail` tails all machines now. (104)
Contributors for this release (by number of commits)
- Dan Blanchard (dan-blanchard)
- Keith Bourgoin (kbourgoin)
- Viktor Shlapakov (vshlapakov)
- Curtis Vogt (omus)
- Andrew Montalenti (amontalenti)
- Tim Hopper (tdhopper)
- Daniel Hodges (hodgesds)
- Arturo Filastò (hellais)
- Wieland Hoffmann (mineo)
- Aiyesha Ma (Aiyesha)
- Cody Wilbourn (codywilbourn)
Thanks to all our contributors!