Marcel

The core of marcel has been overhauled in this release. The old implementation was reyling
too much on copying pipelines by pickling. This was slow because pickling is slow.
It was also slow because the deep copying of namespaces and functions is very tricky
to get right, due to cycles, (e.g. the marcel namespace contains a pipeline which contains
a function, whose globals is the marcel namespace). This was exacerbated by what
I think was a bug in dill in which cycles like this were not handled correctly. In
any case, part of the trickiness here involved the cycles resulting in more and more
"expansion", in which a deep copy of x is made instead of sharing x (either through
my bugs or the possible dill bug). This flaky copying/sharing was also the source
of bugs when a function's globals were not quite in sync with the marcel namespace.

So all that is gone. Pipeline copying is greatly reduced, due to a simple change
in the Pipeline and Op data structures. There is still a Pipeline.copy() function,
but it does not rely on pickling. Pickling is only used when data needs to cross
process boundaries, (jobs, remote execution). Function globals are kept in sync with
the marcel namespace far more reliably now.

So there aren't any visible changes, except: 1) things should generally be running
faster due to far less use of pickling; and 2) subtle bugs due caused by the
older code, e.g. bug 116.

Lots of work on the saving and recalling of streams. Suppose you run this command, to
locate all files under your current directory, recursively.

shell script
ls -fr


This can take a while, and the output can be quite large, so you can put the
command in the background, as you would in bash: hit `ctrl-z` to suspend the
command, and then use the `bg`
command to have the command resume in the background.

But if you do this, then the output continues to go to the console. To fix this
problem, you can redirect the output to an environment variable:

shell script
ls -fr > allfiles


This puts the output of the `ls -fr` command in the environment variable `allfiles`. This
is not a file, it is an environment variable storing the contents of a stream. After you
put this command in the background, you can check on progress, e.g.

shell script
allfiles > tail -5


to see the last 5 `File`'s listed, or to count them:

shell script
allfiles | red count


If what you really want to do is to store the result in a file, you can do that too:

shell script
ls -fr | out -f /tmp/allfiles.txt


There are two differences. First, the environment variable is bound to your
marcel session, and disappears when you exit marcel. Second, the environment
variable stores actual `File` objects. If you store command output in a file,
then that file stores paths only, not `File` objects.

Overhaul handling of store/load variables. These are now handled by a new internal class, Reservoir, which keeps state in a file. Because the streams stored in a reservoir can be arbitrarily large, keeping them in memory was problematic, including slow pickling/unpickling, for execution in a job, or remote execution.

Bug fixes, no new features.

Tab completion wasn't working inside a nested pipeline, e.g. ls -dr ... | args [d: ls ...

Overhauled parser's tracking of context for use by tab completion.

Marcel now has an operator, `args`, that works like Linux's `xargs`.
For example, suppose you have a directory containing log files, and you want to
delete logs that are more than 100 days old. Locating
such files is easy:

shell script
ls -f | select (f: now() - f.mtime > days(100))


What you would like to do is to take each `File` in the resulting stream, and
delete it. The `args` operator takes items arriving on the input
stream, and makes them available to operators. So to do the file removal:

shell script
ls -f | select (f: now() - f.mtime > days(100)) | args [f: rm (f)]


- `ls ... | select ...` produces a stream of `File`s.
- These `File`s flow into the `args` input stream.
- `args` has a pipeline with parameter `f`. Each arriving `File` is bound to `f`, and
the pipeline is executed.
- `rm (f)` removes `File` `f`.

You can also pass all of the qualifying `File`s at once:

shell script
ls -f | select (f: now() - f.mtime > days(100)) | args --all [files: rm (quote_files(files))]


Now, all of the qualifying `File`s are bound to the pipelines `files` parameter.
Because the args pipeline relies on the host OSs `rm` command, carefully quoting
filenames is necessary. `quote_files` takes `files`, a `list` of `File`s, and returns
a string containing all the `File`s names, separated by spaces, and each quoted, (e.g.
to properly handle a filename such as `this filename has four spaces`.)

Actually, you can do the same cleanup by relying on the fact that `File`s implement
the `pathlib.Path` interface, which has an `unlink` method. So this works too:

shell script
ls -f | select (f: now() - f.mtime > days(100)) | map (f: f.unlink())