Jsonnet

Sorry for the churn. Since the last release a buffer overrun was identified, so this release fixes it. Also included are some fixes to jsonnet fmt behavior and a fix for [e::] when e was larger than one token.

New features:
- error will implicitly convert to string
- Verbatim string literals, e.g. " \ \" " can now be given as " \ "" " (note the escaping of "
- std.md5("foo") is added.

Bug fixes:

Fixed the binding of self / super in e after the expansion of [e] +:
Several improvements to the output of jsonnet fmt when -n is used.
Avoid an infinite loop of memory consumption at a ||| syntax error.

Fix a segfault in the GC when using native functions.

Some minor bug fixes.

Imported values are now cached after execution, resulting in a ~2x performance improvement (measured on some real Jsonnet code).

This release renames all "library" jsonnet files to .libsonnet as proposed some months ago. This is just a convention, it does not affect evaluation of Jsonnet configs. As with all conventions, users can adopt it... or ignore it :)

This release also has some new language features:

Named params, default arguments

As in Python, you can provide default arguments to functions, e.g.


local add(x, y=3) = x + y;
add(6)


and also bind arguments to parameters by name instead of by position:


add(x=6)


The syntax and semantics matches Python but without _args and *_kwargs.

Top-level arguments

This is a more principled way of parameterizing an entire config. Previously the only way to do that was with `std.extVar`, but this created a global variable of sorts. Code anywhere in the config could use it and it in large configs this can become a maintenance hurdle. `std.extVar` still exists but there is a new way where the parameter is scoped only to the main or root file of the config, and has to be explicitly threaded to imported files:

Previously a Jsonnet file evaluating to a function would be rejected:


$ jsonnet -e 'function() 42'
RUNTIME ERROR: Couldn't manifest function in JSON output.


Now, such a function is called with no arguments to yield the value that is manifested as JSON. To add arguments, use the new commandline parameters.


Available options for specifying values of 'top-level arguments':
Provide the value as a string:
-A / --tla-str <var>[=<val>] If <val> is omitted, get from environment var <var>
--tla-str-file <var>=<file> Read the string from the file
Provide a value as Jsonnet code:
--tla-code <var>[=<code>] If <code> is omitted, get from environment var <var>
--tla-code-file <var>=<file> Read the code from the file



$ jsonnet -e 'function(a, b, c=3) a + b + c' --tla-code a=1 --tla-code b=2
6


Note that TLAs provide default arguments for the entire config, something that was not possible (but often requested) with std.extVar().

Native functions

Now you can, via the C API (or wrappers, e.g. Python) add your own native functions to Jsonnet. The purpose of these functions is to expose tricky functionality that you wouldn't want to implement in Jsonnet. E.g. compression, or encryption. Note that these functions must be pure. They must have no side-effects and must give the same return value for the same input. Since Jsonnet is a lazy language, native functions may be called at unusual times, in an unusual order, more times than expected, or not at all. If the functionality being exposed is not naturally pure (e.g. random secret generation), please wrap it in a cache so that subsequent calls with the same params give the same result.

Currently native extensions can have an arbitrary number of parameters but each has to be a primitive (i.e. not an object or an array). One option for now is to wrap the native function in Jsonnet code that calls std.toString() on the value, then parse the JSON on the host language side.

Native extensions can however return arbitrary JSON objects.

This release merely fixes some problems with the Bazel build files in the previous release. As such, there is no need for a corresponding Python release.

Firstly, Jsonnet users should be aware of a tiny non-backwards-compatible change: It is no longer possible to put spaces or comments between the characters of each of the following operators :: +: ::: and +::: . The fact this was ever possible was an accident of implementation. In fact, we expect no-one was actually writing code like { x: /*foo */: 3 }.

With that out of the way, let's talk about the many new features this release:

The major announcement is the Jsonnet reformatter, which is in many ways like the Go reformatter, including the way it is invoked:


jsonnet fmt foo.jsonnet


(See --help for more details.)

This tool will clean up whitespace, re-indent, and use syntax sugars where possible. It can control the way strings are quoted and the comment style. However it will not (yet) break or join lines. It is quite extensible so please submit issues for more things you think it should fix. An obvious candidate is alphabetic re-ordering of imports!

One thing for which it's very useful is the process of bootstrapping unformatted JSON code into beautiful Jsonnet. To do this, use a standard JSON reformatter like json_pp to line-break the JSON how you want. Then run jsonnet fmt on it to strip the quotes from field names where possible and put commas on the end of lines. That takes care of the most boring aspects of refactoring JSON into Jsonnet!

Another new feature is to add Python style array slicing. This is the ability to do


local arr = ['it', 'was', 'the', 'best', 'of', 'times'];
assert arr[2:4] == ['the', 'best'];
assert arr[1::2] == ['was', 'best', 'times'];
true


This is designed to be compatible with Python, except the ability to use negative numbers to refer to elements relative to the far end of the array. The rationale: We'd rather have an explicit error to catch broken index arithmetic, and you can always use std.length() when you (rarely) need to refer to the last element of an array.

Did you notice the last example used ' for string literals instead of the " as prescribed by JSON? This new kind of string literal is useful in cases when you want to embed " in your string without escaping. It also has a bit less visual clutter.

Another innovation is --yaml-stream, which was implemented primarily for easy output into kubectl (from the Kubernetes project). The idea is that the Jsonnet script manifests into a JSON array, and this is rendered as a "YAML stream". For more context see yaml.org.


$ jsonnet --yaml-stream -e '[{ animal: "dog", sound: "woof" }, { animal: "cat", sound: "meow" }]'
---
{
"animal": "dog",
"sound": "woof"
}
---
{
"animal": "cat",
"sound": "meow"
}
...


The final new feature: We now have JSON merge patch support (RFC7396) in the standard library.