Webapp
* Multiple problems were fixed around webapp fetching, saving, and pinning. The general end-user experience was also improved by allowing the frontend to show an interstitial screen while the backend was fetching the webapp.
Stdout / stderr usage
* Many related issues around stdout and stderr usage were fixed, such as missing endlines being added, and prompts meant to always be seen were moved to stderr.
Misc
* `qri get` gained a flag `--pretty` to pretty-print json.
* `qri save` with the strict flag set will produce an error if it fails to validate
* `qri add` can use a full reference to specify which version to add
Bug Fixes
* **add:** add specific versions of a dataset using the full reference ([4b1e562](https://github.com/qri-io/qri/commit/4b1e562))
* **config:** `config.Copy()` copies over the path ([2fd3420](https://github.com/qri-io/qri/commit/2fd3420))
* **secrets:** Write secrets warning to stderr instead of stdout ([b43a4b7](https://github.com/qri-io/qri/commit/b43a4b7))
* **unpublish:** Add endline when printing message ([810](https://github.com/qri-io/qri/issues/810)) ([0ced201](https://github.com/qri-io/qri/commit/0ced201))
* **validate:** Allow validate to run when connected ([808](https://github.com/qri-io/qri/issues/808)) ([dedc0c5](https://github.com/qri-io/qri/commit/dedc0c5))
* **webapp:** resolve webapp path before passing path to the handler ([f38dcf4](https://github.com/qri-io/qri/commit/f38dcf4))
* **webapp loading:** pin webapp from dweb, send temporary script if loading ([4fb2921](https://github.com/qri-io/qri/commit/4fb2921))
Code Refactoring
* **cmd:** remove ephemeral `qri connect` flags ([8620aa0](https://github.com/qri-io/qri/commit/8620aa0))
Features
* **pretty:** Pretty flag for get command ([b6f579b](https://github.com/qri-io/qri/commit/b6f579b))
BREAKING CHANGES
* **cmd:** removes:
`--disconnect-after`
`--disable-api`
`--disable-rpc`
`--disable-webapp`
`--disable-p2p`
`--read-only`
`--remote-mode`
These can be configured instead by using the `qri config` command
Removing these temporary flags let's us better reason about the state of the config at any time, as well as helps us be confident that at any time we are not saving over our config with false information.
<a name="v0.8.1"></a>
[v0.8.1](https://github.com/qri-io/qri/compare/v0.8.0...v) (2019-06-11)
This patch release fixes a small-but-critical bug that prevented `qri setup` from working. A few other fixes & bumps made it in, but the main goal was restoring `qri setup` so folks can, you know, set qri up.
Bug Fixes
* **config:** lib.NewInstance option func must check for nil pointers ([69537ce](https://github.com/qri-io/qri/commit/69537ce))
* **lib/diff:** adjust deepdiff.Diff params ([62a13eb](https://github.com/qri-io/qri/commit/62a13eb))
* **setup:** load plugins before attempting to setup IPFS ([795](https://github.com/qri-io/qri/issues/795)) ([69c5fda](https://github.com/qri-io/qri/commit/69c5fda))
* **startf:** bump version number to 0.8.1 ([e27466a](https://github.com/qri-io/qri/commit/e27466a))
Features
* **diff:** return source data in diff response ([d3eae83](https://github.com/qri-io/qri/commit/d3eae83))
* **diff:** return source data in diff response ([d1d2da5](https://github.com/qri-io/qri/commit/d1d2da5))
<a name="0.8.0"></a>
[0.8.0](https://github.com/qri-io/qri/compare/v0.7.3...v0.8.0) (2019-06-05)
Version 0.8.0 is our best-effort to close out the first set of public features.
Automatic Updates ([RFC0024](https://github.com/qri-io/rfcs/blob/master/text/0024-scheduled-updates.md))
Qri can now keep your data up to date for you. 0.8.0 overhauls `qri update` into a service that schedules & runs updates in the background on your computer. Qri runs datasets and maintains a log of changes.
schedule shell scripts
Scheduling datasets that have starlark transforms is the ideal workflow in terms of portability, but a new set of use cases open by adding the capacity to schedule & execute shell scripts within the same cron environment.
Starlark changes
We've made two major changes, and one small API-breaking change. Bad news first:
`ds.set_body` has different optional arguments
`ds.set_body(csv_string, raw=True, data_format="csv")` is now `ds.set_body(csv_string, parse_as="csv")`. We think think this makes more sense, and that the previous API was confusing enough that we needed to completely deprecate it. Any prior transform scripts that used `raw` or `data_format` arguments will need to update.
new beautiful soup-like HTML package
Our `html` package is difficult to use, and we plan to deprecate it in a future release. In it's place we've introduced `bsoup`, a new package that implements parts of the [beautiful soup 4 api](https://www.crummy.com/software/BeautifulSoup/bs4). It's _much_ easier use, and will be familiar to anyone coming from the world of python.
the "ds" passed to a transform is now the previous dataset version
The `ds` that's passed to is now the existing dataset, awaiting transformation. For technical reasons, `ds` used to be a blank dataset. In this version we've addressed those issues, which makes examining the current state a dataset possible without any extra `load_dataset` work. This makes things like append-only datasets a one-liner:
python
def transform(ds,ctx):
ds.set_body(ds.get_body().append(["new row"]))
CLI uses '$PAGER' on POSIX systems
Lots of Qri output is, well, long, so we now check for the presence of the `$PAGER` environment variable and use it to show "scrolling" data where appropriate. While we're at it we've cleaned up output to make things a little more readable. Windows should be unaffected by this change. If you ever want to _avoid_ pagination, I find the easiest way to do so is by piping to `cat`. For example:
$ qri ls | cat
Happy paging!
Switch to go modules
Our project has now switched entirely to using go modules. In the process we've deprecated `gx`, the distributed package manager we formerly used to fetch qri dependencies. This should dramatically simplify the process of building Qri from source by bringing dependency management into alignment with idiomatic go practices.
Dataset Strict mode
`dataset.structure` has a new boolean field: `strict`. If `strict` is `true`, a dataset _must_ pass validation against the specified schema in order to save. When a dataset Dataset is in strict mode, Qri can assume that all data in the body is valid. Being able to make this assumption will allow us to provide additional functionality and performance speedups in the future. If your dataset has no errors, be sure to set `strict` to `true`.
Bug Fixes
* **`doesCommandExist`:** fix to `exec.Command` ([4319db3](https://github.com/qri-io/qri/commit/4319db3))
* **`printToPager`:** different syntax needed for different systems ([86644a0](https://github.com/qri-io/qri/commit/86644a0))
* **api/export:** `/export/` api endpoint now sends data! ([7718b26](https://github.com/qri-io/qri/commit/7718b26))
* **api/list:** fix/add pagination for api `/list` endpoint ([6187c09](https://github.com/qri-io/qri/commit/6187c09))
* **base:** cron dataset command shouldn't use quotes in args ([d0568dc](https://github.com/qri-io/qri/commit/d0568dc))
* **base.DatasetLog:** fix limit and offset logic ([f7b042d](https://github.com/qri-io/qri/commit/f7b042d))
* **base.ListDataset:** add offset and limit to base.ListDataset ([dab742a](https://github.com/qri-io/qri/commit/dab742a))
* **base/log:** fix error that didn't check if PreviousPath exists before loading it ([748e58b](https://github.com/qri-io/qri/commit/748e58b))
* **canonicalize:** Correct peername when profileID matches ([affc3c3](https://github.com/qri-io/qri/commit/affc3c3))
* **cmd:** don't use FgWhite, breaks light-colored termimals ([32b8793](https://github.com/qri-io/qri/commit/32b8793))
* **cmd:** listing page 2 starts numbering at the offset ([b37d186](https://github.com/qri-io/qri/commit/b37d186))
* **cmd/get:** must catch `DatasetRequest` error ([f200acc](https://github.com/qri-io/qri/commit/f200acc))
* **config:** Check err when parsing config to avoid segfault ([f88e6e6](https://github.com/qri-io/qri/commit/f88e6e6))
* **config:** fix api.Copy not copying all fields ([4f30f9f](https://github.com/qri-io/qri/commit/4f30f9f))
* **config:** make setter actually write file ([12230e3](https://github.com/qri-io/qri/commit/12230e3))
* **connect:** use lib.NewInstance with qri connect ([3fc3de4](https://github.com/qri-io/qri/commit/3fc3de4))
* **cron:** connect cron to log file, actually run returned command in lib ([41c9366](https://github.com/qri-io/qri/commit/41c9366))
* **docs:** Building on windows, rpi, and brew install instructions ([8953591](https://github.com/qri-io/qri/commit/8953591))
* **events:** event PeerIDs serialize properly to strings ([7f4a23e](https://github.com/qri-io/qri/commit/7f4a23e))
* **fill.Struct:** If present, use field tag as the field name ([432da1d](https://github.com/qri-io/qri/commit/432da1d))
* **linux:** Ignore errors from setting rlimit, needs root on linux ([02108a9](https://github.com/qri-io/qri/commit/02108a9))
* **list:** qri list new command-line interface ([daf2b70](https://github.com/qri-io/qri/commit/daf2b70))
* **Makefile:** Require go version 1.11 ([0cc2f9b](https://github.com/qri-io/qri/commit/0cc2f9b))
* **p2p:** Improve comment for IPFSCoreAPI accessor ([aec34c5](https://github.com/qri-io/qri/commit/aec34c5))
* **p2p:** pass missing context to ipfsfs.GoOnline ([43a4b2e](https://github.com/qri-io/qri/commit/43a4b2e))
* **peers:** add default limit to peers, fix test ([fe8f6d8](https://github.com/qri-io/qri/commit/fe8f6d8))
* **ref:** CanonicalizeProfile just handles renames all the time ([be1736b](https://github.com/qri-io/qri/commit/be1736b))
* **render:** supply default 'html' format if none exists ([2b82b41](https://github.com/qri-io/qri/commit/2b82b41))
* **repo/test:** fix rebase mistake, func should be named `NewTestRepoWithHistory` ([a39a6ae](https://github.com/qri-io/qri/commit/a39a6ae))
* **transform:** Pass prev dataset to ExecScript ([6ff666c](https://github.com/qri-io/qri/commit/6ff666c))
* **update:** fix update run missing type assignment, absolutize paths ([11ec50c](https://github.com/qri-io/qri/commit/11ec50c))
* **vet:** fix go vet error ([8f46850](https://github.com/qri-io/qri/commit/8f46850))
* plumb context into network methods ([fb036e3](https://github.com/qri-io/qri/commit/fb036e3))
Code Refactoring
* **cmd:** change `limit` and `offset` flags to `page` and `page-size` ([62d5da8](https://github.com/qri-io/qri/commit/62d5da8))
* **lib.AbsPath:** deprecate lib.AbsPath, use qfs.AbsPath ([112362a](https://github.com/qri-io/qri/commit/112362a))
Features
* **/list/:** add `term` param to `/list/` api, for local dataset search ([f66f857](https://github.com/qri-io/qri/commit/f66f857))
* **api:** add update endpoints ([f5f2024](https://github.com/qri-io/qri/commit/f5f2024))
* **base/fill:** Overhaul error handling ([1deb967](https://github.com/qri-io/qri/commit/1deb967))
* **cmd:** add --repo and --ipfs-path flags ([1711f30](https://github.com/qri-io/qri/commit/1711f30))
* **cmd:** overhaul update command ([4218a28](https://github.com/qri-io/qri/commit/4218a28))
* **config:** add update configuration details ([5786d68](https://github.com/qri-io/qri/commit/5786d68))
* **cron:** add cron package for scheduling dataset updates ([7b38164](https://github.com/qri-io/qri/commit/7b38164))
* **cron:** add flatbuffer HTTP api server & client ([50388e5](https://github.com/qri-io/qri/commit/50388e5))
* **cron:** add incrementing job.RunNumber, refactor job field names ([5f6b403](https://github.com/qri-io/qri/commit/5f6b403))
* **cron:** store dataset SaveParams as job options ([df629a7](https://github.com/qri-io/qri/commit/df629a7))
* **cron:** store logs and files of stdout logs ([a9f3c52](https://github.com/qri-io/qri/commit/a9f3c52))
* **cron.FbStore:** store cron jobs as flatbuffers ([755186d](https://github.com/qri-io/qri/commit/755186d))
* **cron.file:** FileJobStore for saving jobs to a backing CBOR file ([9d4de77](https://github.com/qri-io/qri/commit/9d4de77))
* **fill.Struct:** Support interface{} fields, require map string keys ([46166d1](https://github.com/qri-io/qri/commit/46166d1))
* **instance:** instroduce Instance, deprecate global config ([53a8e4b](https://github.com/qri-io/qri/commit/53a8e4b))
* **lib:** sew initial cron service into lib ([5e8a871](https://github.com/qri-io/qri/commit/5e8a871))
* **peers:** fix limit/offset bugs, add sending output to `less` when listing peer cache ([2e623db](https://github.com/qri-io/qri/commit/2e623db))
* **update:** add update log command and api endpoints ([56c4840](https://github.com/qri-io/qri/commit/56c4840))
* **update:** build out update package API, use it ([3004fd8](https://github.com/qri-io/qri/commit/3004fd8))
* **update:** cleanup CLI output ([fa2ebad](https://github.com/qri-io/qri/commit/fa2ebad))
* **update:** detect and report 'no changes' on dataset jobs ([ae63512](https://github.com/qri-io/qri/commit/ae63512))
* **update:** experimental support for multi-repo updates with --use-repo ([9ae8d44](https://github.com/qri-io/qri/commit/9ae8d44))
* **update:** introduce update package & update daemon ([dc6066c](https://github.com/qri-io/qri/commit/dc6066c))
* **update:** support updates via shell scripts ([c94edd8](https://github.com/qri-io/qri/commit/c94edd8))
BREAKING CHANGES
* **cron:** Field names of cron.Job have been refactored, which will break repo/update/logs.qfb and repo/update/jobs.qfb files. Delete them to fix.. This will only affect users who have been building from source between releases.
* **connect:** "qri connect" no longer has flags for setting port numbers
* **api:** /update endpoint is moved to /update/run
* **cmd:** On the cli, all `limit` and `offset` flags have been changed to `page` and `page-size`.
* **lib.AbsPath:** lib.AbsPath is removed, use github.com/qri-io/qfs.AbsPath instead
<a name="0.7.3"></a>
[0.7.3](https://github.com/qri-io/qri/compare/v0.7.2...v0.7.3) (2019-04-03)
This release is all about 3 Rs:
* Rendering
* Remotes
* `load_dataset`
This release we've focused on improving dataset visualiing, setting the stage with better defaults and a cleaner API for creating custom viz. We think expressing dataset vizualiations as self-contained `html` makes Qri datasets an order of magnitude more useful, and can't wait for you to try it.
Along with the usual bug fixes, a few nice bonuses have landed, like supplying [multiple `--file` args](https://github.com/qri-io/qri/pull/718) to qri save to combine dataset input files, and `qri get rendered` to show rendered viz. Anyway, on to the big stuff:
Default Rendering ([RFC0011](https://github.com/qri-io/rfcs/blob/master/text/0011-html_viz.md))
Whenever you create a new dataset version, Qri will now create a default viz component if you don't provide one. Unless run with `--no-render`, Qri will now execute that template, and store the result in a file called `index.html` in your dataset. This makes your dataset _much_ more fun when viewed directly on the d.web, which is outside of Qri entirely.
This is because IPFS HTTP gateways are sensitive to `index.html`. When you use qri to make a dataset, your dataset comes with a self-contained visualization that others can see without downloading Qri at all.
We think this dramatically increases the usefulness of a dataset, and increases the chances that others will want to share & disseminate your work by making your dataset a more-complete offering in the data value chain. These embedded default visualizations drop the time it takes to create a readable dataset to one step.
That being said, we've intentionally made the default visualization rather bland. The reason for this is twofold. First, to keep the file size of the `index.html` small (less than 1KB). Second, we want you to customize it. We'll refine the default template over time, but we hope you'll use viz to tell a story with your data.
Users may understandably want to disable default vizualizations. To achieve this `qri save` and `qri update` have a new flag: `--no-render`. No render will prevent the execution of any viz template. This will save ~1KB per version, at the cost of usability.
Overhauled HTML Template API ([RFC0011](https://github.com/qri-io/rfcs/blob/master/text/0011-html_viz.mdtemplate-api))
Keeping with the theme of better viz, we've also taken time to overhaul our template API. Given that this is a public API, we took some time to think about what it would mean to try to render Qri templates _outside_ of our go implementation. While no code does this today, we wanted to make sure it would be easier in the future, so we took steps to define an API that generally avoids use of the go templating `.`, instead presenting a `ds` object with json-case accessors. Taking things like this:
<h1>{{ .Meta.Title }}</h1>
to this:
<h1>{{ ds.meta.title }}</h1>
This change brings the template syntax closer to the way we work with datasets in other places (eg: in `dataset.yaml` files and starlark transform scripts), which should help cut down on the mental overhead of working with a dataset in all these locations. We think this up-front work on our template API will make it easier to start writing custom templates. We don't have docs up yet, but the RFC [reference](https://github.com/qri-io/rfcs/blob/master/text/0011-html_viz.mdreference-level-explanation) section outlines the API in detail.
Experimental Remote Mode ([RFC0022](https://github.com/qri-io/rfcs/blob/master/text/0022-remotes.md))
The registry is nice and all, but we need more ways to push data around. In this release we're launching a new expriment called "remotes" that start into this work. Remotes act as a way for any user of Qri to setup their own server that keeps datasets alive, providing availability and ownership over data within a set of nodes that they control.
Currently we consider this feature "advanced only" as it comes with a number of warnings and some special setup configuration. For more info, [check the RFC](https://github.com/qri-io/rfcs/blob/master/text/0022-remotes.md), and if you're interested in running a remote, [hop on discord](https://discordapp.com/invite/etap8Gb) and say "hey I want to run a remote".
Starlark `load_dataset` ([RFC0023](https://github.com/qri-io/rfcs/blob/master/text/0023-starlark_load_dataset.md))
We've made a breaking API change in Starlark that deprecates `qri.load_dataset_body`, and introduce a new global function: `load_dataset`. This new API makes it clear that `load_dataset` both loads the dataset and declares it as a dependency of this script. This is an important step toward making datasets a first-class citizen in the qri ecosystem. Here's an example of the new syntax:
python
load("http.star", "http")
load a dataset into a variable named "fhv"
fhv = load_dataset("b5/nyc_for_hire_vehicles")
def download(ctx):
use the fhv dataset to inform an http request
vins = ["%s,%s" % (entry['vin'], entry['model_yearl']), for entry in fhv.body()]
res = http.post("https://vpic.nhtsa.dot.gov/api/vehicles/DecodeVINValuesBatch/", form_body={
'format': 'JSON',
'DATA': vins.join(";")
})
return res.json()
def transform(ds, ctx):
ds.set_body(ctx.download)
Users who were previously using `qri.load_dataset_body` will need to update their scripts to use the new syntax. The easiest way to do that is by adding a new version to your dataset history with the updated script:
$ qri get transform.script me/dataset > transform.star
make updates to transform.star file & save
$ qri save --file transform.script me/dataset
Three easy steps, and your dataset log tells the story of the upgrade.
Bug Fixes
* **api:** Listen on localhost for API and RPC ([04a4500](https://github.com/qri-io/qri/commit/04a4500))
* **fill_struct:** Work with non-pointer structs and pointers to non-structs ([0d14bac](https://github.com/qri-io/qri/commit/0d14bac))
* **get:** If format is misspelled, display an error ([b8fcceb](https://github.com/qri-io/qri/commit/b8fcceb))
* **save:** Better error message when saving with wrong file ([75cb06a](https://github.com/qri-io/qri/commit/75cb06a))
* **tests:** Save with viz needs html format, update testdata file ([f7cd486](https://github.com/qri-io/qri/commit/f7cd486))
Code Refactoring
* **render:** remove limit, offset, all parmeters ([165abce](https://github.com/qri-io/qri/commit/165abce))
Features
* **cmd/get:** add `rendered` to selector in `qri get` command ([90719a8](https://github.com/qri-io/qri/commit/90719a8))
* **dag info:** add label for rendered viz output ([7157cd7](https://github.com/qri-io/qri/commit/7157cd7))
* **daginfo:** add `qri daginfo` command that returns a dag.Info of a dataset ([e4d9e27](https://github.com/qri-io/qri/commit/e4d9e27))
* **daginfo:** add summary view for `qri daginfo` ([9f9b32f](https://github.com/qri-io/qri/commit/9f9b32f))
* **dagInfo:** add ability to get a subDag at a specific label ([7a07ee8](https://github.com/qri-io/qri/commit/7a07ee8))
* **remote:** Beginning of "remote" mode implementation ([49dbef9](https://github.com/qri-io/qri/commit/49dbef9))
* **remote:** Bump version number, review fixes. ([21a4448](https://github.com/qri-io/qri/commit/21a4448))
* **remote:** Deserialize response from remote using json.Decode ([2d73e84](https://github.com/qri-io/qri/commit/2d73e84))
* **remote:** Parse config using FillStruct. Remotes field. ([a2efbf3](https://github.com/qri-io/qri/commit/a2efbf3))
* **remote:** Retrieve sessionID & diff first, pass both to dsync ([7f3d8bc](https://github.com/qri-io/qri/commit/7f3d8bc))
* **remote:** Send actual dag.Info to remote, perform dsync ([5cec2de](https://github.com/qri-io/qri/commit/5cec2de))
* **remote:** Switch to --remote-accept-size-max flag ([b26cbb9](https://github.com/qri-io/qri/commit/b26cbb9))
* **remotes:** base/fill/PathValue used by Config, to enable Remotes ([4d45dce](https://github.com/qri-io/qri/commit/4d45dce))
* **remotes:** Complete dsync by writing to ds_refs and pinning ([b47a3e4](https://github.com/qri-io/qri/commit/b47a3e4))
* **save:** add `no-render` option to `qri save` command ([a8f36b2](https://github.com/qri-io/qri/commit/a8f36b2))
* **save:** More tests for saving: viz, transform. Update comments. ([91246fe](https://github.com/qri-io/qri/commit/91246fe))
* **save:** Save a dataset with multiple file arguments ([23735b7](https://github.com/qri-io/qri/commit/23735b7))
* **save:** Tests for saving datasets with multiple file arguments ([9fe54bc](https://github.com/qri-io/qri/commit/9fe54bc))
BREAKING CHANGES
* **render:** qri render limit, offset, all parameters have been removed
<a name="0.7.2"></a>
[0.7.2](https://github.com/qri-io/qri/compare/v0.7.1...v0.7.2) (2019-03-14)
Better piping for `qri save`
Minor set of fixes aimed at making the command line work better. The biggest command we've fixed is `qri save --dry-run`. Qri now properly routes diagnostic output to `stderr` (things like the spinner, transform exectution, and completion messages), while sending the _result_ of the dry run to `stdout`. This makes the following much more powerful:
`qri save --dry-run --file transform.star me/dataset > dry_run_output.json`
If save works, the `dry_run_output.json` will now be a valid `json` file representing the result of the dry run, and you'll still see progress output in your terminal window. We'll be working to bring this separation of diagnostic & results output to all commands in the near future.
Run transforms directly with the `--file` flag
One other small note, if you feed a file that ends in `.star` to `qri save --file`, Qri will assume you mean it to be a transform script, and execute it as such. This cuts out the need for a `dataset.yaml` file, and makes working with transforms a little easier to reason about. If you need to provide configuration details to the transform, you'll still need to create a `dataset.yaml` file & specify your `.star` file there.
Bug Fixes
* **get:** better defaults for get body. ([5631d08](https://github.com/qri-io/qri/commit/5631d08))
* **stderr:** write save diagnostic output to stderr ([3dac8b8](https://github.com/qri-io/qri/commit/3dac8b8))
Features
* **ReadDatasetFile:** infer .star & .html files to be bare tf & vz components ([89fcba8](https://github.com/qri-io/qri/commit/89fcba8))
BREAKING CHANGES
* **get:** cli body command is now removed
<a name="0.7.1"></a>
[0.7.1](https://github.com/qri-io/qri/compare/v0.7.0...v0.7.1) (2019-03-07)
0.7.1 is on the larger side for a patch release, shipping fixes & features aimed at doing more with datasets. With this release qri is better at comparing data, and more flexible when it comes to both saving & exporting data.
Overhauled Diff
We've completely overhauled our `diff` command, using a new algorithm that's capable of diffing large volumes of structured data quickly. We've also adjusted the command itself to work with structured data both inside and outside of qri.
Two caveats come with this new feature:
* This code is new. We have tests that show our differ produces _valid_ diff scripts, but sometimes diff doesn't produce diffs that are fun to read. We'd love your feedback on improving the new diff command.
* We haven't really figured out a proper way to _visualize_ these diffs yet.
Caveats aside, new diff is mad decent. Here's a few examples:
get more info on diff:
$ qri diff --help
diff dataset body against it's previous version:
$ qri diff body me/annual_pop
diff two dataset meta sections:
$ qri diff meta me/population_2016 me/population_2017
diff two local json files:
$ qri diff a.json b.json
diff a json & a csv file:
$ qri diff data.csv b.json
output diff as json
$ qri diff a.json b.json --format json
print just a diff summary
$ qri diff me/annual_pop --summary
Revamped Export
We've overhauled our export command to up it's utility. We've also introduced the terms "foreign" and "native" export to refer to the _fidelity_ of an export. This round of improvements has focused on foreign exports, which converts qri datasets into common formats. For example, now you can (finally) express a complete dataset as json:
$ qri export --format json me/annual_pop
We've also made the role `qri export` plays work better in relation to `qri get`. If you want a _complete_ dataset, use `qri export`. If you want a piece of a dataset (for example, just the body), use `qri dataset`.
Shorthand Save Files
We've added a way to save changes to datasets that only affect individual components. If you only want to change, say the meta section of a dataset, you can now create a file that only affects the meta component. In the past, you'd have no choice but to construct a complete dataset document, and only alter the `meta` section. Instead you can now save a file like this:
json
{
"qri": "md:0", // this "md:0" value for the "qri" key tells qri this only affects meta
"title" : "this is a new title",
"description": "new decription"
}
and give that to save:
`$ qri save --file meta.json me/dataset`
Qri will interpret this file and apply the changes to the meta component. This pairs very nicely with `qri get` to make workflows that only affect specific components, you could use this to change a dataset's schema, which lives in the `structure` component:
$ qri get structure me/annual_pop > structure.yaml
save changes to structure.yaml
$ qri save --file structure.yaml me/annual_pop
Starlark improvements:
We've added a `json` package to starlark, and given `math` a new method `round()`, which is not built into starlark as a language primtive. If you want to round numbers in starlark scripts, do this:
python
load("math.star", "math")
print(math.round(3.5)) prints: 4
Fixes
We've also shipped a bunch of little bug fixes that should add up to a better experience. All the changes described above also apply to the qri JSON api as well. Happy versioning!
Bug Fixes
* **actions.SaveDataset:** delay inferring values until after writing changes ([86f57fa](https://github.com/qri-io/qri/commit/86f57fa))
* **api:** restore api /diff endpoint ([e71fd8e](https://github.com/qri-io/qri/commit/e71fd8e))
* **base.ReadEntries:** Simplify ReadEntries by using PagedReader in render ([4f914a6](https://github.com/qri-io/qri/commit/4f914a6))
* **cli persistent flags:** properly register global flags ([f4f1ed7](https://github.com/qri-io/qri/commit/f4f1ed7)), closes [506](https://github.com/qri-io/qri/issues/506)
* **export:** Export --zip flag ([8514c76](https://github.com/qri-io/qri/commit/8514c76))
* **export:** Overhaul export command ([f7df3f9](https://github.com/qri-io/qri/commit/f7df3f9))
* **export:** Test cases for export ([12a279f](https://github.com/qri-io/qri/commit/12a279f))
* **export:** Update api zip handler, and cmd integration test ([82cd4cc](https://github.com/qri-io/qri/commit/82cd4cc))
* **get:** Apply format to get when using a selector ([ed1fd31](https://github.com/qri-io/qri/commit/ed1fd31))
* **get:** Get returns 404 if dataset is not found ([291296b](https://github.com/qri-io/qri/commit/291296b))
* **get dataset:** return a datasetRef, not a dataset! ([6e51a31](https://github.com/qri-io/qri/commit/6e51a31))
* **info:** Remove the info command, list does it now ([ef34a05](https://github.com/qri-io/qri/commit/ef34a05))
* **list:** Flag --num-versions shows number of versions of each dataset ([2f5a8b1](https://github.com/qri-io/qri/commit/2f5a8b1))
* **list:** Improve usability of `list` command ([d773363](https://github.com/qri-io/qri/commit/d773363))
* **list:** Rename lib parameter to ShowNumVersions ([7214f93](https://github.com/qri-io/qri/commit/7214f93))
Features
* **cmd:** add summary flag to diff command ([f793c0f](https://github.com/qri-io/qri/commit/f793c0f))
* **diff:** added diffStat string, support for diffing files ([2d16df5](https://github.com/qri-io/qri/commit/2d16df5))
* **diff:** overhauling diff to use difff ([b825e65](https://github.com/qri-io/qri/commit/b825e65))
* **fill_struct:** Additional tests, cover some edge-cases ([e1454de](https://github.com/qri-io/qri/commit/e1454de))
* **fill_struct:** Fix qri key, multiple small fixes ([71d78e7](https://github.com/qri-io/qri/commit/71d78e7))
* **fill_struct:** Json and yaml deserialization rewrite ([ace2c1e](https://github.com/qri-io/qri/commit/ace2c1e))
* **fill_struct:** Rename to SetArbitrary ([990ad24](https://github.com/qri-io/qri/commit/990ad24))
* **fill_struct:** Support bool, float, slices ([314be2a](https://github.com/qri-io/qri/commit/314be2a))
* **lib.Datasets.Save:** add force flag to skip empty-commit checking ([6fd5b1f](https://github.com/qri-io/qri/commit/6fd5b1f))
<a name="0.7.0"></a>
[0.7.0](https://github.com/qri-io/qri/compare/v0.6.2...v0.7.0) (2019-02-05)
We've bumped this release to 0.7.0 to reflect some major-league refactoring going on deeper in the stack. The main goal of this release has been to drive stability up, and put the Qri codebase on firmer foundation for refinement.
Major Stability Improvements
Much of our refactoring has gone into removing & consolidating code, making it easier reason about. The effect of this are fewer unintended interactions between different subsystems, resulting in a version of qri that behaves much better, especially in the corner cases. All of these improvements are made on both the JSON API, the CLI, _and_ while operating over RPC (we use RPC when `qri connect` is running in another terminal).
Commands like `qri get` and `qri export` in particular work consistently, and can be used to greater effect. For example, running `qri use` on a dataset let's you drop the repeated typing of a dataset name, and play with `get` to explore the dataset faster:
$ qri use me/dataset
$ qri get meta --format yaml
$ qri get body --format json --limit 20 --offset 3
It's much to fetch scripts with `get` and write them to a file for local work:
$ qri get transform.script > transform.star
Speaking of transforms, `qri update` on a local dataset is now true alias for `qri save --recall=tf`. Eliminating the alternate codepath for `qri update` has made update work far better for re-running transforms.
Export now has a far friendlier `--format` flag for getting a dataset document into a common data format. This'll give you a JSON interpretation of your dataset:
qri export --format=json me/dataset
We're inching closer to our overall goal of building Qri into a series of well-composed packages with a cohesive user interface. Lots more work to do, partly because there's more that we _can_ do now that our code is better composed. We'd encourage you to play with the CLI if you haven't yet taken it for a spin.
XLSX support
Along with the improved export command, we now have early support for Excel `.xlsx` documents! this now works:
qri export --format=xlsx peer/dataset
this now works too (so long as there's a sheet named "sheet1"), and an existing history in another format like JSON, CSV, or CBOR:
qri save --body dataset.xlsx --keep-format peer/dataset
Lots of work to do here, but early support for excel is an exciting addition.
Bug Fixes
* **get:** Datasets whose name contains a field should work with get ([97410fb](https://github.com/qri-io/qri/commit/97410fb))
* **get:** Fix get command using a dotted path ([c130349](https://github.com/qri-io/qri/commit/c130349))
* **get:** Turn `body` into a selector of the `get` command ([7793845](https://github.com/qri-io/qri/commit/7793845))
* **local update:** rework local update to be a type of save ([fa9ca11](https://github.com/qri-io/qri/commit/fa9ca11))
* **save:** Improve error message if new ds has no body or structure ([31332fd](https://github.com/qri-io/qri/commit/31332fd))
* **save:** Infer more values, such as Schema, when appropriate ([1c95539](https://github.com/qri-io/qri/commit/1c95539))
Features
* **export:** initial foreign export support ([9382483](https://github.com/qri-io/qri/commit/9382483))
* **get:** support getting script file fields ([da8ae46](https://github.com/qri-io/qri/commit/da8ae46))
<a name="0.6.2"></a>
[0.6.2](https://github.com/qri-io/qri/compare/v0.6.1...v0.6.2) (2019-01-22)