Based on our recent discussion around Beancount branches being confusing, I did the following to branches:
- v2: remains the same (no change); but v2 is now even less subject to freeze exceptions than before, because v3 is now officially born and PyPI releases will be made for it. The default reply to patches to v2 will be "move that to v3."
- cpp: I turned the previous master branch's state with all C++ and Bazel build into a new 'cpp' branch.
All work on the C++ rewrite -- if it is to continue -- will live on that branch.
If things do move to Rust, I will probably salvage bits and pieces of that manually into a 'rust' branch.
In particular, the schema defined in the proto files is something I'd like to carry forward (nicely
documented and distinguishes between the pre and post interpolation data structures explicitly),
and perhaps the new reFlex parser will be useful at some point.
- v3: a new branch, to contain the state of the master branch MINUS the C++ bits.
As stated above, the C++ code remains only on branch 'cpp'.
This is now the official release branch and PyPI releases will be made of it.
- master: this branch is now where new work on v3 might live.
I'm not sure we really need it or what it should mean, but it should track v3 for now.
I've reviewed missing PyPI releases and made sure we have these:
https://pypi.org/project/beancount/
https://pypi.org/project/beangulp/
https://pypi.org/project/beanquery/
https://pypi.org/project/beanprice/
https://pypi.org/project/beangrow/
https://pypi.org/project/beancount2ledger/
The inclusion of v3 releases for PyPI and new PyPI releases for subprojects should normalize the versioning situation in Beancount. Note that packaging and following other people's conventions are not my forte (and I don't use PyPI for my own projects I just use source), if you have any suggestions or something's missing please reply to this thread.
Thanks for your patience,
2022-02-15
- Released 2.3.5 with only a few minor bugfixes. Mainly released for 3.10.
2021-03-20
- Fixed 639: Released 2.3.4.
2021-01-09
- Fixed bug in booking that assumed number is set on cost. I did this while
running booking manually on entries freshly created on an importer (never ahd
done that before, will be customary in v3).
2021-01-08
- Fixed recently introduced bug with the export script that reordered the
columns.
- Fixed bug with printer leaving trailing whitespace. Was annoying.
2020-12-20
- Merged half of branch pr128_seltzered_beanpriceupdate on this side of
Beancount. The other side will be patched to
http://github.com/beancount/beanprice. This adds some trimming functions to
beancount.ops.lifetimes.
2020-12-14
- Fixed 588: Catch more narrow exceptions from plugins and render full stack
when caught.
2020-11-25
- Fixed 584 - a bug which was occurring because a pathological example using a
very large precision was used in inferring the precision of a currency and a
subsequent attempt to quantize a large number exceeded the default precision.
2020-11-22
- Fixed bug in parser.parse_file() which when passed a filename in would leave
the file unclosed.
- Fixed latent bug with key/value parsing (incorrect grammar rule, was working
anyway somehow).
- Internal changes to lexer and parser to minimize diffs with ongoing work in
'master' / C++ scanner/parser.
2020-11-17
- Made it legal to insert a balance entry after account closure. It is a common
case that transactions happen on the last day of an account and the balance
directive operates at the beginning of the day. It was arguable whether
directives should be allowed after closure; notes and documents were allowed
because statements and phone calls may still occur after an account is
closed, but I suppose you can make the argument that a statement balance can
also occur at a later date, so voila, make it legal. You still cannot post
transactions to a closed account, which is the thing that really matters.
2020-11-15
- Render the tolerance in Balance directives (merged from Aaron Lindsay).
2020-11-01
- In the process of removing dependencies to the beancount.ops.holdings module,
significantly sped up the net-worth-over-time script by using simple
aggregation to an inventory (which is how it should be done).
- Allow balance_by_account() method to compress positions that aren't booked,
as an option.
- Added split() method to Inventory, like segregate_units but with no
constraints.
- Fixed 255: Added a new beancount.core.prices.project() method to project
prices to another cost currency.
2020-10-26
- Fixed Github Actions workflows and Wheel generation.
- Released 2.3.3 to trigger new binaries.
2020-10-25
- Removed the part of setup_test checks running the binaries with --help, as
they fail starting at 3.8. Someone with more time would create a test using
pytest-virtualenv.
2020-10-17
- Fixed 566: Duplicate postings on entries should count toward distinct
hashes.
2020-10-10
- Moved Emacs support to its own repository
(http://github.com/beancount/beancount-mode).
2020-09-12
- Added metadata on implicitly generated Price directives (from implicit_prices
plugin).
2020-09-12
- Released 2.3.2.
2020-09-07
- Fixed 401: Fixed initialization bugs on min() and max() functions for the SQL
aggregator.
- Fixed 364: Fixed important core bug in how the CostSpec were converted to
Cost for cost specs with total and per-unit cost numbers, like this {106.935
6.98 USD}. For short positions, the cost was computed as negative. In a
sense this was a naive bug, but a very important one. This occurred only for
short positions.
2020-08-19
- Added an option to the printer to allow an arbitraty Python value to be
output. This is useful when creating values programmatically with types not
supported by the syntax.
2020-07-25
- Release 2.3.1 as dependency for beanprice.
- Moved beancount.prices.find_prices to beancount.ops.find_prices in
preparation of moving price fetching code to a new 'beanprice' repository.
2020-07-17
- Fixed bug 508 in beancount.plugins.unique_prices.
2020-07-06
- Updated export script to honor lowercase "ignore" instead of "IGNORE" and set
default tax type to "taxable" instead of "TAXABLE" so that changes in PR 472
don't pick up the metadata as currencies.
2020-07-05
- Removed bean-report and all associated codes. Use bean-query instead (or
Fava).
- Removed bean-web and bean-bake (for v3). Use Fava instead.
- Moved beancount.web.scrape to beancount.utils.scrape in preparation of
removing beancount.web.
- Removed experiments/portfolio for its dependency on beancount.reports code
(this script was obsolete anyway).
- Moved get_assets_holdings() function out of reports code to preserve it for
use by an experimental script (in preparation for removing beancount.reports).
- Moved beancount.scripts.tutorial to beancount.reports.tutorial in preparation
of removing beancount.reports and beancount.web.
- Moved beancount.reports.context to beancount.parser.context in preparation of
removing beancount.reports and beancount.web.
- Moved beancount.reports.table to beancount.utils.table in preparation of
removing beancount.reports and beancount.web.
2020-07-04
- Fixed 494, invalid account used for clamping the period view
(Equity:Conversions:Previous instead of Equity:Conversions:Current).
2020-06-27
- Merged PR 484, some parser improvements by dnicolodi.
- Merged PR 488 from branch beancount/pr135_ppmarciniak_fixnetworth by
pmarciniak.
- Merged branch pr131_tohojo_csvtweaks (BitBucket PR 131) by tohojo, fixed
encoding bug in reading the file for the date and allow customizing the
number parser.
2020-06-26
- Merged 480 - parser: Read input data from Python file objects by dnicolodi.
2020-06-10
- Released 2.3.0. See 459.
2020-06-03
- Added a 'hooks' argument to beancount.ingest.scripts_utils.ingest() replacing
'detect_duplicates_func'. 'find_duplicate_entries()' is still inserted if no
other filter functions are specified (I'd like to remove this eventually; not
changing behavior here for now.)
- Added an example for how to run the importers explictly via a Python script
calling the ingest() function.
- Fixed bug with explicit invocation of importers via ingest() function when
calling the 'file' sub command.
- Some installations of pdfminer install 'pdf2txt.py' as 'pdf2txt'. Adjusted
ingestion example scripts.
2020-06-02
- Modified recently added --no-cache option to also delete the cache if it
exists. Honor the same environment variables. The motivation behind this
change is that while --no-cache is useful for debugging plugins, it can be
disconcerting that if you remove it on the next run it picks up the old
existing cache. Deleting it on usage of --no-cache is reasonable behavior and
avoids the introduction of a --clear-cache option; this should be sufficient.
2020-05-25
- Fixed 449: Added options to bean-check: --no-cache and --cache-filename to
override the cache location. These are added to bean-check only and are
useful during development. The 'BEANCOUNT_DISABLE_LOAD_CACHE' and
'BEANCOUNT_LOAD_CACHE_FILENAME' environment variables are still honored by
the loader and work across all other programs.
2020-05-23
- Fixed 184: Added explicit test for Inventory.average().
- Fixed 404: Iex price source not working any more.
- Fixed 281: Query "id" column does not identify transactions uniquely. Made
the default hash_entry() function include metadata. Exclude metadata and
derived fields explicitly in the context of comparison.
- Fixed 182: Eliding an amount with total price raises TypeError.
- Added options to filter accounts in "onecommodity" plugin: skip accounts with
explicitly declared commodities, metadata of "onecommodity: FALSE" and added
regular expression to specify the list of accounts to match (default: all).
2020-05-22
- Migration of code repository to Github.
2020-05-02
- Minor performance improvment by avoiding calling __iadd__ in a few places.
- Fixed bug due to the removal of __version__ in apiclient:
https://github.com/googleapis/google-api-python-client/commit/1d8ec6874e1c6081893de7cd7cbc86d1f6580320
- Changes all imports from 'apiclient' to 'googleapiclient', which is now the
recommended way.
- Fixed some warnings.
- Disabled expensive assertions in Inventory class and implemented a special
case for adding to an empty inventory to avoid expensive aggregation. This
can occur in aggregation loops and is unnecessarily expensive when adding a
single, very large inventory.
- Made balance check aggregation avoid an initial empty inventory object, and
changed the balance check code to call the realization utility function
compute_balance().
2020-04-11
- Fixed 425: Tentative bug fix for crash on decref over null traceback. The
Python API for this function may return some null values. Not taking any
chances. This should fix this particular bug.
2020-03-28
- Allow the "divert_expenses" plugin to override diversions on postings that
aren't for expenses accounts. I needed this to account to a deposit to my
kid's daycare, which is an asset account in his diverted ledger.
2020-03-22
- Added docstrings on some import code (while installing a new importer; my
stuff is sometimes poorly documented I'm afraid).
2020-02-23
- Added new features to the CSV importer:
* A new 'invert_sign' option to invert the sign of the amount.
* A new field, 'CATEGORY', which goes unrendered.
* Replace newlines from all narrations by semi-colons.
These were driven by recent changes to the Amex CSV format.
2019-12-29
- Added a CSV field for a unique reference number. If present, this is set as a
link on the transaction.
2019-12-25
- Applied patch from Daniele Nicolodi
(https://bitbucket.org/blais/beancount/pull-requests/139/parser-work-around-python38-bug-causing-a)
to work around typing constraints around signed size types introduced in
Python 3.8. See also https://bugs.python.org/issue38913.
- Fixed bug with running python3 as a subprocess during testing. Fixes an error
that occurs with the tutorial file generation.
2019-11-29
- Added 'encoding' option to CSV importer. Amex encodes as 'latin1', not
'utf-8' and this turned out to be necessary.
2019-11-10
- Added an option to upload-to-sheets script to apply a minimum to the number
of rows resized to. The problem was I had another sheet longer than the
auto-updated data sheet, and when new rows were being inserted, rows in the
referencing sheet were being incremented automatically to refer to the now
displaced previous rows in the updated data sheet (even if they used to refer
to rows after the last row in the data sheet). Avoid resizing in the number
of rows up or down sheet should avoid most of these problems (except when at
the boundary, but then the minimum number of rows could just be increased).
This is quick fix that should work in the medium term.
2019-10-20
- More changes to unittest while porting to Bazel build (for C++ support).
2019-10-11
- Added __main__ block to all tests because ongoing port to Bazel build
requires it.
2019-10-10
- Moved test data for b.utils.file_type for Bazel support. It otherwise assumes
file_type.py is a package.
2019-08-21
- Released 2.2.3.
- Fixed bug in split_expenses plugin.
2019-06-01
- Upgraded Bison to 3.4 (built from source). Silenced warnings.
2019-05-01
- Fixed 389: Patch for b.p.forecast improvements.
- Added rolled back change to default value of flag used when using the 'txn'
syntax. See issue 295 for a discussion.
2019-03-16
- Had a very quick shot at implementing trade tracking, added the list of
matches positions from the booking methods. This doesn't work yet, I need to
make changes to b.c.Inventory in order to make this work. Partial
changelist.
2019-03-03
- Fixed 378: Rendering metadata with null values wasn't properly implemented.
In the process, I explicit parsing of None values by allowing "NULL".
- Fixed 309: Catch PermissionError to work around
https://bugs.python.org/issue25481.
2019-02-27
- Implemented a quick prototype of the current trading accounts method of
Peter Selinger in 'beancount.plugins.currency_accounts'. A few warts remain:
* FIXME: We need to handle these important cases (they're not frivolous,
this is a prototype), probably by inserting some exceptions with
collaborating code in the booking (e.g. insert some metadata that
disables price conversions on those postings).
* FIXME(2): Ouch! Some of the residual seeps through here, where there
are more than a single currency block. This needs fixing too. You can
easily mitigate some of this to some extent, by excluding transactions
which don't have any price conversion in them.
2019-02-26
- Fixed 376: Coerce implicit boolean to true boolean.
2019-02-10
- Fixed 374: Option --all treatment was broken in 210. Also catch and print
errors.
- Fixed rare failure of shutil.rmtree() in encryption cleanup test.
2019-02-03
- Fixed 372: Removed formatted string literal to keep compat with Python 3.5.
2019-01-29
- Fixed 362: Predicate matching a parent account was matching accounts with
word breaks but not at the boundary of an account name. This was causing
a rare (but important) failure in how Pad balances were calculated.
2019-01-28
- Released 2.2.1 for bug fixes.
2019-01-25
- Improve output of tags and links for bean-report ledger and hledger.
2019-01-22
- Fixed problem with invalid format strings with 0-width field when rendering
MISSING.
2019-01-05
- Merged pull request 85
(https://bitbucket.org/blais/beancount/pull-requests/85) from hardikar,
adding function grepn() to the shell.
- Merged pull request 86
(https://bitbucket.org/blais/beancount/pull-requests/86) from hardikar,
adding function subst() to the shell.
- Merged pull request 89
(https://bitbucket.org/blais/beancount/pull-requests/89) from hardikar,
adding functions date_diff(), date_add() to the shell.
- Implemented best compression for zipfile in bean-bake as per Justus
Pendleton's suggestion. For context, see this:
https://groups.google.com/d/msgid/beancount/de75671d-532a-4deb-bd0f-fd9377e63753%40googlegroups.com?utm_medium=email&utm_source=footer
2018-12-18
- Fixed 214: Add directory of current filename to sys.path with new option
"insert_pythonpath". When this option is encountered in a file, we prepend
the path of the file in which it is encountered at the front of the
PYTHONPATH for the rest of the duration of the problem (we could consider
resetting this after parsing and processing the plugins; not doing it now).
- Added detection of hitting the root if the search for the repository root
failed.
- Fixed 341: find_repository_root() fails on source tarball. Renamed PKGINFO
to PKG-INFO to match what pip sdist produces.
- Change version number from 2.2.0-devel to 2.2.0-dev which apparently is
recognized by pip (thanks Jakob Schnitzer).
https://bitbucket.org/blais/beancount/pull-requests/91/default/diff#comment-85707662
2018-12-12
- Suppressed almost all the Python 3.8 warnings.
2018-11-16
- Added more logging in loader (done while debugging something else).
2018-10-26
- Fixed 302: Fixed second bug with font-locking for account names (starting
with digit).
- Fixed bug ingest that would cause relative filenames to fail.
See this thread for details:
https://groups.google.com/d/msgid/beancount/709141d5-88a4-444a-acef-94bf43a9b388%40googlegroups.com?utm_medium=email&utm_source=footer
- Fixed bug in ingest whereby the custom duplicates detection function wasn't
being called.
2018-10-25
- Fixed unit test failure on BitBucket pipelines using a temp directory
context manager.
- Applied PR by Richard W to install the mixins package (install was broken).
- Fixed 302: beancount.el does not fontify account names with single letter
components (based on bug report by arianaut).
2018-10-13
- Fixed 158: Missing MIME type for QFX files.
2018-10-04
- Merged new_ingest_mixins: Added new implementations of mixins for ingestion:
* beancount.ingest.importers.mixins.identifier which implements identify().
* beancount.ingest.importers.mixins.filing which implements file_account()
and file_name().
* beancount.ingest.importers.mixins.config which implement a simple schema
with a configuration.
These new mixins are intended to be more independent and more easily
composable than the older ones. The beancount.ingest.importers.regexp has
been removed. If you need it in interim, I made a copy under
experiments/ingest/regexp.py.
* Version: 2.1.3
2018-10-04
- Fixed 323: Escape strings in printer.
- Minor bug fix for beancount.prices.price on null result.
2018-09-19
- Removed TRUNCATE(), after realizing ROOT() can already do the same thing.
2018-09-18
- Added TRUNCATE() function to SQL shell for Vikas Rawal.
2018-09-03
- Added a parameter to the cmptest assertions to disable real booking and
allow a weak local form of booking conversion that allows interpolation.
(I'm using this in another project to assert some incomplete transactions.)
2018-08-30
- Made assertions on entries off of the TestCase class, as free functions (so
that they can be used in another project).
2018-08-26
- Fixed tests for pipeline running pytest (somehow the curses functions fail,
probably due to better environment isolation from pytest).
- Implementation of beancount.ingest.regression_pytest, a pytest replacement
for beancount.ingest.regression, which was for nose. This is simpler and
better and more straightforward (no conditionals; all regression files must
have all accompanying expect files and all must pass). See the module
docstring for new instructions on how to use this.
This should not break the older tests, as I haven't changed the older
support. It (beancount.ingest.regression) will be removed eventually.
(This completes porting Beancount to use pytest instead of nosetests.)
2018-08-25
- Today we're switching to pytest as the official test runner. All the tests
pass, and it even runs faster. It's about time!
- Removed all references and workaround to nose.
2018-08-23
- Oops... removing forgotten trace in bean-web.
- Fixed 325: Implemented support for including encrypted files (for Milind Kamble
and using some of this changes).
2018-08-17
- Added 'issuer' column to the export script. I'm going to use this to produce
a deaggregated list of exposure to individual stocks from a list of ETFs.
2018-08-25
- Upgrade requirement to bison-3.0.5 for development.
2018-08-12
- After many years of using something like it to track my investment portfolio
using a Google Sheets doc, I cleaned up, rewrote and move the asset list
exporting script to beancount.projects.export and deleted the old version
under experiments/upload.
2018-08-05
- Fixed 322: Adjustments to StopIteration treatment from within generators
were breaking bean-example generation code.
- Implemented a new plugin which asserts that on accounts with 'NONE' booking
method reducing postings' cost basis is within the vicinity of the average
cost basis in the account. This allows you to approximate the 'AVERAGE'
booking method avoid too much cost basis leakage until it's implemented
properly.
2018-08-02
- Created a new 'commodity_attr' plugin to asset the presence and validity
(in a set, like an enum) of metadata attributes on Commodity directives.
I export my holdings joined to a table of per-Commodity attributes and I've
made a mistake this week that ended up showing up as an overestimated
portfolio value on my portfolio tracking Google Sheets doc.
2018-07-31
- Fixed bug in new ingest() function found as I refactored my importers.
- Fixed bug with the pipeline making sure to set the CWD of the subprocess
when we run an import test.
- Fixed bug with previous commit containing stray 'string' reference. (My
environment has autoimports by default.)
- Implemented a beancount.ingest.scripts_utils.ingest() function that will act
as a main program, so you can write your own importer scripts from top to
bottom without having to rely on bean-identiy, bean-extract or bean-file.
The purpose of this change was to allow the user to insert an optional
argument 'detect_duplicates_func' in order to implement their own duplicates
detection. See discussion under
https://bitbucket.org/blais/beancount/pull-requests/75/. In order to install
your duplicates detector, just call ingest at the end of your importer
script, like this:
my_importers = [ ... ]
ingest(my_importers, my_duplicates_func)
I could have chosen to add a "hook" option and such to Beancount, but I
preferred moving the whole import framework one level closer to code. I also
like that it's not just an imported module; it's really just your script,
with three subcommands. You could add more custom import-level stuff in
there. The ingest() function is a main program, and commands like
bean-extract myimporters.py ~/Downloads --reverse -e myledger.beancount
are invoked similarly by calling the Python script directly, like this:
./myimporters.py -d ~/Downloads extract --reverse -e myledger.beancount
These two should be equivalent.
The older way of doing things--that is, calling bean-identify, bean-extract
and bean-file--are still supported regardless of whether you've inserted a
call to ingest() in your importers configuration or not. This is made to
work via a rather labyrinthine trampoline of functions calls (in particular,
getting the arguments setup right was surprisingly tricky).
I'm still debating whether this should be "the new way", or just for those
wishing to build their own duplicate detection heuristic. Feedback (on the
list) would be interesting.
2018-07-29
- Updated to flex-2.6.4.
2018-07-28
- Added a utility to import modules by dotted string name.
2018-07-19
* Fixed 319: Incorrect version number from 2.1.2-devel to 2.1.3-devel.
2018-07-28
- Added automated unit test for calling identify() from the importers test
suite, expecting True, always.
2018-05-31
- Created a new plugin: "fill_account" to automatically insert a posting with
a default account name where there's a single posting. Will use this while
traveling as a lot of the expenses come from cash; this should significantly
shorten such files.
- Changed template for version number from {date|short} to {date(date, "%s%)}
as per discussion on 304 (thanks to Jason Chu for looking up the detail for
this).
2018-05-29
- Refactored the call to find duplicates in imported files so that it can be
called only once with all the results from all the importers. This should
eventually allow a user-provided function that can merge entries between
importers. Existing behavior should be unaffected (though there may be a
small delay until everything gets printed out; all the importers are invoked
before any output is written). See 298.
2018-05-28
- Fixed bug with tempdir being a symlink on Mac, which was making one test
fail.
- Fixed unit tests which failed due to timestamps getting rendered with an
offset on a Mac while on a train in Japan.
- Render tags and links on Document directive.
- Minor bug fix: don't fail on rendering a MISSING instnace.
- In 'divert_expenses' plugin, allow limiting diversions to one posting only.
- Fixed unit test for git checkouts. This should fix the Travis-CI build.
* Version: 2.1.2
2018-05-12
- Fixed setup.py to catch exceptions in the absence of git as well.
* Version: 2.1.1
2018-05-12
- Fixed installation with missing version string.
* Version: 2.1.0
2018-05-12
- Added back lexer test from Adrian Medrano Calvo's UTF8 changes to head.
2018-05-08
- Fixed 287: In the price fetcher, store the datetime instances as
naive-in-UTC in order to avoid a serialization problem with older Python
versions.
2018-05-06
- Changes to support git versioning if hg versioning is not available.
2018-05-05
- Fixed 289: Add option to find beancount version. All scripts now support
--version.
- Amended PR64 to remove the need for __getitem__(). Clarified docstring.
- Merged PR64 from Jakob Schnitzer: Inventory: use dict instead of list as a
base.
2018-05-04
- Fixed 290: Validation of account name components was incorrectly swapped.
2018-05-03
- Amended PR74: Adding a new unit test, fail if tags / link appear after the
first posting. Performance amendment to tags/links and clarification to the
freezing of tags & links and their default values.
- Merged PR74 / Fixed 99: Allow tags and links to be spread over several
lines.
- Amended PR14: added simple unit testing for regexp, changes in parsing UTF-8
account type to match changes in Beancount head.
- Merged PR14 by Adrian Medrano Calvo: Support account names to consist of
Unicode letters and numbers.
2018-05-02
- Removed a few accidental __author__ metadatas from files, normalized to
__copyright__ as all other files.
2018-05-01
- Added patch for clean build for lexer. 2.6.1 results in a build warning.
- Merged PR73 (Ethan Glasser-Camp) - Cleanups to lexer and more restrictive
syntax for comments within a transaction's indented block.
- Fixed failing test re. new grammar.
- Merged PR29: Update grammar to allow comments in the middle of transactions
(Ethan Glasser-Camp).
- Merged PR71: Use enhanced extract method which passes existing entries
(Patrick Ruckstuhl).
2018-04-29
- Added polymorphic POSSIGN(_, account) function to the SQL shell in order to
support correcting for per-account sign.
- Implemented polymorphic NEG() function in the SQL shell, which works on
instances of Decimal, Amount, Position, Inventory.
- Changed the SQL shell's operators to dispatch to Python's default operator
selection instead of coercing to Decimal instances.
2018-04-28
- Added QUARTER() function to the SQL shell.
(See https://groups.google.com/d/msg/beancount/fTlSnQRPayo/1yN8zd76AAAJ.)
- Updated version of flex on Ubuntu (2.6.1) generates warnings. Fixed a few.
But Beancount now fails on account names with a dash in them; added a test
for those. (The lexer is not the issue; I suspect a bison update from 4/17.)
2018-04-18
- Fixed 157: Building a price map with zero prices should not fail,
especially when done accidentally.
- Fixed 264: Converted zip compression in bake from using an external zip
tool to using the 'zipfile' module packaged with Python. I'm doing this to
fix the CI build as well.
2018-04-17
- Fixed 265: Refined the timezone specification of the price sources. In
doing this, I documented it in the beancount.prices.source module and now
pass in a timezone-aware datetime instance instead of a date, and the source
is now required to also require a timezone-aware datetime instance. I fixed
all the tests, and added a utility to simulate running them from different
timezones.
- Implemented multi-file support for bean-doctor context command. Use it like
this:
bean-doctor context <toplevel.beancount> <file.beancount>:<lineno>
The previous syntax is still supported; like this:
bean-doctor context <toplevel.beancount> <lineno>
2018-04-05
- Since I added a dependency on 'requests', forgot to add to checkdeps. Done.
2018-04-04
- Handle error on an invalid implementation of a price source (e.g., in this
case an obsoleted price source string of 'google' imports something else and
referenceing .Source fails).
- Bubble up error from Yahoo price source.
- Fixed 267: Implemented a price source from the IEX exchange.
2018-04-03
- Implemented Quandl importer on its REST API. Another implementation, based
on the Quandl Python client API was proposed by Hugo Ideler (see PR34), but
I didn't want to add a dependency on the client library (nor on Pandas), and
after reading the description on Quandl API's website, I figured the library
was simple enough the output could be parsed directly. I wrote a new
implementation from scratch, parsing just the time-series datasets.
- Merged PR68 from Martin Michlmayr, added Travis Yaml support file.
- Fixed Emacs functionality to uncomment regions (change submitted by Stefan
Monnier on the list).
2018-04-02
- Fixed 193: Removed the broken Google Finance price source importer. Google
has removed this service in the favor of a special Search page and this has
been broken for the recent little while.
- Fixed 203: Updated the Yahoo price source implementation to use the
undocumented v7 and v8 APIs; moved away from the "ichart" and "download"
APIs. The Yahoo importer is back in function.
This introduced a new dependency on the awesome 'requests' library, which
really, was quite inevitable).
2018-04-01
- Merged PR65: Use parse_version to compare versions; then replicated
parse_version() locally to avoid the dependency on setuptools.
- Merged PR15 by Johannes Harms:
Added the following features to the forecasting plugin:
- Yearly repetition of recurring transactions (instead of just monthly)
- Users may optionally specify an end date
- Users may optionally specify the number of repetitions
"I used this code to calculate scenarios for a possible real estate purchase
and I contribute a cleaned-up version as an example file."
- Merged PR54 & Fixed 232: Align quoted commodities correctly in ledger price
statements (from Martin Michlmayr).
- Merged PR57: query_execute: Allow ordering by columns which may be None
(from Jakob Schnitzer). Made some adjustments to generalize on all
datatypes.
- Merged PR62: Also allow 0/0 in SAFEDIV() (from Jakob Schnitzer), with small
adjustments to catch only the specific invalid operations on Decimal.
- Merged PR64 from yagebu (Jakob Schnitzer), which makes the Inventory class
rely on the fact that Position is immutable in order to avoid unnecessary
copies. This yields an 18% speedup with no difference.
- Removed unused and invalid mutating method Position.set_units(). Amazing
obsolete stuff like this can still be found.
2018-03-31
- Fixed 10: "print from close_date" causes problems on account open. This was
stemming from a bug in the implementation of the FROM clause execution. I
changed the "entry" context functions to also accept a RowContext object, as
the simple functions may be called in either entry or posting context.
(The SQL shell remains the part of Beancount which is severely undertested
and wading into that code once again has convinced me I need to rewrite the
shell from scratch, it'll be simpler and better. The distinction between
entry and posting context will go away in the next version.)
- Fixed 120: Documents directive with a trailing slash fails.
2018-03-28
- Fixed failing test in amount_test probably due to some recent merges.
2018-03-27
- Completed PR 59 which adds support for globbing patterns in includes,
inserting a chdir() around glob.glob() to handle expansion of relative
filenames. (Credits to Martin Michlmayr for submitting the patch in the
first place.)
* Version: 2.0.0
2018-03-27
- Released 2.0.0. Today we're starting versioning Beancount. Many people have
been waiting for this in order to manage their dependencies (e.g. Fava
users).
We're loosely adopting semantic versioning (https://semver.org/) where bug
fixes will be posted as 0.0.x releases, and new features that don't break
existing functionality (nor change it too much) will be posted as 0.x.0
releases.
Note: There's nothing special about this particular point in time, other
than I haven't added much features in the last six months and have mainly
been fixing bugs and accepting bug reports in order to make sure things are
stable. There are plenty remaining bug reports and it's not perfect.
Addressing some of these will involve making changes that could break
existing functionality. For this reason, I prefer to release now and make
those changes as part of point releases later on.
2018-03-26
- Fixed 245: Synchronized the definition of a valid account regular
expression between the lexer and the later enhanced version (in Python)
which checks the configured prefixes. A valid subaccount name may start with
either a capital letter or a number.
2018-03-23
- Removed deprecated methods from beancount.core (position, inventory,
interpolation).
2018-03-22
- Fixed 249: Error messages involving postings should start counting at 1
instead of 0. (I agree.)
- Added aliases for COMMODITY_META() vs. CURRENCY_META() and CURRENCY() and
COMMODITY(). These terms are used interchangeably. In the next version of
the shell, only one of them will be available.
- Removed the ability to explicitly set a different price currency than a cost
currency. I'm stunned my test for this is gone, I must have removed this by
accident? No idea. It should not be allowed.
2018-03-19
- Fixed 218, added a SAFEDIV function to the shell language.
- Fixed 239, HELP command should tolerate semicolons and whitespace.
(Also, don't print shell commands in test (remove docstrings), we want the
name of the tests to list, not the input.)
2018-03-13
- Removed SIMPLE booking method and all traces of it, including the
'booking_algorithm' option. Only the later (I hesitate to say "newer"
because it's been around for years) FULL booking method is supported
onwards.
- Fixed 35: Implemented implicit group-by syntax. That is, if you omit the
GROUP BY clause, it is automatically inferred to be the non-aggregate
columns. This make writing the type of simple queries we do much more
convenient and allows you to be a little sloppy.
- Fixed bug with unit test util for docutil, which was stripping the first
empty line of the docstring (and failing).
2018-03-02
- Made the 'readline' library conditional, for Windows support.
See PR46.
2018-02-23
- Fixed 219: Restored sorting of the entries in the extractor. File order
will have to be maintained by extracting the line no, which is used as a
secondary key.
2018-02-19
- Added prototype for converting Google Docs doc from docx and rst
conversions, using the docx blocks to insert indented versions in the rst
file.
- Removed deprecated beancount.ops.prices module.
2018-02-17
- Added an option to the CSV importer for the string separating the narration
(default: "; ").
- Merged a change submitted by droogmic to add options for the dateutils
parser, in order to disambiguate between day-first (DD-MM-YYYY) vs.
month-first (MM-DD-YYYY) date representations.
- Created a new plugin "check_closing" that automatically inserts a zero
balance check after postings which are known - and flagged, using metadata -
to be closing trades.
- Fixed bug in "sellgains" plugin whereby a price of zero units failed an
assertion. Added a test.
2018-02-12
- Added a side-effects local variable saver utility for writing importers.
- Fixed 212: Clarified error message for error in covering non-aggregate
targets.
- The Importer.extract() method now accepts a new parameters with the prior
entries (or None, if not specified) as 'existing_entries'. It's free to use
that as it pleases in order to flag entries as duplicates. The entries
returned by Importer.extract() will be checked for __duplicate__ metadata
and automatically inserted to that set if it is present. This allows the
importer to return some duplicate entries for context - which will be
rendered as such in the output, e.g. commented out - without having to
necessarily throw them away.
(The previous signature is still supported in order to avoid breaking
everyone's importers.)
2018-01-31
- Modified 'divert_expenses' plugin to store the original account name as
metadata of diverted postings, so that they can be picked up by the
extract_tagged script and brought back into the extract file.
2018-01-29
- Created a new 'divert_expenses' plugin to redirect Expenses postings from
transactions tagged with a particular tag to another account. See this
thread for details:
https://docs.google.com/drawings/d/18fTrrGlmz0jFbfcGGHTffbdRwbmST8r9_3O26Dd1Xww/edit?usp=sharing
- Added account_types.is_account_type() function, which should be used above
account_types.get_account_type().
2018-01-23
- Removed the insertion of 'balance' metadata from the importer. We use it for
inserting Balance entries, not attach as metadata.
- Disabled Balance insertion CSV test;
03d9a5c3b2784a6f6a05c12eaeb7776f0478a96d broke the insertion of Balance
entries without inserting 'balance' metadata. We'll probably have to revert the
sorting changes and that. For now, fix the test.
- Fixed 210: Changed the bean-price option --undeclared to accept a string
value, the name of the Python source module to use for fetching undeclared
commodities.
- Added an error on using explicit merge cost syntax (this one: {..., *}).
It's not supported yet, like AVERAGE cost booking (I probably should not
have enabled the syntax until it was done. I was hopeful and moving fast at
the time).
2018-01-16
- CSV importer: Added an option not to add balance metadata to output.
- CSV importer: Also, fixed a bug I introduced in the recent PR merge with
rendering the name of the importer.
- SQL importer: Made "location" field support transactions with metadata with
missing filename and line no. Some plugins might be doing that.
2018-01-08
- Fixed 209: Don't require 'ido. User will have to make sure that
beancount-use-ido is false.
2018-01-07
- Merged https://bitbucket.org/blais/beancount/pull-requests/28/allow-one-character-top-level-account,
allowing one character top-level account names in the syntax. This doesn't
mess with the flag token generateion because the account name is required to
have at least two components.
2017-12-09
- Fixed rendering of price directives to use the display context precision.
2017-11-24
- Silenced annoying warning about file_cache version incompatibility in
upload-to-sheets.
2017-10-28
- Fixed bug with identification based on the first line of a file.
- Fixed bug with 'context' on a directive that's not a transaction.
2017-09-30
- Fixed a bug in reduction booking that would fail when the units are omitted.
2017-09-17
- Fixed bug in Balance check, detect when an account does not exist and handle
it properly.
- Added bean-difference script, used to reconcile transactions in multiple
files after a long trip.
2017-09-06
- Added ABS() implementation to SQL shell for Position and Inventory types.
See https://groups.google.com/d/msg/beancount/0i4w98LmS6E/JJGOKSB8CQAJ
2017-08-08
- Accept account name components as numbers only. The grammar safely supports
this because of the leading semi-colon, all the tests pass, and everything
runs on my own very large input files.
- Fixed a failing test with MIME mis-categorization which occurred under
Ubuntu.
2017-07-25
- Fixed 181: Support underscores in Python module names for price sources
(thanks to Stephan Mueller for reporting).
2017-07-23
- Fixed 179: Scientific notation should never appear in rendered output.
2017-07-02
- Fixed minor bug with rendering postings with missing numbers.
2017-07-06
- More changes on 173: use a stringify two-level macro in order to place the
parser source hash as a string in the source file. Hopefully this should
also work under MSDEV.
2017-07-05
- Fixed 173: Added a potential fix for building under MSDEV. Thanks to Alan
Ray for pointing out a method to build with MSDEV under Windows.
2017-06-25
- Fixed a minor bug in the tolerance inference code that wouldn't accept a
posting with a missing number only, e.g. Assets:Checking USD 1.32 CAD.
- Added a "--no-colon" option to both bean-web and bean-bake in order to
support Windows filesystems which do not allow colons in filenames.
2017-06-22
- Added --numberify option to bean-query, which for CSV output will remove the
currencies of any Amount column. This is great for then uploading to a
spreadsheet.
2017-05-28
- Merged patch from Felix Eckhofer that adds an option to bean-format to make
it use a fixed width. I've been wanting this myself for a while actually, I
think a fixed value makes more sense than what I'd done previously and
should eventually be made the default.
2017-05-24
- Implemented beancount.plugins.auto and beancount.plugins.pedantic. I've been
wanting to do this for a while. When writing quick-and-dirty little tests,
inserting 'plugin "beancount.plugins.auto"' is quick and does the job, and
if you want absolutely all the verifications and to make Beancount as
pedantic as possible, you should be able to adopt 'plugin
"beancount.plugins.pedantic"'.
In order to do that I had to make the loader support plugin definitions
whose __plugins__ attribute is a function reference, not just a string. It
supports both now.
- A number like this now fails to parse: 1,245,449,72.00 USD. I added a test
in the lexer to tighten the parser. This has been brought up before and will
potentially avoid input errors.
2017-05-20
- Clarified error message for price format specified on the command-line.
2017-05-08
- Made the open/close map return by the getters not a defaultdict by default,
so that accidentally looking up elements won't mutate it.
2017-04-30
- Moved src/python/beancount/... to beancount/...
The purpose is to structure this project in the most common way
people expect it (most Python OSS projects have this structure, with the
package name directory at the root). I've had this other structure for a
while mainly for historical reasons. I wanted to make this clean before
baking a final 2.0 release and switching to numbered releases.
- Moved src/elisp/beancount.el to editors/emacs/beancount.el and made
necessary adjustments.
- Merged incomplete changes from booking branch to "default." The work on
supporting self-reduction (see
http://furius.ca/benacount/doc/self-reductions) won't be carried during 2.0,
it will be subject of a subsequent release.
- This concludes the merging of all the branches, which I've done in order to
move the source to a more regular project directory (under /beancount).
2017-04-29
- Fixed 27: IN membership operator did not have a proper precedence definition.
2017-03-19
- Fixed minor bug with CSV importer that would occur when the file has no
valid columns.
2017-01-26
- (internal) Moved the implementation of booking methods to its own
beancount.parser.booking_methods module.
- (internal) Split up the booking methods to functions for exhaustive testing
and allowing others to implement new methods more easily if desired.
2017-01-22
- Added new price source fetcher from OANDA's API. It supports a large number
of currency pairs with tickers like this "XXX_YYY", such as "EUR_USD". Use
it like the other price sources, e.g., like this:
bean-price -e 'USD:oanda/EUR_USD'
- Began working on the conversion of Google Docs to reStructuredText for
inclusion in Dominik Aumayr's docs.
2017-01-21
- Fixed 155: Use entry_points instead of scripts in setup.py.
- Added an extra check for Balance directives, verifying that asserted
currencies are accepted currencies for that account (if the account's
corresponding currencies have been declared explicitly). In other words,
this will now fail instead of being silently ignored:
2017-01-21 open Assets:Checking CAD
2017-01-21 balance Assets:Checking 0.00 USD
- Fixed bug in sheets_upload.py / upload-to-sheets whereby all the fields of a
sheet being updated just in order to resize it was causing the sheet to be
reindexed at the front of the list of sheets.
- Merged 'arithmetic_ops' branch with no effect, in order to move all files
and release. This will have to be revisited later.
2017-01-19
- Fixed bug with generation of column names in sheets uploader script.
2017-01-16
- Removed tools/gauth (used to be beancount.docs.gauth). All users of this are
slightly different, and they're utility scripts. Unfactoring.
- Removed beancount.docs from Beancount library. These are internal tools used
to manage documentation, should not be included in the installed codebase.
- Some cleanup of internal tools.
- Added some basic unit tetsts for beancount.tools.sheets_upload.
- Moved bin/upload-to-sheets to src/python/beancount/tools/sheets_upload.py
and replaced it with an associated stub for it.
- Moved bin/treeify to src/python/beancount/tools/treeify.py. The script is
still usable in a standalone form (no new dependencies have been added). To
have all the source code in one place is more convenient for maintenance.
- Created a beancount.tools package to contain all standalone tools which
don't depend on Beancount modules or packages.
- Removed branch "returns2" and moved associated projects script
beancount.projects.returns to experiments/returns. This code isn't robust
enough to work for everyone, and so therefore I'm moving it back to
experiments until I have time to produce something that will work very
robustly, and on all my own accounts as well.
2017-01-15
(from branch "conversions")
- Created and test beancount.core.convert Posting and Position conversion
functions: get_units(), get_cost(), get_weight(), get_value() (for market
value).
- Implemented Inventory.reduce() and associated test, which works in
conjunction with those.
- Implemented more straightforward versions of convert_position() and
convert_amount() in module beancount.core.convert, to convert positions and
amounts to other currencies.
- Deprecated Inventory.units(); use Inventory.reduce(convert.get_units)
- Deprecated Inventory.cost(); use Inventory.reduce(convert.get_cost)
- Deprecated Inventory.get_units() to Inventory.get_currency_units() in order to
avoid future name confusion.
- Deprecated Position.at_cost(); use convert.get_cost(position) instead.
- Deprecated Position.get_cost(); use convert.get_cost(position) instead.
- Deprecated b.c.interpolation.compute_cost_basis(); use
Inventory.(pos for pos in positions if pos.cost is not
None).reduce(convert.get_cost)
instead.
- Deprecated get_posting_weight(); use convert.get_weight() instead.
- Removed b.ops.holdings.get_pholding_market_value(), which was unused.
If you need a market-value function, use convert.get_value() instead.
- Converted all the old code to these new converted functions.
- Moved module beancount.ops.prices to beancount.core.prices, and added a
deprecation warning.
- Deprecated shell function PRICE(). Renamed it to GETPRICE() in order to
avoid confusion.
- Removed beancount.core.prices.convert_amount(). Use
beancount.core.convert.convert_amount() instead. Note that a failure doesn't
return None anymore, but an unmodified amount. Also notice the reordering of
parameters to be consistent with the other functions in that module.
- Removed beancount.core.prices.get_position_market_value();
use convert.get_value() instead.
- Removed beancount.core.prices.get_inventory_market_value();
use Inventory.reduce(convert.get_value, price_map) instead.
- Removed private functions
beancount.query.query_env.convert_amount(),
beancount.query.query_env.convert_position(), and
beancount.query.query_env.value_position(),
beancount.query.query_env.value_inventory().
These used to issue warnings if the conversion could not be carried out; now
they simply keeps the units of the position instead.
2017-01-15
- Fixed all tests for Python-3.6.
Beancount has not officially been ported to Python-3.6.
2017-01-14
- Fixed bug with generation of example that would produce an incorrect initial
balance for banking accounts. The example test should have been broken. (No
idea why this wasn't deteted earlier, probably by chance.)
2017-01-08
- Completed "Intro to Double-Entry Accounting" doc.
http://furius.ca/beancount/doc/intro
- Added a --reverse/--descending flag to bean-extract to print the extracted
entries in reverse order.
2016-12-17
- Added a --title option to upload-to-sheets in order to conveniently set the
document's title.
- Removed obsolete variable LEGACY_DEFAULT_TOLERANCES which had been
forgotten.
2016-12-15
- Made beancount.ingest.extract print out a stack trace when an unexpected
exception is raised.
- Fixed bugs in beancount.query.query_render where expand wasn't being passed
through the rendering functions.
- Deprecated bin/upload-csv-to-google-sheet; I moved the last version to
beancount/experiments/gapis in case you've come to be dependent on its
particularities. It doesn't need any of the Beancount libraries, so you can
just copy it somewhere and keep using it if so desired.
- Replaced "bin/upload-csv-to-google-sheet" to "bin/upload-to-sheets". The
newer script is a complete rewrite which does not require the antiquated
gdata library, and which uses the latest and greatest version of the Google
Sheets API (v4). You don't need gdata anymore. Moreover, this one runs in
Python3 instead of depending on Python2.
The script has slightly improved invocation semantics, and I'll using it
regularly to share CSV files with others, most often CSV files produced by
bean-query.
2016-12-11
- Changed b.p.split_expenses outputs to be more useful.
2016-12-10
- Made Amount, Posting, Cost and CostSpec all immutable classes that inherit
from typing.NamedTuple.
- Removed mutable method Posting.add().
2016-12-05
- Merged changes made for calculating wash sales a long time ago. I had
implemnted a temporary kludge to carry dates on all augmenting postings,
left it in a branch so I could use it there for taxes every year. Now that
the booking branch is merged, this is supported by default, so I could undo
that kludge. Some changes remained:
* The Inventory.add_amount() method used to return the modified position.
Now it returns the position before being modified, which is more useful.
* The list-wash-sales.py script has been updated. It now works on 'default'.
* The 'carry_date_and_book_cost' branch has been merged away.
2016-12-04
- Made .tags and .links attributes of b.data.Transaction and b.data.Document
directives ensured to always be a set when they're empty. If there are no
tags (or links), those attributes are set to b.data.EMPTY_SET, whose value
is a frozenset(). This should make the processing of those attributes a bit
simpler.
Note that if you create transactions from plugins and their tags and/or
links attributes are set None, they will now fail. The right thing to do is
to set their value to an empty set, or reuse the immutable
beancount.core.data.EMPTY_SET object.
If you need to circumvent this temporarily, set the following option in your
file, which will allow None again:
option "allow_deprecated_none_for_tags_and_links" "TRUE"
See {1989ba9e0e79} for details.
- Updated all copyright and license notices on all files.
- Added a script to automatically verify that all source files include a
copyright notice.
- Removed ancestral Beancount documentation from eons ago. These notes and
conversationsa are now wildly irrelevant to the current state of Beancount,
and all the documentation from Google Docs is vastly more recent and
obsoletes all these old snippets and notes.
- Moved all development tools under tools/, from etc/ and
experiments/code-health/.
2016-12-03
- Fixed fava bug 429, avoid empty conversion entries if the cost of the
conversion balance is zero. See:
https://github.com/beancount/fava/issues/429.
2016-11-20
- Fixed bugs in bean-format; in order to do this properly I made bean-format
reuse the globally defined regular expressions for currency and account in
the rest of Beancount's code. This makes bean-format depend on the Beancount
installation. Bah, it's reasonable. This issue is related to issue 146.
- Added ANY_META() function to the SQL shell's environment; ANY_META() allows
you to query a posting's metadata, falling back on the parent entry's
metadata. This idea was suggested by Stefano Zacchiroli in
https://groups.google.com/d/msg/beancount/jh-n4rvbQAI/Hz5Mmd60BAAJ. I also
added the first unit tests for environment functions--I'll have to write all
of those at some point.
- Made unsupported cost specifications in the SIMPLE booking algorithm issue
an appropriate error. For example, instead of just ignoring the cost spec in
this:
2014-10-15 * "buy widgets"
Assets:Inventory 10 WIDGET {} ;; Not supported.
Assets:Cash -80 GBP
It will issue an error. See
https://groups.google.com/d/msg/beancount/9NBcT-SXZMQ/sy7Z3dIdBQAJ for
context.
2016-11-12
- Made the Emacs prefix key configurable via a new beancount-mode-map-prefix
variable. See discussion here:
https://groups.google.com/d/msg/beancount/_7TscZ9phNE/K8PkcRMpBAAJ
2016-11-06
- Fixed 139: Full booking parsing had been more liberal than the old SIMPLE
booking algorithm can interpret. In the old code, explicitly disallow some
interpolation cases which cannot be handled.
- Fixed 129: Ignore custom directives for Ledger and HLedger conversion.
2016-11-05
- Fixed 142: bean-check should return non-zero exit code upon failure.
2016-10-30
- Merge the current version of the 'booking' branch and made the "FULL" booking
algorithm the default one.
* The "experiment_booking_algorithm" option has been renamed to
"booking_algorithm". Its default value is now "FULL"; everyone will be
using the new booking code. If for some reason you have a transitional
need to revert to the older booking method use this option.
option "booking_algorithm" "SIMPLE"
* The new booking algorithm is _MUCH_ more powerful than the one old. I'll
describe it soon in a new document, but briefly:
- It supports many more interpolation capabilities; try omitting amounts
or currencies which you think it ought be able to figure out
automatically and you might find it does.
- All positions carry a date. This means that if you have transactions
that would merge because they weren't specified with a date, like this
you may face a problem:
2016-01-15 *
Assets:Investments 70 UUG {10.00 USD}
...
2016-02-16 *
Assets:Investments 25 UUG {10.00 USD}
...
2016-03-30 *
Assets:Investments -80 UUG {10.00 USD}
...
Those used to resolve automatically because both augmenting lots would
merge into a single one without a date. Now that all positions have an
acquisition date, the -80 reducing posting is ambiguous, because it
matches against these:
Assets:Investments 70 UUG {10.00 USD, 2016-01-15}
Assets:Investments 25 UUG {10.00 USD, 2016-02-16}
Either you'll have to disambiguate, or let Beancount do so. One way to
do this is to add more specific matching data, and split the posting.
An even easier way to do this is to make the default booking method of
this account to "FIFO" or "LIFO", which will automatically select the
oldest (newest) postings without issuing a warning.
If you find yourself in a bind, please email the mailing-list with a
specific and as small as possible data input that reproduces the
problem.
- Removed parsing of the pipe (|) symbol as a separator for strings on
transaction description lines. This was vestigial from the days long long
ago when Beancount was attempting to be compatible with Ledger syntax; this
is wholly unnecessary how.
You will have to convert transactions like this:
2015-08-25 * "Tunnel" | "Coffees"
Expenses:Food:Coffee 15 CAD
Assets:Cash
Into transactions like this:
2015-08-25 * "Tunnel" "Coffees"
Expenses:Food:Coffee 15 CAD
Assets:Cash
This can be done using a sed script, or as a macro in your favorite text editor.
If you need a temporary respite, you can use the following option, which
will replace all the error messages by a single warning:
option "allow_pipe_separator" "TRUE"
Support for this option will be removed eventually.
- Moved the {{...}}} total cost syntax from a legacy support with only "/"
separator and a date, no label, to full support. Previously, only these syntaxes
were supported for total cost:
Assets:Account 100 MSFT {{2000.00 USD}}
Assets:Account 100 MSFT {{2000.00 USD / 2016-10-30}}
Changes:
* The "/" separator support has been removed, you must now use a comma, like
for the {...} full booking syntax.
* The same syntax as for the regular {...} compound cost is supported;
however, specifying a compound amount will raise an error, and the
per-unit cost is interpreted as a total cost.
For example, this:
Assets:Account 100 MSFT {{2000.00 USD}}
is interpreted as:
Assets:Account 100 MSFT {0 2000.00 USD}
which is equivalent to:
Assets:Account 100 MSFT {20.00 USD}
And the following will raise an error, it is not legal syntax:
Assets:Account 100 MSFT {{0 2000.00 USD}}
Finally, specifying a date and a label can be done, just like for {...}:
Assets:Account 100 MSFT {{2000.00 USD, 2011-10-30, "ref32"}}
The intention here is to deprecate the older limited syntax while keeping
support for total cost input using {{...}}. Based on previous questions, I
suspect that many users are still using {{...}} instead of {{ ...}}, so I
won't remove it just yet.
- Removed very old compatibility code in beancount.core.amount that lived
since I cleaned up the definition of the D() constructor and moved it to
beancount.core.number.
- Removed built-in aliases for auto_accounts and implicit_prices modules, from
beancount.ops to beancount.plugins.
- Removed deprecated option "default_tolerance" (which had been renamed to
"inferred_tolerance_default").
- Removed deprecated option "tolerance".
- Remove deprecated use of 'plugin' as an option. The directive ought to be
the only way.
- Removed deprecated option "experiment_explicit_tolerances".
- Renamed "experiment_booking_algorithm" to "booking_algorithm", and changed
the default value to "FULL". The old "SIMPLE" algorithm will be deprecated
eventually.
- Removed old code that would warn if the user set account_rounding to a full
account name; it just needs be a subaccount.
- Commit to Python 3.5. 3.5.0 released more than a year ago. You need to
update or stick with the 2.0b13 release.
- Remove old code to dispatch between an external Enum class and the built-in
one that's included in 3.5 (commit to 3.5).
- Added type annotations to beancount.core.data. This will grow more over
time. (I also need to add rules to use those.)
2016-10-23
- Modified parsing of transaction strings and links and tags. I did this to
factor out the parsing of tags and links in order to add it to other
transactions and minimize grammar code. There is user-visible impact to this
change: all of a transaction's strings must appear before tags or links. You
cannot place tags or links before or in-between strings anymore. This is a
reasonable restriction and I doubt anybody did anyway.
- Added support for tags and links to the Document directive. This works just
as for Transactions, e.g.
2016-10-23 document Assets:Checking "/statements/oct.pdf" banking ^9fed846348c2
The current tags on the tag stack are also included, like for transactions.
2016-10-10
- Removed "experiment_explicit_tolerances" flag; made this supported by
default permanently.
2016-10-08
- Documented a subtle problem reported about balances against inventory of
mixed at-cost and no-cost, added more context to the error and a unit test
for balance checks with mixed at-cost and no-cost positions.
- Fixed a bug in the full booking matching code whereby an existing position
not held at cost would trip up the matching code. (Credits to Son Nguyen for
finding and reporting.)
- During ingest, when multiple extractors match a particular file, select the
first matching account instead of raising an error.
- Fixed accidentally colliding key-binding for invoking bean-price from Emacs,
to C-c p.
2016-09-25
- Fixed typo in grammar that was parsing a transaction flag as '' into '*'.
- Added "cost_date" and "cost_label" columns in the SQL shell.
2016-09-04
- Implemented new "run" shell command that allows you to run a named query.
For example, if an input file has a Query directive like this:
2016-09-01 query "home" "select account, sum(position) where account ~ 'Home' group by 1"
And you input this command:
run home
The entries will be closed at tue query date (2016-09-01) and the query
automatically run. If you want to always include all the transactions, set a
date far into the future.
This works from the command-line and from the shell, equally. You can also
run all the queries in an input file, like this:
run *
To view the list of available queries, just type the "run" command without
an argument.
2016-08-21
- Fixed a SQL shell bug with extracting UNITS() from a Position column.
- Finally properly fixed the timezone of the Google Finance price fetcher.
(It was done incorrectly and I found my unit tests failing whenever I would
work on Beancount from another timezone.)
2016-08-17
- Added beancount.core.amount.abs() function for absolute values of Amount.
- Added rendering of links to beancount.plugins.split_expenses plugin, and
made it quiet by default. Added --output-stdout option to make it do that
again.
- Created a new experimental script that attempts to match postings between
two ledgers. A pair of (filename, account-regexp) is provided for each, and
then the amounts and links are compared and paired up, and remaining
unmatched postings are printed out. This is very useful to reconcile trip
expenses between a personal ledger and the ledger for a trip or external
project where some of the expenses are paid from the personal ledger as
contributions to the trip or project.
- Made the upload-csv-to-google-sheet script use the basename (without
extension) of the given filenames as sheet names instead of "Sheet X"
numbers. (Note: This entire thing be rewritten using the new Google Sheets
API which won't require gdata and which will be more stable.)
- Added a quick implementation of b.q.query_render.render_csv()
and refactored b.q.query_render.render_text() to use common code. Both
functions now have a new 'expand' option which will create multiple rows for
cells which return a list of strings. The new default behavior is to
concatenate the strings with commas.
The shell (bean-query) has a new 'expand' shell variable to match. Also, the
-f/--format option of it is now supported, and in interactive mode it sets
the default value for 'expand'.
- Since output to CSV is now implemented, I enabled the --output-csv option of
b.p.split_expenses as well.
2016-08-14
- Fixed bug in beancount.utils.file_type for Python3, whereby magic now returns
bytes, not a string
2016-08-04
- Made account name components allowed to have two characters (beyond the
first component). For instance, Assets:Investments:F is now a valid account
name.
2016-07-11
- Added parsing a transaction date column to metadata in the CSV importer.
2016-07-01
- bean-format now accepts input from stdin.
2016-06-30
- Make it possible for a BALANCES statement to have a WHERE clause. BALANCES
is now equivalent to
SELECT account, sum(position) FROM ... WHERE ... GROUP BY account
Inching one step closer to removing that WHERE clause eventually.
2016-06-18
- Added support for columns AMOUNT_DEBIT and AMOUNT_CREDIT in the csv
importer.
- Merged ongoing progress from 'booking' branch; if you don't use any of the
in-progress new booking methods, this should not have any effect.
However, the 'booking_method has changed names and you may have to adjust
its values if you've been tinkering with it:
* 'booking_method' used to take on value "SIMPLE" or "FULL" and dispatch
betwee the current (old) booking implementation and the new one which will
support many other booking methods. This option is now named
'experiment_booking_algorithm' (a new option) and takes on the same
values. The default value is "SIMPLE", which keeps the behavior unchanged.
Use "FULL" if you want to test out the new booking algorithm: the new
algorithms should work except for the AVERAGE cost method. WARNING: This
is still under flux and considered unreleased.
" 'booking_method': This is a different option now, and takes the value of
any one of the booking methods visible here:
https://bitbucket.org/blais/beancount/src/7b9e00e6e19136f680befc1d4322f47cdc294cf6/src/python/beancount/core/data.py?at=booking&fileviewer=file-view-default#data.py-64
2016-06-13
- Ported beancount.docs.download_docs to the v3 drive API and the use of a
service account instead of a full access to a user account. Service accounts
are easier to configure and work with.
- Ported beancount.docs.upload_options to the v3 drive API.
2016-06-05
- Added a new column to the posting context: "other_accounts" is a set of
account strings for the other postings of the transaction.
- Added a new JOINSTR() generic function to the shell, to reduce the repeated
column above to a single comma-separated string.
2016-05-28
- Made various changes to the shell in order to make it possible to convert to
the market value:
* CONVERT() has been converted to convert to the market value, when applied
to a position held at cost. However, it still requires a target currency
and it may fail if you apply it to a position for which the cost currency
differs from that of the target currency, i.e., it will not automatically
chase down the transitive conversions toward the desired target currency.
* A new function VALUE() has been added, which is similar to CONVERT() but
which when applied to a position held at cost converts to the cost
currency at the current price. It's similar to CONVERT() but you don't
have to provide the target currency. And similarly to CONVERT(), you may
provide a particular date, and if at that date there is no available
conversion rate in the prices database, the conversion will fail
analogously and spit out an error message.
* PRICE() has been renamed to GETPRICE() to avoid some confusion. PRICE()
does not convert, it merely gets the price from the price database.
GETPRICE() is thus a more appropriate name.
2016-05-22
- Created the beancount.plugins.mark_unverified plugin as a response to this
thread: https://groups.google.com/d/msg/beancount/hQoOSJE7qNo/-E__v-XhEgAJ
This plugin makes it possible to compute the balances as of the last Balance
directive of each account by adding metadata to postings which appear after
that account's last balance. You'd filter out unverified postings with an SQL
query on meta-data like this:
SELECT account, sum(position)
WHERE NOT meta('unverified')
GROUP BY 1
ORDER BY 1
Note that the resulting balanaces almost certainly don't sum up to zero,
because of differing dates of the Balance directives. Also, note that
accounts without a single Balance directive will not have their postings
marked at all.
2016-05-08
- In the output of bean-price, sort prices exclusively by currency, regardless
of date. This is because on weekends you'll tend to get slightly varying
dates, and manual/occasional price updates are usually put in "chunks" in a
file section. It's easier to visually find the price by scanning by currency
order.
2016-05-05
- Added a shell function to filter a set of strings (for tags or links) with a
regular expression and return the first one. This way you can extract some
of the tags or links with a pattern as columns.
2016-05-01
- Fixed issue 124, a leak in the grammar parser which would leak the entire
set of entries on every instance of parse.
- Fixed some warnings about unclosed files which occur only on default python
branch.
2016-04-29
- Don't just log an extraction error to logging.error() in the regression
tests; let the exception through. This makes building and debugging
importers much easier.
- Created an experimental script to extract date/location information required
for a naturalization application.
2016-04-24
- Rewrote the section on "Core Data Structures" of the Design Doc, to reflect
the newer internals of Beancount, those which have been merged from the
ongoing 'booking' branch.
See http://furius.ca/beancount/doc/design-doc.
2016-04-23
- Added beancount.ingest.importers.fileonly, a simplistic importer which can
be instantiated to just file away some downloads without extracting anything
from them.
2016-04-18
- In bean-doctor context, all the accounts should appear in the
post-transaction balances, even if their balance is empty. This had been
fixed for the pre-transaction balances but I stumbled upon a case of
post-transaction balances not showing an empty account.
2016-04-17
- Added support for the BEANCOUNT_LOAD_CACHE_FILENAME environment variable,
which can be set in order to overridde the location of the pickle cache.
This variable may contain the string "{filename}", and if it is present the
basename of the beancount file will be spliced in. This can be useful if,
for example, you are reading your Beancount input file from a read-only
filesystem and still want to use the cache by specifying an alternative
filename.
- Fixed a bug when reading from a read-only filesystem, the loader cache would
fail to be removed and raise an exception.
- Fixed an important bug in beancount.ingest whereby instances of
b.i.cache.FileMemo weren't being shared as widely as they could have been.
FileMemo instances now have to be created via b.i.cache.get_file() and
FileMemo is now a hidden class _FileMemo. See this discussion for more details:
https://groups.google.com/d/msg/beancount/b-ZqYqag7AU/N-AaGSZOAwAJ
2016-04-10
- Create a new 'beancount.plugins.fix_payees' plugin that allows the user to
provide a set of rules to rewrite and clean up the names of the payees from
the Beancount file.
While it would be ideal to do this at import time, in practice there are a
lot of downloaded files' "memo" fields which get set as the payee
automatically. These "memos" aren't very clean and readable strings: they
sometimes contain numbers, partial city names, and other junk. In order to
produce clean reports where one might want to aggregate by payee, it's
useful to rewrite the payees to make them nice and clean. You can use this
plugin to do that.
- Made the Custom directive accept ACCOUNT tokens. You can now do something
like this:
2014-06-20 custom "budget" Assets:Account2 "balance < 200.00 USD"
Prior to this, the account name had to be represented as a string. (Users of
the burgeoning Custom directive not like this very much.)
Note that account names and regular strings are both output from the parser
as 'str' objects, so in order to make this possible, I had to change the
representation of the values to a pair of (value, dtype). Account names have
a 'dtype' of beancount.core.account.TYPE. (Eventually I will probably want
to make account strings represented with their own custom datatypes, but
that will require a fair amount of work on the codebase; this substitute is
fine for now, as there are no other places where such a disambiguation is
necessary.)
2016-04-04
- Added a test to make sure that building a source dist includes the header
files and C implementation files.
- Sort Document directives after Transactions when they occur on the same day.
The reason for this is that statements typically include transactions on
that final day, and we'd like for them to occur at the right place in the
journals. See originating thread:
https://groups.google.com/d/msg/beancount/3ItOYS9DJx4/VyTJB3ssBQAJ
2016-04-03
- Fixed a bug in "bean-doctor context" which wouldn't report the full set of
accounts when an account's before-balance would be empty.
- Fixed a bug in "bean-doctor linked" whereby the balances wouldn't get
printed out if there was no link. Print the balances before and after; in
fact, maybe this command should be renamed to "balances" and have an option
to support links.
2016-03-31
- Added TODAY() function to extract today's date. Not sure how useful it'll
be.
2016-03-27
- Added a PRICE() function to extract entries from the price database. You can
now query for "SELECT PRICE('USD', 'CAD', date)", for example.
- Added a boolean renderer for the SQL shell.
- Added support for arithmetic operations to the bean-query SQL shell.
2016-03-26
- Added an optional date argument to convert, so you can provide the requested
date in a SQL query, as in "CONVERT(sum(position), "CAD", 2015-06-01)".
- Improved the rendering of the context, removed the weird comments it used to
render with, render an account name if the inventory is empty.
- Ignore .DS_Store files from the file finder in ingest.
2016-03-22
- Support multiple accounts in same ofx file. See
https://bitbucket.org/blais/beancount/pull-requests/10/. I applied this
one-liner: issue a balance for each statement, use the maximum date for the
file_date() method, and added unit tests for this.
2016-03-21
- Fixed bug in beancount.ingest.file whereby two files being filed to the same
destination filename would clobber each other; with this fix, instead, an
error will be generated.
- Create an example of an importer just filing a PDF file using the external
tool PDFminer2. If the tool isn't installed, nothing should "break" (but the
importer won't really do its job). The tests should be skipped
automatically, and the importing will just ignore the PDF file to be filed.
2016-03-20
- Implemented beancount.ingest.importers.regexp and
beancount.ingest.importers.config mixin classes to help support
functionality which used to be available in LedgerHub.
- Removed the temporary beancount.ingest.importers.compat
backwards-compatibility helper for LedgerHub. Not needed.
- Completed the documentation for the new import framework "beancount.ingest".
The document is here:
http://furius.ca/beancount/doc/ingest
2016-03-19
- Centralized the <3.4 and >=3.4 support for Enum class into
beancount.utils.misc_utils.Enum.
- Added an option to the OFX importer to let the instantiator decides whether
a balance directive should not be inserted (NONE), should be inserted at the
declared date of a <ledgerbal> tag (DECLARED), or right after the last entry
imported from the file (LAST).
2016-03-18
- Merged 'web_months' branch modified from changes originally submitted by
Dave Stephens, which implements a monthly view. You enter the monthly view
from the annual view.
2016-03-14
- Added META() and ENTRY_META() functions to the SQL shell to answer a
question on the mailing-list (from richieuk63 at gmail dot com).
- Fixed a bug with the OFX importer whereby the Balance directive should be
generated to a day following the date provided in the LEDGERBAL directive.
- Added forgotten dependency on "beautifulsoup4" in setup.py.
2016-03-13
- Merged more changes from the 'ingest' branch which furthers the work of
integrating the LedgerHub code into Beancount (almost done now).
* Added mimetype() and head() methods to the file cache object passed to the
extractors, as a convenience instead of having clients call the cached
convert() method.
* Added paranoid exception handlers to all the methods called on the
importer classes.
* Added tests for the examples provided in "examples/ingest".
* Refactor some of the code of beancount.ingest.file.
* Fixed bugs with ensuring the filenames are absolute names.
* Cleaned up the output of the commands.
* Made the header and section strings configurable from the import
configuration.
* Cleaned up the OFX importer from LedgerHub, added unit tests for it, and
added it to the beancount.ingest.importers directory. In addition, added
it to the example configuration for ingest under
beancount/examples/ingest.
* Added a dependency on the beautifulsoup4 package (bs4) in order to parse
XML files.
- Added a beancount.core.amount.add() function to add two amounts.
- Made the directive printer be more lenient to support incomplete postings.
- Moved the "src/clojure" and "src/golang" directories to the experiments/
directory; they really are experiments. Should have done this a long time
ago.
2016-03-12
- Converted the amount.amount_*() functions to amount.*(). It's just simpler
and nicer.
- Made the beancount.report.balance_reports text reports rendered using the
precision from the DisplayContext, and realization.dump_balances() require a
formatter argument. More probaby needs to be done on issue 108 to close it,
but that's a good start.
- Made the beancount.plugins.auto_accounts plugin insert entries that will
sort stably.
- Sync up the print report code with the print command code, fixing a bug with
precision not being rendered correctly by the print report.
- Fixed 104: bean-format would fail on numbers with commas in them.
- Fixed 106: The "render_commas" option wasn't being honored. I have no idea
how things came to be this way, but I fixed it. When rendering numbers,
inserting commas for thousands (e.g., "1,000" vs. "1000") is now controlled
by the "render_commas" option, and the default is not to render them
(FALSE). Note: This should affect the web interface, the reports, and the
PRINT command, but so far NOT the output of the SQL query reports. The
reason for this is historical: the query rendering code does not make use of
the DisplayContext; it needs to be converted to do that. See
https://bitbucket.org/blais/beancount/issues/105/
https://bitbucket.org/blais/beancount/issues/106/
https://bitbucket.org/blais/beancount/issues/107/ and
https://groups.google.com/d/msg/beancount/frfN1zc6TEc/d5OjuDnREgAJ for a
discussion.
2016-03-06
- Fixed issue 83: Renamed 'default_tolerance' option to
'inferred_tolerance_default'.
- Added exception handlers for all importer callbacks. Badly written importers
should never completely throw off the tools.
2016-03-05
- Improved handling of corrupted or out-of-date pickle cache file.
- Fixed a test that would only fail in a timzone != EST.
- Made bean-file use the configuration filename's directory as the default
output dir instead of requiring an option. The behavior of providing a
reasonable default makes most sense.
2016-02-28
- Moved beancount.scripts.report to beancount.report.report in order to be
consistent with the way other libraries are invoked. I had to move
beancount.report.report out of the way (to beancount.report.base) to make
space for it.
- Moved beancount.scripts.query to beancount.query.shell in order to be
consistent with the way other libraries are invoked.
- Merged fix by yagebu for bug found in b.ops.summarize() that would not yield
a list of ordered transactions.
- Introduced a new "Custom" directive type. You can enter directives like
this:
2016-02-28 custom "budget" ....
where "..." can be a mixed list of
* string
* date
* boolean
* number
* amount (number + currency)
The first element is a string, which is intended to be used as the runtime
type of the directive. This can be used to prototype new features in plugins
or otherwise.
Right now, no consistency is enforced on the datatypes declared for
directives with the same type name. Also, there is currently no way for a
plugin to declare what a valid set of tokens may be (but I may do this later
on to simplify error reporting for plugin implementors).
2016-02-27
- Fixed rendering bug in bean-web whereby empty cells would receive a 0.00
amount.
- Improved rendering of the account name in the name of the compat importer.
2016-02-24
- Changed the bean-doctor "linked" command to fetch the set of links from the
entire set of entries.
- Added ABS() function to the shell, in order to dig around for transactions
with a particular amount.
2016-02-22
- Fixed issue 98: There was a bug when fetching historical data from a Yahoo
price source was failing, it wasn't handled correctly.
- Merged LedgerHub back into beancount under a new "beancount.ingest" package.
The new protocol for importers can be read at:
/home/blais/p/repos/beancount1/src/python/beancount/ingest/importer.py
There are three new scripts:
bean-identify
bean-extract
bean-file
These correspond one-to-one to their ledgerhub variants.
There is a bit more work to be done to complete this - see the top of the
TODO file in the "ingest" branch if you're interested - and I'll write
documentation on this once all the work is completed.
LedgerHub is now officially dead.
2016-02-20
- Create an beancount.utils.misc_utils.is_sorted() utility function to be used
in the new importer code.
- Match a price source over the entire source string.
2016-02-19
- Now accepting '=' in currency symbol in order to accommodate currencies in
Yahoo, which look like this "CNYUSD=X".
2016-02-15
- Handle an incomplete pickle file by ignoring it and regenerating it silently.
2016-02-14
- Baked a 2.0b5 and 2.0b6 releases in a further attempt to get the included
files to be included by setuptools and for pip3 to be able to make a local
build. setuptools is horrible.
- Baked a 2.0b4 release for a more recent PyPI availability.
- Fixed issue 72 - Thanks to Gary Peck for figuring out what was wrong with
setuptools - and now you should be able to install via pip3 again, which
will pull all the dependencies automatically. If setuptools isn't installed,
this should degrade peacefully to distutil.
2016-02-06
- IMPORTANT API CHANGES FOLLOW:
I made some important changes to the following classes:
beancount.core.position.Position
beancount.core.inventory.Inventory
beancount.core.data.Posting
"Position" objects have been restructured to reflect the structure implied
by the syntax. Instead of a .number (Decimal) and .lot (Lot) objects, a
Position now has a .units (Amount) and .cost (Cost) objects.
The "Lot" class has been removed; it has been replaced by a "Cost" class
with .number (Decimal), .currency (string), .date (datetime.date), .label
(string) attributes.
The "Inventory" class is a list of "Position" objects, and as such it has
received corresponding adjustments and minor changes.
"Posting" objects do not contain a .position attribute anymore. Instead, the
data structure has been flattened to contain .units and .cost objects
directly. As a result, a Posting acts a bit like a subclass of a Position
rather than using composition to contain one.
During the parsing phase, a "Posting" object's .cost attribute does not
point to a "Cost" instance, but rather to a "CostSpec" instance, which
reflects the unresolved and partially incomplete lot specification from the
input syntax that needs to get resolved to a particular lot during the new
booking process.
Here's a pictorial representation of changes. Before we had:
Assets:Checking -40 QQQ {66.32 USD, 2015-12-14} 70.32 USD
`---account---' `number' `--------------lot--------------' `--price--'
`currency'`---cost--' `--date--'
posting (Posting)
.account (str)
.position (Position)
.number (Decimal)
.lot (Lot)
.currency (str)
.cost (Amount)
.number (Decimal)
.currency (str)
.lot_date (datetime.date)
.price (Amount)
.number (Decimal)
.currency (str)
After the changes we have something that looks more like the input:
Assets:Checking -40 QQQ {66.32 USD, 2015-12-14, "0d6f0"} 70.32 USD
`---account---' `-units-' `------------cost-----------------' `--price--'
number currency date label
posting (Posting)
.account (str)
.units (Amount)
.number (Decimal)
.currency (str)
.cost (Cost, CostSpec, or None)
.number (Decimal)
.currency (str)
.date (datetime.date, or None)
.label (str, or None)
.price (Amount, or None)
.number (Decimal)
.currency (str)
And field-by-field conversions:
posting.account -> posting.account
posting.position.number -> posting.units.number
posting.position.lot.currency -> posting.units.currency
posting.position.lot.cost.number -> posting.cost.number
posting.position.lot.cost.currency -> posting.cost.currency
posting.position.lot.lot_date -> posting.cost.date
(none) -> posting.cost.label
posting.price.number -> posting.price.number
posting.price.currency -> posting.price.currency
Notes:
* These changes are mostly driven by the need for the parser to output
incomplete cost specifications. The booking of lots against existing
inventory is isolated in a beancount.parser.booking and before booking we
need to be able to represent missing values from the input syntax, since
the booking process is able to interpolate and fill in these missing
values. It was required to make important changes to the data structures
for this reason. The parser outputs 'posting.cost' with an instance of a
new "CostSpec" class, not "Cost". After booking, all the postings's costs
are replaced by instances of "Cost" and "CostSpec" is gone.
* This representation is much closer to the syntax. It's also flatter and
simpler. The original representation's design was driven by an old idea
that the lots were to be used as the key in Inventory mappings, but as it
turns out, operations on lots are just as well implemented with an
association list and performance isn't a problem. Moreover, lots with a
cost are never merged using such a mapping, only reduced against, so this
was an unnecessary distraction.
* If you were creating Position objects in plugins or other code, please
look at the new interface and make necessary changes to your code. You can
create new Position objects from a pair of Amount and Cost objects.
- beancount.core.position.lot_currency_pair() has been converted to
beancount.core.position.Position.currency_pair().
- Renamed beancount.core.position.Position.cost() to
beancount.core.position.Position.at_cost() due to the conflict resulting
from the Position class changes.
- There is a new option "booking_method" which should be set to the value
"SIMPLE" to maintain the current booking algorithm (it's the default if left
unset), but which can be set to "FULL" to enable the yet-unreleased, new
experimental booking method. Do this at your own risk; this is not ready for
public use, not effectively working yet.
2016-02-05
- Fixed bug with query parser: a single dot (.) would be matched as a number.
It should require at least some digits.
- Fixed bug whereby a missing cost number would be rendered as "0"; if needs
be a null value.
2016-01-31
- Added python-magic as an optional depdendency, in order to support imports
code. Updated http://furius.ca/beancount/doc/install
- Renamed beancount.scripts.checkdeps to beancount.scripts.deps (annoyed with
name too similar to beancount.scripts.check and I trip over it all the
time).
- Moved ledgerhub.core.compress to beancount.ops.compress and its associated
test.
- Moved ledgerhub.identify.filetype to beancount.utils.file_type and its
associated test and test files.
- Small improvement to beancount.utils.test_utils.skipIfRaises() to accept
multiple exceptions.
2016-01-25
- Moved many of the utility code from ledgerhub.utils to beancount.utils. This
is beginning the process of migrating and rewriting LedgerHub and
integrating it into the Beancount source code.
- Moved the 'remove_crdb' utility script from LedgerHub into
beancount/experiments/imports.
2016-01-18
- Committed the Query directive to non-experimental status, for
beancount-web's usage.
2016-01-16
- Created the beancount.plugins.coherent_cost plugin which checks that one's
currencies are either always used with cost, or never at all. This is by far
the most typical situation whereby a currency is always held at cost, or
never (and perhaps converted at price or not).
- Created a function to filter just the transactions, which is such an
incredibly common operation I should have done this a while ago. I should
convert all code to use it.
2016-01-16
- Added a feature to the "beancount.plugins.ira_contribs" plugin to allow the
further restriction that the trigger posting must include a particular flag
in order for the insertion to occur. This is to be used in account that have
sales, whereby just looking at the sign of the posting isn't enough to
decide that an insertion must be made.
2016-01-10
- Addressed issue 96, supporting an explicitly installed C decimal version
when the Python3 installation does not include one. From
beancount.core.number:
Note: Python 3.3 is supposed to guarantee a fast "C" decimal implementation,
as it comes with its source code. However, in practice, many distributions,
including the popular Ubuntu distro, simply don't compile it. Other distros
(e.g., Arch Linux) insist on breaking up Python into multiple packages in
order to manage the dependency to the mpdec library, and make that
installation optional. Moreover, there seems to have been changes in the
dispatching of pure-Python and fast C decimal implementations in Python-3.5's
source code itself. The result is that despite Python-3.3 or above being
installed, a fast C decimal implementation may or may not be present. Thus, we
must detect it here. I believe this subtle problem should go away with
Python-3.5 and above.
What do we do? We import the default 'decimal' module, and if it's the pure
Python implementation, we attempt to import an explicitly installed
'cdecimal', which you can install with
pip3 install m3-cdecimal
Also added a check for this in checkdeps. Hopefully this resolves the issue
once and for all.
2016-01-09
- Fixed a bug (issue 94) in the load cache that wouldn't handle files
disappearing after having been moved.
2016-01-03
- Moved beancount.web.scrape.scrape() to beancount.web.web.scrape_webapp() to
isolate beancount.web.scrape from dependencies, so that other projects may
reuse it.
2015-12-25
- Closed multiple obsolete branches: parse_test_utils, display_context,
performance, datatypes.
- Added some utilities for parsing strings for writing tests that use the
local variables present in the caller (a bit of magic).
- Improved date parsing from command-line options for some scripts; imported
from branch 'text_statements'.
2015-12-24
- Implemented metadata stack and "pushmeta" and "popmeta" directives mirroring
"pushtag" and "poptag". These new directives work similarly, but setting
metadata fields instead of tags themselves. The parser maintains a stack of
values for each metadata field. Values explicitly set on the Transactions
themselves are not overridden by the meta stack.
2015-12-21
- Created a new project script that uses specific metadata on Open directives
to generate a document of a user's assets, institution names, addresses and
contact info, and account types and approximate value. Note: I haven't made
the example file include necessary fields to demo or document this yet.
- The loader now supports loading encrypted files transparently. If a filename
ends with .gpg or .asc (and contains armored content), the file is run
through a gpg subprocess to decrypt it. No temporary files are created. Note
that caching for those files is disabled.
If you're writing a script, you can also call
beancount.loaded.load_encrypted_file() directly to avoid the check.
- Implemented the 'beancount.plugins.merge_meta' plugin that copies metadata
from an external file into a regular file. It is essentially a data join on
the key of accounts for Open and Close directives, and a join on the key of
currency on Commodity directives.
- Added an option to Inventory.to_string() to avoid rendering parentheses.
- Added variable and username expansion support for the loader filename.
- Removed --batch mode for the encryption support; it needs to be potentially
interactive.
2015-12-17
- In prices, removed undocumented support for automatically extracting
Commodities with a "quote" metadata field. This had been used in times past
and had been brought in as legacy, but now neither the export code nor the prices
code use it. Vaccummed away.
2015-12-13
- Fixed a minor bug with the loader cache whereby errors weren't getting
emitted on a cache hit (Issue 92).
2015-12-12
- Made bean-web's --incognito mode replace integer numbers, not just
fractional ones. (See issue 86).
- Output a warning when cdecimal isn't installed. This may occur on some
distribution which break up Python into multiple packages in order to allow
the user to avoid depending on a mpdec library install. Arch Linux suffers
from this problem.
- Made some improvements to the loader caching algorithm in order not to rely
on the percision of timestamps from the filesystem. Filesystems implement
one-second precision which is not sufficient in some case. I now cache both
the time and the size of files in order to check for a cache hit, and this
paves the way for hashing the contents of files as well eventually.
- Began implementing aggregation of some of the attributes of options map
across include files. Both "operating_currency" and "commodities" will be
aggregated correctly. (This was a request for beancount-web.)
- Fixed flaky test that failed only in homecooked install of Python 3.5.1.
- Fixed a minor bug that would not print out costs with zero amounts.
2015-12-11
- In the context page dedicated to rendering the details of an entry, render
an "Open" link on the upper right that can be clicked on in order to invoke
a desktop handler that will open a text editor to the containing file and at
the right line number.
2015-12-10
- Renamed beancount.reports.context.render_entry_context() to
render_file_context() and implemented render_entry_context() to accept an
entry directly instead of a (filename, line-no), so that bug reporters can
insert some print statements. I also removed the dcontext instance required
to be passed, since all the callers were fetching it from options_map. This
should make it easier for those sending bug reports to insert quick calls to
render the context around a particular entry or file location.
2015-12-09
- Fixed installation snafu: forgot to add the beancount.prices and bean-price
install to the setup.py file.
2015-12-07
- Fixed a bug in the load cache, whereby the load cache would not update
itself on an included file. In order to do this, first I made the loader
always return the full correct set of unique filenames processed in a load
call, and then I used this to check the latest modified time.
- I also used this to fix a related bug in the bean-web watcher: If an
included file was being modified, bean-web would not reload itself
automatically. redstreet0 sent a pull request to highlight the problem
(thanks!):
https://bitbucket.org/blais/beancount/pull-requests/6/feature-include-files-refresh/diff
2015-12-03
- Added ROOT(account, num) account manipulation function to the shell.
2015-12-02
- Fixed issue 84: The sorting of entries from multiple files was incorrectly
placed, due to recent changes merged from the booking branch.
- Fixed a bug deleting the load cache (was using an invalid filename).
2015-11-29
- Made the pickle load cache enabled by default. This means you can now remove
the BEANCOUNT_LOAD_CACHE environment variable. The load cache will be
written out automatically, but only if it takes more than the time threshold
(1 sec) to load a Beancount file. If it takes less than this time, no cache
is written. (This saves the creation of these pesky crumbs for files it
doesn't matter for.) If you would like to always disable the load cache, set
the environment variable BEANCOUNT_DISABLE_LOAD_CACHE. I don't recommend
it--this works well and for large files makes loading much, much faster.
2015-11-28
- Implemented a script used to fetch prices online and output them in
Beancount input format. This script is able not only to take an explicit
list of prices to get, but also able to read a Beancount input file and
figure out the list of commodities held at a particular point in time and
fetch only those prices. A "price" metadata field on a Commodity directive
is used to specify how and where to get prices from. It comes with support
for Google Finance and Yahoo! Finance price sources and the syntax supports
custom-written Python modules so you can write your own price fetching
extensions, that integrate with the price fetching tool.
The new code is located under "src/python/beancount/prices" and this new
tool is documented in http://furius.ca/beancount/doc/prices.
(Note: This is brand new code, probably imperfect, but I tried to keep this
simple yet complex enough to be practical. I've been using a variant of it
for a while, I'm going to merge this now and use it myself, and I hope it
works for others.)
2015-11-26
- I redesigned the portfolio export process in order to (a) simplify it
greatly (the process was a bit confusing, between the use of three metadat
fields: "ticker", "export" and "quote"), and (b) to separate it entirely
from the upcoming price fetching script metadata.
The new outcome is very similiar, however, except it uses only a SINGLE
metadata field: "export", and the documentation has been rewritten to
reflect this:
https://docs.google.com/document/d/1mNyE_ONuyEkF_I2l6V_AoAU5HJgI654AOBhHsnNPPqw/edit#
If you have an existing file, here's what you need to do to it:
* Remove all the 'quote' metadata. You won't need it.
* If you have 'ticker' metadata: convert those to 'export'.
* If you have 'export' metadata: keep it.
* If you had an 'export' metadata with a "MONEY" spec, move that into the
'export' field after the symbol to export to.
The example has been ported, so you can refer to that as well.
I think the new process is much simpler to understand and the documentation
shorter and easier to read. It should be a trivial exercise to convert one's
input file to the newer format.
2015-11-22
- Removed beancount.core.data.AttrDict and replace it by a regular dict. This
removes some annoyances when pickling and copying metadata. Metadata is now
represented by a plain old dict. (Fixes issue 79.)
2015-11-09
- Implemented a simple pickle-date on beancount.loader.load_file(), which will
reload the parsed & processed contents of a top-level file if a cache file
is present and more recent than the file itself, or update the contents of
the cache after loading if not. (This works a bit like .pyc byte-compilation
in Python or .elc in Emacs LISP.) This makes the loading of large files
multiple times ~15x faster, which is a big deal since Beancount has become a
little bit slow (I need to spend time optimizing at some point).
This feature is a bit experimental, so for now it only gets turned on if the
BEANCOUNT_LOAD_CACHE environment variable is set.
2015-10-04
- Implemented --output=<filename> option for bean-query tool.
2015-10-02
- Implemented an experimental new Query directive whose purpose is to define
pre-canned SQL queries in the Beancount file itself and to then build tools
to automate the generation of reports of those queries.
For the moment being, I'm just playing with this to convince myself that
this can be powerful and generic enough to provide more generally. I've
found myself running specific queries on speific files, e.g., during trips
and I'd like to automate those to some extend, perhaps replace the script
code that lives in beancount.plugins.split_expenses with this.
Anyhow, this is not supported but if you understand this and really want to
turn it on, set this option in your input file:
option "experimental_query_directive" "TRUE"
The directives aren't used anywhere yet, and you'll have to read source code
to figure out what they look like.
- Implemented a "reload" command in the shell. Typing "reload" will refresh
the data from the latest file contents.
2015-09-29
- Tightened the screws on the Amount() constructor a bit and made changes ini
order to allow it to accept a sentinel value for MISSING values from the
parser. Changes to beancount.core.amount.Amount are:
1. A string for the number is not accepted anymore, that is, Amount(str,
str) is not supported anymore. Amount() now unconditionally requires a
Decimal instance, so you must use Amount(Decimal, str). In any case, this
constrcutros was likely abused only by me in the tests. I'd rather have
a tighter, less permissive constructor.
2. The convenience constructor beancount.core.amount.from_string() is now
officially aliased as beancount.core.amount.A, so you can "from
beancount.core.amount import A". I now commit to using this constructor
and converted all the tests to use it. This is great for writing tests.
Amount really is a fundamental piece that deserves a very short alias
like this.
In summary, how to create Amount instances:
* If you use explicit values, use Amount(Decimal, str), e.g.,
Amount(D('100.00'), 'USD'), Amount(number, currency)
* If you need a convenience constructor, use A(str), e.g.,
A('100.00 USD'), A('32,000 KRW')
2015-09-29
- Added a new column 'description' to the SQL client in order to produce more
compact reports.
Oftentimes the payee is left to an empty value and journals
have a lot of wasted whitespace. Concatenating the payee and the narration
in a single field goes a long way towards producing reports that aren't
nearly as wide. I could have added a CONCAT() function but this is so common
and convenient that a dedicated 'description' alias is justified. You can
use it in posting context ("SELECT description ..." or "SELECT ... WHERE
description ~ '...'") or transaction context ("SELECT ... FROM description ~
'...'")..
2015-09-27
- Implemented a plugin that can book price conversions using a particular
booking method, in this case, FIFO. This idea stemmed from a suggestion I
made on the mailing-list a few weeks ago, which was to process a particular
account's price conversions to insert cost basis specifications and in doing
so to book reducing postings to FIFO order automatically. This could all be
done via a plugin, so I did it. (See tag {c395de691449} on the mailing-list
for more context.)
You can find it under "beancount.plugins.book_conversions". You use it like
this: first you load the plugin:
plugin "beancount.plugins.book_conversions" "Assets:Bitcoin,Income:Bitcoin"
The configuration requires you provide two accounts:
1. An assets account to process, whereby all price conversions (in any currency)
occurring in that account will have their cost basis automatically inserted,
and,
2. An Income account that is inserted on reducing transactions to absorb the
profit or loss of the sale.
Then you book your transactions as simple price conversions, without worrying
about cost basis, for example, this input:
2015-09-04 *
Assets:Bank -750.00 USD
Assets:Bitcoin 3.000000 BTC 250.00 USD
2015-09-05 *
Assets:Bank -780.00 USD
Assets:Bitcoin 3.000000 BTC 260.00 USD
2015-09-20 *
Assets:Bitcoin -2.000000 BTC 300.00 USD
Expenses:Something
2015-09-21 *
Assets:Bitcoin -2.000000 BTC 310.00 USD
Expenses:Something
2015-09-22 *
Assets:Bitcoin -2.000000 BTC 330.00 USD
Expenses:Something
Would get translated by the plugin into the equivalent of this:
2015-09-04 *
Assets:Bitcoin 3.000000 BTC {250.00 USD} 250.00 USD
trades: "trade-ff4c3e592147,trade-6b9c8689ed50"
Assets:Bank -750.00 USD
2015-09-05 *
Assets:Bitcoin 3.000000 BTC {260.00 USD} 260.00 USD
trades: "trade-91e2245de593,trade-d622a89c614d"
Assets:Bank -780.00 USD
2015-09-20 *
Assets:Bitcoin -2.000000 BTC {250.00 USD} 300.00 USD
trades: "trade-ff4c3e592147"
Income:Bitcoin -100.00000000 USD
Expenses:Something 600.00000000 USD
2015-09-21 *
Assets:Bitcoin -1.000000 BTC {250.00 USD} 310.00 USD
trades: "trade-6b9c8689ed50"
Assets:Bitcoin -1.000000 BTC {260.00 USD} 310.00 USD
trades: "trade-91e2245de593"
Income:Bitcoin -110.00000000 USD
Expenses:Something 620.00000000 USD
2015-09-22 *
Assets:Bitcoin -2.000000 BTC {260.00 USD} 330.00 USD
trades: "trade-d622a89c614d"
Income:Bitcoin -140.00000000 USD
Expenses:Something 660.00000000 USD
Note that the price annotations are kept as in the input file. The plugin only
inserts the cost basis and splits postings where required.
Secondly, a unique trade identifier is allocated for each matching pair of
postings that were booked and is added to the metadata of those postings.
Its value is a comma-separated list. This metadata is useful because it
allows you to load the entries and from it, to build a list of trades to
produce a suitable report for reporting.
So the plugin also contains code to do this. You can invoke it as a main
program, like this:
python3 -m beancount.plugins.book_conversions <filename.beancount>
On the file above, it should generate the following output, which I believe is
the report you're really after for reporting purposes:
Units Currency Cost Currency Buy Date Buy Price Sell Date Sell Price P/L
-------- -------- ------------- ---------- --------- ---------- ---------- ------
2.000000 BTC USD 2015-09-04 250.00 2015-09-20 300.00 100.00
1.000000 BTC USD 2015-09-04 250.00 2015-09-21 310.00 60.00
1.000000 BTC USD 2015-09-05 260.00 2015-09-21 310.00 50.00
2.000000 BTC USD 2015-09-05 260.00 2015-09-22 330.00 140.00
-------- -------- ------------- ---------- --------- ---------- ---------- ------
Use the '--format=csv' option if you want to produce output suitable for
including in a spreadsheet. If you need to customize the output further, see the
source code (file:book_conversions.py). It is not possible to generate this list
of trades using the SQL client yet, to change the output columns you would have
to create a standalone script; you can copy just the main() function in the
plugin if you need to do that and import the rest. (Note: The function that
extracts the list of matches provides both the posting and its parent
Transaction object, so you can output data from the transaction if you want,
e.g. the narration of the selling transaction.)
2015-09-14
- Merged more changes to beancount.el (Emacs support) by Stefan Monnier:
* Use syntax-tables for comments and strings.
* Make ido use optional in Emacs config.
* Quote filenames on command lines
Thank you Stefan!
2015-09-12
- Added to beancount.plugins.split_expenses a new script that is able to
generate the typical kinds of reports you would need to provide to
participants in a shared project or travel on their specific expenses and
contributions, as well as a breakdown of their expenses per category
(expenses account) and final balances for each participants. Invoke it like
this:
python3 -m beancount.plugins.split_expenses <filename>
The script lives in the same file as the split_expenses plugin, because it
is tightly related to it. It presumes the input file will process through
this plugin. Try to run it on the example files under
beancount/examples/sharing.
- Added a new file "examples/sharing/duxbury2015.beancount" which contains a
new example of sharing expenses on a trip between three people.
- Starter cleanup of beancount.el contributed by Stefan Monnier. More changes
to come, this is basically a no-op patch. It just moves code around and
makes other such cosmetic changes to silence some compiler warnings, prepare
for subsequent changes, better follow coding conventions etc. Detail:
* beancount.el: Add some standard file headers, license, and
sectioning comments. Sprinkle a few FIXMEs while I'm here.
(beancount-font-lock-defaults): Rename from beancount-font-lock-keywords,
since this corresponds to font-lock-keywords and not to
font-lock-defaults.
(beancount-directive-names, beancount-font-lock-keywords,
beancount-accounts): Move variables before their first use (to silence
the compiler).
(beancount-mode-map): Use a proper defvar instead of define-minor-mode's
half-broken :keymap.
(beancount-mode): Merge the make-local-variables into their corresponding
`set'.
(beancount-accounts): Move make-variable-buffer-local to top-level since
it's a global operation.
(beancount): Declare this group, already used in define-minor-mode.
(beancount-hash-keys): Mark unused arg `v' with an underscore.
(beancount-transaction-set-flag): Don't use replace-string (designed for
interactive use) from Elisp.
(beancount-for-line-in-region): Use copy-marker and eobp.
(beancount-align-numbers): Break lines to avoid hiding arguments.
(compilation-read-command): Declare variable to silence byte-compiler.
2015-09-06
- Filter out platform-specific comments from the code generated by GNU Bison,
in order to avoid unnecessary diffs when I switch between Linux and Mac OS X.
2015-09-05
- Changed the semantics of the account_rounding account, so that it matches
that of the other account options, that is, they all specify a sub-account
of the Equity account. So if you had
option "account_rounding" "Equity:RoundingError"
in your input file, change this to:
option "account_rounding" "RoundingError"
To ease the transition, the parser now checks if the first component is one
of the five categories and if so, issues a warning and removes the prefix.
(One side-effect of this change is that the rounding error account may only
be an Equity account, but I'm hoping this won't be a problem for anyone.)
2015-08-31
- The column "weight" was broken; fixed by yagebu.
2015-08-30
- Fixed a very minor bug in split_expenses plugin whereby the generated
postings did not contain the __automatic__ metadata field, and in some
particular situations, their automatically calculated values would end up
being used for inferring the tolerances.
2015-08-15
- Changed the semantics of the parsing stage, in a fairly profound way.
This should have no visible changes to users, but people writing scripts
should revise their code if they were using
beancount.parser.parser.parse_*() functions.
Just to be clear: beancount.loader.load_*() has not changed. If you just use
the loader, there are no changes. Changes are only at the parser level.
Here's what's going on and why: The parser used to carry out interpolation
of missing numbers on postings for each transaction locally, while parsing
the transactions. This was done by calling by calling
beancount.core.interpolate.balance_incomplete_postings(), here:
https://bitbucket.org/blais/beancount/src/ee2073aae080aaa8e260abe8a501abf872948f0e/src/python/beancount/parser/grammar.py?at=default#grammar.py-803
Loading a list of entries was carried out in two steps:
,-----------------------load----------------------.
(recursively)
,---------------------. ,---------. ,------------.
(input)--->| parse + interpolate |-->| plugins |-->| validation |--> entries
`---------------------' `---------' `------------'
First, the parser would run on the input and process all the input files
recursively (processing includes). "Interpolation", the process of filling
in missing numbers, was carried out at that stage, during parsing, and only
locally, that is, for each transaction in isolation. "Booking" of lots, that
is, selecting which of an account's inventory lots to match and reduce, was
explicit. This booking could not take advantage of the accumulated
inventories in order to vary its behavior. You had to specify the entire lot
information unambiguously.
After this, in a second stage, the plugins were run on the entries and a
final validation step was run at the end.
To implement the booking proposal
(http://furius.ca/beancount/doc/proposal-booking), we want for the user to
be able to provide a partial specification of lots to be matched against an
account's accumulated inventory at the date the transaction is to be
applied. The idea is that if there is no ambiguity, the user should be able
to specify very little information about a lot (for example if there is a
single lot in the account when processing the transaction an empty spec of
"{}" should be sufficient). Moreover, where the specification is ambiguous,
we also want to support automatic selection of lots according a method
specified by the user, e.g., FIFO booking, LIFO booking, etc.
For this to work, we need to have parsed all the inputs to some sort of
incomplete specification, a representation of inputs that hasn't yet been
resolved to specific lots, in order to carry out booking. The parser has
been modified to output such incomplete postings:
,-------------------------load-------------------------.
(recursively)
,-------. ,-------------. ,---------. ,------------.
(input)--->| parse |-->| booking |-->| plugins |-->| validation |--> entries
`-------' | + | `---------' `------------'
| interpolate |
`-------------'
incomplete
entries
Because "interpolation" runs on the result of specific lot values, "booking"
must run before it, and so they are inter-related. Thus, booking and
inteprolation has been moved to a dedicated step that runs on the list of
incomplete entries and resolves them to the regular entries to be processed
further by plugins and validation.
This also has a nice side-effect: the booking step is where all the
complexity is, and it is now isolated and I will be able to test and
experiment on it in isolation. This is where all the fun will be.
A description of the incomplete specifications output by the parser can be
found here in the parser.py file, this is the description of the
intermediate state of postings whose lots haven't yet been matched and
resolved to specific inventory lots:
https://bitbucket.org/blais/beancount/src/18282452e265959b69d3d10c6d9cf32e5815c522/src/python/beancount/parser/parser.py?at=booking
Essentially, a posting's 'lot' attribute contains a "LotSpec" tuple instead
of a "Lot" tuple, and several numbers may be left unfilled (for interpolate
values). I don't imagine anyone will ever have to manipulate such
intermediate entries, only the booking code.
The previous booking algorithm has been moved to the booking stage and the
semantics should be identical to what they used to be. This is still the
default algorithm--it just runs in its own dedicated stage, still operating
locally on each transation. You should observe no difference in behaviour.
I've merged these changes now in order to minimize the differences between
the booking and default branches and because I was able to do it without
changing any of the semantics (despite the large number of lines and tests
modified). This was a necessary refactoring. Because that code is now
isolated to its own stage I should be able to begin implementing the more
complex state-dependent booking algorithms in the 'booking' branch. (I'm
excited about this.) I could even implement different booking heuristics and
switch between them.
Because the parser used to spit out regular, complete entries,
parser.parsedoc() was used in much of the tests instead of loader.loaddoc(),
so that many of those tests would not have to concern themselves with making
sure the input passed the validation stage run by the loader. For example,
creating Open directives just to create some Transaction test object wasn't
necessary in the tests. All of those tests had to be revised and I made them
all depend on beancount.loader.load_*() instead of the now weakened
parser.parse_*() functions which output incomplete, unbooked entries. This
cleans up the dependencies a bit as well. If you wrote your own unit tests
and were using parser.parsedoc(), you should convert them to use
loader.loaddoc(). In order for this change not to go unnoticed (and for
naming consistency with parse_string() and parse_file()) I'll probably
rename parser.parsedoc() to parser.parse_doc() and ditto with the loader.
(Note: In some cases I've had to specifically setup the input of some tests
in "raw" plugin processing mode to avoid triggering some unwanted and
unrelated errors. I'm tempted to remove even more from the default plugins.
This may happen in a future CL.)
- Renamed beancount.parser.parser.parsedoc() to parse_doc() and
beancount.loader.loaddoc() to load_doc(), for consistency with the other
parse_*() and load_*() functions. Kept a stub that will issue a warning if
you use it.
2015-07-23
- In the conversion to the new booking syntax, I had inadventently removed
support for the total cost "{{ ... }}" syntax. I brought this back next to
the new booking syntax, and uncommented tests that had been made to skip.
2015-07-21
- Merged ongoing work from the 'booking' branch that will eventually change
the semantics of inventory booking; in the interest of a smooth transition,
and for me to be able to use either branch interchangeably, I introduced a
few changes that should have no user-visible effect on 'default':
* The parser.parse_string() and parser.parse_file() routines don't
interpolate anymore. For this reason, the tests all had to be adjusted not
to include interpolation. The interpolation of incomplete postings
(grammar.interpolation()) has moved to a separate phase and is now run by
loader._load() *AFTER* parsing. This is key to implementing fuzzy matching
semantics for matching reducing lots: we need to have all the incomplete
transactions parsed and sorted in order to select matching lots in date
order.
This only affects you if you wrote scripts against the parser
interface directly (this is highly unlikely).
* For writing unit tests, parser.parsedoc() and loader.loaddoc() are now
decorator factories. This allowed me to add options to parser.parsedoc()
to perform interpolation, and when not specified, to check that entries
with interpolation are not present in the tests. Also, I coudl merge the
functionality of parser.parsedoc_noerrors() and loader.loaddoc_noerrors()
in their respective equivalents. The docstring tests now validate that
there are no errors in the docstrings by default. This makes the tests
tighter (a few bugs in the tests themselves were found and fixed).
* The new syntax for cost specification that will be in effect for the
inventory booking proposal is now supported. The syntax is
backward-compatible with the previous one. Previously, the following
syntaxes were supported for specifying the cost and optionally a lot
acquisition date:
Assets:Investments 1 HOOL {123.00 USD}
Assets:Investments 1 HOOL {123.00 USD / 2013-07-20}
Instead of these options, the new syntax supports a comma-separated list
of cost-specifiers which can be one of
<cost> -> e.g. 123.00 USD
<lot-date> -> e.g. 2015-07-20
<label> -> e.g. "first-lot"
<merge-cost> -> e.g. *
In order to keep current input working, either a comma (,) or a slash (/)
is supported to separate the components. This is valid:
Assets:Investments 1 HOOL {2013-07-20 / 123.00 USD / "first-lot"}
Those can be provided in any order. For example, these are all also valid
syntaxes for cost:
Assets:Investments 1 HOOL {}
Assets:Investments 1 HOOL {"first-lot"}
Assets:Investments 1 HOOL {2013-07-20, 123.00 USD}
Assets:Investments 1 HOOL {2013-07-20, "first-lot"}
Assets:Investments 1 HOOL {*}
Assets:Investments 1 HOOL {*, 123.00 USD}
Moreover, the cost amount now supports a compound amount that is expressed
not in terms of each unit of the currency, but in terms of the total
amount over all units, for example, this is how you could fold in the cost
of a commission:
Assets:Investments 1 HOOL {123.00 9.95 USD}
The syntax for a compound amount follows this pattern:
[<per-unit-cost>] [<total-cost>] <currency>
The numbers are both optional. If no '' separator is present, the total
cost component is assumed to be zero. This will eventually subsume the
{{...}} total cost syntax, by specifying only the total cost portion of
the compound amount:
Assets:Investments 100 HOOL { 12300.00 USD}
Of course, this combines with the other spec formats, so this is valid:
Assets:Investments 1 HOOL {123.00 9.95 USD, 2015-07-22}
Finally, while the new syntax is supported in the parser, the old
semantics for inventory lot specification is still in order. If you
provide an unsupported combination of lot specifiers (e.g., you use a
label, a compound amount, or a merge-cost marker), an error will be issued
accordingly.
So just use the cost and date as previously; I will bring in new semantics
incrementally, semantics that will take advantage of this new syntax. I
will try to do so in a way that minimizes changes.
- I removed forwarded symbols from beancount.parser. Generally, in order to be
able to mock functions, you should always import packages, not symbols.
2015-07-20
- Numbers rendered from bean-query are now rendered using the display context
inferred from the input file. This means numbers are rounded nicely, using
the most common precision seen in the input file.
- I fixed a bug on writing UTF-8 output to the console on Mac OS X and how it
interacts with the 'less' pager.
2015-07-12
- Implemented support for arithmetic operations: +, -, *, / and parenthesis
groupings are now supported, anywhere that a number can be seen in the input
file, including postings, costs & prices, and balance numbers.
2015-07-12
- Changed how web & report rendering code renders numbers so that it uses the
DisplayContext inferred from the numbers in the input file. Numbers in the
reports were previously using the default precision and that led to all
sorts of inconsistencies. The numbers are now rendered using the most common
precision which has been seen in the input files. (This work had been
planned for a while, as part of converting everything to use the
DisplayContext. Next up will be converting the shell's rendering routines.)
2015-07-11
- WARNING: API CHANGES AHEAD. If you wrote plugins, you will need to remove
the 'entry' parameter to constructors of Posting().
I removed the Posting.entry back-reference of Postings to their owning
Transaction objects. This makes programmatically creating our data
structures much more straight-forward and avoids us having to run pirouettes
around which object to create first. It also removes all manners of "fixing
up" transactions and removes one ultimatley unnecessary degree of freedom
from plugins outputs: we don't have to check that the plugins generate
objects with correct back-references. It also makes the documentation
simpler: I won't have to explain backreferences and how to deal with them.
Finally, it avoids the circular reference which could potentially make
Python's GC do some work for no other reason.
This happened because it occurred to me that the only really important place
where back-references were used were in the beancount.core.realization code,
and that in the RealAccount objects I could easily replace the Posting
instances in lists of "postings or entries" into a new tuple object
"TxnPosting" that contains a reference to the parent transaction and a
reference to the posting. This voids the need for a back-reference, and
makes the API really straightforward.
- WARNING: USER-LEVEL CHANGES FOLLOW. I repatriated the following optional
plugins to the beancount.plugins package, so that all the plugins are in the
same place:
beancount.ops.auto_accounts -> beancount.plugins.auto_accounts
beancount.ops.implicit_prices -> beancount.plugins.implicit_prices
This is so that the "beancount.ops" package contains only plugins run by
default (non-optional), and so that the "beancount.plugins" only contains
optional plugins the user must provide.
In doing so, I also made the "implicit_prices" plugin NOT INCLUDED BY
DEFAULT. This plugin creates automatically Price directives for all Postings
which have a <price> annotation, thus feeding those prices into the price
database. If you want to keep this behaviour -- and you probably should --
you need to add the following in your input file:
plugin "beancount.plugins.implicit_prices"
(I'm considering moving move of the default plugins to be optional, in
particular, beancount.ops.documents, not sure yet.)
2015-07-05
- Implemented support for parsing strings from files in various encodings.
Only string objects encoded in non-ASCII are supported, e.g., payee,
narrations, description, etc. This means the account names are still
restricted to ASCII, in particular. This works with UTF8 encoding (tested),
latin1 encoding (tested), and probably more. The support is at the lexer
level. You may now supply an 'encoding' parameter to parser.parse_file(),
parser.parse_string(), loader.load_file(), loader.load_string() functions.
Passing in an invalid encoding should not mail it fail (though the contents
of the strings will be empty). Unit tests for all those cases have been
added.
- Completely reviewed all lexer and grammar error handling code. The parser
code used to bail out with abort() on an unhandled exception and dump core
after issuing its error message (this is ugly but was never a big issue
since this occurred pretty rarely). All errors are now handled gracefully
(invalid lexer token, exception raised in lexer builder, invalid gramamr
rule, exception raised in grammar builder Python code). A comprehensive
suite of unit tests has been built to catch all these. You should never
experience a core dump anymore. A number of other unexpected cases has also
been dealt with.
- The lexer used to have a fixed-size buffer for parsing its strings. The
buffer is now dynamically allocated. Lexing should also be slightly faster.
- Mmeory management for the parser has been reviewed. Memory clean up for
temporaries generated during parsing has been reinstated (it was disabled
somehow) and cleanup rules are invoked even in the case of errors. This may
make the parser somewhat slower.
2015-07-02
- Built support for setuptools and rolled back changes. Setuptools is broken
beyond what I can bear. Forget this. You'll have to install dependencies by
hand. It's a one-liner, like this:
pip3 install python-dateutil bottle ply lxml
2015-07-01
- Wrote a post-mortem/assessment doc on LedgerHub, detailed future plans for
LedgerHub and plans to eventually merge the codebase into Beancount again.
See: http://furius.ca/ledgerhub/doc/postmortem
2015-06-27
- Complete the design doc and made it public.
2015-06-26
- Created a generic script that can be used to automate the upload of
arbitrary CSV files to a Google Spreadsheet. This can be used along with the
holdings reports in order to create a spreadsheet tracking your assets.
Unfortunately, this script has to be written in Python 2.x, because of
the unavailability of the gdata-python-client library which is not available
at this time for Python 3.x. The script is self-contained. If you want
to use this, you must install the following libraries in your Python 2.x
install:
* gdata-python-client
* oauth2client
* httplib2
2015-06-25
- Posting objects now have (filename, lineno) in their metadata. This could
potentially be helpful for people who want to insert text programmatically
in their input file without having to write their own parser.
See: https://groups.google.com/d/msg/beancount/zNCFk-YGdr8/8P2_vGGH9xoJ
2015-06-15
- All the code to bake a static HTML archive of a bean-web interface has been
rewritten.
* There used to be a number of bugs caused by the wget tool. These bugs have
been fixed.
* The new code does not depend on wget -- Beancount does not require wget
anymore. Explicit code to do web scraping has been implemented. On the
other hand, a new dependency is introduced: lxml. This dependency was
already required for development and install in the same way as all the
other dependencies: using pip3. Should be a breeze to install on all
platforms.
* A verification layer has been tacked on after the web scraping step to
check that there are no dangling links.
* Intentionally skipped links are now converted to plain text, such that
there are no dangling links for pages we aren't saving (these just used
to be ignored).
I repeat: Beancount does not depend on wget anymore.
Please install lxml with: "pip3 install lxml"
- The 'exportpf' report which is used to produce OFX that can be imported in
Google Finance now issues a warning if you did not provide either 'ticker'
or 'export' metadata for holdings that get exported. The reason for the new
adminition is that it trips up the OFX import and when Google Finance sees
a ticker it does not understand, the import fails. At least with the warning
we're telling the user they should be providing tickers or export directives
for all their commodities if they export the portfolio.
2015-06-10
- Removed dependency on curses. If you don't have curses, everything should
work fine now, errors are being caught. Curses was only used to provide good
default values for rendering tables in terminals on e.g. bean-report
journal and to decide whether to use a pager in multi-page bean-query.
The default values are fixed if curses isn't available, e.g. 80 chars wide,
but as previously, you can override them manually. I also removed the
dependency from checkdeps and from the install document.
2015-06-09
- Rewrote the Installation Instructions document completely to make it
simpler. Some users had found it daunting.
http://furius.ca/beancount/doc/install
- Tested Windows installation using Cygwin and added it to the installation
instructions.
http://furius.ca/beancount/doc/install
2015-06-07
- Moved the following representation related symbols to their own module:
beancount.core.amount.D -> beancount.core.number.D
beancount.core.amount.Decimal -> beancount.core.number.Decimal
beancount.core.amount.ZERO -> beancount.core.number.ZERO
beancount.core.amount.ONE -> beancount.core.number.ONE
beancount.core.amount.HALF -> beancount.core.number.HALF
beancount.core.amount.round_to -> beancount.core.number.round_to
I also moved the tests. All the code specifically related to numbers is
now in beancount.core.number. I also removed the relative imports, as
Ledgerhub is importing from the Beancount source code now.
This is not a user-visible change. However, if you wrote scripts against
the codebase, you will need to rename your imports. In the meantime I added
stubs which issue a warning if you import from the obsoleted names.
- Moved documentation scripts from experiments to the main Beancount codebase.
There is a new package for it: beancount.docs. These codes are used to maintain
the Beancount documentation and there is common code for authentication in there,
code that I plan to use for automating the exporting ot portfolios.
This introduces some optional dependencies:
* google-api-python-client
* oauth2client
* httplib2
These are optional, and at the moment none of the actual Beancount tools
make any use of them. I moved this into the main Beancount codes because
the common authentication code will be used to export portfolios directly
to a Google spreadsheet in Google Drive, which will be super convenient
to build custom portfolio tracking views.
2015-06-06
- Added an 'inferred_tolerance_multiplier' option to allow the user to adjust
the magnitude of the inferred tolerance.
When the tolerance values aren't specified explicitly via the
'default_tolerance' option, the tolerance is inferred from the numbers in
the input file. For example, if a transaction has posting with a value
like '32.424 CAD', the tolerance for CAD will be inferred to be 0.001
times some multiplier. This is the muliplier value.
We normally assume that the institution we're reproducing this posting
from applies rounding, and so the default value for the multiplier is
0.5, that is, half of the smallest digit encountered.
You can now customize this multiplier by changing this option, typically
expanding it to account for amounts slightly beyond the usual tolerance,
for example, if you deal with institutions with bad of unexpected rounding
behaviour.
- Promoted the 'experiment_infer_tolerance_from_cost' option to
'infer_tolerance_from_cost': not an experiment anymore. See thread on
mailing-list w/ mharris. This feature works well and actually solves real
problems for users.
A corresponding section has been added in the documentation at
http://furius.ca/beancount/doc/tolerances
2015-06-04
- When using the 'experiment_infer_tolerance_from_cost' option, the inferred
tolerance is now expanded to include the SUM of all maximum rounding errors
from postings held at cost, not just the maximum. Users with multiple
rounding events in a single transactions could occur rounding that many
times. We assume no more rounding events than the number of postings held at
cost.
- I wrote a new document to explain how to deal with restricted stock units
vesting events:
http://furius.ca/beancount/doc/vesting
2015-06-03
- bean-doctor context now outputs weight and residual numbers at their full
precision instead of their rounded one.
2015-06-02
- Created a new document to keep track of user contributions:
http://furius.ca/beancount/doc/contrib
2015-05-30
- Converted the Syntax Cheat Sheet to a Google Doc and removed it here.
- Added a check to bake the source code's checksums into the C extension
module and upon loading beancount.parser.parser check that the checksums
match. This automatically issues a warning on all programs that use the
parser if the compiled extension module is out-of-date and the user needs to
rebuild. This should explain the occasional unexpected situation where the
extension module doesn't match the Python code, for those running from
source or during development and in switching branches.
- Added an experimental boolean option: 'experiment_infer_tolerance_from_cost'
that expands the inferred tolerance to include values inferred from the cost
and price amounts on those cost and price currencies.
Here is the doc for it:
Enable an EXPERIMENTAL feature that expands the maximum tolerance inferred
on transactions to include values on cost currencies inferred by postings
held at-cost or converted at price. Those postings can imply a tolerance
value by multiplying the smallest digit of the unit by the cost or price value
and taking half of that value.
For example, if a posting has an amount of "2.345 RGAGX {45.00 USD}"
attached to it, it implies a tolerance of 0.001 x 45.00 / 2 = 0.045 USD
and this is added to the mix to enlarge the tolerance allowed for units of
USD on that transaction. All the normally inferred tolerances (see
http://furius.ca/beancount/doc/tolerances) are still taken into account.
Enabling this flag only makes the tolerances potentially wider.
WARNING: This feature may go away in the future. It is an exploration to see
if it is truly useful.
- Added a dependency check for 'curses' library and updated documentation.
This is for OpenSuse which packages curses separately.
2015-05-22
- Fixed a simple bug (47) where the default value used for legacy tolerances
wasn't being initialized properly and added a test for it.
2015-05-20
- Fixed a potential important bug (48) just introduced a few days ago whereby
enabling the 'account_rounding' option would inadvertently disable balance
checks. Thanks to Matthew Harris for finding and reporting it.
- Added a 'dedent' option to both beancount.parser.parser.parse_string() and
beancount.loader.load_string(). This is a convenience for writing quick
tests.
2015-05-17
- MAJOR CHANGES: Precision and tolerances in Beancount have been entirely
revamped. The logic used to compute tolerances on transaction balances and
balance assertions is new. This previously discussed proposal:
http://furius.ca/beancount/doc/proposal-rounding
has been implemented. The new behaviour is documented here:
http://furius.ca/beancount/doc/tolerances
Here is a high-level summary of changes (see doc above for details):
* The tolerance used for transaction balances is inferred automatically from
the amounts on the transactions themselves. When not possible, a default
tolerance is looked up from values you can provide (as well as a global
default).
* There is a new 'default_tolerances' option to provide tolerances to be
used used when it cannot be inferred automatically. You use it like this:
option "default_tolerance" "*:0.005"
option "default_tolerance" "INR:0.5"
...
* The 'tolerance' option has been deprecated. It now has no effect.
* Balance assertions and Pad directives use a similar inference mechanism,
where the tolerance is based on the precision of the expected number in
the input.
* Rounding errors may be automatically posted to a special account (new
postings are inserted during parsing where necessary). This is disabled by
default; you enable this by setting the new 'account_rounding' option to
the account you want the rounding errors to be posted to:
option "account_rounding" "Equity:Rounding-Errors"
* These changes may cause some of your transactions to report as not
balancing, because the tolerances and now inferred automatically and the
inferred tolerances may be smaller than the fixed value we used
previously. YOU NEED TO MAKE SUITABLE ADJUSTMENTS TO YOUR INPUT FILE.
However, because this might cause some minor disruption, I have provided
an option to allow users to revert Beancount to its previous tolerance
behavior for a few months:
option "use_legacy_fixed_tolerances" "TRUE"
* There is a new EXPERIMENTAL feature which makes Beancount accept an
explicit tolerance value on balance assertions, like this:
YYYY-MM-DD balance <account> <number> ~ <tolerance> <currency>
For example,
2015-05-17 balance Assets:Vanguard 23.2134 ~ 0.002 RGAGX
By default, this feature is not enabled; this is being tested out and it
might go away eventually. To enable it, add the following experiment flag:
option "experiment_explicit_tolerances" "TRUE"
- The 'plugin' option has now been deprecated and will issue a warning if you
use it. You must now use the 'plugin' directive instead.
- The parser now supports boolean types, with this syntax: TRUE, FALSE. This
can be useful to set metadata fields.
- Postings that were inserted automatically have some metadata set on them to
distinguish this fact ('__automatic__').
2015-05-12
- Allow currency names up to 24 characters (not 12). This is often required
for people using financial options.
2015-05-11
- Added parsing of boolean types: TRUE, FALSE will produce values of
Python "True" and "False". This is useful to used as metadata values.
- Removed extraneous reference release in key_value_value in parser.
2015-05-03
- Added Equity to the list of accounts that count as proceeds for
beancount.plugins.sellgains plugin. This is so that old transactions
may use an unknown equity account and still pass the sellgains check.
2015-04-19
- Fixed a bug with the pad directive that did not look for account name
boundaries when looking at a parent account.
- In output of bean-doctor context, now print the residual amount.
2015-04-18
- Implemented limited support for arithmetic expressions. For now, only
support division. So A/B where "A" and "B" are numbers should now work.
In order to support full arithmetic expressions, we need to reimplement
the lexer in order to distinguish between "*" as a TAG or as a multiplication
token, and operator precedence to deal with terms vs. exprs. This will
probably be done soon. (I need division for a particular task right now, so
it is being added in a backwards-compatilble way).
2015-04-18
- Merged a new plugin (beancount.plugins.sellgains) that automatically
cross-checks the purchase price of a lot against all the other legs that are
in non-income accounts. This is useful to verify the correctness of price
annotations on reductions at cost, because not only the transactions have to
balance in cost (that is, including the income portion), they then also have
to balance without the income portion, using the price you attached (if you
somehow reported an incorrect price you will get an error).
This is best explained with an example:
1999-07-31 * "Sell"
Assets:US:BRS:Company:ESPP -81 ADSK {26.3125 USD} 26.4375 USD
Assets:US:BRS:Company:Cash 2141.36 USD
Expenses:Financial:Fees 0.08 USD
Income:US:Company:ESPP:PnL -10.125 USD
The regular balance verification checks all the legs against the cost basis:
-18 x 26.3125 + 2141.36 + 0.08 - 10.125 = 0 USD
This is the default behavior, and is still required. With the sellgains
plugin, an additional check also verifies that
-18 x 26.4375 + 2141.36 + 0.08 = 0 USD
Note how the price (26.4375) is used instead of the cost basis, and how
income legs are ignore in the calculation.
Transactions with reductions without a price and without an income leg will
be ignored, so only those which can be verified will be verified.
2015-04-13
- Beancount now allows the cost to the set to zero. This is useful for stock
options that one receives from an employer that have no value attached to
them.
Additionally, I'm considering allowing negative cost as well, which may be
necessary to handle adjusted cost bases, but I haven't added this just yet.
I'm not 100% convinced (adjustments are probably always upward).
2015-04-16
- Added a new command to bean-doctor: "linked" which prints out all
transactions linked to the one at a given line number, in a format Emacs can
grok with filename and location like errors so you can easily navigate
between the linked transactions.
Correspondingly, added a "bean-linked" command to the Emacs support mode.
2015-04-13
- A minor improvment to bean-format has been implemented: Whitespace before
postings is now automatically normalized to the most frequent spacing
present in the document. This allows you to be somewhat sloppy on the
postings themselves when entering new transactions.
2015-02-16
- Finally completed documentation on how to export your portfolio to Google
Finance, as
http://furius.ca/beancount/doc/export,
or equivalently as
https://docs.google.com/document/d/1eZIDRmQZxR6cmDyOJf7U3BnCm4PDMah2twxYFfKPJtM/
2015-02-10
- Revamped the export-to-OFX report to showcase a more explicit, flexible and
easily debuggable configuration. Also supports the declaration of money
instrument equivalents within the file itself. This will be documented here:
https://docs.google.com/document/d/1eZIDRmQZxR6cmDyOJf7U3BnCm4PDMah2twxYFfKPJtM/
2015-02-03
- Fixed an unreported bug with Emacs support whereby a number appearing within
a narration string would get aligned along with the other amounts.
2015-02-01
- Implemented an "include" directive. The include directive does not parse the
files in order... rather, each parse run on an input file accumulates the
list of includes to be parsed and processed the includes after parsing the
file. Because we're order independent, this should work. Note however that
the options should be set in the root file.
This is documented here:
https://docs.google.com/document/d/1wAMVrKIA2qtRGmoVDSUBJGmYZSygUaR0uOMW1GV3YE0/edit#heading=h.86lelow4097r
2015-01-24
- I created a new branch 'ledgerhub' to provide a version of Beancount that i
guaranteed to be compatible with LedgerHub.
Previously, LedgerHub was synchronizing a copy of some of the files under
beancount.core in order to avoid depending on Beancount. This turned out to
be difficult to maintain, as Beancount is still evolving relatively fast. So
I've decided to return to the original state of affairs: LedgerHub will
depend on a Beancount install in order to reuse its core libraries. (The
original reason for not doing so was that I thought I'd want to fork
those core libraries to support Ledger-specific features, but at this point
it's dubious whether I will need that at all --or whether I couldn't just
add generic support in Beancount's core libraries themselve to do that-- and
this will be an easier way to synchronize the two codebases going forward).
If you're a LedgerHub user, please checkout Beancount at its 'ledgerhub'
branch and make your PYTHONPATH depend on that.
2015-01-22
- Added a dependency check for PLY and documented in the Install doc
(http://furius.ca/beancount/doc/install).
2015-01-18
- Added a new, optional "Commodity" directive, which can be used to declare
commodity/currencies/instruments. The real purpose of this optional
directive is to hang per-commodity meta-data off of them. Some scripts are
much simplified by being able to define custom attributes for each
commodity, e.g. "name", "asset-class", "currency-exposure", and such. There
is a new function beancount.core.getters.get_commodity_map() which can be
used to gather a map of commodity to metadata.
You otherwise don't have to add those declarations; the new directive is
entirely optional and its addition should not break any existing feature
nor input file. It is an error to insert duplicate Commodity directives
for the same commodity name. I'm mainly doing this now so I can migrate
the portfolio dashboard script from experiments into the mainline while
configuring it using metadata.
The directive is documented here:
http://furius.ca/beancount/doc/language#heading=h.a3si01ejc035
- I added a development dependency on pyflakes and validated all code on
'default' to pass a flawless pyflakes run.
2015-01-17
- I added a paragraph about Releases in the Install document. Basically, I
don't do releases: the "default" branch is stable, just clone on "default"
and update regularly, I do all "risky" feature development in branches and
only merge to "default" when fully stable. All major changes on "default"
are announced in this file.
2015-01-11
- **IMPORTANT** Fixed an inconsistency with the signs of total cost syntax,
like this:
<account> -1000 USD -1123.24 CAD
It was required to use the negative sign in order to get the correct value.
This makes no sense: per-unit cost and price amounts were unsigned, and total
cost values were only allowed to be positive. Also, Ledger requires all
costs and prices to be positive/unsigned, which could be confusing for
Ledger users. I had no particularly good reason to require the same sign
for total cost: this was only an oversight.
So I changed this so that total price, like total cost, per-unit price,
per-unit cost, are all positive/unsigned. An error is now raised i a price
or cost, total or per-unit, are provided negative. You need to convert your
input files to unsigned values, like this (for the above example):
<account> -1000 USD 1123.24 CAD
If you pull this change and have negative total prices, your input file will
break. In order to circumvent the change, you may set the
BEANCOUNT_ALLOW_NEGATIVE_PRICES environment variable. I will remove this
workaround in a few months.
(This bug reported by Matthew Harris.)
- Fixed a bug when total cost {{...}} syntax was being used with a negative
number of units: it would raise an error, although it was legal to do so.
For example, this is legal and was raising an error:
<account> -10 MSFT {{455.24 USD}}
It probably did not show up much because few people use the total cost
method, and the tests happened only to use positive numbers of units.
- Fixed issue 5: bug in beancount.plugins.leafonly plugin when posting to a
non-leaf account that had no Open directive. (This bug reported by Nathan
Grigg.)
2015-01-04
- Merged the 'shell' branch into 'default', providing a new tool to extract
information from a Beancount input file: bean-query. This is the first
revision of the SQL-like query language for Beancount. Documentation for it
is available under http://furius.ca/beancount/doc/query.
2014-12-28
- Merged various functions (in beancount.ops.lifetimes) used to automatically
compute the date intervals that particular commodities are in used. This is
intended to be used to write scripts to automatically fetch all the price
entries needed during the lifetime of a particular Ledger. A script has been
created under experiments (src/experiments/fetch-missing-prices.py) which
you should be able to modify to your favourite price fetcher.
- The 'export_holdings' report now exports an extra holding as a cash
equivalent mutual fund in order to take into account of the value of
holdings that could not be converted to symbols the online portfolio
manager software knows about. The cash position brings the portfolio to
close to its total value.
2014-12-26
- Merged the latest version of the 'display_context' branch into 'default'.
These changes make the DisplayContext object a factory for DisplayFormatter
objects which are then used to render numbers. This is a little more
sophisticated than the previous approach and will be reusable in the
rendering routines for the interactive SQL shell's result.
2014-12-25
- The 'metadata' branch has been merged into 'default'. Metadata is now
officially supported in Beancount.
* Metadata can be created on all directives and postings. An attribute of
'.meta' can be used to access all parsed metadata. The object is a dict
which can also have its fields accessed via attributes.
* The '.source' attribute on all directives has been removed and '.meta' now
replaces it. The 'filename' and 'lineno' attributes which were previously
available on the beancount.core.data.Source tuple have been folded into
the metadata dict under the same names.
* Metadata values can be of various datatypes and these will be parsed and
available in Python:
- Strings
- Account names (appears as str)
- Currency (appears as str)
- Dates (a datetime.date instance)
- Tags (appears as str)
- Numbers (a decimal.Decimal instance)
- Amount (a beancount.core.amount.Amount instance)
* Attributes without a value will be parsed and have a value of 'None'. If
an attribute is repeated multiple times, only the first value for this
attribute will be parsed and retained.
* Apart from the 'filename' and 'lineno' attributes, no other "special"
usage of metadata will be made by Beancount. Metadata is intended to be
for the user to attach arbitrary data to directives. The query language
will eventually support query arbitrary metadata attributes, but no special
meaning will be assumed.
2014-12-19
- Created a new "export_holdings" report type that spits out an OFX format of
the holdings that can be consumed by external portfolio definition services
such as Google Finance (for example). This is useful, as it allows you to
monitor changes in your portfolio from a mobile device.
Note that in some of these services, a commodity that isn't a valid ticker
symbol fails the import process. Also, a commodity that is a mutual fund
recorded in the OFX file as a stock will similar fail the import process. We
need to find a way to fetch this info from the file itself. When metadata
will get merged, we should be able to get it from the account names, where
we could attach a property to the account's corresponding Open directive.
In the meantime, and as a kludge to start using this right away, place a
note for each currency (at any date, in any account) with the text in the
following format:
YYYY-MM-DD note <account> "Export <commodity>: IGNORE"
YYYY-MM-DD note <account> "Export <commodity>: MUTUAL_FUND"
This will get removed later.
2014-12-07
- After much more work on the SQL query language (in 'shell' branch), I need
to access Inventory positions by index, so I removed the getitem ([])
operator from Inventory, which was unused in the codebase. (If you have
scripts this *may* affect you; translate calls to [] into calls to
.get_position()).
2014-11-29
- Completed work on calculating returns (merged branch 'returns' into
default). I created a new package called "projects" to hold these special
tasks, those that don't integrate in the plugins framework, and created a
beancount.projects.returns library that can be inovoked as a script, like
this:
python3 -m beancount.projects.returns <filename.beancount> ...
For the arguments, you will have to read the docstring. You essentially need
to provide two or three regular expressions to list the set of accounts
that define you portfolio as well as other lists of accounts that should be
considered internal flows. See the docstring for details. I will eventually
move much of that explanation into a Google Doc and detail it in more depth.
There is also a script that runs it on the example beancount file, as well
as examples in the unit tests.
2014-11-27
- Started the "booking" branch which will implement the
http://furius.ca/beancount/doc/proposal-booking proposal. The first step was
to add an argument to the Open directive to specify the default booking
method to use when there are either 0, 2 or more matching lots in an
inventory. The only value supported by this are currently "STRICT" (the
default) or "NONE" which effectively disables inventory booking and is
equivalent to Ledger's method of accumulating lots. The new syntax looked
like this and is backward compatible with previous synta for it:
YYYY-MM-DD open <account> [<commodities>] ["<booking-method>"]
Though the result may be inventories with positive and negative positions in
the same commodity, this is useful for working with accounts booked at their
average cost (until we have correct treatment of basis for those with
dedicated syntax for merging lots). The total number of units and the total
cost should be correct.
This state of the 'booking' branch was merged into the 'default' branch,
which is why it appears in this CHANGE entry.
2014-11-21
- Merged "metadata" branch into default, Beancount now parses the metadata but
does not store it anywhere. This was done so that early adopters can start
to enter metadata right away even if it is not usable yet.
2014-11-13
- Added a new "beancount.plugins.ira_contribs" module that automatically
inserts postings for tracking pre-tax money contributions.
2014-11-11
- The precision of rendered numbers isn't quantized nor subject to hard-coded
maximums anymore. I cleaned this up. There is a new "DisplayContext" object
that maintains the most common and maximum precisions to be used to render
each currency, and this precision is derived automatically from the numbers
used in the input file (the most common is the mode of the distribution of
numbers seen). The alignment of the printer's amounts is now also fixed (it
rendered correct transactions lately but alignment was a little borked), and
the output is now kept as tightly as possible close to the account names.
- Numbers don't render with commas by default anymore. There is a new option:
'render_commas' that you can set to have commas in numbers rendered in all
the routines that output numbers.
2014-11-09
- A new directive has been created: "plugin" that accepts the an optional
plugin-configuration string. Instead of
option "plugin" "<module>"
option "plugin" "<module>:<config>"
Now use:
plugin "<module>"
plugin "<module>" "<config>"
The plugin option will remain functional for a while. The configuration
string is as previously a free-form string whose meaning is to be defined by
the plugin itself.
- The parser now stores the distribution of the precisions of all the numbers
it parses, and this adds a little bit of overhead (about 4%). With a large
file, you might notice a little tiny bit of extra slowness. The reason I
created this is in order to automatically infer reasonable defaults for the
display precision (used in the SQL shell at the moment).
- Renamed the 'bean-query' tool to 'bean-report'. There is a good reason for
this: there is a new tool coming that will provide an command-shell to make
interactive queries on your ledger contents. This tool will be
called--appropriately--"bean-query". I decided that the two tools should be
separate: "bean-report" will be used to render custom-coded reports (nothing
changes), and "bean-query" will be used to build reports out of SQL-like
queries. I'm hoping that eventually bean-query functionality will entirely
subsume bean-report reports, but that is unclear at this point.
- Made default rendering for changes and balances of HTML reports by units().
Rendering all the lots was horrible in even moderately complex stock trading
accounts with a few positions. Now we aggregate by currency ("units").
In order for the user to be able to view the previously available lot
detail from the web interface, we now provide a link to the "context",
which is a link specific to a transaction that renders the balances for all
the accounts just before the transaction is applied, and just after.
- (Internal) Renamed beancount.core.complete to beancount.core.interpolate.
I've been using the term interpolate for a while now, in the docs, and it's
way more telling than complete.
(Beginning to summarize user-facing changes 2014-11-09.)