Beancount-reds-importers

Improvements

- improvements to pdfreader. [gary-roach]

Refactored pdfreader to be more extensible and added a mercurycards importer which uses the pdfreader to read credit card transactions.
- turn on emit_filing_account by default. [Red S]

Else, we are dependent on the order of postings determined by Python
- feat! : add generic json reader. [dev]

BREAKING CHANGE: old jsonreader.py is rename into schwabjsonreader.py
- fidelity_brokerage_csv importer (ofx is limited to 90 days) [Red S]
- minor: add deep_identify to ibkr. [Red S]
- ibkr flex query web API downloader (reds-ibkr-flexquery-download) [Red S]

the new command is: reds-ibkr-flexquery-download
- ibkr: add transfers; minor refactor. [Red S]
- IBKR add balance assertions for positions. [Red S]
- ibkr balance assertion for cash balance. [Red S]
- xml reader + ibkr flex query importer. [Red S]

Fixes

- silent exit if there's nothing to print. [Justus Pendleton]

Only print table header if there are actually accounts
that need updates.
- 114: rerun checks on push to PRs. [Red S]
- failing unit tests because of emit_filing_account change. [Red S]

- emit_filing_account was turned on by default in 07d972d, which broke
unit tests
- 109: raise notimplemented for get_transactions() [Red S]
- schwab_csv_brokerage: add more transaction types. [Red S]
- constrain Beancount dependency version. [dev]

The project doesn't work with beancount v3.
- '' in securities bug. [Red S]
- csvreader: strip before parsing dates. [Red S]
- wip, fix fidelity_cma_csv. [Red S]

TODO:
- this needs a pytest so tests don't go stale
- needs much more fixing, or removal
- remove blank ticker, common with brokerage-bank combos. [Red S]
- ibkr trade commissions fix. [Red S]
- add interest to schwab_csv_brokerage. [Red S]

Other

- Merge pull request 117 from hoostus/main. [Red S]

Only print table header if there are actually accounts that need updates
- Merge pull request 107 from dev590t/main. [Red S]

Support JSON File

New Features

- add genericpdf paycheck importer. [Ammon Sarver]
- add bamboohr paycheck importer. [Ammon Sarver]
- add pdfreader libreader importer. [Ammon Sarver]
- minor: add 'show_configured' in paycheck transaction builder. [Red S]

New Importers
- add CSV importer for Digital Credit Union (89) [Harlan Lieberman-Berg]
- add importer for Alliant Federal Credit Union (88) [Harlan Lieberman-Berg]
- add schwab csv credit line importer. [Red S]

Improvements

- minor: add_custom_postings to banking.py. [Red S]
- minor: add identification based on filename to schwab importers. [Red S]
- minor: overridable add_custom_postings() [Red S]
- minor: add tdameritrade to template.cfg. [Red S]


Fixes

- update requirements to add back lost packages. [Ammon Sarver]
- schwab_csv_creditline balance sign. [Red S]
- add file encoding support to csvreader. [Ad Timmering]
- only emit filing account metadata if configured 97. [Red S]
- one-file-per-account broke with smart_importer 97. [Red S]
- timestamp issue in balance assertions. [Red S]
- balance date on test. [Red S]
- schwab tests. [Red S]
- schwab doesn't use a header in their csv any more. [Red S]
- schwab_csv_checking format changed 91. [Red S]

Other
- enforce formatting with ruff. [Rane Brown]
- format with isort, use pyproject.toml. [Rane Brown]
- switch to ruff for linting (90) [Harlan Lieberman-Berg]
- style: reformat to 99 col width (previously 88 col) [Red S]

Improvements

- Handle the INVEXPENSE field (79) [Jacob Farkas]
* Handle the INVEXPENSE field
* Rename the 'expenses' property to 'invexpense' to match the OFX tag
- handle 'fee' ofx type in investments. [Red S]
- configurable balance assertion dates. [Red S]
- smarter balance assertion dates. [Red S]
We find the statement's end date from the OFX file. However, banks and
credit cards typically have pending transactions that are not included
in downloads. When we download the next statement, new transactions may
appear prior to the balance assertion date that we generate for this
statement. To attempt to avoid this, we set the balance assertion date
to either two days before the statement's end date or the last
transaction's date, whichever is later.
- needs_update: add --all-accounts. [Red S]

Fixes

- update requirements.txt; also replace == with >= 87. [Red S]
- 85, balance dates were returning function instead of date. [Red S]
- update unit tests to match smart date. [Red S]
- add 'xfer' to ofx types. [Red S]
- make "invexpense" optional with a smart default. [Red S]
- resolved several gotchas with balance assertion dates with ofx files. [Red S]
- 80 document xlrd requirement. [Red S]
xlrd is optional, and thus not in requirements.txt, but is needed for
xls files and for testing

New Features

- bean-download needs-update to determine accounts needing an update based on the latest
balance assertion in the journal. [Red S]

New Importers

- Fidelity cma csv importer (47) [kantskernel]
* adding support for importering Fidelity Cash Management Acct (checking). CSV.
* cleaned up importer and merged required csvreader change with the version from main

- add Schwab CSV Positions importer. [Red S]

Breaking Changes
- reorganized directories. Changes import lines. [Red S]

Change your import lines from:

1. For most importers
from beancount_reds_importers import fidelity
to:
from beancount_reds_importers.importers import fidelity

2. For institutions with alternative importers:
from beancount_reds_importers.importers.schwab import schwab_csv_balances, schwab_csv_brokerage

BREAKING CHANGE: users will need to change import lines

Improvements

- Improve Vanguard support. [Matt Brown]

Remove noisy Vanguard memos. Show Vanguard transaction type as payee.
- Add 401k support to investments. [Matt Brown]

Support 401(k) source sub-accounts in account config values.

Add some human-readable error handling if a config variable is missing.
- add get_tags to transactionbuilders (68) [Sarah E Busch]

* feat: add get_tags to banking.py

* feat: add get_tags to investments.py

* feat: add get_tags to paycheck.py
- schwab_csv_brokerage: more descriptive payee/memo. [Red S]
- tests: tests for schwab_csv_brokerage 59. [Red S]

With these tests, 59 is reproduced. Fix pending.
- overridable target account in banking importer. [Red S]
- get_available_cash now takes as input settlement_fund_balance. [Red S]

different institutions seem to compute this in different ways.
overridable method now includes settlement_fund_balance for use if
needed in importers
- use filing_account to file. [Red S]
- overridable main_account in banking. [Red S]
- overridable currency and date fields in csvreader. [Red S]
- remove double quotes from column headers. [Red S]
- SCB/UOB: add 'convert_currencies' as an option; fix conversions. [Red S]
- check for account number in csv files. [Red S]

added untested, commented out code for xls
- perform table extraction in csvreader; scbbank now uses it. [Red S]

Start at a given header, and extract a table until either the end of the
file or until a blank line occurs.

Useful high level extractor.
- scb/uob: enable custom header; xls/xlsx: respect header. [Red S]
- relax header finding condition in csv. [Red S]
- united overseas bank (uob) importers: card, srs. [Red S]
- standard chartered importers (bank, card) [Red S]
- united overseas bank csv importer. [Red S]
- xls reader (previous one was xlsx reader) [Red S]
- enable price conversions in banking transaction builder. See 38. [Red S]
- default currency to CURRENCY_NOT_CONFIGURED. [Red S]
- csvreader now can search for the header. [Red S]

Eliminates the need to skip a set number of head/tail lines.
- attempt to get currency from input file instead of config. [Red S]
- Improve 401k and Vanguard support [Matt Brown]

Fixes

- intergrate needs_update into bean-download cli. [Red S]
- file_account was returning '{ticker}'. refactored config vars. [Red S]

TODO: refactor setting config vars better. format_raw_account() is
closely related.
- uobbank importer: add unit tests, fix several issues 70. [Red S]

- decimal places
- still needs a full solution. The problem is, petl uses xlwt/xlrd to
read excel, which use excel datatypes, which we don't want, as we
want just the string, which we can then pass on to decimal.decimal
- not sure exactly how decimals are stored in excel

- don't end up with long floats
- ignore xls spurious warning 70. [Red S]

- ignore WARNING *** file size (92598) not 512 + multiple of sector size (512)
- vanguard importer: clean up repeated memo. [Red S]
- filter out XMLParsedAsHTMLWarning from ofxparse. 40. [Red S]

https://github.com/jseutter/ofxparse/issues/170
https://github.com/jseutter/ofxparse/pull/108
- schwab_csv_checking: add tests, and fixes 63. [Red S]

- fixed skipping "pending transactions" rows
- move Balance namedtuple from importers to banking.py
- 52 unit price immutable. [Red S]
- schwab_csv_brokerage transfers 59. [Red S]
- counter -> next(counter) 62. [Red S]
- add 'Qualified Dividend' type to Schwab CSV Brokerage 60. [Red S]
- allow file_date() to be called without initialization 61. [Red S]
- minor fixes to schwab_csv_brokerage. [Red S]
- unitedoverseas xls doesn't convert amounts in strings. [Red S]
- remove special chars from column names. [Red S]
- ugly hack to handle 41 until that is resolved upstream. [Red S]
- banking: don't call self.build_metadata unless there is metadata. [Red S]
- scbcard: ignore unposted. [Red S]
- securities not found handling IndexError exception not caught. [Red S]
- Merge pull request 77 from mariolopjr/main. [Red S]

Add buydebt which corresponds to CD for Fidelity
- add buydebt which corresponds to CD for fidelity. [Mario Lopez]

- 48 clean up get_balance_positions() [Red S]
- default currency to CURRENCY_NOT_CONFIGURED. [Red S]
- uobbank withdrawal/deposit. [Red S]
- typo in vanguard_screenscrape.py. [Wiebe Stolp]

Other

- bean_download: catch config not in file gracefully. [Red S]
- style: Short-circuit in investments initialize to reduce indentation. [Matt Brown]
- test: Add Vanguard 401(k) test (65) [Matt Brown]
- Add test for capitalonebank, which also demonstrates 57 (58) [Matt Brown]
- Importer.extract: Explicitly name args to Transaction (55) [Matt Brown]

This shows that payee and narration are switched, as documented.
https://github.com/beancount/beancount/blob/v2/beancount/core/data.py is
the reference. This confused my use of smart_importer completion from an
existing ledger and took me some time to figure out.

I think the arguments should be swithed but I can see this would be
difficult to do while maintaining backwards compatibility.
- Add a simple regression test for the ally importer. (56) [Matt Brown]

This uses a copy of transactions.qfx from example/.
- chore: directory structure. [Red S]

---------

New Importers

- add schwab_checking_csv. [Red S]
- importer: amazon gift card importer. [Red S]
- add Discover credit card csv importer. [Red S]
- importer: add capitalone 360 ofx. [Red S]
- add becu (Boeing Employees Credit Union) (34) [Patrick Baker]

Improvements

- Better missing security reporting (43) [thehilll]

* if get_ticker_info_from_id finds an id not present in fund_info.py try to use self.ofx.security_list
to report more useful information. At least in Fidelity ofx files both the symbol and security name
are present, so in most cases the required additions to fund_info.py are already in the file.

* Slightly better reportings...print a summary line before list of securities

* remove unused f string

* 1. Comments to explain what is going on
2. Key the extracted securities dict by CUSIP (not symbol) which matches what is found
in the transaction entries (not for bonds symbol generally matches CUSIP)
3. Values of dict are now a tuple of (symbol, cusip, name) which for stocks should be all
that is needed for fund_info.py

* Remove another unnecessary f string

- Convert value to str before applying regex in remove_non_numeric (32) [Balazs Keresztury]
- add overridable security_narration() method; use it in Fidelity. [Red S]
- add ability to use smart importer with investments (36) [Patrick Baker]
- bean-download accepts a comma separated institution list. [Red S]
- bean-download display. [Red S]
- add skip_transaction to banking. [Red S]
- minor: workday importer now specifies filing account. [Red S]
- minor: add overridable post process step for csv. [Red S]
- minor: add etrade to direct download template. [Red S]
- minor: Chase filename regex is now [Cc]hase. [Red S]
- minor: allow overriding payee in investments. [Red S]

Fixes

- fidelity rounding errors are frequently higher than current value. [Red S]
- date format wasn't getting set during bean-file. [Red S]
- 41 add 'check' to ofx transaction types. [Red S]
- 41 add 'payment' to ofx transaction types. [Red S]
for investment accounts with banking features
- 40 check if ofx file provides balances instead of assuming it does. [Red S]
- 37 document `filename_pattern` in the demo example. [Red S]
- schwab_csv: update csv fields; custom payee. [Red S]
- csvreader returned blank when skip_comments was not present. [Red S]
- handle vanguard annoyance. [Red S]
- workday: skip last row to zero. [Red S]
- banking: fix the payee/narration order issue caused by 9057fcf j 9057fcf: 33. allow importers to override payee and narration fields. [Red S]
- catch ofxparse exceptions. [Red S]
- fix; schwab_checking_csv: withdrawals should be negative. [Red S]
- schwab_csv bug fixes. [Red S]
- 33. allow importers to override payee and narration fields. [Red S]
- when matching ofx security_id to funds_db prefer a complete match, if none found fall back to substring (45) [thehilll]

Improvements

- util: bean-download: display download progress bars; pretty output. [Red S]
- util: bean-download template improvements. [Red S]

- separate template file
- working examples
- util: ofx_summarize: sort by date. [Red S]

Other

- build: deps quoting. [Red S]
- build: requirements; gitchangelog. [Red S]
- doc: add changelog, and gitchangelog config. [Red S]
- doc: gitchangelog exclude 'feat' [Red S]
- README updates. [Red S]
- refactor: print verbose. [Red S]