Paper-qa

Highlights

Added a new metadata provider [OpenAlex](https://docs.openalex.org/) for scholarly work, researchers, institutions, journals, and research topics.

- Responses can include open access information and raw pdf locations.
- Doesn't require authentication, but does prioritize requests with an email in the `mailto` URL parameter, exposed as an environment variable `OPENALEX_MAILTO`

Implemented an opt-in bypass around the `litellm.Router` for LLM completions (see https://github.com/Future-House/paper-qa/pull/563)

What's Changed

* Fixed pickle-ability of `LiteLLMModel` by jamesbraza in https://github.com/Future-House/paper-qa/pull/560
* Refactoring `LiteLLMModel` before removing `Router` by jamesbraza in https://github.com/Future-House/paper-qa/pull/561
* Adds openalex client as a default client by nadolskit in https://github.com/Future-House/paper-qa/pull/555
* Moving to `setup-uv` and `hynek/build-and-inspect-python-package` in CI by jamesbraza in https://github.com/Future-House/paper-qa/pull/564
* Ability to bypass usage of `litellm.Router` by jamesbraza in https://github.com/Future-House/paper-qa/pull/563
* Propagating `hynek/build-and-inspect-python-package`'s output location to `pypa/gh-action-pypi-publish` by jamesbraza in https://github.com/Future-House/paper-qa/pull/565
* Downloading `Package` artifact for `pypa/gh-action-pypi-publish` by jamesbraza in https://github.com/Future-House/paper-qa/pull/566

**Full Changelog**: https://github.com/Future-House/paper-qa/compare/v5.1.1...v5.2.0

What's Changed

* Lock file maintenance by renovate in https://github.com/Future-House/paper-qa/pull/545
* Validating for broken index by jamesbraza in https://github.com/Future-House/paper-qa/pull/544
* Added example how to use ollama hosted models by grg-ffb in https://github.com/Future-House/paper-qa/pull/536
* Making parsing resistant to failed inference of citations by whitead in https://github.com/Future-House/paper-qa/pull/551
* Exposed log verbosity configuration function by jamesbraza in https://github.com/Future-House/paper-qa/pull/552
* Cleaned up log verbosity code by jamesbraza in https://github.com/Future-House/paper-qa/pull/554

New Contributors
* grg-ffb made their first contribution in https://github.com/Future-House/paper-qa/pull/536

**Full Changelog**: https://github.com/Future-House/paper-qa/compare/v5.1.0...v5.1.1

Highlights

In-housed rate limits management

- Centers on a moving window algorithm with either a Redis or in-memory state
- Supports dynamically defined rates for different models or providers.
- New bundled configurations for different OpenAI rate limit tiers
- Accomplished using new third party dependencies [`coredis`](https://github.com/alisaifee/coredis) and [`limits`](https://github.com/alisaifee/limits)

Refactored `Settings` to allow for increased flexibility

- Indexing
- Indexes can use relative paths, enabling sharing across machines
- Paper search now no longer rebuilds the index every invocation
- Index parameter now are grouped in `IndexSettings`
- This release begins a deprecation cycle for the original hyperparameters
- Index builds now have a `rich.Progress` bar
- Parsing
- Chunking and embedding can now be deferred to inference time
- Agents
- Agents now have a `max_timesteps` parameter to upper-bound trajectory length
- Default agent is now a simple tool calling agent (`ToolSelector`), instead of a deterministic sequence of tool calls ("fake" agent)

Several bug fixes centered on retry-able errors:

- Flaky Semantic Scholar and Crossref SSL errors and connection reset errors
- LLM completions and text embeddings

What's Changed

* Cleaning up 489's implementation by jamesbraza in https://github.com/Future-House/paper-qa/pull/503
* chore(deps): lock file maintenance by renovate in https://github.com/Future-House/paper-qa/pull/504
* chore(deps): lock file maintenance by renovate in https://github.com/Future-House/paper-qa/pull/506
* chore(deps): update all non-major dependencies by renovate in https://github.com/Future-House/paper-qa/pull/505
* Filtering two more `DeprecationWarning`s by jamesbraza in https://github.com/Future-House/paper-qa/pull/509
* Refactor to create `settings.agent.index` grouping by jamesbraza in https://github.com/Future-House/paper-qa/pull/510
* Removed extra `save_index` calls, and added missing `changed` by jamesbraza in https://github.com/Future-House/paper-qa/pull/513
* Not rebuilding `SearchIndex` every `paper_search` by jamesbraza in https://github.com/Future-House/paper-qa/pull/512
* Updated citation to arxiv preprint by whitead in https://github.com/Future-House/paper-qa/pull/514
* Aviary agent `max_timesteps` and fixed `test_gather_evidence_rejects_empty_docs` by jamesbraza in https://github.com/Future-House/paper-qa/pull/515
* Moved `reset_log_levels` to `usefixtures` by jamesbraza in https://github.com/Future-House/paper-qa/pull/517
* Decomposing `Answer.could_not_answer` by jamesbraza in https://github.com/Future-House/paper-qa/pull/516
* Fixing `IndexSettings.use_absolute_paper_directory` leading to relative index file paths by jamesbraza in https://github.com/Future-House/paper-qa/pull/518
* Moving `run_ldp_agent` to center on `RolloutManager` by jamesbraza in https://github.com/Future-House/paper-qa/pull/519
* Retrying on known Semantic Scholar flaky SSL error in `get_s2_doc_details_from_doi` by jamesbraza in https://github.com/Future-House/paper-qa/pull/522
* Converted `PyMuPDF` message to warning logs by jamesbraza in https://github.com/Future-House/paper-qa/pull/523
* `rich.Progress` bar for monitoring index builds by jamesbraza in https://github.com/Future-House/paper-qa/pull/521
* Better descriptions and log messages by jamesbraza in https://github.com/Future-House/paper-qa/pull/524
* Made it possible to skip chunking by whitead in https://github.com/Future-House/paper-qa/pull/526
* Retrying on `aiohttp.ClientConnectionResetError` by jamesbraza in https://github.com/Future-House/paper-qa/pull/529
* Add rate limits for LLMs and Embedding Models by mskarlin in https://github.com/Future-House/paper-qa/pull/520
* Disallowing confusing `None` from `IndexSettings.index_directory`, and `IndexSettings.get_named_index_directory` by jamesbraza in https://github.com/Future-House/paper-qa/pull/531
* Add router_kwargs in separate control flow step by mskarlin in https://github.com/Future-House/paper-qa/pull/532
* Propagating `AgentSettings.agent_type` default for synchrony by jamesbraza in https://github.com/Future-House/paper-qa/pull/533
* Adding retrying of `aembedding` if it fails by jamesbraza in https://github.com/Future-House/paper-qa/pull/535
* Add limits+coredis to mypy by mskarlin in https://github.com/Future-House/paper-qa/pull/537
* Lock file maintenance by renovate in https://github.com/Future-House/paper-qa/pull/534
* Controlling for `pymupdf` version in `test_pdf_reader_match_doc_details` VCR by jamesbraza in https://github.com/Future-House/paper-qa/pull/538
* Lock file maintenance by renovate in https://github.com/Future-House/paper-qa/pull/539
* Fixed yet another `api.semanticscholar.org:443 ssl:default` error via retrying by jamesbraza in https://github.com/Future-House/paper-qa/pull/540

**Full Changelog**: https://github.com/Future-House/paper-qa/compare/v5.0.10...v5.1.0

What's Changed

* Discovered Renovate `:automergeMinor` and preventing `openai` version bumps by jamesbraza in https://github.com/Future-House/paper-qa/pull/493
* Fixing `LitQATaskDataset` deserialization from config by jamesbraza in https://github.com/Future-House/paper-qa/pull/494
* chore(deps): update all non-major dependencies by renovate in https://github.com/Future-House/paper-qa/pull/498
* Broken reader ut by nadolskit in https://github.com/Future-House/paper-qa/pull/497
* Fixing `LitQATaskDatasetcompute_trajectory_metrics` crash with bad status extraction by jamesbraza in https://github.com/Future-House/paper-qa/pull/500
* For autogenerated `Router` kwargs, specifying `timeout` of 60-sec by jamesbraza in https://github.com/Future-House/paper-qa/pull/501

**Full Changelog**: https://github.com/Future-House/paper-qa/compare/v5.0.9...v5.0.10

What's Changed

* Fixing `tests/tests/cassettes` issue by using absolute path by jamesbraza in https://github.com/Future-House/paper-qa/pull/482
* Retrying on known Crossref flaky SSL error in `doi_to_bibtex` by jamesbraza in https://github.com/Future-House/paper-qa/pull/479
* Cleaning up and testing `get_directory_index` by jamesbraza in https://github.com/Future-House/paper-qa/pull/483
* Modernizing Renovate config by jamesbraza in https://github.com/Future-House/paper-qa/pull/487
* Allowing `parse_text` to be given a `str` path by jamesbraza in https://github.com/Future-House/paper-qa/pull/491
* Refactor to expose `agents.RichHandler` by jamesbraza in https://github.com/Future-House/paper-qa/pull/489

**Full Changelog**: https://github.com/Future-House/paper-qa/compare/v5.0.8...v5.0.9

What's Changed

* Documenting and cleaning up manifest file logic by jamesbraza in https://github.com/Future-House/paper-qa/pull/448
* Latest dependencies for `pylint` 3.3 by jamesbraza in https://github.com/Future-House/paper-qa/pull/463
* Down-pinning `openai` 1.47 since it breaks CI by jamesbraza in https://github.com/Future-House/paper-qa/pull/466
* Lock file maintenance by renovate in https://github.com/Future-House/paper-qa/pull/462
* chore: add .gitattributes for cassettes file by devstein in https://github.com/Future-House/paper-qa/pull/468
* Documenting Python 3.11+ in README by jamesbraza in https://github.com/Future-House/paper-qa/pull/467
* Fixing flaky `test_tool_failure` by jamesbraza in https://github.com/Future-House/paper-qa/pull/465
* Documenting manifest CSV pathing a bit more by jamesbraza in https://github.com/Future-House/paper-qa/pull/469
* Handling S2 `KeyError` crash during indexing by jamesbraza in https://github.com/Future-House/paper-qa/pull/472
* Fixing `pymupdf.mupdf.FzErrorFormat` crash by recasting as an `ImpossibleParsingError` by jamesbraza in https://github.com/Future-House/paper-qa/pull/474
* Updating `test_tool_failure` cassette by jamesbraza in https://github.com/Future-House/paper-qa/pull/476
* Simplifying the indexing of `action` tokens by jamesbraza in https://github.com/Future-House/paper-qa/pull/477
* Truncating failing `test_evaluation` via `max_rollout_steps` by jamesbraza in https://github.com/Future-House/paper-qa/pull/475

New Contributors

* devstein made their first contribution in https://github.com/Future-House/paper-qa/pull/468

**Full Changelog**: https://github.com/Future-House/paper-qa/compare/v5.0.7...v5.0.8