Pebblo

Key Highlights

* Support to request anonymization on a per-SafeLoader invocation
* Pebblo TextLoader support
* Updated Entity Classifier
* Added support for Azure Client Secret
* Added support for email-address
* Improved credit-card matching
* Sqlite state storage is now a supported and preferred option

Deprecation Notice

* File-based state storage will be deprecated in the next release, `v0.1.21`. It is recommended to switch to `type: db` in the pebblo configuration. See https://daxa-ai.github.io/pebblo/config#storage for more details.

What's Changed
* Revert docker_release.yml changes by srics in https://github.com/daxa-ai/pebblo/pull/551
* Add Ruff sort imports command to the format_diff target by Raj725 in https://github.com/daxa-ai/pebblo/pull/549
* added changes for documentation 0.19 by gr8nishan in https://github.com/daxa-ai/pebblo/pull/556
* added multi-arch build for docker. by rajburnwal07 in https://github.com/daxa-ai/pebblo/pull/553
* New field added client-secret for Azure client secret ID. by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/555
* Samples: text loader sample using PebbloTextLoader by Raj725 in https://github.com/daxa-ai/pebblo/pull/539
* Adding anonymize_snippet as a input filed in /loader/doc API by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/558
* Adding deprecation message for db storage type as file in config by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/559
* Added Pebblo Docs Version - 0.1.19 by rutujaac in https://github.com/daxa-ai/pebblo/pull/563
* docs: update README.md by eltociear in https://github.com/daxa-ai/pebblo/pull/562
* Added the last released versions of langchain-community to run tests by Rohan-sss1 in https://github.com/daxa-ai/pebblo/pull/561
* added sonarscan workflow. by rajburnwal07 in https://github.com/daxa-ai/pebblo/pull/566
* Adding anonymizeSnippets in reports config by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/569
* Returning default value if no config path is passed by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/571
* samples: Added readme file, metadata to Text loader sample by Raj725 in https://github.com/daxa-ai/pebblo/pull/565
* Update env var for Sonarscan workflow by rajburnwal07 in https://github.com/daxa-ai/pebblo/pull/570
* Add LinkedIn shield button by srics in https://github.com/daxa-ai/pebblo/pull/572
* Updated anonymizeSnippets location from classifier to reports in config.yaml by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/573
* Updating file deprecation msg by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/574
* Update LinkedIn URL by sridhar-daxa in https://github.com/daxa-ai/pebblo/pull/576
* Update pebblo version to 0.1.20rc1 by srics in https://github.com/daxa-ai/pebblo/pull/577
* DB query optimization and reducing sqlalchemy logs by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/575
* Added changes for Email Address and Credit Card Number by gr8nishan in https://github.com/daxa-ai/pebblo/pull/580
* Docs: table for Pebblo Safe loader args/parameters description by Raj725 in https://github.com/daxa-ai/pebblo/pull/568
* added changes for email address documentation by gr8nishan in https://github.com/daxa-ai/pebblo/pull/582
* Update pebblo version to 0.1.20 by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/583

New Contributors
* Rohan-sss1 made their first contribution in https://github.com/daxa-ai/pebblo/pull/561

**Full Changelog**: https://github.com/daxa-ai/pebblo/compare/v0.1.19...v0.1.20

Key Highlights
* Support for new Entities types:
* GitHub Finegrained Personal Access Token
* RSA, EC, DSA Private Keys
* OpenSSH Private Key
* Google Account Private Key
* Improved configuration validation
* Improvements to SQLite DB state storage subsystem
* Security fixes

What's Changed
* Ensure the build module is installed before running build command by Raj725 in https://github.com/daxa-ai/pebblo/pull/511
* Update Pebblo Docs Footer by rutujaac in https://github.com/daxa-ai/pebblo/pull/517
* 515 raise if --config is passed but file does not exist by aramnhammer in https://github.com/daxa-ai/pebblo/pull/516
* Upversion pebblo to 0.1.19rc1 by srics in https://github.com/daxa-ai/pebblo/pull/525
* Bugfix/514 move weasyprint dependency check to config validation by aramnhammer in https://github.com/daxa-ai/pebblo/pull/524
* added changes for github and slack token by gr8nishan in https://github.com/daxa-ai/pebblo/pull/512
* Confidence score changes for DB by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/523
* SQLite - Minor fixes by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/521
* Fix entity details response to return based on location order by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/528
* Remove pinned pkg versions to avoid dependency error by srics in https://github.com/daxa-ai/pebblo/pull/513
* Fix the ruff command error by Raj725 in https://github.com/daxa-ai/pebblo/pull/520
* DB load history by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/529
* Add support for private keys by gr8nishan in https://github.com/daxa-ai/pebblo/pull/527
* Fixed timeit issue by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/531
* Fixed text overflow on snippet details "snippet" string by rutujaac in https://github.com/daxa-ai/pebblo/pull/526
* Fix Data source path overflow issue by rutujaac in https://github.com/daxa-ai/pebblo/pull/500
* Dependabot fixes by Raj725 in https://github.com/daxa-ai/pebblo/pull/534
* Add lint-fix target by srics in https://github.com/daxa-ai/pebblo/pull/532
* added changes for new api for entity and topic classification by gr8nishan in https://github.com/daxa-ai/pebblo/pull/518
* Fixed the issue where app was being shown multiple times on local UI by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/535
* Added new Pebblo Docs version by rutujaac in https://github.com/daxa-ai/pebblo/pull/536
* Fix sorting on findings-total by rutujaac in https://github.com/daxa-ai/pebblo/pull/537
* Swagger changes in pebblo server APIs by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/530
* Adding mode knob in config under classifier by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/538
* Showing anonymized data in DB apps by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/533
* Adding classifier mode changes for file type storage by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/540
* Fixed integration tests by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/541
* Show zero state for App Finding tabs by rutujaac in https://github.com/daxa-ai/pebblo/pull/544
* Fix Security Issues by rutujaac in https://github.com/daxa-ai/pebblo/pull/542
* Show zero in tables on Local UI by rutujaac in https://github.com/daxa-ai/pebblo/pull/545
* Update version to 0.1.19 by srics in https://github.com/daxa-ai/pebblo/pull/546
* Fixing client_version error by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/547

New Contributors
* aramnhammer made their first contribution in https://github.com/daxa-ai/pebblo/pull/516

**Full Changelog**: https://github.com/daxa-ai/pebblo/compare/v0.1.18...v0.1.19

<img width="1656" alt="image" src="https://github.com/user-attachments/assets/166062f0-26cd-48fc-a34b-19d9a6fd1b90">


Key Highlights

* Confidence score in Local UI for document snippets
* IP address as a supported entity
* Improved Entity Classifier accuracy
* SQLite DB as an option for state storage (beta)
* Stability improvements in Pebblo Client in Langchain
* Versioned Pebblo documentation

What's Changed
* Update docker-bake.hcl by siddheshwar-more in https://github.com/daxa-ai/pebblo/pull/456
* made changes to docs for prompt governance by gr8nishan in https://github.com/daxa-ai/pebblo/pull/457
* added changes to merge classes NDA_AGREEMENT, SETTLEMENT_AGREEMENT, D… by gr8nishan in https://github.com/daxa-ai/pebblo/pull/458
* Adding test log config in UTs by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/461
* Integration test fix by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/462
* Adding client version to report.json file by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/465
* Setup: Documentation Versioning by rutujaac in https://github.com/daxa-ai/pebblo/pull/464
* Pebblo 0.1.18 to main by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/486
* Pebblo Docs - Hide "Version not actively maintained" note by rutujaac in https://github.com/daxa-ai/pebblo/pull/487
* Sqlite Changes by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/472
* Retrieval App Bug Fix by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/495
* Added documentation for LlamaIndex Pebblo Safe DataLoader by yograjopcito in https://github.com/daxa-ai/pebblo/pull/469
* Update client version field in PDF reports and local UI app details page by rutujaac in https://github.com/daxa-ai/pebblo/pull/463
* DB Retriever app prompt with findings fix by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/496
* Loader doc db response by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/499
* Fixed version error in case of empty clientVersion by rutujaac in https://github.com/daxa-ai/pebblo/pull/501
* Fixed loader/doc issues w.r.t. loadId by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/504
* Fixing loader API changes by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/502
* added slack notification in case of build failure. by rajburnwal07 in https://github.com/daxa-ai/pebblo/pull/490
* Added new version - python 3.12.5 for UTs by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/509
* Updated pydantic version and relevant model changes by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/505
* Update pebblo version to 0.1.18 by srics in https://github.com/daxa-ai/pebblo/pull/510
* Add Langchain tab in Pebblo Docs by rutujaac in https://github.com/daxa-ai/pebblo/pull/497
* Fix Snippets Listing Page - Snippets Count issue by rutujaac in https://github.com/daxa-ai/pebblo/pull/503
* Documentation for storage type by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/508
* Confidence Score fix to highlight only the entity which is present in the label name by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/498
* Updated doc for 0.1.18 version by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/506

New Contributors
* rajburnwal07 made their first contribution in https://github.com/daxa-ai/pebblo/pull/490

**Full Changelog**: https://github.com/daxa-ai/pebblo/compare/v0.1.17...v0.1.18

<img width="1340" alt="image" src="https://github.com/user-attachments/assets/88ada5b1-1c4a-40ac-a31b-645781c04802">
<img width="1334" alt="image" src="https://github.com/user-attachments/assets/95c657ad-155a-456a-afbc-889e786acfab">


Key Highlights

* Prompt Governance (beta) to find and flag restricted entities like secret-keys in prompt text
* Checksum based RAG context to document source mapping
* Logging enhancements, to both console and file logs with rotation
* Docker config updates
* Documentation updates based on user community feedback
* Security fixes for dependency packages
* Enhanced samples including for SharePoint-Postgres


What's Changed
* Pinned python 3.12 version for UTs by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/401
* Excluding RAG apps from ruff check by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/405
* docs: Add SharePointLoader and Postgres PGVector supported list by Raj725 in https://github.com/daxa-ai/pebblo/pull/404
* Replacing doc with checksum by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/402
* Updated samples: Added SharePoint-Postgres, Rearranged existing samples, Updated docs by Raj725 in https://github.com/daxa-ai/pebblo/pull/406
* docs: update README.md by eltociear in https://github.com/daxa-ai/pebblo/pull/407
* Sharepoint samples: Updated readme, requirements and token check by Raj725 in https://github.com/daxa-ai/pebblo/pull/410
* Remove text-capitalise on app names on Pebblo UI by rutujaac in https://github.com/daxa-ai/pebblo/pull/408
* Index for sample apps by Raj725 in https://github.com/daxa-ai/pebblo/pull/409
* Update README and requirements for samples by srics in https://github.com/daxa-ai/pebblo/pull/411
* Fix for blank retrieval app page by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/413
* Pinned python-bidi version to 0.4.2 by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/419
* Sort Apps on app listing page based on findings by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/417
* Add Delete App by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/416
* Fix Dependabot findings in Pebblo samples by Raj725 in https://github.com/daxa-ai/pebblo/pull/422
* Fix Dependabot findings in Pebblo by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/423
* Docker latest tag from pipeline by siddheshwar-more in https://github.com/daxa-ai/pebblo/pull/418
* Display pebblo version by srics in https://github.com/daxa-ai/pebblo/pull/426
* Pebblo UI - Delete App UI by rutujaac in https://github.com/daxa-ai/pebblo/pull/428
* Docker pebblo config to .pebblo by siddheshwar-more in https://github.com/daxa-ai/pebblo/pull/431
* Update pyproject.toml by srics in https://github.com/daxa-ai/pebblo/pull/435
* Updating pebblo version by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/439
* Update tests.yml by siddheshwar-more in https://github.com/daxa-ai/pebblo/pull/442
* Enhanced PDF reporting docker pebblo by siddheshwar-more in https://github.com/daxa-ai/pebblo/pull/420
* Pebblo UI : Fix http to https auto upgrade issue by rutujaac in https://github.com/daxa-ai/pebblo/pull/412
* Delete App for Retrieval Apps by rutujaac in https://github.com/daxa-ai/pebblo/pull/438
* Docker docs update by siddheshwar-more in https://github.com/daxa-ai/pebblo/pull/440
* Log messages updated by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/441
* Sort prompt retrievals based on time by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/447
* Sort Loader apps based on total findings by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/446
* Adding langchain client version by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/414
* Add combined file and console logger by srics in https://github.com/daxa-ai/pebblo/pull/433
* Prompt governance by gr8nishan in https://github.com/daxa-ai/pebblo/pull/430
* Word break: all if overflows column width by rutujaac in https://github.com/daxa-ai/pebblo/pull/382
* Added changes for try catch in al app retrieval by gr8nishan in https://github.com/daxa-ai/pebblo/pull/451
* Updating pebblo config and logs by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/427
* Update docs for cacheDir and logging changes by srics in https://github.com/daxa-ai/pebblo/pull/453
* Update release version to 0.1.17 by srics in https://github.com/daxa-ai/pebblo/pull/454
* Update Pebblo client key by rutujaac in https://github.com/daxa-ai/pebblo/pull/452
* Revert "Update Pebblo client key" by srics in https://github.com/daxa-ai/pebblo/pull/455

New Contributors
* eltociear made their first contribution in https://github.com/daxa-ai/pebblo/pull/407

**Full Changelog**: https://github.com/daxa-ai/pebblo/compare/v0.1.16...v0.1.17


Key Highlights

1. Support for Sharepoint loader with authorized identities and extended metadata
2. Updated `pebblo-classifier` to Version 9 with improved topic accuracy
3. Support for Postgress PGVector VectorDB for identity and semantic-topic enforcement
4. Documentation updates for `PebbloRetrieval` chain with samples for identity and semantic-topic enforcement

What's Changed
* Adding lint fixes by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/380
* Updated UT and minor linting fixes by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/381
* Updated documentation for Pebblo UI by rutujaac in https://github.com/daxa-ai/pebblo/pull/387
* Documentation for PebbloRetrievalQA chain by Raj725 in https://github.com/daxa-ai/pebblo/pull/388
* SafeRetriever docs and README updates by srics in https://github.com/daxa-ai/pebblo/pull/389
* Updating field validator for prompt API by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/379
* Handle bubble chart height for last number of finding data by rutujaac in https://github.com/daxa-ai/pebblo/pull/383
* Skip pkg cache to avoid hash mistmatch by srics in https://github.com/daxa-ai/pebblo/pull/384
* Fix test pypi pkg naming to include yr.mo.dt.hh.mm by srics in https://github.com/daxa-ai/pebblo/pull/385
* Add GoogleDriveLoader to the list of supported loaders by Pebblo by rahul-trip in https://github.com/daxa-ai/pebblo/pull/390
* Cleaning up discovery API response by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/396
* added changes to integrate v9 version of topic classifier by gr8nishan in https://github.com/daxa-ai/pebblo/pull/397
* Updating sensitive data for different classification by dineshkrr in https://github.com/daxa-ai/pebblo/pull/394
* Update langchain library versions and imports in the sample apps by Raj725 in https://github.com/daxa-ai/pebblo/pull/398
* Add docs for Semantic Enforcement RAG using PebbloRetrievalQA by Raj725 in https://github.com/daxa-ai/pebblo/pull/392
* add sharepoint retriever app. by rahul-trip in https://github.com/daxa-ai/pebblo/pull/399
* Update release version by srics in https://github.com/daxa-ai/pebblo/pull/400

New Contributors
* dineshkrr made their first contribution in https://github.com/daxa-ai/pebblo/pull/394

**Full Changelog**: https://github.com/daxa-ai/pebblo/compare/v0.1.15...v0.1.16



Key Highlights

* Dedicated `Safe Retriever` Local UI page for Visualization of AI apps using `PebbloRetrievalChain`
* Visualize top users using LLM inference, including their `authorization groups`
* Visualize top documents retrieved from Vector DB to facilitate data governance and compliance
* Kubernetes deployment of pebblo server docker image
* Updated samples with upstreamed `langchain-google` pkg for authorization enabled `GoogleDriveLoader`
* Extended metadata support for `GoogleDriveLoader` - document paths, names, and sizes
* Initial Pebblo support for LlamaIndex
* Topic and Entity name enhancements in pebblo-schema


What's Changed
* Log and print statement were not working in topic classfiier by gr8nishan in https://github.com/daxa-ai/pebblo/pull/309
* Changes for showing pebblo server and client version on DAXA UI. by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/319
* Entity Classifier Fix by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/325
* Removed text capitalize on file name in local UI by rutujaac in https://github.com/daxa-ai/pebblo/pull/327
* Updating entities and Topics names from human readable format to programming format by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/306
* Added keyword to string mapping in reports and ui by rutujaac in https://github.com/daxa-ai/pebblo/pull/326
* Example for Llama PebbloSafeReader by yograjopcito in https://github.com/daxa-ai/pebblo/pull/313
* Docker repo fix by siddheshwar-more in https://github.com/daxa-ai/pebblo/pull/328
* Use GoogleDriveLoader from langchain_google_community by Raj725 in https://github.com/daxa-ai/pebblo/pull/337
* Added make command to format all/changed files by Raj725 in https://github.com/daxa-ai/pebblo/pull/340
* Update rag.md for pebblo environment variable by rahul-trip in https://github.com/daxa-ai/pebblo/pull/348
* Update Identity and Semantic enforcement samples (PebbloRetrievalQA) by Raj725 in https://github.com/daxa-ai/pebblo/pull/335
* Docs and version updates by srics in https://github.com/daxa-ai/pebblo/pull/357
* Remove spell check in CI lint pipeline by srics in https://github.com/daxa-ai/pebblo/pull/367
* Unit test trigger on tests by siddheshwar-more in https://github.com/daxa-ai/pebblo/pull/366
* Pebblo k8s deployment by siddheshwar-more in https://github.com/daxa-ai/pebblo/pull/362
* Topic classifier fixes - Normal text in findings, Lint errors by Raj725 in https://github.com/daxa-ai/pebblo/pull/370
* Pebblo redirection and download report URL fix by rutujaac in https://github.com/daxa-ai/pebblo/pull/354
* Pebblo Safe Retriever details in Local UI by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/353
* Adding python dateutil package by dristysrivastava in https://github.com/daxa-ai/pebblo/pull/374
* Fix Local UI issues by rutujaac in https://github.com/daxa-ai/pebblo/pull/375
* Update samples: Import PebbloRetrievalQA from langchain_community by Raj725 in https://github.com/daxa-ai/pebblo/pull/364
* Capitalize app names for local UI by rutujaac in https://github.com/daxa-ai/pebblo/pull/376
* source_aggr_size renamed to source_aggregate_size in loader/doc API. by shreyas-damle in https://github.com/daxa-ai/pebblo/pull/378
* Add sort by date recent for active users and docs by rutujaac in https://github.com/daxa-ai/pebblo/pull/377

New Contributors
* gr8nishan made their first contribution in https://github.com/daxa-ai/pebblo/pull/309
* yograjopcito made their first contribution in https://github.com/daxa-ai/pebblo/pull/313

**Full Changelog**: https://github.com/daxa-ai/pebblo/compare/v0.1.14...v0.1.15