Pytokencounter

This update improves file handling within the `ReadTextFile` function by adding a check for empty or blank files. Previously, if a file was empty, `chardet` would incorrectly raise an error when attempting to detect the encoding, as the content was `None`. This fix ensures better robustness and prevents unnecessary failures in workflows processing multiple files.

**Key Updates & Enhancements**
- **Improved File Handling**
- Added a check in `ReadTextFile` to handle empty or blank files gracefully.
- Prevents `chardet` from raising an encoding detection error on empty files.
- Ensures smoother processing when handling diverse file inputs.

**Bug Fixes**
- Fixed an issue where empty files caused encoding detection failures.

This update improves token count aggregation in directory and file processing, ensuring accurate and structured JSON output.

Key Updates & Enhancements

Improved Token Count Aggregation
- `GetNumTokenDir` now includes a `"numTokens"` key at the top level, correctly summing all files and subdirectories while respecting the `recursive` parameter.
- `GetNumTokenFiles` properly propagates `mapTokens` when processing directories, ensuring accurate aggregation of `"numTokens"` for nested structures, also following the `recursive` parameter logic.

This update improves JSON output handling for CLI functions, ensuring consistent and structured formatting across results.

Key Updates & Enhancements

Improved JSON Formatting
- Functions returning `OrderedDict` or `list` are now properly formatted using `json.dumps(..., indent=4)`.
- Standardized JSON output across CLI functions for improved readability.

Bug Fixes
- Fixed inconsistencies in output formatting.
- Prevented unnecessary JSON indentation for non-JSON return values.

This release reintroduces the list-based exclusion mechanism that was removed in **v1.6.1**. Files with unsupported extensions are now bypassed for enhanced performance, while decoding error handling remains in place to manage unreadable files.

Key Changes

- **List-Based Exclusion:** Quickly skips files with known unsupported extensions.
- **Decoding Error Handling:** Continues to safely bypass files that trigger errors.
- **Optimized Performance:** Balances efficiency and robustness for improved token counting.

Summary

Enhances processing efficiency while maintaining high reliability.

This update improves handling of binary/unreadable files when `excludeBinary=True`. Instead of relying on a predefined list of extensions, the implementation now catches decoding errors and skips the file if it cannot be read as text.

Key Updates & Enhancements

Improved Binary Handling
- When `excludeBinary=True`, files that cannot be decoded are skipped instead of relying on an extension-based exclusion list.
- Prevents incorrect exclusions and ensures that unreadable binary files are properly skipped.

Bug Fixes
- Fixed an issue where some files were incorrectly allowed despite being unreadable, causing errors.
- Resolved cases where certain valid text files were mistakenly excluded.
- Improved error handling and logging to indicate when a file is skipped due to decoding failure.