Repopack

This update brings a significant new feature that enhances the tool's utility for working with large language models.

New Features

Token Counting Support (39)

Thanks to the fantastic contribution from joshellington, Repopack now includes token counting functionality:

- Added token counts for individual files and the entire repository
- Updated CLI output to display token information in the summary and top files list

This feature is particularly useful for users working within context limits of large language models like GPT-4.

A huge thank you to joshellington for this valuable contribution!

Improvements

Enhanced ignore files support (34, 35)
- Updated the file filtering logic to consider `.gitignore`, `.repopackignore` files in all directories, not just the root.
- This change ensures that all ignore files, regardless of their location in the project structure, are properly respected during the file filtering process.

Notes
- This update improves the handling of ignore files in your project structure. You may notice changes in which files are included or excluded if you have ignore files in subdirectories.

Changes

gitignore Syntax Compliance
- Fixed handling of `.gitignore` and `.repopackignore` files to properly comply with gitignore syntax.
- This ensures more accurate and consistent file filtering based on your ignore patterns.

Change File Filtering Logic
- Migrated custom ignore pattern processing from gitignore syntax to glob patterns.

Notes

- This update may affect which files are included/excluded in the packed output. We recommend reviewing your ignore patterns to ensure desired behavior.
- If you're using a custom configuration, make sure to update your `ignore.customPatterns` to use glob patterns instead of gitignore syntax.

It's focusing on improved file filtering capabilities and overall performance enhancements.

New Features

Support `include` (22, 30)
- Introduced the `--include` CLI option for specifying files to include using glob patterns.
- Added support for `include` patterns in the configuration file.

To pack specific files or directories using glob patterns:
bash
repopack --include "src/**/*.ts,**/*.md"


Special thanks to IsaacSante for their contributions.

Improvements

Performance Optimization
- Replaced the `ignore` package with `globby` for more efficient file filtering.

It's focusing on security improvements and output refinements.

Improvements
Security Enhancements: Exclude Suspicious Files
- Implemented a filter to exclude potentially sensitive files from the output.
- This improvement prevents the inclusion of suspicious files, enhancing overall security.

Fixes
Prevent Recursive Output Issues
- Modified the `getFilePaths` function in `src/core/packager.ts` to exclude the output file from processing.
- This fix resolves a recursive issue where the output file was being included in itself.

Changes
XML Escaping Removal
- Removed XML escaping for file paths and contents in repository files.
- This change improves readability and AI comprehension of the generated output.

Note: This update assumes that input data (file paths, contents, etc.) does not contain characters that would break XML syntax. If there's a possibility of such characters, we may implement a more robust solution in the future, such as using CDATA sections for file contents.

Our CLI has learned the art of style. It's now fluent in both plain and XML.

What's New

- Added `--style` option to CLI
- Choose between `plain` and `xml` output styles
- Usage: `repopack --style xml`

Happy packing!