Indox

🚀 New Features:

Unstructured Data Handling:

Unstructured Library Integration: We’ve integrated the Unstructured library into our data loaders, enhancing the ability to process and manage unstructured data formats. This addition allows for more flexible and robust data handling, making it easier to work with diverse data sources.
New Splitter Added:

AI21 Splitter: Introducing the AI21 Splitter, a cutting-edge tool designed to enhance text segmentation capabilities. This splitter leverages advanced AI21 models to provide even more precise and context-aware splitting, improving the quality of downstream NLP tasks.
🛠️ General Improvements:

Optimized Data Loader Functionality: We have further refined the functionality of our data loaders, significantly improving their efficiency. These optimizations reduce processing time and enhance performance, ensuring smoother and more reliable data pipelines.
Indox continues to evolve, providing users with more powerful tools for managing data workflows. With this update, we're expanding the platform's capabilities to better handle unstructured data and offering more sophisticated text processing options.

Stay tuned for more updates as we continue to enhance the Indox platform!

🚀 New Features:

Unstructured Data Handling:

Unstructured Library Integration: We’ve integrated the Unstructured library into our data loaders, enhancing the ability to process and manage unstructured data formats. This addition allows for more flexible and robust data handling, making it easier to work with diverse data sources.
New Splitter Added:

AI21 Splitter: Introducing the AI21 Splitter, a cutting-edge tool designed to enhance text segmentation capabilities. This splitter leverages advanced AI21 models to provide even more precise and context-aware splitting, improving the quality of downstream NLP tasks.
🛠️ General Improvements:

Optimized Data Loader Functionality: We have further refined the functionality of our data loaders, significantly improving their efficiency. These optimizations reduce processing time and enhance performance, ensuring smoother and more reliable data pipelines.
Indox continues to evolve, providing users with more powerful tools for managing data workflows. With this update, we're expanding the platform's capabilities to better handle unstructured data and offering more sophisticated text processing options.

Stay tuned for more updates as we continue to enhance the Indox platform!

🚀 **New Features:**

**1. New Splitters Added:**
- We've expanded the range of text splitters available to provide more flexibility in data processing:
- `semantic_text_splitter`: A splitter that leverages semantic understanding for more meaningful text segmentation.
- `RecursiveCharacterTextSplitter`: Recursively splits text by character, ensuring finer control over text segmentation.
- `CharacterTextSplitter`: A simple character-based splitter.
- `MarkdownTextSplitter`: Specially designed for Markdown files, ensuring accurate and clean splitting based on Markdown syntax.

**2. New Vector Store Integration:**
- **Weaviate**: Weaviate has been added to our list of supported vector stores, offering more options for storing and retrieving vector embeddings. This integration allows for more sophisticated and scalable machine-learning workflows.

**3. Refactored Data Loaders and Connectors:**
- We’ve overhauled how data loaders and connectors work for enhanced performance and usability. The separation of these components has been optimized further, enabling smoother data handling and better performance. Bugs have been fixed, and the overall architecture has been streamlined to make the platform more robust and user-friendly.

🛠️ **General Improvements:**

- **Enhanced Flexibility and Optimization**: The updates provide more granular control over data management workflows, optimizing the performance and usability of data loaders and connectors. This enhancement allows users to create more customized and efficient data pipelines.
- **Broader Integration Capabilities**: With the addition of Weaviate and new text splitters, along with the continuous improvement of existing features, Indox is now more versatile than ever, catering to a broader range of use cases.

---

Enjoy the new features and enhancements in Indox v0.1.23! Dive in and explore the improved flexibility and expanded capabilities now available to you.

🚀 New Features:

Modular Data Loading and Splitting:
- **Data Loader and Data Connector Separation:** We've restructured the data handling process by separating the Data Loader and Data Connector modules. Previously, data loading and splitting were handled together; now, they can be performed independently, providing greater flexibility and control.
- **New Data Loaders Added:**
- **PyPdf2**
- **PyPdf4**
- **PdfMiner**
- **PdfPlumber**
- **OpenPyXl**
- **Docx**
- **Csv**
- **Txt**
- **Json**
- **Bs4**
- **Md**
- **Pptx**
- **Sql**
- **Joblib**
- **Scipy**

Enhanced Data Connectors:
- **New Data Connectors Added:**
- **WikipediaReader**
- **YoutubeTranscriptReader**
- **ArxivReader**
- **TwitterTweetReader**
- **GutenbergReader**
- **GithubClient & GithubRepositoryReader**
- **DiscordChannelReader**
- **GoogleDoc**
- **GoogleSheet**
- **GoogleDrive**
- **GoogleChat**
- **MapsTextSearch**

Expanded Embedding Model Support:
- **New Embedding Models Supported:**
- **Cohere**
- **Elasticsearch**
- **Clarifai**
- **GPT4All**
- **Ollama**

New Vector Store Integrations:
- **New Vector Stores Supported:**
- **MongoDB**
- **Pathway**
- **Milvus**

Direct Document Storage:
- **Database.add() Method:** You can now store documents directly using the `database.add()` method. This allows users to define different databases by calling Indox just once, streamlining the process and enhancing usability.

🛠️ General Improvements:
- **Enhanced Flexibility:** The separation of data loading and splitting functions enables more customized workflows and better data management.
- **Broader Integration Capabilities:** With the addition of new data loaders, connectors, embedding models, and vector stores, Indox is more versatile and adaptable to a wide range of use cases.

🚀 New Features:
- **MongoDB Integration**:
- Added support for MongoDB as a storage backend, allowing more flexible and scalable data management.

- **Weaviate Vectorstore Support**:
- Integrated Weaviate as a vector store option, enabling more efficient and scalable vector-based retrieval.

🛠️ Bug Fixes:
- **Unstructured Library Fixes**:
- Addressed several bugs related to the use of the `unstructured` library, improving the reliability and accuracy of data processing.

⚙️ Performance Optimizations:
- **Workflow Optimization**:
- Optimized the overall workflow to enhance performance, resulting in faster data retrieval and processing.

📝 General Improvements:
- **Code Refinement**:
- Various code improvements and refactoring for better maintainability and readability.

🚀 inDox Version 0.1.17 Release Notes

We're thrilled to announce the release of inDox version 0.1.17! This update focuses on performance improvements to make your experience even better.

✨ Enhancements:

⚡ Improved Performance
- Significant optimizations have been made to ensure the app runs faster and more smoothly.

💡 Lighter Installation
- The app is now lighter and easier to install via pip, reducing installation time and resource usage.

---

Get ready for the next update, which will include the exciting new feature: inDoxEval!

Thank you for your continued support and feedback. We're dedicated to enhancing inDox with every release!

AI Indox MachineLearning NLP OpenSource TechNews Update