Implementation of an Automated Information Search System with Artificial Intelligence

 

Implementation of an Automated Information Search System with Artificial Intelligence

Author: Francisco Prats Quílez

Introduction

In the information age, efficient document management is crucial for business productivity. Organizations accumulate large volumes of data in the form of documents, making it imperative to have advanced tools to manage and extract relevant information quickly and accurately. This study analyzes the implementation of an automated information search system in documents using local artificial intelligence models, integrated into the Project Manager AI application.

Objective

The objective of this project is to develop an automated solution that enables the efficient management and search of information in documents stored in various formats (docx and pdf). The goal is to optimize the process of loading, monitoring, querying, and updating documents using advanced natural language processing (NLP) techniques, particularly large language models (LLM) and vector databases.

Development

1. Creation and Selection of Workspace
• The process begins using the Project Manager AI application in the document management section.
• The user can create a new workspace or select an existing one, specifying a path where the documents will be loaded.

2. Document Loading
• Documents in docx and pdf formats are converted to vectors and stored in a Pinecone vector database.
• The interface allows monitoring documents, adding new ones, deleting or opening existing ones, and performing searches by document name.
3. Updating and Monitoring
• By clicking "Reload List," documents that have been modified are reloaded into the vector database.
• There is an option to perform an automatic reload every 10 minutes to keep the database updated.
4. Duplicate Documentation Detection
• An LLM model identifies duplicate documents within the workspace, improving organization and avoiding redundancies.
5. Information Queries
• The RAG (Retrieval-Augmented Generation) technique is used to query the loaded documentation.
• The system generates a query vector and searches for matches in the vector database.
• Once the relevant document or section is identified, a local LLM model processes and displays the response in the interface.

Conclusions

The implementation of the automated information search system in documents has proven to be a powerful tool for document management. The ability to convert documents to vectors and store them in a vector database allows for quick and accurate searches. Additionally, the integration of LLM models significantly enhances the quality and relevance of the responses obtained. The system not only facilitates the management of large volumes of documents but also ensures continuous updating and duplicate elimination, optimizing workflow and reducing the time spent searching for information. When dealing with information that is often confidential, an open-source local model called Llama3 is used.

Future Development

For future improvements, the following areas can be considered:

1. Optimization of Prompts
• Improve the prompts used by LLM models to generate more precise queries and obtain more relevant answers.
2. Expansion of Document Formats
• Expand the system's compatibility to other document formats such as HTML, TXT, etc.
3. Integration with Other Systems
• Integrate the system with other document management platforms and collaboration tools such as SharePoint, Google Drive, and Slack.
4. Improvements in the Vector Database
• Implement advanced vectorization and search techniques to enhance the speed and accuracy of queries.
5. User Interface
• Develop a more intuitive and feature-rich user interface to improve the end-user experience.

In summary, the implementation of an automated information search system in documents with artificial intelligence not only improves efficiency and accuracy in document management but also opens the door to future innovations and continuous improvements.