Charles Englebert
3ec95c9edf
* ✨(backend) add trivial vector embedding add a trivial vector embedding with constant [0.0, 0.0] * 🙈(core) gitignore ignore files related to sqlite and pdb * ✨(backend) introduce hybrid search handle full-text along with seamntic search * ✨(backend) install basic embedding model I need an embedding to performe semantic search. I need a simple model from hugging face. * ✨(backend) embbed the text embed the text of the query and the document. * 🐛(backend) fix filters and refactor view filter were broken by previous commits. This fixes them. * ♻️(backend) refactor pipeline creation pipeline had to be refactored. This refactors it. * ✨(backend) improve filtering filtering were done once after hybrid computation. For efficiency it should be done of each subquery. * 🔧(setting) move variables to settings NLP_SEARCH_PIPELINE_ID and HYBRID_SEARCH_WEIGHTS is moved to setting file so user can param Find. * ✨(backend) use albert api we choose to rely on Albert API instead of installing a model in local. It is less effort to maintain. * ♻️(backend) hide EMBEDDING_API_KEY EMBEDDING_API_KEY should not be visible. * ✨(backend) move opensearch functions to a service user of find should be able to disable seamntic search. If it is not properly setted it is also turned off without impatcing full-text search. * 🧪(backend) test add test so the app is tested. * 📝(backend) add documentation add documentation so the app is documented. * ♻️(backend) remove local model the local model is no longer useful. Its file must be removed. * 🧪(backend) one more test The app must be tested more. This tests the app more. * ✨(bakckend) handle k value k value must be handled so the user can have a control over the number of results. * ♻️(backend) clean branch beanch had to be cleaned bery very much * 🔧(infra) define EMBEDDING_API_KEY EMBEDDING_API_KEY must be hiden. This hide EMBEDDING_API_KEY * 🚨(backend) fix linters linters must be fixed. This commit fixes them. * 📝(core) add changelog changelog must be updated. This updates the changelog. * 🐛(backend) fix linters more linters had to be fixed more. This fixes linters more. * 🐛(backend) fix tests test must be fixed. This fixes the tests. * ♻️(backend) improve variable managment variable managment must be improve. This improve variable managment. * ♻️(backend) remove embedding from schemas embedding must be removed from schemas. This removes embedding from shemas. * ✨(backend) add reindex_with_embedding command We must be able to enable hybrid search if it was disabled or chnage the embedding model. To do so we must reindex all documents with a new embedding. reindex_with_embedding does that. * ✨(backend) add create_pipeline command We must be able to create the command pipeline once and not check at all request. * 🧪(backend) tests I add a test and fix other tests * 🚨(backend) linters linters must be fixed. This fixes linters. * ✨(backend) remove pagination Semantic search has an impact of pagination. Pagination will be perfomed in services consuming Find API (Doc, Drive etc...) * ✨(backend) improve reindexing we want to handle error case and model change. I introduce a embedding_model field to keep track of the embedding state. * 🧪(backend) test more the command must be tested more. This tests the command more. * 🧪(backend) test concurent update do not lead do data loss updates on a document mught be done by a user while reindexing. I check the latest data is not lost. using if_seq_no and if_primary_term is not only not useful but whould require reruning the command. * ✨(backend) improve reindexing again reindexing must preserve the latest updates. I reintroduce the no_seq update field. * ♻️(backend) various small improvments I make various small improvments.
322 B
322 B
Architecture
Global system architecture
flowchart TD
Docs -- REST API --> Back("Backend (Django)")
Back --> DB("Database (PostgreSQL)")
Back -- REST API --> Opensearch
Back <--> Celery --> DB
User -- HTTP --> Dashboard --> Opensearch
Back -- REST API --> Embedding Endpoint