Charles Englebert
3ec95c9edf
* ✨(backend) add trivial vector embedding add a trivial vector embedding with constant [0.0, 0.0] * 🙈(core) gitignore ignore files related to sqlite and pdb * ✨(backend) introduce hybrid search handle full-text along with seamntic search * ✨(backend) install basic embedding model I need an embedding to performe semantic search. I need a simple model from hugging face. * ✨(backend) embbed the text embed the text of the query and the document. * 🐛(backend) fix filters and refactor view filter were broken by previous commits. This fixes them. * ♻️(backend) refactor pipeline creation pipeline had to be refactored. This refactors it. * ✨(backend) improve filtering filtering were done once after hybrid computation. For efficiency it should be done of each subquery. * 🔧(setting) move variables to settings NLP_SEARCH_PIPELINE_ID and HYBRID_SEARCH_WEIGHTS is moved to setting file so user can param Find. * ✨(backend) use albert api we choose to rely on Albert API instead of installing a model in local. It is less effort to maintain. * ♻️(backend) hide EMBEDDING_API_KEY EMBEDDING_API_KEY should not be visible. * ✨(backend) move opensearch functions to a service user of find should be able to disable seamntic search. If it is not properly setted it is also turned off without impatcing full-text search. * 🧪(backend) test add test so the app is tested. * 📝(backend) add documentation add documentation so the app is documented. * ♻️(backend) remove local model the local model is no longer useful. Its file must be removed. * 🧪(backend) one more test The app must be tested more. This tests the app more. * ✨(bakckend) handle k value k value must be handled so the user can have a control over the number of results. * ♻️(backend) clean branch beanch had to be cleaned bery very much * 🔧(infra) define EMBEDDING_API_KEY EMBEDDING_API_KEY must be hiden. This hide EMBEDDING_API_KEY * 🚨(backend) fix linters linters must be fixed. This commit fixes them. * 📝(core) add changelog changelog must be updated. This updates the changelog. * 🐛(backend) fix linters more linters had to be fixed more. This fixes linters more. * 🐛(backend) fix tests test must be fixed. This fixes the tests. * ♻️(backend) improve variable managment variable managment must be improve. This improve variable managment. * ♻️(backend) remove embedding from schemas embedding must be removed from schemas. This removes embedding from shemas. * ✨(backend) add reindex_with_embedding command We must be able to enable hybrid search if it was disabled or chnage the embedding model. To do so we must reindex all documents with a new embedding. reindex_with_embedding does that. * ✨(backend) add create_pipeline command We must be able to create the command pipeline once and not check at all request. * 🧪(backend) tests I add a test and fix other tests * 🚨(backend) linters linters must be fixed. This fixes linters. * ✨(backend) remove pagination Semantic search has an impact of pagination. Pagination will be perfomed in services consuming Find API (Doc, Drive etc...) * ✨(backend) improve reindexing we want to handle error case and model change. I introduce a embedding_model field to keep track of the embedding state. * 🧪(backend) test more the command must be tested more. This tests the command more. * 🧪(backend) test concurent update do not lead do data loss updates on a document mught be done by a user while reindexing. I check the latest data is not lost. using if_seq_no and if_primary_term is not only not useful but whould require reruning the command. * ✨(backend) improve reindexing again reindexing must preserve the latest updates. I reintroduce the no_seq update field. * ♻️(backend) various small improvments I make various small improvments.
Find
Find can index documents from several applications sharing a common OIDC federation and allows users to search documents with their access rights accross all applications in the federation.
Find is built on top of Django Rest Framework.
Getting started
Prerequisite
Make sure you have a recent version of Docker and Docker Compose installed on your laptop:
$ docker -v
Docker version 27.4.1, build b9d17ea
$ docker compose version
Docker Compose version v2.32.1
⚠️ You may need to run the following commands with
sudobut this can be avoided by assigning your user to thedockergroup. See docker Documentation
Project bootstrap
The easiest way to start working on the project is to use GNU Make:
$ make bootstrap
This command builds the app container, installs dependencies, performs
database migrations and compile translations. It's a good idea to use this
command each time you are pulling code from the project repository to avoid
dependency-releated or migration-releated issues.
Your Docker services should now be up and running 🎉
Adding content
You can create a basic demo site by running:
$ make demo
Finally, you can check all available Make rules using:
$ make help
Django admin
You can access the Django admin site at http://localhost:8071/admin.
You first need to create a superuser account:
$ make superuser
Contributing
This project is intended to be community-driven, so please, do not hesitate to get in touch if you have any question related to our implementation or design decisions.
License
This work is released under the MIT License (see LICENSE).