Charles Englebert
3ec95c9edf
* ✨(backend) add trivial vector embedding add a trivial vector embedding with constant [0.0, 0.0] * 🙈(core) gitignore ignore files related to sqlite and pdb * ✨(backend) introduce hybrid search handle full-text along with seamntic search * ✨(backend) install basic embedding model I need an embedding to performe semantic search. I need a simple model from hugging face. * ✨(backend) embbed the text embed the text of the query and the document. * 🐛(backend) fix filters and refactor view filter were broken by previous commits. This fixes them. * ♻️(backend) refactor pipeline creation pipeline had to be refactored. This refactors it. * ✨(backend) improve filtering filtering were done once after hybrid computation. For efficiency it should be done of each subquery. * 🔧(setting) move variables to settings NLP_SEARCH_PIPELINE_ID and HYBRID_SEARCH_WEIGHTS is moved to setting file so user can param Find. * ✨(backend) use albert api we choose to rely on Albert API instead of installing a model in local. It is less effort to maintain. * ♻️(backend) hide EMBEDDING_API_KEY EMBEDDING_API_KEY should not be visible. * ✨(backend) move opensearch functions to a service user of find should be able to disable seamntic search. If it is not properly setted it is also turned off without impatcing full-text search. * 🧪(backend) test add test so the app is tested. * 📝(backend) add documentation add documentation so the app is documented. * ♻️(backend) remove local model the local model is no longer useful. Its file must be removed. * 🧪(backend) one more test The app must be tested more. This tests the app more. * ✨(bakckend) handle k value k value must be handled so the user can have a control over the number of results. * ♻️(backend) clean branch beanch had to be cleaned bery very much * 🔧(infra) define EMBEDDING_API_KEY EMBEDDING_API_KEY must be hiden. This hide EMBEDDING_API_KEY * 🚨(backend) fix linters linters must be fixed. This commit fixes them. * 📝(core) add changelog changelog must be updated. This updates the changelog. * 🐛(backend) fix linters more linters had to be fixed more. This fixes linters more. * 🐛(backend) fix tests test must be fixed. This fixes the tests. * ♻️(backend) improve variable managment variable managment must be improve. This improve variable managment. * ♻️(backend) remove embedding from schemas embedding must be removed from schemas. This removes embedding from shemas. * ✨(backend) add reindex_with_embedding command We must be able to enable hybrid search if it was disabled or chnage the embedding model. To do so we must reindex all documents with a new embedding. reindex_with_embedding does that. * ✨(backend) add create_pipeline command We must be able to create the command pipeline once and not check at all request. * 🧪(backend) tests I add a test and fix other tests * 🚨(backend) linters linters must be fixed. This fixes linters. * ✨(backend) remove pagination Semantic search has an impact of pagination. Pagination will be perfomed in services consuming Find API (Doc, Drive etc...) * ✨(backend) improve reindexing we want to handle error case and model change. I introduce a embedding_model field to keep track of the embedding state. * 🧪(backend) test more the command must be tested more. This tests the command more. * 🧪(backend) test concurent update do not lead do data loss updates on a document mught be done by a user while reindexing. I check the latest data is not lost. using if_seq_no and if_primary_term is not only not useful but whould require reruning the command. * ✨(backend) improve reindexing again reindexing must preserve the latest updates. I reintroduce the no_seq update field. * ♻️(backend) various small improvments I make various small improvments.
87 lines
870 B
Plaintext
87 lines
870 B
Plaintext
# Byte-compiled / optimized / DLL files
|
|
__pycache__/
|
|
*.py[cod]
|
|
*$py.class
|
|
|
|
# C extensions
|
|
*.so
|
|
|
|
# Distribution / packaging
|
|
.Python
|
|
build/
|
|
develop-eggs/
|
|
dist/
|
|
downloads/
|
|
eggs/
|
|
.eggs/
|
|
lib/
|
|
lib64/
|
|
parts/
|
|
sdist/
|
|
var/
|
|
wheels/
|
|
pip-wheel-metadata/
|
|
share/python-wheels/
|
|
*.egg-info/
|
|
.installed.cfg
|
|
*.egg
|
|
MANIFEST
|
|
.DS_Store
|
|
.next/
|
|
|
|
# Translations # Translations
|
|
*.pot
|
|
|
|
# Environments
|
|
.env
|
|
.venv
|
|
env/
|
|
venv/
|
|
ENV/
|
|
env.bak/
|
|
venv.bak/
|
|
env.d/development/*
|
|
!env.d/development/*.dist
|
|
env.d/terraform
|
|
|
|
# npm
|
|
node_modules
|
|
|
|
# Mails
|
|
src/backend/core/templates/mail/
|
|
|
|
# Typescript client
|
|
src/frontend/tsclient
|
|
|
|
# Swagger
|
|
**/swagger.json
|
|
|
|
# Logs
|
|
*.log
|
|
|
|
# Terraform
|
|
.terraform
|
|
*.tfstate
|
|
*.tfstate.backup
|
|
|
|
# Test & lint
|
|
.coverage
|
|
.pylint.d
|
|
.pytest_cache
|
|
db.sqlite3
|
|
*history.sqlite
|
|
.mypy_cache
|
|
|
|
# Site media
|
|
/data/
|
|
|
|
# IDEs
|
|
.idea/
|
|
.vscode/
|
|
*.iml
|
|
.devcontainer
|
|
.vscode-server
|
|
|
|
#pdb
|
|
*.pdbhistory
|