* ✨(backend) add evaluate-search-engine command
I want to automize the search evaluation. This new command
computes performance metrics.
* ✨(backend) improve evaluation
I add more data to my evaluations.
* 📝(backend) add changelog
add changelog and various fixes.
* ✨(backend) add evaluation data from service-public.fr
I need better data with longer content to work on chuncking
* ✨(backend) handle multi-embedding
I breack document content into peaces and embed each peace separatly.
Search is them based on the mest match.
* 📝(docs) add documentation
I add documentation about chunking
* 🚨(backend) fix things
thigs were broken. I fixed this.
* 📝(backend) documentation
I document the documentation of it
* 🚨(backend) fix rebase
the rebase has messed things up. I fixed those things.
* ♻️(backend) refactor language code handling and improve test cases
I fix things to fix things
* ♻️(backend) refactor
I am doing refactoooooooooooor
* ✨(backend) add trivial vector embedding
add a trivial vector embedding with constant [0.0, 0.0]
* 🙈(core) gitignore
ignore files related to sqlite and pdb
* ✨(backend) introduce hybrid search
handle full-text along with seamntic search
* ✨(backend) install basic embedding model
I need an embedding to performe semantic search.
I need a simple model from hugging face.
* ✨(backend) embbed the text
embed the text of the query and the document.
* 🐛(backend) fix filters and refactor view
filter were broken by previous commits.
This fixes them.
* ♻️(backend) refactor pipeline creation
pipeline had to be refactored.
This refactors it.
* ✨(backend) improve filtering
filtering were done once after hybrid computation.
For efficiency it should be done of each
subquery.
* 🔧(setting) move variables to settings
NLP_SEARCH_PIPELINE_ID and HYBRID_SEARCH_WEIGHTS is moved
to setting file so user can param Find.
* ✨(backend) use albert api
we choose to rely on Albert API instead of installing
a model in local. It is less effort to maintain.
* ♻️(backend) hide EMBEDDING_API_KEY
EMBEDDING_API_KEY should not be visible.
* ✨(backend) move opensearch functions to a service
user of find should be able to disable seamntic search.
If it is not properly setted it is also turned off
without impatcing full-text search.
* 🧪(backend) test
add test so the app is tested.
* 📝(backend) add documentation
add documentation so the app is documented.
* ♻️(backend) remove local model
the local model is no longer useful.
Its file must be removed.
* 🧪(backend) one more test
The app must be tested more. This tests the app more.
* ✨(bakckend) handle k value
k value must be handled so the user can have a control
over the number of results.
* ♻️(backend) clean branch
beanch had to be cleaned bery very much
* 🔧(infra) define EMBEDDING_API_KEY
EMBEDDING_API_KEY must be hiden. This hide EMBEDDING_API_KEY
* 🚨(backend) fix linters
linters must be fixed. This commit fixes them.
* 📝(core) add changelog
changelog must be updated. This updates the changelog.
* 🐛(backend) fix linters more
linters had to be fixed more. This fixes linters more.
* 🐛(backend) fix tests
test must be fixed. This fixes the tests.
* ♻️(backend) improve variable managment
variable managment must be improve. This improve variable managment.
* ♻️(backend) remove embedding from schemas
embedding must be removed from schemas. This removes embedding from shemas.
* ✨(backend) add reindex_with_embedding command
We must be able to enable hybrid search if it was disabled
or chnage the embedding model. To do so we must reindex all
documents with a new embedding. reindex_with_embedding does that.
* ✨(backend) add create_pipeline command
We must be able to create the command pipeline once and
not check at all request.
* 🧪(backend) tests
I add a test and fix other tests
* 🚨(backend) linters
linters must be fixed. This fixes linters.
* ✨(backend) remove pagination
Semantic search has an impact of pagination.
Pagination will be perfomed in services consuming Find API (Doc, Drive etc...)
* ✨(backend) improve reindexing
we want to handle error case and model change. I introduce a
embedding_model field to keep track of the embedding state.
* 🧪(backend) test more
the command must be tested more. This tests the command more.
* 🧪(backend) test concurent update do not lead do data loss
updates on a document mught be done by a user while reindexing.
I check the latest data is not lost. using if_seq_no and if_primary_term
is not only not useful but whould require reruning the command.
* ✨(backend) improve reindexing again
reindexing must preserve the latest updates.
I reintroduce the no_seq update field.
* ♻️(backend) various small improvments
I make various small improvments.
The previous commit has allowed to run Docs and Find projects
stacks together with a tweak : Find must use the keycloak OIDC
endpoints of Docs on the 'nginx' domain for ResourceServer authentication
and introspection.
This configuration broke the OIDC token claims validation because the iss
from the token of the 'impress' user is 'localhost' and not 'nginx'.
We fix it by using the same endpoint : http://localhost:8083/realms/impress
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Populate a service configuration "docs" for development.
Move OIDC endpoints to the "impress" realm to allow introspection
of Docs tokens.
Upgrade version of dependencies (fix some security issues).
Signed-off-by: Fabre Florian <ffabre@hybird.org>
New fulltext search view for indexed documents with OIDC authentication
Extract token information through introspection to get the audience & user info
Limit access to documents :
- public & authenticated with linkreach to the user
- owned by the user
Check intersection between the allowed services linked to the
audience/client_id and the requested ones.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
This is built from our boilerplate and adds a first working
prototype of indexing (single document or bulk) and searching
(with basic filtering and sorting. No authentication yet).
Run `make demo` to generate an index for development.
Charles Englebert