* ✨(backend) add evaluate-search-engine command
I want to automize the search evaluation. This new command
computes performance metrics.
* ✨(backend) improve evaluation
I add more data to my evaluations.
* 📝(backend) add changelog
add changelog and various fixes.
* ✨(backend) add evaluation data from service-public.fr
I need better data with longer content to work on chuncking
* ✨(backend) handle multi-embedding
I breack document content into peaces and embed each peace separatly.
Search is them based on the mest match.
* 📝(docs) add documentation
I add documentation about chunking
* 🚨(backend) fix things
thigs were broken. I fixed this.
* 📝(backend) documentation
I document the documentation of it
* 🚨(backend) fix rebase
the rebase has messed things up. I fixed those things.
* ♻️(backend) refactor language code handling and improve test cases
I fix things to fix things
* ♻️(backend) refactor
I am doing refactoooooooooooor
* ✨(backend) add evaluate-search-engine command
I want to automize the search evaluation. This new command
computes performance metrics.
* ✨(backend) improve evaluation
I add more data to my evaluations.
* 📝(backend) add changelog
add changelog and various fixes.
* ✨(backend) improve full text
I introduce two analyszers to improve the full text search.
* 📝(backend) add changelog
I update the changelog
* 🧪(backend) fix tests and linters
I fix tests and linters
* ♻️(backend) various fixes
I fix a buch of small things
* 🔧(backend) define settings
I define settings to remove magic numbers
* ✨(backend) copy evaluation
I copy the evaluation command
* ✨(backend) index multi-language
I index in multi-language
* ✨(backend) flatten the data structure
I changed my mind. I want a flat structure.
* ♻️(backend) handle search
the search must be updated so everything works
* 🧪(backend) more tests
I add more tests so the feature is tested more
* 📝(backend) docuemntation
I docuemnt so the feature is documented
* ♻️(backend) various fixes
I did many mistakes. There are now fixed.
* 🚨(backend) fix things
things were a bit broken but I ixed them
* ✨(backend) detect language
I change the logic.
I detect the language instead of receiving it as queryparams
* 🚨(backend) fix things
things are broken and I fixe them here
* 📝(backend) better documentation
I improve the documentation a little bit
* 🧪(backend) test
more test is better. I add tests.
* 🚨(backend) fix things
fiiiiiiiiiiiiiiiiiiiiiiiix things
* ♻️(backend) simplify language_code
we do not need language variations
* ♻️(backend) fix things
things are broken. now they are fixed.
* ✨(backend) add evaluate-search-engine command
I want to automize the search evaluation. This new command
computes performance metrics.
* ✨(backend) improve evaluation
I add more data to my evaluations.
* 📝(backend) add changelog
add changelog and various fixes.
* ✨(backend) add more data
I add more data to evaluation
* ✨(backend) add index management flags
I add --keep-index and --force-reindex flags
* ♻️(backend) remove dependences from test/utils
I remove dependences from test/utils
* 📝(backend) documenting
I add documentation of the command
* ♻️(backend) break unique documents file into text files
I change the data structure of the documents
* 🚨(backend) fix things
things were broken but here I fix them
* ♻️(backend) evaluation app
I move the command to an evaluation app
* 🧪(backend) add tests
I add test on the command
* 🚨(backend) fix thing
thinghs must be fixed.
The create_demo command will now create a 'drive' service for development
purpose.
Add setup documentation.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
* ✨(backend) add trivial vector embedding
add a trivial vector embedding with constant [0.0, 0.0]
* 🙈(core) gitignore
ignore files related to sqlite and pdb
* ✨(backend) introduce hybrid search
handle full-text along with seamntic search
* ✨(backend) install basic embedding model
I need an embedding to performe semantic search.
I need a simple model from hugging face.
* ✨(backend) embbed the text
embed the text of the query and the document.
* 🐛(backend) fix filters and refactor view
filter were broken by previous commits.
This fixes them.
* ♻️(backend) refactor pipeline creation
pipeline had to be refactored.
This refactors it.
* ✨(backend) improve filtering
filtering were done once after hybrid computation.
For efficiency it should be done of each
subquery.
* 🔧(setting) move variables to settings
NLP_SEARCH_PIPELINE_ID and HYBRID_SEARCH_WEIGHTS is moved
to setting file so user can param Find.
* ✨(backend) use albert api
we choose to rely on Albert API instead of installing
a model in local. It is less effort to maintain.
* ♻️(backend) hide EMBEDDING_API_KEY
EMBEDDING_API_KEY should not be visible.
* ✨(backend) move opensearch functions to a service
user of find should be able to disable seamntic search.
If it is not properly setted it is also turned off
without impatcing full-text search.
* 🧪(backend) test
add test so the app is tested.
* 📝(backend) add documentation
add documentation so the app is documented.
* ♻️(backend) remove local model
the local model is no longer useful.
Its file must be removed.
* 🧪(backend) one more test
The app must be tested more. This tests the app more.
* ✨(bakckend) handle k value
k value must be handled so the user can have a control
over the number of results.
* ♻️(backend) clean branch
beanch had to be cleaned bery very much
* 🔧(infra) define EMBEDDING_API_KEY
EMBEDDING_API_KEY must be hiden. This hide EMBEDDING_API_KEY
* 🚨(backend) fix linters
linters must be fixed. This commit fixes them.
* 📝(core) add changelog
changelog must be updated. This updates the changelog.
* 🐛(backend) fix linters more
linters had to be fixed more. This fixes linters more.
* 🐛(backend) fix tests
test must be fixed. This fixes the tests.
* ♻️(backend) improve variable managment
variable managment must be improve. This improve variable managment.
* ♻️(backend) remove embedding from schemas
embedding must be removed from schemas. This removes embedding from shemas.
* ✨(backend) add reindex_with_embedding command
We must be able to enable hybrid search if it was disabled
or chnage the embedding model. To do so we must reindex all
documents with a new embedding. reindex_with_embedding does that.
* ✨(backend) add create_pipeline command
We must be able to create the command pipeline once and
not check at all request.
* 🧪(backend) tests
I add a test and fix other tests
* 🚨(backend) linters
linters must be fixed. This fixes linters.
* ✨(backend) remove pagination
Semantic search has an impact of pagination.
Pagination will be perfomed in services consuming Find API (Doc, Drive etc...)
* ✨(backend) improve reindexing
we want to handle error case and model change. I introduce a
embedding_model field to keep track of the embedding state.
* 🧪(backend) test more
the command must be tested more. This tests the command more.
* 🧪(backend) test concurent update do not lead do data loss
updates on a document mught be done by a user while reindexing.
I check the latest data is not lost. using if_seq_no and if_primary_term
is not only not useful but whould require reruning the command.
* ✨(backend) improve reindexing again
reindexing must preserve the latest updates.
I reintroduce the no_seq update field.
* ♻️(backend) various small improvments
I make various small improvments.
Charles Englebert