Update all GitHub Actions to their latest major versions for improved
performance, security patches, and Node.js runtime compatibility.
Signed-off-by: Stephan Meijer <me@stephanmeijer.com>
* ✨(backend) add path filter
allow search filters on path for Docs compatibility
* ✅(backend) test
I improve existing and add new tests
* ✨(backend) add search_type params
We want to control the type of search to use at run time
to feature flag Docs users with full-test or with hybrid
search.
I am adding a search_type params on the search route.
* 🚨(backend) linters
fiiiiiiiiiix liiiiiinters
* ✨(backend) allow delete documents by tags
the assistant does not know the ids. It needs to be able to detele
documents by tags.
* ♻️(backend) refactor
I refactor to have a better code.
* 🚨(backend) various fixes
I am fixing things for review.
Migreate usage of pip to uv in github actions. How python is setup is
also changed. Doing like this, we will just have to upgrade the python
version requirement in the pyproject file
We want to migrate our projects from pip to uv to take the benefits of
the lock file and have reproducible installations.
A first uv.lock file is comitted and the Dockerfile and compose are
modified to work with uv
* ✨(backend) add tags field for result filtering (#29)
* 🚨(backend) remove dead files
I left files I should have removed
* ✨(backend) add tags field
I add a tag field in the index and a filtering.
A tag is keyword that can be applied to a document.
A document can have several tags.
Tags allow filtering related documents.
The Conversations app needs tags for its rag tool.
* ✨(backend) enhance document indexing with error handling and changelog
I enhance document indexing with error handling and changelog
* 📝(backend) improve docstrings and Changelog
Docstrings and Changelog had to be improved.
This commit improves them.
* 🐛(backend) prevent information lick
return user message with less info and log error
* 🚨(backend) return undeleted document IDs
We need better responses in case of unexpected behaviour.
I enhance document deletion API by returning undeleted
document IDs
* ✨(backend) enhance document indexing with error handling and changelog
I enhance document indexing with error handling and changelog
* ✨(backend) adapt to conversation RAG
the document rag tool of conversation expect a content
and a dedicated service with token access is needed.
I am also adding loggers.
* 🚨(backend) fix tests
demo tests are broken. Here is the fix.
Allow devops team to have a quick insight if DB connections
are properly set.
This is a temporary commit and will be rewritten in a proper way
later (work was already started for "people" project).
* 🚨(backend) remove dead files
I left files I should have removed
* ✨(backend) add tags field
I add a tag field in the index and a filtering.
A tag is keyword that can be applied to a document.
A document can have several tags.
Tags allow filtering related documents.
The Conversations app needs tags for its rag tool.
* ✨(backend) enhance document indexing with error handling and changelog
I enhance document indexing with error handling and changelog
* 📝(backend) improve docstrings and Changelog
Docstrings and Changelog had to be improved.
This commit improves them.
* ✨(backend) add evaluate-search-engine command
I want to automize the search evaluation. This new command
computes performance metrics.
* ✨(backend) improve evaluation
I add more data to my evaluations.
* 📝(backend) add changelog
add changelog and various fixes.
* ✨(backend) add evaluation data from service-public.fr
I need better data with longer content to work on chuncking
* ✨(backend) handle multi-embedding
I breack document content into peaces and embed each peace separatly.
Search is them based on the mest match.
* 📝(docs) add documentation
I add documentation about chunking
* 🚨(backend) fix things
thigs were broken. I fixed this.
* 📝(backend) documentation
I document the documentation of it
* 🚨(backend) fix rebase
the rebase has messed things up. I fixed those things.
* ♻️(backend) refactor language code handling and improve test cases
I fix things to fix things
* ♻️(backend) refactor
I am doing refactoooooooooooor
* ✨(backend) add evaluate-search-engine command
I want to automize the search evaluation. This new command
computes performance metrics.
* ✨(backend) improve evaluation
I add more data to my evaluations.
* 📝(backend) add changelog
add changelog and various fixes.
* ✨(backend) improve full text
I introduce two analyszers to improve the full text search.
* 📝(backend) add changelog
I update the changelog
* 🧪(backend) fix tests and linters
I fix tests and linters
* ♻️(backend) various fixes
I fix a buch of small things
* 🔧(backend) define settings
I define settings to remove magic numbers
* ✨(backend) copy evaluation
I copy the evaluation command
* ✨(backend) index multi-language
I index in multi-language
* ✨(backend) flatten the data structure
I changed my mind. I want a flat structure.
* ♻️(backend) handle search
the search must be updated so everything works
* 🧪(backend) more tests
I add more tests so the feature is tested more
* 📝(backend) docuemntation
I docuemnt so the feature is documented
* ♻️(backend) various fixes
I did many mistakes. There are now fixed.
* 🚨(backend) fix things
things were a bit broken but I ixed them
* ✨(backend) detect language
I change the logic.
I detect the language instead of receiving it as queryparams
* 🚨(backend) fix things
things are broken and I fixe them here
* 📝(backend) better documentation
I improve the documentation a little bit
* 🧪(backend) test
more test is better. I add tests.
* 🚨(backend) fix things
fiiiiiiiiiiiiiiiiiiiiiiiix things
* ♻️(backend) simplify language_code
we do not need language variations
* ♻️(backend) fix things
things are broken. now they are fixed.
* ✨(backend) add evaluate-search-engine command
I want to automize the search evaluation. This new command
computes performance metrics.
* ✨(backend) improve evaluation
I add more data to my evaluations.
* 📝(backend) add changelog
add changelog and various fixes.
* ✨(backend) add more data
I add more data to evaluation
* ✨(backend) add index management flags
I add --keep-index and --force-reindex flags
* ♻️(backend) remove dependences from test/utils
I remove dependences from test/utils
* 📝(backend) documenting
I add documentation of the command
* ♻️(backend) break unique documents file into text files
I change the data structure of the documents
* 🚨(backend) fix things
things were broken but here I fix them
* ♻️(backend) evaluation app
I move the command to an evaluation app
* 🧪(backend) add tests
I add test on the command
* 🚨(backend) fix thing
thinghs must be fixed.
Service opensearch index is now defined by the property 'index_name' and
prefixed by the new setting OPENSEARCH_INDEX_PREFIX (default: 'find').
Fix parallel execution of tests.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
The create_demo command will now create a 'drive' service for development
purpose.
Add setup documentation.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
* ✨(backend) add trivial vector embedding
add a trivial vector embedding with constant [0.0, 0.0]
* 🙈(core) gitignore
ignore files related to sqlite and pdb
* ✨(backend) introduce hybrid search
handle full-text along with seamntic search
* ✨(backend) install basic embedding model
I need an embedding to performe semantic search.
I need a simple model from hugging face.
* ✨(backend) embbed the text
embed the text of the query and the document.
* 🐛(backend) fix filters and refactor view
filter were broken by previous commits.
This fixes them.
* ♻️(backend) refactor pipeline creation
pipeline had to be refactored.
This refactors it.
* ✨(backend) improve filtering
filtering were done once after hybrid computation.
For efficiency it should be done of each
subquery.
* 🔧(setting) move variables to settings
NLP_SEARCH_PIPELINE_ID and HYBRID_SEARCH_WEIGHTS is moved
to setting file so user can param Find.
* ✨(backend) use albert api
we choose to rely on Albert API instead of installing
a model in local. It is less effort to maintain.
* ♻️(backend) hide EMBEDDING_API_KEY
EMBEDDING_API_KEY should not be visible.
* ✨(backend) move opensearch functions to a service
user of find should be able to disable seamntic search.
If it is not properly setted it is also turned off
without impatcing full-text search.
* 🧪(backend) test
add test so the app is tested.
* 📝(backend) add documentation
add documentation so the app is documented.
* ♻️(backend) remove local model
the local model is no longer useful.
Its file must be removed.
* 🧪(backend) one more test
The app must be tested more. This tests the app more.
* ✨(bakckend) handle k value
k value must be handled so the user can have a control
over the number of results.
* ♻️(backend) clean branch
beanch had to be cleaned bery very much
* 🔧(infra) define EMBEDDING_API_KEY
EMBEDDING_API_KEY must be hiden. This hide EMBEDDING_API_KEY
* 🚨(backend) fix linters
linters must be fixed. This commit fixes them.
* 📝(core) add changelog
changelog must be updated. This updates the changelog.
* 🐛(backend) fix linters more
linters had to be fixed more. This fixes linters more.
* 🐛(backend) fix tests
test must be fixed. This fixes the tests.
* ♻️(backend) improve variable managment
variable managment must be improve. This improve variable managment.
* ♻️(backend) remove embedding from schemas
embedding must be removed from schemas. This removes embedding from shemas.
* ✨(backend) add reindex_with_embedding command
We must be able to enable hybrid search if it was disabled
or chnage the embedding model. To do so we must reindex all
documents with a new embedding. reindex_with_embedding does that.
* ✨(backend) add create_pipeline command
We must be able to create the command pipeline once and
not check at all request.
* 🧪(backend) tests
I add a test and fix other tests
* 🚨(backend) linters
linters must be fixed. This fixes linters.
* ✨(backend) remove pagination
Semantic search has an impact of pagination.
Pagination will be perfomed in services consuming Find API (Doc, Drive etc...)
* ✨(backend) improve reindexing
we want to handle error case and model change. I introduce a
embedding_model field to keep track of the embedding state.
* 🧪(backend) test more
the command must be tested more. This tests the command more.
* 🧪(backend) test concurent update do not lead do data loss
updates on a document mught be done by a user while reindexing.
I check the latest data is not lost. using if_seq_no and if_primary_term
is not only not useful but whould require reruning the command.
* ✨(backend) improve reindexing again
reindexing must preserve the latest updates.
I reintroduce the no_seq update field.
* ♻️(backend) various small improvments
I make various small improvments.
The previous commit has allowed to run Docs and Find projects
stacks together with a tweak : Find must use the keycloak OIDC
endpoints of Docs on the 'nginx' domain for ResourceServer authentication
and introspection.
This configuration broke the OIDC token claims validation because the iss
from the token of the 'impress' user is 'localhost' and not 'nginx'.
We fix it by using the same endpoint : http://localhost:8083/realms/impress
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Populate a service configuration "docs" for development.
Move OIDC endpoints to the "impress" realm to allow introspection
of Docs tokens.
Upgrade version of dependencies (fix some security issues).
Signed-off-by: Fabre Florian <ffabre@hybird.org>
New fulltext search view for indexed documents with OIDC authentication
Extract token information through introspection to get the audience & user info
Limit access to documents :
- public & authenticated with linkreach to the user
- owned by the user
Check intersection between the allowed services linked to the
audience/client_id and the requested ones.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
In indexation view, raise a validation error only when both the
title & content of a document are empty.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
We need to make a POST to search documents so that we can post a
list of documents the current user has already "visited" This is
necessary to limited the number of documents we return among the
ones available to any authenticated or anonymous user.
We need to index the tree structure information as well as an
active field that can be set to False when the item is deleted on
the remote service. We could delete the item from our search index
but it is safer to keep all documents synchronized and not only
those which are not deleted.
renovate[bot]