* 🚨(backend) remove dead files
I left files I should have removed
* ✨(backend) add tags field
I add a tag field in the index and a filtering.
A tag is keyword that can be applied to a document.
A document can have several tags.
Tags allow filtering related documents.
The Conversations app needs tags for its rag tool.
* ✨(backend) enhance document indexing with error handling and changelog
I enhance document indexing with error handling and changelog
* 📝(backend) improve docstrings and Changelog
Docstrings and Changelog had to be improved.
This commit improves them.
* ✨(backend) add evaluate-search-engine command
I want to automize the search evaluation. This new command
computes performance metrics.
* ✨(backend) improve evaluation
I add more data to my evaluations.
* 📝(backend) add changelog
add changelog and various fixes.
* ✨(backend) add evaluation data from service-public.fr
I need better data with longer content to work on chuncking
* ✨(backend) handle multi-embedding
I breack document content into peaces and embed each peace separatly.
Search is them based on the mest match.
* 📝(docs) add documentation
I add documentation about chunking
* 🚨(backend) fix things
thigs were broken. I fixed this.
* 📝(backend) documentation
I document the documentation of it
* 🚨(backend) fix rebase
the rebase has messed things up. I fixed those things.
* ♻️(backend) refactor language code handling and improve test cases
I fix things to fix things
* ♻️(backend) refactor
I am doing refactoooooooooooor
* ✨(backend) add evaluate-search-engine command
I want to automize the search evaluation. This new command
computes performance metrics.
* ✨(backend) improve evaluation
I add more data to my evaluations.
* 📝(backend) add changelog
add changelog and various fixes.
* ✨(backend) improve full text
I introduce two analyszers to improve the full text search.
* 📝(backend) add changelog
I update the changelog
* 🧪(backend) fix tests and linters
I fix tests and linters
* ♻️(backend) various fixes
I fix a buch of small things
* 🔧(backend) define settings
I define settings to remove magic numbers
* ✨(backend) copy evaluation
I copy the evaluation command
* ✨(backend) index multi-language
I index in multi-language
* ✨(backend) flatten the data structure
I changed my mind. I want a flat structure.
* ♻️(backend) handle search
the search must be updated so everything works
* 🧪(backend) more tests
I add more tests so the feature is tested more
* 📝(backend) docuemntation
I docuemnt so the feature is documented
* ♻️(backend) various fixes
I did many mistakes. There are now fixed.
* 🚨(backend) fix things
things were a bit broken but I ixed them
* ✨(backend) detect language
I change the logic.
I detect the language instead of receiving it as queryparams
* 🚨(backend) fix things
things are broken and I fixe them here
* 📝(backend) better documentation
I improve the documentation a little bit
* 🧪(backend) test
more test is better. I add tests.
* 🚨(backend) fix things
fiiiiiiiiiiiiiiiiiiiiiiiix things
* ♻️(backend) simplify language_code
we do not need language variations
* ♻️(backend) fix things
things are broken. now they are fixed.
* ✨(backend) add evaluate-search-engine command
I want to automize the search evaluation. This new command
computes performance metrics.
* ✨(backend) improve evaluation
I add more data to my evaluations.
* 📝(backend) add changelog
add changelog and various fixes.
* ✨(backend) add more data
I add more data to evaluation
* ✨(backend) add index management flags
I add --keep-index and --force-reindex flags
* ♻️(backend) remove dependences from test/utils
I remove dependences from test/utils
* 📝(backend) documenting
I add documentation of the command
* ♻️(backend) break unique documents file into text files
I change the data structure of the documents
* 🚨(backend) fix things
things were broken but here I fix them
* ♻️(backend) evaluation app
I move the command to an evaluation app
* 🧪(backend) add tests
I add test on the command
* 🚨(backend) fix thing
thinghs must be fixed.
Service opensearch index is now defined by the property 'index_name' and
prefixed by the new setting OPENSEARCH_INDEX_PREFIX (default: 'find').
Fix parallel execution of tests.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
The create_demo command will now create a 'drive' service for development
purpose.
Add setup documentation.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
* ✨(backend) add trivial vector embedding
add a trivial vector embedding with constant [0.0, 0.0]
* 🙈(core) gitignore
ignore files related to sqlite and pdb
* ✨(backend) introduce hybrid search
handle full-text along with seamntic search
* ✨(backend) install basic embedding model
I need an embedding to performe semantic search.
I need a simple model from hugging face.
* ✨(backend) embbed the text
embed the text of the query and the document.
* 🐛(backend) fix filters and refactor view
filter were broken by previous commits.
This fixes them.
* ♻️(backend) refactor pipeline creation
pipeline had to be refactored.
This refactors it.
* ✨(backend) improve filtering
filtering were done once after hybrid computation.
For efficiency it should be done of each
subquery.
* 🔧(setting) move variables to settings
NLP_SEARCH_PIPELINE_ID and HYBRID_SEARCH_WEIGHTS is moved
to setting file so user can param Find.
* ✨(backend) use albert api
we choose to rely on Albert API instead of installing
a model in local. It is less effort to maintain.
* ♻️(backend) hide EMBEDDING_API_KEY
EMBEDDING_API_KEY should not be visible.
* ✨(backend) move opensearch functions to a service
user of find should be able to disable seamntic search.
If it is not properly setted it is also turned off
without impatcing full-text search.
* 🧪(backend) test
add test so the app is tested.
* 📝(backend) add documentation
add documentation so the app is documented.
* ♻️(backend) remove local model
the local model is no longer useful.
Its file must be removed.
* 🧪(backend) one more test
The app must be tested more. This tests the app more.
* ✨(bakckend) handle k value
k value must be handled so the user can have a control
over the number of results.
* ♻️(backend) clean branch
beanch had to be cleaned bery very much
* 🔧(infra) define EMBEDDING_API_KEY
EMBEDDING_API_KEY must be hiden. This hide EMBEDDING_API_KEY
* 🚨(backend) fix linters
linters must be fixed. This commit fixes them.
* 📝(core) add changelog
changelog must be updated. This updates the changelog.
* 🐛(backend) fix linters more
linters had to be fixed more. This fixes linters more.
* 🐛(backend) fix tests
test must be fixed. This fixes the tests.
* ♻️(backend) improve variable managment
variable managment must be improve. This improve variable managment.
* ♻️(backend) remove embedding from schemas
embedding must be removed from schemas. This removes embedding from shemas.
* ✨(backend) add reindex_with_embedding command
We must be able to enable hybrid search if it was disabled
or chnage the embedding model. To do so we must reindex all
documents with a new embedding. reindex_with_embedding does that.
* ✨(backend) add create_pipeline command
We must be able to create the command pipeline once and
not check at all request.
* 🧪(backend) tests
I add a test and fix other tests
* 🚨(backend) linters
linters must be fixed. This fixes linters.
* ✨(backend) remove pagination
Semantic search has an impact of pagination.
Pagination will be perfomed in services consuming Find API (Doc, Drive etc...)
* ✨(backend) improve reindexing
we want to handle error case and model change. I introduce a
embedding_model field to keep track of the embedding state.
* 🧪(backend) test more
the command must be tested more. This tests the command more.
* 🧪(backend) test concurent update do not lead do data loss
updates on a document mught be done by a user while reindexing.
I check the latest data is not lost. using if_seq_no and if_primary_term
is not only not useful but whould require reruning the command.
* ✨(backend) improve reindexing again
reindexing must preserve the latest updates.
I reintroduce the no_seq update field.
* ♻️(backend) various small improvments
I make various small improvments.
The previous commit has allowed to run Docs and Find projects
stacks together with a tweak : Find must use the keycloak OIDC
endpoints of Docs on the 'nginx' domain for ResourceServer authentication
and introspection.
This configuration broke the OIDC token claims validation because the iss
from the token of the 'impress' user is 'localhost' and not 'nginx'.
We fix it by using the same endpoint : http://localhost:8083/realms/impress
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Populate a service configuration "docs" for development.
Move OIDC endpoints to the "impress" realm to allow introspection
of Docs tokens.
Upgrade version of dependencies (fix some security issues).
Signed-off-by: Fabre Florian <ffabre@hybird.org>
New fulltext search view for indexed documents with OIDC authentication
Extract token information through introspection to get the audience & user info
Limit access to documents :
- public & authenticated with linkreach to the user
- owned by the user
Check intersection between the allowed services linked to the
audience/client_id and the requested ones.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
In indexation view, raise a validation error only when both the
title & content of a document are empty.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
We need to make a POST to search documents so that we can post a
list of documents the current user has already "visited" This is
necessary to limited the number of documents we return among the
ones available to any authenticated or anonymous user.
We need to index the tree structure information as well as an
active field that can be set to False when the item is deleted on
the remote service. We could delete the item from our search index
but it is safer to keep all documents synchronized and not only
those which are not deleted.
We need to connect to find from the app container of another project
that wants to index documents to our index. This requires sharing a
common network and exposing our app on it with a service name that
does not clash with the other project.
While developping, we need to run find along other projects that
want to index documents to our index. For the two projects to run
along side each other, we need to avoid port conflicts.
documents can be published under one of three reaches:
- public: anybody can see them
- authenticated: only logged-in users can see them
- restricted: only users listed in the "users" field or belonging
to a group listed in the "groups" field can see them.
The create superuser job should be optional. This job should not be
deployed every time, just when we want. Also, a job can not be patch, so
we have to change its name between version, like for the migrate command
that should be trigger when the appVersion or the chart version is
changing.
We want to secure the connection to opensearch. This will also enable
authentification for the dashboard access. For now we use the default
admin user without custom configuration.
Once started, the cluster can not be paused using kind or ctlptl
command. 2 helpers are added to pause and resume docker containers
running the cluster.
Opensearch was missing in the dev dependencies when deploying in a local
k8s using Tilt. Also the secrets and values have been updated
accordingly to make it works.
Jacques ROUSSEL