Charles Englebert
3ec95c9edf
* ✨(backend) add trivial vector embedding add a trivial vector embedding with constant [0.0, 0.0] * 🙈(core) gitignore ignore files related to sqlite and pdb * ✨(backend) introduce hybrid search handle full-text along with seamntic search * ✨(backend) install basic embedding model I need an embedding to performe semantic search. I need a simple model from hugging face. * ✨(backend) embbed the text embed the text of the query and the document. * 🐛(backend) fix filters and refactor view filter were broken by previous commits. This fixes them. * ♻️(backend) refactor pipeline creation pipeline had to be refactored. This refactors it. * ✨(backend) improve filtering filtering were done once after hybrid computation. For efficiency it should be done of each subquery. * 🔧(setting) move variables to settings NLP_SEARCH_PIPELINE_ID and HYBRID_SEARCH_WEIGHTS is moved to setting file so user can param Find. * ✨(backend) use albert api we choose to rely on Albert API instead of installing a model in local. It is less effort to maintain. * ♻️(backend) hide EMBEDDING_API_KEY EMBEDDING_API_KEY should not be visible. * ✨(backend) move opensearch functions to a service user of find should be able to disable seamntic search. If it is not properly setted it is also turned off without impatcing full-text search. * 🧪(backend) test add test so the app is tested. * 📝(backend) add documentation add documentation so the app is documented. * ♻️(backend) remove local model the local model is no longer useful. Its file must be removed. * 🧪(backend) one more test The app must be tested more. This tests the app more. * ✨(bakckend) handle k value k value must be handled so the user can have a control over the number of results. * ♻️(backend) clean branch beanch had to be cleaned bery very much * 🔧(infra) define EMBEDDING_API_KEY EMBEDDING_API_KEY must be hiden. This hide EMBEDDING_API_KEY * 🚨(backend) fix linters linters must be fixed. This commit fixes them. * 📝(core) add changelog changelog must be updated. This updates the changelog. * 🐛(backend) fix linters more linters had to be fixed more. This fixes linters more. * 🐛(backend) fix tests test must be fixed. This fixes the tests. * ♻️(backend) improve variable managment variable managment must be improve. This improve variable managment. * ♻️(backend) remove embedding from schemas embedding must be removed from schemas. This removes embedding from shemas. * ✨(backend) add reindex_with_embedding command We must be able to enable hybrid search if it was disabled or chnage the embedding model. To do so we must reindex all documents with a new embedding. reindex_with_embedding does that. * ✨(backend) add create_pipeline command We must be able to create the command pipeline once and not check at all request. * 🧪(backend) tests I add a test and fix other tests * 🚨(backend) linters linters must be fixed. This fixes linters. * ✨(backend) remove pagination Semantic search has an impact of pagination. Pagination will be perfomed in services consuming Find API (Doc, Drive etc...) * ✨(backend) improve reindexing we want to handle error case and model change. I introduce a embedding_model field to keep track of the embedding state. * 🧪(backend) test more the command must be tested more. This tests the command more. * 🧪(backend) test concurent update do not lead do data loss updates on a document mught be done by a user while reindexing. I check the latest data is not lost. using if_seq_no and if_primary_term is not only not useful but whould require reruning the command. * ✨(backend) improve reindexing again reindexing must preserve the latest updates. I reintroduce the no_seq update field. * ♻️(backend) various small improvments I make various small improvments.
26 KiB
26 KiB
Find variables
Here we describe all environment variables that can be set for the find application.
find-backend container
These are the environment variables you can set for the find-backend container.
| Option | Description | default |
|---|---|---|
| API_USERS_LIST_LIMIT | Limit on API users | 5 |
| API_USERS_LIST_THROTTLE_RATE_BURST | Throttle rate for api on burst | 30/minute |
| API_USERS_LIST_THROTTLE_RATE_SUSTAINED | Throttle rate for api | 180/hour |
| CACHES_DEFAULT_TIMEOUT | Cache default timeout | 30 |
| CACHES_KEY_PREFIX | The prefix used to every cache keys. | docs |
| DB_ENGINE | Engine to use for database connections | django.db.backends.postgresql_psycopg2 |
| DB_HOST | Host of the database | localhost |
| DB_NAME | Name of the database | impress |
| DB_PASSWORD | Password to authenticate with | pass |
| DB_PORT | Port of the database | 5432 |
| DB_USER | User to authenticate with | dinum |
| DJANGO_ALLOWED_HOSTS | Allowed hosts | [] |
| DJANGO_CELERY_BROKER_TRANSPORT_OPTIONS | Celery broker transport options | {} |
| DJANGO_CELERY_BROKER_URL | Celery broker url | redis://redis:6379/0 |
| DJANGO_CORS_ALLOW_ALL_ORIGINS | Allow all CORS origins | false |
| DJANGO_CORS_ALLOWED_ORIGIN_REGEXES | List of origins allowed for CORS using regulair expressions | [] |
| DJANGO_CORS_ALLOWED_ORIGINS | List of origins allowed for CORS | [] |
| DJANGO_CSRF_TRUSTED_ORIGINS | CSRF trusted origins | [] |
| DJANGO_EMAIL_BACKEND | Email backend library | django.core.mail.backends.smtp.EmailBackend |
| DJANGO_EMAIL_BRAND_NAME | Brand name for email | |
| DJANGO_EMAIL_FROM | Email address used as sender | from@example.com |
| DJANGO_EMAIL_HOST | Hostname of email | |
| DJANGO_EMAIL_HOST_PASSWORD | Password to authenticate with on the email host | |
| DJANGO_EMAIL_HOST_USER | User to authenticate with on the email host | |
| DJANGO_EMAIL_LOGO_IMG | Logo for the email | |
| DJANGO_EMAIL_PORT | Port used to connect to email host | |
| DJANGO_EMAIL_USE_SSL | Use ssl for email host connection | false |
| DJANGO_EMAIL_USE_TLS | Use tls for email host connection | false |
| DJANGO_SECRET_KEY | Secret key | |
| DJANGO_SERVER_TO_SERVER_API_TOKENS | [] | |
| DOCUMENT_IMAGE_MAX_SIZE | Maximum size of document in bytes | 10485760 |
| FRONTEND_CSS_URL | To add a external css file to the app | |
| FRONTEND_HOMEPAGE_FEATURE_ENABLED | Frontend feature flag to display the homepage | false |
| FRONTEND_THEME | Frontend theme to use | |
| LANGUAGE_CODE | Default language | en-us |
| LOGGING_LEVEL_LOGGERS_APP | Application logging level. options are "DEBUG", "INFO", "WARN", "ERROR", "CRITICAL" | INFO |
| LOGGING_LEVEL_LOGGERS_ROOT | Default logging level. options are "DEBUG", "INFO", "WARN", "ERROR", "CRITICAL" | INFO |
| LOGIN_REDIRECT_URL | Login redirect url | |
| LOGIN_REDIRECT_URL_FAILURE | Login redirect url on failure | |
| LOGOUT_REDIRECT_URL | Logout redirect url | |
| MALWARE_DETECTION_BACKEND | The malware detection backend use from the django-lasuite package | lasuite.malware_detection.backends.dummy.DummyBackend |
| MALWARE_DETECTION_PARAMETERS | A dict containing all the parameters to initiate the malware detection backend | {"callback_path": "core.malware_detection.malware_detection_callback",} |
| MEDIA_BASE_URL | ||
| NO_WEBSOCKET_CACHE_TIMEOUT | Cache used to store current editor session key when only users without websocket are editing a document | 120 |
| OIDC_ALLOW_DUPLICATE_EMAILS | Allow duplicate emails | false |
| OIDC_AUTH_REQUEST_EXTRA_PARAMS | OIDC extra auth parameters | {} |
| OIDC_CREATE_USER | Create used on OIDC | false |
| OIDC_DRF_AUTH_BACKEND | DRF Authentication backend class for OIDC | lasuite.oidc_login.backends.OIDCAuthenticationBackend |
| OIDC_FALLBACK_TO_EMAIL_FOR_IDENTIFICATION | Fallback to email for identification | true |
| OIDC_OP_AUTHORIZATION_ENDPOINT | Authorization endpoint for OIDC | |
| OIDC_OP_INTROSPECTION_ENDPOINT | Introspection endpoint for OIDC | |
| OIDC_OP_JWKS_ENDPOINT | JWKS endpoint for OIDC | |
| OIDC_OP_LOGOUT_ENDPOINT | Logout endpoint for OIDC | |
| OIDC_OP_TOKEN_ENDPOINT | Token endpoint for OIDC | |
| OIDC_OP_USER_ENDPOINT | User endpoint for OIDC | |
| OIDC_REDIRECT_ALLOWED_HOSTS | Allowed hosts for OIDC redirect url | [] |
| OIDC_REDIRECT_REQUIRE_HTTPS | Require https for OIDC redirect url | false |
| OIDC_RP_CLIENT_ID | Client id used for OIDC | impress |
| OIDC_RP_CLIENT_SECRET | Client secret used for OIDC | |
| OIDC_RP_SCOPES | Scopes requested for OIDC | openid email |
| OIDC_RP_SIGN_ALGO | verification algorithm used OIDC tokens | RS256 |
| OIDC_RS_BACKEND_CLASS | Resource server backend class for OIDC | core.authentication.FinderResourceServerBackend |
| OIDC_RS_AUDIENCE_CLAIM | The claim used to identify the audience | client_id |
| OIDC_RS_CLIENT_ID | Client id used for OIDC resource server | |
| OIDC_RS_CLIENT_SECRET | Client secret used for OIDC resource server | |
| OIDC_RS_SIGNING_ALGO | Signing algorithm | ES256 |
| OIDC_RS_SCOPES | Required scopes for authentication | ["openid"] |
| OIDC_RS_PRIVATE_KEY_STR | Private key for encryption/decryption | |
| OIDC_RS_ENCRYPTION_KEY_TYPE | Encryption key type (RSA, EC, etc.) | RSA |
| OIDC_RS_ENCRYPTION_ALGO | Encryption algorithm | RSA-OAEP |
| OIDC_RS_ENCRYPTION_ENCODING | Encryption encoding algorithm | A256GCM |
| OIDC_STORE_ID_TOKEN | Store OIDC token | true |
| OIDC_USE_NONCE | Use nonce for OIDC | true |
| OIDC_USERINFO_FULLNAME_FIELDS | OIDC token claims to create full name | ["first_name", "last_name"] |
| OIDC_USERINFO_SHORTNAME_FIELD | OIDC token claims to create shortname | first_name |
| OIDC_VERIFY_SSL | Enable SSL validation for OIDC | true |
| OIDC_TIMEOUT | Timeout delay for OID requests | |
| OIDC_PROXY | Defines a proxy for all requests to the OpenID Connect provider | |
| OPENSEARCH_HOST | Opensearch database url | default |
| OPENSEARCH_PORT | Port of Opensearch database | 9200 |
| OPENSEARCH_USER | Opensearch database user | admin |
| OPENSEARCH_PASSWORD | Opensearch database user password | |
| OPENSEARCH_USE_SSL | Enable SSL connection for Opensearch database | true |
| POSTHOG_KEY | Posthog key for analytics | |
| REDIS_URL | Cache url | redis://redis:6379/1 |
| SENTRY_DSN | Sentry host | |
| SESSION_COOKIE_AGE | duration of the cookie session | 606012 |
| SPECTACULAR_SETTINGS_ENABLE_DJANGO_DEPLOY_CHECK | false | |
| STORAGES_STATICFILES_BACKEND | whitenoise.storage.CompressedManifestStaticFilesStorage | |
| THEME_CUSTOMIZATION_CACHE_TIMEOUT | Cache duration for the customization settings | 86400 |
| THEME_CUSTOMIZATION_FILE_PATH | Full path to the file customizing the theme. An example is provided in src/backend/impress/configuration/theme/default.json | BASE_DIR/impress/configuration/theme/default.json |
| TRASHBIN_CUTOFF_DAYS | Trashbin cutoff | 30 |
| USER_OIDC_ESSENTIAL_CLAIMS | Essential claims in OIDC token | [] |
| Y_PROVIDER_API_BASE_URL | Y Provider url | |
| Y_PROVIDER_API_KEY | Y provider API key | |
| HYBRID_SEARCH_ENABLED | Flag to enable hybrid (an then semantic) search | True |
| HYBRID_SEARCH_WEIGHTS | Weights used in the weighted sum of the hybrid search score | [0.3, 0.7] |
| EMBEDDING_API_PATH | URL of the embedding api | https://albert.api.etalab.gouv.fr/v1/embeddings |
| EMBEDDING_API_KEY | API key of the embedding api | |
| EMBEDDING_REQUEST_TIMEOUT | time out in seconds of the embedding requests | 10 |
| EMBEDDING_API_MODEL_NAME | Name of the embedding model used on the api | embeddings-small |
| EMBEDDING_DIMENSION | Size of the embedding vector | 1024 |