find/docs/setup-indexer.md

Using the Find indexer

This guide explains how to setup the Find service which provide an API for indexation and fulltext search of documents from various sources in a secure way : only the documents within the scope of the user's OIDC token are visible.

Setup Opensearch

General

Add the following variables to your Django settings to configure Find and enable full-text search.

# Login for opensearch
OPENSEARCH_USER=opensearch-user
OPENSEARCH_PASSWORD=your-opensearch-password

# Host configuration
OPENSEARCH_HOST=opensearch
OPENSEARCH_PORT=9200

# Enable SSL for opensearch connection (False in dev mode)
OPENSEARCH_USE_SSL=True

# Prefix for the index name of the registered services.
OPENSEARCH_INDEX_PREFIX=find

Language

Language specific operations are applied to document titles and contents to improve search results. The language is automatically detected by Find. If the language can not be detected no language specific operation are applied and the indexing process is not affected.

Find supports french, english, german and dutch.

The search process is not language specific, a query can get documents of any language.

Language detection estimates a confidence between 0 and 1. If the confidence is below a threshold the language is considered unrecognized. This threshold can be controlled with LANGUAGE_DETECTION_CONFIDENCE_THRESHOLD environment variable.

LANGUAGE_DETECTION_CONFIDENCE_THRESHOLD=0.75

Semantic search

Find offers a semantic search feature. You can either use pure full-text search or a hybrid full-text + semantic search. To enable the hybrid search, add the following settings.

# Enable flag
HYBRID_SEARCH_ENABLED = True

# weighted sum: full_text_weight, semantic_search_weight
HYBRID_SEARCH_WEIGHTS = 0.7,0.3

# Embedding
CHUNK_SIZE=512
CHUNK_OVERLAP=50
EMBEDDING_API_PATH = https://embedding.api.example.com/full/path/
EMBEDDING_API_KEY = your-embedding-api-key
EMBEDDING_REQUEST_TIMEOUT = 10
EMBEDDING_API_MODEL_NAME = embedding-api-model-name
EMBEDDING_DIMENSION = 1024

The hybrid search computes a score for full-text and semantic search and combines them through a weighted sum. HYBRID_SEARCH_WEIGHTS contains the weights of full-text and semantic respectively.

You need to use an embedding api similar to https://albert.api.etalab.gouv.fr/documentation#tag/Embeddings/operation/embeddings_v1_embeddings_post.

document chunking

The indexing process embeds documents by converting their content into vector representations (embeddings). When a document exceeds the character dimension defined by CHUNK_SIZE, it's divided into smaller segments (chunks), with each chunk embedded independently. Each chunk must be smaller than the embedding model's context window .

The chunking algorithm works recursively. It attempts to create the largest possible segments first, then subdivides them further if they still exceed the size limit defined by CHUNK_SIZE. Segments can share overlapping content between them (set CHUNK_OVERLAP=0 to disable overlapping).

During the search, the matching of a document is the matching of its best chunk.

trigrams

Find uses trigrams to improve the robustness of the full text search engine to spelling variations and errors. It can be configured by two environment variables.

TRIGRAMS_BOOST=0.25
TRIGRAMS_MINIMUM_SHOULD_MATCH=0.75%

TRIGRAMS_BOOST is weight boost applied to the trigram score in the document matching. TRIGRAMS_MINIMUM_SHOULD_MATCH is the minimal number or proportion of trigrams having to match to score. It is either an absolute number or proportion.

Setup indexation API

Other applications can index their files through the /index/ endpoint with a simple token authentication.

For each application a new Service must be created through the admin interface (see http://localhost:9071/admin/core/service/add/)

Field	Description
Name	Name of the service and also the name of the index in Opensearch database
Is active	Toggle service availability
Client id	Calling service client_id (e.g impress for docs)
Allowed services for search	List of sub-services. Will add the results from all these index to the search results.
Token (read-only)	Random token for calling service authentication

And add the key in the calling application Django settings.

Development Mode (Docs + Find)

The command make demo will create a working service configuration for docs and drive with predefined secret keys

# Docs
SEARCH_INDEXER_SECRET="find-api-key-for-docs-with-exactly-50-chars-length"

# Drive
SEARCH_INDEXER_SECRET="find-api-key-for-driv-with-exactly-50-chars-length"

Setup search API

The /search/ endpoint is an OIDC ResourceServer view and needs extra Django settings (see lasuite for details)

OIDC_OP_JWKS_ENDPOINT=http://nginx:8083/realms/impress/protocol/openid-connect/certs
OIDC_OP_AUTHORIZATION_ENDPOINT=http://nginx:8083/realms/impress/protocol/openid-connect/auth
OIDC_OP_TOKEN_ENDPOINT=http://nginx:8083/realms/impress/protocol/openid-connect/token
OIDC_OP_USER_ENDPOINT=http://nginx:8083/realms/impress/protocol/openid-connect/userinfo

# To run Find in development mode along other projects like docs/impress
# we should to use OIDC endpoints on a common keycloak realm. e.g :
# OIDC_OP_URL = http://nginx:8083/realms/impress
#
# This will cause a conflict with the 'iss' claim validation rule because the docs realm
# gives {'iss': 'http://localhost:8083/realms/impress'} so it must be the same
OIDC_OP_URL=http://localhost:8083/realms/impress

# Introspection endpoint is needed to get the "audience" and "sub" from the user token
OIDC_OP_INTROSPECTION_ENDPOINT=http://nginx:8083/realms/impress/protocol/openid-connect/token/introspect

# In development, the resource server use insecure settings
# OIDC_VERIFY_SSL=False

# Resource server
OIDC_RS_SCOPES="openid"
OIDC_RS_SIGN_ALGO=RS256

# This backend allows authentication without any model in database. 
OIDC_RS_BACKEND_CLASS="core.authentication.FinderResourceServerBackend"

Development mode (Docs + Find)

Docs and Find projects stacks can be run together but must have the same keycloak server.

So the endpoints of Docs on the 'nginx' domain for ResourceServer authentication and introspection are also used for Find.

IMPORTANT: Keep OIDC_OP_URL on 'localhost' or it will break the OIDC token claims validation : the iss claim from the token of the 'impress' users are 'localhost' and not 'nginx'.