Compare commits

..

31 Commits

Author SHA1 Message Date
Fabre Florian d575545832 🔧(backend) setup Find app dockers to work with Docs
Populate a service configuration "docs" for development.
Move OIDC endpoints to the "impress" realm to allow introspection
of Docs tokens.
Upgrade version of dependencies (fix some security issues).

Signed-off-by: Fabre Florian <ffabre@hybird.org>
2025-10-07 07:06:32 +02:00
Fabre Florian 0dd513a4d3 (backend) improve unit tests
Add tests for schema validation
Use strict list comparison in some search access control tests

Signed-off-by: Fabre Florian <ffabre@hybird.org>
2025-10-02 15:51:27 +02:00
Fabre Florian 7c2a60eb6f (backend) setup search api view as OIDC resource server
New fulltext search view for indexed documents with OIDC authentication
Extract token information through introspection to get the audience & user info
Limit access to documents :
 - public & authenticated with linkreach to the user
 - owned by the user
Check intersection between the allowed services linked to the
audience/client_id and the requested ones.

Signed-off-by: Fabre Florian <ffabre@hybird.org>
2025-09-23 10:00:45 +02:00
Fabre Florian 23c4412114 (views) be a bit more permissive for indexable document content
In indexation view, raise a validation error only when both the
title & content of a document are empty.

Signed-off-by: Fabre Florian <ffabre@hybird.org>
2025-09-15 10:00:26 +02:00
Samuel Paccoud - DINUM fec8c375a2 (views) add tests for access control for an authenticated user
The implementation is partial and fakes resource server but we can
already test the filtering logic in OpenSearch.
2025-08-07 18:22:00 +02:00
Samuel Paccoud - DINUM 3cb3bf3d8f ♻️(schema) stop forcing users to be uuid (represented by a sub)
The list of users allowed to access a document is a sub and is
not guaranteed to be a UUID.
2025-08-07 18:22:00 +02:00
Samuel Paccoud - DINUM b8b335d724 ♻️(views) split views in 2 urls: /index and /search
We need to make a POST to search documents so that we can post a
list of documents the current user has already "visited" This is
necessary to limited the number of documents we return among the
ones available to any authenticated or anonymous user.
2025-08-07 18:22:00 +02:00
Samuel Paccoud - DINUM 89923e694b (schema) add fields to the document
We need to index the tree structure information as well as an
active field that can be set to False when the item is deleted on
the remote service. We could delete the item from our search index
but it is safer to keep all documents synchronized and not only
those which are not deleted.
2025-08-07 17:42:02 +02:00
Samuel Paccoud - DINUM 6b80aa280a (backend) allow passing list of indices via the query string
The client should be able to choose on which indices, among those
to which it has access (check to be added later), the query should
run.
2025-08-05 22:56:41 +02:00
Samuel Paccoud - DINUM c8d1af667c 🧑‍💻(backend) simplify index name by using service name
We added a "find-" prefix for no good reason.
2025-08-05 22:54:53 +02:00
Samuel Paccoud - DINUM 5242417738 🧑‍💻(backend) rename Reach enum class to ReachEnum
enums.Reach was a bit too similar to enums.REACT
2025-08-05 22:54:53 +02:00
Samuel Paccoud - DINUM 7eaa284357 ⬆️(backend) replace "check" by "condition" in CheckConstraint
This change is required before upgrading to Django 6.0
2025-08-05 22:54:53 +02:00
Samuel Paccoud - DINUM d48c837c3e ♻️(backend) pass full service object as request.auth
We were passing the service name which is not what is expected on
this request property.
2025-08-05 16:34:32 +02:00
Samuel Paccoud - DINUM 51dee6475b 🧑‍💻(compose) allow connecting to find from another compose project
We need to connect to find from the app container of another project
that wants to index documents to our index. This requires sharing a
common network and exposing our app on it with a service name that
does not clash with the other project.
2025-07-19 19:08:01 +02:00
Samuel Paccoud - DINUM d8050bf63d 🧑‍💻(compose) allow running in parallel to other lasuite projects
While developping, we need to run find along other projects that
want to index documents to our index. For the two projects to run
along side each other, we need to avoid port conflicts.
2025-07-19 19:05:53 +02:00
Samuel Paccoud - DINUM e31cf57bcd ♻️(schemas) replace "is_public" field by "reach"
documents can be published under one of three reaches:
- public: anybody can see them
- authenticated: only logged-in users can see them
- restricted: only users listed in the "users" field or belonging
  to a group listed in the "groups" field can see them.
2025-07-19 19:04:39 +02:00
Manuel Raynaud f3fa0eca0c 🔖(helm) upgrade chart to version 0.0.2
We added new features to the helm chart. In order to use them we have to
release a new version.
2025-01-27 11:27:50 +01:00
Manuel Raynaud 234cc23b5e 🔧(helm) add annotations on job to force to replace them
A job can not be updated, to resolve this, we can add the argocd
annotation that will replace and force to recreate the jobs.
2025-01-27 11:10:12 +01:00
Manuel Raynaud abc0b772ef 🔧(actions) remove notify argocd action
The notify argocd steps is not used and can be removed.
2025-01-15 11:53:29 +01:00
Manuel Raynaud 460ba8d337 ♻️(helm) make some jobs optional and with a more unique name
The create superuser job should be optional. This job should not be
deployed every time, just when we want. Also, a job can not be patch, so
we have to change its name between version, like for the migrate command
that should be trigger when the appVersion or the chart version is
changing.
2025-01-15 11:53:29 +01:00
Manuel Raynaud cfe59dc986 🔧(back) redefine logging in production environment
The logging settings was not redefined leading to loosing logs in
production. This settings is changed to handle logs on the console
2025-01-15 10:19:55 +01:00
Manuel Raynaud faa4d0ac92 🔧(back) configure settings to detect current hostname in ALLOWED_HOSTS
In the ALLOWED_HOSTS settings we want to detect the current hostname and
allow it.
2025-01-14 16:13:00 +01:00
Manuel Raynaud 4fa36f0381 🔧(action) use helm chart master branch action 2025-01-14 10:23:44 +01:00
Manuel Raynaud 7674f8aae9 🔧(actions) allow override release in chart action 2025-01-14 10:13:05 +01:00
Manuel Raynaud e7ac83340f 🔧(action) allow to manually trigger helm action 2025-01-14 10:06:25 +01:00
Manuel Raynaud bc60d658cf 🔧(actions) push helm chart when code is merged in main branch 2025-01-14 09:57:37 +01:00
Manuel Raynaud 76f5f8d393 🔧(helm) enable security plugin in opensearch
We want to secure the connection to opensearch. This will also enable
authentification for the dashboard access. For now we use the default
admin user without custom configuration.
2025-01-14 09:48:15 +01:00
Manuel Raynaud 1a1c5641f7 💚(helm) fix helm linting
The helm lint was not working, values were missing about the
createsuperuser job
2025-01-14 09:48:15 +01:00
Manuel Raynaud 98627d015c ♻️(helm) move secrets to extra chart
The secrets are used only in developement. For other environment we will
manage them differently.
2025-01-14 09:48:15 +01:00
Manuel Raynaud de105be159 🚀(deploy) publish helm chart on gh-pages
We wanto to publish the helm chart on the gh-pages branch. For this we
use a github actions responsible to do it. Inpired by
numerique-gouv/meet
2025-01-14 09:48:15 +01:00
Manuel Raynaud a58e61c014 🔥(helm) remove non dev environments
Non dev environments will be managed externally, we don't want to deal
with secrets in open source repo anymore.
2025-01-14 09:48:15 +01:00
106 changed files with 7855 additions and 28 deletions
+36
View File
@@ -0,0 +1,36 @@
# Python
__pycache__
*.pyc
**/__pycache__
**/*.pyc
venv
.venv
# System-specific files
.DS_Store
**/.DS_Store
# Docker
docker compose.*
env.d
# Docs
docs
*.md
*.log
# Development/test cache & configurations
data
.cache
.circleci
.git
.vscode
.iml
.idea
db.sqlite3
.mypy_cache
.pylint.d
.pytest_cache
# Frontend
node_modules
+6
View File
@@ -0,0 +1,6 @@
<!---
Thanks for filing an issue 😄 ! Before you submit, please read the following:
Check the other issue templates if you are trying to submit a bug report, feature request, or question
Search open/closed issues before submitting since someone might have asked the same thing before!
-->
+28
View File
@@ -0,0 +1,28 @@
---
name: 🐛 Bug Report
about: If something is not working as expected 🤔.
---
## Bug Report
**Problematic behavior**
A clear and concise description of the behavior.
**Expected behavior/code**
A clear and concise description of what you expected to happen (or code).
**Steps to Reproduce**
1. Do this...
2. Then this...
3. And then the bug happens!
**Environment**
- Drive version:
- Platform:
**Possible Solution**
<!--- Only if you have suggestions on a fix for the bug -->
**Additional context/Screenshots**
Add any other context about the problem here. If applicable, add screenshots to help explain.
+23
View File
@@ -0,0 +1,23 @@
---
name: ✨ Feature Request
about: I have a suggestion (and may want to build it 💪)!
---
## Feature Request
**Is your feature request related to a problem or unsupported use case? Please describe.**
A clear and concise description of what the problem is. For example: I need to do some task and I have an issue...
**Describe the solution you'd like**
A clear and concise description of what you want to happen. Add any considered drawbacks.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Discovery, Documentation, Adoption, Migration Strategy**
If you can, explain how users will be able to use this and possibly write out a version the docs (if applicable).
Maybe a screenshot or design?
**Do you want to work on it through a Pull Request?**
<!-- Make sure to coordinate with us before you spend too much time working on an implementation! -->
@@ -0,0 +1,22 @@
---
name: 🤗 Support Question
about: If you have a question 💬, or something was not clear from the docs!
---
<!-- ^ Click "Preview" for a nicer view! ^
We primarily use GitHub as an issue tracker. If however you're encountering an issue not covered in the docs, we may be able to help! -->
---
Please make sure you have read our [main Readme](https://github.com/suitenumerique/find).
Also make sure it was not already answered in [an open or close issue](https://github.com/suitenumerique/find/issues).
If your question was not covered, and you feel like it should be, fire away! We'd love to improve our docs! 👌
**Topic**
What's the general area of your question: for example, docker setup, database schema, search functionality,...
**Question**
Try to be as specific as possible so we can help you as best we can. Please be patient 🙏
+11
View File
@@ -0,0 +1,11 @@
## Purpose
Description...
## Proposal
Description...
- [] item 1...
- [] item 2...
+52
View File
@@ -0,0 +1,52 @@
name: Deploy
on:
push:
tags:
- 'preprod'
- 'production'
jobs:
notify-argocd:
runs-on: ubuntu-latest
steps:
-
uses: actions/create-github-app-token@v1
id: app-token
with:
app-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.PRIVATE_KEY }}
owner: ${{ github.repository_owner }}
repositories: "drive,secrets"
-
name: Checkout repository
uses: actions/checkout@v2
with:
submodules: recursive
token: ${{ steps.app-token.outputs.token }}
-
name: Load sops secrets
uses: rouja/actions-sops@main
with:
secret-file: secrets/numerique-gouv/drive/secrets.enc.env
age-key: ${{ secrets.SOPS_PRIVATE }}
-
name: Call argocd github webhook
run: |
data='{"ref": "'$GITHUB_REF'","repository": {"html_url":"'$GITHUB_SERVER_URL'/'$GITHUB_REPOSITORY'"}}'
sig=$(echo -n ${data} | openssl dgst -sha1 -hmac ''${ARGOCD_WEBHOOK_SECRET}'' | awk '{print "X-Hub-Signature: sha1="$2}')
curl -X POST -H 'X-GitHub-Event:push' -H "Content-Type: application/json" -H "${sig}" --data "${data}" $ARGOCD_WEBHOOK_URL
sig=$(echo -n ${data} | openssl dgst -sha1 -hmac ''${ARGOCD_PRODUCTION_WEBHOOK_SECRET}'' | awk '{print "X-Hub-Signature: sha1="$2}')
curl -X POST -H 'X-GitHub-Event:push' -H "Content-Type: application/json" -H "${sig}" --data "${data}" $ARGOCD_PRODUCTION_WEBHOOK_URL
start-test-on-preprod:
needs:
- notify-argocd
runs-on: ubuntu-latest
if: startsWith(github.event.ref, 'refs/tags/preprod')
steps:
-
name: Debug
run: |
echo "Start test when preprod is ready"
+50
View File
@@ -0,0 +1,50 @@
name: Docker Hub Workflow
on:
workflow_dispatch:
push:
branches:
- 'main'
tags:
- 'v*'
pull_request:
branches:
- 'main'
env:
DOCKER_USER: 1001:127
jobs:
build-and-push-backend:
runs-on: ubuntu-latest
steps:
-
name: Checkout repository
uses: actions/checkout@v4
-
name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: lasuite/find
-
name: Login to DockerHub
if: github.event_name != 'pull_request'
run: echo "${{ secrets.DOCKER_HUB_PASSWORD }}" | docker login -u "${{ secrets.DOCKER_HUB_USER }}" --password-stdin
-
name: Run trivy scan
uses: numerique-gouv/action-trivy-cache@main
with:
docker-build-args: '--target backend-production -f Dockerfile'
docker-image-name: 'docker.io/lasuite/find:${{ github.sha }}'
continue-on-error: true
-
name: Build and push
uses: docker/build-push-action@v6
with:
context: .
target: backend-production
build-args: DOCKER_USER=${{ env.DOCKER_USER }}:-1000
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
+142
View File
@@ -0,0 +1,142 @@
name: Main Workflow
on:
push:
branches:
- main
pull_request:
branches:
- "*"
jobs:
lint-git:
runs-on: ubuntu-latest
if: github.event_name == 'pull_request' # Makes sense only for pull requests
steps:
- name: Checkout repository
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: show
run: git log
- name: Enforce absence of print statements in code
run: |
! git diff origin/${{ github.event.pull_request.base.ref }}..HEAD -- . ':(exclude)**/find.yml' | grep "print("
- name: Check absence of fixup commits
run: |
! git log | grep 'fixup!'
- name: Install gitlint
run: pip install --user requests gitlint
- name: Lint commit messages added to main
run: ~/.local/bin/gitlint --commits origin/${{ github.event.pull_request.base.ref }}..HEAD
check-changelog:
runs-on: ubuntu-latest
if: |
contains(github.event.pull_request.labels.*.name, 'noChangeLog') == false &&
github.event_name == 'pull_request'
steps:
- name: Checkout repository
uses: actions/checkout@v3
with:
fetch-depth: 50
- name: Check that the CHANGELOG has been modified in the current branch
run: git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.event.after }} | grep 'CHANGELOG.md'
lint-changelog:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v2
- name: Check CHANGELOG max line length
run: |
max_line_length=$(cat CHANGELOG.md | grep -Ev "^\[.*\]: https://github.com" | wc -L)
if [ $max_line_length -ge 80 ]; then
echo "ERROR: CHANGELOG has lines longer than 80 characters."
exit 1
fi
lint-back:
runs-on: ubuntu-latest
defaults:
run:
working-directory: src/backend
steps:
- name: Checkout repository
uses: actions/checkout@v2
- name: Install Python
uses: actions/setup-python@v3
with:
python-version: "3.12"
- name: Install development dependencies
run: pip install --user .[dev]
- name: Check code formatting with ruff
run: ~/.local/bin/ruff format . --diff
- name: Lint code with ruff
run: ~/.local/bin/ruff check .
- name: Lint code with pylint
run: ~/.local/bin/pylint .
test-back:
runs-on: ubuntu-latest
defaults:
run:
working-directory: src/backend
services:
postgres:
image: postgres:15
env:
POSTGRES_DB: find
POSTGRES_USER: dinum
POSTGRES_PASSWORD: pass
ports:
- 5432:5432
# needed because the postgres container does not provide a healthcheck
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
opensearch:
image: opensearchproject/opensearch:latest
env:
discovery.type: single-node
plugins.security.disabled: true
plugins.security.ssl.http.enabled: false
OPENSEARCH_INITIAL_ADMIN_PASSWORD: find.PASS123
ports:
- 9200:9200
env:
DJANGO_CONFIGURATION: Test
DJANGO_SETTINGS_MODULE: find.settings
DJANGO_SECRET_KEY: ThisIsAnExampleKeyForTestPurposeOnly
OIDC_OP_JWKS_ENDPOINT: /endpoint-for-test-purpose-only
DB_HOST: localhost
DB_NAME: find
DB_USER: dinum
DB_PASSWORD: pass
DB_PORT: 5432
STORAGES_STATICFILES_BACKEND: django.contrib.staticfiles.storage.StaticFilesStorage
OPENSEARCH_HOST: localhost
OPENSEARCH_PASSWORD: find.PASS123
OPENSEARCH_USE_SSL: false
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Create writable /data
run: |
sudo mkdir -p /data/media && \
sudo mkdir -p /data/static
- name: Install Python
uses: actions/setup-python@v3
with:
python-version: "3.12"
- name: Install development dependencies
run: pip install --user .[dev]
- name: Run tests
run: ~/.local/bin/pytest
+40
View File
@@ -0,0 +1,40 @@
name: Release Chart
run-name: Release Chart
on:
workflow_dispatch:
push:
branches:
- 'main'
tags:
- 'v*'
pull_request:
branches:
- 'main'
jobs:
release:
if: github.event_name != 'pull_request'
permissions:
contents: write
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Cleanup
run: rm -rf ./src/helm/extra
- name: Install Helm
uses: azure/setup-helm@v4
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
- name: Publish Helm charts
uses: numerique-gouv/helm-gh-pages@master
with:
charts_dir: ./src/helm
linting: on
token: ${{ secrets.GITHUB_TOKEN }}
+81
View File
@@ -0,0 +1,81 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
.DS_Store
.next/
# Translations # Translations
*.pot
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
env.d/development/*
!env.d/development/*.dist
env.d/terraform
# npm
node_modules
# Mails
src/backend/core/templates/mail/
# Typescript client
src/frontend/tsclient
# Swagger
**/swagger.json
# Logs
*.log
# Terraform
.terraform
*.tfstate
*.tfstate.backup
# Test & lint
.coverage
.pylint.d
.pytest_cache
db.sqlite3
.mypy_cache
# Site media
/data/
# IDEs
.idea/
.vscode/
*.iml
.devcontainer
+78
View File
@@ -0,0 +1,78 @@
# All these sections are optional, edit this file as you like.
[general]
# Ignore certain rules, you can reference them by their id or by their full name
# ignore=title-trailing-punctuation, T3
# verbosity should be a value between 1 and 3, the commandline -v flags take precedence over this
# verbosity = 2
# By default gitlint will ignore merge commits. Set to 'false' to disable.
# ignore-merge-commits=true
# By default gitlint will ignore fixup commits. Set to 'false' to disable.
# ignore-fixup-commits=true
# By default gitlint will ignore squash commits. Set to 'false' to disable.
# ignore-squash-commits=true
# Enable debug mode (prints more output). Disabled by default.
# debug=true
# Set the extra-path where gitlint will search for user defined rules
# See http://jorisroovers.github.io/gitlint/user_defined_rules for details
extra-path=gitlint/
# [title-max-length]
# line-length=80
[title-must-not-contain-word]
# Comma-separated list of words that should not occur in the title. Matching is case
# insensitive. It's fine if the keyword occurs as part of a larger word (so "WIPING"
# will not cause a violation, but "WIP: my title" will.
words=wip
#[title-match-regex]
# python like regex (https://docs.python.org/2/library/re.html) that the
# commit-msg title must be matched to.
# Note that the regex can contradict with other rules if not used correctly
# (e.g. title-must-not-contain-word).
#regex=
# [B1]
# B1 = body-max-line-length
# line-length=120
# [body-min-length]
# min-length=5
# [body-is-missing]
# Whether to ignore this rule on merge commits (which typically only have a title)
# default = True
# ignore-merge-commits=false
# [body-changed-file-mention]
# List of files that need to be explicitly mentioned in the body when they are changed
# This is useful for when developers often erroneously edit certain files or git submodules.
# By specifying this rule, developers can only change the file when they explicitly reference
# it in the commit message.
# files=gitlint/rules.py,README.md
# [author-valid-email]
# python like regex (https://docs.python.org/2/library/re.html) that the
# commit author email address should be matched to
# For example, use the following regex if you only want to allow email addresses from foo.com
# regex=[^@]+@foo.com
[ignore-by-title]
# Allow empty body & wrong title pattern only when bots (pyup/greenkeeper)
# upgrade dependencies
regex=^(⬆️.*|Update (.*) from (.*) to (.*)|(chore|fix)\(package\): update .*)$
ignore=B6,UC1
# [ignore-by-body]
# Ignore certain rules for commits of which the body has a line that matches a regex
# E.g. Match bodies that have a line that that contain "release"
# regex=(.*)release(.*)
#
# Ignore certain rules, you can reference them by their id or by their full name
# Use 'all' to ignore all rules
# ignore=T1,body-min-length
+21
View File
@@ -0,0 +1,21 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0),
and this project adheres to
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
# Unreleased
## Added
- backend application
- helm chart
- 🐛(backend) fix missing index creation in 'index/' view
- ✨(backend) allow indexation of documents with either empty content or title.
- ✨(api) new fulltext 'search/' view with OIDC resource server authentication
- ✨(backend) limit access to documents : public & authenticated with a
linkreach & owned ones
- ✨(backend) limit search to the calling app (audience) and a configured
list of services
+137
View File
@@ -0,0 +1,137 @@
# Django find
# ---- base image to inherit from ----
FROM python:3.12-slim-bookworm AS base
# Upgrade pip to its latest release to speed up dependencies installation
RUN python -m pip install --upgrade pip
# Upgrade system packages to install security updates
RUN apt-get update && \
apt-get -y upgrade && \
rm -rf /var/lib/apt/lists/*
# ---- Back-end builder image ----
FROM base AS back-builder
WORKDIR /builder
# Copy required python dependencies
COPY ./src/backend /builder
RUN mkdir /install && \
pip install --prefix=/install .
# ---- static link collector ----
FROM base AS link-collector
ARG FIND_STATIC_ROOT=/data/static
# Install libpangocairo & rdfind
RUN apt-get update && \
apt-get install -y \
libpangocairo-1.0-0 \
rdfind && \
rm -rf /var/lib/apt/lists/*
# Copy installed python dependencies
COPY --from=back-builder /install /usr/local
# Copy find application (see .dockerignore)
COPY ./src/backend /app/
WORKDIR /app
# collectstatic
RUN DJANGO_CONFIGURATION=Build \
DJANGO_JWT_PRIVATE_SIGNING_KEY=Dummy \
OPENSEARCH_PASSWORD=Dummy \
python manage.py collectstatic --noinput
# Replace duplicated file by a symlink to decrease the overall size of the
# final image
RUN rdfind -makesymlinks true -followsymlinks true -makeresultsfile false ${FIND_STATIC_ROOT}
# ---- Core application image ----
FROM base AS core
ENV PYTHONUNBUFFERED=1
# Install required system libs
RUN apt-get update && \
apt-get install -y \
gettext \
libcairo2 \
libffi-dev \
libgdk-pixbuf2.0-0 \
libpango-1.0-0 \
libpangocairo-1.0-0 \
shared-mime-info && \
rm -rf /var/lib/apt/lists/*
# Copy entrypoint
COPY ./docker/files/usr/local/bin/entrypoint /usr/local/bin/entrypoint
# Give the "root" group the same permissions as the "root" user on /etc/passwd
# to allow a user belonging to the root group to add new users; typically the
# docker user (see entrypoint).
RUN chmod g=u /etc/passwd
# Copy installed python dependencies
COPY --from=back-builder /install /usr/local
# Copy find application (see .dockerignore)
COPY ./src/backend /app/
WORKDIR /app
# We wrap commands run in this container by the following entrypoint that
# creates a user on-the-fly with the container user ID (see USER) and root group
# ID.
ENTRYPOINT [ "/usr/local/bin/entrypoint" ]
# ---- Development image ----
FROM core AS backend-development
# Switch back to the root user to install development dependencies
USER root:root
# Install psql
RUN apt-get update && \
apt-get install -y postgresql-client && \
rm -rf /var/lib/apt/lists/*
# Uninstall find and re-install it in editable mode along with development
# dependencies
RUN pip uninstall -y find
RUN pip install -e .[dev]
# Restore the un-privileged user running the application
ARG DOCKER_USER
USER ${DOCKER_USER}
# Target database host (e.g. database engine following docker compose services
# name) & port
ENV DB_HOST=postgresql \
DB_PORT=5432
# Run django development server
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
# ---- Production image ----
FROM core AS backend-production
ARG FIND_STATIC_ROOT=/data/static
# Gunicorn
RUN mkdir -p /usr/local/etc/gunicorn
COPY docker/files/usr/local/etc/gunicorn/find.py /usr/local/etc/gunicorn/find.py
# Un-privileged user running the application
ARG DOCKER_USER
USER ${DOCKER_USER}
# Copy statics
COPY --from=link-collector ${FIND_STATIC_ROOT} ${FIND_STATIC_ROOT}
# The default command runs gunicorn WSGI server in find's main module
CMD ["gunicorn", "-c", "/usr/local/etc/gunicorn/find.py", "find.wsgi:application"]
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2024 Direction Interministérielle du Numérique - Gouvernement Français
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+264
View File
@@ -0,0 +1,264 @@
# /!\ /!\ /!\ /!\ /!\ /!\ /!\ DISCLAIMER /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\
#
# This Makefile is only meant to be used for DEVELOPMENT purpose as we are
# changing the user id that will run in the container.
#
# PLEASE DO NOT USE IT FOR YOUR CI/PRODUCTION/WHATEVER...
#
# /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\
#
# Note to developers:
#
# While editing this file, please respect the following statements:
#
# 1. Every variable should be defined in the ad hoc VARIABLES section with a
# relevant subsection
# 2. Every new rule should be defined in the ad hoc RULES section with a
# relevant subsection depending on the targeted service
# 3. Rules should be sorted alphabetically within their section
# 4. When a rule has multiple dependencies, you should:
# - duplicate the rule name to add the help string (if required)
# - write one dependency per line to increase readability and diffs
# 5. .PHONY rule statement should be written after the corresponding rule
# ==============================================================================
# VARIABLES
BOLD := \033[1m
RESET := \033[0m
GREEN := \033[1;32m
# -- Database
DB_HOST = postgresql
DB_PORT = 5432
# -- Docker
# Get the current user ID to use for docker run and docker exec commands
DOCKER_UID = $(shell id -u)
DOCKER_GID = $(shell id -g)
DOCKER_USER = $(DOCKER_UID):$(DOCKER_GID)
COMPOSE = DOCKER_USER=$(DOCKER_USER) docker compose
COMPOSE_EXEC = $(COMPOSE) exec
COMPOSE_EXEC_APP = $(COMPOSE_EXEC) app
COMPOSE_RUN = $(COMPOSE) run --rm
COMPOSE_RUN_APP = $(COMPOSE_RUN) app
COMPOSE_RUN_CROWDIN = $(COMPOSE_RUN) crowdin crowdin
WAIT_DB = @$(COMPOSE_RUN) dockerize -wait tcp://$(DB_HOST):$(DB_PORT) -timeout 60s
# -- Backend
MANAGE = $(COMPOSE_RUN_APP) python manage.py
# ==============================================================================
# RULES
default: help
data/opensearch:
@mkdir -p data/opensearch
data/static:
@mkdir -p data/static
# -- Project
create-env-files: ## Copy the dist env files to env files
create-env-files: \
env.d/development/common \
env.d/development/crowdin \
env.d/development/postgresql
.PHONY: create-env-files
bootstrap: ## Prepare Docker images for the project
bootstrap: \
data/opensearch \
data/static \
create-env-files \
build \
migrate \
demo \
back-i18n-generate \
back-i18n-compile
.PHONY: bootstrap
# -- Docker/compose
build: ## build the app container
@$(COMPOSE) build app --no-cache
.PHONY: build
down: ## stop and remove containers, networks, images, and volumes
@$(COMPOSE) down
.PHONY: down
logs: ## display app logs (follow mode)
@$(COMPOSE) logs -f app
.PHONY: logs
run: ## start the wsgi (production) and development server
@$(COMPOSE) up --force-recreate -d celery
@echo "Wait for postgresql to be up..."
@$(WAIT_DB)
.PHONY: run
status: ## an alias for "docker compose ps"
@$(COMPOSE) ps
.PHONY: status
stop: ## stop the development server using Docker
@$(COMPOSE) stop
.PHONY: stop
# -- Backend
demo: ## flush db then create a demo for load testing purpose
@$(MAKE) resetdb
@$(MANAGE) create_demo
.PHONY: demo
# Nota bene: Black should come after isort just in case they don't agree...
lint: ## lint back-end python sources
lint: \
lint-ruff-format \
lint-ruff-check \
lint-pylint
.PHONY: lint
lint-ruff-format: ## format back-end python sources with ruff
@echo 'lint:ruff-format started…'
@$(COMPOSE_RUN_APP) ruff format .
.PHONY: lint-ruff-format
lint-ruff-check: ## lint back-end python sources with ruff
@echo 'lint:ruff-check started…'
@$(COMPOSE_RUN_APP) ruff check . --fix
.PHONY: lint-ruff-check
lint-pylint: ## lint back-end python sources with pylint only on changed files from main
@echo 'lint:pylint started…'
bin/pylint .
.PHONY: lint-pylint
test: ## run project tests
@$(MAKE) test-back-parallel
.PHONY: test
test-back: ## run back-end tests
@args="$(filter-out $@,$(MAKECMDGOALS))" && \
bin/pytest $${args:-${1}}
.PHONY: test-back
test-back-parallel: ## run all back-end tests in parallel
@args="$(filter-out $@,$(MAKECMDGOALS))" && \
bin/pytest -n auto $${args:-${1}}
.PHONY: test-back-parallel
makemigrations: ## run django makemigrations for the find project.
@echo "$(BOLD)Running makemigrations$(RESET)"
@$(COMPOSE) up -d postgresql
@$(WAIT_DB)
@$(MANAGE) makemigrations
.PHONY: makemigrations
migrate: ## run django migrations for the find project.
@echo "$(BOLD)Running migrations$(RESET)"
@$(COMPOSE) up -d postgresql
@$(WAIT_DB)
@$(MANAGE) migrate
.PHONY: migrate
superuser: ## Create an admin superuser with password "admin"
@echo "$(BOLD)Creating a Django superuser$(RESET)"
@$(WAIT_DB)
@$(MANAGE) shell -c "from core.models import User; not User.objects.filter(username='admin').exists() and User.objects.create_superuser('admin', 'admin@example.com', 'admin')"
.PHONY: superuser
back-i18n-compile: ## compile the gettext files
@$(MANAGE) compilemessages --ignore="venv/**/*"
.PHONY: back-i18n-compile
back-i18n-generate: ## create the .pot files used for i18n
@$(MANAGE) makemessages -a --keep-pot
.PHONY: back-i18n-generate
shell: ## connect to database shell
@$(MANAGE) shell #_plus
.PHONY: dbshell
# -- Database
dbshell: ## connect to database shell
docker compose exec app python manage.py dbshell
.PHONY: dbshell
resetdb: FLUSH_ARGS ?=
resetdb: ## flush database and create a superuser "admin"
@echo "$(BOLD)Flush database$(RESET)"
@$(MANAGE) flush $(FLUSH_ARGS)
@${MAKE} superuser
.PHONY: resetdb
env.d/development/common:
cp --update=none env.d/development/common.dist env.d/development/common
env.d/development/postgresql:
cp --update=none env.d/development/postgresql.dist env.d/development/postgresql
# -- Internationalization
env.d/development/crowdin:
cp --update=none env.d/development/crowdin.dist env.d/development/crowdin
crowdin-download: ## Download translated message from crowdin
@$(COMPOSE_RUN_CROWDIN) download -c crowdin/config.yml
.PHONY: crowdin-download
crowdin-download-sources: ## Download sources from Crowdin
@$(COMPOSE_RUN_CROWDIN) download sources -c crowdin/config.yml
.PHONY: crowdin-download-sources
crowdin-upload: ## Upload source translations to crowdin
@$(COMPOSE_RUN_CROWDIN) upload sources -c crowdin/config.yml
.PHONY: crowdin-upload
i18n-compile: ## compile all translations
i18n-compile: \
back-i18n-compile
.PHONY: i18n-compile
i18n-generate: ## create the .pot files used for i18n
i18n-generate: \
back-i18n-generate
.PHONY: i18n-generate
i18n-download-and-compile: ## download all translated messages and compile them to be used by all applications
i18n-download-and-compile: \
crowdin-download \
i18n-compile
.PHONY: i18n-download-and-compile
i18n-generate-and-upload: ## generate source translations for all applications and upload them to Crowdin
i18n-generate-and-upload: \
i18n-generate \
crowdin-upload
.PHONY: i18n-generate-and-upload
# -- Misc
clean: ## restore repository state as it was freshly cloned
git clean -idx
.PHONY: clean
help:
@echo "$(BOLD)find Makefile"
@echo "Please use 'make $(BOLD)target$(RESET)' where $(BOLD)target$(RESET) is one of:"
@grep -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(firstword $(MAKEFILE_LIST)) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "$(GREEN)%-30s$(RESET) %s\n", $$1, $$2}'
.PHONY: help
# -- K8S
build-k8s-cluster: ## build the kubernetes cluster using kind
./bin/start-kind.sh
.PHONY: build-k8s-cluster
start-tilt: ## start the kubernetes cluster using kind
tilt up -f ./bin/Tiltfile
.PHONY: build-k8s-cluster
+75
View File
@@ -0,0 +1,75 @@
# Find
Find can index documents from several applications sharing a common OIDC federation
and allows users to search documents with their access rights accross all applications
in the federation.
Find is built on top of [Django Rest
Framework](https://www.django-rest-framework.org/).
## Getting started
### Prerequisite
Make sure you have a recent version of Docker and [Docker
Compose](https://docs.docker.com/compose/install) installed on your laptop:
```bash
$ docker -v
Docker version 27.4.1, build b9d17ea
$ docker compose version
Docker Compose version v2.32.1
```
> ⚠️ You may need to run the following commands with `sudo` but this can be
> avoided by assigning your user to the `docker` group. See docker
> [Documentation](https://docs.docker.com/engine/install/linux-postinstall/)
### Project bootstrap
The easiest way to start working on the project is to use GNU Make:
```bash
$ make bootstrap
```
This command builds the `app` container, installs dependencies, performs
database migrations and compile translations. It's a good idea to use this
command each time you are pulling code from the project repository to avoid
dependency-releated or migration-releated issues.
Your Docker services should now be up and running 🎉
### Adding content
You can create a basic demo site by running:
$ make demo
Finally, you can check all available Make rules using:
```bash
$ make help
```
### Django admin
You can access the Django admin site at
[http://localhost:8071/admin](http://localhost:8071/admin).
You first need to create a superuser account:
```bash
$ make superuser
```
## Contributing
This project is intended to be community-driven, so please, do not hesitate to
get in touch if you have any question related to our implementation or design
decisions.
## License
This work is released under the MIT License (see [LICENSE](./LICENSE)).
+17
View File
@@ -0,0 +1,17 @@
# Upgrade
All instructions to upgrade this project from one release to the next will be
documented in this file. Upgrades must be run sequentially, meaning you should
not skip minor/major releases while upgrading (fix releases can be skipped).
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
For most upgrades, you just need to run the django migrations with
the following command inside your docker container:
`python manage.py migrate`
(Note : in your development environment, you can `make migrate`.)
## [Unreleased]
+47
View File
@@ -0,0 +1,47 @@
load('ext://uibutton', 'cmd_button', 'bool_input', 'location')
load('ext://namespace', 'namespace_create', 'namespace_inject')
namespace_create('find')
docker_build(
'localhost:5001/find-backend:latest',
context='..',
dockerfile='../Dockerfile',
only=['./src/backend', './docker'],
target = 'backend-production',
live_update=[
sync('../src/backend', '/app'),
run(
'pip install .',
trigger=['../src/backend/pyproject.toml']
)
]
)
k8s_yaml(local('cd ../src/helm && helmfile -n find -e dev template .'))
migrate_db = '''
set -eu
# get k8s pod name from tilt resource name
POD_NAME="$(tilt get kubernetesdiscovery find-backend -ojsonpath='{.status.pods[0].name}')"
kubectl -n find exec "$POD_NAME" -- python manage.py migrate --no-input
'''
cmd_button('Migrate db',
argv=['sh', '-c', migrate_db],
resource='find-backend',
icon_name='developer_board',
text='Run database migration',
)
create_demo = '''
set -eu
# get k8s pod name from tilt resource name
POD_NAME="$(tilt get kubernetesdiscovery find-backend -ojsonpath='{.status.pods[0].name}')"
kubectl -n find exec "$POD_NAME" -- python manage.py create_demo --force
'''
cmd_button('Create demo fixtures',
argv=['sh', '-c', create_demo],
resource='find-backend',
icon_name='developer_board',
text='Run demo fixtures',
)
+93
View File
@@ -0,0 +1,93 @@
#!/usr/bin/env bash
set -eo pipefail
REPO_DIR="$(cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd)"
UNSET_USER=0
COMPOSE_FILE="${REPO_DIR}/compose.yml"
COMPOSE_PROJECT="find"
# _set_user: set (or unset) default user id used to run docker commands
#
# usage: _set_user
#
# You can override default user ID (the current host user ID), by defining the
# USER_ID environment variable.
#
# To avoid running docker commands with a custom user, please set the
# $UNSET_USER environment variable to 1.
function _set_user() {
if [ $UNSET_USER -eq 1 ]; then
USER_ID=""
return
fi
# USER_ID = USER_ID or `id -u` if USER_ID is not set
USER_ID=${USER_ID:-$(id -u)}
echo "🙋(user) ID: ${USER_ID}"
}
# docker_compose: wrap docker compose command
#
# usage: docker_compose [options] [ARGS...]
#
# options: docker compose command options
# ARGS : docker compose command arguments
function _docker_compose() {
echo "🐳(compose) project: '${COMPOSE_PROJECT}' file: '${COMPOSE_FILE}'"
docker compose \
-p "${COMPOSE_PROJECT}" \
-f "${COMPOSE_FILE}" \
--project-directory "${REPO_DIR}" \
"$@"
}
# _dc_run: wrap docker compose run command
#
# usage: _dc_run [options] [ARGS...]
#
# options: docker compose run command options
# ARGS : docker compose run command arguments
function _dc_run() {
_set_user
user_args="--user=$USER_ID"
if [ -z $USER_ID ]; then
user_args=""
fi
_docker_compose run --rm $user_args "$@"
}
# _dc_exec: wrap docker compose exec command
#
# usage: _dc_exec [options] [ARGS...]
#
# options: docker compose exec command options
# ARGS : docker compose exec command arguments
function _dc_exec() {
_set_user
echo "🐳(compose) exec command: '\$@'"
user_args="--user=$USER_ID"
if [ -z $USER_ID ]; then
user_args=""
fi
_docker_compose exec $user_args "$@"
}
# _django_manage: wrap django's manage.py command with docker compose
#
# usage : _django_manage [ARGS...]
#
# ARGS : django's manage.py command arguments
function _django_manage() {
_dc_run app python manage.py "$@"
}
Executable
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# shellcheck source=bin/_config.sh
source "$(dirname "${BASH_SOURCE[0]}")/_config.sh"
_docker_compose "$@"
+30
View File
@@ -0,0 +1,30 @@
#!/bin/bash
mkdir -p "$(dirname -- "${BASH_SOURCE[0]}")/../.git/hooks/"
PRE_COMMIT_FILE="$(dirname -- "${BASH_SOURCE[0]}")/../.git/hooks/pre-commit"
cat <<'EOF' >$PRE_COMMIT_FILE
#!/bin/bash
# directories containing potential secrets
DIRS="."
bold=$(tput bold)
normal=$(tput sgr0)
# allow to read user input, assigns stdin to keyboard
exec </dev/tty
for d in $DIRS; do
# find files containing secrets that should be encrypted
for f in $(find "${d}" -type f -regex ".*\.enc\..*"); do
if ! $(grep -q "unencrypted_suffix" $f); then
printf '\xF0\x9F\x92\xA5 '
echo "File $f has non encrypted secrets!"
exit 1
fi
done
done
EOF
chmod +x $PRE_COMMIT_FILE
Executable
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# shellcheck source=bin/_config.sh
source "$(dirname "${BASH_SOURCE[0]}")/_config.sh"
_django_manage "$@"
+6
View File
@@ -0,0 +1,6 @@
#!/bin/sh
set -o errexit
echo "Pausing docker containers running k8s cluster managed with kind"
docker pause $(docker ps --filter name=kind-* --filter status=running --format "{{.Names}}")
Executable
+38
View File
@@ -0,0 +1,38 @@
#!/usr/bin/env bash
# shellcheck source=bin/_config.sh
source "$(dirname "${BASH_SOURCE[0]}")/_config.sh"
declare diff_from
declare -a paths
declare -a args
# Parse options
for arg in "$@"
do
case $arg in
--diff-only=*)
diff_from="${arg#*=}"
shift
;;
-*)
args+=("$arg")
shift
;;
*)
paths+=("$arg")
shift
;;
esac
done
if [[ -n "${diff_from}" ]]; then
# Run pylint only on modified files located in src/backend
# (excluding deleted files and migration files)
# shellcheck disable=SC2207
paths=($(git diff "${diff_from}" --name-only --diff-filter=d -- src/backend ':!**/migrations/*.py' | grep -E '^src/backend/.*\.py$'))
fi
# Fix docker vs local path when project sources are mounted as a volume
read -ra paths <<< "$(echo "${paths[@]}" | sed "s|src/backend/||g")"
_dc_run app pylint "${paths[@]}" "${args[@]}"
Executable
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env bash
source "$(dirname "${BASH_SOURCE[0]}")/_config.sh"
_dc_run \
-e DJANGO_CONFIGURATION=Test \
app \
pytest "$@"
+6
View File
@@ -0,0 +1,6 @@
#!/bin/sh
set -o errexit
echo "Resuming docker containers running k8s cluster managed with kind"
docker unpause $(docker ps --filter name=kind-* --filter status=paused --format "{{.Names}}")
+103
View File
@@ -0,0 +1,103 @@
#!/bin/sh
set -o errexit
CURRENT_DIR=$(pwd)
echo "0. Create ca"
# 0. Create ca
mkcert -install
cd /tmp
mkcert "127.0.0.1.nip.io" "*.127.0.0.1.nip.io"
cd $CURRENT_DIR
echo "1. Create registry container unless it already exists"
# 1. Create registry container unless it already exists
reg_name='kind-registry'
reg_port='5001'
if [ "$(docker inspect -f '{{.State.Running}}' "${reg_name}" 2>/dev/null || true)" != 'true' ]; then
docker run \
-d --restart=always -p "127.0.0.1:${reg_port}:5000" --network bridge --name "${reg_name}" \
registry:2
fi
echo "2. Create kind cluster with containerd registry config dir enabled"
# 2. Create kind cluster with containerd registry config dir enabled
# TODO: kind will eventually enable this by default and this patch will
# be unnecessary.
#
# See:
# https://github.com/kubernetes-sigs/kind/issues/2875
# https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration
# See: https://github.com/containerd/containerd/blob/main/docs/hosts.md
cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"
nodes:
- role: control-plane
image: kindest/node:v1.27.3
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "ingress-ready=true"
extraPortMappings:
- containerPort: 80
hostPort: 80
protocol: TCP
- containerPort: 443
hostPort: 443
protocol: TCP
- role: worker
image: kindest/node:v1.27.3
- role: worker
image: kindest/node:v1.27.3
EOF
echo "3. Add the registry config to the nodes"
# 3. Add the registry config to the nodes
#
# This is necessary because localhost resolves to loopback addresses that are
# network-namespace local.
# In other words: localhost in the container is not localhost on the host.
#
# We want a consistent name that works from both ends, so we tell containerd to
# alias localhost:${reg_port} to the registry container when pulling images
REGISTRY_DIR="/etc/containerd/certs.d/localhost:${reg_port}"
for node in $(kind get nodes); do
docker exec "${node}" mkdir -p "${REGISTRY_DIR}"
cat <<EOF | docker exec -i "${node}" cp /dev/stdin "${REGISTRY_DIR}/hosts.toml"
[host."http://${reg_name}:5000"]
EOF
done
echo "4. Connect the registry to the cluster network if not already connected"
# 4. Connect the registry to the cluster network if not already connected
# This allows kind to bootstrap the network but ensures they're on the same network
if [ "$(docker inspect -f='{{json .NetworkSettings.Networks.kind}}' "${reg_name}")" = 'null' ]; then
docker network connect "kind" "${reg_name}"
fi
echo "5. Document the local registry"
# 5. Document the local registry
# https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: local-registry-hosting
namespace: kube-public
data:
localRegistryHosting.v1: |
host: "localhost:${reg_port}"
help: "https://kind.sigs.k8s.io/docs/user/local-registry/"
EOF
echo "6. Install ingress-nginx"
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
kubectl -n ingress-nginx create secret tls mkcert --key /tmp/127.0.0.1.nip.io+1-key.pem --cert /tmp/127.0.0.1.nip.io+1.pem
kubectl -n ingress-nginx patch deployments.apps ingress-nginx-controller --type 'json' -p '[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--default-ssl-certificate=ingress-nginx/mkcert"}]'
Executable
+25
View File
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
set -eo pipefail
source "$(dirname "${BASH_SOURCE[0]}")/_config.sh"
project=$(_set_openstack_project)
echo "Using \"${project}\" project..."
source "${TERRAFORM_DIRECTORY}/${project}/openrc.sh"
# Run Terraform commands in the Hashicorp docker container via docker compose
# shellcheck disable=SC2068
DOCKER_USER="$(id -u):$(id -g)" \
PROJECT="${project}" \
docker compose run --rm \
-e OS_AUTH_URL \
-e OS_IDENTITY_API_VERSION \
-e OS_USER_DOMAIN_NAME \
-e OS_PROJECT_DOMAIN_NAME \
-e OS_TENANT_ID \
-e OS_TENANT_NAME \
-e OS_USERNAME \
-e OS_PASSWORD \
-e OS_REGION_NAME \
terraform-state "$@"
+4
View File
@@ -0,0 +1,4 @@
#!/bin/bash
git submodule update --init --recursive
git submodule foreach 'git fetch origin; git checkout $(git rev-parse --abbrev-ref HEAD); git reset --hard origin/$(git rev-parse --abbrev-ref HEAD); git submodule update --recursive; git clean -dfx'
+12
View File
@@ -0,0 +1,12 @@
#!/usr/bin/env bash
source "$(dirname "${BASH_SOURCE[0]}")/_config.sh"
_dc_run \
-e DJANGO_CONFIGURATION=Test \
app \
python manage.py spectacular \
--api-version 'v1.0' \
--urlconf 'find.api_urls' \
--format openapi-json \
--file /app/core/tests/swagger/swagger.json
+3
View File
@@ -0,0 +1,3 @@
#!/bin/bash
find . -name "*.enc.*" -exec sops updatekeys -y {} \;
+119
View File
@@ -0,0 +1,119 @@
name: find
services:
postgresql:
image: postgres:15
env_file:
- env.d/development/postgresql
ports:
- "25432:5432"
opensearch:
image: opensearchproject/opensearch:latest
environment:
- discovery.type=single-node
- plugins.security.disabled=true
- plugins.security.ssl.http.enabled=false
- OPENSEARCH_INITIAL_ADMIN_PASSWORD=find.PASS123
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- ./data/opensearch:/usr/share/opensearch/data
ports:
- "9200:9200"
- "9600:9600"
healthcheck:
test: ["CMD", "curl", "--head", "-fsS", "http://localhost:9200/"]
start_period: 5s
interval: 2s
timeout: 2s
retries: 10
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:latest
ports:
- "5601:5601"
environment:
OPENSEARCH_HOSTS: '["http://opensearch:9200"]'
DISABLE_SECURITY_DASHBOARDS_PLUGIN: "true"
depends_on:
opensearch:
condition: service_healthy
restart: true
redis:
image: redis:5
app:
build:
context: .
target: backend-development
args:
DOCKER_USER: ${DOCKER_USER:-1000}
user: ${DOCKER_USER:-1000}
image: find:backend-development
environment:
- PYLINTHOME=/app/.pylint.d
- DJANGO_CONFIGURATION=Development
env_file:
- env.d/development/common
- env.d/development/postgresql
ports:
- "8081:8000"
networks:
default: {}
lasuite-net:
aliases:
- find
volumes:
- ./src/backend:/app
- ./data/static:/data/static
depends_on:
postgresql:
condition: service_started
redis:
condition: service_started
opensearch:
condition: service_healthy
celery:
user: ${DOCKER_USER:-1000}
image: find:backend-development
command: ["celery", "-A", "find.celery_app", "worker", "-l", "DEBUG"]
environment:
- DJANGO_CONFIGURATION=Development
env_file:
- env.d/development/common
- env.d/development/postgresql
volumes:
- ./src/backend:/app
- ./data/static:/data/static
depends_on:
- app
dockerize:
image: jwilder/dockerize
crowdin:
image: crowdin/cli:3.16.0
volumes:
- ".:/app"
env_file:
- env.d/development/crowdin
user: "${DOCKER_USER:-1000}"
working_dir: /app
node:
image: node:18
user: "${DOCKER_USER:-1000}"
environment:
HOME: /tmp
volumes:
- ".:/app"
networks:
lasuite-net:
name: lasuite-net
driver: bridge
+23
View File
@@ -0,0 +1,23 @@
#
# Your crowdin's credentials
#
api_token_env: CROWDIN_API_TOKEN
project_id_env: CROWDIN_PROJECT_ID
base_path_env: CROWDIN_BASE_PATH
#
# Choose file structure in crowdin
# e.g. true or false
#
preserve_hierarchy: true
#
# Files configuration
#
files: [
{
source : "/backend/locale/django.pot",
dest: "/backend-find.pot",
translation : "/backend/locale/%locale_with_underscore%/LC_MESSAGES/django.po"
},
]
+35
View File
@@ -0,0 +1,35 @@
#!/bin/sh
#
# The container user (see USER in the Dockerfile) is an un-privileged user that
# does not exists and is not created during the build phase (see Dockerfile).
# Hence, we use this entrypoint to wrap commands that will be run in the
# container to create an entry for this user in the /etc/passwd file.
#
# The following environment variables may be passed to the container to
# customize running user account:
#
# * USER_NAME: container user name (default: default)
# * HOME : container user home directory (default: none)
#
# To pass environment variables, you can either use the -e option of the docker run command:
#
# docker run --rm -e USER_NAME=foo -e HOME='/home/foo' find:latest python manage.py migrate
#
# or define new variables in an environment file to use with docker or docker compose:
#
# # env.d/production
# USER_NAME=foo
# HOME=/home/foo
#
# docker run --rm --env-file env.d/production find:latest python manage.py migrate
#
echo "🐳(entrypoint) creating user running in the container..."
if ! whoami > /dev/null 2>&1; then
if [ -w /etc/passwd ]; then
echo "${USER_NAME:-default}:x:$(id -u):$(id -g):${USER_NAME:-default} user:${HOME}:/sbin/nologin" >> /etc/passwd
fi
fi
echo "🐳(entrypoint) running your command: ${*}"
exec "$@"
@@ -0,0 +1,16 @@
# Gunicorn-django settings
bind = ["0.0.0.0:8000"]
name = "find"
python_path = "/app"
# Run
graceful_timeout = 90
timeout = 90
workers = 3
# Logging
# Using '-' for the access log file makes gunicorn log accesses to stdout
accesslog = "-"
# Using '-' for the error log file makes gunicorn log errors to stderr
errorlog = "-"
loglevel = "info"
+44
View File
@@ -0,0 +1,44 @@
# Django
DJANGO_ALLOWED_HOSTS=*
DJANGO_SECRET_KEY=ThisIsAnExampleKeyForDevPurposeOnly
DJANGO_SETTINGS_MODULE=find.settings
DJANGO_SUPERUSER_PASSWORD=admin
# Python
PYTHONPATH=/app
# find settings
# Backend url
FIND_BASE_URL="http://localhost:8072"
# Opensearch
OPENSEARCH_PASSWORD=find.PASS123
OPENSEARCH_USE_SSL=false
# OIDC
OIDC_OP_JWKS_ENDPOINT=http://nginx:8083/realms/impress/protocol/openid-connect/certs
OIDC_OP_AUTHORIZATION_ENDPOINT=http://localhost:8083/realms/impress/protocol/openid-connect/auth
OIDC_OP_TOKEN_ENDPOINT=http://nginx:8083/realms/impress/protocol/openid-connect/token
OIDC_OP_USER_ENDPOINT=http://nginx:8083/realms/impress/protocol/openid-connect/userinfo
OIDC_OP_URL=http://nginx:8083/realms/impress
OIDC_OP_INTROSPECTION_ENDPOINT=http://nginx:8083/realms/impress/protocol/openid-connect/token/introspect
OIDC_RP_CLIENT_ID=find
OIDC_RP_CLIENT_SECRET=ThisIsAnExampleKeyForDevPurposeOnly
OIDC_RP_SIGN_ALGO=RS256
OIDC_RP_SCOPES="openid email"
OIDC_REDIRECT_ALLOWED_HOSTS=["http://localhost:8083", "http://localhost:3000"]
OIDC_AUTH_REQUEST_EXTRA_PARAMS={"acr_values": "eidas1"}
# OIDC Resource server
OIDC_DRF_AUTH_BACKEND="lasuite.oidc_login.backends.OIDCAuthenticationBackend"
OIDC_RS_SCOPES="openid"
OIDC_RS_CLIENT_ID=impress
OIDC_RS_CLIENT_SECRET=ThisIsAnExampleKeyForDevPurposeOnly
OIDC_RS_SIGN_ALGO=RS256
OIDC_RS_BACKEND_CLASS="core.authentication.FinderResourceServerBackend"
OIDC_RS_ENCRYPTION_KEY_TYPE="RSA"
+3
View File
@@ -0,0 +1,3 @@
CROWDIN_API_TOKEN=Your-Api-Token
CROWDIN_PROJECT_ID=Your-Project-Id
CROWDIN_BASE_PATH=/app/src
+11
View File
@@ -0,0 +1,11 @@
# Postgresql db container configuration
POSTGRES_DB=find
POSTGRES_USER=dinum
POSTGRES_PASSWORD=pass
# App database configuration
DB_HOST=postgresql
DB_NAME=find
DB_USER=dinum
DB_PASSWORD=pass
DB_PORT=5432
BIN
View File
Binary file not shown.
BIN
View File
Binary file not shown.
BIN
View File
Binary file not shown.
+37
View File
@@ -0,0 +1,37 @@
"""
Gitlint extra rule to validate that the message title is of the form
"<gitmoji>(<scope>) <subject>"
"""
from __future__ import unicode_literals
import re
import requests
from gitlint.rules import CommitMessageTitle, LineRule, RuleViolation
class GitmojiTitle(LineRule):
"""
This rule will enforce that each commit title is of the form "<gitmoji>(<scope>) <subject>"
where gitmoji is an emoji from the list defined in https://gitmoji.carloscuesta.me and
subject should be all lowercase
"""
id = "UC1"
name = "title-should-have-gitmoji-and-scope"
target = CommitMessageTitle
def validate(self, title, _commit):
"""
Download the list possible gitmojis from the project's github repository and check that
title contains one of them.
"""
gitmojis = requests.get(
"https://raw.githubusercontent.com/carloscuesta/gitmoji/master/packages/gitmojis/src/gitmojis.json"
).json()["gitmojis"]
emojis = [item["emoji"] for item in gitmojis]
pattern = r"^({:s})\(.*\)\s[a-z].*$".format("|".join(emojis))
if not re.search(pattern, title):
violation_msg = 'Title does not match regex "<gitmoji>(<scope>) <subject>"'
return [RuleViolation(self.id, violation_msg, title)]
-28
View File
@@ -1,28 +0,0 @@
apiVersion: v1
entries:
find:
- apiVersion: v2
created: "2026-03-16T08:58:33.405623957Z"
digest: 7db8aa465f4f5cf0b56d36ba9eebe0ab7d94536f6b5954362420c1089dacb2c4
name: find
type: application
urls:
- https://suitenumerique.github.io/find/find-0.0.3.tgz
version: 0.0.3
- apiVersion: v2
created: "2025-12-16T18:38:32.210537113Z"
digest: 9a00ea763fb6a7cb79275a0ccc5f7d83cc7bc1e8ac995544f9a6dcfcf1401f19
name: find
type: application
urls:
- https://suitenumerique.github.io/find/find-0.0.2.tgz
version: 0.0.2
- apiVersion: v2
created: "2025-01-27T10:10:29.611386242Z"
digest: bfbe5e73c4a7f306788cecfad9d65b22a89ec167257c3e03e72b097501ce1186
name: find
type: application
urls:
- https://suitenumerique.github.io/find/find-0.0.1.tgz
version: 0.0.1
generated: "2026-03-16T08:58:33.404908133Z"
+13
View File
@@ -0,0 +1,13 @@
{
"extends": ["github>numerique-gouv/renovate-configuration"],
"dependencyDashboard": true,
"labels": ["dependencies", "noChangeLog"],
"packageRules": [
{
"enabled": false,
"groupName": "ignored python dependencies",
"matchManagers": ["pep621"],
"matchPackageNames": []
}
]
}
+472
View File
@@ -0,0 +1,472 @@
[MASTER]
# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code
extension-pkg-whitelist=
# Add files or directories to the blacklist. They should be base names, not
# paths.
ignore=migrations
# Add files or directories matching the regex patterns to the blacklist. The
# regex matches against base names, not paths.
ignore-patterns=
# Python code to execute, usually for sys.path manipulation such as
# pygtk.require().
#init-hook=
# Use multiple processes to speed up Pylint. Specifying 0 will auto-detect the
# number of processors available to use.
jobs=0
# List of plugins (as comma separated values of python modules names) to load,
# usually to register additional checkers.
load-plugins=pylint_django,pylint.extensions.no_self_use
# Pickle collected data for later comparisons.
persistent=yes
# Specify a configuration file.
#rcfile=
# When enabled, pylint would attempt to guess common misconfiguration and emit
# user-friendly hints instead of false-positive error messages
suggestion-mode=yes
# Allow loading of arbitrary C extensions. Extensions are imported into the
# active Python interpreter and may run arbitrary code.
unsafe-load-any-extension=no
[MESSAGES CONTROL]
# Only show warnings with the listed confidence levels. Leave empty to show
# all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED
confidence=
# Disable the message, report, category or checker with the given id(s). You
# can either give multiple identifiers separated by comma (,) or put this
# option multiple times (only on the command line, not in the configuration
# file where it should appear only once).You can also use "--disable=all" to
# disable everything first and then reenable specific checks. For example, if
# you want to run only the similarities checker, you can use "--disable=all
# --enable=similarities". If you want to run only the classes checker, but have
# no Warning level messages displayed, use"--disable=all --enable=classes
# --disable=W"
disable=bad-inline-option,
deprecated-pragma,
django-not-configured,
file-ignored,
locally-disabled,
no-self-use,
raw-checker-failed,
suppressed-message,
useless-suppression
# Enable the message, report, category or checker with the given id(s). You can
# either give multiple identifier separated by comma (,) or put this option
# multiple time (only on the command line, not in the configuration file where
# it should appear only once). See also the "--disable" option for examples.
enable=c-extension-no-member
[REPORTS]
# Python expression which should return a note less than 10 (10 is the highest
# note). You have access to the variables errors warning, statement which
# respectively contain the number of errors / warnings messages and the total
# number of statements analyzed. This is used by the global evaluation report
# (RP0004).
evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10)
# Template used to display messages. This is a python new-style format string
# used to format the message information. See doc for all details
#msg-template=
# Set the output format. Available formats are text, parseable, colorized, json
# and msvs (visual studio).You can also give a reporter class, eg
# mypackage.mymodule.MyReporterClass.
output-format=text
# Tells whether to display a full report or only the messages
reports=no
# Activate the evaluation score.
score=yes
[REFACTORING]
# Maximum number of nested blocks for function / method body
max-nested-blocks=5
# Complete name of functions that never returns. When checking for
# inconsistent-return-statements if a never returning function is called then
# it will be considered as an explicit return statement and no message will be
# printed.
never-returning-functions=optparse.Values,sys.exit
[LOGGING]
# Logging modules to check that the string format arguments are in logging
# function parameter format
logging-modules=logging
[SPELLING]
# Limits count of emitted suggestions for spelling mistakes
max-spelling-suggestions=4
# Spelling dictionary name. Available dictionaries: none. To make it working
# install python-enchant package.
spelling-dict=
# List of comma separated words that should not be checked.
spelling-ignore-words=
# A path to a file that contains private dictionary; one word per line.
spelling-private-dict-file=
# Tells whether to store unknown words to indicated private dictionary in
# --spelling-private-dict-file option instead of raising a message.
spelling-store-unknown-words=no
[MISCELLANEOUS]
# List of note tags to take in consideration, separated by a comma.
notes=FIXME,
XXX,
TODO
[TYPECHECK]
# List of decorators that produce context managers, such as
# contextlib.contextmanager. Add to this list to register other decorators that
# produce valid context managers.
contextmanager-decorators=contextlib.contextmanager
# List of members which are set dynamically and missed by pylint inference
# system, and so shouldn't trigger E1101 when accessed. Python regular
# expressions are accepted.
generated-members=
# Tells whether missing members accessed in mixin class should be ignored. A
# mixin class is detected if its name ends with "mixin" (case insensitive).
ignore-mixin-members=yes
# This flag controls whether pylint should warn about no-member and similar
# checks whenever an opaque object is returned when inferring. The inference
# can return multiple potential results while evaluating a Python object, but
# some branches might not be evaluated, which results in partial inference. In
# that case, it might be useful to still emit no-member and other checks for
# the rest of the inferred objects.
ignore-on-opaque-inference=yes
# List of class names for which member attributes should not be checked (useful
# for classes with dynamically set attributes). This supports the use of
# qualified names.
ignored-classes=optparse.Values,thread._local,_thread._local,responses,
Template,Contact
# List of module names for which member attributes should not be checked
# (useful for modules/projects where namespaces are manipulated during runtime
# and thus existing member attributes cannot be deduced by static analysis. It
# supports qualified module names, as well as Unix pattern matching.
ignored-modules=
# Show a hint with possible names when a member name was not found. The aspect
# of finding the hint is based on edit distance.
missing-member-hint=yes
# The minimum edit distance a name should have in order to be considered a
# similar match for a missing member name.
missing-member-hint-distance=1
# The total number of similar names that should be taken in consideration when
# showing a hint for a missing member.
missing-member-max-choices=1
[VARIABLES]
# List of additional names supposed to be defined in builtins. Remember that
# you should avoid to define new builtins when possible.
additional-builtins=
# Tells whether unused global variables should be treated as a violation.
allow-global-unused-variables=yes
# List of strings which can identify a callback function by name. A callback
# name must start or end with one of those strings.
callbacks=cb_,
_cb
# A regular expression matching the name of dummy variables (i.e. expectedly
# not used).
dummy-variables-rgx=_+$|(_[a-zA-Z0-9_]*[a-zA-Z0-9]+?$)|dummy|^ignored_|^unused_
# Argument names that match this expression will be ignored. Default to name
# with leading underscore
ignored-argument-names=_.*|^ignored_|^unused_
# Tells whether we should check for unused import in __init__ files.
init-import=no
# List of qualified module names which can have objects that can redefine
# builtins.
redefining-builtins-modules=six.moves,past.builtins,future.builtins
[FORMAT]
# Expected format of line ending, e.g. empty (any line ending), LF or CRLF.
expected-line-ending-format=
# Regexp for a line that is allowed to be longer than the limit.
ignore-long-lines=^\s*(# )?<?https?://\S+>?$
# Number of spaces of indent required inside a hanging or continued line.
indent-after-paren=4
# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1
# tab).
indent-string=' '
# Maximum number of characters on a single line.
max-line-length=100
# Maximum number of lines in a module
max-module-lines=1000
# Allow the body of a class to be on the same line as the declaration if body
# contains single statement.
single-line-class-stmt=no
# Allow the body of an if to be on the same line as the test if there is no
# else.
single-line-if-stmt=no
[SIMILARITIES]
# Ignore comments when computing similarities.
ignore-comments=yes
# Ignore docstrings when computing similarities.
ignore-docstrings=yes
# Ignore imports when computing similarities.
ignore-imports=yes
# Minimum lines number of a similarity.
# First implementations of CMS wizards have common fields we do not want to factorize for now
min-similarity-lines=35
[BASIC]
# Naming style matching correct argument names
argument-naming-style=snake_case
# Regular expression matching correct argument names. Overrides argument-
# naming-style
#argument-rgx=
# Naming style matching correct attribute names
attr-naming-style=snake_case
# Regular expression matching correct attribute names. Overrides attr-naming-
# style
#attr-rgx=
# Bad variable names which should always be refused, separated by a comma
bad-names=foo,
bar,
baz,
toto,
tutu,
tata
# Naming style matching correct class attribute names
class-attribute-naming-style=any
# Regular expression matching correct class attribute names. Overrides class-
# attribute-naming-style
#class-attribute-rgx=
# Naming style matching correct class names
class-naming-style=PascalCase
# Regular expression matching correct class names. Overrides class-naming-style
#class-rgx=
# Naming style matching correct constant names
const-naming-style=UPPER_CASE
# Regular expression matching correct constant names. Overrides const-naming-
# style
const-rgx=(([A-Z_][A-Z0-9_]*)|(__.*__)|urlpatterns|logger)$
# Minimum line length for functions/classes that require docstrings, shorter
# ones are exempt.
docstring-min-length=-1
# Naming style matching correct function names
function-naming-style=snake_case
# Regular expression matching correct function names. Overrides function-
# naming-style
#function-rgx=
# Good variable names which should always be accepted, separated by a comma
good-names=i,
j,
k,
cm,
ex,
Run,
_
# Include a hint for the correct naming format with invalid-name
include-naming-hint=no
# Naming style matching correct inline iteration names
inlinevar-naming-style=any
# Regular expression matching correct inline iteration names. Overrides
# inlinevar-naming-style
#inlinevar-rgx=
# Naming style matching correct method names
method-naming-style=snake_case
# Regular expression matching correct method names. Overrides method-naming-
# style
method-rgx=([a-z_][a-z0-9_]{2,50}|setUp|set[Uu]pClass|tearDown|tear[Dd]ownClass|assert[A-Z]\w*|maxDiff|test_[a-z0-9_]+)$
# Naming style matching correct module names
module-naming-style=snake_case
# Regular expression matching correct module names. Overrides module-naming-
# style
#module-rgx=
# Colon-delimited sets of names that determine each other's naming style when
# the name regexes allow several styles.
name-group=
# Regular expression which should only match function or class names that do
# not require a docstring.
no-docstring-rgx=^_
# List of decorators that produce properties, such as abc.abstractproperty. Add
# to this list to register other decorators that produce valid properties.
property-classes=abc.abstractproperty
# Naming style matching correct variable names
variable-naming-style=snake_case
# Regular expression matching correct variable names. Overrides variable-
# naming-style
#variable-rgx=
[IMPORTS]
# Allow wildcard imports from modules that define __all__.
allow-wildcard-with-all=no
# Analyse import fallback blocks. This can be used to support both Python 2 and
# 3 compatible code, which means that the block might have code that exists
# only in one or another interpreter, leading to false positives when analysed.
analyse-fallback-blocks=no
# Deprecated modules which should not be used, separated by a comma
deprecated-modules=optparse,tkinter.tix
# Create a graph of external dependencies in the given file (report RP0402 must
# not be disabled)
ext-import-graph=
# Create a graph of every (i.e. internal and external) dependencies in the
# given file (report RP0402 must not be disabled)
import-graph=
# Create a graph of internal dependencies in the given file (report RP0402 must
# not be disabled)
int-import-graph=
# Force import order to recognize a module as part of the standard
# compatibility libraries.
known-standard-library=
# Force import order to recognize a module as part of a third party library.
known-third-party=enchant
[CLASSES]
# List of method names used to declare (i.e. assign) instance attributes.
defining-attr-methods=__init__,
__new__,
setUp
# List of member names, which should be excluded from the protected access
# warning.
exclude-protected=_asdict,
_fields,
_replace,
_source,
_make
# List of valid names for the first argument in a class method.
valid-classmethod-first-arg=cls
# List of valid names for the first argument in a metaclass class method.
valid-metaclass-classmethod-first-arg=mcs
[DESIGN]
# Maximum number of arguments for function / method
max-args=5
# Maximum number of attributes for a class (see R0902).
max-attributes=7
# Maximum number of boolean expressions in a if statement
max-bool-expr=5
# Maximum number of branch for function / method body
max-branches=12
# Maximum number of locals for function / method body
max-locals=15
# Maximum number of parents for a class (see R0901).
max-parents=7
# Maximum number of public methods for a class (see R0904).
max-public-methods=20
# Maximum number of return / yield for function / method body
max-returns=6
# Maximum number of statements in function / method body
max-statements=50
# Minimum number of public methods for a class (see R0903).
min-public-methods=0
[EXCEPTIONS]
# Exceptions that will emit a warning when being caught. Defaults to
# "Exception"
overgeneral-exceptions=builtins.Exception
+3
View File
@@ -0,0 +1,3 @@
include LICENSE
include README.md
recursive-include src/backend/find *.html *.png *.gif *.css *.ico *.jpg *.jpeg *.po *.mo *.eot *.svg *.ttf *.woff *.woff2
View File
View File
+16
View File
@@ -0,0 +1,16 @@
"""Admin config for find's core app"""
from django.contrib import admin
from .models import Service
@admin.register(Service)
class ServiceAdmin(admin.ModelAdmin):
"""Register the serivce model for the admin site"""
list_display = ("name", "created_at", "is_active")
search_fields = ("name",)
list_filter = ("is_active", "created_at")
ordering = ("-created_at",)
readonly_fields = ("created_at", "token")
+37
View File
@@ -0,0 +1,37 @@
"""Impress core API endpoints"""
from django.core.exceptions import ValidationError
from pydantic import ValidationError as PydanticValidationError
from rest_framework import exceptions as drf_exceptions
from rest_framework import status
from rest_framework import views as drf_views
from rest_framework.response import Response
def exception_handler(exc, context):
"""Handle Django ValidationError as an accepted exception.
For the parameters, see ``exception_handler``
This code comes from twidi's gist:
https://gist.github.com/twidi/9d55486c36b6a51bdcb05ce3a763e79f
"""
if isinstance(exc, ValidationError):
detail = exc.message_dict
if hasattr(exc, "message"):
detail = exc.message
elif hasattr(exc, "messages"):
detail = exc.messages
exc = drf_exceptions.ValidationError(detail=detail)
elif isinstance(exc, PydanticValidationError):
return Response(
[
{key: error[key] for key in ("msg", "type", "loc")}
for error in exc.errors()
],
status=status.HTTP_400_BAD_REQUEST,
)
return drf_views.exception_handler(exc, context)
+76
View File
@@ -0,0 +1,76 @@
"""Token authentication."""
import logging
from django.contrib.auth.models import AnonymousUser
from django.core.exceptions import ObjectDoesNotExist
from lasuite.oidc_resource_server.backend import ResourceServerBackend
from rest_framework import authentication, exceptions
from .models import Service
logger = logging.getLogger(__name__)
class ServiceTokenAuthentication(authentication.BaseAuthentication):
"""A custom authentication looking for valid tokens among registered services"""
model = Service
def authenticate(self, request):
"""Authenticate token from the "Authorization" header."""
token = request.headers.get("Authorization")
if not token:
raise exceptions.NotAuthenticated()
token = token.split(" ")[-1] # Extract token if prefixed with "Token"
return self.authenticate_credentials(token)
def authenticate_credentials(self, token):
"""Check that the token is registered and valid."""
try:
service = self.model.objects.only("name").get(token=token, is_active=True)
except self.model.DoesNotExist as excpt:
raise exceptions.AuthenticationFailed("Invalid token.") from excpt
# We don't associate tokens with a user
return AnonymousUser(), service
class ResourceUserManager:
"""Fake manager that simply returns an instance of user object with the right sub"""
def get(self, sub):
"""Returns a ResourceUser for this sub"""
return ResourceUser(sub=sub)
class ResourceUser:
"""Fake user model for the ResourceServerBackend.get_object() method"""
DoesNotExist = ObjectDoesNotExist
objects = ResourceUserManager()
def __init__(self, sub: str):
self.sub = sub
self.pk = None
self.is_authenticated = True
class FinderResourceServerBackend(ResourceServerBackend):
"""
Custom resource server that uses a ResourceUser object instead of
a user model from the database
"""
def __init__(self):
try:
super().__init__()
except Exception as e:
logger.error(e)
raise
self.UserModel = ResourceUser
+32
View File
@@ -0,0 +1,32 @@
"""Enums for find's core app."""
from enum import Enum
# Reach
class ReachEnum(str, Enum):
"""Publication options for indexed documents"""
PUBLIC = "public"
AUTHENTICATED = "authenticated"
RESTRICTED = "restricted"
# Fields
CREATED_AT = "created_at"
DEPTH = "depth"
PATH = "path"
NUMCHILD = "numchild"
REACH = "reach"
SIZE = "size"
TITLE = "title"
UPDATED_AT = "updated_at"
USERS = "users"
GROUPS = "groups"
RELEVANCE = "relevance"
ORDER_BY_OPTIONS = (RELEVANCE, TITLE, CREATED_AT, UPDATED_AT, SIZE, REACH)
SOURCE_FIELDS = (TITLE, SIZE, DEPTH, PATH, NUMCHILD, CREATED_AT, UPDATED_AT, REACH)
+58
View File
@@ -0,0 +1,58 @@
"""Factories for the find's core app"""
from uuid import uuid4
from django.utils import timezone
from django.utils.text import slugify
import factory
from faker import Faker
from . import enums, models
fake = Faker()
class DocumentSchemaFactory(factory.DictFactory):
"""
A factory for generating dictionaries that represent a document for
indexation for testing and development purposes.
"""
id = factory.LazyFunction(lambda: str(uuid4()))
title = factory.Sequence(lambda n: f"Test title {n!s}")
path = factory.Sequence(lambda n: f"000{n}")
content = factory.Sequence(lambda n: f"Test content {n!s}")
created_at = factory.LazyFunction(
lambda: fake.date_time_this_decade(tzinfo=timezone.get_current_timezone())
)
size = factory.LazyFunction(lambda: fake.random_int(min=0, max=1024**2))
users = factory.LazyFunction(lambda: [str(uuid4()) for _ in range(3)])
groups = factory.LazyFunction(lambda: [slugify(fake.word()) for _ in range(3)])
reach = factory.Iterator(list(enums.ReachEnum))
depth = 1
numchild = 0
is_active = True
@factory.lazy_attribute
def updated_at(self):
"""Ensure updated_at is after created_at and before now"""
return fake.date_time_between(
start_date=self.created_at,
end_date=timezone.now(),
tzinfo=timezone.get_current_timezone(),
)
class ServiceFactory(factory.django.DjangoModelFactory):
"""
A factory for generating service instances for testing and development purposes.
"""
name = factory.Sequence(lambda n: f"test-index-{n!s}")
created_at = factory.Faker("date_time_this_year", tzinfo=None)
is_active = True
client_id = "some_client_id"
class Meta:
model = models.Service
@@ -0,0 +1,64 @@
# Generated by Django 5.0.7 on 2024-09-01 05:31
import django.contrib.auth.models
import django.contrib.auth.validators
import django.utils.timezone
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = [
('auth', '0012_alter_user_first_name_max_length'),
]
operations = [
migrations.CreateModel(
name='Service',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.SlugField(max_length=20, unique=True)),
('token', models.CharField(max_length=50)),
('created_at', models.DateTimeField(auto_now_add=True)),
('is_active', models.BooleanField(default=True)),
],
options={
'verbose_name': 'service',
'verbose_name_plural': 'services',
'db_table': 'find_service',
'ordering': ['-is_active', '-created_at'],
},
),
migrations.CreateModel(
name='User',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('password', models.CharField(max_length=128, verbose_name='password')),
('last_login', models.DateTimeField(blank=True, null=True, verbose_name='last login')),
('is_superuser', models.BooleanField(default=False, help_text='Designates that this user has all permissions without explicitly assigning them.', verbose_name='superuser status')),
('username', models.CharField(error_messages={'unique': 'A user with that username already exists.'}, help_text='Required. 150 characters or fewer. Letters, digits and @/./+/-/_ only.', max_length=150, unique=True, validators=[django.contrib.auth.validators.UnicodeUsernameValidator()], verbose_name='username')),
('first_name', models.CharField(blank=True, max_length=150, verbose_name='first name')),
('last_name', models.CharField(blank=True, max_length=150, verbose_name='last name')),
('email', models.EmailField(blank=True, max_length=254, verbose_name='email address')),
('is_staff', models.BooleanField(default=False, help_text='Designates whether the user can log into this admin site.', verbose_name='staff status')),
('is_active', models.BooleanField(default=True, help_text='Designates whether this user should be treated as active. Unselect this instead of deleting accounts.', verbose_name='active')),
('date_joined', models.DateTimeField(default=django.utils.timezone.now, verbose_name='date joined')),
('groups', models.ManyToManyField(blank=True, help_text='The groups this user belongs to. A user will get all permissions granted to each of their groups.', related_name='user_set', related_query_name='user', to='auth.group', verbose_name='groups')),
('user_permissions', models.ManyToManyField(blank=True, help_text='Specific permissions for this user.', related_name='user_set', related_query_name='user', to='auth.permission', verbose_name='user permissions')),
],
options={
'verbose_name': 'user',
'verbose_name_plural': 'users',
'abstract': False,
},
managers=[
('objects', django.contrib.auth.models.UserManager()),
],
),
migrations.AddConstraint(
model_name='service',
constraint=models.CheckConstraint(condition=models.Q(('token__length', 50)), name='token_length_exact_50'),
),
]
@@ -0,0 +1,23 @@
# Generated by Django 5.1.4 on 2025-09-05 04:41
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('core', '0001_initial'),
]
operations = [
migrations.AddField(
model_name='service',
name='client_id',
field=models.CharField(blank=True, null=True),
),
migrations.AddField(
model_name='service',
name='services',
field=models.ManyToManyField(blank=True, to='core.service', verbose_name='Allowed services for search'),
),
]
+63
View File
@@ -0,0 +1,63 @@
"""Models for find's core app"""
import secrets
import string
from django.contrib.auth.models import AbstractUser
from django.db import models
from django.db.models.functions import Length
from django.utils.text import slugify
from django.utils.translation import gettext_lazy as _
models.CharField.register_lookup(Length)
TOKEN_LENGTH = 50
class User(AbstractUser):
"""User for the find application"""
class Service(models.Model):
"""Service registered to index its documents to our find"""
name = models.SlugField(max_length=20, unique=True)
token = models.CharField(max_length=TOKEN_LENGTH)
created_at = models.DateTimeField(auto_now_add=True)
is_active = models.BooleanField(default=True)
client_id = models.CharField(blank=True, null=True)
services = models.ManyToManyField(
"self",
verbose_name=_("Allowed services for search"),
blank=True,
)
class Meta:
db_table = "find_service"
verbose_name = _("service")
verbose_name_plural = _("services")
ordering = ["-is_active", "-created_at"]
constraints = [
models.CheckConstraint(
condition=models.Q(token__length=TOKEN_LENGTH),
name="token_length_exact_50",
),
]
def __str__(self):
return self.name
def save(self, *args, **kwargs):
"""Automatically slugify the service name and generate a token on creation"""
self.name = slugify(self.name)
if not self.token:
self.token = self.generate_secure_token()
super().save(*args, **kwargs)
@staticmethod
def generate_secure_token():
"""Generate a secure token with with Python secret module"""
characters = (
string.ascii_letters + string.digits + r"""!#%&'()*+,-./:;<=>?@[\]^_`{|}~"""
)
token = "".join(secrets.choice(characters) for _ in range(TOKEN_LENGTH))
return token
+54
View File
@@ -0,0 +1,54 @@
"""Opensearch related utils."""
from django.conf import settings
from opensearchpy import OpenSearch
from opensearchpy.exceptions import NotFoundError
client = OpenSearch(
hosts=[{"host": settings.OPENSEARCH_HOST, "port": settings.OPENSEARCH_PORT}],
http_auth=(settings.OPENSEARCH_USER, settings.OPENSEARCH_PASSWORD),
timeout=50,
use_ssl=settings.OPENSEARCH_USE_SSL,
verify_certs=False,
)
def ensure_index_exists(index_name):
"""Create index if it does not exist"""
try:
client.indices.get(index=index_name)
except NotFoundError:
client.indices.create(
index=index_name,
body={
"mappings": {
"dynamic": "strict",
"properties": {
"id": {"type": "keyword"},
"title": {
"type": "keyword", # Primary field for exact matches and sorting
"fields": {
"text": {
"type": "text"
} # Sub-field for full-text search
},
},
"depth": {"type": "integer"},
"path": {
"type": "keyword",
"fields": {"text": {"type": "text"}},
},
"numchild": {"type": "integer"},
"content": {"type": "text"},
"created_at": {"type": "date"},
"updated_at": {"type": "date"},
"size": {"type": "long"},
"users": {"type": "keyword"},
"groups": {"type": "keyword"},
"reach": {"type": "keyword"},
"is_active": {"type": "boolean"},
},
}
},
)
+12
View File
@@ -0,0 +1,12 @@
"""Permission classes for find's core app"""
from rest_framework import permissions
class IsAuthAuthenticated(permissions.BasePermission):
"""
Allows access only to auth authenticated users.
"""
def has_permission(self, request, view):
return bool(request.auth)
+118
View File
@@ -0,0 +1,118 @@
"""Pydantic model to validate documents before indexation."""
from typing import Annotated, List, Literal, Optional
from django.utils import timezone
from django.utils.text import slugify
from pydantic import (
UUID4,
AwareDatetime,
BaseModel,
BeforeValidator,
ConfigDict,
Field,
conint,
field_validator,
model_validator,
)
from . import enums
class DocumentSchema(BaseModel):
"""Schema for validating the documents submitted to our API for indexing"""
id: UUID4
title: Annotated[str, Field(max_length=300, min_length=0)]
depth: Annotated[int, Field(ge=0)]
path: Annotated[str, Field(max_length=300)]
numchild: Annotated[int, Field(ge=0)]
content: Annotated[str, Field(min_length=0)]
created_at: AwareDatetime
updated_at: AwareDatetime
size: Annotated[int, Field(ge=0, le=100 * 1024**3)] # File size limited to 100GB
users: List[Annotated[str, Field(max_length=50)]] = Field(default_factory=list)
groups: List[Annotated[str, Field(pattern=r"^[a-z0-9]+(?:-[a-z0-9]+)*$")]] = Field(
default_factory=list
)
reach: Optional[enums.ReachEnum] = Field(default=enums.ReachEnum.RESTRICTED)
is_active: bool
model_config = ConfigDict(
str_min_length=1, str_strip_whitespace=True, use_enum_values=True
)
@field_validator("title")
@staticmethod
def normalize_title(value):
"""Normalize the title field by stripping whitespace and converting to lowercase"""
return value.strip().lower()
@field_validator("created_at", "updated_at")
@staticmethod
def must_be_past(value, info):
"""Validate that `created_at` and `updated_at` fields are in the past"""
if value >= timezone.now():
raise ValueError(f"{info.field_name} must be earlier than now")
return value
@model_validator(mode="after")
def check_empty_content(self):
"""Validate that either `title` or `content` are not empty."""
if not self.title and not self.content:
raise ValueError("Either title or content should have at least 1 character")
return self
@model_validator(mode="after")
def check_update_at_after_created_at(self):
"""Date and time of last modification should be later than date and time of creation"""
if self.created_at > self.updated_at:
raise ValueError("updated_at must be later than created_at")
return self
@field_validator("groups")
@staticmethod
def validate_groups(groups):
"""Validate that group slugs are properly formatted as lowercase and hyphen-separated"""
validated_groups = []
for value in groups:
slug = slugify(value)
if value != slug:
raise ValueError(
f"Groups must be slugs (lowercase, hyphen-separated): {slug:s}"
)
validated_groups.append(value)
return validated_groups
def cleanlist(value):
"""Build a list of strings from a string, None (empty list) or a list of objects."""
if isinstance(value, str):
# Convert comma-separated strings to list
return [s.strip() for s in value.split(",") if s.strip()]
if isinstance(value, list):
# Clean up list of strings
return [str(s).strip() for s in value if s is not None and str(s).strip()]
if value is None:
return []
raise ValueError()
StringListParameter = Annotated[List[str], BeforeValidator(cleanlist)]
class SearchQueryParametersSchema(BaseModel):
"""Schema for validating the querystring on the search API endpoint"""
q: str
services: StringListParameter = Field(default_factory=list)
visited: StringListParameter = Field(default_factory=list)
reach: Optional[enums.ReachEnum] = None
order_by: Optional[Literal[enums.ORDER_BY_OPTIONS]] = Field(default=enums.RELEVANCE)
order_direction: Optional[Literal["asc", "desc"]] = Field(default="desc")
page_number: Optional[conint(ge=1)] = Field(default=1)
page_size: Optional[conint(ge=1, le=100)] = Field(default=50)
View File
+22
View File
@@ -0,0 +1,22 @@
"""Fixtures for tests in the find core application"""
import pytest
from lasuite.oidc_resource_server.authentication import (
get_resource_server_backend,
)
@pytest.fixture(name="jwt_rs_backend")
def jwt_resource_server_backend_fixture(settings):
"""Fixture to switch the backend to the JWTResourceServerBackend."""
_original_backend = str(settings.OIDC_RS_BACKEND_CLASS)
settings.OIDC_RS_BACKEND_CLASS = (
"lasuite.oidc_resource_server.backend.JWTResourceServerBackend"
)
get_resource_server_backend.cache_clear()
yield
settings.OIDC_RS_BACKEND_CLASS = _original_backend
get_resource_server_backend.cache_clear()
@@ -0,0 +1,432 @@
"""Tests indexing documents in OpenSearch over the API"""
import datetime
from unittest import mock
from django.utils import timezone
import pytest
from rest_framework.test import APIClient
from core import factories, opensearch
pytestmark = pytest.mark.django_db
def test_api_documents_index_bulk_anonymous():
"""Anonymous requests should not be allowed to index documents in bulk."""
documents = factories.DocumentSchemaFactory.build_batch(3)
response = APIClient().post("/api/v1.0/documents/index/", documents, format="json")
assert response.status_code == 403
assert response.json() == {
"detail": "Authentication credentials were not provided."
}
def test_api_documents_index_bulk_invalid_token():
"""Requests with invalid tokens should not be allowed to index documents in bulk."""
documents = factories.DocumentSchemaFactory.build_batch(3)
response = APIClient().post(
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION="Bearer invalid",
format="json",
)
assert response.status_code == 403
assert response.json() == {"detail": "Invalid token."}
def test_api_documents_index_bulk_success():
"""A registered service should be able to index documents in bulk with a valid token."""
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(3)
response = APIClient().post(
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 207
responses = response.json()
assert [d["status"] for d in responses] == ["success"] * 3
def test_api_documents_index_bulk_ensure_index():
"""A registered service should be create the opensearch index if need."""
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(3)
# Delete the index
opensearch.client.indices.delete(index="*test*")
with pytest.raises(opensearch.NotFoundError):
opensearch.client.indices.get(index="test-service")
response = APIClient().post(
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 207
responses = response.json()
assert len(responses) == 3
assert [d["status"] for d in responses] == ["success"] * 3
# The index has been rebuilt
opensearch.client.indices.get(index="test-service")
@pytest.mark.parametrize(
"field, invalid_value, error_type, error_message",
[
(
"id",
"0f9b1c9d-030f-427a-8a0e-6b7c202c5daz", # invalid UUID b/c contains a z
"uuid_parsing",
(
"Input should be a valid UUID, invalid character: expected an optional "
"prefix of `urn:uuid:` followed by [0-9a-fA-F-], found `z` at 36"
),
),
("title", 1, "string_type", "Input should be a valid string"),
(
"title",
"a" * 301,
"string_too_long",
"String should have at most 300 characters",
),
(
"depth",
-1,
"greater_than_equal",
"Input should be greater than or equal to 0",
),
(
"depth",
"a",
"int_parsing",
"Input should be a valid integer, unable to parse string as an integer",
),
(
"path",
"a" * 301,
"string_too_long",
"String should have at most 300 characters",
),
(
"numchild",
-1,
"greater_than_equal",
"Input should be greater than or equal to 0",
),
(
"numchild",
"a",
"int_parsing",
"Input should be a valid integer, unable to parse string as an integer",
),
("content", 1, "string_type", "Input should be a valid string"),
(
"created_at",
"invalid_date",
"datetime_from_date_parsing",
"Input should be a valid datetime or date, invalid character in year",
),
(
"updated_at",
"invalid_date",
"datetime_from_date_parsing",
"Input should be a valid datetime or date, invalid character in year",
),
(
"size",
64448894017.3,
"int_from_float",
"Input should be a valid integer, got a number with a fractional part",
),
(
"size",
"not an integer",
"int_parsing",
"Input should be a valid integer, unable to parse string as an integer",
),
(
"users",
"33052c8b-3181-4420-aede-f8396fc0f9a1",
"list_type",
"Input should be a valid list",
),
(
"users",
["a" * 51],
"string_too_long",
"String should have at most 50 characters",
),
(
"groups",
["not_a_slug"],
"string_pattern_mismatch",
"String should match pattern '^[a-z0-9]+(?:-[a-z0-9]+)*$'",
),
(
"groups",
"not-a-list",
"list_type",
"Input should be a valid list",
),
(
"reach",
"invalid",
"enum",
"Input should be 'public', 'authenticated' or 'restricted'",
),
(
"is_active",
"invalid",
"bool_parsing",
"Input should be a valid boolean, unable to interpret input",
),
],
)
def test_api_documents_index_bulk_invalid_document(
field, invalid_value, error_type, error_message
):
"""Test bulk document indexing with various invalid fields."""
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(3)
# Modify the first document with the invalid value for the specified field
documents[0][field] = invalid_value
response = APIClient().post(
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 400
responses = response.json()
assert [d["status"] for d in responses] == ["error", "valid", "valid"]
assert responses[0]["status"] == "error"
assert len(responses[0]["errors"]) == 1
assert responses[0]["errors"][0]["msg"] == error_message
assert responses[0]["errors"][0]["type"] == error_type
@pytest.mark.parametrize(
"field",
[
"id",
"title",
"depth",
"path",
"numchild",
"content",
"size",
"created_at",
"updated_at",
"is_active",
],
)
def test_api_documents_index_bulk_required(field):
"""Test bulk document indexing with a required field missing."""
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(3)
del documents[0][field]
response = APIClient().post(
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 400
responses = response.json()
assert [d["status"] for d in responses] == ["error", "valid", "valid"]
assert responses[0]["status"] == "error"
assert len(responses[0]["errors"]) == 1
assert responses[0]["errors"][0]["msg"] == "Field required"
assert responses[0]["errors"][0]["type"] == "missing"
@pytest.mark.parametrize(
"field,default_value",
[
("users", []),
("groups", []),
("reach", "restricted"),
],
)
def test_api_documents_index_bulk_default(field, default_value):
"""Test bulk document indexing while removing optional fields that have default values."""
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(3)
del documents[0][field]
response = APIClient().post(
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 207
responses = response.json()
assert [d["status"] for d in responses] == ["success"] * 3
indexed_document = opensearch.client.get(
index=service.name, id=responses[0]["_id"]
)["_source"]
assert indexed_document[field] == default_value
def test_api_documents_index_bulk_updated_at_before_created_at():
"""Test bulk document indexing with updated_at before created_at."""
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(3)
documents[0]["updated_at"] = documents[0]["created_at"] - datetime.timedelta(
seconds=1
)
response = APIClient().post(
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 400
responses = response.json()
assert [d["status"] for d in responses] == ["error", "valid", "valid"]
assert responses[0]["status"] == "error"
assert len(responses[0]["errors"]) == 1
assert (
responses[0]["errors"][0]["msg"]
== "Value error, updated_at must be later than created_at"
)
assert responses[0]["errors"][0]["type"] == "value_error"
@pytest.mark.parametrize(
"field",
["created_at", "updated_at"],
)
def test_api_documents_index_bulk_datetime_future(field):
"""Test bulk document indexing with datetimes in the future."""
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(3)
now = timezone.now()
documents[0][field] = now + datetime.timedelta(seconds=3)
response = APIClient().post(
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 400
responses = response.json()
assert [d["status"] for d in responses] == ["error", "valid", "valid"]
assert responses[0]["status"] == "error"
assert len(responses[0]["errors"]) == 1
assert (
responses[0]["errors"][0]["msg"]
== f"Value error, {field:s} must be earlier than now"
)
assert responses[0]["errors"][0]["type"] == "value_error"
def test_api_documents_index_empty_content_check():
"""Test bulk document indexing with both empty title & content."""
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(3)
documents[0]["content"] = ""
documents[0]["title"] = ""
response = APIClient().post(
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 400
responses = response.json()
assert [d["status"] for d in responses] == ["error", "valid", "valid"]
assert responses[0]["status"] == "error"
assert len(responses[0]["errors"]) == 1
assert (
responses[0]["errors"][0]["msg"]
== "Value error, Either title or content should have at least 1 character"
)
assert responses[0]["errors"][0]["type"] == "value_error"
def test_api_documents_index_opensearch_errors():
"""Test bulk document indexing errors"""
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(3)
with mock.patch.object(opensearch.client, "bulk") as mock_bulk:
mock_bulk.return_value = {
"items": [
{"index": {"status": 201}},
{
"index": {
"status": 400,
}
},
{"index": {"status": 403, "error": {"reason": "This is forbidden"}}},
]
}
response = APIClient().post(
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 207
responses = response.json()
assert responses == [
{
"_id": documents[0]["id"],
"index": 0,
"status": "success",
},
{
"_id": documents[1]["id"],
"index": 1,
"status": "error",
"message": "Unknown error",
},
{
"_id": documents[2]["id"],
"index": 2,
"status": "error",
"message": "This is forbidden",
},
]
@@ -0,0 +1,375 @@
"""Tests indexing documents in OpenSearch over the API"""
import datetime
from django.utils import timezone
import pytest
from rest_framework.test import APIClient
from core import factories, opensearch
pytestmark = pytest.mark.django_db
def test_api_documents_index_single_anonymous():
"""Anonymous requests should not be allowed to index documents."""
document = factories.DocumentSchemaFactory.build()
response = APIClient().post("/api/v1.0/documents/index/", document, format="json")
assert response.status_code == 403
assert response.json() == {
"detail": "Authentication credentials were not provided."
}
def test_api_documents_index_single_invalid_token():
"""Requests with invalid tokens should not be allowed to index documents."""
document = factories.DocumentSchemaFactory.build()
response = APIClient().post(
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION="Bearer invalid",
format="json",
)
assert response.status_code == 403
assert response.json() == {"detail": "Invalid token."}
def test_api_documents_index_single_success():
"""A registered service should be able to index document with a valid token."""
service = factories.ServiceFactory(name="test-service")
document = factories.DocumentSchemaFactory.build()
response = APIClient().post(
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 201
assert response.json()["_id"] == str(document["id"])
def test_api_documents_index_bulk_ensure_index():
"""A registered service should be create the opensearch index if need."""
service = factories.ServiceFactory(name="test-service")
document = factories.DocumentSchemaFactory.build()
# Delete the index
opensearch.client.indices.delete(index="*test*")
with pytest.raises(opensearch.NotFoundError):
opensearch.client.indices.get(index="test-service")
response = APIClient().post(
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 201
assert response.json()["_id"] == str(document["id"])
# The index has been rebuilt
data = opensearch.client.indices.get(index="test-service")
assert data["test-service"]["mappings"] == {
"dynamic": "strict",
"properties": {
"id": {"type": "keyword"},
"title": {
"type": "keyword", # Primary field for exact matches and sorting
"fields": {
"text": {"type": "text"} # Sub-field for full-text search
},
},
"depth": {"type": "integer"},
"path": {
"type": "keyword",
"fields": {"text": {"type": "text"}},
},
"numchild": {"type": "integer"},
"content": {"type": "text"},
"created_at": {"type": "date"},
"updated_at": {"type": "date"},
"size": {"type": "long"},
"users": {"type": "keyword"},
"groups": {"type": "keyword"},
"reach": {"type": "keyword"},
"is_active": {"type": "boolean"},
},
}
@pytest.mark.parametrize(
"field, invalid_value, error_type, error_message",
[
(
"id",
"0f9b1c9d-030f-427a-8a0e-6b7c202c5daz", # invalid UUID b/c contains a z*
"uuid_parsing",
(
"Input should be a valid UUID, invalid character: expected an optional "
"prefix of `urn:uuid:` followed by [0-9a-fA-F-], found `z` at 36"
),
),
("title", 1, "string_type", "Input should be a valid string"),
(
"title",
"a" * 301,
"string_too_long",
"String should have at most 300 characters",
),
(
"depth",
-1,
"greater_than_equal",
"Input should be greater than or equal to 0",
),
(
"depth",
"a",
"int_parsing",
"Input should be a valid integer, unable to parse string as an integer",
),
(
"path",
"a" * 301,
"string_too_long",
"String should have at most 300 characters",
),
(
"numchild",
-1,
"greater_than_equal",
"Input should be greater than or equal to 0",
),
(
"numchild",
"a",
"int_parsing",
"Input should be a valid integer, unable to parse string as an integer",
),
("content", 1, "string_type", "Input should be a valid string"),
(
"created_at",
"invalid_date",
"datetime_from_date_parsing",
"Input should be a valid datetime or date, invalid character in year",
),
(
"updated_at",
"invalid_date",
"datetime_from_date_parsing",
"Input should be a valid datetime or date, invalid character in year",
),
(
"size",
64448894017.3,
"int_from_float",
"Input should be a valid integer, got a number with a fractional part",
),
(
"size",
"not an integer",
"int_parsing",
"Input should be a valid integer, unable to parse string as an integer",
),
(
"users",
"33052c8b-3181-4420-aede-f8396fc0f9a1",
"list_type",
("Input should be a valid list"),
),
(
"users",
["a" * 51],
"string_too_long",
"String should have at most 50 characters",
),
(
"groups",
["not_a_slug"],
"string_pattern_mismatch",
"String should match pattern '^[a-z0-9]+(?:-[a-z0-9]+)*$'",
),
(
"groups",
"not-a-list",
"list_type",
("Input should be a valid list"),
),
(
"reach",
"invalid",
"enum",
"Input should be 'public', 'authenticated' or 'restricted'",
),
(
"is_active",
"invalid",
"bool_parsing",
"Input should be a valid boolean, unable to interpret input",
),
],
)
def test_api_documents_index_single_invalid_document(
field, invalid_value, error_type, error_message
):
"""Test document indexing with various invalid fields."""
service = factories.ServiceFactory(name="test-service")
document = factories.DocumentSchemaFactory.build()
# Modify the document with the invalid value for the specified field
document[field] = invalid_value
response = APIClient().post(
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 400
assert response.data[0]["msg"] == error_message
assert response.data[0]["type"] == error_type
@pytest.mark.parametrize(
"field",
[
"id",
"title",
"depth",
"path",
"numchild",
"content",
"size",
"created_at",
"updated_at",
"is_active",
],
)
def test_api_documents_index_single_required(field):
"""Test document indexing with a required field missing."""
service = factories.ServiceFactory(name="test-service")
document = factories.DocumentSchemaFactory.build()
del document[field]
response = APIClient().post(
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 400
assert response.data[0]["msg"] == "Field required"
assert response.data[0]["type"] == "missing"
@pytest.mark.parametrize(
"field,default_value",
[
("users", []),
("groups", []),
("reach", "restricted"),
],
)
def test_api_documents_index_single_default(field, default_value):
"""Test document indexing while removing optional fields that have default values."""
service = factories.ServiceFactory(name="test-service")
document = factories.DocumentSchemaFactory.build()
del document[field]
response = APIClient().post(
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 201
assert response.json()["_id"] == str(document["id"])
indexed_document = opensearch.client.get(
index=service.name, id=str(document["id"])
)["_source"]
assert indexed_document[field] == default_value
def test_api_documents_index_single_udpated_at_before_created():
"""Test document indexing with updated_at before created_at."""
service = factories.ServiceFactory(name="test-service")
document = factories.DocumentSchemaFactory.build()
document["updated_at"] = document["created_at"] - datetime.timedelta(seconds=1)
response = APIClient().post(
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 400
assert (
response.data[0]["msg"]
== "Value error, updated_at must be later than created_at"
)
assert response.data[0]["type"] == "value_error"
@pytest.mark.parametrize(
"field",
["created_at", "updated_at"],
)
def test_api_documents_index_single_datetime_future(field):
"""Test document indexing with datetimes in the future."""
service = factories.ServiceFactory(name="test-service")
document = factories.DocumentSchemaFactory.build()
now = timezone.now()
document[field] = now + datetime.timedelta(seconds=3)
response = APIClient().post(
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 400
assert response.data[0]["msg"] == f"Value error, {field:s} must be earlier than now"
assert response.data[0]["type"] == "value_error"
def test_api_documents_index_empty_content_check():
"""Test document indexing with both empty title & content."""
service = factories.ServiceFactory(name="test-service")
document = factories.DocumentSchemaFactory.build()
document["content"] = ""
document["title"] = ""
response = APIClient().post(
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 400
assert (
response.data[0]["msg"]
== "Value error, Either title or content should have at least 1 character"
)
assert response.data[0]["type"] == "value_error"
@@ -0,0 +1,701 @@
"""
Test suite for searching documents in OpenSearch over the API.
Don't use pytest parametrized tests because batch generation and indexing
of documents is slow and better done only once.
"""
import operator
import random
import pytest
import responses
from rest_framework.test import APIClient
from core import enums, factories
from .utils import build_authorization_bearer, prepare_index, setup_oicd_resource_server
pytestmark = pytest.mark.django_db
@responses.activate
def test_api_documents_search_auth_invalid_parameters(settings):
"""Invalid service parameters should result in a 401 error"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
settings.OIDC_RS_CLIENT_ID = None
settings.OIDC_RS_CLIENT_SECRET = None
service = factories.ServiceFactory(name="test-service")
prepare_index(service.name, [])
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "a quick fox"},
format="json",
HTTP_AUTHORIZATION="Bearer unknown",
)
assert response.status_code == 401
assert response.json() == {"detail": "Resource Server is improperly configured"}
@responses.activate
def test_api_documents_search_query_unknown_user(settings):
"""Searching a document without an existing user should result in a 401 error"""
setup_oicd_resource_server(
responses,
settings,
sub="unknown",
introspect=lambda request, user_info: (404, {}, ""),
)
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
prepare_index(service.name, [])
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "a quick fox"},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 401
assert response.json() == {"detail": "Login failed"}
@responses.activate
def test_api_documents_search_services_invalid_parameters(settings):
"""Invalid pagination parameters should result in a 400 error"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
factories.ServiceFactory(name="test-service")
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "a quick fox", "services": {}},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
assert response.json() == [
{
"loc": ["services"],
"msg": "Value error, ",
"type": "value_error",
}
]
@responses.activate
def test_api_documents_search_reached_docs_invalid_parameters(settings):
"""Invalid pagination parameters should result in a 400 error"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
factories.ServiceFactory(name="test-service")
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "a quick fox", "visited": {}},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
assert response.json() == [
{
"loc": ["visited"],
"msg": "Value error, ",
"type": "value_error",
}
]
@responses.activate
def test_api_documents_search_query_title(settings):
"""Searching a document by its title should work as expected"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
document = factories.DocumentSchemaFactory.build(
title="The quick brown fox",
content="the wolf",
reach=random.choice(["public", "authenticated"]),
)
# Add other documents
other_fox_document = factories.DocumentSchemaFactory.build(
title="The blue fox",
content="the wolf",
reach=random.choice(["public", "authenticated"]),
)
no_fox_document = factories.DocumentSchemaFactory.build(
title="The brown goat",
content="the wolf",
reach=random.choice(["public", "authenticated"]),
)
documents = [document, other_fox_document, no_fox_document]
prepare_index(service.name, documents)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "a quick fox", "visited": [doc["id"] for doc in documents]},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert len(response.json()) == 2
fox_data = response.json()[0]
assert list(fox_data.keys()) == ["_index", "_id", "_score", "_source", "fields"]
assert fox_data["_id"] == str(document["id"])
assert fox_data["_source"] == {
"depth": 1,
"numchild": 0,
"path": document["path"],
"size": document["size"],
"created_at": document["created_at"].isoformat(),
"updated_at": document["updated_at"].isoformat(),
"reach": document["reach"],
"title": "The quick brown fox",
}
assert fox_data["fields"] == {"number_of_users": [3], "number_of_groups": [3]}
other_fox_data = response.json()[1]
assert list(other_fox_data.keys()) == [
"_index",
"_id",
"_score",
"_source",
"fields",
]
assert other_fox_data["_id"] == str(other_fox_document["id"])
assert other_fox_data["_source"] == {
"depth": 1,
"numchild": 0,
"path": other_fox_document["path"],
"size": other_fox_document["size"],
"created_at": other_fox_document["created_at"].isoformat(),
"updated_at": other_fox_document["updated_at"].isoformat(),
"reach": other_fox_document["reach"],
"title": "The blue fox",
}
assert other_fox_data["fields"] == {"number_of_users": [3], "number_of_groups": [3]}
@responses.activate
def test_api_documents_search_query_content(settings):
"""Searching a document by its content should work as expected"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
document = factories.DocumentSchemaFactory.build(
title="the wolf",
content="The quick brown fox",
reach=random.choice(["public", "authenticated"]),
)
# Add other documents
other_fox_document = factories.DocumentSchemaFactory.build(
title="the wolf",
content="The blue fox",
reach=random.choice(["public", "authenticated"]),
)
no_fox_document = factories.DocumentSchemaFactory.build(
title="the wolf",
content="The brown goat",
reach=random.choice(["public", "authenticated"]),
)
documents = [document, other_fox_document, no_fox_document]
prepare_index(service.name, documents)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "a quick fox", "visited": [doc["id"] for doc in documents]},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert len(response.json()) == 2
fox_data = response.json()[0]
assert list(fox_data.keys()) == ["_index", "_id", "_score", "_source", "fields"]
assert fox_data["_id"] == str(document["id"])
assert fox_data["_source"] == {
"depth": 1,
"numchild": 0,
"path": document["path"],
"size": document["size"],
"created_at": document["created_at"].isoformat(),
"updated_at": document["updated_at"].isoformat(),
"reach": document["reach"],
"title": document["title"],
}
assert fox_data["fields"] == {"number_of_users": [3], "number_of_groups": [3]}
other_fox_data = response.json()[1]
assert list(other_fox_data.keys()) == [
"_index",
"_id",
"_score",
"_source",
"fields",
]
assert other_fox_data["_id"] == str(other_fox_document["id"])
assert other_fox_data["_source"] == {
"depth": 1,
"numchild": 0,
"path": other_fox_document["path"],
"size": other_fox_document["size"],
"created_at": other_fox_document["created_at"].isoformat(),
"updated_at": other_fox_document["updated_at"].isoformat(),
"reach": other_fox_document["reach"],
"title": other_fox_document["title"],
}
assert other_fox_data["fields"] == {"number_of_users": [3], "number_of_groups": [3]}
@responses.activate
def test_api_documents_search_ordering_by_fields(settings):
"""It should be possible to order by several fields"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(
4, reach=random.choice(["public", "authenticated"])
)
prepare_index(service.name, documents)
parameters = [
(enums.TITLE, "asc"),
(enums.TITLE, "desc"),
(enums.CREATED_AT, "asc"),
(enums.CREATED_AT, "desc"),
(enums.UPDATED_AT, "asc"),
(enums.UPDATED_AT, "desc"),
(enums.SIZE, "asc"),
(enums.SIZE, "desc"),
(enums.REACH, "asc"),
(enums.REACH, "desc"),
]
for field, direction in parameters:
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"order_by": field,
"order_direction": direction,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
data = response.json()
assert len(data) == 4
# Check that results are sorted by the field as expected
compare = operator.le if direction == "asc" else operator.ge
for i in range(len(data) - 1):
assert compare(data[i]["_source"][field], data[i + 1]["_source"][field])
@responses.activate
def test_api_documents_search_ordering_by_relevance(settings):
"""It should be possible to order by relevance (score)"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(
4, reach=random.choice(["public", "authenticated"])
)
prepare_index(service.name, documents)
for direction in ["asc", "desc"]:
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"order_by": "relevance",
"order_direction": direction,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
data = response.json()
assert len(data) == 4
# Check that results are sorted by score as expected
compare = operator.le if direction == "asc" else operator.ge
for i in range(len(data) - 1):
assert compare(data[i]["_score"], data[i + 1]["_score"])
@responses.activate
def test_api_documents_search_ordering_by_unknown_field(settings):
"""Trying to sort by an unknown field should return a 400 error"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
# Setup: Initialize the service and documents only once
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(
2, reach=random.choice(["public", "authenticated"])
)
prepare_index(service.name, documents)
# Define the parameters manually
directions = ["asc", "desc"]
# Perform the parameterized tests
for direction in directions:
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"order_by": "unknown",
"order_direction": direction,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
assert response.json() == [
{
"loc": ["order_by"],
"msg": (
"Input should be 'relevance', 'title', 'created_at', "
"'updated_at', 'size' or 'reach'"
),
"type": "literal_error",
}
]
@responses.activate
def test_api_documents_search_ordering_by_unknown_direction(settings):
"""Trying to sort with an unknown direction should return a 400 error"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(
2, reach=random.choice(["public", "authenticated"])
)
prepare_index(service.name, documents)
for field in enums.ORDER_BY_OPTIONS:
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"order_by": field,
"order_direction": "unknown",
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
assert response.json() == [
{
"loc": ["order_direction"],
"msg": "Input should be 'asc' or 'desc'",
"type": "literal_error",
}
]
@responses.activate
def test_api_documents_search_filtering_by_reach(settings):
"""It should be possible to filter results by their reach"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(
4, reach=random.choice(["public", "authenticated"])
)
prepare_index(service.name, documents)
for reach in enums.ReachEnum:
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"reach": reach.value,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
data = response.json()
for result in data:
assert reach == result["_source"]["reach"]
# Pagination
@responses.activate
def test_api_documents_search_pagination_basic(settings):
"""Pagination should correctly return documents for the specified page and page size"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(
9, reach=random.choice(["public", "authenticated"])
)
ids = [str(doc["id"]) for doc in documents]
prepare_index(service.name, documents)
# Request the first page with a page size of 3
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"page_number": 1,
"page_size": 3,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
data = response.json()
assert len(data) == 3 # Page size is 3
assert [r["_id"] for r in data] == ids[0:3]
# Request the second page with a page size of 3
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"page_number": 2,
"page_size": 3,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
data = response.json()
assert len(data) == 3
assert [r["_id"] for r in data] == ids[3:6]
# Request the third page with a page size of 5 (should contain the remaining 3 documents)
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"page_number": 3,
"page_size": 3,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
data = response.json()
assert len(data) == 3
assert [r["_id"] for r in data] == ids[6:9]
@responses.activate
def test_api_documents_search_pagination_last_page_edge_case(settings):
"""Requesting the last page should return the correct number of remaining documents"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(
8, reach=random.choice(["public", "authenticated"])
)
ids = [str(doc["id"]) for doc in documents]
prepare_index(service.name, documents)
# Request the first page with a page size of 3
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"page_number": 1,
"page_size": 3,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert len(response.json()) == 3
assert [r["_id"] for r in response.json()] == ids[0:3]
# Request the third page with a page size of 3 (should contain the last 1 document)
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"page_number": 3,
"page_size": 3,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert len(response.json()) == 2 # Only 2 documents should be on the last page
assert [r["_id"] for r in response.json()] == ids[6:]
@responses.activate
def test_api_documents_search_pagination_out_of_bounds(settings):
"""
Requesting a page number that exceeds the total number of pages should return an empty list
"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(
4, reach=random.choice(["public", "authenticated"])
)
prepare_index(service.name, documents)
# Request the fourth page with a page size of 2 (there are only 2 pages)
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"page_number": 4,
"page_size": 2,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert len(response.json()) == 0 # No documents should be returned
@responses.activate
def test_api_documents_search_pagination_invalid_parameters(settings):
"""Invalid pagination parameters should result in a 400 error"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(
4, reach=random.choice(["public", "authenticated"])
)
prepare_index(service.name, documents)
parameters = [
(
"invalid",
10,
"int_parsing",
"Input should be a valid integer, unable to parse string as an integer",
),
(
1,
"invalid",
"int_parsing",
"Input should be a valid integer, unable to parse string as an integer",
),
(-1, 10, "greater_than_equal", "Input should be greater than or equal to 1"),
(1, -10, "greater_than_equal", "Input should be greater than or equal to 1"),
(0, 10, "greater_than_equal", "Input should be greater than or equal to 1"),
(1, 0, "greater_than_equal", "Input should be greater than or equal to 1"),
]
for page_number, page_size, error_type, error_message in parameters:
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "page_number": page_number, "page_size": page_size},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
assert response.data[0]["msg"] == error_message
assert response.data[0]["type"] == error_type
@responses.activate
def test_api_documents_search_pagination_with_filtering(settings):
"""Pagination should work correctly when combined with filtering by reach"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
public_documents = factories.DocumentSchemaFactory.build_batch(3, reach="public")
public_ids = [str(doc["id"]) for doc in public_documents]
private_documents = factories.DocumentSchemaFactory.build_batch(
2, reach="authenticated"
)
prepare_index(service.name, public_documents + private_documents)
# Filter by public documents, request first page
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"reach": "public",
"page_number": 1,
"page_size": 2,
"visited": public_ids,
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert len(response.json()) == 2
assert [r["_id"] for r in response.json()] == public_ids[0:2]
# Request second page for public documents (remaining 1 document)
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"reach": "public",
"page_number": 2,
"page_size": 2,
"visited": public_ids,
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert len(response.json()) == 1
assert [r["_id"] for r in response.json()] == public_ids[2:]
@@ -0,0 +1,438 @@
"""
Test suite for access control when searching documents over the API.
Don't use pytest parametrized tests because batch generation and indexing
of documents is slow and better done only once.
"""
import pytest
import responses
from rest_framework.test import APIClient
from core import enums, factories
from .utils import (
build_authorization_bearer,
delete_test_indices,
prepare_index,
setup_oicd_resource_server,
)
pytestmark = pytest.mark.django_db
def test_api_documents_search_access_control_anonymous():
"""Anonymous users should not be allowed to search documents even public."""
service = factories.ServiceFactory(name="test-service")
documents = []
for reach in enums.ReachEnum:
documents.extend(factories.DocumentSchemaFactory.build_batch(3, reach=reach))
prepare_index(service.name, documents)
response = APIClient().post("/api/v1.0/documents/search/?q=*")
assert response.status_code == 401
@responses.activate
def test_api_documents_search_access_control(settings):
"""
Authenticated users should only see documents:
- for which they are listed in the "users" field
- that have a reach set to "authenticated" or "public"
- only configured services providers are allowed (e.g docs)
(groups is not yet implemnted)
"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents_reach = factories.DocumentSchemaFactory.build_batch(6)
documents_open = [
doc for doc in documents_reach if doc["reach"] in ["authenticated", "public"]
]
documents_user = factories.DocumentSchemaFactory.build_batch(
6, users=["user_sub", "user_sub2"]
)
expected_ids = [doc["id"] for doc in documents_open + documents_user]
prepare_index(service.name, documents_user + documents_reach)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "visited": [d["id"] for d in documents_open]},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert sorted([d["_id"] for d in response.json()]) == sorted(expected_ids)
@responses.activate
@pytest.mark.parametrize(
"doc_ids,visited,expected",
[
(["a", "b"], [], []),
(["a", "b"], "", []),
(["a", "b"], None, []),
(["a", "b"], ["other"], []),
([], ["a"], []),
(["a", "b"], ["a"], ["a"]),
(["a", "b"], ["a", "b", "c"], ["a", "b"]),
(["a", "b"], "a,b,c", ["a", "b"]),
],
)
def test_api_documents_search_access__only_visited_public(
doc_ids, visited, expected, settings
):
"""
Authenticated users should only see documents with reach="public"
that are in "visited" list.
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="docs")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service", client_id="docs")
docs = [
factories.DocumentSchemaFactory(
reach=[enums.ReachEnum.PUBLIC, enums.ReachEnum.AUTHENTICATED], id=doc_id
)
for doc_id in doc_ids
]
prepare_index(service.name, docs)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "visited": visited},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200, response.json()
assert sorted([d["_id"] for d in response.json()]) == sorted(expected)
@responses.activate
def test_api_documents_search_access__any_owner_public(settings):
"""
Authenticated users should only see documents with reach="public"
that are in "visited" list.
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="docs")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service", client_id="docs")
docs = factories.DocumentSchemaFactory.build_batch(
6,
reach=enums.ReachEnum.PUBLIC,
users=["user_sub"],
)
other_docs = factories.DocumentSchemaFactory.build_batch(
6,
reach=enums.ReachEnum.PUBLIC,
users=["other_sub"],
)
prepare_index(service.name, docs + other_docs)
expected = [d["id"] for d in docs]
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "visited": []},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200, response.json()
assert sorted([d["_id"] for d in response.json()]) == sorted(expected)
@responses.activate
def test_api_documents_search_access__services(settings):
"""
Authenticated users should only see documents of audience
service providers (e.g docs)
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="a-client")
token = build_authorization_bearer()
service_a = factories.ServiceFactory(name="test-index-a", client_id="a-client")
service_b = factories.ServiceFactory(name="test-index-b", client_id="b-client")
service_a_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
service_b_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
expected_ids = [doc["id"] for doc in service_a_docs]
prepare_index(service_a.name, service_a_docs)
prepare_index(service_b.name, service_b_docs, cleanup=False)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*"},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert sorted([d["_id"] for d in response.json()]) == sorted(expected_ids)
@responses.activate
def test_api_documents_search_access__missing_index(settings):
"""
When the service has no opensearch index, returns an empty list.
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="a-client")
token = build_authorization_bearer()
factories.ServiceFactory(name="test-index-a", client_id="a-client")
delete_test_indices()
# a-client has no index. ignore it.
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*"},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert response.json() == []
@responses.activate
def test_api_documents_search_access__related_services(settings):
"""
Authenticated users should only see documents of audience
service providers and its related services (e.g drive + docs)
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="c-client")
token = build_authorization_bearer()
service_a = factories.ServiceFactory(name="test-index-a", client_id="a-client")
service_b = factories.ServiceFactory(name="test-index-b", client_id="b-client")
service_c = factories.ServiceFactory(name="test-index-c", client_id="c-client")
service_c.services.set([service_a])
service_c.save()
service_a_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
service_b_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
service_c_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
expected_ids = [doc["id"] for doc in service_a_docs + service_c_docs]
prepare_index(service_a.name, service_a_docs)
prepare_index(service_b.name, service_b_docs, cleanup=False)
prepare_index(service_c.name, service_c_docs, cleanup=False)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*"},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert sorted([d["_id"] for d in response.json()]) == sorted(expected_ids)
@responses.activate
def test_api_documents_search_access__related_missing_index(settings):
"""
When the service has no opensearch index, returns the related services data.
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="a-client")
token = build_authorization_bearer()
service_a = factories.ServiceFactory(name="test-index-a", client_id="a-client")
service_b = factories.ServiceFactory(name="test-index-b", client_id="b-client")
service_c = factories.ServiceFactory(name="test-index-c", client_id="c-client")
service_c.services.set([service_a])
service_c.save()
service_b_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
service_c_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
expected_ids = [doc["id"] for doc in service_c_docs]
prepare_index(service_b.name, service_b_docs)
prepare_index(service_c.name, service_c_docs, cleanup=False)
# a-client has no index. ignore it.
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*"},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert sorted([d["_id"] for d in response.json()]) == sorted(expected_ids)
@responses.activate
def test_api_documents_search_access__request_services(settings):
"""
Authenticated users should only see documents of audience
from requested services : 'services' parameter.
Raise 400 error if not all requested services are authorized.
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="c-client")
token = build_authorization_bearer()
service_a = factories.ServiceFactory(name="test-index-a", client_id="a-client")
service_b = factories.ServiceFactory(name="test-index-b", client_id="b-client")
service_c = factories.ServiceFactory(name="test-index-c", client_id="c-client")
service_a_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
service_b_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
service_c_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
expected_ids = [doc["id"] for doc in service_c_docs]
prepare_index(service_a.name, service_a_docs)
prepare_index(service_b.name, service_b_docs, cleanup=False)
prepare_index(service_c.name, service_c_docs, cleanup=False)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "services": ["test-index-c"]},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert sorted([d["_id"] for d in response.json()]) == sorted(expected_ids)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "services": ["test-index-c", "test-index-b"]},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
assert response.json() == {"detail": "Some requested services are not available"}
@responses.activate
def test_api_documents_search_access__request_inactive_services(settings):
"""
Authenticated users should only see documents of audience
from requested services : 'services' parameter.
Raise 400 error if not all requested services are active.
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="client")
token = build_authorization_bearer()
factories.ServiceFactory(name="test-index", client_id="client", is_active=False)
factories.ServiceFactory(name="test-index-b", client_id="b-client")
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "services": ["test-index"]},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
assert response.json() == {"detail": "Service is not available"}
# Event without explicit argument, the client service from the request is not active
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*"},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
assert response.json() == {"detail": "Service is not available"}
@responses.activate
def test_api_documents_search_access__authenticated(settings):
"""
Authenticated users should only see documents
- for which they are listed in the "users" field
- that have a reach set to "authenticated" or "public" AND in visited list
- only configured services providers are allowed (e.g docs)
(groups is not yet implemnted)
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="docs")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service", client_id="docs")
documents_open = factories.DocumentSchemaFactory.build_batch(
2, reach=enums.ReachEnum.PUBLIC
) + factories.DocumentSchemaFactory.build_batch(
2, reach=enums.ReachEnum.AUTHENTICATED
)
documents_restricted = factories.DocumentSchemaFactory.build_batch(
2, reach=enums.ReachEnum.RESTRICTED
)
documents_user = factories.DocumentSchemaFactory.build_batch(
6, users=["user_sub", "user_sub2"]
)
documents = documents_user + documents_open + documents_restricted
prepare_index(service.name, documents_user + documents_open + documents_restricted)
# Only owned documents (reach is ignored)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*"},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert sorted([d["_id"] for d in response.json()]) == sorted(
[doc["id"] for doc in documents_user]
)
# Owned documents and visited public/authenticated ones.
# Restricted ones from another owner are filtered (even if given as visited ones)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "visited": [d["id"] for d in documents]},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert sorted([d["_id"] for d in response.json()]) == sorted(
[doc["id"] for doc in documents_user + documents_open]
)
@@ -0,0 +1,41 @@
"""Tests Service model for find's core app."""
from django.db import DataError, IntegrityError
import pytest
from core import factories
pytestmark = pytest.mark.django_db
def test_models_services_name_unique():
"""The name field should be unique across services."""
service = factories.ServiceFactory()
with pytest.raises(IntegrityError):
factories.ServiceFactory(name=service.name)
def test_models_services_name_slugified():
"""The name field should be slugified."""
service = factories.ServiceFactory(name="My service name")
assert service.name == "my-service-name"
def test_models_services_token_50_characters_exact():
"""The token field should be 50 characters long."""
service = factories.ServiceFactory()
assert len(service.token) == 50
def test_models_services_token_50_characters_less():
"""The token field should not be less than 50 characters long."""
with pytest.raises(IntegrityError):
factories.ServiceFactory(token="a" * 49)
def test_models_services_token_50_characters_more():
"""The token field should be 50 characters long."""
with pytest.raises(DataError):
factories.ServiceFactory(token="a" * 51)
+26
View File
@@ -0,0 +1,26 @@
"""Test pydantic models & helpers"""
import pytest
from core.schemas import cleanlist
def test_cleanlist_empty():
"""Empty data should return an empty list"""
assert cleanlist(None) == []
assert cleanlist([]) == []
assert cleanlist("") == []
def test_cleanlist_error():
"""Invalid data should raise"""
with pytest.raises(ValueError):
cleanlist(123)
def test_cleanlist():
"""Should return a list of stripped strings and remove the empty items"""
assert cleanlist([1, 2, 3]) == ["1", "2", "3"]
assert cleanlist(" 1, 2,3 ") == ["1", "2", "3"]
assert cleanlist(["1 ", " 2", "3 "]) == ["1", "2", "3"]
assert cleanlist([None, 2, 3, ""]) == ["2", "3"]
+221
View File
@@ -0,0 +1,221 @@
"""Tests Service model for find's core app."""
import base64
import json
from functools import partial
from typing import List
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import rsa
from joserfc import jwe as jose_jwe
from joserfc import jwt as jose_jwt
from joserfc.jwk import RSAKey
from jwt.utils import to_base64url_uint
from opensearchpy.helpers import bulk
from core import opensearch
def delete_test_indices():
"""Drop all search index containing the 'test' word"""
opensearch.client.indices.delete(index="*test*")
def prepare_index(index_name, documents: List, cleanup=True):
"""Prepare the search index before testing a query on it."""
if cleanup:
delete_test_indices()
opensearch.ensure_index_exists(index_name)
# Index new documents
actions = [
{
"_op_type": "index",
"_index": index_name,
"_id": doc["id"],
"_source": {k: v for k, v in doc.items() if k != "id"},
}
for doc in documents
]
bulk(opensearch.client, actions)
# Force refresh again so all changes are visible to search
opensearch.client.indices.refresh(index=index_name)
count = opensearch.client.count(index=index_name)["count"]
assert count == len(documents), f"Expected {len(documents)}, got {count}"
def build_authorization_bearer(token="some_token"):
"""
Build an Authorization Bearer header value from a token.
This can be used like this:
client.post(
...
HTTP_AUTHORIZATION=f"Bearer {build_authorization_bearer('some_token')}",
)
"""
return base64.b64encode(token.encode("utf-8")).decode("utf-8")
def setup_oicd_jwt_resource_server(
responses,
settings,
sub="some_sub",
audience="some_client_id",
):
"""
Setup settings for a resource server with JWT backend.
Simulate an encrypted token introspection.
NOTE : Use it with @responses.activate or the fake introspection view will not work.
"""
token_data = {
"sub": sub,
"iss": "https://oidc.example.com",
"aud": audience,
"client_id": "some_service_provider",
"scope": "docs",
"active": True,
}
private_key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
unencrypted_pem_private_key = private_key.private_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PrivateFormat.TraditionalOpenSSL,
encryption_algorithm=serialization.NoEncryption(),
)
pem_public_key = private_key.public_key().public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo,
)
settings.OIDC_RS_PRIVATE_KEY_STR = unencrypted_pem_private_key.decode("utf-8")
settings.OIDC_RS_ENCRYPTION_KEY_TYPE = "RSA"
settings.OIDC_RS_ENCRYPTION_ENCODING = "A256GCM"
settings.OIDC_RS_ENCRYPTION_ALGO = "RSA-OAEP"
settings.OIDC_RS_SIGNING_ALGO = "RS256"
settings.OIDC_RS_CLIENT_ID = audience
settings.OIDC_RS_CLIENT_SECRET = "some_client_secret"
settings.OIDC_RS_SCOPES = ["openid", "docs", "email"]
settings.OIDC_OP_URL = "https://oidc.example.com"
settings.OIDC_OP_JWKS_ENDPOINT = "https://oidc.example.com/jwks"
settings.OIDC_OP_INTROSPECTION_ENDPOINT = "https://oidc.example.com/introspect"
settings.OIDC_VERIFY_SSL = False
settings.OIDC_TIMEOUT = 5
settings.OIDC_PROXY = None
settings.OIDC_CREATE_USER = False
# Mock the JWKS endpoint
public_numbers = private_key.public_key().public_numbers()
responses.add(
responses.GET,
settings.OIDC_OP_JWKS_ENDPOINT,
body=json.dumps(
{
"keys": [
{
"kty": settings.OIDC_RS_ENCRYPTION_KEY_TYPE,
"alg": settings.OIDC_RS_SIGNING_ALGO,
"use": "sig",
"kid": "1234567890",
"n": to_base64url_uint(public_numbers.n).decode("ascii"),
"e": to_base64url_uint(public_numbers.e).decode("ascii"),
}
]
}
),
)
def encrypt_jwt(json_data):
"""Encrypt the JWT token for the backend to decrypt."""
token = jose_jwt.encode(
{
"kid": "1234567890",
"alg": settings.OIDC_RS_SIGNING_ALGO,
},
json_data,
RSAKey.import_key(unencrypted_pem_private_key),
algorithms=[settings.OIDC_RS_SIGNING_ALGO],
)
return jose_jwe.encrypt_compact(
protected={
"alg": settings.OIDC_RS_ENCRYPTION_ALGO,
"enc": settings.OIDC_RS_ENCRYPTION_ENCODING,
},
plaintext=token,
public_key=RSAKey.import_key(pem_public_key),
algorithms=[
settings.OIDC_RS_ENCRYPTION_ALGO,
settings.OIDC_RS_ENCRYPTION_ENCODING,
],
)
responses.add(
responses.POST,
"https://oidc.example.com/introspect",
body=encrypt_jwt(
{
"iss": "https://oidc.example.com",
"aud": audience, # settings.OIDC_RS_CLIENT_ID
"token_introspection": token_data,
}
),
)
def setup_oicd_resource_server(
responses,
settings,
sub="some_sub",
audience="some_client_id",
introspect=None,
): # pylint: disable=too-many-arguments
"""
Setup settings for a resource server.
Simulate a token introspection.
NOTE : Use it with @responses.activate or the fake introspection view will not work.
"""
token_data = {
"sub": sub,
"iss": "https://oidc.example.com",
"aud": audience,
"client_id": audience,
"scope": "docs",
"active": True,
}
settings.OIDC_RS_ENCRYPTION_KEY_TYPE = "RSA"
settings.OIDC_RS_ENCRYPTION_ENCODING = "A256GCM"
settings.OIDC_RS_ENCRYPTION_ALGO = "RSA-OAEP"
settings.OIDC_RS_SIGNING_ALGO = "RS256"
settings.OIDC_RS_CLIENT_ID = audience
settings.OIDC_RS_CLIENT_SECRET = "some_client_secret"
settings.OIDC_RS_SCOPES = ["openid", "docs", "email"]
settings.OIDC_OP_URL = "https://oidc.example.com"
settings.OIDC_OP_INTROSPECTION_ENDPOINT = "https://oidc.example.com/introspect"
settings.OIDC_VERIFY_SSL = False
settings.OIDC_TIMEOUT = 5
settings.OIDC_PROXY = None
settings.OIDC_CREATE_USER = False
if callable(introspect):
responses.add_callback(
responses.POST,
"https://oidc.example.com/introspect",
callback=partial(introspect, user_info=token_data),
)
else:
responses.add(
responses.POST,
"https://oidc.example.com/introspect",
body=json.dumps(token_data),
)
+11
View File
@@ -0,0 +1,11 @@
"""URL configuration for find's core app."""
from django.urls import include, path
from .views import IndexDocumentView, SearchDocumentView
urlpatterns = [
path("documents/index/", IndexDocumentView.as_view(), name="document"),
path("documents/search/", SearchDocumentView.as_view(), name="document"),
path("", include("lasuite.oidc_resource_server.urls")),
]
+315
View File
@@ -0,0 +1,315 @@
"""Views for find's core app."""
import logging
from django.core.exceptions import SuspiciousOperation
from lasuite.oidc_resource_server.authentication import ResourceServerAuthentication
from lasuite.oidc_resource_server.mixins import ResourceServerMixin
from pydantic import ValidationError as PydanticValidationError
from rest_framework import status, views
from rest_framework.response import Response
from . import enums, schemas
from .authentication import ServiceTokenAuthentication
from .models import Service
from .opensearch import client, ensure_index_exists
from .permissions import IsAuthAuthenticated
logger = logging.getLogger(__name__)
class IndexDocumentView(views.APIView):
"""
API view for indexing documents in OpenSearch.
- Handles both single document and bulk document indexing.
- The index is dynamically determined based on the service authentication token,
ensuring that each service has its own isolated index.
"""
authentication_classes = [ServiceTokenAuthentication]
permission_classes = [IsAuthAuthenticated]
# pylint: disable=too-many-locals
def post(self, request, *args, **kwargs):
"""
API view for indexing documents into OpenSearch index of the authenticated service.
This view supports both single document indexing and bulk indexing. It handles
the following scenarios based on the type of request data:
1. **Single Document Indexing**: If the request contains a single document (as a
dictionary), it will be indexed into an OpenSearch index named `find-{auth_token}`.
On success, the indexed document is returned with a `201 Created` status. If an
error occurs, a `400 Bad Request` response with an error message is returned.
2. **Bulk Indexing**: If the request contains a list of documents, each document is
validated and indexed in bulk. The response includes a detailed status of each
document, indicating whether it was successfully indexed or if an error occurred.
The HTTP status code for the bulk indexing operation is `207 Multi-Status`, and the
response body contains information about the success or failure of each individual
document.
Methods:
-------
post(request, *args, **kwargs):
Handles POST requests to index either a single document or a list of documents.
- **Single Document**: Expects a dictionary representing a document.
- **Bulk Indexing**: Expects a list of dictionaries, each representing a document.
Request Data:
-------------
For single document indexing:
- Content-Type: application/json
- Body: JSON object representing a document.
For bulk indexing:
- Content-Type: application/json
- Body: JSON array of JSON objects, each representing a document.
Responses:
-----------
- **Single Document Indexing**:
- 201 Created: Returns the indexed document.
- 400 Bad Request: Returns an error message if the document is invalid or if
indexing fails.
- **Bulk Indexing**:
- 207 Multi-Status if all documents formatting is correct, 400 Bad Request otherwise:
- Returns a list of results for all documents, with details of success and indexing
errors.
"""
index_name = request.auth.name
if isinstance(request.data, list):
# Bulk indexing several documents
results = []
actions = []
has_errors = False
for i, document_data in enumerate(request.data):
try:
document = schemas.DocumentSchema(**document_data)
except PydanticValidationError as excpt:
errors = [
{key: error[key] for key in ("msg", "type", "loc")}
for error in excpt.errors()
]
results.append({"index": i, "status": "error", "errors": errors})
has_errors = True
else:
document_dict = document.model_dump()
_id = document_dict.pop("id")
actions.append({"index": {"_id": _id}})
actions.append(document_dict)
results.append({"index": i, "_id": _id, "status": "valid"})
if has_errors:
return Response(results, status=status.HTTP_400_BAD_REQUEST)
# Build index if needed.
ensure_index_exists(index_name)
response = client.bulk(index=index_name, body=actions)
for i, item in enumerate(response["items"]):
if item["index"]["status"] != 201:
results[i]["status"] = "error"
results[i]["message"] = (
item["index"].get("error", {}).get("reason", "Unknown error")
)
else:
results[i]["status"] = "success"
return Response(results, status=status.HTTP_207_MULTI_STATUS)
# Indexing a single document
document = schemas.DocumentSchema(**request.data)
document_dict = document.model_dump()
_id = document_dict.pop("id")
# Build index if needed.
ensure_index_exists(index_name)
client.index(index=index_name, body=document_dict, id=_id)
return Response(
{"status": "created", "_id": _id}, status=status.HTTP_201_CREATED
)
class SearchDocumentView(ResourceServerMixin, views.APIView):
"""
API view for searching documents in OpenSearch.
- Enables searching through indexed documents with support for various filters
and sorting options.
- The search results can be sorted or filtered via querystring parameters.
"""
authentication_classes = [ResourceServerAuthentication]
permission_classes = [IsAuthAuthenticated]
def _get_opensearch_indices(self, audience, services):
# Get request user service
try:
user_service = Service.objects.get(client_id=audience, is_active=True)
except Service.DoesNotExist as e:
logger.warning("Login failed: No service %s found", audience)
raise SuspiciousOperation("Service is not available") from e
# Find allowed sub-services for this service
allowed_services = set(user_service.services.values_list("name", flat=True))
allowed_services.add(user_service.name)
if services:
available = set(services).intersection(allowed_services)
if len(available) < len(services):
raise SuspiciousOperation("Some requested services are not available")
return allowed_services
def post(self, request, *args, **kwargs):
"""
Handle POST requests to perform a search on indexed documents with optional filtering
and ordering.
The search query should be provided as a "q" parameter. The method constructs a
search request to OpenSearch using the specified query, with the option to filter by
'reach' and order by 'relevance', 'created_at', 'updated_at', or 'size'.
The results are further filtered by 'users' and 'groups' based on the authentication
header.
Body Parameters:
---------------
q : str
The search query string. This is a required parameter.
reach : str, optional
Filter results based on the 'reach' field.
order_by : str, optional
Order results by 'relevance', 'created_at', 'updated_at', or 'size'.
Defaults to 'relevance' if not specified.
order_direction : str, optional
Order direction, 'asc' for ascending or 'desc' for descending.
Defaults to 'desc'.
page_number : int, optional
The page number to retrieve.
Defaults to 1 if not specified.
page_size : int, optional
The number of results to return per page.
Defaults to 50 if not specified.
services: List[str], optional
List of services on which we intend to run the query (current service if left empty)
visited: List[sub], optional
List of public/authenticated documents the user has visited to limit
the document returned to the ones the current user has seen.
Built from linkreach list of a document in docs app.
Returns:
--------
Response : rest_framework.response.Response
- 200 OK: Returns a list of search results matching the query.
- 400 Bad Request: If the query parameter 'q' is not provided or invalid.
"""
# Get list of groups related to the user from SCIM provider (consider caching result)
audience = self._get_service_provider_audience()
user_sub = self.request.user.sub
groups = []
# //////////////////////////////////////////////////
# Extract and validate query parameters using Pydantic schema
params = schemas.SearchQueryParametersSchema(**request.data)
# Compute pagination parameters
from_value = (params.page_number - 1) * params.page_size
size_value = params.page_size
# Get index list for search query
try:
search_indices = self._get_opensearch_indices(
audience, services=params.services
)
except SuspiciousOperation as e:
return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST)
# Prepare the search query
search_body = {
"_source": enums.SOURCE_FIELDS, # limit the fields to return
"script_fields": {
"number_of_users": {"script": {"source": "doc['users'].size()"}},
"number_of_groups": {"script": {"source": "doc['groups'].size()"}},
},
"query": {"bool": {"must": [], "filter": []}},
"sort": [],
"from": from_value,
"size": size_value,
}
# Adding the text query
if params.q == "*":
search_body["query"]["bool"]["must"].append({"match_all": {}})
else:
search_body["query"]["bool"]["must"].append(
{
"multi_match": {
"query": params.q,
# Give title more importance over content by a power of 3
"fields": ["title.text^3", "content"],
}
}
)
# Add sorting logic based on relevance or specified field
if params.order_by == enums.RELEVANCE:
search_body["sort"].append({"_score": {"order": params.order_direction}})
else:
search_body["sort"].append(
{params.order_by: {"order": params.order_direction}}
)
# Apply access control based on documents reach
search_body["query"]["bool"]["must"].append(
{
"bool": {
"should": [
# Access control on public & authenticated reach
{
"bool": {
"must_not": {
"term": {enums.REACH: enums.ReachEnum.RESTRICTED},
},
# Limit search to already visited documents.
"must": {
"terms": {
"_id": sorted(params.visited),
}
},
},
},
# Access control on restricted search : either user or group should match
{"term": {enums.USERS: user_sub}},
{"terms": {enums.GROUPS: groups}},
],
# At least one of the 2 optional should clauses must apply
"minimum_should_match": 1,
}
}
)
# Optional filter by reach if explicitly provided in the query
if params.reach is not None:
search_body["query"]["bool"]["filter"].append(
{"term": {enums.REACH: params.reach}}
)
# Always filter out inactive documents
search_body["query"]["bool"]["filter"].append({"term": {"is_active": True}})
response = client.search( # pylint: disable=unexpected-keyword-arg
index=",".join(search_indices),
body=search_body,
# Argument added by the query_params() decorator of opensearch and
# not in the method declaration.
ignore_unavailable=True,
)
return Response(response["hits"]["hits"], status=status.HTTP_200_OK)
View File
+10
View File
@@ -0,0 +1,10 @@
"""Parameters that define how the demo site will be built."""
NB_OBJECTS = {
"documents": 1000,
"services": 5
}
DEV_SERVICES = (
{"name": "docs", "client_id": "impress", "token": "find-api-key-for-docs-with-exactly-50-chars-length"},
)
@@ -0,0 +1,205 @@
# ruff: noqa: S311, S106
"""create_demo management command"""
import logging
import random
import time
from uuid import uuid4
from django.conf import settings
from django.core.management.base import BaseCommand, CommandError
from django.utils import timezone
from django.utils.text import slugify
from faker import Faker
from opensearchpy.helpers import bulk
from core import enums, factories, opensearch
from demo import defaults
fake = Faker()
logger = logging.getLogger("find.commands.demo.create_demo")
class BulkIndexing:
"""A utility class to index to OpenSearch in bulk by just pushing to a queue."""
BATCH_SIZE = 20000
def __init__(self, stdout, *args, **kwargs):
"""Define actions as a list."""
self.actions = []
self.stdout = stdout
self.logger = logging.getLogger(__name__)
def bulk_index(self):
"""Actually index documents in bulk to OpenSearch."""
_success, failed = bulk(opensearch.client, self.actions, stats_only=False)
if failed:
self.handle_failures(failed)
# Clear the actions list after bulk indexing and display progress
self.actions.clear()
self.stdout.write(".", ending="")
def handle_failures(self, failed):
"""Handle the failed bulk operations."""
for failure in failed:
self.logger.error("Failed to index document: %s", failure)
def push(self, index_name, _id, document):
"""Add a document to queue so that it gets created in bulk."""
self.actions.append(
{
"_op_type": "index",
"_index": index_name,
"_id": _id,
"_source": document,
}
)
if len(self.actions) >= self.BATCH_SIZE:
self.bulk_index()
def flush(self):
"""Flush any remaining documents in the queue."""
if self.actions:
self.bulk_index()
def __del__(self):
"""Ensure that any remaining documents are indexed when the instance is deleted."""
self.flush()
class Timeit:
"""A utility context manager/method decorator to time execution."""
total_time = 0
def __init__(self, stdout, sentence=None):
"""Set the sentence to be displayed for timing information."""
self.sentence = sentence
self.start = None
self.stdout = stdout
def __call__(self, func):
"""Behavior on call for use as a method decorator."""
def timeit_wrapper(*args, **kwargs):
"""wrapper to trigger/stop the timer before/after function call."""
self.__enter__()
result = func(*args, **kwargs)
self.__exit__(None, None, None)
return result
return timeit_wrapper
def __enter__(self):
"""Start timer upon entering context manager."""
self.start = time.perf_counter()
if self.sentence:
self.stdout.write(self.sentence, ending=".")
def __exit__(self, exc_type, exc_value, exc_tb):
"""Stop timer and display result upon leaving context manager."""
if exc_type is not None:
raise exc_type(exc_value)
end = time.perf_counter()
elapsed_time = end - self.start
if self.sentence:
self.stdout.write(f" Took {elapsed_time:g} seconds")
self.__class__.total_time += elapsed_time
return elapsed_time
def generate_document():
"""Generate a realistic document dictionary faster than factory_boy does it."""
created_at = fake.past_datetime(tzinfo=timezone.get_current_timezone())
updated_at = fake.date_time_between(
start_date=created_at,
end_date=timezone.now(),
tzinfo=timezone.get_current_timezone(),
)
return {
"title": fake.sentence(nb_words=10, variable_nb_words=True),
"content": "\n".join(fake.paragraphs(nb=5)),
"created_at": created_at,
"updated_at": updated_at,
"size": random.randint(0, 100 * 1024**2),
"users": [str(uuid4()) for _ in range(3)],
"groups": [slugify(fake.word()) for _ in range(3)],
"reach": random.choice(list(enums.ReachEnum)).value,
}
def create_demo(stdout):
"""
Create a database with demo data for developers to work in a realistic environment.
"""
opensearch.client.indices.delete("*")
with Timeit(stdout, "Creating services"):
services = factories.ServiceFactory.create_batch(
defaults.NB_OBJECTS["services"]
)
for service in services:
opensearch.ensure_index_exists(service.name)
opensearch.client.indices.refresh(index=service.name)
with Timeit(stdout, "Creating documents"):
actions = BulkIndexing(stdout)
for _ in range(defaults.NB_OBJECTS["documents"]):
service = random.choice(services)
document = generate_document()
actions.push(service.name, uuid4(), document)
actions.flush()
with Timeit(stdout, "Creating dev services"):
for conf in defaults.DEV_SERVICES:
service = factories.ServiceFactory(**conf)
opensearch.ensure_index_exists(service.name)
opensearch.client.indices.refresh(index=service.name)
# Check and report on indexed documents
total_indexed = 0
for service in services:
opensearch.client.indices.refresh(index=service.name)
indexed = opensearch.client.count(index=service.name)["count"]
stdout.write(f" - {service.name:s}: {indexed:d} documents")
total_indexed += indexed
stdout.write(f" TOTAL: {total_indexed:d} documents")
class Command(BaseCommand):
"""A management command to create a demo database."""
help = __doc__
def add_arguments(self, parser):
"""Add argument to require forcing execution when not in debug mode."""
parser.add_argument(
"-f",
"--force",
action="store_true",
default=False,
help="Force command execution despite DEBUG is set to False",
)
def handle(self, *args, **options):
"""Handling of the management command."""
if not settings.DEBUG and not options["force"]:
raise CommandError(
(
"This command is not meant to be used in production environment "
"except you know what you are doing, if so use --force parameter"
)
)
create_demo(self.stdout)
View File
@@ -0,0 +1,29 @@
"""Test the `create_demo` management command"""
from unittest import mock
from django.core.management import call_command
from django.test import override_settings
import pytest
from core import models, opensearch
from demo import defaults
pytestmark = pytest.mark.django_db
TEST_NB_OBJECTS = {
"documents": 4,
"services": 2,
}
@override_settings(DEBUG=True)
@mock.patch.dict(defaults.NB_OBJECTS, TEST_NB_OBJECTS)
def test_commands_create_demo():
"""The create_demo management command should create objects as expected."""
call_command("create_demo")
assert models.Service.objects.count() == 2
assert opensearch.client.count()["count"] == 4
View File
+23
View File
@@ -0,0 +1,23 @@
"""find celery configuration file."""
import os
from celery import Celery
from configurations.importer import install
# Set the default Django settings module for the 'celery' program.
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "find.settings")
os.environ.setdefault("DJANGO_CONFIGURATION", "Development")
install(check_options=True)
app = Celery("find")
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object("django.conf:settings", namespace="CELERY")
# Load task modules from all registered Django apps.
app.autodiscover_tasks()
+637
View File
@@ -0,0 +1,637 @@
"""
Django settings for find project.
Generated by 'django-admin startproject' using Django 3.1.5.
For more information on this file, see
https://docs.djangoproject.com/en/3.1/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/3.1/ref/settings/
"""
import json
import os
from socket import gethostbyname, gethostname
from django.utils.translation import gettext_lazy as _
import sentry_sdk
from configurations import Configuration, values
from sentry_sdk.integrations.django import DjangoIntegration
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA_DIR = os.path.join("/", "data")
def get_release():
"""
Get the current release of the application
By release, we mean the release from the version.json file à la Mozilla [1]
(if any). If this file has not been found, it defaults to "NA".
[1]
https://github.com/mozilla-services/Dockerflow/blob/master/docs/version_object.md
"""
# Try to get the current release from the version.json file generated by the
# CI during the Docker image build
try:
with open(os.path.join(BASE_DIR, "version.json"), encoding="utf8") as version:
return json.load(version)["version"]
except FileNotFoundError:
return "NA" # Default: not available
class Base(Configuration):
"""
This is the base configuration every configuration (aka environment) should inherit from. It
is recommended to configure third-party applications by creating a configuration mixins in
./configurations and compose the Base configuration with those mixins.
It depends on an environment variable that SHOULD be defined:
* DJANGO_SECRET_KEY
You may also want to override default configuration by setting the following environment
variables:
* DJANGO_SENTRY_DSN
* DB_NAME
* DB_HOST
* DB_PASSWORD
* DB_USER
"""
DEBUG = False
USE_SWAGGER = False
API_VERSION = "v1.0"
# Security
ALLOWED_HOSTS = values.ListValue([])
SECRET_KEY = values.Value(None)
SILENCED_SYSTEM_CHECKS = values.ListValue([])
# Application definition
ROOT_URLCONF = "find.urls"
WSGI_APPLICATION = "find.wsgi.application"
# Database
DATABASES = {
"default": {
"ENGINE": values.Value(
"django.db.backends.postgresql_psycopg2",
environ_name="DB_ENGINE",
environ_prefix=None,
),
"NAME": values.Value("find", environ_name="DB_NAME", environ_prefix=None),
"USER": values.Value("dinum", environ_name="DB_USER", environ_prefix=None),
"PASSWORD": values.Value(
"pass", environ_name="DB_PASSWORD", environ_prefix=None
),
"HOST": values.Value(
"localhost", environ_name="DB_HOST", environ_prefix=None
),
"PORT": values.Value(5432, environ_name="DB_PORT", environ_prefix=None),
}
}
DEFAULT_AUTO_FIELD = "django.db.models.AutoField"
# Static files (CSS, JavaScript, Images)
STATIC_URL = "/static/"
STATIC_ROOT = os.path.join(DATA_DIR, "static")
MEDIA_URL = "/media/"
MEDIA_ROOT = os.path.join(DATA_DIR, "media")
SITE_ID = 1
STORAGES = {
"default": {
"BACKEND": "storages.backends.s3.S3Storage",
},
"staticfiles": {
"BACKEND": values.Value(
"whitenoise.storage.CompressedManifestStaticFilesStorage",
environ_name="STORAGES_STATICFILES_BACKEND",
),
},
}
# Internationalization
# https://docs.djangoproject.com/en/3.1/topics/i18n/
# Languages
LANGUAGE_CODE = values.Value("en-us")
# Careful! Languages should be ordered by priority, as this tuple is used to get
# fallback/default languages throughout the app.
LANGUAGES = values.SingleNestedTupleValue(
(
("en-us", _("English")),
("fr-fr", _("French")),
)
)
LOCALE_PATHS = (os.path.join(BASE_DIR, "locale"),)
TIME_ZONE = "UTC"
USE_I18N = True
USE_TZ = True
# Templates
TEMPLATES = [
{
"BACKEND": "django.template.backends.django.DjangoTemplates",
"DIRS": [os.path.join(BASE_DIR, "templates")],
"OPTIONS": {
"context_processors": [
"django.contrib.auth.context_processors.auth",
"django.contrib.messages.context_processors.messages",
"django.template.context_processors.csrf",
"django.template.context_processors.debug",
"django.template.context_processors.i18n",
"django.template.context_processors.media",
"django.template.context_processors.request",
"django.template.context_processors.tz",
],
"loaders": [
"django.template.loaders.filesystem.Loader",
"django.template.loaders.app_directories.Loader",
],
},
},
]
MIDDLEWARE = [
"django.middleware.security.SecurityMiddleware",
"whitenoise.middleware.WhiteNoiseMiddleware",
"django.contrib.sessions.middleware.SessionMiddleware",
"django.middleware.locale.LocaleMiddleware",
"django.middleware.clickjacking.XFrameOptionsMiddleware",
"corsheaders.middleware.CorsMiddleware",
"django.middleware.common.CommonMiddleware",
"django.middleware.csrf.CsrfViewMiddleware",
"django.contrib.auth.middleware.AuthenticationMiddleware",
"django.contrib.messages.middleware.MessageMiddleware",
"dockerflow.django.middleware.DockerflowMiddleware",
]
AUTHENTICATION_BACKENDS = [
"django.contrib.auth.backends.ModelBackend",
]
# Django applications from the highest priority to the lowest
INSTALLED_APPS = [
# find
"core",
"demo",
# Third party apps
"corsheaders",
"dockerflow.django",
"drf_spectacular",
"rest_framework",
# Django
"django.contrib.admin",
"django.contrib.auth",
"django.contrib.contenttypes",
"django.contrib.postgres",
"django.contrib.sessions",
"django.contrib.sites",
"django.contrib.messages",
"django.contrib.staticfiles",
# OIDC third party
"mozilla_django_oidc",
]
# Cache
CACHES = {
"default": {"BACKEND": "django.core.cache.backends.locmem.LocMemCache"},
}
REST_FRAMEWORK = {
"DEFAULT_AUTHENTICATION_CLASSES": (
"rest_framework.authentication.TokenAuthentication",
),
"DEFAULT_PARSER_CLASSES": [
"rest_framework.parsers.JSONParser",
],
"DEFAULT_PERMISSION_CLASSES": [
"rest_framework.permissions.IsAuthenticated",
],
"EXCEPTION_HANDLER": "core.api.exception_handler",
"DEFAULT_PAGINATION_CLASS": "rest_framework.pagination.PageNumberPagination",
"PAGE_SIZE": 20,
"DEFAULT_VERSIONING_CLASS": "rest_framework.versioning.URLPathVersioning",
"DEFAULT_SCHEMA_CLASS": "drf_spectacular.openapi.AutoSchema",
}
# OpenSearch settings
OPENSEARCH_HOST = values.Value(
default="opensearch", environ_name="OPENSEARCH_HOST", environ_prefix=None
)
OPENSEARCH_PORT = values.Value(
default=9200, environ_name="OPENSEARCH_PORT", environ_prefix=None
)
OPENSEARCH_USER = values.Value(
default="admin", environ_name="OPENSEARCH_USER", environ_prefix=None
)
OPENSEARCH_PASSWORD = values.SecretValue(
environ_name="OPENSEARCH_PASSWORD", environ_prefix=None
)
OPENSEARCH_USE_SSL = values.BooleanValue(
default=True, environ_name="OPENSEARCH_USE_SSL", environ_prefix=None
)
SPECTACULAR_SETTINGS = {
"TITLE": "Find API",
"DESCRIPTION": "This is the find API schema.",
"VERSION": "0.1.0",
"SERVE_INCLUDE_SCHEMA": False,
"ENABLE_DJANGO_DEPLOY_CHECK": values.BooleanValue(
default=False,
environ_name="SPECTACULAR_SETTINGS_ENABLE_DJANGO_DEPLOY_CHECK",
),
"COMPONENT_SPLIT_REQUEST": True,
# OTHER SETTINGS
"SWAGGER_UI_DIST": "SIDECAR", # shorthand to use the sidecar instead
"SWAGGER_UI_FAVICON_HREF": "SIDECAR",
"REDOC_DIST": "SIDECAR",
}
AUTH_USER_MODEL = "core.User"
# CORS
CORS_ALLOW_CREDENTIALS = True
CORS_ALLOW_ALL_ORIGINS = values.BooleanValue(True)
CORS_ALLOWED_ORIGINS = values.ListValue([])
CORS_ALLOWED_ORIGIN_REGEXES = values.ListValue([])
# Sentry
SENTRY_DSN = values.Value(None, environ_name="SENTRY_DSN")
# Celery
CELERY_BROKER_URL = values.Value("redis://redis:6379/0")
CELERY_BROKER_TRANSPORT_OPTIONS = values.DictValue({})
# Session
SESSION_ENGINE = "django.contrib.sessions.backends.cache"
SESSION_CACHE_ALIAS = "default"
SESSION_COOKIE_AGE = 60 * 60 * 12
# OIDC - Authorization Code Flow
OIDC_CREATE_USER = values.BooleanValue(
default=True,
environ_name="OIDC_CREATE_USER",
)
OIDC_RP_SIGN_ALGO = values.Value(
"RS256", environ_name="OIDC_RP_SIGN_ALGO", environ_prefix=None
)
OIDC_RP_CLIENT_ID = values.Value(
"find", environ_name="OIDC_RP_CLIENT_ID", environ_prefix=None
)
OIDC_RP_CLIENT_SECRET = values.Value(
None,
environ_name="OIDC_RP_CLIENT_SECRET",
environ_prefix=None,
)
OIDC_OP_URL = values.Value(None, environ_name="OIDC_OP_URL", environ_prefix=None)
OIDC_OP_JWKS_ENDPOINT = values.Value(
environ_name="OIDC_OP_JWKS_ENDPOINT", environ_prefix=None
)
OIDC_OP_INTROSPECTION_ENDPOINT = values.Value(
environ_name="OIDC_OP_INTROSPECTION_ENDPOINT", environ_prefix=None
)
OIDC_OP_AUTHORIZATION_ENDPOINT = values.Value(
environ_name="OIDC_OP_AUTHORIZATION_ENDPOINT", environ_prefix=None
)
OIDC_OP_TOKEN_ENDPOINT = values.Value(
None, environ_name="OIDC_OP_TOKEN_ENDPOINT", environ_prefix=None
)
OIDC_OP_USER_ENDPOINT = values.Value(
None, environ_name="OIDC_OP_USER_ENDPOINT", environ_prefix=None
)
OIDC_OP_LOGOUT_ENDPOINT = values.Value(
None, environ_name="OIDC_OP_LOGOUT_ENDPOINT", environ_prefix=None
)
OIDC_AUTH_REQUEST_EXTRA_PARAMS = values.DictValue(
{}, environ_name="OIDC_AUTH_REQUEST_EXTRA_PARAMS", environ_prefix=None
)
OIDC_RP_SCOPES = values.Value(
"openid email", environ_name="OIDC_RP_SCOPES", environ_prefix=None
)
# OIDC - Resource server
OIDC_DRF_AUTH_BACKEND = values.Value(
"lasuite.oidc_login.backends.OIDCAuthenticationBackend",
environ_name="OIDC_DRF_AUTH_BACKEND",
environ_prefix=None,
)
OIDC_RS_BACKEND_CLASS = values.Value(
"core.authentication.FinderResourceServerBackend",
environ_name="OIDC_RS_BACKEND_CLASS",
environ_prefix=None,
)
OIDC_RS_AUDIENCE_CLAIM = values.Value(
"client_id",
environ_name="OIDC_RS_AUDIENCE_CLAIM",
environ_prefix=None,
)
OIDC_RS_CLIENT_ID = values.Value(
None, environ_name="OIDC_RS_CLIENT_ID", environ_prefix=None
)
OIDC_RS_CLIENT_SECRET = values.Value(
None,
environ_name="OIDC_RS_CLIENT_SECRET",
environ_prefix=None,
)
OIDC_RS_SIGNING_ALGO = values.Value(
default="ES256", environ_name="OIDC_RS_SIGNING_ALGO", environ_prefix=None
)
OIDC_RS_SCOPES = values.ListValue(
["openid"], environ_name="OIDC_RS_SCOPES", environ_prefix=None
)
OIDC_RS_PRIVATE_KEY_STR = values.Value(
default=None,
environ_name="OIDC_RS_PRIVATE_KEY_STR",
environ_prefix=None,
)
OIDC_RS_ENCRYPTION_KEY_TYPE = values.Value(
default="RSA",
environ_name="OIDC_RS_ENCRYPTION_KEY_TYPE",
environ_prefix=None,
)
OIDC_RS_ENCRYPTION_ALGO = values.Value(
default="RSA-OAEP",
environ_name="OIDC_RS_ENCRYPTION_ALGO",
environ_prefix=None,
)
OIDC_RS_ENCRYPTION_ENCODING = values.Value(
default="A256GCM",
environ_name="OIDC_RS_ENCRYPTION_ENCODING",
environ_prefix=None,
)
OIDC_VERIFY_SSL = values.BooleanValue(
True, environ_name="OIDC_VERIFY_SSL", environ_prefix=None
)
OIDC_TIMEOUT = values.Value(None, environ_name="OIDC_TIMEOUT", environ_prefix=None)
OIDC_PROXY = values.Value(None, environ_name="OIDC_PROXY", environ_prefix=None)
LOGIN_REDIRECT_URL = values.Value(
None, environ_name="LOGIN_REDIRECT_URL", environ_prefix=None
)
LOGIN_REDIRECT_URL_FAILURE = values.Value(
None, environ_name="LOGIN_REDIRECT_URL_FAILURE", environ_prefix=None
)
LOGOUT_REDIRECT_URL = values.Value(
None, environ_name="LOGOUT_REDIRECT_URL", environ_prefix=None
)
OIDC_USE_NONCE = values.BooleanValue(
default=True, environ_name="OIDC_USE_NONCE", environ_prefix=None
)
OIDC_REDIRECT_REQUIRE_HTTPS = values.BooleanValue(
default=False, environ_name="OIDC_REDIRECT_REQUIRE_HTTPS", environ_prefix=None
)
OIDC_REDIRECT_ALLOWED_HOSTS = values.ListValue(
default=[], environ_name="OIDC_REDIRECT_ALLOWED_HOSTS", environ_prefix=None
)
OIDC_STORE_ID_TOKEN = values.BooleanValue(
default=True, environ_name="OIDC_STORE_ID_TOKEN", environ_prefix=None
)
ALLOW_LOGOUT_GET_METHOD = values.BooleanValue(
default=True, environ_name="ALLOW_LOGOUT_GET_METHOD", environ_prefix=None
)
# Logging
# We want to make it easy to log to console but by default we log production
# to Sentry and don't want to log to console.
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"simple": {
"format": "{asctime} {name} {levelname} {message}",
"style": "{",
},
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"formatter": "simple",
},
},
# Override root logger to send it to console
"root": {
"handlers": ["console"],
"level": values.Value(
"INFO", environ_name="LOGGING_LEVEL_LOGGERS_ROOT", environ_prefix=None
),
},
"loggers": {
"find": {
"handlers": ["console"],
"level": values.Value(
"INFO",
environ_name="LOGGING_LEVEL_LOGGERS_APP",
environ_prefix=None,
),
"propagate": True,
},
},
}
# pylint: disable=invalid-name
@property
def ENVIRONMENT(self):
"""Environment in which the application is launched."""
return self.__class__.__name__.lower()
# pylint: disable=invalid-name
@property
def RELEASE(self):
"""
Return the release information.
Delegate to the module function to enable easier testing.
"""
return get_release()
@classmethod
def post_setup(cls):
"""Post setup configuration.
This is the place where you can configure settings that require other
settings to be loaded.
"""
super().post_setup()
# The SENTRY_DSN setting should be available to activate sentry for an environment
if cls.SENTRY_DSN is not None:
sentry_sdk.init(
dsn=cls.SENTRY_DSN,
environment=cls.__name__.lower(),
release=get_release(),
integrations=[DjangoIntegration()],
)
with sentry_sdk.configure_scope() as scope:
scope.set_extra("application", "backend")
class Build(Base):
"""Settings used when the application is built.
This environment should not be used to run the application. Just to build it with non-blocking
settings.
"""
SECRET_KEY = values.Value("DummyKey")
STORAGES = {
"default": {
"BACKEND": "django.core.files.storage.FileSystemStorage",
},
"staticfiles": {
"BACKEND": values.Value(
"whitenoise.storage.CompressedManifestStaticFilesStorage",
environ_name="STORAGES_STATICFILES_BACKEND",
),
},
}
class Development(Base):
"""
Development environment settings
We set DEBUG to True and configure the server to respond from all hosts.
"""
ALLOWED_HOSTS = ["*"]
CORS_ALLOW_ALL_ORIGINS = True
CSRF_TRUSTED_ORIGINS = ["http://localhost:8072", "http://localhost:3000"]
DEBUG = True
SESSION_COOKIE_NAME = "find_sessionid"
USE_SWAGGER = True
def __init__(self):
# pylint: disable=invalid-name
self.INSTALLED_APPS += ["django_extensions", "drf_spectacular_sidecar"]
class Test(Base):
"""Test environment settings"""
PASSWORD_HASHERS = [
"django.contrib.auth.hashers.MD5PasswordHasher",
]
USE_SWAGGER = True
CELERY_TASK_ALWAYS_EAGER = values.BooleanValue(True)
def __init__(self):
# pylint: disable=invalid-name
self.INSTALLED_APPS += ["drf_spectacular_sidecar"]
class ContinuousIntegration(Test):
"""
Continuous Integration environment settings
nota bene: it should inherit from the Test environment.
"""
class Production(Base):
"""
Production environment settings
You must define the ALLOWED_HOSTS environment variable in Production
configuration (and derived configurations):
ALLOWED_HOSTS=["foo.com", "foo.fr"]
"""
# Security
ALLOWED_HOSTS = [
*values.ListValue([], environ_name="ALLOWED_HOSTS"),
gethostbyname(gethostname()),
]
CSRF_TRUSTED_ORIGINS = values.ListValue([])
SECURE_BROWSER_XSS_FILTER = True
SECURE_CONTENT_TYPE_NOSNIFF = True
# SECURE_PROXY_SSL_HEADER allows to fix the scheme in Django's HttpRequest
# object when your application is behind a reverse proxy.
#
# Keep this SECURE_PROXY_SSL_HEADER configuration only if :
# - your Django app is behind a proxy.
# - your proxy strips the X-Forwarded-Proto header from all incoming requests
# - Your proxy sets the X-Forwarded-Proto header and sends it to Django
#
# In other cases, you should comment the following line to avoid security issues.
# SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
# Modern browsers require to have the `secure` attribute on cookies with `Samesite=none`
CSRF_COOKIE_SECURE = True
SESSION_COOKIE_SECURE = True
# Privacy
SECURE_REFERRER_POLICY = "same-origin"
CACHES = {
"default": {
"BACKEND": "django_redis.cache.RedisCache",
"LOCATION": values.Value(
"redis://redis:6379/1",
environ_name="REDIS_URL",
environ_prefix=None,
),
"OPTIONS": {
"CLIENT_CLASS": "django_redis.client.DefaultClient",
},
},
}
class Feature(Production):
"""
Feature environment settings
nota bene: it should inherit from the Production environment.
"""
class Staging(Production):
"""
Staging environment settings
nota bene: it should inherit from the Production environment.
"""
class PreProduction(Production):
"""
Pre-production environment settings
nota bene: it should inherit from the Production environment.
"""
class Demo(Production):
"""
Demonstration environment settings
nota bene: it should inherit from the Production environment.
"""
STORAGES = {
"default": {
"BACKEND": "django.core.files.storage.FileSystemStorage",
},
"staticfiles": {
"BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
},
}
+38
View File
@@ -0,0 +1,38 @@
"""URL configuration for the find project"""
from django.conf import settings
from django.contrib import admin
from django.urls import include, path, re_path
from drf_spectacular.views import (
SpectacularJSONAPIView,
SpectacularRedocView,
SpectacularSwaggerView,
)
urlpatterns = [
path("admin/", admin.site.urls),
path(f"api/{settings.API_VERSION}/", include("core.urls")),
]
if settings.USE_SWAGGER or settings.DEBUG:
urlpatterns += [
path(
f"{settings.API_VERSION}/swagger.json",
SpectacularJSONAPIView.as_view(
api_version=settings.API_VERSION,
urlconf="core.urls",
),
name="client-api-schema",
),
path(
f"{settings.API_VERSION}//swagger/",
SpectacularSwaggerView.as_view(url_name="client-api-schema"),
name="swagger-ui-schema",
),
re_path(
f"{settings.API_VERSION}//redoc/",
SpectacularRedocView.as_view(url_name="client-api-schema"),
name="redoc-schema",
),
]
+17
View File
@@ -0,0 +1,17 @@
"""
WSGI config for the find project.
It exposes the WSGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/3.1/howto/deployment/wsgi/
"""
import os
from configurations.wsgi import get_wsgi_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "find.settings")
os.environ.setdefault("DJANGO_CONFIGURATION", "Development")
application = get_wsgi_application()
+15
View File
@@ -0,0 +1,15 @@
#!/usr/bin/env python
"""
find's sandbox management script.
"""
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "find.settings")
os.environ.setdefault("DJANGO_CONFIGURATION", "Development")
from configurations.management import execute_from_command_line
execute_from_command_line(sys.argv)
+133
View File
@@ -0,0 +1,133 @@
#
# find package
#
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
[project]
name = "find"
version = "0.0.1"
authors = [{ "name" = "DINUM", "email" = "dev@mail.numerique.gouv.fr" }]
classifiers = [
"Development Status :: 5 - Production/Stable",
"Framework :: Django",
"Framework :: Django :: 5",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Natural Language :: English",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
]
description = "An application to print markdown to pdf from a set of managed templates."
keywords = ["Django", "Contacts", "Templates", "RBAC"]
license = { file = "LICENSE" }
readme = "README.md"
requires-python = ">=3.10"
dependencies = [
"celery[redis]==5.5.3",
"django-configurations==2.5.1",
"django-cors-headers==4.7.0",
"redis==5.2.1",
"django-redis==6.0.0",
"django==5.2.6",
"django-lasuite[all]==0.0.14",
"djangorestframework==3.16.0",
"drf_spectacular==0.28.0",
"dockerflow==2024.4.2",
"factory_boy==3.3.1",
"gunicorn==23.0.0",
"mozilla-django-oidc==4.0.1",
"psycopg[binary]==3.2.9",
"pydantic==2.10.5",
"pyjwt==2.10.1",
"requests==2.32.4",
"sentry-sdk==2.32.0",
"url-normalize==1.4.3",
"opensearch-py==2.8.0",
"whitenoise==6.8.2",
]
[project.urls]
"Bug Tracker" = "https://github.com/suitenumerique/find/issues/new"
"Changelog" = "https://github.com/suitenumerique/find/blob/main/CHANGELOG.md"
"Homepage" = "https://github.com/suitenumerique/find"
"Repository" = "https://github.com/suitenumerique/find"
[project.optional-dependencies]
dev = [
"django-extensions==4.1",
"drf-spectacular-sidecar==2025.7.1",
"faker==33.3.0",
"ipdb==0.13.13",
"ipython==8.31.0",
"pyfakefs==5.9.1",
"pylint-django==2.6.1",
"pylint==3.3.7",
"pytest-cov==6.2.1",
"pytest-django==4.11.1",
"pytest==8.4.1",
"pytest-icdiff==0.9",
"pytest-xdist==3.8.0",
"responses==0.25.7",
"ruff==0.12.2",
"types-requests==2.32.4.20250611",
]
[tool.setuptools]
packages = { find = { where = ["."], exclude = ["tests"] } }
zip-safe = true
[tool.distutils.bdist_wheel]
universal = true
[tool.ruff]
exclude = [
".git",
".venv",
"build",
"venv",
"__pycache__",
"*/migrations/*",
]
line-length = 88
[tool.ruff.lint]
select = [
"B", # flake8-bugbear
"BLE", # flake8-blind-except
"C4", # flake8-comprehensions
"DJ", # flake8-django
"I", # isort
"PLC", # pylint-convention
"PLE", # pylint-error
"PLR", # pylint-refactoring
"PLW", # pylint-warning
"RUF100", # Ruff unused-noqa
"RUF200", # Ruff check pyproject.toml
"S", # flake8-bandit
"SLF", # flake8-self
"T20", # flake8-print
]
ignore= ["DJ001", "PLR2004"]
[tool.ruff.lint.isort]
section-order = ["future","standard-library","django","third-party","find","first-party","local-folder"]
sections = { find=["core"], django=["django"] }
[tool.ruff.lint.per-file-ignores]
"**/tests/*" = ["S", "SLF"]
[tool.pytest.ini_options]
addopts = [
"-v",
"--cov-report",
"term-missing",
# Allow test files to have the same name in different directories.
"--import-mode=importlib",
]
python_files = [
"test_*.py",
"tests.py",
]
+7
View File
@@ -0,0 +1,7 @@
#!/usr/bin/env python
"""Setup file for the find module. All configuration stands in the setup.cfg file."""
# coding: utf-8
from setuptools import setup # pylint: disable=import-error
setup()
+77
View File
@@ -0,0 +1,77 @@
djangoSuperUserEmail: ENC[AES256_GCM,data:7b1xfYmr1g0RlBmsHBRA39ZPV/6+1DrtHQ==,iv:/GW7oLxPTZYmRWVPvyAQMoZl1owHM4Fo0XAOtyEh2rA=,tag:DaqoW+dglyAOXMm5+mrDfA==,type:str]
djangoSuperUserPass: ENC[AES256_GCM,data:RQgX,iv:q3CdfmwGfHSTjLXTimDk/1MyoFLviRuwmZa2E7GUzhY=,tag:HCtdtqgSxdJIHFhI8xpegQ==,type:str]
djangoSecretKey: ENC[AES256_GCM,data:mtJCf6mKfj/fJkg4wmfIvvU1vkUEF77BI8TUFikp/M3nPveDXhKmy3Cw3cXFpOYiFZ0=,iv:qwPRKsPS1Jhylj5asbmknXm1xOX3nfp9iccuorUrcj0=,tag:ENVfAt4i3PttoqD8+Kc4wQ==,type:str]
opensearch:
password: ENC[AES256_GCM,data:y4iWSEwW5u1W3vPx,iv:1A9qIUELbqSIdKL+CXn1CshR3g1jMJVgw+VsJHZU9Q8=,tag:6S6RKSC5mnUZP3epcpd5jQ==,type:str]
logstash:
password: ENC[AES256_GCM,data:gLHdKHExMEMcvvDz,iv:Ib7YKW0mjoxj4E9JUfjyxB1kW3+DNe0FuYVS1n6AzME=,tag:WPTgVtsbx0hIATGFpoxGxg==,type:str]
opensearchDashboards:
password: ENC[AES256_GCM,data:Xt4RjZR60rEthVmC,iv:ppWHKKFzL+WhDhZgG0ie7kOc8icB757micF5JW7HDOY=,tag:XwRA7CjwA1jig5EmIwZROA==,type:str]
oidc:
clientId: ENC[AES256_GCM,data:wndPCbysbWDybdHglcG+wkMWk1rrD40hKqFxct9T3TLEGOk/,iv:RH1OdBX1GYIT90sSq0AGz49fFi6dL0m49Pegs6Ko9tQ=,tag:/tKytQwoZkBX1Tf96gAjIA==,type:str]
clientSecret: ENC[AES256_GCM,data:MUJ0wsg+LC2QZ1jZ0Twd3FS3dQevmJq9/97qVI3ARHuJIVlQz0Qah4vE7/iR+sn7ME2o1s1AzV4c1Yx/F3nHBg==,iv:LvinICSzF/8EvrHZD4Jp6lt7g3yxSOEgVHPrc3SShjo=,tag:yvkyyBXmhEkmGL7jZevUCA==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age15fyxdwmg5mvldtqqus87xspuws2u0cpvwheehrtvkexj4tnsqqysw6re2x
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwdFdnd1RnMkE1blN2YWhK
Q0xYYnpTbk0rVkxEQ2w4a0hrcjZ4TURxQ1ZnCmNyakFSMmU2S0c0cWhLRFpSckk4
bU1kOXlZeFU1S05DN1RCYWpaMTJhM1EKLS0tIHZWREtDVFU3U1djYWsrNzRxeTBW
TmRSbzlRUGpMZHF0NFdDTVpwQURmM1kKUJbmN8UE/7zhVbK5QRzfdCSTiHM3OWai
4ENxdmo1A58De7GE+kb4yMIym6aFa+LnQ0qCVkuZ+Y6i4Ipu/9s3DQ==
-----END AGE ENCRYPTED FILE-----
- recipient: age16hnlml8yv4ynwy0seer57g8qww075crd0g7nsundz3pj4wk7m3vqftszg7
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsd0ZMZlJscC9LQktDVHVr
UFEzSEI5QUk1VFA5YU93Y1RBNCtpQzg3RDNvCjdKOStUcmsxamNpLzE4VWs2WWNF
MTA3eGNmMU9zSndMZVVvYy9ZMVRuNzgKLS0tICtuT1ZlbVJQMTZNVWRJZllnUXZ1
eDIxczdXOTEza0lYYzZRWERaVUVKdVEKLICxvqmzEOTqDBgcaTMU8XlbpSQPiH6u
gHwTXezsaOwjzdviyAeyhNPq3N1vauM2L8NW/0sWSq9qFZAi92uw6A==
-----END AGE ENCRYPTED FILE-----
- recipient: age1plkp8td6zzfcavjusmsfrlk54t9vn8jjxm8zaz7cmnr7kzl2nfnsd54hwg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBYcVF3UURjNzFPbzJqRmZu
RGd0QWptWXRQZXVDWVJ4bHA5MllWOEU4N25NCldJa1ZNb0xURFcxMERjbFVMZk01
WHpKY3BWai83SmE4bDl2R2pKR1NGb1UKLS0tIFpJZHpRRUN1aGhTUytTSVA1K1U3
bUJNZUtsSHFZMnRFaUZtZEZmSzN5RncK2hBeMQHxCWCabS0wWV6LC3Shhj5g4LKc
J44oWmIx/Aaz3Jnf73E4aE8/wW5L593oqP3wzT4dermd9OKcEvU2gQ==
-----END AGE ENCRYPTED FILE-----
- recipient: age12g6f5fse25tgrwweleh4jls3qs52hey2edh759smulwmk5lnzadslu2cp3
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBvTlJuQ0pndFFpQ1JxZ3hR
dVZjVDVVd09JcmhQVWpDYnpxSjkrM25VTFZnCkk2SU9QVUJQQWdyUjVPWFRNbU44
RnlKcEJ6RHJyU2lMOTNwbkE1QnBzK2sKLS0tIC80eXNIVTZQa0JjbnpZQkJkRlJq
a1NGa2pNOE9reG84QlMzaDRhQmhjZDgKFihYgODIysk5jPWKSvTckayDx0jKRtKc
FUmsLQC87bJdY6/HGnRSB3xI7ZJYlX12G2lfshOY0ZpQtkRgLtzOBg==
-----END AGE ENCRYPTED FILE-----
- recipient: age1hnhuzj96ktkhpyygvmz0x9h8mfvssz7ss6emmukags644mdhf4msajk93r
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRcmlzOG44S0VZUVJvTXZ0
aDEybUhBY3kzWG5TMDRpZUtiai9DTE1sU0RFCklKV2Y0eUZOUHhRSEZPamQxMXNp
QmxsZEppMWkrTzRRWnJLTHByNVRuZ1EKLS0tIHQ0bThtaUl2eXE4TlZWck5CbUh4
ZXZnKyt0a0hyTEZRbGhvNGkvTEFsem8Kc0SgSfI8U+nfB9ZqID/dh0IYWl4dBhl3
kpHAn9OiaKXZs9dt6Um2H/ACOOKjRlRYoaybEAeIP5d5RLV8dDVcOw==
-----END AGE ENCRYPTED FILE-----
- recipient: age19drtvac5c0e6t4da24lw8jq9k3hrcwyd3pdj78kxy9kllalxx3dsz9ttx4
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBHelFUdlVrTkIvdnlETDli
dGJHc3pTRjV1VWRNQWhTWkVVNkxIb3QzNTJjCm4zdWVFbmo0Y1IzdzhYYjBuM3B5
QlhHaHVqaXhMUGRPU1lQZUN6RUJ5SDgKLS0tIGd6ZWVEc2pWcDNGSE5OMWhBZm12
b0tsOEExQ1gyTEx3VVR3c1MrRkw3QkUK7pLI3R0p+07URlC3nTsXEnxkbX3KuIxv
JjsDoCnG+p45moQYzSh6tokfIxJKQDt4EJCuQCgHU5twb17aWt9Q7A==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2025-01-13T13:30:46Z"
mac: ENC[AES256_GCM,data:B0OjEp+CaCGgJ8FPHoeAgTnwAwBUQY2ArNeqKU0GNFr38lHfmnvccWvQcHyGxlGIwpyTOnEOKuSLq8EowxeLUTCTai0nM8jaq292pwb2TjZ2U57Y02RhnOiDsHjkomCVdSephaXiAZCGYrlNWWRUALcYTy2AryYY22gB6akb7OU=,iv:u7NCh2lythDSQUaRtfxAg7scFRijqlOczLRGlUgIzB0=,tag:uoKgZYaUj0xQfu4C+qqEsw==,type:str]
pgp: []
unencrypted_suffix: _unencrypted
version: 3.9.3
@@ -0,0 +1,96 @@
image:
repository: localhost:5001/find-backend
pullPolicy: Always
tag: "latest"
backend:
replicas: 1
envVars:
DJANGO_CSRF_TRUSTED_ORIGINS: https://find.127.0.0.1.nip.io,http://find.127.0.0.1.nip.io
DJANGO_CONFIGURATION: Production
DJANGO_ALLOWED_HOSTS: "*"
DJANGO_SECRET_KEY:
secretKeyRef:
name: find-backend
key: DJANGO_SECRET_KEY
DJANGO_SETTINGS_MODULE: find.settings
DJANGO_SUPERUSER_EMAIL:
secretKeyRef:
name: find-backend
key: DJANGO_SUPERUSER_EMAIL
DJANGO_SUPERUSER_PASSWORD:
secretKeyRef:
name: find-backend
key: DJANGO_SUPERUSER_PASSWORD
DJANGO_EMAIL_HOST: "mailcatcher"
DJANGO_EMAIL_PORT: 1025
DJANGO_EMAIL_USE_SSL: False
OIDC_OP_JWKS_ENDPOINT: https://fca.integ01.dev-agentconnect.fr/api/v2/jwks
OIDC_OP_AUTHORIZATION_ENDPOINT: https://fca.integ01.dev-agentconnect.fr/api/v2/authorize
OIDC_OP_TOKEN_ENDPOINT: https://fca.integ01.dev-agentconnect.fr/api/v2/token
OIDC_OP_USER_ENDPOINT: https://fca.integ01.dev-agentconnect.fr/api/v2/userinfo
OIDC_OP_LOGOUT_ENDPOINT: https://fca.integ01.dev-agentconnect.fr/api/v2/session/end
OIDC_RP_CLIENT_ID:
secretKeyRef:
name: find-backend
key: OIDC_RP_CLIENT_ID
OIDC_RP_CLIENT_SECRET:
secretKeyRef:
name: find-backend
key: OIDC_RP_CLIENT_SECRET
OIDC_RP_SIGN_ALGO: RS256
OIDC_RP_SCOPES: "openid email"
OIDC_REDIRECT_ALLOWED_HOSTS: https://find.127.0.0.1.nip.io
OIDC_AUTH_REQUEST_EXTRA_PARAMS: "{'acr_values': 'eidas1'}"
LOGIN_REDIRECT_URL: https://find.127.0.0.1.nip.io
LOGIN_REDIRECT_URL_FAILURE: https://find.127.0.0.1.nip.io
LOGOUT_REDIRECT_URL: https://find.127.0.0.1.nip.io
DB_HOST: postgres-postgresql
DB_NAME: find
DB_USER: dinum
DB_PASSWORD: pass
DB_PORT: 5432
POSTGRES_DB: find
POSTGRES_USER: dinum
POSTGRES_PASSWORD: pass
REDIS_URL: redis://default:pass@redis-master:6379/1
STORAGES_STATICFILES_BACKEND: django.contrib.staticfiles.storage.StaticFilesStorage
OPENSEARCH_HOST: opensearch
OPENSEARCH_USE_SSL: False
OPENSEARCH_PORT: 9200
OPENSEARCH_PASSWORD:
secretKeyRef:
name: find-backend
key: OPENSEARCH_PASSWORD
migrate:
command:
- "/bin/bash"
- "-c"
- |
python manage.py migrate --no-input
restartPolicy: Never
command:
- "gunicorn"
- "-c"
- "/usr/local/etc/gunicorn/find.py"
- "find.wsgi:application"
- "--reload"
createsuperuser:
enabled: true
command:
- "/bin/bash"
- "-c"
- |
python manage.py shell -c "from core.models import User; not User.objects.filter(username='admin').exists() and User.objects.create_superuser('admin', 'admin@example.com', 'admin')"
restartPolicy: Never
ingress:
enabled: true
host: find.127.0.0.1.nip.io
ingressAdmin:
enabled: true
host: find.127.0.0.1.nip.io
@@ -0,0 +1,9 @@
security:
existingSecret: opensearch
dashboards:
enabled: true
ingress:
enabled: true
tls: true
selfSigned: true
hostname: find-dashboard.127.0.0.1.nip.io
+5
View File
@@ -0,0 +1,5 @@
apiVersion: v2
name: extra
description: A Helm chart to add some manifests to find
type: application
version: 0.1.0
@@ -0,0 +1,12 @@
apiVersion: v1
kind: Secret
metadata:
name: find-backend
namespace: {{ .Release.Namespace | quote }}
stringData:
DJANGO_SUPERUSER_EMAIL: {{ .Values.djangoSuperUserEmail }}
DJANGO_SUPERUSER_PASSWORD: {{ .Values.djangoSuperUserPass }}
DJANGO_SECRET_KEY: {{ .Values.djangoSecretKey }}
OIDC_RP_CLIENT_ID: {{ .Values.oidc.clientId }}
OIDC_RP_CLIENT_SECRET: {{ .Values.oidc.clientSecret }}
OPENSEARCH_PASSWORD: {{ .Values.opensearch.password }}
@@ -0,0 +1,9 @@
apiVersion: v1
kind: Secret
metadata:
name: opensearch
namespace: {{ .Release.Namespace | quote }}
stringData:
opensearch-password: {{ .Values.opensearch.password }}
opensearch-dashboards-password: {{ .Values.opensearchDashboards.password }}
logstash-password: {{ .Values.opensearch.logstash.password }}
+4
View File
@@ -0,0 +1,4 @@
apiVersion: v2
type: application
name: find
version: 0.0.2
+84
View File
@@ -0,0 +1,84 @@
# Find helm chart
## Parameters
### General configuration
| Name | Description | Value |
| ------------------------------------------ | ---------------------------------------------------- | ---------------------- |
| `image.repository` | Repository to use to pull find's container image | `lasuite/find-backend` |
| `image.tag` | find's container tag | `latest` |
| `image.pullPolicy` | Container image pull policy | `IfNotPresent` |
| `image.credentials.username` | Username for container registry authentication | |
| `image.credentials.password` | Password for container registry authentication | |
| `image.credentials.registry` | Registry url for which the credentials are specified | |
| `image.credentials.name` | Name of the generated secret for imagePullSecrets | |
| `nameOverride` | Override the chart name | `""` |
| `fullnameOverride` | Override the full application name | `""` |
| `ingress.enabled` | whether to enable the Ingress or not | `false` |
| `ingress.className` | IngressClass to use for the Ingress | `nil` |
| `ingress.host` | Host for the Ingress | `find.example.com` |
| `ingress.path` | Path to use for the Ingress | `/` |
| `ingress.hosts` | Additional host to configure for the Ingress | `[]` |
| `ingress.tls.enabled` | Whether to enable TLS for the Ingress | `true` |
| `ingress.tls.additional[].secretName` | Secret name for additional TLS config | |
| `ingress.tls.additional[].hosts[]` | Hosts for additional TLS config | |
| `ingress.customBackends` | Add custom backends to ingress | `[]` |
| `ingressAdmin.enabled` | whether to enable the Ingress or not | `false` |
| `ingressAdmin.className` | IngressClass to use for the Ingress | `nil` |
| `ingressAdmin.host` | Host for the Ingress | `find.example.com` |
| `ingressAdmin.path` | Path to use for the Ingress | `/admin` |
| `ingressAdmin.hosts` | Additional host to configure for the Ingress | `[]` |
| `ingressAdmin.tls.enabled` | Whether to enable TLS for the Ingress | `true` |
| `ingressAdmin.tls.additional[].secretName` | Secret name for additional TLS config | |
| `ingressAdmin.tls.additional[].hosts[]` | Hosts for additional TLS config | |
### backend
| Name | Description | Value |
| ----------------------------------------------------- | ---------------------------------------------------------------------------------- | ----------------------------------------------- |
| `backend.command` | Override the backend container command | `[]` |
| `backend.args` | Override the backend container args | `[]` |
| `backend.replicas` | Amount of backend replicas | `3` |
| `backend.shareProcessNamespace` | Enable share process namespace between containers | `false` |
| `backend.sidecars` | Add sidecars containers to backend deployment | `[]` |
| `backend.migrateJobAnnotations` | Annotations for the migrate job | `{}` |
| `backend.securityContext` | Configure backend Pod security context | `nil` |
| `backend.envVars` | Configure backend container environment variables | `undefined` |
| `backend.envVars.BY_VALUE` | Example environment variable by setting value directly | |
| `backend.envVars.FROM_CONFIGMAP.configMapKeyRef.name` | Name of a ConfigMap when configuring env vars from a ConfigMap | |
| `backend.envVars.FROM_CONFIGMAP.configMapKeyRef.key` | Key within a ConfigMap when configuring env vars from a ConfigMap | |
| `backend.envVars.FROM_SECRET.secretKeyRef.name` | Name of a Secret when configuring env vars from a Secret | |
| `backend.envVars.FROM_SECRET.secretKeyRef.key` | Key within a Secret when configuring env vars from a Secret | |
| `backend.podAnnotations` | Annotations to add to the backend Pod | `{}` |
| `backend.service.type` | backend Service type | `ClusterIP` |
| `backend.service.port` | backend Service listening port | `80` |
| `backend.service.targetPort` | backend container listening port | `8000` |
| `backend.service.annotations` | Annotations to add to the backend Service | `{}` |
| `backend.migrate.command` | backend migrate command | `["python","manage.py","migrate","--no-input"]` |
| `backend.migrate.restartPolicy` | backend migrate job restart policy | `Never` |
| `backend.probes.liveness.path` | Configure path for backend HTTP liveness probe | `/__heartbeat__` |
| `backend.probes.liveness.targetPort` | Configure port for backend HTTP liveness probe | `undefined` |
| `backend.probes.liveness.initialDelaySeconds` | Configure initial delay for backend liveness probe | `10` |
| `backend.probes.liveness.initialDelaySeconds` | Configure timeout for backend liveness probe | `10` |
| `backend.probes.startup.path` | Configure path for backend HTTP startup probe | `undefined` |
| `backend.probes.startup.targetPort` | Configure port for backend HTTP startup probe | `undefined` |
| `backend.probes.startup.initialDelaySeconds` | Configure initial delay for backend startup probe | `undefined` |
| `backend.probes.startup.initialDelaySeconds` | Configure timeout for backend startup probe | `undefined` |
| `backend.probes.readiness.path` | Configure path for backend HTTP readiness probe | `/__lbheartbeat__` |
| `backend.probes.readiness.targetPort` | Configure port for backend HTTP readiness probe | `undefined` |
| `backend.probes.readiness.initialDelaySeconds` | Configure initial delay for backend readiness probe | `10` |
| `backend.probes.readiness.initialDelaySeconds` | Configure timeout for backend readiness probe | `10` |
| `backend.resources` | Resource requirements for the backend container | `{}` |
| `backend.nodeSelector` | Node selector for the backend Pod | `{}` |
| `backend.tolerations` | Tolerations for the backend Pod | `[]` |
| `backend.affinity` | Affinity for the backend Pod | `{}` |
| `backend.persistence` | Additional volumes to create and mount on the backend. Used for debugging purposes | `{}` |
| `backend.persistence.volume-name.size` | Size of the additional volume | |
| `backend.persistence.volume-name.type` | Type of the additional volume, persistentVolumeClaim or emptyDir | |
| `backend.persistence.volume-name.mountPath` | Path where the volume should be mounted to | |
| `backend.extraVolumeMounts` | Additional volumes to mount on the backend. | `[]` |
| `backend.extraVolumes` | Additional volumes to mount on the backend. | `[]` |
| `backend.createsuperuser.enabled` | Whether to enable the create superuser command | `false` |
| `backend.createsuperuser.command` | Command to run to create superuser | `["python","manage.py","createsuperuser"]` |
| `backend.createsuperuser.restartPolicy` | Restart policy for the createsuperuser job | `Never` |
+10
View File
@@ -0,0 +1,10 @@
#!/bin/bash
docker image ls | grep readme-generator-for-helm
if [ "$?" -ne "0" ]; then
git clone https://github.com/bitnami/readme-generator-for-helm.git /tmp/readme-generator-for-helm
cd /tmp/readme-generator-for-helm
docker build -t readme-generator-for-helm:latest .
cd $(dirname -- "${BASH_SOURCE[0]}")
fi
docker run --rm -it -v ./values.yaml:/app/values.yaml -v ./README.md:/app/README.md readme-generator-for-helm:latest readme-generator -v values.yaml -r README.md
+182
View File
@@ -0,0 +1,182 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "find.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "find.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "find.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create version.
*/}}
{{- define "find.version" -}}
{{- printf "%s-%s" .Chart.Version .Chart.AppVersion | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
find.labels
*/}}
{{- define "find.labels" -}}
helm.sh/chart: {{ include "find.chart" . }}
{{ include "find.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "find.selectorLabels" -}}
app.kubernetes.io/name: {{ include "find.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
transform dictionnary of environment variables
Usage : {{ include "find.env.transformDict" .Values.envVars }}
Example:
envVars:
# Using simple strings as env vars
ENV_VAR_NAME: "envVar value"
# Using a value from a configMap
ENV_VAR_FROM_CM:
configMapKeyRef:
name: cm-name
key: "key_in_cm"
# Using a value from a secret
ENV_VAR_FROM_SECRET:
secretKeyRef:
name: secret-name
key: "key_in_secret"
*/}}
{{- define "find.env.transformDict" -}}
{{- range $key, $value := . }}
- name: {{ $key | quote }}
{{- if $value | kindIs "map" }}
valueFrom: {{ $value | toYaml | nindent 4 }}
{{- else }}
value: {{ $value | quote }}
{{- end }}
{{- end }}
{{- end }}
{{/*
find env vars
*/}}
{{- define "find.common.env" -}}
{{- $topLevelScope := index . 0 -}}
{{- $workerScope := index . 1 -}}
{{- include "find.env.transformDict" $workerScope.envVars -}}
{{- end }}
{{/*
Common labels
Requires array with top level scope and component name
*/}}
{{- define "find.common.labels" -}}
{{- $topLevelScope := index . 0 -}}
{{- $component := index . 1 -}}
{{- include "find.labels" $topLevelScope }}
app.kubernetes.io/component: {{ $component }}
{{- end }}
{{/*
Common selector labels
Requires array with top level scope and component name
*/}}
{{- define "find.common.selectorLabels" -}}
{{- $topLevelScope := index . 0 -}}
{{- $component := index . 1 -}}
{{- include "find.selectorLabels" $topLevelScope }}
app.kubernetes.io/component: {{ $component }}
{{- end }}
{{- define "find.probes.abstract" -}}
{{- if .exec -}}
exec:
{{- toYaml .exec | nindent 2 }}
{{- else if .tcpSocket -}}
tcpSocket:
{{- toYaml .tcpSocket | nindent 2 }}
{{- else -}}
httpGet:
path: {{ .path }}
port: {{ .targetPort }}
{{- end }}
initialDelaySeconds: {{ .initialDelaySeconds | eq nil | ternary 0 .initialDelaySeconds }}
timeoutSeconds: {{ .timeoutSeconds | eq nil | ternary 1 .timeoutSeconds }}
{{- end }}
{{/*
Full name for the backend
Requires top level scope
*/}}
{{- define "find.backend.fullname" -}}
{{ include "find.fullname" . }}-backend
{{- end }}
{{/*
Full name for the webrtc
Requires top level scope
*/}}
{{- define "find.webrtc.fullname" -}}
{{ include "find.fullname" . }}-webrtc
{{- end }}
{{/*
Usage : {{ include "find.secret.dockerconfigjson.name" (dict "fullname" (include "find.fullname" .) "imageCredentials" .Values.path.to.the.image1) }}
*/}}
{{- define "find.secret.dockerconfigjson.name" }}
{{- if (default (dict) .imageCredentials).name }}{{ .imageCredentials.name }}{{ else }}{{ .fullname | trunc 63 | trimSuffix "-" }}-dockerconfig{{ end -}}
{{- end }}
{{/*
Usage : {{ include "find.secret.dockerconfigjson" (dict "fullname" (include "find.fullname" .) "imageCredentials" .Values.path.to.the.image1) }}
*/}}
{{- define "find.secret.dockerconfigjson" }}
{{- if .imageCredentials -}}
apiVersion: v1
kind: Secret
metadata:
name: {{ template "find.secret.dockerconfigjson.name" (dict "fullname" .fullname "imageCredentials" .imageCredentials) }}
annotations:
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-weight": "-5"
"helm.sh/hook-delete-policy": before-hook-creation
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: {{ template "find.secret.dockerconfigjson.data" .imageCredentials }}
{{- end -}}
{{- end }}
@@ -0,0 +1,136 @@
{{- $envVars := include "find.common.env" (list . .Values.backend) -}}
{{- $fullName := include "find.backend.fullname" . -}}
{{- $component := "backend" -}}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ $fullName }}
namespace: {{ .Release.Namespace | quote }}
labels:
{{- include "find.common.labels" (list . $component) | nindent 4 }}
spec:
replicas: {{ .Values.backend.replicas }}
selector:
matchLabels:
{{- include "find.common.selectorLabels" (list . $component) | nindent 6 }}
template:
metadata:
annotations:
{{- with .Values.backend.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "find.common.selectorLabels" (list . $component) | nindent 8 }}
spec:
{{- if $.Values.image.credentials }}
imagePullSecrets:
- name: {{ include "find.secret.dockerconfigjson.name" (dict "fullname" (include "find.fullname" .) "imageCredentials" $.Values.image.credentials) }}
{{- end}}
shareProcessNamespace: {{ .Values.backend.shareProcessNamespace }}
containers:
{{- with .Values.backend.sidecars }}
{{- toYaml . | nindent 8 }}
{{- end }}
- name: {{ .Chart.Name }}
image: "{{ (.Values.backend.image | default dict).repository | default .Values.image.repository }}:{{ (.Values.backend.image | default dict).tag | default .Values.image.tag }}"
imagePullPolicy: {{ (.Values.backend.image | default dict).pullPolicy | default .Values.image.pullPolicy }}
{{- with .Values.backend.command }}
command:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.backend.args }}
args:
{{- toYaml . | nindent 12 }}
{{- end }}
env:
{{- if $envVars}}
{{- $envVars | indent 12 }}
{{- end }}
{{- with .Values.backend.securityContext }}
securityContext:
{{- toYaml . | nindent 12 }}
{{- end }}
ports:
- name: http
containerPort: {{ .Values.backend.service.targetPort }}
protocol: TCP
{{- if .Values.backend.probes.liveness }}
livenessProbe:
{{- include "find.probes.abstract" (merge .Values.backend.probes.liveness (dict "targetPort" .Values.backend.service.targetPort )) | nindent 12 }}
{{- end }}
{{- if .Values.backend.probes.readiness }}
readinessProbe:
{{- include "find.probes.abstract" (merge .Values.backend.probes.readiness (dict "targetPort" .Values.backend.service.targetPort )) | nindent 12 }}
{{- end }}
{{- if .Values.backend.probes.startup }}
startupProbe:
{{- include "find.probes.abstract" (merge .Values.backend.probes.startup (dict "targetPort" .Values.backend.service.targetPort )) | nindent 12 }}
{{- end }}
{{- with .Values.backend.resources }}
resources:
{{- toYaml . | nindent 12 }}
{{- end }}
volumeMounts:
{{- range $index, $value := .Values.mountFiles }}
- name: "files-{{ $index }}"
mountPath: {{ $value.path }}
subPath: content
{{- end }}
{{- range $name, $volume := .Values.backend.persistence }}
- name: "{{ $name }}"
mountPath: "{{ $volume.mountPath }}"
{{- end }}
{{- range .Values.backend.extraVolumeMounts }}
- name: {{ .name }}
mountPath: {{ .mountPath }}
subPath: {{ .subPath | default "" }}
readOnly: {{ .readOnly }}
{{- end }}
{{- with .Values.backend.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.backend.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.backend.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
volumes:
{{- range $index, $value := .Values.mountFiles }}
- name: "files-{{ $index }}"
configMap:
name: "{{ include "find.fullname" $ }}-files-{{ $index }}"
{{- end }}
{{- range $name, $volume := .Values.backend.persistence }}
- name: "{{ $name }}"
{{- if eq $volume.type "emptyDir" }}
emptyDir: {}
{{- else }}
persistentVolumeClaim:
claimName: "{{ $fullName }}-{{ $name }}"
{{- end }}
{{- end }}
{{- range .Values.backend.extraVolumes }}
- name: {{ .name }}
{{- if .existingClaim }}
persistentVolumeClaim:
claimName: {{ .existingClaim }}
{{- else if .hostPath }}
hostPath:
{{ toYaml .hostPath | nindent 12 }}
{{- else if .csi }}
csi:
{{- toYaml .csi | nindent 12 }}
{{- else if .configMap }}
configMap:
{{- toYaml .configMap | nindent 12 }}
{{- else if .emptyDir }}
emptyDir:
{{- toYaml .emptyDir | nindent 12 }}
{{- else }}
emptyDir: {}
{{- end }}
{{- end }}
+122
View File
@@ -0,0 +1,122 @@
{{- $envVars := include "find.common.env" (list . .Values.backend) -}}
{{- $fullName := include "find.backend.fullname" . -}}
{{- $component := "backend" -}}
apiVersion: batch/v1
kind: Job
metadata:
name: {{ $fullName }}-migrate-{{ include "find.version" .}}
namespace: {{ .Release.Namespace | quote }}
{{- with .Values.backend.migrateJobAnnotations }}
annotations:
argocd.argoproj.io/sync-options: Replace=true,Force=true
{{- toYaml . | nindent 4 }}
{{- end }}
labels:
{{- include "find.common.labels" (list . $component) | nindent 4 }}
spec:
template:
metadata:
annotations:
{{- with .Values.backend.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "find.common.selectorLabels" (list . $component) | nindent 8 }}
spec:
{{- if $.Values.image.credentials }}
imagePullSecrets:
- name: {{ include "find.secret.dockerconfigjson.name" (dict "fullname" (include "find.fullname" .) "imageCredentials" $.Values.image.credentials) }}
{{- end}}
shareProcessNamespace: {{ .Values.backend.shareProcessNamespace }}
containers:
{{- with .Values.backend.sidecars }}
{{- toYaml . | nindent 8 }}
{{- end }}
- name: {{ .Chart.Name }}
image: "{{ (.Values.backend.image | default dict).repository | default .Values.image.repository }}:{{ (.Values.backend.image | default dict).tag | default .Values.image.tag }}"
imagePullPolicy: {{ (.Values.backend.image | default dict).pullPolicy | default .Values.image.pullPolicy }}
{{- with .Values.backend.migrate.command }}
command:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.backend.args }}
args:
{{- toYaml . | nindent 12 }}
{{- end }}
env:
{{- if $envVars}}
{{- $envVars | indent 12 }}
{{- end }}
{{- with .Values.backend.securityContext }}
securityContext:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.backend.resources }}
resources:
{{- toYaml . | nindent 12 }}
{{- end }}
volumeMounts:
{{- range $index, $value := .Values.mountFiles }}
- name: "files-{{ $index }}"
mountPath: {{ $value.path }}
subPath: content
{{- end }}
{{- range $name, $volume := .Values.backend.persistence }}
- name: "{{ $name }}"
mountPath: "{{ $volume.mountPath }}"
{{- end }}
{{- range .Values.backend.extraVolumeMounts }}
- name: {{ .name }}
mountPath: {{ .mountPath }}
subPath: {{ .subPath | default "" }}
readOnly: {{ .readOnly }}
{{- end }}
{{- with .Values.backend.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.backend.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.backend.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
restartPolicy: {{ .Values.backend.migrate.restartPolicy }}
volumes:
{{- range $index, $value := .Values.mountFiles }}
- name: "files-{{ $index }}"
configMap:
name: "{{ include "find.fullname" $ }}-files-{{ $index }}"
{{- end }}
{{- range $name, $volume := .Values.backend.persistence }}
- name: "{{ $name }}"
{{- if eq $volume.type "emptyDir" }}
emptyDir: {}
{{- else }}
persistentVolumeClaim:
claimName: "{{ $fullName }}-{{ $name }}"
{{- end }}
{{- end }}
{{- range .Values.backend.extraVolumes }}
- name: {{ .name }}
{{- if .existingClaim }}
persistentVolumeClaim:
claimName: {{ .existingClaim }}
{{- else if .hostPath }}
hostPath:
{{ toYaml .hostPath | nindent 12 }}
{{- else if .csi }}
csi:
{{- toYaml .csi | nindent 12 }}
{{- else if .configMap }}
configMap:
{{- toYaml .configMap | nindent 12 }}
{{- else if .emptyDir }}
emptyDir:
{{- toYaml .emptyDir | nindent 12 }}
{{- else }}
emptyDir: {}
{{- end }}
{{- end }}
@@ -0,0 +1,124 @@
{{- $envVars := include "find.common.env" (list . .Values.backend) -}}
{{- $fullName := include "find.backend.fullname" . -}}
{{- $component := "backend" -}}
{{- if .Values.backend.createsuperuser.enabled }}
apiVersion: batch/v1
kind: Job
metadata:
name: {{ $fullName }}-createsuperuser
namespace: {{ .Release.Namespace | quote }}
{{- with .Values.backend.migrateJobAnnotations }}
annotations:
argocd.argoproj.io/sync-options: Replace=true,Force=true
{{- toYaml . | nindent 4 }}
{{- end }}
labels:
{{- include "find.common.labels" (list . $component) | nindent 4 }}
spec:
template:
metadata:
annotations:
{{- with .Values.backend.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "find.common.selectorLabels" (list . $component) | nindent 8 }}
spec:
{{- if $.Values.image.credentials }}
imagePullSecrets:
- name: {{ include "find.secret.dockerconfigjson.name" (dict "fullname" (include "find.fullname" .) "imageCredentials" $.Values.image.credentials) }}
{{- end}}
shareProcessNamespace: {{ .Values.backend.shareProcessNamespace }}
containers:
{{- with .Values.backend.sidecars }}
{{- toYaml . | nindent 8 }}
{{- end }}
- name: {{ .Chart.Name }}
image: "{{ (.Values.backend.image | default dict).repository | default .Values.image.repository }}:{{ (.Values.backend.image | default dict).tag | default .Values.image.tag }}"
imagePullPolicy: {{ (.Values.backend.image | default dict).pullPolicy | default .Values.image.pullPolicy }}
{{- with .Values.backend.createsuperuser.command }}
command:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.backend.args }}
args:
{{- toYaml . | nindent 12 }}
{{- end }}
env:
{{- if $envVars}}
{{- $envVars | indent 12 }}
{{- end }}
{{- with .Values.backend.securityContext }}
securityContext:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.backend.resources }}
resources:
{{- toYaml . | nindent 12 }}
{{- end }}
volumeMounts:
{{- range $index, $value := .Values.mountFiles }}
- name: "files-{{ $index }}"
mountPath: {{ $value.path }}
subPath: content
{{- end }}
{{- range $name, $volume := .Values.backend.persistence }}
- name: "{{ $name }}"
mountPath: "{{ $volume.mountPath }}"
{{- end }}
{{- range .Values.backend.extraVolumeMounts }}
- name: {{ .name }}
mountPath: {{ .mountPath }}
subPath: {{ .subPath | default "" }}
readOnly: {{ .readOnly }}
{{- end }}
{{- with .Values.backend.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.backend.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.backend.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
restartPolicy: {{ .Values.backend.createsuperuser.restartPolicy }}
volumes:
{{- range $index, $value := .Values.mountFiles }}
- name: "files-{{ $index }}"
configMap:
name: "{{ include "find.fullname" $ }}-files-{{ $index }}"
{{- end }}
{{- range $name, $volume := .Values.backend.persistence }}
- name: "{{ $name }}"
{{- if eq $volume.type "emptyDir" }}
emptyDir: {}
{{- else }}
persistentVolumeClaim:
claimName: "{{ $fullName }}-{{ $name }}"
{{- end }}
{{- end }}
{{- range .Values.backend.extraVolumes }}
- name: {{ .name }}
{{- if .existingClaim }}
persistentVolumeClaim:
claimName: {{ .existingClaim }}
{{- else if .hostPath }}
hostPath:
{{ toYaml .hostPath | nindent 12 }}
{{- else if .csi }}
csi:
{{- toYaml .csi | nindent 12 }}
{{- else if .configMap }}
configMap:
{{- toYaml .configMap | nindent 12 }}
{{- else if .emptyDir }}
emptyDir:
{{- toYaml .emptyDir | nindent 12 }}
{{- else }}
emptyDir: {}
{{- end }}
{{- end }}
{{- end }}

Some files were not shown because too many files have changed in this diff Show More