Compare commits

...

16 Commits

Author SHA1 Message Date
Fabre Florian d575545832 🔧(backend) setup Find app dockers to work with Docs
Populate a service configuration "docs" for development.
Move OIDC endpoints to the "impress" realm to allow introspection
of Docs tokens.
Upgrade version of dependencies (fix some security issues).

Signed-off-by: Fabre Florian <ffabre@hybird.org>
2025-10-07 07:06:32 +02:00
Fabre Florian 0dd513a4d3 (backend) improve unit tests
Add tests for schema validation
Use strict list comparison in some search access control tests

Signed-off-by: Fabre Florian <ffabre@hybird.org>
2025-10-02 15:51:27 +02:00
Fabre Florian 7c2a60eb6f (backend) setup search api view as OIDC resource server
New fulltext search view for indexed documents with OIDC authentication
Extract token information through introspection to get the audience & user info
Limit access to documents :
 - public & authenticated with linkreach to the user
 - owned by the user
Check intersection between the allowed services linked to the
audience/client_id and the requested ones.

Signed-off-by: Fabre Florian <ffabre@hybird.org>
2025-09-23 10:00:45 +02:00
Fabre Florian 23c4412114 (views) be a bit more permissive for indexable document content
In indexation view, raise a validation error only when both the
title & content of a document are empty.

Signed-off-by: Fabre Florian <ffabre@hybird.org>
2025-09-15 10:00:26 +02:00
Samuel Paccoud - DINUM fec8c375a2 (views) add tests for access control for an authenticated user
The implementation is partial and fakes resource server but we can
already test the filtering logic in OpenSearch.
2025-08-07 18:22:00 +02:00
Samuel Paccoud - DINUM 3cb3bf3d8f ♻️(schema) stop forcing users to be uuid (represented by a sub)
The list of users allowed to access a document is a sub and is
not guaranteed to be a UUID.
2025-08-07 18:22:00 +02:00
Samuel Paccoud - DINUM b8b335d724 ♻️(views) split views in 2 urls: /index and /search
We need to make a POST to search documents so that we can post a
list of documents the current user has already "visited" This is
necessary to limited the number of documents we return among the
ones available to any authenticated or anonymous user.
2025-08-07 18:22:00 +02:00
Samuel Paccoud - DINUM 89923e694b (schema) add fields to the document
We need to index the tree structure information as well as an
active field that can be set to False when the item is deleted on
the remote service. We could delete the item from our search index
but it is safer to keep all documents synchronized and not only
those which are not deleted.
2025-08-07 17:42:02 +02:00
Samuel Paccoud - DINUM 6b80aa280a (backend) allow passing list of indices via the query string
The client should be able to choose on which indices, among those
to which it has access (check to be added later), the query should
run.
2025-08-05 22:56:41 +02:00
Samuel Paccoud - DINUM c8d1af667c 🧑‍💻(backend) simplify index name by using service name
We added a "find-" prefix for no good reason.
2025-08-05 22:54:53 +02:00
Samuel Paccoud - DINUM 5242417738 🧑‍💻(backend) rename Reach enum class to ReachEnum
enums.Reach was a bit too similar to enums.REACT
2025-08-05 22:54:53 +02:00
Samuel Paccoud - DINUM 7eaa284357 ⬆️(backend) replace "check" by "condition" in CheckConstraint
This change is required before upgrading to Django 6.0
2025-08-05 22:54:53 +02:00
Samuel Paccoud - DINUM d48c837c3e ♻️(backend) pass full service object as request.auth
We were passing the service name which is not what is expected on
this request property.
2025-08-05 16:34:32 +02:00
Samuel Paccoud - DINUM 51dee6475b 🧑‍💻(compose) allow connecting to find from another compose project
We need to connect to find from the app container of another project
that wants to index documents to our index. This requires sharing a
common network and exposing our app on it with a service name that
does not clash with the other project.
2025-07-19 19:08:01 +02:00
Samuel Paccoud - DINUM d8050bf63d 🧑‍💻(compose) allow running in parallel to other lasuite projects
While developping, we need to run find along other projects that
want to index documents to our index. For the two projects to run
along side each other, we need to avoid port conflicts.
2025-07-19 19:05:53 +02:00
Samuel Paccoud - DINUM e31cf57bcd ♻️(schemas) replace "is_public" field by "reach"
documents can be published under one of three reaches:
- public: anybody can see them
- authenticated: only logged-in users can see them
- restricted: only users listed in the "users" field or belonging
  to a group listed in the "groups" field can see them.
2025-07-19 19:04:39 +02:00
25 changed files with 1920 additions and 385 deletions
+7
View File
@@ -12,3 +12,10 @@ and this project adheres to
- backend application
- helm chart
- 🐛(backend) fix missing index creation in 'index/' view
- ✨(backend) allow indexation of documents with either empty content or title.
- ✨(api) new fulltext 'search/' view with OIDC resource server authentication
- ✨(backend) limit access to documents : public & authenticated with a
linkreach & owned ones
- ✨(backend) limit search to the calling app (audience) and a configured
list of services
+30 -7
View File
@@ -1,10 +1,12 @@
name: find
services:
postgresql:
image: postgres:15
env_file:
- env.d/development/postgresql
ports:
- "15432:5432"
- "25432:5432"
opensearch:
image: opensearchproject/opensearch:latest
@@ -22,6 +24,12 @@ services:
ports:
- "9200:9200"
- "9600:9600"
healthcheck:
test: ["CMD", "curl", "--head", "-fsS", "http://localhost:9200/"]
start_period: 5s
interval: 2s
timeout: 2s
retries: 10
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:latest
@@ -31,7 +39,9 @@ services:
OPENSEARCH_HOSTS: '["http://opensearch:9200"]'
DISABLE_SECURITY_DASHBOARDS_PLUGIN: "true"
depends_on:
- opensearch
opensearch:
condition: service_healthy
restart: true
redis:
image: redis:5
@@ -51,15 +61,23 @@ services:
- env.d/development/common
- env.d/development/postgresql
ports:
- "8071:8000"
- "8081:8000"
networks:
default: {}
lasuite-net:
aliases:
- find
volumes:
- ./src/backend:/app
- ./data/static:/data/static
depends_on:
- postgresql
- opensearch
- redis
postgresql:
condition: service_started
redis:
condition: service_started
opensearch:
condition: service_healthy
celery:
user: ${DOCKER_USER:-1000}
image: find:backend-development
@@ -94,3 +112,8 @@ services:
HOME: /tmp
volumes:
- ".:/app"
networks:
lasuite-net:
name: lasuite-net
driver: bridge
+17 -4
View File
@@ -17,10 +17,12 @@ OPENSEARCH_PASSWORD=find.PASS123
OPENSEARCH_USE_SSL=false
# OIDC
OIDC_OP_JWKS_ENDPOINT=http://nginx:8083/realms/find/protocol/openid-connect/certs
OIDC_OP_AUTHORIZATION_ENDPOINT=http://localhost:8083/realms/find/protocol/openid-connect/auth
OIDC_OP_TOKEN_ENDPOINT=http://nginx:8083/realms/find/protocol/openid-connect/token
OIDC_OP_USER_ENDPOINT=http://nginx:8083/realms/find/protocol/openid-connect/userinfo
OIDC_OP_JWKS_ENDPOINT=http://nginx:8083/realms/impress/protocol/openid-connect/certs
OIDC_OP_AUTHORIZATION_ENDPOINT=http://localhost:8083/realms/impress/protocol/openid-connect/auth
OIDC_OP_TOKEN_ENDPOINT=http://nginx:8083/realms/impress/protocol/openid-connect/token
OIDC_OP_USER_ENDPOINT=http://nginx:8083/realms/impress/protocol/openid-connect/userinfo
OIDC_OP_URL=http://nginx:8083/realms/impress
OIDC_OP_INTROSPECTION_ENDPOINT=http://nginx:8083/realms/impress/protocol/openid-connect/token/introspect
OIDC_RP_CLIENT_ID=find
OIDC_RP_CLIENT_SECRET=ThisIsAnExampleKeyForDevPurposeOnly
@@ -29,3 +31,14 @@ OIDC_RP_SCOPES="openid email"
OIDC_REDIRECT_ALLOWED_HOSTS=["http://localhost:8083", "http://localhost:3000"]
OIDC_AUTH_REQUEST_EXTRA_PARAMS={"acr_values": "eidas1"}
# OIDC Resource server
OIDC_DRF_AUTH_BACKEND="lasuite.oidc_login.backends.OIDCAuthenticationBackend"
OIDC_RS_SCOPES="openid"
OIDC_RS_CLIENT_ID=impress
OIDC_RS_CLIENT_SECRET=ThisIsAnExampleKeyForDevPurposeOnly
OIDC_RS_SIGN_ALGO=RS256
OIDC_RS_BACKEND_CLASS="core.authentication.FinderResourceServerBackend"
OIDC_RS_ENCRYPTION_KEY_TYPE="RSA"
+46 -5
View File
@@ -1,11 +1,17 @@
"""Token authentication."""
from django.contrib.auth.models import AnonymousUser
import logging
from django.contrib.auth.models import AnonymousUser
from django.core.exceptions import ObjectDoesNotExist
from lasuite.oidc_resource_server.backend import ResourceServerBackend
from rest_framework import authentication, exceptions
from .models import Service
logger = logging.getLogger(__name__)
class ServiceTokenAuthentication(authentication.BaseAuthentication):
"""A custom authentication looking for valid tokens among registered services"""
@@ -25,11 +31,46 @@ class ServiceTokenAuthentication(authentication.BaseAuthentication):
def authenticate_credentials(self, token):
"""Check that the token is registered and valid."""
try:
service_name = (
self.model.objects.only("name").get(token=token, is_active=True).name
)
service = self.model.objects.only("name").get(token=token, is_active=True)
except self.model.DoesNotExist as excpt:
raise exceptions.AuthenticationFailed("Invalid token.") from excpt
# We don't associate tokens with a user
return AnonymousUser(), service_name
return AnonymousUser(), service
class ResourceUserManager:
"""Fake manager that simply returns an instance of user object with the right sub"""
def get(self, sub):
"""Returns a ResourceUser for this sub"""
return ResourceUser(sub=sub)
class ResourceUser:
"""Fake user model for the ResourceServerBackend.get_object() method"""
DoesNotExist = ObjectDoesNotExist
objects = ResourceUserManager()
def __init__(self, sub: str):
self.sub = sub
self.pk = None
self.is_authenticated = True
class FinderResourceServerBackend(ResourceServerBackend):
"""
Custom resource server that uses a ResourceUser object instead of
a user model from the database
"""
def __init__(self):
try:
super().__init__()
except Exception as e:
logger.error(e)
raise
self.UserModel = ResourceUser
+24 -4
View File
@@ -1,12 +1,32 @@
"""Enums for find's core app."""
RELEVANCE = "relevance"
from enum import Enum
# Reach
class ReachEnum(str, Enum):
"""Publication options for indexed documents"""
PUBLIC = "public"
AUTHENTICATED = "authenticated"
RESTRICTED = "restricted"
# Fields
CREATED_AT = "created_at"
IS_PUBLIC = "is_public"
DEPTH = "depth"
PATH = "path"
NUMCHILD = "numchild"
REACH = "reach"
SIZE = "size"
TITLE = "title"
UPDATED_AT = "updated_at"
USERS = "users"
GROUPS = "groups"
ORDER_BY_OPTIONS = (RELEVANCE, TITLE, CREATED_AT, UPDATED_AT, SIZE, IS_PUBLIC)
SOURCE_FIELDS = (TITLE, SIZE, CREATED_AT, UPDATED_AT, IS_PUBLIC)
RELEVANCE = "relevance"
ORDER_BY_OPTIONS = (RELEVANCE, TITLE, CREATED_AT, UPDATED_AT, SIZE, REACH)
SOURCE_FIELDS = (TITLE, SIZE, DEPTH, PATH, NUMCHILD, CREATED_AT, UPDATED_AT, REACH)
+10 -5
View File
@@ -8,7 +8,7 @@ from django.utils.text import slugify
import factory
from faker import Faker
from .models import Service
from . import enums, models
fake = Faker()
@@ -19,16 +19,20 @@ class DocumentSchemaFactory(factory.DictFactory):
indexation for testing and development purposes.
"""
id = factory.LazyFunction(uuid4)
id = factory.LazyFunction(lambda: str(uuid4()))
title = factory.Sequence(lambda n: f"Test title {n!s}")
path = factory.Sequence(lambda n: f"000{n}")
content = factory.Sequence(lambda n: f"Test content {n!s}")
created_at = factory.LazyFunction(
lambda: fake.date_time_this_decade(tzinfo=timezone.get_current_timezone())
)
size = factory.LazyFunction(lambda: fake.random_int(min=0, max=1024**2))
users = factory.LazyFunction(lambda: [uuid4() for _ in range(3)])
users = factory.LazyFunction(lambda: [str(uuid4()) for _ in range(3)])
groups = factory.LazyFunction(lambda: [slugify(fake.word()) for _ in range(3)])
is_public = factory.Faker("boolean")
reach = factory.Iterator(list(enums.ReachEnum))
depth = 1
numchild = 0
is_active = True
@factory.lazy_attribute
def updated_at(self):
@@ -48,6 +52,7 @@ class ServiceFactory(factory.django.DjangoModelFactory):
name = factory.Sequence(lambda n: f"test-index-{n!s}")
created_at = factory.Faker("date_time_this_year", tzinfo=None)
is_active = True
client_id = "some_client_id"
class Meta:
model = Service
model = models.Service
+1 -1
View File
@@ -59,6 +59,6 @@ class Migration(migrations.Migration):
),
migrations.AddConstraint(
model_name='service',
constraint=models.CheckConstraint(check=models.Q(('token__length', 50)), name='token_length_exact_50'),
constraint=models.CheckConstraint(condition=models.Q(('token__length', 50)), name='token_length_exact_50'),
),
]
@@ -0,0 +1,23 @@
# Generated by Django 5.1.4 on 2025-09-05 04:41
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('core', '0001_initial'),
]
operations = [
migrations.AddField(
model_name='service',
name='client_id',
field=models.CharField(blank=True, null=True),
),
migrations.AddField(
model_name='service',
name='services',
field=models.ManyToManyField(blank=True, to='core.service', verbose_name='Allowed services for search'),
),
]
+11 -7
View File
@@ -24,6 +24,12 @@ class Service(models.Model):
token = models.CharField(max_length=TOKEN_LENGTH)
created_at = models.DateTimeField(auto_now_add=True)
is_active = models.BooleanField(default=True)
client_id = models.CharField(blank=True, null=True)
services = models.ManyToManyField(
"self",
verbose_name=_("Allowed services for search"),
blank=True,
)
class Meta:
db_table = "find_service"
@@ -32,7 +38,8 @@ class Service(models.Model):
ordering = ["-is_active", "-created_at"]
constraints = [
models.CheckConstraint(
check=models.Q(token__length=TOKEN_LENGTH), name="token_length_exact_50"
condition=models.Q(token__length=TOKEN_LENGTH),
name="token_length_exact_50",
),
]
@@ -49,11 +56,8 @@ class Service(models.Model):
@staticmethod
def generate_secure_token():
"""Generate a secure token with with Python secret module"""
characters = string.ascii_letters + string.digits + string.punctuation
characters = (
string.ascii_letters + string.digits + r"""!#%&'()*+,-./:;<=>?@[\]^_`{|}~"""
)
token = "".join(secrets.choice(characters) for _ in range(TOKEN_LENGTH))
return token
@property
def index_name(self):
"""Compute index name from service name"""
return f"find-{self.name:s}"
+11 -3
View File
@@ -25,21 +25,29 @@ def ensure_index_exists(index_name):
"mappings": {
"dynamic": "strict",
"properties": {
"id": {"type": "keyword"},
"title": {
"type": "keyword", # Primary field for exact matches and sorting
"fields": {
"text": {
"type": "text" # Sub-field for full-text search
}
"type": "text"
} # Sub-field for full-text search
},
},
"depth": {"type": "integer"},
"path": {
"type": "keyword",
"fields": {"text": {"type": "text"}},
},
"numchild": {"type": "integer"},
"content": {"type": "text"},
"created_at": {"type": "date"},
"updated_at": {"type": "date"},
"size": {"type": "long"},
"users": {"type": "keyword"},
"groups": {"type": "keyword"},
"is_public": {"type": "boolean"},
"reach": {"type": "keyword"},
"is_active": {"type": "boolean"},
},
}
},
+39 -16
View File
@@ -9,6 +9,7 @@ from pydantic import (
UUID4,
AwareDatetime,
BaseModel,
BeforeValidator,
ConfigDict,
Field,
conint,
@@ -23,16 +24,20 @@ class DocumentSchema(BaseModel):
"""Schema for validating the documents submitted to our API for indexing"""
id: UUID4
title: Annotated[str, Field(max_length=300)]
content: str
title: Annotated[str, Field(max_length=300, min_length=0)]
depth: Annotated[int, Field(ge=0)]
path: Annotated[str, Field(max_length=300)]
numchild: Annotated[int, Field(ge=0)]
content: Annotated[str, Field(min_length=0)]
created_at: AwareDatetime
updated_at: AwareDatetime
size: Annotated[int, Field(ge=0, le=100 * 1024**3)] # File size limited to 100GB
users: List[UUID4] = Field(default_factory=list)
users: List[Annotated[str, Field(max_length=50)]] = Field(default_factory=list)
groups: List[Annotated[str, Field(pattern=r"^[a-z0-9]+(?:-[a-z0-9]+)*$")]] = Field(
default_factory=list
)
is_public: bool = Field(default=False)
reach: Optional[enums.ReachEnum] = Field(default=enums.ReachEnum.RESTRICTED)
is_active: bool
model_config = ConfigDict(
str_min_length=1, str_strip_whitespace=True, use_enum_values=True
@@ -52,6 +57,13 @@ class DocumentSchema(BaseModel):
raise ValueError(f"{info.field_name} must be earlier than now")
return value
@model_validator(mode="after")
def check_empty_content(self):
"""Validate that either `title` or `content` are not empty."""
if not self.title and not self.content:
raise ValueError("Either title or content should have at least 1 character")
return self
@model_validator(mode="after")
def check_update_at_after_created_at(self):
"""Date and time of last modification should be later than date and time of creation"""
@@ -74,22 +86,33 @@ class DocumentSchema(BaseModel):
return validated_groups
def cleanlist(value):
"""Build a list of strings from a string, None (empty list) or a list of objects."""
if isinstance(value, str):
# Convert comma-separated strings to list
return [s.strip() for s in value.split(",") if s.strip()]
if isinstance(value, list):
# Clean up list of strings
return [str(s).strip() for s in value if s is not None and str(s).strip()]
if value is None:
return []
raise ValueError()
StringListParameter = Annotated[List[str], BeforeValidator(cleanlist)]
class SearchQueryParametersSchema(BaseModel):
"""Schema for validating the querystring on the search API endpoint"""
q: str
is_public: Optional[bool] = None
order_by: Optional[Literal[enums.ORDER_BY_OPTIONS]] = Field(default="relevance")
services: StringListParameter = Field(default_factory=list)
visited: StringListParameter = Field(default_factory=list)
reach: Optional[enums.ReachEnum] = None
order_by: Optional[Literal[enums.ORDER_BY_OPTIONS]] = Field(default=enums.RELEVANCE)
order_direction: Optional[Literal["asc", "desc"]] = Field(default="desc")
page_number: Optional[conint(ge=1)] = Field(default=1)
page_size: Optional[conint(ge=1, le=100)] = Field(default=50)
@model_validator(mode="before")
@staticmethod
def handle_lists(values):
"""Make sure we get strings and ignore multiple values."""
for key, value in values.items():
if isinstance(value, list):
# Take the first item if it's a list
values[key] = value[0] if value else None
return values
+22
View File
@@ -0,0 +1,22 @@
"""Fixtures for tests in the find core application"""
import pytest
from lasuite.oidc_resource_server.authentication import (
get_resource_server_backend,
)
@pytest.fixture(name="jwt_rs_backend")
def jwt_resource_server_backend_fixture(settings):
"""Fixture to switch the backend to the JWTResourceServerBackend."""
_original_backend = str(settings.OIDC_RS_BACKEND_CLASS)
settings.OIDC_RS_BACKEND_CLASS = (
"lasuite.oidc_resource_server.backend.JWTResourceServerBackend"
)
get_resource_server_backend.cache_clear()
yield
settings.OIDC_RS_BACKEND_CLASS = _original_backend
get_resource_server_backend.cache_clear()
@@ -1,6 +1,7 @@
"""Tests indexing documents in OpenSearch over the API"""
import datetime
from unittest import mock
from django.utils import timezone
@@ -16,7 +17,7 @@ def test_api_documents_index_bulk_anonymous():
"""Anonymous requests should not be allowed to index documents in bulk."""
documents = factories.DocumentSchemaFactory.build_batch(3)
response = APIClient().post("/api/v1.0/documents/", documents, format="json")
response = APIClient().post("/api/v1.0/documents/index/", documents, format="json")
assert response.status_code == 403
assert response.json() == {
@@ -29,7 +30,7 @@ def test_api_documents_index_bulk_invalid_token():
documents = factories.DocumentSchemaFactory.build_batch(3)
response = APIClient().post(
"/api/v1.0/documents/",
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION="Bearer invalid",
format="json",
@@ -45,7 +46,30 @@ def test_api_documents_index_bulk_success():
documents = factories.DocumentSchemaFactory.build_batch(3)
response = APIClient().post(
"/api/v1.0/documents/",
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 207
responses = response.json()
assert [d["status"] for d in responses] == ["success"] * 3
def test_api_documents_index_bulk_ensure_index():
"""A registered service should be create the opensearch index if need."""
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(3)
# Delete the index
opensearch.client.indices.delete(index="*test*")
with pytest.raises(opensearch.NotFoundError):
opensearch.client.indices.get(index="test-service")
response = APIClient().post(
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
@@ -54,8 +78,10 @@ def test_api_documents_index_bulk_success():
assert response.status_code == 207
responses = response.json()
assert len(responses) == 3
for result in response.json():
assert result["status"] == "success"
assert [d["status"] for d in responses] == ["success"] * 3
# The index has been rebuilt
opensearch.client.indices.get(index="test-service")
@pytest.mark.parametrize(
@@ -77,6 +103,36 @@ def test_api_documents_index_bulk_success():
"string_too_long",
"String should have at most 300 characters",
),
(
"depth",
-1,
"greater_than_equal",
"Input should be greater than or equal to 0",
),
(
"depth",
"a",
"int_parsing",
"Input should be a valid integer, unable to parse string as an integer",
),
(
"path",
"a" * 301,
"string_too_long",
"String should have at most 300 characters",
),
(
"numchild",
-1,
"greater_than_equal",
"Input should be greater than or equal to 0",
),
(
"numchild",
"a",
"int_parsing",
"Input should be a valid integer, unable to parse string as an integer",
),
("content", 1, "string_type", "Input should be a valid string"),
(
"created_at",
@@ -110,12 +166,9 @@ def test_api_documents_index_bulk_success():
),
(
"users",
["33052c8b-3181-4420-aede-f8396fc0f9az"], # invalid UUID b/c contains a z
"uuid_parsing",
(
"Input should be a valid UUID, invalid character: expected an optional "
"prefix of `urn:uuid:` followed by [0-9a-fA-F-], found `z` at 36"
),
["a" * 51],
"string_too_long",
"String should have at most 50 characters",
),
(
"groups",
@@ -130,8 +183,14 @@ def test_api_documents_index_bulk_success():
"Input should be a valid list",
),
(
"is_public",
"invalid_boolean",
"reach",
"invalid",
"enum",
"Input should be 'public', 'authenticated' or 'restricted'",
),
(
"is_active",
"invalid",
"bool_parsing",
"Input should be a valid boolean, unable to interpret input",
),
@@ -148,7 +207,7 @@ def test_api_documents_index_bulk_invalid_document(
documents[0][field] = invalid_value
response = APIClient().post(
"/api/v1.0/documents/",
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
@@ -156,18 +215,28 @@ def test_api_documents_index_bulk_invalid_document(
assert response.status_code == 400
responses = response.json()
assert [d["status"] for d in responses] == ["error", "valid", "valid"]
assert responses[0]["status"] == "error"
assert len(responses[0]["errors"]) == 1
assert responses[0]["errors"][0]["msg"] == error_message
assert responses[0]["errors"][0]["type"] == error_type
for i in [1, 2]:
assert responses[i]["status"] == "valid"
@pytest.mark.parametrize(
"field", ["id", "title", "content", "size", "created_at", "updated_at"]
"field",
[
"id",
"title",
"depth",
"path",
"numchild",
"content",
"size",
"created_at",
"updated_at",
"is_active",
],
)
def test_api_documents_index_bulk_required(field):
"""Test bulk document indexing with a required field missing."""
@@ -177,7 +246,7 @@ def test_api_documents_index_bulk_required(field):
del documents[0][field]
response = APIClient().post(
"/api/v1.0/documents/",
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
@@ -185,21 +254,20 @@ def test_api_documents_index_bulk_required(field):
assert response.status_code == 400
responses = response.json()
assert [d["status"] for d in responses] == ["error", "valid", "valid"]
assert responses[0]["status"] == "error"
assert len(responses[0]["errors"]) == 1
assert responses[0]["errors"][0]["msg"] == "Field required"
assert responses[0]["errors"][0]["type"] == "missing"
for i in [1, 2]:
assert responses[i]["status"] == "valid"
@pytest.mark.parametrize(
"field,default_value",
[
("users", []),
("groups", []),
("is_public", False),
("reach", "restricted"),
],
)
def test_api_documents_index_bulk_default(field, default_value):
@@ -210,7 +278,7 @@ def test_api_documents_index_bulk_default(field, default_value):
del documents[0][field]
response = APIClient().post(
"/api/v1.0/documents/",
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
@@ -218,12 +286,10 @@ def test_api_documents_index_bulk_default(field, default_value):
assert response.status_code == 207
responses = response.json()
assert len(responses) == 3
for result in response.json():
assert result["status"] == "success"
assert [d["status"] for d in responses] == ["success"] * 3
indexed_document = opensearch.client.get(
index=service.index_name, id=responses[0]["_id"]
index=service.name, id=responses[0]["_id"]
)["_source"]
assert indexed_document[field] == default_value
@@ -238,7 +304,7 @@ def test_api_documents_index_bulk_updated_at_before_created_at():
)
response = APIClient().post(
"/api/v1.0/documents/",
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
@@ -246,6 +312,9 @@ def test_api_documents_index_bulk_updated_at_before_created_at():
assert response.status_code == 400
responses = response.json()
assert [d["status"] for d in responses] == ["error", "valid", "valid"]
assert responses[0]["status"] == "error"
assert len(responses[0]["errors"]) == 1
assert (
@@ -254,9 +323,6 @@ def test_api_documents_index_bulk_updated_at_before_created_at():
)
assert responses[0]["errors"][0]["type"] == "value_error"
for i in [1, 2]:
assert responses[i]["status"] == "valid"
@pytest.mark.parametrize(
"field",
@@ -271,7 +337,7 @@ def test_api_documents_index_bulk_datetime_future(field):
documents[0][field] = now + datetime.timedelta(seconds=3)
response = APIClient().post(
"/api/v1.0/documents/",
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
@@ -279,6 +345,8 @@ def test_api_documents_index_bulk_datetime_future(field):
assert response.status_code == 400
responses = response.json()
assert [d["status"] for d in responses] == ["error", "valid", "valid"]
assert responses[0]["status"] == "error"
assert len(responses[0]["errors"]) == 1
assert (
@@ -287,5 +355,78 @@ def test_api_documents_index_bulk_datetime_future(field):
)
assert responses[0]["errors"][0]["type"] == "value_error"
for i in [1, 2]:
assert responses[i]["status"] == "valid"
def test_api_documents_index_empty_content_check():
"""Test bulk document indexing with both empty title & content."""
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(3)
documents[0]["content"] = ""
documents[0]["title"] = ""
response = APIClient().post(
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 400
responses = response.json()
assert [d["status"] for d in responses] == ["error", "valid", "valid"]
assert responses[0]["status"] == "error"
assert len(responses[0]["errors"]) == 1
assert (
responses[0]["errors"][0]["msg"]
== "Value error, Either title or content should have at least 1 character"
)
assert responses[0]["errors"][0]["type"] == "value_error"
def test_api_documents_index_opensearch_errors():
"""Test bulk document indexing errors"""
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(3)
with mock.patch.object(opensearch.client, "bulk") as mock_bulk:
mock_bulk.return_value = {
"items": [
{"index": {"status": 201}},
{
"index": {
"status": 400,
}
},
{"index": {"status": 403, "error": {"reason": "This is forbidden"}}},
]
}
response = APIClient().post(
"/api/v1.0/documents/index/",
documents,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 207
responses = response.json()
assert responses == [
{
"_id": documents[0]["id"],
"index": 0,
"status": "success",
},
{
"_id": documents[1]["id"],
"index": 1,
"status": "error",
"message": "Unknown error",
},
{
"_id": documents[2]["id"],
"index": 2,
"status": "error",
"message": "This is forbidden",
},
]
@@ -16,7 +16,7 @@ def test_api_documents_index_single_anonymous():
"""Anonymous requests should not be allowed to index documents."""
document = factories.DocumentSchemaFactory.build()
response = APIClient().post("/api/v1.0/documents/", document, format="json")
response = APIClient().post("/api/v1.0/documents/index/", document, format="json")
assert response.status_code == 403
assert response.json() == {
@@ -29,7 +29,7 @@ def test_api_documents_index_single_invalid_token():
document = factories.DocumentSchemaFactory.build()
response = APIClient().post(
"/api/v1.0/documents/",
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION="Bearer invalid",
format="json",
@@ -45,7 +45,7 @@ def test_api_documents_index_single_success():
document = factories.DocumentSchemaFactory.build()
response = APIClient().post(
"/api/v1.0/documents/",
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
@@ -55,6 +55,58 @@ def test_api_documents_index_single_success():
assert response.json()["_id"] == str(document["id"])
def test_api_documents_index_bulk_ensure_index():
"""A registered service should be create the opensearch index if need."""
service = factories.ServiceFactory(name="test-service")
document = factories.DocumentSchemaFactory.build()
# Delete the index
opensearch.client.indices.delete(index="*test*")
with pytest.raises(opensearch.NotFoundError):
opensearch.client.indices.get(index="test-service")
response = APIClient().post(
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 201
assert response.json()["_id"] == str(document["id"])
# The index has been rebuilt
data = opensearch.client.indices.get(index="test-service")
assert data["test-service"]["mappings"] == {
"dynamic": "strict",
"properties": {
"id": {"type": "keyword"},
"title": {
"type": "keyword", # Primary field for exact matches and sorting
"fields": {
"text": {"type": "text"} # Sub-field for full-text search
},
},
"depth": {"type": "integer"},
"path": {
"type": "keyword",
"fields": {"text": {"type": "text"}},
},
"numchild": {"type": "integer"},
"content": {"type": "text"},
"created_at": {"type": "date"},
"updated_at": {"type": "date"},
"size": {"type": "long"},
"users": {"type": "keyword"},
"groups": {"type": "keyword"},
"reach": {"type": "keyword"},
"is_active": {"type": "boolean"},
},
}
@pytest.mark.parametrize(
"field, invalid_value, error_type, error_message",
[
@@ -74,6 +126,36 @@ def test_api_documents_index_single_success():
"string_too_long",
"String should have at most 300 characters",
),
(
"depth",
-1,
"greater_than_equal",
"Input should be greater than or equal to 0",
),
(
"depth",
"a",
"int_parsing",
"Input should be a valid integer, unable to parse string as an integer",
),
(
"path",
"a" * 301,
"string_too_long",
"String should have at most 300 characters",
),
(
"numchild",
-1,
"greater_than_equal",
"Input should be greater than or equal to 0",
),
(
"numchild",
"a",
"int_parsing",
"Input should be a valid integer, unable to parse string as an integer",
),
("content", 1, "string_type", "Input should be a valid string"),
(
"created_at",
@@ -107,12 +189,9 @@ def test_api_documents_index_single_success():
),
(
"users",
["33052c8b-3181-4420-aede-f8396fc0f9az"], # invalid UUID b/c contains a z*
"uuid_parsing",
(
"Input should be a valid UUID, invalid character: expected an optional "
"prefix of `urn:uuid:` followed by [0-9a-fA-F-], found `z` at 36"
),
["a" * 51],
"string_too_long",
"String should have at most 50 characters",
),
(
"groups",
@@ -127,8 +206,14 @@ def test_api_documents_index_single_success():
("Input should be a valid list"),
),
(
"is_public",
"invalid_boolean",
"reach",
"invalid",
"enum",
"Input should be 'public', 'authenticated' or 'restricted'",
),
(
"is_active",
"invalid",
"bool_parsing",
"Input should be a valid boolean, unable to interpret input",
),
@@ -145,7 +230,7 @@ def test_api_documents_index_single_invalid_document(
document[field] = invalid_value
response = APIClient().post(
"/api/v1.0/documents/",
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
@@ -157,7 +242,19 @@ def test_api_documents_index_single_invalid_document(
@pytest.mark.parametrize(
"field", ["id", "title", "content", "size", "created_at", "updated_at"]
"field",
[
"id",
"title",
"depth",
"path",
"numchild",
"content",
"size",
"created_at",
"updated_at",
"is_active",
],
)
def test_api_documents_index_single_required(field):
"""Test document indexing with a required field missing."""
@@ -167,7 +264,7 @@ def test_api_documents_index_single_required(field):
del document[field]
response = APIClient().post(
"/api/v1.0/documents/",
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
@@ -183,7 +280,7 @@ def test_api_documents_index_single_required(field):
[
("users", []),
("groups", []),
("is_public", False),
("reach", "restricted"),
],
)
def test_api_documents_index_single_default(field, default_value):
@@ -194,7 +291,7 @@ def test_api_documents_index_single_default(field, default_value):
del document[field]
response = APIClient().post(
"/api/v1.0/documents/",
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
@@ -204,7 +301,7 @@ def test_api_documents_index_single_default(field, default_value):
assert response.json()["_id"] == str(document["id"])
indexed_document = opensearch.client.get(
index=service.index_name, id=str(document["id"])
index=service.name, id=str(document["id"])
)["_source"]
assert indexed_document[field] == default_value
@@ -217,7 +314,7 @@ def test_api_documents_index_single_udpated_at_before_created():
document["updated_at"] = document["created_at"] - datetime.timedelta(seconds=1)
response = APIClient().post(
"/api/v1.0/documents/",
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
@@ -244,7 +341,7 @@ def test_api_documents_index_single_datetime_future(field):
document[field] = now + datetime.timedelta(seconds=3)
response = APIClient().post(
"/api/v1.0/documents/",
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
@@ -253,3 +350,26 @@ def test_api_documents_index_single_datetime_future(field):
assert response.status_code == 400
assert response.data[0]["msg"] == f"Value error, {field:s} must be earlier than now"
assert response.data[0]["type"] == "value_error"
def test_api_documents_index_empty_content_check():
"""Test document indexing with both empty title & content."""
service = factories.ServiceFactory(name="test-service")
document = factories.DocumentSchemaFactory.build()
document["content"] = ""
document["title"] = ""
response = APIClient().post(
"/api/v1.0/documents/index/",
document,
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
format="json",
)
assert response.status_code == 400
assert (
response.data[0]["msg"]
== "Value error, Either title or content should have at least 1 character"
)
assert response.data[0]["type"] == "value_error"
@@ -6,65 +6,149 @@ of documents is slow and better done only once.
"""
import operator
import random
import pytest
from opensearchpy.helpers import bulk
import responses
from rest_framework.test import APIClient
from core import enums, factories, opensearch
from core import enums, factories
from .utils import build_authorization_bearer, prepare_index, setup_oicd_resource_server
pytestmark = pytest.mark.django_db
def prepare_index(index_name, documents):
"""Prepare the search index before testing a query on it."""
@responses.activate
def test_api_documents_search_auth_invalid_parameters(settings):
"""Invalid service parameters should result in a 401 error"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
documents = documents if isinstance(documents, list) else [documents]
settings.OIDC_RS_CLIENT_ID = None
settings.OIDC_RS_CLIENT_SECRET = None
# Ensure existence of an empty index
opensearch.client.indices.delete(index=index_name)
opensearch.ensure_index_exists(index_name)
service = factories.ServiceFactory(name="test-service")
prepare_index(service.name, [])
# Bulk index documents
actions = []
for document in documents:
_source = document.copy()
_id = _source.pop("id")
actions.append(
{
"_op_type": "index",
"_index": index_name,
"_id": _id,
"_source": _source,
}
)
bulk(opensearch.client, actions)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "a quick fox"},
format="json",
HTTP_AUTHORIZATION="Bearer unknown",
)
opensearch.client.indices.refresh(index=index_name)
assert opensearch.client.count(index=index_name)["count"] == len(documents)
assert response.status_code == 401
assert response.json() == {"detail": "Resource Server is improperly configured"}
def test_api_documents_search_query_title():
@responses.activate
def test_api_documents_search_query_unknown_user(settings):
"""Searching a document without an existing user should result in a 401 error"""
setup_oicd_resource_server(
responses,
settings,
sub="unknown",
introspect=lambda request, user_info: (404, {}, ""),
)
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
prepare_index(service.name, [])
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "a quick fox"},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 401
assert response.json() == {"detail": "Login failed"}
@responses.activate
def test_api_documents_search_services_invalid_parameters(settings):
"""Invalid pagination parameters should result in a 400 error"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
factories.ServiceFactory(name="test-service")
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "a quick fox", "services": {}},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
assert response.json() == [
{
"loc": ["services"],
"msg": "Value error, ",
"type": "value_error",
}
]
@responses.activate
def test_api_documents_search_reached_docs_invalid_parameters(settings):
"""Invalid pagination parameters should result in a 400 error"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
factories.ServiceFactory(name="test-service")
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "a quick fox", "visited": {}},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
assert response.json() == [
{
"loc": ["visited"],
"msg": "Value error, ",
"type": "value_error",
}
]
@responses.activate
def test_api_documents_search_query_title(settings):
"""Searching a document by its title should work as expected"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
document = factories.DocumentSchemaFactory.build(
title="The quick brown fox", content="the wolf"
title="The quick brown fox",
content="the wolf",
reach=random.choice(["public", "authenticated"]),
)
# Add other documents
other_fox_document = factories.DocumentSchemaFactory.build(
title="The blue fox", content="the wolf"
title="The blue fox",
content="the wolf",
reach=random.choice(["public", "authenticated"]),
)
no_fox_document = factories.DocumentSchemaFactory.build(
title="The brown goat", content="the wolf"
title="The brown goat",
content="the wolf",
reach=random.choice(["public", "authenticated"]),
)
documents = [document, other_fox_document, no_fox_document]
prepare_index(service.index_name, documents)
prepare_index(service.name, documents)
response = APIClient().get(
"/api/v1.0/documents/",
{"q": "a quick fox"},
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "a quick fox", "visited": [doc["id"] for doc in documents]},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
@@ -74,10 +158,13 @@ def test_api_documents_search_query_title():
assert list(fox_data.keys()) == ["_index", "_id", "_score", "_source", "fields"]
assert fox_data["_id"] == str(document["id"])
assert fox_data["_source"] == {
"depth": 1,
"numchild": 0,
"path": document["path"],
"size": document["size"],
"created_at": document["created_at"].isoformat(),
"updated_at": document["updated_at"].isoformat(),
"is_public": document["is_public"],
"reach": document["reach"],
"title": "The quick brown fox",
}
assert fox_data["fields"] == {"number_of_users": [3], "number_of_groups": [3]}
@@ -92,35 +179,51 @@ def test_api_documents_search_query_title():
]
assert other_fox_data["_id"] == str(other_fox_document["id"])
assert other_fox_data["_source"] == {
"depth": 1,
"numchild": 0,
"path": other_fox_document["path"],
"size": other_fox_document["size"],
"created_at": other_fox_document["created_at"].isoformat(),
"updated_at": other_fox_document["updated_at"].isoformat(),
"is_public": other_fox_document["is_public"],
"reach": other_fox_document["reach"],
"title": "The blue fox",
}
assert other_fox_data["fields"] == {"number_of_users": [3], "number_of_groups": [3]}
def test_api_documents_search_query_content():
@responses.activate
def test_api_documents_search_query_content(settings):
"""Searching a document by its content should work as expected"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
document = factories.DocumentSchemaFactory.build(
title="the wolf", content="The quick brown fox"
title="the wolf",
content="The quick brown fox",
reach=random.choice(["public", "authenticated"]),
)
# Add other documents
other_fox_document = factories.DocumentSchemaFactory.build(
title="the wolf", content="The blue fox"
title="the wolf",
content="The blue fox",
reach=random.choice(["public", "authenticated"]),
)
no_fox_document = factories.DocumentSchemaFactory.build(
title="the wolf", content="The brown goat"
title="the wolf",
content="The brown goat",
reach=random.choice(["public", "authenticated"]),
)
prepare_index(service.index_name, [document, other_fox_document, no_fox_document])
response = APIClient().get(
"/api/v1.0/documents/",
{"q": "a quick fox"},
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
documents = [document, other_fox_document, no_fox_document]
prepare_index(service.name, documents)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "a quick fox", "visited": [doc["id"] for doc in documents]},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
@@ -130,10 +233,13 @@ def test_api_documents_search_query_content():
assert list(fox_data.keys()) == ["_index", "_id", "_score", "_source", "fields"]
assert fox_data["_id"] == str(document["id"])
assert fox_data["_source"] == {
"depth": 1,
"numchild": 0,
"path": document["path"],
"size": document["size"],
"created_at": document["created_at"].isoformat(),
"updated_at": document["updated_at"].isoformat(),
"is_public": document["is_public"],
"reach": document["reach"],
"title": document["title"],
}
assert fox_data["fields"] == {"number_of_users": [3], "number_of_groups": [3]}
@@ -148,20 +254,29 @@ def test_api_documents_search_query_content():
]
assert other_fox_data["_id"] == str(other_fox_document["id"])
assert other_fox_data["_source"] == {
"depth": 1,
"numchild": 0,
"path": other_fox_document["path"],
"size": other_fox_document["size"],
"created_at": other_fox_document["created_at"].isoformat(),
"updated_at": other_fox_document["updated_at"].isoformat(),
"is_public": other_fox_document["is_public"],
"reach": other_fox_document["reach"],
"title": other_fox_document["title"],
}
assert other_fox_data["fields"] == {"number_of_users": [3], "number_of_groups": [3]}
def test_api_documents_search_ordering_by_fields():
@responses.activate
def test_api_documents_search_ordering_by_fields(settings):
"""It should be possible to order by several fields"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(4)
prepare_index(service.index_name, documents)
documents = factories.DocumentSchemaFactory.build_batch(
4, reach=random.choice(["public", "authenticated"])
)
prepare_index(service.name, documents)
parameters = [
(enums.TITLE, "asc"),
@@ -172,66 +287,96 @@ def test_api_documents_search_ordering_by_fields():
(enums.UPDATED_AT, "desc"),
(enums.SIZE, "asc"),
(enums.SIZE, "desc"),
(enums.IS_PUBLIC, "asc"),
(enums.IS_PUBLIC, "desc"),
(enums.REACH, "asc"),
(enums.REACH, "desc"),
]
for field, direction in parameters:
response = APIClient().get(
f"/api/v1.0/documents/?q=*&order_by={field}&order_direction={direction}",
HTTP_AUTHORIZATION=f"Bearer {service.token}",
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"order_by": field,
"order_direction": direction,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
responses = response.json()
assert len(responses) == 4
data = response.json()
assert len(data) == 4
# Check that results are sorted by the field as expected
compare = operator.le if direction == "asc" else operator.ge
for i in range(len(responses) - 1):
assert compare(
responses[i]["_source"][field], responses[i + 1]["_source"][field]
)
for i in range(len(data) - 1):
assert compare(data[i]["_source"][field], data[i + 1]["_source"][field])
def test_api_documents_search_ordering_by_relevance():
@responses.activate
def test_api_documents_search_ordering_by_relevance(settings):
"""It should be possible to order by relevance (score)"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(4)
prepare_index(service.index_name, documents)
documents = factories.DocumentSchemaFactory.build_batch(
4, reach=random.choice(["public", "authenticated"])
)
prepare_index(service.name, documents)
for direction in ["asc", "desc"]:
response = APIClient().get(
f"/api/v1.0/documents/?q=*&order_by=relevance&order_direction={direction}",
HTTP_AUTHORIZATION=f"Bearer {service.token}",
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"order_by": "relevance",
"order_direction": direction,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
responses = response.json()
assert len(responses) == 4
data = response.json()
assert len(data) == 4
# Check that results are sorted by score as expected
compare = operator.le if direction == "asc" else operator.ge
for i in range(len(responses) - 1):
assert compare(responses[i]["_score"], responses[i + 1]["_score"])
for i in range(len(data) - 1):
assert compare(data[i]["_score"], data[i + 1]["_score"])
def test_api_documents_search_ordering_by_unknown_field():
@responses.activate
def test_api_documents_search_ordering_by_unknown_field(settings):
"""Trying to sort by an unknown field should return a 400 error"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
# Setup: Initialize the service and documents only once
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(2)
prepare_index(service.index_name, documents)
documents = factories.DocumentSchemaFactory.build_batch(
2, reach=random.choice(["public", "authenticated"])
)
prepare_index(service.name, documents)
# Define the parameters manually
directions = ["asc", "desc"]
# Perform the parameterized tests
for direction in directions:
response = APIClient().get(
f"/api/v1.0/documents/?q=*&order_by=unknown&order_direction={direction}",
HTTP_AUTHORIZATION=f"Bearer {service.token}",
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"order_by": "unknown",
"order_direction": direction,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
@@ -240,23 +385,36 @@ def test_api_documents_search_ordering_by_unknown_field():
"loc": ["order_by"],
"msg": (
"Input should be 'relevance', 'title', 'created_at', "
"'updated_at', 'size' or 'is_public'"
"'updated_at', 'size' or 'reach'"
),
"type": "literal_error",
}
]
def test_api_documents_search_ordering_by_unknown_direction():
@responses.activate
def test_api_documents_search_ordering_by_unknown_direction(settings):
"""Trying to sort with an unknown direction should return a 400 error"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(2)
prepare_index(service.index_name, documents)
documents = factories.DocumentSchemaFactory.build_batch(
2, reach=random.choice(["public", "authenticated"])
)
prepare_index(service.name, documents)
for field in enums.ORDER_BY_OPTIONS:
response = APIClient().get(
f"/api/v1.0/documents/?q=*&order_by={field}&order_direction=unknown",
HTTP_AUTHORIZATION=f"Bearer {service.token}",
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"order_by": field,
"order_direction": "unknown",
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
@@ -269,87 +427,131 @@ def test_api_documents_search_ordering_by_unknown_direction():
]
def test_api_documents_search_filtering_by_is_public():
"""It should be possible to filter results by their publication status"""
@responses.activate
def test_api_documents_search_filtering_by_reach(settings):
"""It should be possible to filter results by their reach"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(4)
prepare_index(service.index_name, documents)
documents = factories.DocumentSchemaFactory.build_batch(
4, reach=random.choice(["public", "authenticated"])
)
prepare_index(service.name, documents)
parameters = [
[True, "True"],
[False, "False"],
[True, "true"],
[False, "false"],
[True, "1"],
[False, "0"],
]
for public_boolean, public_string in parameters:
response = APIClient().get(
f"/api/v1.0/documents/?q=*&is_public={public_string}",
HTTP_AUTHORIZATION=f"Bearer {service.token}",
for reach in enums.ReachEnum:
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"reach": reach.value,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
responses = response.json()
data = response.json()
for result in responses:
assert public_boolean == result["_source"]["is_public"]
for result in data:
assert reach == result["_source"]["reach"]
# Pagination
def test_api_documents_search_pagination_basic():
@responses.activate
def test_api_documents_search_pagination_basic(settings):
"""Pagination should correctly return documents for the specified page and page size"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(9)
documents = factories.DocumentSchemaFactory.build_batch(
9, reach=random.choice(["public", "authenticated"])
)
ids = [str(doc["id"]) for doc in documents]
prepare_index(service.index_name, documents)
prepare_index(service.name, documents)
# Request the first page with a page size of 3
response = APIClient().get(
"/api/v1.0/documents/?q=*&page_number=1&page_size=3",
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"page_number": 1,
"page_size": 3,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
responses = response.json()
assert len(responses) == 3 # Page size is 3
assert [r["_id"] for r in responses] == ids[0:3]
data = response.json()
assert len(data) == 3 # Page size is 3
assert [r["_id"] for r in data] == ids[0:3]
# Request the second page with a page size of 3
response = APIClient().get(
"/api/v1.0/documents/?q=*&page_number=2&page_size=3",
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"page_number": 2,
"page_size": 3,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
responses = response.json()
assert len(responses) == 3
assert [r["_id"] for r in responses] == ids[3:6]
data = response.json()
assert len(data) == 3
assert [r["_id"] for r in data] == ids[3:6]
# Request the third page with a page size of 5 (should contain the remaining 3 documents)
response = APIClient().get(
"/api/v1.0/documents/?q=*&page_number=3&page_size=3",
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"page_number": 3,
"page_size": 3,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
responses = response.json()
assert len(responses) == 3
assert [r["_id"] for r in responses] == ids[6:9]
data = response.json()
assert len(data) == 3
assert [r["_id"] for r in data] == ids[6:9]
def test_api_documents_search_pagination_last_page_edge_case():
@responses.activate
def test_api_documents_search_pagination_last_page_edge_case(settings):
"""Requesting the last page should return the correct number of remaining documents"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(8)
documents = factories.DocumentSchemaFactory.build_batch(
8, reach=random.choice(["public", "authenticated"])
)
ids = [str(doc["id"]) for doc in documents]
prepare_index(service.index_name, documents)
prepare_index(service.name, documents)
# Request the first page with a page size of 3
response = APIClient().get(
"/api/v1.0/documents/?q=*&page_number=1&page_size=3",
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"page_number": 1,
"page_size": 3,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
@@ -357,9 +559,16 @@ def test_api_documents_search_pagination_last_page_edge_case():
assert [r["_id"] for r in response.json()] == ids[0:3]
# Request the third page with a page size of 3 (should contain the last 1 document)
response = APIClient().get(
"/api/v1.0/documents/?q=*&page_number=3&page_size=3",
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"page_number": 3,
"page_size": 3,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
@@ -367,29 +576,48 @@ def test_api_documents_search_pagination_last_page_edge_case():
assert [r["_id"] for r in response.json()] == ids[6:]
def test_api_documents_search_pagination_out_of_bounds():
@responses.activate
def test_api_documents_search_pagination_out_of_bounds(settings):
"""
Requesting a page number that exceeds the total number of pages should return an empty list
"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(4)
prepare_index(service.index_name, documents)
documents = factories.DocumentSchemaFactory.build_batch(
4, reach=random.choice(["public", "authenticated"])
)
prepare_index(service.name, documents)
# Request the fourth page with a page size of 2 (there are only 2 pages)
response = APIClient().get(
"/api/v1.0/documents/?q=*&page_number=4&page_size=2",
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"page_number": 4,
"page_size": 2,
"visited": [doc["id"] for doc in documents],
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert len(response.json()) == 0 # No documents should be returned
def test_api_documents_search_pagination_invalid_parameters():
@responses.activate
def test_api_documents_search_pagination_invalid_parameters(settings):
"""Invalid pagination parameters should result in a 400 error"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents = factories.DocumentSchemaFactory.build_batch(4)
prepare_index(service.index_name, documents)
documents = factories.DocumentSchemaFactory.build_batch(
4, reach=random.choice(["public", "authenticated"])
)
prepare_index(service.name, documents)
parameters = [
(
@@ -411,9 +639,11 @@ def test_api_documents_search_pagination_invalid_parameters():
]
for page_number, page_size, error_type, error_message in parameters:
response = APIClient().get(
f"/api/v1.0/documents/?q=*&page_number={page_number}&page_size={page_size}",
HTTP_AUTHORIZATION=f"Bearer {service.token}",
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "page_number": page_number, "page_size": page_size},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
@@ -421,27 +651,49 @@ def test_api_documents_search_pagination_invalid_parameters():
assert response.data[0]["type"] == error_type
def test_api_documents_search_pagination_with_filtering():
"""Pagination should work correctly when combined with filtering by is_public"""
@responses.activate
def test_api_documents_search_pagination_with_filtering(settings):
"""Pagination should work correctly when combined with filtering by reach"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
public_documents = factories.DocumentSchemaFactory.build_batch(3, is_public=True)
public_documents = factories.DocumentSchemaFactory.build_batch(3, reach="public")
public_ids = [str(doc["id"]) for doc in public_documents]
private_documents = factories.DocumentSchemaFactory.build_batch(2, is_public=False)
prepare_index(service.index_name, public_documents + private_documents)
private_documents = factories.DocumentSchemaFactory.build_batch(
2, reach="authenticated"
)
prepare_index(service.name, public_documents + private_documents)
# Filter by public documents, request first page
response = APIClient().get(
"/api/v1.0/documents/?q=*&is_public=true&page_number=1&page_size=2",
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"reach": "public",
"page_number": 1,
"page_size": 2,
"visited": public_ids,
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert len(response.json()) == 2
assert [r["_id"] for r in response.json()] == public_ids[0:2]
# Request second page for public documents (remaining 1 document)
response = APIClient().get(
"/api/v1.0/documents/?q=*&is_public=true&page_number=2&page_size=2",
HTTP_AUTHORIZATION=f"Bearer {service.token:s}",
response = APIClient().post(
"/api/v1.0/documents/search/",
{
"q": "*",
"reach": "public",
"page_number": 2,
"page_size": 2,
"visited": public_ids,
},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
@@ -0,0 +1,438 @@
"""
Test suite for access control when searching documents over the API.
Don't use pytest parametrized tests because batch generation and indexing
of documents is slow and better done only once.
"""
import pytest
import responses
from rest_framework.test import APIClient
from core import enums, factories
from .utils import (
build_authorization_bearer,
delete_test_indices,
prepare_index,
setup_oicd_resource_server,
)
pytestmark = pytest.mark.django_db
def test_api_documents_search_access_control_anonymous():
"""Anonymous users should not be allowed to search documents even public."""
service = factories.ServiceFactory(name="test-service")
documents = []
for reach in enums.ReachEnum:
documents.extend(factories.DocumentSchemaFactory.build_batch(3, reach=reach))
prepare_index(service.name, documents)
response = APIClient().post("/api/v1.0/documents/search/?q=*")
assert response.status_code == 401
@responses.activate
def test_api_documents_search_access_control(settings):
"""
Authenticated users should only see documents:
- for which they are listed in the "users" field
- that have a reach set to "authenticated" or "public"
- only configured services providers are allowed (e.g docs)
(groups is not yet implemnted)
"""
setup_oicd_resource_server(responses, settings, sub="user_sub")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service")
documents_reach = factories.DocumentSchemaFactory.build_batch(6)
documents_open = [
doc for doc in documents_reach if doc["reach"] in ["authenticated", "public"]
]
documents_user = factories.DocumentSchemaFactory.build_batch(
6, users=["user_sub", "user_sub2"]
)
expected_ids = [doc["id"] for doc in documents_open + documents_user]
prepare_index(service.name, documents_user + documents_reach)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "visited": [d["id"] for d in documents_open]},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert sorted([d["_id"] for d in response.json()]) == sorted(expected_ids)
@responses.activate
@pytest.mark.parametrize(
"doc_ids,visited,expected",
[
(["a", "b"], [], []),
(["a", "b"], "", []),
(["a", "b"], None, []),
(["a", "b"], ["other"], []),
([], ["a"], []),
(["a", "b"], ["a"], ["a"]),
(["a", "b"], ["a", "b", "c"], ["a", "b"]),
(["a", "b"], "a,b,c", ["a", "b"]),
],
)
def test_api_documents_search_access__only_visited_public(
doc_ids, visited, expected, settings
):
"""
Authenticated users should only see documents with reach="public"
that are in "visited" list.
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="docs")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service", client_id="docs")
docs = [
factories.DocumentSchemaFactory(
reach=[enums.ReachEnum.PUBLIC, enums.ReachEnum.AUTHENTICATED], id=doc_id
)
for doc_id in doc_ids
]
prepare_index(service.name, docs)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "visited": visited},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200, response.json()
assert sorted([d["_id"] for d in response.json()]) == sorted(expected)
@responses.activate
def test_api_documents_search_access__any_owner_public(settings):
"""
Authenticated users should only see documents with reach="public"
that are in "visited" list.
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="docs")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service", client_id="docs")
docs = factories.DocumentSchemaFactory.build_batch(
6,
reach=enums.ReachEnum.PUBLIC,
users=["user_sub"],
)
other_docs = factories.DocumentSchemaFactory.build_batch(
6,
reach=enums.ReachEnum.PUBLIC,
users=["other_sub"],
)
prepare_index(service.name, docs + other_docs)
expected = [d["id"] for d in docs]
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "visited": []},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200, response.json()
assert sorted([d["_id"] for d in response.json()]) == sorted(expected)
@responses.activate
def test_api_documents_search_access__services(settings):
"""
Authenticated users should only see documents of audience
service providers (e.g docs)
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="a-client")
token = build_authorization_bearer()
service_a = factories.ServiceFactory(name="test-index-a", client_id="a-client")
service_b = factories.ServiceFactory(name="test-index-b", client_id="b-client")
service_a_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
service_b_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
expected_ids = [doc["id"] for doc in service_a_docs]
prepare_index(service_a.name, service_a_docs)
prepare_index(service_b.name, service_b_docs, cleanup=False)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*"},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert sorted([d["_id"] for d in response.json()]) == sorted(expected_ids)
@responses.activate
def test_api_documents_search_access__missing_index(settings):
"""
When the service has no opensearch index, returns an empty list.
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="a-client")
token = build_authorization_bearer()
factories.ServiceFactory(name="test-index-a", client_id="a-client")
delete_test_indices()
# a-client has no index. ignore it.
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*"},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert response.json() == []
@responses.activate
def test_api_documents_search_access__related_services(settings):
"""
Authenticated users should only see documents of audience
service providers and its related services (e.g drive + docs)
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="c-client")
token = build_authorization_bearer()
service_a = factories.ServiceFactory(name="test-index-a", client_id="a-client")
service_b = factories.ServiceFactory(name="test-index-b", client_id="b-client")
service_c = factories.ServiceFactory(name="test-index-c", client_id="c-client")
service_c.services.set([service_a])
service_c.save()
service_a_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
service_b_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
service_c_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
expected_ids = [doc["id"] for doc in service_a_docs + service_c_docs]
prepare_index(service_a.name, service_a_docs)
prepare_index(service_b.name, service_b_docs, cleanup=False)
prepare_index(service_c.name, service_c_docs, cleanup=False)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*"},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert sorted([d["_id"] for d in response.json()]) == sorted(expected_ids)
@responses.activate
def test_api_documents_search_access__related_missing_index(settings):
"""
When the service has no opensearch index, returns the related services data.
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="a-client")
token = build_authorization_bearer()
service_a = factories.ServiceFactory(name="test-index-a", client_id="a-client")
service_b = factories.ServiceFactory(name="test-index-b", client_id="b-client")
service_c = factories.ServiceFactory(name="test-index-c", client_id="c-client")
service_c.services.set([service_a])
service_c.save()
service_b_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
service_c_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
expected_ids = [doc["id"] for doc in service_c_docs]
prepare_index(service_b.name, service_b_docs)
prepare_index(service_c.name, service_c_docs, cleanup=False)
# a-client has no index. ignore it.
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*"},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert sorted([d["_id"] for d in response.json()]) == sorted(expected_ids)
@responses.activate
def test_api_documents_search_access__request_services(settings):
"""
Authenticated users should only see documents of audience
from requested services : 'services' parameter.
Raise 400 error if not all requested services are authorized.
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="c-client")
token = build_authorization_bearer()
service_a = factories.ServiceFactory(name="test-index-a", client_id="a-client")
service_b = factories.ServiceFactory(name="test-index-b", client_id="b-client")
service_c = factories.ServiceFactory(name="test-index-c", client_id="c-client")
service_a_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
service_b_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
service_c_docs = factories.DocumentSchemaFactory.build_batch(
3, reach=enums.ReachEnum.AUTHENTICATED, users=["user_sub"]
)
expected_ids = [doc["id"] for doc in service_c_docs]
prepare_index(service_a.name, service_a_docs)
prepare_index(service_b.name, service_b_docs, cleanup=False)
prepare_index(service_c.name, service_c_docs, cleanup=False)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "services": ["test-index-c"]},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert sorted([d["_id"] for d in response.json()]) == sorted(expected_ids)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "services": ["test-index-c", "test-index-b"]},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
assert response.json() == {"detail": "Some requested services are not available"}
@responses.activate
def test_api_documents_search_access__request_inactive_services(settings):
"""
Authenticated users should only see documents of audience
from requested services : 'services' parameter.
Raise 400 error if not all requested services are active.
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="client")
token = build_authorization_bearer()
factories.ServiceFactory(name="test-index", client_id="client", is_active=False)
factories.ServiceFactory(name="test-index-b", client_id="b-client")
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "services": ["test-index"]},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
assert response.json() == {"detail": "Service is not available"}
# Event without explicit argument, the client service from the request is not active
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*"},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 400
assert response.json() == {"detail": "Service is not available"}
@responses.activate
def test_api_documents_search_access__authenticated(settings):
"""
Authenticated users should only see documents
- for which they are listed in the "users" field
- that have a reach set to "authenticated" or "public" AND in visited list
- only configured services providers are allowed (e.g docs)
(groups is not yet implemnted)
"""
setup_oicd_resource_server(responses, settings, sub="user_sub", audience="docs")
token = build_authorization_bearer()
service = factories.ServiceFactory(name="test-service", client_id="docs")
documents_open = factories.DocumentSchemaFactory.build_batch(
2, reach=enums.ReachEnum.PUBLIC
) + factories.DocumentSchemaFactory.build_batch(
2, reach=enums.ReachEnum.AUTHENTICATED
)
documents_restricted = factories.DocumentSchemaFactory.build_batch(
2, reach=enums.ReachEnum.RESTRICTED
)
documents_user = factories.DocumentSchemaFactory.build_batch(
6, users=["user_sub", "user_sub2"]
)
documents = documents_user + documents_open + documents_restricted
prepare_index(service.name, documents_user + documents_open + documents_restricted)
# Only owned documents (reach is ignored)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*"},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert sorted([d["_id"] for d in response.json()]) == sorted(
[doc["id"] for doc in documents_user]
)
# Owned documents and visited public/authenticated ones.
# Restricted ones from another owner are filtered (even if given as visited ones)
response = APIClient().post(
"/api/v1.0/documents/search/",
{"q": "*", "visited": [d["id"] for d in documents]},
format="json",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
assert response.status_code == 200
assert sorted([d["_id"] for d in response.json()]) == sorted(
[doc["id"] for doc in documents_user + documents_open]
)
@@ -23,12 +23,6 @@ def test_models_services_name_slugified():
assert service.name == "my-service-name"
def test_models_services_index_name():
"""The index name should be computed as a property from the service name."""
service = factories.ServiceFactory(name="My service name")
assert service.index_name == "find-my-service-name"
def test_models_services_token_50_characters_exact():
"""The token field should be 50 characters long."""
service = factories.ServiceFactory()
+26
View File
@@ -0,0 +1,26 @@
"""Test pydantic models & helpers"""
import pytest
from core.schemas import cleanlist
def test_cleanlist_empty():
"""Empty data should return an empty list"""
assert cleanlist(None) == []
assert cleanlist([]) == []
assert cleanlist("") == []
def test_cleanlist_error():
"""Invalid data should raise"""
with pytest.raises(ValueError):
cleanlist(123)
def test_cleanlist():
"""Should return a list of stripped strings and remove the empty items"""
assert cleanlist([1, 2, 3]) == ["1", "2", "3"]
assert cleanlist(" 1, 2,3 ") == ["1", "2", "3"]
assert cleanlist(["1 ", " 2", "3 "]) == ["1", "2", "3"]
assert cleanlist([None, 2, 3, ""]) == ["2", "3"]
+221
View File
@@ -0,0 +1,221 @@
"""Tests Service model for find's core app."""
import base64
import json
from functools import partial
from typing import List
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import rsa
from joserfc import jwe as jose_jwe
from joserfc import jwt as jose_jwt
from joserfc.jwk import RSAKey
from jwt.utils import to_base64url_uint
from opensearchpy.helpers import bulk
from core import opensearch
def delete_test_indices():
"""Drop all search index containing the 'test' word"""
opensearch.client.indices.delete(index="*test*")
def prepare_index(index_name, documents: List, cleanup=True):
"""Prepare the search index before testing a query on it."""
if cleanup:
delete_test_indices()
opensearch.ensure_index_exists(index_name)
# Index new documents
actions = [
{
"_op_type": "index",
"_index": index_name,
"_id": doc["id"],
"_source": {k: v for k, v in doc.items() if k != "id"},
}
for doc in documents
]
bulk(opensearch.client, actions)
# Force refresh again so all changes are visible to search
opensearch.client.indices.refresh(index=index_name)
count = opensearch.client.count(index=index_name)["count"]
assert count == len(documents), f"Expected {len(documents)}, got {count}"
def build_authorization_bearer(token="some_token"):
"""
Build an Authorization Bearer header value from a token.
This can be used like this:
client.post(
...
HTTP_AUTHORIZATION=f"Bearer {build_authorization_bearer('some_token')}",
)
"""
return base64.b64encode(token.encode("utf-8")).decode("utf-8")
def setup_oicd_jwt_resource_server(
responses,
settings,
sub="some_sub",
audience="some_client_id",
):
"""
Setup settings for a resource server with JWT backend.
Simulate an encrypted token introspection.
NOTE : Use it with @responses.activate or the fake introspection view will not work.
"""
token_data = {
"sub": sub,
"iss": "https://oidc.example.com",
"aud": audience,
"client_id": "some_service_provider",
"scope": "docs",
"active": True,
}
private_key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
unencrypted_pem_private_key = private_key.private_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PrivateFormat.TraditionalOpenSSL,
encryption_algorithm=serialization.NoEncryption(),
)
pem_public_key = private_key.public_key().public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo,
)
settings.OIDC_RS_PRIVATE_KEY_STR = unencrypted_pem_private_key.decode("utf-8")
settings.OIDC_RS_ENCRYPTION_KEY_TYPE = "RSA"
settings.OIDC_RS_ENCRYPTION_ENCODING = "A256GCM"
settings.OIDC_RS_ENCRYPTION_ALGO = "RSA-OAEP"
settings.OIDC_RS_SIGNING_ALGO = "RS256"
settings.OIDC_RS_CLIENT_ID = audience
settings.OIDC_RS_CLIENT_SECRET = "some_client_secret"
settings.OIDC_RS_SCOPES = ["openid", "docs", "email"]
settings.OIDC_OP_URL = "https://oidc.example.com"
settings.OIDC_OP_JWKS_ENDPOINT = "https://oidc.example.com/jwks"
settings.OIDC_OP_INTROSPECTION_ENDPOINT = "https://oidc.example.com/introspect"
settings.OIDC_VERIFY_SSL = False
settings.OIDC_TIMEOUT = 5
settings.OIDC_PROXY = None
settings.OIDC_CREATE_USER = False
# Mock the JWKS endpoint
public_numbers = private_key.public_key().public_numbers()
responses.add(
responses.GET,
settings.OIDC_OP_JWKS_ENDPOINT,
body=json.dumps(
{
"keys": [
{
"kty": settings.OIDC_RS_ENCRYPTION_KEY_TYPE,
"alg": settings.OIDC_RS_SIGNING_ALGO,
"use": "sig",
"kid": "1234567890",
"n": to_base64url_uint(public_numbers.n).decode("ascii"),
"e": to_base64url_uint(public_numbers.e).decode("ascii"),
}
]
}
),
)
def encrypt_jwt(json_data):
"""Encrypt the JWT token for the backend to decrypt."""
token = jose_jwt.encode(
{
"kid": "1234567890",
"alg": settings.OIDC_RS_SIGNING_ALGO,
},
json_data,
RSAKey.import_key(unencrypted_pem_private_key),
algorithms=[settings.OIDC_RS_SIGNING_ALGO],
)
return jose_jwe.encrypt_compact(
protected={
"alg": settings.OIDC_RS_ENCRYPTION_ALGO,
"enc": settings.OIDC_RS_ENCRYPTION_ENCODING,
},
plaintext=token,
public_key=RSAKey.import_key(pem_public_key),
algorithms=[
settings.OIDC_RS_ENCRYPTION_ALGO,
settings.OIDC_RS_ENCRYPTION_ENCODING,
],
)
responses.add(
responses.POST,
"https://oidc.example.com/introspect",
body=encrypt_jwt(
{
"iss": "https://oidc.example.com",
"aud": audience, # settings.OIDC_RS_CLIENT_ID
"token_introspection": token_data,
}
),
)
def setup_oicd_resource_server(
responses,
settings,
sub="some_sub",
audience="some_client_id",
introspect=None,
): # pylint: disable=too-many-arguments
"""
Setup settings for a resource server.
Simulate a token introspection.
NOTE : Use it with @responses.activate or the fake introspection view will not work.
"""
token_data = {
"sub": sub,
"iss": "https://oidc.example.com",
"aud": audience,
"client_id": audience,
"scope": "docs",
"active": True,
}
settings.OIDC_RS_ENCRYPTION_KEY_TYPE = "RSA"
settings.OIDC_RS_ENCRYPTION_ENCODING = "A256GCM"
settings.OIDC_RS_ENCRYPTION_ALGO = "RSA-OAEP"
settings.OIDC_RS_SIGNING_ALGO = "RS256"
settings.OIDC_RS_CLIENT_ID = audience
settings.OIDC_RS_CLIENT_SECRET = "some_client_secret"
settings.OIDC_RS_SCOPES = ["openid", "docs", "email"]
settings.OIDC_OP_URL = "https://oidc.example.com"
settings.OIDC_OP_INTROSPECTION_ENDPOINT = "https://oidc.example.com/introspect"
settings.OIDC_VERIFY_SSL = False
settings.OIDC_TIMEOUT = 5
settings.OIDC_PROXY = None
settings.OIDC_CREATE_USER = False
if callable(introspect):
responses.add_callback(
responses.POST,
"https://oidc.example.com/introspect",
callback=partial(introspect, user_info=token_data),
)
else:
responses.add(
responses.POST,
"https://oidc.example.com/introspect",
body=json.dumps(token_data),
)
+5 -3
View File
@@ -1,9 +1,11 @@
"""URL configuration for find's core app."""
from django.urls import path
from django.urls import include, path
from .views import DocumentView
from .views import IndexDocumentView, SearchDocumentView
urlpatterns = [
path("documents/", DocumentView.as_view(), name="document"),
path("documents/index/", IndexDocumentView.as_view(), name="document"),
path("documents/search/", SearchDocumentView.as_view(), name="document"),
path("", include("lasuite.oidc_resource_server.urls")),
]
+124 -49
View File
@@ -1,43 +1,35 @@
"""Views for find's core app."""
import logging
from django.core.exceptions import SuspiciousOperation
from lasuite.oidc_resource_server.authentication import ResourceServerAuthentication
from lasuite.oidc_resource_server.mixins import ResourceServerMixin
from pydantic import ValidationError as PydanticValidationError
from rest_framework import status, views
from rest_framework.response import Response
from urllib3.exceptions import ReadTimeoutError
from . import enums, schemas
from .authentication import ServiceTokenAuthentication
from .models import Service
from .opensearch import client, ensure_index_exists
from .permissions import IsAuthAuthenticated
logger = logging.getLogger(__name__)
class DocumentView(views.APIView):
class IndexDocumentView(views.APIView):
"""
API view for managing documents in an OpenSearch index.
This view provides functionality for both indexing and searching documents
within an OpenSearch index dedicated to the authenticated service. The class
supports the following operations:
1. **Document Indexing (POST)**:
API view for indexing documents in OpenSearch.
- Handles both single document and bulk document indexing.
- The index is dynamically determined based on the service authentication token,
ensuring that each service has its own isolated index.
2. **Document Search (GET)**:
- Enables searching through indexed documents with support for various filters
and sorting options.
- The search results can be sorted or filtered via querystring parameters.
ensuring that each service has its own isolated index.
"""
authentication_classes = [ServiceTokenAuthentication]
permission_classes = [IsAuthAuthenticated]
@property
def index_name(self):
"""Compute index name from the service name extracted during authentication"""
return f"find-{self.request.auth}"
# pylint: disable=too-many-locals
def post(self, request, *args, **kwargs):
"""
@@ -87,6 +79,7 @@ class DocumentView(views.APIView):
- Returns a list of results for all documents, with details of success and indexing
errors.
"""
index_name = request.auth.name
if isinstance(request.data, list):
# Bulk indexing several documents
@@ -110,11 +103,14 @@ class DocumentView(views.APIView):
actions.append({"index": {"_id": _id}})
actions.append(document_dict)
results.append({"index": i, "_id": _id, "status": "valid"})
if has_errors:
return Response(results, status=status.HTTP_400_BAD_REQUEST)
ensure_index_exists(self.index_name)
response = client.bulk(index=self.index_name, body=actions)
# Build index if needed.
ensure_index_exists(index_name)
response = client.bulk(index=index_name, body=actions)
for i, item in enumerate(response["items"]):
if item["index"]["status"] != 201:
results[i]["status"] = "error"
@@ -130,33 +126,65 @@ class DocumentView(views.APIView):
document = schemas.DocumentSchema(**request.data)
document_dict = document.model_dump()
_id = document_dict.pop("id")
try:
client.index(index=self.index_name, body=document_dict, id=_id)
except ReadTimeoutError:
ensure_index_exists(self.index_name)
client.index(index=self.index_name, body=document_dict, id=_id)
# Build index if needed.
ensure_index_exists(index_name)
client.index(index=index_name, body=document_dict, id=_id)
return Response(
{"status": "created", "_id": _id}, status=status.HTTP_201_CREATED
)
def get(self, request, *args, **kwargs):
class SearchDocumentView(ResourceServerMixin, views.APIView):
"""
API view for searching documents in OpenSearch.
- Enables searching through indexed documents with support for various filters
and sorting options.
- The search results can be sorted or filtered via querystring parameters.
"""
authentication_classes = [ResourceServerAuthentication]
permission_classes = [IsAuthAuthenticated]
def _get_opensearch_indices(self, audience, services):
# Get request user service
try:
user_service = Service.objects.get(client_id=audience, is_active=True)
except Service.DoesNotExist as e:
logger.warning("Login failed: No service %s found", audience)
raise SuspiciousOperation("Service is not available") from e
# Find allowed sub-services for this service
allowed_services = set(user_service.services.values_list("name", flat=True))
allowed_services.add(user_service.name)
if services:
available = set(services).intersection(allowed_services)
if len(available) < len(services):
raise SuspiciousOperation("Some requested services are not available")
return allowed_services
def post(self, request, *args, **kwargs):
"""
Handle GET requests to perform a search on indexed documents with optional filtering
Handle POST requests to perform a search on indexed documents with optional filtering
and ordering.
The search query should be provided as a query parameter 'q'. The method constructs a
The search query should be provided as a "q" parameter. The method constructs a
search request to OpenSearch using the specified query, with the option to filter by
'is_public' and order by 'relevance', 'created_at', 'updated_at', or 'size'.
'reach' and order by 'relevance', 'created_at', 'updated_at', or 'size'.
The results are further filtered by 'users' and 'groups' based on the authentication
header.
Query Parameters:
Body Parameters:
---------------
q : str
The search query string. This is a required parameter.
is_public : bool, optional
Filter results based on the 'is_public' field.
reach : str, optional
Filter results based on the 'reach' field.
order_by : str, optional
Order results by 'relevance', 'created_at', 'updated_at', or 'size'.
Defaults to 'relevance' if not specified.
@@ -169,6 +197,12 @@ class DocumentView(views.APIView):
page_size : int, optional
The number of results to return per page.
Defaults to 50 if not specified.
services: List[str], optional
List of services on which we intend to run the query (current service if left empty)
visited: List[sub], optional
List of public/authenticated documents the user has visited to limit
the document returned to the ones the current user has seen.
Built from linkreach list of a document in docs app.
Returns:
--------
@@ -176,13 +210,27 @@ class DocumentView(views.APIView):
- 200 OK: Returns a list of search results matching the query.
- 400 Bad Request: If the query parameter 'q' is not provided or invalid.
"""
# Get list of groups related to the user from SCIM provider (consider caching result)
audience = self._get_service_provider_audience()
user_sub = self.request.user.sub
groups = []
# //////////////////////////////////////////////////
# Extract and validate query parameters using Pydantic schema
params = schemas.SearchQueryParametersSchema(**request.GET)
params = schemas.SearchQueryParametersSchema(**request.data)
# Compute pagination parameters
from_value = (params.page_number - 1) * params.page_size
size_value = params.page_size
# Get index list for search query
try:
search_indices = self._get_opensearch_indices(
audience, services=params.services
)
except SuspiciousOperation as e:
return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST)
# Prepare the search query
search_body = {
"_source": enums.SOURCE_FIELDS, # limit the fields to return
@@ -218,23 +266,50 @@ class DocumentView(views.APIView):
{params.order_by: {"order": params.order_direction}}
)
# Filter by is_public if provided
if params.is_public is not None:
# Apply access control based on documents reach
search_body["query"]["bool"]["must"].append(
{
"bool": {
"should": [
# Access control on public & authenticated reach
{
"bool": {
"must_not": {
"term": {enums.REACH: enums.ReachEnum.RESTRICTED},
},
# Limit search to already visited documents.
"must": {
"terms": {
"_id": sorted(params.visited),
}
},
},
},
# Access control on restricted search : either user or group should match
{"term": {enums.USERS: user_sub}},
{"terms": {enums.GROUPS: groups}},
],
# At least one of the 2 optional should clauses must apply
"minimum_should_match": 1,
}
}
)
# Optional filter by reach if explicitly provided in the query
if params.reach is not None:
search_body["query"]["bool"]["filter"].append(
{"term": {enums.IS_PUBLIC: params.is_public}}
{"term": {enums.REACH: params.reach}}
)
# Filter by users and groups based on authentication$
# user = request.user
# groups = user.get_teams()
# Always filter out inactive documents
search_body["query"]["bool"]["filter"].append({"term": {"is_active": True}})
# search_body['query']['bool']['filter'].append({
# "terms": {"users": [user.sub]}
# })
response = client.search( # pylint: disable=unexpected-keyword-arg
index=",".join(search_indices),
body=search_body,
# Argument added by the query_params() decorator of opensearch and
# not in the method declaration.
ignore_unavailable=True,
)
# search_body['query']['bool']['filter'].append({
# "terms": {"groups": list(groups)}
# })
response = client.search(index=self.index_name, body=search_body)
return Response(response["hits"]["hits"], status=status.HTTP_200_OK)
+5 -1
View File
@@ -2,5 +2,9 @@
NB_OBJECTS = {
"documents": 1000,
"services": 5,
"services": 5
}
DEV_SERVICES = (
{"name": "docs", "client_id": "impress", "token": "find-api-key-for-docs-with-exactly-50-chars-length"},
)
@@ -14,7 +14,7 @@ from django.utils.text import slugify
from faker import Faker
from opensearchpy.helpers import bulk
from core import factories, opensearch
from core import enums, factories, opensearch
from demo import defaults
@@ -133,7 +133,7 @@ def generate_document():
"size": random.randint(0, 100 * 1024**2),
"users": [str(uuid4()) for _ in range(3)],
"groups": [slugify(fake.word()) for _ in range(3)],
"is_public": fake.boolean(),
"reach": random.choice(list(enums.ReachEnum)).value,
}
@@ -147,24 +147,31 @@ def create_demo(stdout):
services = factories.ServiceFactory.create_batch(
defaults.NB_OBJECTS["services"]
)
for service in services:
opensearch.ensure_index_exists(service.index_name)
opensearch.client.indices.refresh(index=service.index_name)
opensearch.ensure_index_exists(service.name)
opensearch.client.indices.refresh(index=service.name)
with Timeit(stdout, "Creating documents"):
actions = BulkIndexing(stdout)
for _ in range(defaults.NB_OBJECTS["documents"]):
service = random.choice(services)
document = generate_document()
actions.push(service.index_name, uuid4(), document)
actions.push(service.name, uuid4(), document)
actions.flush()
with Timeit(stdout, "Creating dev services"):
for conf in defaults.DEV_SERVICES:
service = factories.ServiceFactory(**conf)
opensearch.ensure_index_exists(service.name)
opensearch.client.indices.refresh(index=service.name)
# Check and report on indexed documents
total_indexed = 0
for service in services:
opensearch.client.indices.refresh(index=service.index_name)
indexed = opensearch.client.count(index=service.index_name)["count"]
stdout.write(f" - {service.index_name:s}: {indexed:d} documents")
opensearch.client.indices.refresh(index=service.name)
indexed = opensearch.client.count(index=service.name)["count"]
stdout.write(f" - {service.name:s}: {indexed:d} documents")
total_indexed += indexed
stdout.write(f" TOTAL: {total_indexed:d} documents")
+101 -36
View File
@@ -296,9 +296,13 @@ class Base(Configuration):
environ_name="OIDC_RP_CLIENT_SECRET",
environ_prefix=None,
)
OIDC_OP_URL = values.Value(None, environ_name="OIDC_OP_URL", environ_prefix=None)
OIDC_OP_JWKS_ENDPOINT = values.Value(
environ_name="OIDC_OP_JWKS_ENDPOINT", environ_prefix=None
)
OIDC_OP_INTROSPECTION_ENDPOINT = values.Value(
environ_name="OIDC_OP_INTROSPECTION_ENDPOINT", environ_prefix=None
)
OIDC_OP_AUTHORIZATION_ENDPOINT = values.Value(
environ_name="OIDC_OP_AUTHORIZATION_ENDPOINT", environ_prefix=None
)
@@ -317,6 +321,65 @@ class Base(Configuration):
OIDC_RP_SCOPES = values.Value(
"openid email", environ_name="OIDC_RP_SCOPES", environ_prefix=None
)
# OIDC - Resource server
OIDC_DRF_AUTH_BACKEND = values.Value(
"lasuite.oidc_login.backends.OIDCAuthenticationBackend",
environ_name="OIDC_DRF_AUTH_BACKEND",
environ_prefix=None,
)
OIDC_RS_BACKEND_CLASS = values.Value(
"core.authentication.FinderResourceServerBackend",
environ_name="OIDC_RS_BACKEND_CLASS",
environ_prefix=None,
)
OIDC_RS_AUDIENCE_CLAIM = values.Value(
"client_id",
environ_name="OIDC_RS_AUDIENCE_CLAIM",
environ_prefix=None,
)
OIDC_RS_CLIENT_ID = values.Value(
None, environ_name="OIDC_RS_CLIENT_ID", environ_prefix=None
)
OIDC_RS_CLIENT_SECRET = values.Value(
None,
environ_name="OIDC_RS_CLIENT_SECRET",
environ_prefix=None,
)
OIDC_RS_SIGNING_ALGO = values.Value(
default="ES256", environ_name="OIDC_RS_SIGNING_ALGO", environ_prefix=None
)
OIDC_RS_SCOPES = values.ListValue(
["openid"], environ_name="OIDC_RS_SCOPES", environ_prefix=None
)
OIDC_RS_PRIVATE_KEY_STR = values.Value(
default=None,
environ_name="OIDC_RS_PRIVATE_KEY_STR",
environ_prefix=None,
)
OIDC_RS_ENCRYPTION_KEY_TYPE = values.Value(
default="RSA",
environ_name="OIDC_RS_ENCRYPTION_KEY_TYPE",
environ_prefix=None,
)
OIDC_RS_ENCRYPTION_ALGO = values.Value(
default="RSA-OAEP",
environ_name="OIDC_RS_ENCRYPTION_ALGO",
environ_prefix=None,
)
OIDC_RS_ENCRYPTION_ENCODING = values.Value(
default="A256GCM",
environ_name="OIDC_RS_ENCRYPTION_ENCODING",
environ_prefix=None,
)
OIDC_VERIFY_SSL = values.BooleanValue(
True, environ_name="OIDC_VERIFY_SSL", environ_prefix=None
)
OIDC_TIMEOUT = values.Value(None, environ_name="OIDC_TIMEOUT", environ_prefix=None)
OIDC_PROXY = values.Value(None, environ_name="OIDC_PROXY", environ_prefix=None)
LOGIN_REDIRECT_URL = values.Value(
None, environ_name="LOGIN_REDIRECT_URL", environ_prefix=None
)
@@ -342,6 +405,44 @@ class Base(Configuration):
default=True, environ_name="ALLOW_LOGOUT_GET_METHOD", environ_prefix=None
)
# Logging
# We want to make it easy to log to console but by default we log production
# to Sentry and don't want to log to console.
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"simple": {
"format": "{asctime} {name} {levelname} {message}",
"style": "{",
},
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"formatter": "simple",
},
},
# Override root logger to send it to console
"root": {
"handlers": ["console"],
"level": values.Value(
"INFO", environ_name="LOGGING_LEVEL_LOGGERS_ROOT", environ_prefix=None
),
},
"loggers": {
"find": {
"handlers": ["console"],
"level": values.Value(
"INFO",
environ_name="LOGGING_LEVEL_LOGGERS_APP",
environ_prefix=None,
),
"propagate": True,
},
},
}
# pylint: disable=invalid-name
@property
def ENVIRONMENT(self):
@@ -423,23 +524,6 @@ class Development(Base):
class Test(Base):
"""Test environment settings"""
LOGGING = values.DictValue(
{
"version": 1,
"disable_existing_loggers": False,
"handlers": {
"console": {
"class": "logging.StreamHandler",
},
},
"loggers": {
"find": {
"handlers": ["console"],
"level": "DEBUG",
},
},
}
)
PASSWORD_HASHERS = [
"django.contrib.auth.hashers.MD5PasswordHasher",
]
@@ -511,25 +595,6 @@ class Production(Base):
},
}
LOGGING = values.DictValue(
{
"version": 1,
"disable_existing_loggers": False,
"handlers": {
"console": {
"class": "logging.StreamHandler",
},
},
"loggers": {
"find": {
"handlers": ["console"],
"level": "INFO",
"propagate": True,
},
},
}
)
class Feature(Production):
"""
+21 -20
View File
@@ -7,7 +7,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "find"
version = "1.1.0"
version = "0.0.1"
authors = [{ "name" = "DINUM", "email" = "dev@mail.numerique.gouv.fr" }]
classifiers = [
"Development Status :: 5 - Production/Stable",
@@ -25,23 +25,24 @@ license = { file = "LICENSE" }
readme = "README.md"
requires-python = ">=3.10"
dependencies = [
"celery[redis]==5.4.0",
"celery[redis]==5.5.3",
"django-configurations==2.5.1",
"django-cors-headers==4.6.0",
"django-cors-headers==4.7.0",
"redis==5.2.1",
"django-redis==5.4.0",
"django==5.1.4",
"djangorestframework==3.15.2",
"django-redis==6.0.0",
"django==5.2.6",
"django-lasuite[all]==0.0.14",
"djangorestframework==3.16.0",
"drf_spectacular==0.28.0",
"dockerflow==2024.4.2",
"factory_boy==3.3.1",
"gunicorn==23.0.0",
"mozilla-django-oidc==4.0.1",
"psycopg[binary]==3.2.3",
"psycopg[binary]==3.2.9",
"pydantic==2.10.5",
"pyjwt==2.10.1",
"requests==2.32.3",
"sentry-sdk==2.19.2",
"requests==2.32.4",
"sentry-sdk==2.32.0",
"url-normalize==1.4.3",
"opensearch-py==2.8.0",
"whitenoise==6.8.2",
@@ -55,22 +56,22 @@ dependencies = [
[project.optional-dependencies]
dev = [
"django-extensions==3.2.3",
"drf-spectacular-sidecar==2024.12.1",
"django-extensions==4.1",
"drf-spectacular-sidecar==2025.7.1",
"faker==33.3.0",
"ipdb==0.13.13",
"ipython==8.31.0",
"pyfakefs==5.7.3",
"pyfakefs==5.9.1",
"pylint-django==2.6.1",
"pylint==3.3.3",
"pytest-cov==6.0.0",
"pytest-django==4.9.0",
"pytest==8.3.4",
"pylint==3.3.7",
"pytest-cov==6.2.1",
"pytest-django==4.11.1",
"pytest==8.4.1",
"pytest-icdiff==0.9",
"pytest-xdist==3.6.1",
"responses==0.25.3",
"ruff==0.9.0",
"types-requests==2.32.0.20241016",
"pytest-xdist==3.8.0",
"responses==0.25.7",
"ruff==0.12.2",
"types-requests==2.32.4.20250611",
]
[tool.setuptools]