Compare commits

..

11 Commits

Author SHA1 Message Date
Jacques ROUSSEL 090bae6028 👷(helm) fix typo in the ci
Fix the following issue :
```
The workflow is not valid. .github/workflows/release-helmchart.yml
(Line: 25, Col: 12): Job 'release' depends on unknown job
'helmfile-lint'.
```
2025-01-09 18:12:55 +01:00
lebaudantoine 3e4a7058d2 ♻️(ci) merge helmfile-lint and release chart workflows
Merge both flows related to the Helmfile.
Also, make sure lint job is run before the release one.
2025-01-06 10:19:09 +01:00
Jacques ROUSSEL 25a4e2dfc6 👷(helm) improve local stack
Use the common create_cluster.sh in order to improve cooperation
between teams.

Also, mount extra volume, to avoid setting ssl_verify to false,
while using request module in Python.
2025-01-06 10:19:09 +01:00
Jacques ROUSSEL 14e83ecaff 📝(doc) add a self-host tutorial
Add a documentation to deploy a self-hosted visio instance in a
standalone way (without AI features)
2025-01-06 10:19:09 +01:00
Jacques ROUSSEL 8bd90bd2ff 👷(helm) add a github workflow to publish a chart
We have a dedicated deployment repository, also containing
the Helm chart. To avoid duplicating and maintaining twice
a chart, we decided to publish our Helm chart.

At first we tried the official chart releaser action, however,
this ended in creating a new release on Github for each chart
update, which wasn't acceptable.
2025-01-06 10:19:09 +01:00
Jacques ROUSSEL b51f127872 🔧(helm) offer a standalone dev environment
Offer a standalone dev environment or a dinum specific dev
environment with ProConnect authentication.

Needed to refactor the way secrets are managed in the project,
and also re-organize the Helm chart to make it totally standalone.

Particulary useful for external wanting to run the project.
Work done by @rouja.
2025-01-06 10:19:09 +01:00
lebaudantoine 4c0230d537 (backend) post email to marketing tools while signing up new users
Submitting new users to the marketing service is currently handled
during signup and is performed only once.

This is a pragmatic first implementation, yet imperfect.

In the future, this should be improved by delegating the call to a Celery
worker or an async task.
2024-12-31 15:09:51 +01:00
lebaudantoine 7309df4115 ♻️(backend) add MarketingService protocol and Brevo implementation
Introduced a MarketingService protocol for typed marketing operations,
allowing easy integration of alternative services.

Implemented a Brevo wrapper following the protocol to decouple
the codebase from the sdk. These implementations are simple and pragmatic.
Feel free to refactor them.
2024-12-31 15:09:51 +01:00
lebaudantoine 18b2dfc497 (backend) install brevo client
Brevo (ex-sendinblue) is a common solution used for marketing and
communications.
2024-12-31 15:09:51 +01:00
lebaudantoine 3282da7c56 🚨(backend) fix Django UnorderedObjectListWarning on models
Found this solution googling on Stack Overflow.

Without a default ordering on a model, Django raises a warning, that
pagination may yield inconsistent results.
2024-12-31 15:09:35 +01:00
renovate[bot] 7f8a6e8685 ⬆️(dependencies) update python dependencies 2024-12-30 13:32:37 +01:00
12 changed files with 532 additions and 28 deletions
+1 -15
View File
@@ -10,21 +10,7 @@ on:
- ./src/helm/meet/**
jobs:
lint-helmfile:
runs-on: ubuntu-latest
container:
image: ghcr.io/helmfile/helmfile:latest
steps:
- uses: numerique-gouv/action-helmfile-lint@main
with:
app-id: ${{ secrets.APP_ID }}
age-key: ${{ secrets.SOPS_PRIVATE }}
private-key: ${{ secrets.PRIVATE_KEY }}
helmfile-src: "src/helm"
repositories: "meet,secrets"
release:
needs: helmfile-lint
if: github.event_name == 'push'
permissions:
contents: write
@@ -47,5 +33,5 @@ jobs:
uses: numerique-gouv/helm-gh-pages@add-overwrite-option
with:
charts_dir: ./src/helm
linting: off
linting: on
token: ${{ secrets.GITHUB_TOKEN }}
+30 -1
View File
@@ -1,7 +1,7 @@
"""Authentication Backends for the Meet core app."""
from django.conf import settings
from django.core.exceptions import SuspiciousOperation
from django.core.exceptions import ImproperlyConfigured, SuspiciousOperation
from django.utils.translation import gettext_lazy as _
import requests
@@ -10,6 +10,11 @@ from mozilla_django_oidc.auth import (
)
from core.models import User
from core.services.marketing_service import (
ContactCreationError,
ContactData,
get_marketing_service,
)
class OIDCAuthenticationBackend(MozillaOIDCAuthenticationBackend):
@@ -86,6 +91,10 @@ class OIDCAuthenticationBackend(MozillaOIDCAuthenticationBackend):
password="!", # noqa: S106
**claims,
)
if settings.SIGNUP_NEW_USER_TO_MARKETING_EMAIL:
self.signup_to_marketing_email(email)
elif not user:
return None
@@ -96,6 +105,26 @@ class OIDCAuthenticationBackend(MozillaOIDCAuthenticationBackend):
return user
@staticmethod
def signup_to_marketing_email(email):
"""Pragmatic approach to newsletter signup during authentication flow.
Details:
1. Uses a very short timeout (1s) to prevent blocking the auth process
2. Silently fails if the marketing service is down/slow to prioritize user experience
3. Trade-off: May miss some signups but ensures auth flow remains fast
Note: For a more robust solution, consider using Async task processing (Celery/Django-Q)
"""
try:
marketing_service = get_marketing_service()
contact_data = ContactData(
email=email, attributes={"VISIO_SOURCE": ["SIGNIN"]}
)
marketing_service.create_contact(contact_data, timeout=1)
except (ContactCreationError, ImproperlyConfigured, ImportError):
pass
def get_existing_user(self, sub, email):
"""Fetch existing user by sub or email."""
try:
+2
View File
@@ -189,6 +189,7 @@ class User(AbstractBaseUser, BaseModel, auth_models.PermissionsMixin):
class Meta:
db_table = "meet_user"
ordering = ("-created_at",)
verbose_name = _("user")
verbose_name_plural = _("users")
@@ -304,6 +305,7 @@ class ResourceAccess(BaseModel):
class Meta:
db_table = "meet_resource_access"
ordering = ("-created_at",)
verbose_name = _("Resource access")
verbose_name_plural = _("Resource accesses")
constraints = [
@@ -0,0 +1,134 @@
"""Marketing service in charge of pushing data for marketing automation."""
import logging
from dataclasses import dataclass
from functools import lru_cache
from typing import Dict, List, Optional, Protocol
from django.conf import settings
from django.core.exceptions import ImproperlyConfigured
from django.utils.module_loading import import_string
import brevo_python
logger = logging.getLogger(__name__)
class ContactCreationError(Exception):
"""Raised when the contact creation fails."""
@dataclass
class ContactData:
"""Contact data for marketing service integration."""
email: str
attributes: Optional[Dict[str, str]] = None
list_ids: Optional[List[int]] = None
update_enabled: bool = True
class MarketingServiceProtocol(Protocol):
"""Interface for marketing automation service integrations."""
def create_contact(
self, contact_data: ContactData, timeout: Optional[int] = None
) -> dict:
"""Create or update a contact.
Args:
contact_data: Contact information and attributes
timeout: API request timeout in seconds
Returns:
dict: Service response
Raises:
ContactCreationError: If contact creation fails
"""
class BrevoMarketingService:
"""Brevo marketing automation integration.
Handles:
- Contact management and segmentation
- Marketing campaigns and automation
- Email communications
Configuration via Django settings:
- BREVO_API_KEY: API authentication
- BREVO_API_CONTACT_LIST_IDS: Default contact lists
- BREVO_API_CONTACT_ATTRIBUTES: Default contact attributes
"""
def __init__(self):
"""Initialize Brevo (ex-sendinblue) marketing service."""
if not settings.BREVO_API_KEY:
raise ImproperlyConfigured("Brevo API key is required")
configuration = brevo_python.Configuration()
configuration.api_key["api-key"] = settings.BREVO_API_KEY
self._api_client = brevo_python.ApiClient(configuration)
def create_contact(self, contact_data: ContactData, timeout=None) -> dict:
"""Create or update a Brevo contact.
Args:
contact_data: Contact information and attributes
timeout: API request timeout in seconds
Returns:
dict: Brevo API response
Raises:
ContactCreationError: If contact creation fails
ImproperlyConfigured: If required settings are missing
Note:
Contact attributes must be pre-configured in Brevo.
Changes to attributes can impact existing workflows.
"""
if not settings.BREVO_API_CONTACT_LIST_IDS:
raise ImproperlyConfigured(
"Default Brevo List IDs must be configured in settings."
)
contact_api = brevo_python.ContactsApi(self._api_client)
attributes = {
**settings.BREVO_API_CONTACT_ATTRIBUTES,
**(contact_data.attributes or {}),
}
list_ids = (contact_data.list_ids or []) + settings.BREVO_API_CONTACT_LIST_IDS
contact = brevo_python.CreateContact(
email=contact_data.email,
attributes=attributes,
list_ids=list_ids,
update_enabled=contact_data.update_enabled,
)
api_configurations = {}
if timeout is not None:
api_configurations["_request_timeout"] = timeout
try:
response = contact_api.create_contact(contact, **api_configurations)
except brevo_python.rest.ApiException as err:
logger.exception("Failed to create contact in Brevo")
raise ContactCreationError("Failed to create contact in Brevo") from err
return response
@lru_cache(maxsize=1)
def get_marketing_service() -> MarketingServiceProtocol:
"""Return cached instance of configured marketing service."""
marketing_service_cls = import_string(settings.MARKETING_SERVICE_CLASS)
return marketing_service_cls()
@@ -1,12 +1,15 @@
"""Unit tests for the Authentication Backends."""
from django.core.exceptions import SuspiciousOperation
from unittest import mock
from django.core.exceptions import ImproperlyConfigured, SuspiciousOperation
import pytest
from core import models
from core.authentication.backends import OIDCAuthenticationBackend
from core.factories import UserFactory
from core.services import marketing_service
pytestmark = pytest.mark.django_db
@@ -412,3 +415,139 @@ def test_update_user_when_no_update_needed(django_assert_num_queries, claims):
user.refresh_from_db()
assert user.email == "john.doe@example.com"
@mock.patch.object(OIDCAuthenticationBackend, "signup_to_marketing_email")
def test_marketing_signup_new_user_enabled(mock_signup, monkeypatch, settings):
"""Test marketing signup for new user with settings enabled."""
settings.SIGNUP_NEW_USER_TO_MARKETING_EMAIL = True
klass = OIDCAuthenticationBackend()
email = "test@example.com"
def get_userinfo_mocked(*args):
return {"sub": "123", "email": email}
monkeypatch.setattr(OIDCAuthenticationBackend, "get_userinfo", get_userinfo_mocked)
user = klass.get_or_create_user("test-token", None, None)
assert user.email == email
mock_signup.assert_called_once_with(email)
@mock.patch.object(OIDCAuthenticationBackend, "signup_to_marketing_email")
def test_marketing_signup_new_user_disabled(mock_signup, monkeypatch, settings):
"""Test no marketing signup for new user with settings disabled."""
settings.SIGNUP_NEW_USER_TO_MARKETING_EMAIL = False
klass = OIDCAuthenticationBackend()
email = "test@example.com"
def get_userinfo_mocked(*args):
return {"sub": "123", "email": email}
monkeypatch.setattr(OIDCAuthenticationBackend, "get_userinfo", get_userinfo_mocked)
user = klass.get_or_create_user("test-token", None, None)
assert user.email == email
mock_signup.assert_not_called()
@mock.patch.object(OIDCAuthenticationBackend, "signup_to_marketing_email")
def test_marketing_signup_new_user_default_disabled(mock_signup, monkeypatch):
"""Test no marketing signup for new user with settings by default disabled."""
klass = OIDCAuthenticationBackend()
email = "test@example.com"
def get_userinfo_mocked(*args):
return {"sub": "123", "email": email}
monkeypatch.setattr(OIDCAuthenticationBackend, "get_userinfo", get_userinfo_mocked)
user = klass.get_or_create_user("test-token", None, None)
assert user.email == email
mock_signup.assert_not_called()
@pytest.mark.parametrize(
"is_signup_enabled",
[True, False],
)
@mock.patch.object(OIDCAuthenticationBackend, "signup_to_marketing_email")
def test_marketing_signup_existing_user(
mock_signup, monkeypatch, settings, is_signup_enabled
):
"""Test no marketing signup for existing user regardless of settings."""
settings.SIGNUP_NEW_USER_TO_MARKETING_EMAIL = is_signup_enabled
klass = OIDCAuthenticationBackend()
db_user = UserFactory(email="test@example.com")
def get_userinfo_mocked(*args):
return {"sub": db_user.sub, "email": db_user.email}
monkeypatch.setattr(OIDCAuthenticationBackend, "get_userinfo", get_userinfo_mocked)
user = klass.get_or_create_user("test-token", None, None)
assert user == db_user
mock_signup.assert_not_called()
@mock.patch("core.authentication.backends.get_marketing_service")
def test_signup_to_marketing_email_success(mock_marketing):
"""Test successful marketing signup."""
email = "test@example.com"
# Call the method
OIDCAuthenticationBackend.signup_to_marketing_email(email)
# Verify service interaction
mock_service = mock_marketing.return_value
mock_service.create_contact.assert_called_once()
@pytest.mark.parametrize(
"error",
[
ImportError,
ImproperlyConfigured,
],
)
@mock.patch("core.authentication.backends.get_marketing_service")
def test_marketing_signup_handles_service_initialization_errors(
mock_marketing, error, settings
):
"""Tests errors that occur when trying to get/initialize the marketing service."""
settings.SIGNUP_NEW_USER_TO_MARKETING_EMAIL = True
mock_marketing.side_effect = error
# Should not raise any exception
OIDCAuthenticationBackend.signup_to_marketing_email("test@example.com")
@pytest.mark.parametrize(
"error",
[
marketing_service.ContactCreationError,
ImproperlyConfigured,
ImportError,
],
)
@mock.patch("core.authentication.backends.get_marketing_service")
def test_marketing_signup_handles_contact_creation_errors(
mock_marketing, error, settings
):
"""Tests errors that occur during the contact creation process."""
settings.SIGNUP_NEW_USER_TO_MARKETING_EMAIL = True
mock_marketing.return_value.create_contact.side_effect = error
# Should not raise any exception
OIDCAuthenticationBackend.signup_to_marketing_email("test@example.com")
@@ -0,0 +1,187 @@
"""
Test marketing services.
"""
# pylint: disable=W0621,W0613
from unittest import mock
from django.conf import settings
from django.core.exceptions import ImproperlyConfigured
import brevo_python
import pytest
from core.services.marketing_service import (
BrevoMarketingService,
ContactCreationError,
ContactData,
get_marketing_service,
)
def test_init_missing_api_key(settings):
"""Test initialization with missing API key."""
settings.BREVO_API_KEY = None
with pytest.raises(ImproperlyConfigured, match="Brevo API key is required"):
BrevoMarketingService()
def test_create_contact_missing_list_ids(settings):
"""Test contact creation with missing list IDs."""
settings.BREVO_API_KEY = "test-api-key"
settings.BREVO_API_CONTACT_LIST_IDS = None
settings.BREVO_API_CONTACT_ATTRIBUTES = {"source": "test"}
valid_contact_data = ContactData(
email="test@example.com",
attributes={"first_name": "Test"},
list_ids=[1, 2],
update_enabled=True,
)
brevo_service = BrevoMarketingService()
with pytest.raises(
ImproperlyConfigured, match="Default Brevo List IDs must be configured"
):
brevo_service.create_contact(valid_contact_data)
@mock.patch("brevo_python.ContactsApi")
def test_create_contact_success(mock_contact_api):
"""Test successful contact creation."""
mock_api = mock_contact_api.return_value
settings.BREVO_API_KEY = "test-api-key"
settings.BREVO_API_CONTACT_LIST_IDS = [1, 2, 3, 4]
settings.BREVO_API_CONTACT_ATTRIBUTES = {"source": "test"}
valid_contact_data = ContactData(
email="test@example.com",
attributes={"first_name": "Test"},
list_ids=[1, 2],
update_enabled=True,
)
brevo_service = BrevoMarketingService()
mock_api.create_contact.return_value = {"id": "test-id"}
response = brevo_service.create_contact(valid_contact_data)
assert response == {"id": "test-id"}
mock_api.create_contact.assert_called_once()
contact_arg = mock_api.create_contact.call_args[0][0]
assert contact_arg.email == "test@example.com"
assert contact_arg.attributes == {
**settings.BREVO_API_CONTACT_ATTRIBUTES,
**valid_contact_data.attributes,
}
assert set(contact_arg.list_ids) == {1, 2, 3, 4}
assert contact_arg.update_enabled is True
@mock.patch("brevo_python.ContactsApi")
def test_create_contact_with_timeout(mock_contact_api):
"""Test contact creation with timeout."""
mock_api = mock_contact_api.return_value
settings.BREVO_API_KEY = "test-api-key"
settings.BREVO_API_CONTACT_LIST_IDS = [1, 2, 3, 4]
settings.BREVO_API_CONTACT_ATTRIBUTES = {"source": "test"}
valid_contact_data = ContactData(
email="test@example.com",
attributes={"first_name": "Test"},
list_ids=[1, 2],
update_enabled=True,
)
brevo_service = BrevoMarketingService()
brevo_service.create_contact(valid_contact_data, timeout=30)
mock_api.create_contact.assert_called_once()
assert mock_api.create_contact.call_args[1]["_request_timeout"] == 30
@mock.patch("brevo_python.ContactsApi")
def test_create_contact_api_error(mock_contact_api):
"""Test contact creation API error handling."""
mock_api = mock_contact_api.return_value
settings.BREVO_API_KEY = "test-api-key"
settings.BREVO_API_CONTACT_LIST_IDS = [1, 2, 3, 4]
settings.BREVO_API_CONTACT_ATTRIBUTES = {"source": "test"}
valid_contact_data = ContactData(
email="test@example.com",
attributes={"first_name": "Test"},
list_ids=[1, 2],
update_enabled=True,
)
brevo_service = BrevoMarketingService()
mock_api.create_contact.side_effect = brevo_python.rest.ApiException()
with pytest.raises(ContactCreationError, match="Failed to create contact in Brevo"):
brevo_service.create_contact(valid_contact_data)
@pytest.fixture
def clear_marketing_cache():
"""Clear marketing service cache between tests."""
get_marketing_service.cache_clear()
yield
get_marketing_service.cache_clear()
def test_get_marketing_service_caching(clear_marketing_cache):
"""Test marketing service caching behavior."""
settings.BREVO_API_KEY = "test-api-key"
settings.MARKETING_SERVICE_CLASS = (
"core.services.marketing_service.BrevoMarketingService"
)
service1 = get_marketing_service()
service2 = get_marketing_service()
assert service1 is service2
assert isinstance(service1, BrevoMarketingService)
def test_get_marketing_service_invalid_class(clear_marketing_cache):
"""Test handling of invalid service class."""
settings.MARKETING_SERVICE_CLASS = "invalid.service.path"
with pytest.raises(ImportError):
get_marketing_service()
@mock.patch("core.services.marketing_service.import_string")
def test_service_instantiation_called_once(mock_import_string, clear_marketing_cache):
"""Test service class is instantiated only once."""
settings.BREVO_API_KEY = "test-api-key"
settings.MARKETING_SERVICE_CLASS = (
"core.services.marketing_service.BrevoMarketingService"
)
get_marketing_service.cache_clear()
mock_service_cls = mock.Mock()
mock_service_instance = mock.Mock()
mock_service_cls.return_value = mock_service_instance
mock_import_string.return_value = mock_service_cls
service1 = get_marketing_service()
service2 = get_marketing_service()
mock_import_string.assert_called_once_with(settings.MARKETING_SERVICE_CLASS)
mock_service_cls.assert_called_once()
assert service1 is service2
assert service1 is mock_service_instance
+1 -1
View File
@@ -66,7 +66,7 @@ def test_api_users_list_query_email():
assert response.status_code == 200
user_ids = [user["id"] for user in response.json()["results"]]
assert user_ids == [str(nicole.id), str(frank.id)]
assert user_ids == [str(frank.id), str(nicole.id)]
def test_api_users_retrieve_me_anonymous():
+26
View File
@@ -457,6 +457,32 @@ class Base(Configuration):
None, environ_name="SUMMARY_SERVICE_API_TOKEN", environ_prefix=None
)
# Marketing and communication settings
SIGNUP_NEW_USER_TO_MARKETING_EMAIL = values.BooleanValue(
False,
environ_name="SIGNUP_NEW_USERS_TO_NEWSLETTER",
environ_prefix=None,
help_text=(
"When enabled, new users are automatically added to mailing list "
"for product updates, marketing communications, and customized emails. "
),
)
MARKETING_SERVICE_CLASS = values.Value(
"core.services.marketing_service.BrevoMarketingService",
environ_name="MARKETING_SERVICE_CLASS",
environ_prefix=None,
)
BREVO_API_KEY = values.Value(
None, environ_name="BREVO_API_KEY", environ_prefix=None
)
BREVO_API_CONTACT_LIST_IDS = values.ListValue(
[],
environ_name="BREVO_API_CONTACT_LIST_IDS",
environ_prefix=None,
converter=lambda x: int(x), # pylint: disable=unnecessary-lambda
)
BREVO_API_CONTACT_ATTRIBUTES = values.DictValue({"VISIO_USER": True})
# pylint: disable=invalid-name
@property
def ENVIRONMENT(self):
+8 -7
View File
@@ -25,8 +25,9 @@ license = { file = "LICENSE" }
readme = "README.md"
requires-python = ">=3.10"
dependencies = [
"boto3==1.35.76",
"boto3==1.35.90",
"Brotli==1.1.0",
"brevo-python==1.1.2",
"celery[redis]==5.4.0",
"django-configurations==2.5.1",
"django-cors-headers==4.6.0",
@@ -56,8 +57,8 @@ dependencies = [
"WeasyPrint>=60.2",
"whitenoise==6.8.2",
"mozilla-django-oidc==4.0.1",
"livekit-api==0.8.0",
"aiohttp==3.11.10",
"livekit-api==0.8.1",
"aiohttp==3.11.11",
]
[project.urls]
@@ -72,17 +73,17 @@ dev = [
"drf-spectacular-sidecar==2024.12.1",
"freezegun==1.5.1",
"ipdb==0.13.13",
"ipython==8.30.0",
"pyfakefs==5.7.2",
"ipython==8.31.0",
"pyfakefs==5.7.3",
"pylint-django==2.6.1",
"pylint==3.3.2",
"pylint==3.3.3",
"pytest-cov==6.0.0",
"pytest-django==4.9.0",
"pytest==8.3.4",
"pytest-icdiff==0.9",
"pytest-xdist==3.6.1",
"responses==0.25.3",
"ruff==0.8.2",
"ruff==0.8.4",
"types-requests==2.32.0.20241016",
]
+3 -3
View File
@@ -9,15 +9,15 @@ dependencies = [
"pydantic-settings>=2.1.0",
"celery==5.4.0",
"redis==5.2.1",
"minio==7.2.12",
"openai==1.57.1",
"minio==7.2.13",
"openai==1.58.1",
"requests==2.32.3",
"sentry-sdk[fastapi, celery]==2.19.0",
]
[project.optional-dependencies]
dev = [
"ruff==0.8.2",
"ruff==0.8.4",
]
[build-system]