Compare commits
17 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 lebaudantoine
|
1f603ce17b | ||
|
Arnaud Robin
|
672ca7dfe4 | ||
|
Arnaud Robin
|
7405011cd2 | ||
|
Arnaud Robin
|
1d5aebcfdc | ||
|
Arnaud Robin
|
0d6881382b | ||
|
Arnaud Robin
|
c428f39456 | ||
|
Arnaud Robin
|
cd9b80b966 | ||
|
Florent
|
f80603b4bb | ||
|
Berry den Hartog
|
3299ae7c39 | ||
|
Jacques ROUSSEL
|
93ca4f2bf4 | ||
|
lebaudantoine
|
19a4e442c5 | ||
|
lebaudantoine
|
53f1c9c1d4 | ||
|
lebaudantoine
|
28538b63da | ||
|
renovate[bot]
|
94ae5d52c2 | ||
|
lebaudantoine
|
6b38a3c996 | ||
|
lebaudantoine
|
0af30ec366 | ||
|
lebaudantoine
|
33e017464f |
@@ -119,12 +119,9 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
if: github.event_name != 'pull_request'
|
||||
steps:
|
||||
-
|
||||
name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
-
|
||||
name: Call argocd github webhook
|
||||
run: |
|
||||
data='{"ref": "'$GITHUB_REF'","repository": {"html_url":"'$GITHUB_SERVER_URL'/numerique-gouv/lasuite-deploiement"}}'
|
||||
sig=$(echo -n ${data} | openssl dgst -sha1 -hmac "${{ secrets.ARGOCD_PREPROD_WEBHOOK_SECRET }}" | awk '{print "X-Hub-Signature: sha1="$2}')
|
||||
curl -X POST -H 'X-GitHub-Event:push' -H "Content-Type: application/json" -H "${sig}" --data "${data}" ${{ vars.ARGOCD_PREPROD_WEBHOOK_URL }}
|
||||
- uses: numerique-gouv/action-argocd-webhook-notification@main
|
||||
id: notify
|
||||
with:
|
||||
deployment_repo_path: "${{ secrets.DEPLOYMENT_REPO_URL }}"
|
||||
argocd_webhook_secret: "${{ secrets.ARGOCD_PREPROD_WEBHOOK_SECRET }}"
|
||||
argocd_url: "${{ vars.ARGOCD_PREPROD_WEBHOOK_URL }}"
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2021-2023 DINUM/Etalab
|
||||
Copyright (c) 2024-2025 DINUM/Etalab
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2021-2023 DINUM/Etalab
|
||||
Copyright (c) 2024-2025 DINUM/Etalab
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
FROM livekit/livekit-server:latest
|
||||
FROM livekit/livekit-server:v1.8.0
|
||||
|
||||
# We inject the nip.io certificate manually because the livekit chart doesn't support volume mounting
|
||||
COPY rootCA.pem /etc/ssl/certs/
|
||||
|
||||
+106
-9
@@ -2,7 +2,6 @@
|
||||
|
||||
This document is a step-by-step guide that describes how to install Visio on a k8s cluster without AI features.
|
||||
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- k8s cluster with an nginx-ingress controller
|
||||
@@ -43,11 +42,11 @@ It will expire on 23 March 2027 🗓
|
||||
2. Create kind cluster with containerd registry config dir enabled
|
||||
Creating cluster "visio" ...
|
||||
✓ Ensuring node image (kindest/node:v1.27.3) 🖼
|
||||
✓ Preparing nodes 📦
|
||||
✓ Writing configuration 📜
|
||||
✓ Starting control-plane 🕹️
|
||||
✓ Installing CNI 🔌
|
||||
✓ Installing StorageClass 💾
|
||||
✓ Preparing nodes 📦
|
||||
✓ Writing configuration 📜
|
||||
✓ Starting control-plane 🕹️
|
||||
✓ Installing CNI 🔌
|
||||
✓ Installing StorageClass 💾
|
||||
Set kubectl context to "kind-visio"
|
||||
You can now use your cluster with:
|
||||
|
||||
@@ -95,13 +94,14 @@ ingress-nginx-admission-create-jgnc9 0/1 Completed 0 2m44s
|
||||
ingress-nginx-admission-patch-wrt47 0/1 Completed 0 2m44s
|
||||
ingress-nginx-controller-57c548c4cd-9xwt6 1/1 Running 0 2m44s
|
||||
```
|
||||
When your k8s cluster is ready, you can start the deployment. This cluster is special because it uses the *.127.0.0.1.nip.io domain and mkcert certificates to have full HTTPS support and easy domain name management.
|
||||
|
||||
Please remember that *.127.0.0.1.nip.io will always resolve to 127.0.0.1, except in the k8s cluster where we configure CoreDNS to answer with the ingress-nginx service IP.
|
||||
When your k8s cluster is ready, you can start the deployment. This cluster is special because it uses the \*.127.0.0.1.nip.io domain and mkcert certificates to have full HTTPS support and easy domain name management.
|
||||
|
||||
Please remember that \*.127.0.0.1.nip.io will always resolve to 127.0.0.1, except in the k8s cluster where we configure CoreDNS to answer with the ingress-nginx service IP.
|
||||
|
||||
## Preparation
|
||||
|
||||
### What will you use to authenticate your users ?
|
||||
### What will you use to authenticate your users ?
|
||||
|
||||
Visio uses OIDC, so if you already have an OIDC provider, obtain the necessary information to use it. In the next step, we will see how to configure Django (and thus Visio) to use it. If you do not have a provider, we will show you how to deploy a local Keycloak instance (this is not a production deployment, just a demo).
|
||||
|
||||
@@ -153,6 +153,7 @@ keycloak-0 1/1 Running 0 26m
|
||||
keycloak-postgresql-0 1/1 Running 0 26m
|
||||
redis-master-0 1/1 Running 0 35s
|
||||
```
|
||||
|
||||
When the redis is ready we can deploy livekit-server.
|
||||
|
||||
```
|
||||
@@ -194,6 +195,7 @@ livekit-livekit-server-5c5fb87f7f-ct6x5 1/1 Running 0 15m
|
||||
postgresql-0 1/1 Running 0 50s
|
||||
redis-master-0 1/1 Running 0 19
|
||||
```
|
||||
|
||||
From here important information you will need are :
|
||||
|
||||
```
|
||||
@@ -231,3 +233,98 @@ meet-admin <none> meet.127.0.0.1.nip.io localhost 80, 44
|
||||
```
|
||||
|
||||
You can use Visio on https://meet.127.0.0.1.nip.io. The provisioning user in keycloak is meet/meet.
|
||||
|
||||
## All options
|
||||
|
||||
These are the environmental options available on meet backend.
|
||||
|
||||
| Option | Description | default |
|
||||
| ----------------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| DJANGO_ALLOWED_HOSTS | Host that are allowed | [] |
|
||||
| DJANGO_SECRET_KEY | Secret key used | |
|
||||
| DJANGO_SILENCED_SYSTEM_CHECKS | Silence system checks | [] |
|
||||
| DJANGO_ALLOW_UNSECURE_USER_LISTING | | false |
|
||||
| DB_ENGINE | Database engine used | django.db.backends.postgresql_psycopg2 |
|
||||
| DB_NAME | name of the database | meet |
|
||||
| DB_USER | user used to connect to database | dinum |
|
||||
| DB_PASSWORD | password used to connect to the database | pass |
|
||||
| DB_HOST | hostname of the database | localhost |
|
||||
| DB_PORT | port to connect to database | 5432 |
|
||||
| STORAGES_STATICFILES_BACKEND | Static file serving engine | whitenoise.storage.CompressedManifestStaticFilesStorage |
|
||||
| AWS_S3_ENDPOINT_URL | S3 host endpoint | |
|
||||
| AWS_S3_ACCESS_KEY_ID | s3 access key | |
|
||||
| AWS_S3_SECRET_ACCESS_KEY | s3 secret key | |
|
||||
| AWS_S3_REGION_NAME | s3 region | |
|
||||
| AWS_STORAGE_BUCKET_NAME | s3 bucket name | |
|
||||
| DJANGO_LANGUAGE_CODE | Default language | en-us |
|
||||
| REDIS_URL | redis endpoint | redis://redis:6379/1 |
|
||||
| REQUEST_ENTRY_THROTTLE_RATES | throttle rates | 150/minute |
|
||||
| SPECTACULAR_SETTINGS_ENABLE_DJANGO_DEPLOY_CHECK | deploy check | False |
|
||||
| FRONTEND_ANALYTICS | analytics information | {} |
|
||||
| FRONTEND_SUPPORT | crisp frontend support | {} |
|
||||
| FRONTEND_SILENCE_LIVEKIT_DEBUG | silence livekit debug | false |
|
||||
| DJANGO_EMAIL_BACKEND | email backend library | django.core.mail.backends.smtp.EmailBackend |
|
||||
| DJANGO_EMAIL_HOST | host of the email server | |
|
||||
| DJANGO_EMAIL_HOST_USER | user to connect to the email server | |
|
||||
| DJANGO_EMAIL_HOST_PASSWORD | password to connect tto the email server | |
|
||||
| DJANGO_EMAIL_PORT | por tot connect to the email server | |
|
||||
| DJANGO_EMAIL_USE_TLS | enable tls on email connection | false |
|
||||
| DJANGO_EMAIL_USE_SSL | enable tls on email connection | false |
|
||||
| DJANGO_EMAIL_FROM | email from account | from@example.com |
|
||||
| DJANGO_CORS_ALLOW_ALL_ORIGINS | allow all cors origins | true |
|
||||
| DJANGO_CORS_ALLOWED_ORIGINS | origins to allow in string | [] |
|
||||
| DJANGO_CORS_ALLOWED_ORIGIN_REGEXES | origins to allow in regex | [] |
|
||||
| SENTRY_DSN | sentry server | |
|
||||
| DJANGO_CELERY_BROKER_URL | celery broker host | redis://redis:6379/0 |
|
||||
| DJANGO_CELERY_BROKER_TRANSPORT_OPTIONS | celery broker options | {} |
|
||||
| OIDC_CREATE_USER | create oidc user if not exists | false |
|
||||
| OIDC_VERIFY_SSL | verify ssl for oidc | true |
|
||||
| OIDC_FALLBACK_TO_EMAIL_FOR_IDENTIFICATION | fallback to email for identification | false |
|
||||
| OIDC_RP_SIGN_ALGO | token verification algoritm used by oidc | RS256 |
|
||||
| OIDC_RP_CLIENT_ID | oidc client | meet |
|
||||
| OIDC_RP_CLIENT_SECRET | oidc client secret | |
|
||||
| OIDC_OP_JWKS_ENDPOINT | oidc endpoint for JWKS | |
|
||||
| OIDC_OP_AUTHORIZATION_ENDPOINT | oidc endpoint for authorization | |
|
||||
| OIDC_OP_TOKEN_ENDPOINT | oidc endpoint for token | |
|
||||
| OIDC_OP_USER_ENDPOINT | oidc endpoint for user | |
|
||||
| OIDC_OP_LOGOUT_ENDPOINT | oidc endpoint for logout | |
|
||||
| OIDC_AUTH_REQUEST_EXTRA_PARAMS | extra parameters for oidc request | |
|
||||
| OIDC_RP_SCOPES | oidc scopes | openid email |
|
||||
| LOGIN_REDIRECT_URL | login redirect url | |
|
||||
| LOGIN_REDIRECT_URL_FAILURE | login redurect url for failure | |
|
||||
| LOGOUT_REDIRECT_URL | url to redirect to on logout | |
|
||||
| OIDC_USE_NONCE | use nonce for oidc | true |
|
||||
| OIDC_REDIRECT_REQUIRE_HTTPS | require https for oidc | false |
|
||||
| OIDC_REDIRECT_ALLOWED_HOSTS | allowed redirect hosts for oidc | [] |
|
||||
| OIDC_STORE_ID_TOKEN | store oidc ID token | true |
|
||||
| ALLOW_LOGOUT_GET_METHOD | allow logout through get method | true |
|
||||
| OIDC_REDIRECT_FIELD_NAME | direct field for oidc | returnTo |
|
||||
| OIDC_USERINFO_FULLNAME_FIELDS | full name claim from OIDC token | ["given_name", "usual_name"] |
|
||||
| OIDC_USERINFO_SHORTNAME_FIELD | shortname claim from OIDC token | given_name |
|
||||
| LIVEKIT_API_KEY | livekit api key | |
|
||||
| LIVEKIT_API_SECRET | livekit api secret | |
|
||||
| LIVEKIT_API_URL | livekit api url | |
|
||||
| RESOURCE_DEFAULT_ACCESS_LEVEL | default resource access level for rooms | public |
|
||||
| ALLOW_UNREGISTERED_ROOMS | Allow usage of unregistered rooms | true |
|
||||
| RECORDING_ENABLE | record meeting option | false |
|
||||
| RECORDING_OUTPUT_FOLDER | folder to store meetings | recordings |
|
||||
| RECORDING_VERIFY_SSL | verify ssl for recording storage | true |
|
||||
| RECORDING_WORKER_CLASSES | worker classes for recording | {"screen_recording": "core.recording.worker.services.VideoCompositeEgressService","transcript": "core.recording.worker.services.AudioCompositeEgressService"} |
|
||||
| RECORDING_EVENT_PARSER_CLASS | storage event engine for recording | core.recording.event.parsers.MinioParser |
|
||||
| RECORDING_ENABLE_STORAGE_EVENT_AUTH | enable storage event authorization | true |
|
||||
| RECORDING_STORAGE_EVENT_ENABLE | enable recording storage events | false |
|
||||
| RECORDING_STORAGE_EVENT_TOKEN | recording storage event token | |
|
||||
| SUMMARY_SERVICE_ENDPOINT | summary service endpoint | |
|
||||
| SUMMARY_SERVICE_API_TOKEN | api token voor summary service | |
|
||||
| SIGNUP_NEW_USER_TO_MARKETING_EMAIL | signup users to marketing emails | false |
|
||||
| MARKETING_SERVICE_CLASS | markering class | core.services.marketing.BrevoMarketingService |
|
||||
| BREVO_API_KEY | breva api key for marketing emails | |
|
||||
| BREVO_API_CONTACT_LIST_IDS | brevo api contact list ids | [] |
|
||||
| DJANGO_BREVO_API_CONTACT_ATTRIBUTES | brevo contact attributes | {"VISIO_USER": True} |
|
||||
| BREVO_API_TIMEOUT | brevo timeout | 1 |
|
||||
| LOBBY_KEY_PREFIX | lobby prefix | room_lobby |
|
||||
| LOBBY_WAITING_TIMEOUT | lobby waiting tumeout | 3 |
|
||||
| LOBBY_DENIED_TIMEOUT | lobby deny timeout | 5 |
|
||||
| LOBBY_ACCEPTED_TIMEOUT | lobby accept timeout | 21600 |
|
||||
| LOBBY_NOTIFICATION_TYPE | lobby notification types | participantWaiting |
|
||||
| LOBBY_COOKIE_NAME | lobby cookie name | lobbyParticipantId |
|
||||
|
||||
@@ -121,7 +121,9 @@ class OIDCAuthenticationBackend(MozillaOIDCAuthenticationBackend):
|
||||
contact_data = ContactData(
|
||||
email=email, attributes={"VISIO_SOURCE": ["SIGNIN"]}
|
||||
)
|
||||
marketing_service.create_contact(contact_data, timeout=1)
|
||||
marketing_service.create_contact(
|
||||
contact_data, timeout=settings.BREVO_API_TIMEOUT
|
||||
)
|
||||
except (ContactCreationError, ImproperlyConfigured, ImportError):
|
||||
pass
|
||||
|
||||
|
||||
@@ -12,7 +12,12 @@ from django.conf import settings
|
||||
from django.core.cache import cache
|
||||
|
||||
from asgiref.sync import async_to_sync
|
||||
from livekit.api import LiveKitAPI, SendDataRequest, TwirpError # pylint: disable=E0611
|
||||
from livekit.api import ( # pylint: disable=E0611
|
||||
ListRoomsRequest,
|
||||
LiveKitAPI,
|
||||
SendDataRequest,
|
||||
TwirpError,
|
||||
)
|
||||
|
||||
from core import models, utils
|
||||
|
||||
@@ -343,7 +348,18 @@ class LobbyService:
|
||||
}
|
||||
|
||||
lkapi = LiveKitAPI(**settings.LIVEKIT_CONFIGURATION)
|
||||
|
||||
try:
|
||||
room_response = await lkapi.room.list_rooms(
|
||||
ListRoomsRequest(
|
||||
names=[str(room_id)],
|
||||
)
|
||||
)
|
||||
|
||||
# Check if the room exists
|
||||
if not room_response.rooms:
|
||||
return
|
||||
|
||||
await lkapi.room.send_data(
|
||||
SendDataRequest(
|
||||
room=str(room_id),
|
||||
|
||||
@@ -776,6 +776,43 @@ def test_update_participant_status_success(mock_cache, lobby_service, participan
|
||||
lobby_service._get_cache_key.assert_called_once_with(room.id, participant_id)
|
||||
|
||||
|
||||
@mock.patch("core.services.lobby.LiveKitAPI")
|
||||
def test_notify_participants_success_no_room(mock_livekit_api, lobby_service):
|
||||
"""Test the notify_participants method when the LiveKit room doesn't exist yet."""
|
||||
room = RoomFactory(access_level=RoomAccessLevel.RESTRICTED)
|
||||
|
||||
# Set up the mock LiveKitAPI and its behavior
|
||||
mock_api_instance = mock.Mock()
|
||||
mock_api_instance.room = mock.Mock()
|
||||
mock_api_instance.room.send_data = mock.AsyncMock()
|
||||
|
||||
# Create a proper response object with an empty rooms list
|
||||
class MockResponse:
|
||||
"""LiveKit API response mock with empty rooms list."""
|
||||
|
||||
rooms = []
|
||||
|
||||
mock_api_instance.room.list_rooms = mock.AsyncMock(return_value=MockResponse())
|
||||
mock_api_instance.aclose = mock.AsyncMock()
|
||||
mock_livekit_api.return_value = mock_api_instance
|
||||
|
||||
# Act
|
||||
lobby_service.notify_participants(room.id)
|
||||
|
||||
# Assert
|
||||
# Verify the API was initialized with correct configuration
|
||||
mock_livekit_api.assert_called_once_with(**settings.LIVEKIT_CONFIGURATION)
|
||||
|
||||
# Verify that the service checked for existing rooms
|
||||
mock_api_instance.room.list_rooms.assert_called_once()
|
||||
|
||||
# Verify the send_data method was not called since no room exists
|
||||
mock_api_instance.room.send_data.assert_not_called()
|
||||
|
||||
# Verify the connection was properly closed
|
||||
mock_api_instance.aclose.assert_called_once()
|
||||
|
||||
|
||||
@mock.patch("core.services.lobby.LiveKitAPI")
|
||||
def test_notify_participants_success(mock_livekit_api, lobby_service):
|
||||
"""Test successful participant notification."""
|
||||
@@ -784,6 +821,14 @@ def test_notify_participants_success(mock_livekit_api, lobby_service):
|
||||
mock_api_instance = mock.Mock()
|
||||
mock_api_instance.room = mock.Mock()
|
||||
mock_api_instance.room.send_data = mock.AsyncMock()
|
||||
|
||||
class MockResponse:
|
||||
"""LiveKit API response mock with non-empty rooms list."""
|
||||
|
||||
rooms = ["room-1"]
|
||||
|
||||
mock_api_instance.room.list_rooms = mock.AsyncMock(return_value=MockResponse())
|
||||
|
||||
mock_api_instance.aclose = mock.AsyncMock()
|
||||
mock_livekit_api.return_value = mock_api_instance
|
||||
|
||||
@@ -793,6 +838,9 @@ def test_notify_participants_success(mock_livekit_api, lobby_service):
|
||||
# Verify the API was called correctly
|
||||
mock_livekit_api.assert_called_once_with(**settings.LIVEKIT_CONFIGURATION)
|
||||
|
||||
# Verify that the service checked for existing rooms
|
||||
mock_api_instance.room.list_rooms.assert_called_once()
|
||||
|
||||
# Verify the send_data method was called
|
||||
mock_api_instance.room.send_data.assert_called_once()
|
||||
send_data_request = mock_api_instance.room.send_data.call_args[0][0]
|
||||
@@ -817,6 +865,14 @@ def test_notify_participants_error(mock_livekit_api, lobby_service):
|
||||
mock_api_instance.room.send_data = mock.AsyncMock(
|
||||
side_effect=TwirpError(msg="test error", code=123)
|
||||
)
|
||||
|
||||
class MockResponse:
|
||||
"""LiveKit API response mock with non-empty rooms list."""
|
||||
|
||||
rooms = ["room-1"]
|
||||
|
||||
mock_api_instance.room.list_rooms = mock.AsyncMock(return_value=MockResponse())
|
||||
|
||||
mock_api_instance.aclose = mock.AsyncMock()
|
||||
mock_livekit_api.return_value = mock_api_instance
|
||||
|
||||
@@ -829,6 +885,9 @@ def test_notify_participants_error(mock_livekit_api, lobby_service):
|
||||
# Verify the API was called correctly
|
||||
mock_livekit_api.assert_called_once_with(**settings.LIVEKIT_CONFIGURATION)
|
||||
|
||||
# Verify that the service checked for existing rooms
|
||||
mock_api_instance.room.list_rooms.assert_called_once()
|
||||
|
||||
# Verify send_data was called
|
||||
mock_api_instance.room.send_data.assert_called_once()
|
||||
|
||||
|
||||
@@ -498,6 +498,9 @@ class Base(Configuration):
|
||||
converter=lambda x: int(x), # pylint: disable=unnecessary-lambda
|
||||
)
|
||||
BREVO_API_CONTACT_ATTRIBUTES = values.DictValue({"VISIO_USER": True})
|
||||
BREVO_API_TIMEOUT = values.PositiveIntegerValue(
|
||||
1, environ_name="BREVO_API_TIMEOUT", environ_prefix=None
|
||||
)
|
||||
|
||||
# Lobby configurations
|
||||
LOBBY_KEY_PREFIX = values.Value(
|
||||
|
||||
@@ -7,7 +7,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "meet"
|
||||
version = "0.1.15"
|
||||
version = "0.1.17"
|
||||
authors = [{ "name" = "DINUM", "email" = "dev@mail.numerique.gouv.fr" }]
|
||||
classifiers = [
|
||||
"Development Status :: 5 - Production/Stable",
|
||||
|
||||
Generated
+1274
-2069
File diff suppressed because it is too large
Load Diff
+17
-17
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "meet",
|
||||
"private": true,
|
||||
"version": "0.1.15",
|
||||
"version": "0.1.17",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "panda codegen && vite",
|
||||
@@ -16,45 +16,45 @@
|
||||
"@livekit/components-react": "2.8.1",
|
||||
"@livekit/components-styles": "1.1.4",
|
||||
"@livekit/track-processors": "0.3.3",
|
||||
"@pandacss/preset-panda": "0.53.0",
|
||||
"@react-aria/toast": "3.0.0-beta.19",
|
||||
"@pandacss/preset-panda": "0.53.3",
|
||||
"@react-aria/toast": "3.0.1",
|
||||
"@remixicon/react": "4.6.0",
|
||||
"@tanstack/react-query": "5.67.1",
|
||||
"@tanstack/react-query": "5.69.0",
|
||||
"crisp-sdk-web": "1.0.25",
|
||||
"hoofd": "1.7.3",
|
||||
"i18next": "24.2.2",
|
||||
"i18next": "24.2.3",
|
||||
"i18next-browser-languagedetector": "8.0.4",
|
||||
"i18next-parser": "9.3.0",
|
||||
"i18next-resources-to-backend": "1.2.1",
|
||||
"livekit-client": "2.9.5",
|
||||
"posthog-js": "1.225.1",
|
||||
"livekit-client": "2.9.8",
|
||||
"posthog-js": "1.232.6",
|
||||
"react": "18.3.1",
|
||||
"react-aria-components": "1.6.0",
|
||||
"react-aria-components": "1.7.1",
|
||||
"react-dom": "18.3.1",
|
||||
"react-i18next": "15.1.1",
|
||||
"use-sound": "5.0.0",
|
||||
"valtio": "2.1.3",
|
||||
"valtio": "2.1.4",
|
||||
"wouter": "3.6.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@pandacss/dev": "0.53.0",
|
||||
"@tanstack/eslint-plugin-query": "5.66.1",
|
||||
"@tanstack/react-query-devtools": "5.67.1",
|
||||
"@types/node": "22.13.9",
|
||||
"@pandacss/dev": "0.53.3",
|
||||
"@tanstack/eslint-plugin-query": "5.68.0",
|
||||
"@tanstack/react-query-devtools": "5.69.0",
|
||||
"@types/node": "22.13.13",
|
||||
"@types/react": "18.3.12",
|
||||
"@types/react-dom": "18.3.1",
|
||||
"@typescript-eslint/eslint-plugin": "8.26.0",
|
||||
"@typescript-eslint/parser": "8.26.0",
|
||||
"@typescript-eslint/eslint-plugin": "8.28.0",
|
||||
"@typescript-eslint/parser": "8.28.0",
|
||||
"@vitejs/plugin-react": "4.3.4",
|
||||
"eslint": "8.57.0",
|
||||
"eslint-config-prettier": "10.0.2",
|
||||
"eslint-config-prettier": "10.1.1",
|
||||
"eslint-plugin-jsx-a11y": "6.10.2",
|
||||
"eslint-plugin-react-hooks": "5.2.0",
|
||||
"eslint-plugin-react-refresh": "0.4.19",
|
||||
"postcss": "8.5.3",
|
||||
"prettier": "3.5.3",
|
||||
"typescript": "5.8.2",
|
||||
"vite": "6.2.0",
|
||||
"vite": "6.2.3",
|
||||
"vite-tsconfig-paths": "5.1.4"
|
||||
}
|
||||
}
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 20 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 457 B |
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
@@ -48,7 +48,7 @@ export const useUser = (
|
||||
const isLoggedOut = isLoggedIn === false
|
||||
|
||||
return {
|
||||
...query,
|
||||
refetch: query.refetch,
|
||||
user: isLoggedOut ? undefined : (query.data as ApiUser | undefined),
|
||||
isLoggedIn,
|
||||
logout,
|
||||
|
||||
@@ -298,7 +298,7 @@ export const Join = ({
|
||||
case ApiLobbyStatus.TIMEOUT:
|
||||
return (
|
||||
<VStack alignItems="center" textAlign="center">
|
||||
<H lvl={1} margin={false}>
|
||||
<H lvl={1} margin={false} centered>
|
||||
{t('timeoutInvite.title')}
|
||||
</H>
|
||||
<Text as="p" variant="note">
|
||||
@@ -310,7 +310,7 @@ export const Join = ({
|
||||
case ApiLobbyStatus.DENIED:
|
||||
return (
|
||||
<VStack alignItems="center" textAlign="center">
|
||||
<H lvl={1} margin={false}>
|
||||
<H lvl={1} margin={false} centered>
|
||||
{t('denied.title')}
|
||||
</H>
|
||||
<Text as="p" variant="note">
|
||||
@@ -322,7 +322,7 @@ export const Join = ({
|
||||
case ApiLobbyStatus.WAITING:
|
||||
return (
|
||||
<VStack alignItems="center" textAlign="center">
|
||||
<H lvl={1} margin={false}>
|
||||
<H lvl={1} margin={false} centered>
|
||||
{t('waiting.title')}
|
||||
</H>
|
||||
<Text
|
||||
@@ -346,7 +346,7 @@ export const Join = ({
|
||||
}}
|
||||
>
|
||||
<VStack marginBottom={1}>
|
||||
<H lvl={1} margin="sm">
|
||||
<H lvl={1} margin="sm" centered>
|
||||
{t('heading')}
|
||||
</H>
|
||||
<Field
|
||||
|
||||
@@ -0,0 +1,345 @@
|
||||
import { ProcessorOptions, Track } from 'livekit-client'
|
||||
import posthog from 'posthog-js'
|
||||
import {
|
||||
FilesetResolver,
|
||||
FaceLandmarker,
|
||||
FaceLandmarkerResult,
|
||||
} from '@mediapipe/tasks-vision'
|
||||
import {
|
||||
CLEAR_TIMEOUT,
|
||||
SET_TIMEOUT,
|
||||
TIMEOUT_TICK,
|
||||
timerWorkerScript,
|
||||
} from './TimerWorker'
|
||||
import {
|
||||
BackgroundProcessorInterface,
|
||||
BackgroundOptions,
|
||||
ProcessorType,
|
||||
} from '.'
|
||||
|
||||
const PROCESSING_WIDTH = 256 * 3
|
||||
const PROCESSING_HEIGHT = 144 * 3
|
||||
|
||||
const FACE_LANDMARKS_CANVAS_ID = 'face-landmarks-local'
|
||||
|
||||
export class FaceLandmarksProcessor implements BackgroundProcessorInterface {
|
||||
options: BackgroundOptions
|
||||
name: string
|
||||
processedTrack?: MediaStreamTrack | undefined
|
||||
|
||||
source?: MediaStreamTrack
|
||||
sourceSettings?: MediaTrackSettings
|
||||
videoElement?: HTMLVideoElement
|
||||
videoElementLoaded?: boolean
|
||||
|
||||
// Canvas containing the video processing result
|
||||
outputCanvas?: HTMLCanvasElement
|
||||
outputCanvasCtx?: CanvasRenderingContext2D
|
||||
|
||||
faceLandmarker?: FaceLandmarker
|
||||
faceLandmarkerResult?: FaceLandmarkerResult
|
||||
|
||||
// The resized image of the video source
|
||||
sourceImageData?: ImageData
|
||||
|
||||
timerWorker?: Worker
|
||||
|
||||
type: ProcessorType
|
||||
|
||||
// Effect images
|
||||
glassesImage?: HTMLImageElement
|
||||
mustacheImage?: HTMLImageElement
|
||||
beretImage?: HTMLImageElement
|
||||
constructor(opts: BackgroundOptions) {
|
||||
this.name = 'face_landmarks'
|
||||
this.options = opts
|
||||
this.type = ProcessorType.FACE_LANDMARKS
|
||||
this._initEffectImages()
|
||||
}
|
||||
|
||||
private _initEffectImages() {
|
||||
this.glassesImage = new Image()
|
||||
this.glassesImage.src = '/assets/glasses.png'
|
||||
this.glassesImage.crossOrigin = 'anonymous'
|
||||
|
||||
this.mustacheImage = new Image()
|
||||
this.mustacheImage.src = '/assets/mustache.png'
|
||||
this.mustacheImage.crossOrigin = 'anonymous'
|
||||
|
||||
this.beretImage = new Image()
|
||||
this.beretImage.src = '/assets/beret.png'
|
||||
this.beretImage.crossOrigin = 'anonymous'
|
||||
}
|
||||
|
||||
static get isSupported() {
|
||||
return true // Face landmarks should work in all modern browsers
|
||||
}
|
||||
|
||||
async init(opts: ProcessorOptions<Track.Kind>) {
|
||||
if (!opts.element) {
|
||||
throw new Error('Element is required for processing')
|
||||
}
|
||||
|
||||
this.source = opts.track as MediaStreamTrack
|
||||
this.sourceSettings = this.source!.getSettings()
|
||||
this.videoElement = opts.element as HTMLVideoElement
|
||||
|
||||
this._createMainCanvas()
|
||||
|
||||
const stream = this.outputCanvas!.captureStream()
|
||||
const tracks = stream.getVideoTracks()
|
||||
if (tracks.length == 0) {
|
||||
throw new Error('No tracks found for processing')
|
||||
}
|
||||
this.processedTrack = tracks[0]
|
||||
|
||||
await this.initFaceLandmarker()
|
||||
this._initWorker()
|
||||
|
||||
posthog.capture('face-landmarks-init')
|
||||
}
|
||||
|
||||
_initWorker() {
|
||||
this.timerWorker = new Worker(timerWorkerScript, {
|
||||
name: 'FaceLandmarks',
|
||||
})
|
||||
this.timerWorker.onmessage = (data) => this.onTimerMessage(data)
|
||||
if (this.videoElementLoaded) {
|
||||
this.timerWorker!.postMessage({
|
||||
id: SET_TIMEOUT,
|
||||
timeMs: 1000 / 30,
|
||||
})
|
||||
} else {
|
||||
this.videoElement!.onloadeddata = () => {
|
||||
this.videoElementLoaded = true
|
||||
this.timerWorker!.postMessage({
|
||||
id: SET_TIMEOUT,
|
||||
timeMs: 1000 / 30,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
onTimerMessage(response: { data: { id: number } }) {
|
||||
if (response.data.id === TIMEOUT_TICK) {
|
||||
this.process()
|
||||
}
|
||||
}
|
||||
|
||||
async initFaceLandmarker() {
|
||||
const vision = await FilesetResolver.forVisionTasks(
|
||||
'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision/wasm'
|
||||
)
|
||||
this.faceLandmarker = await FaceLandmarker.createFromOptions(vision, {
|
||||
baseOptions: {
|
||||
modelAssetPath:
|
||||
'https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/latest/face_landmarker.task',
|
||||
delegate: 'GPU',
|
||||
},
|
||||
runningMode: 'VIDEO',
|
||||
outputFaceBlendshapes: true,
|
||||
outputFacialTransformationMatrixes: true,
|
||||
})
|
||||
}
|
||||
|
||||
async sizeSource() {
|
||||
this.outputCanvasCtx?.drawImage(
|
||||
this.videoElement!,
|
||||
0,
|
||||
0,
|
||||
this.videoElement!.videoWidth,
|
||||
this.videoElement!.videoHeight,
|
||||
0,
|
||||
0,
|
||||
PROCESSING_WIDTH,
|
||||
PROCESSING_HEIGHT
|
||||
)
|
||||
|
||||
this.sourceImageData = this.outputCanvasCtx?.getImageData(
|
||||
0,
|
||||
0,
|
||||
PROCESSING_WIDTH,
|
||||
PROCESSING_HEIGHT
|
||||
)
|
||||
}
|
||||
|
||||
async detectFaces() {
|
||||
const startTimeMs = performance.now()
|
||||
this.faceLandmarkerResult = this.faceLandmarker!.detectForVideo(
|
||||
this.sourceImageData!,
|
||||
startTimeMs
|
||||
)
|
||||
}
|
||||
|
||||
private drawEffect(
|
||||
leftPoint: { x: number; y: number },
|
||||
rightPoint: { x: number; y: number },
|
||||
image: HTMLImageElement,
|
||||
widthScale: number,
|
||||
heightScale: number,
|
||||
yOffset: number = 0
|
||||
) {
|
||||
// Calculate distance between points
|
||||
const distance = Math.sqrt(
|
||||
Math.pow(rightPoint.x - leftPoint.x, 2) +
|
||||
Math.pow(rightPoint.y - leftPoint.y, 2)
|
||||
)
|
||||
|
||||
// Scale image based on distance
|
||||
const width = distance * PROCESSING_WIDTH * widthScale
|
||||
const height = width * heightScale
|
||||
|
||||
// Calculate center position between points
|
||||
const centerX = (leftPoint.x + rightPoint.x) / 2
|
||||
const centerY = (leftPoint.y + rightPoint.y) / 2 + yOffset
|
||||
|
||||
// Draw image
|
||||
this.outputCanvasCtx!.save()
|
||||
this.outputCanvasCtx!.translate(
|
||||
centerX * PROCESSING_WIDTH,
|
||||
centerY * PROCESSING_HEIGHT
|
||||
)
|
||||
|
||||
// Calculate rotation angle based on point positions
|
||||
const angle = Math.atan2(
|
||||
rightPoint.y - leftPoint.y,
|
||||
rightPoint.x - leftPoint.x
|
||||
)
|
||||
this.outputCanvasCtx!.rotate(angle)
|
||||
|
||||
// Draw image centered at the midpoint between points
|
||||
this.outputCanvasCtx!.drawImage(
|
||||
image,
|
||||
-width / 2,
|
||||
-height / 2,
|
||||
width,
|
||||
height
|
||||
)
|
||||
|
||||
this.outputCanvasCtx!.restore()
|
||||
}
|
||||
|
||||
async drawFaceLandmarks() {
|
||||
// Draw the original video frame at the canvas size
|
||||
this.outputCanvasCtx!.drawImage(
|
||||
this.videoElement!,
|
||||
0,
|
||||
0,
|
||||
this.videoElement!.videoWidth,
|
||||
this.videoElement!.videoHeight,
|
||||
0,
|
||||
0,
|
||||
PROCESSING_WIDTH,
|
||||
PROCESSING_HEIGHT
|
||||
)
|
||||
|
||||
if (!this.faceLandmarkerResult?.faceLandmarks) {
|
||||
return
|
||||
}
|
||||
|
||||
// Draw face landmarks (optional, for debugging)
|
||||
this.outputCanvasCtx!.strokeStyle = '#00FF00'
|
||||
this.outputCanvasCtx!.lineWidth = 2
|
||||
|
||||
for (const face of this.faceLandmarkerResult.faceLandmarks) {
|
||||
// Find eye landmarks
|
||||
const leftEye = face[468]
|
||||
const rightEye = face[473]
|
||||
|
||||
// Find mouth landmarks for mustache
|
||||
const leftMoustache = face[92]
|
||||
const rightMoustache = face[322]
|
||||
|
||||
// Find forehead landmarks for beret
|
||||
const leftForehead = face[103]
|
||||
const rightForehead = face[332]
|
||||
|
||||
if (leftEye && rightEye && this.options.showGlasses) {
|
||||
this.drawEffect(leftEye, rightEye, this.glassesImage!, 2.5, 0.7)
|
||||
}
|
||||
|
||||
if (leftMoustache && rightMoustache && this.options.showFrench) {
|
||||
this.drawEffect(
|
||||
leftMoustache,
|
||||
rightMoustache,
|
||||
this.mustacheImage!,
|
||||
1.5,
|
||||
0.5
|
||||
)
|
||||
}
|
||||
|
||||
if (leftForehead && rightForehead && this.options.showFrench) {
|
||||
this.drawEffect(
|
||||
leftForehead,
|
||||
rightForehead,
|
||||
this.beretImage!,
|
||||
2.1,
|
||||
0.7,
|
||||
-0.1
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async process() {
|
||||
await this.sizeSource()
|
||||
await this.detectFaces()
|
||||
await this.drawFaceLandmarks()
|
||||
|
||||
this.timerWorker!.postMessage({
|
||||
id: SET_TIMEOUT,
|
||||
timeMs: 1000 / 30,
|
||||
})
|
||||
}
|
||||
|
||||
_createMainCanvas() {
|
||||
this.outputCanvas = document.querySelector(
|
||||
`#${FACE_LANDMARKS_CANVAS_ID}`
|
||||
) as HTMLCanvasElement
|
||||
if (!this.outputCanvas) {
|
||||
this.outputCanvas = this._createCanvas(
|
||||
FACE_LANDMARKS_CANVAS_ID,
|
||||
PROCESSING_WIDTH,
|
||||
PROCESSING_HEIGHT
|
||||
)
|
||||
}
|
||||
this.outputCanvasCtx = this.outputCanvas.getContext('2d')!
|
||||
}
|
||||
|
||||
_createCanvas(id: string, width: number, height: number) {
|
||||
const element = document.createElement('canvas')
|
||||
element.setAttribute('id', id)
|
||||
element.setAttribute('width', '' + width)
|
||||
element.setAttribute('height', '' + height)
|
||||
return element
|
||||
}
|
||||
|
||||
update(opts: BackgroundOptions): void {
|
||||
this.options = opts
|
||||
}
|
||||
|
||||
async restart(opts: ProcessorOptions<Track.Kind>) {
|
||||
await this.destroy()
|
||||
return this.init(opts)
|
||||
}
|
||||
|
||||
async destroy() {
|
||||
this.timerWorker?.postMessage({
|
||||
id: CLEAR_TIMEOUT,
|
||||
})
|
||||
|
||||
this.timerWorker?.terminate()
|
||||
this.faceLandmarker?.close()
|
||||
}
|
||||
|
||||
clone() {
|
||||
return new FaceLandmarksProcessor(this.options)
|
||||
}
|
||||
|
||||
serialize() {
|
||||
return {
|
||||
type: this.type,
|
||||
options: this.options,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -3,10 +3,13 @@ import { Track, TrackProcessor } from 'livekit-client'
|
||||
import { BackgroundBlurTrackProcessorJsWrapper } from './BackgroundBlurTrackProcessorJsWrapper'
|
||||
import { BackgroundCustomProcessor } from './BackgroundCustomProcessor'
|
||||
import { BackgroundVirtualTrackProcessorJsWrapper } from './BackgroundVirtualTrackProcessorJsWrapper'
|
||||
import { FaceLandmarksProcessor } from './FaceLandmarksProcessor'
|
||||
|
||||
export type BackgroundOptions = {
|
||||
blurRadius?: number
|
||||
imagePath?: string
|
||||
showGlasses?: boolean
|
||||
showFrench?: boolean
|
||||
}
|
||||
|
||||
export interface ProcessorSerialized {
|
||||
@@ -25,11 +28,16 @@ export interface BackgroundProcessorInterface
|
||||
export enum ProcessorType {
|
||||
BLUR = 'blur',
|
||||
VIRTUAL = 'virtual',
|
||||
FACE_LANDMARKS = 'faceLandmarks',
|
||||
}
|
||||
|
||||
export class BackgroundProcessorFactory {
|
||||
static isSupported() {
|
||||
return ProcessorWrapper.isSupported || BackgroundCustomProcessor.isSupported
|
||||
return (
|
||||
ProcessorWrapper.isSupported ||
|
||||
BackgroundCustomProcessor.isSupported ||
|
||||
FaceLandmarksProcessor.isSupported
|
||||
)
|
||||
}
|
||||
|
||||
static getProcessor(
|
||||
@@ -50,6 +58,10 @@ export class BackgroundProcessorFactory {
|
||||
if (BackgroundCustomProcessor.isSupported) {
|
||||
return new BackgroundCustomProcessor(opts)
|
||||
}
|
||||
} else if (type === ProcessorType.FACE_LANDMARKS) {
|
||||
if (FaceLandmarksProcessor.isSupported) {
|
||||
return new FaceLandmarksProcessor(opts)
|
||||
}
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
+103
-2
@@ -5,17 +5,22 @@ import {
|
||||
BackgroundProcessorFactory,
|
||||
BackgroundProcessorInterface,
|
||||
ProcessorType,
|
||||
BackgroundOptions,
|
||||
} from '../blur'
|
||||
import { css } from '@/styled-system/css'
|
||||
import { Text, P, ToggleButton, H } from '@/primitives'
|
||||
import { styled } from '@/styled-system/jsx'
|
||||
import { BackgroundOptions } from '@livekit/track-processors'
|
||||
import { BlurOn } from '@/components/icons/BlurOn'
|
||||
import { BlurOnStrong } from '@/components/icons/BlurOnStrong'
|
||||
import { useTrackToggle } from '@livekit/components-react'
|
||||
import { Loader } from '@/primitives/Loader'
|
||||
import { useSyncAfterDelay } from '@/hooks/useSyncAfterDelay'
|
||||
import { RiProhibited2Line } from '@remixicon/react'
|
||||
import {
|
||||
RiProhibited2Line,
|
||||
RiGlassesLine,
|
||||
RiGoblet2Fill,
|
||||
} from '@remixicon/react'
|
||||
import { useHasFaceLandmarksAccess } from '../../hooks/useHasFaceLandmarksAccess'
|
||||
|
||||
enum BlurRadius {
|
||||
NONE = 0,
|
||||
@@ -50,6 +55,7 @@ export const EffectsConfiguration = ({
|
||||
const { toggle, enabled } = useTrackToggle({ source: Track.Source.Camera })
|
||||
const [processorPending, setProcessorPending] = useState(false)
|
||||
const processorPendingReveal = useSyncAfterDelay(processorPending)
|
||||
const hasFaceLandmarksAccess = useHasFaceLandmarksAccess()
|
||||
|
||||
useEffect(() => {
|
||||
const videoElement = videoRef.current
|
||||
@@ -139,9 +145,42 @@ export const EffectsConfiguration = ({
|
||||
}
|
||||
|
||||
const tooltipLabel = (type: ProcessorType, options: BackgroundOptions) => {
|
||||
if (type === ProcessorType.FACE_LANDMARKS) {
|
||||
const effect = options.showGlasses ? 'glasses' : 'french'
|
||||
return t(
|
||||
`faceLandmarks.${effect}.${isSelected(type, options) ? 'clear' : 'apply'}`
|
||||
)
|
||||
}
|
||||
return t(`${type}.${isSelected(type, options) ? 'clear' : 'apply'}`)
|
||||
}
|
||||
|
||||
const getFaceLandmarksOptions = () => {
|
||||
const processor = getProcessor()
|
||||
if (processor?.serialize().type === ProcessorType.FACE_LANDMARKS) {
|
||||
return processor.serialize().options as {
|
||||
showGlasses?: boolean
|
||||
showFrench?: boolean
|
||||
}
|
||||
}
|
||||
return { showGlasses: false, showFrench: false }
|
||||
}
|
||||
|
||||
const toggleFaceLandmarkEffect = async (effect: 'glasses' | 'french') => {
|
||||
const currentOptions = getFaceLandmarksOptions()
|
||||
const newOptions = {
|
||||
...currentOptions,
|
||||
[effect === 'glasses' ? 'showGlasses' : 'showFrench']:
|
||||
!currentOptions[effect === 'glasses' ? 'showGlasses' : 'showFrench'],
|
||||
}
|
||||
|
||||
if (!newOptions.showGlasses && !newOptions.showFrench) {
|
||||
// If both effects are off stop the processor
|
||||
await clearEffect()
|
||||
} else {
|
||||
await toggleEffect(ProcessorType.FACE_LANDMARKS, newOptions)
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div
|
||||
className={css(
|
||||
@@ -302,6 +341,68 @@ export const EffectsConfiguration = ({
|
||||
</ToggleButton>
|
||||
</div>
|
||||
</div>
|
||||
{hasFaceLandmarksAccess && (
|
||||
<div
|
||||
className={css({
|
||||
marginTop: '1.5rem',
|
||||
})}
|
||||
>
|
||||
<H
|
||||
lvl={3}
|
||||
style={{
|
||||
marginBottom: '1rem',
|
||||
}}
|
||||
variant="bodyXsBold"
|
||||
>
|
||||
{t('faceLandmarks.title')}
|
||||
</H>
|
||||
<div
|
||||
className={css({
|
||||
display: 'flex',
|
||||
gap: '1.25rem',
|
||||
})}
|
||||
>
|
||||
<ToggleButton
|
||||
variant="bigSquare"
|
||||
aria-label={tooltipLabel(ProcessorType.FACE_LANDMARKS, {
|
||||
showGlasses: true,
|
||||
showFrench: false,
|
||||
})}
|
||||
tooltip={tooltipLabel(ProcessorType.FACE_LANDMARKS, {
|
||||
showGlasses: true,
|
||||
showFrench: false,
|
||||
})}
|
||||
isDisabled={processorPendingReveal}
|
||||
onChange={async () =>
|
||||
await toggleFaceLandmarkEffect('glasses')
|
||||
}
|
||||
isSelected={getFaceLandmarksOptions().showGlasses}
|
||||
data-attr="toggle-glasses"
|
||||
>
|
||||
<RiGlassesLine />
|
||||
</ToggleButton>
|
||||
<ToggleButton
|
||||
variant="bigSquare"
|
||||
aria-label={tooltipLabel(ProcessorType.FACE_LANDMARKS, {
|
||||
showGlasses: false,
|
||||
showFrench: true,
|
||||
})}
|
||||
tooltip={tooltipLabel(ProcessorType.FACE_LANDMARKS, {
|
||||
showGlasses: false,
|
||||
showFrench: true,
|
||||
})}
|
||||
isDisabled={processorPendingReveal}
|
||||
onChange={async () =>
|
||||
await toggleFaceLandmarkEffect('french')
|
||||
}
|
||||
isSelected={getFaceLandmarksOptions().showFrench}
|
||||
data-attr="toggle-french"
|
||||
>
|
||||
<RiGoblet2Fill />
|
||||
</ToggleButton>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
<div
|
||||
className={css({
|
||||
marginTop: '1.5rem',
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
import { useFeatureFlagEnabled } from 'posthog-js/react'
|
||||
import { useIsAnalyticsEnabled } from '@/features/analytics/hooks/useIsAnalyticsEnabled'
|
||||
|
||||
export const useHasFaceLandmarksAccess = () => {
|
||||
const featureEnabled = useFeatureFlagEnabled('face-landmarks')
|
||||
const isAnalyticsEnabled = useIsAnalyticsEnabled()
|
||||
|
||||
return featureEnabled || !isAnalyticsEnabled
|
||||
}
|
||||
@@ -58,14 +58,14 @@ export class SdkReverseClient {
|
||||
* To be used in SDK scope.
|
||||
*/
|
||||
export function useEnsureAuth() {
|
||||
const { isLoggedIn, ...other } = useUser({
|
||||
const { isLoggedIn, refetch } = useUser({
|
||||
fetchUserOptions: { attemptSilent: false },
|
||||
})
|
||||
|
||||
const startSSO = () => {
|
||||
return new Promise<void>((resolve) => {
|
||||
SdkReverseClient.waitForAuthenticationAck().then(async () => {
|
||||
await other.refetch()
|
||||
await refetch()
|
||||
resolve()
|
||||
})
|
||||
const params = `scrollbars=no,resizable=no,status=no,location=no,toolbar=no,menubar=no,
|
||||
|
||||
@@ -135,22 +135,33 @@
|
||||
}
|
||||
},
|
||||
"effects": {
|
||||
"activateCamera": "",
|
||||
"notAvailable": "",
|
||||
"heading": "",
|
||||
"activateCamera": "Ihre Kamera ist deaktiviert. Wählen Sie eine Option, um sie zu aktivieren.",
|
||||
"notAvailable": "Videoeffekte werden in Kürze in Ihrem Browser verfügbar sein. Wir arbeiten daran! In der Zwischenzeit können Sie Google Chrome für beste Leistung oder Firefox verwenden :(",
|
||||
"heading": "Unschärfe",
|
||||
"blur": {
|
||||
"title": "",
|
||||
"light": "",
|
||||
"normal": "",
|
||||
"apply": "",
|
||||
"clear": ""
|
||||
"title": "Hintergrundunschärfe",
|
||||
"light": "Leichte Unschärfe",
|
||||
"normal": "Unschärfe",
|
||||
"apply": "Unschärfe aktivieren",
|
||||
"clear": "Unschärfe deaktivieren"
|
||||
},
|
||||
"virtual": {
|
||||
"title": "",
|
||||
"apply": "",
|
||||
"clear": ""
|
||||
"title": "Virtueller Hintergrund",
|
||||
"apply": "Virtuellen Hintergrund aktivieren",
|
||||
"clear": "Virtuellen Hintergrund deaktivieren"
|
||||
},
|
||||
"experimental": ""
|
||||
"faceLandmarks": {
|
||||
"title": "Visuelle Effekte",
|
||||
"glasses": {
|
||||
"apply": "Brille hinzufügen",
|
||||
"clear": "Brille entfernen"
|
||||
},
|
||||
"french": {
|
||||
"apply": "Französische Touch hinzufügen",
|
||||
"clear": "Französische Touch entfernen"
|
||||
}
|
||||
},
|
||||
"experimental": "Experimentelle Funktion. Eine v2 kommt für vollständige Browserunterstützung und verbesserte Qualität."
|
||||
},
|
||||
"sidePanel": {
|
||||
"heading": {
|
||||
|
||||
@@ -149,6 +149,17 @@
|
||||
"apply": "Enable virtual background",
|
||||
"clear": "Disable virtual background"
|
||||
},
|
||||
"faceLandmarks": {
|
||||
"title": "Visual Effects",
|
||||
"glasses": {
|
||||
"apply": "Add Glasses",
|
||||
"clear": "Remove Glasses"
|
||||
},
|
||||
"french": {
|
||||
"apply": "Add French touch",
|
||||
"clear": "Remove French touch"
|
||||
}
|
||||
},
|
||||
"experimental": "Experimental feature. A v2 is coming for full browser support and improved quality."
|
||||
},
|
||||
"sidePanel": {
|
||||
|
||||
@@ -134,22 +134,33 @@
|
||||
}
|
||||
},
|
||||
"effects": {
|
||||
"activateCamera": "Votre camera est désactivée. Choisissez une option pour l'activer.",
|
||||
"notAvailable": "Les effets vidéo seront bientôt disponible sur votre navigateur. Nous y travaillons ! En attendant, vous pouvez utiliser Google Chrome pour une meilleure performance ou Firefox :(",
|
||||
"activateCamera": "Votre caméra est désactivée. Choisissez une option pour l'activer.",
|
||||
"notAvailable": "Les effets vidéo seront bientôt disponibles sur votre navigateur. Nous y travaillons ! En attendant, vous pouvez utiliser Google Chrome pour de meilleures performances ou Firefox :(",
|
||||
"heading": "Flou",
|
||||
"blur": {
|
||||
"title": "Flou d'arrière-plan",
|
||||
"light": "Léger flou",
|
||||
"light": "Flou léger",
|
||||
"normal": "Flou",
|
||||
"apply": "Appliquer le flou",
|
||||
"apply": "Activer le flou",
|
||||
"clear": "Désactiver le flou"
|
||||
},
|
||||
"virtual": {
|
||||
"title": "Arrière-plan virtuel",
|
||||
"apply": "Appliquer l'arrière plan virtuel",
|
||||
"clear": "Désactiver l'arrière plan virtuel"
|
||||
"apply": "Activer l'arrière-plan virtuel",
|
||||
"clear": "Désactiver l'arrière-plan virtuel"
|
||||
},
|
||||
"experimental": "Fonctionnalité expérimentale. Une v2 arrive pour un support complet sur tous les navigateurs et une meilleur qualité."
|
||||
"faceLandmarks": {
|
||||
"title": "Effets visuels",
|
||||
"glasses": {
|
||||
"apply": "Ajouter des lunettes",
|
||||
"clear": "Retirer les lunettes"
|
||||
},
|
||||
"french": {
|
||||
"apply": "Ajouter la touche française",
|
||||
"clear": "Retirer la touche française"
|
||||
}
|
||||
},
|
||||
"experimental": "Fonctionnalité expérimentale. Une v2 arrive pour un support complet des navigateurs et une meilleure qualité."
|
||||
},
|
||||
"sidePanel": {
|
||||
"heading": {
|
||||
|
||||
@@ -134,22 +134,33 @@
|
||||
}
|
||||
},
|
||||
"effects": {
|
||||
"activateCamera": "Uw camera is uitgeschakeld. Kies een optie om hem in te schakelen.",
|
||||
"notAvailable": "Video-effecten zijn binnenkort beschikbaar in uw browser. We werken hier aan! Ondertussen kunt u Chrome gebruiken voor de beste prestaties, of Firefox :(",
|
||||
"activateCamera": "Uw camera is uitgeschakeld. Kies een optie om deze in te schakelen.",
|
||||
"notAvailable": "Video-effecten zijn binnenkort beschikbaar in uw browser. We werken eraan! In de tussentijd kunt u Google Chrome gebruiken voor de beste prestaties of Firefox :(",
|
||||
"heading": "Vervaging",
|
||||
"blur": {
|
||||
"title": "Achtergrond vervagen",
|
||||
"light": "Licht vervagen",
|
||||
"normal": "Vervagen",
|
||||
"apply": "Schakel vervaging in",
|
||||
"clear": "Schakel vervaging uit"
|
||||
"title": "Achtergrondvervaging",
|
||||
"light": "Lichte vervaging",
|
||||
"normal": "Vervaging",
|
||||
"apply": "Vervaging inschakelen",
|
||||
"clear": "Vervaging uitschakelen"
|
||||
},
|
||||
"virtual": {
|
||||
"title": "Virtuele achtergrond",
|
||||
"apply": "Schakel virtuele achtergrond in",
|
||||
"clear": "Schakel virtuele achtergrond uit"
|
||||
"apply": "Virtuele achtergrond inschakelen",
|
||||
"clear": "Virtuele achtergrond uitschakelen"
|
||||
},
|
||||
"experimental": "Experimentele functionaliteit. Een v2 komt er aan met volledige browser-ondersteuning en verbeterde kwaliteit."
|
||||
"faceLandmarks": {
|
||||
"title": "Visuele effecten",
|
||||
"glasses": {
|
||||
"apply": "Bril toevoegen",
|
||||
"clear": "Bril verwijderen"
|
||||
},
|
||||
"french": {
|
||||
"apply": "Franse stijl toevoegen",
|
||||
"clear": "Franse stijl verwijderen"
|
||||
}
|
||||
},
|
||||
"experimental": "Experimentele functie. Een v2 komt eraan voor volledige browserondersteuning en verbeterde kwaliteit."
|
||||
},
|
||||
"sidePanel": {
|
||||
"heading": {
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "mail_mjml",
|
||||
"version": "0.1.15",
|
||||
"version": "0.1.17",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "mail_mjml",
|
||||
"version": "0.1.15",
|
||||
"version": "0.1.17",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@html-to/text-cli": "0.5.4",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "mail_mjml",
|
||||
"version": "0.1.15",
|
||||
"version": "0.1.17",
|
||||
"description": "An util to generate html and text django's templates from mjml templates",
|
||||
"type": "module",
|
||||
"dependencies": {
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "sdk",
|
||||
"version": "0.1.15",
|
||||
"version": "0.1.17",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "sdk",
|
||||
"version": "0.1.15",
|
||||
"version": "0.1.17",
|
||||
"license": "ISC",
|
||||
"workspaces": [
|
||||
"./library",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "sdk",
|
||||
"version": "0.1.15",
|
||||
"version": "0.1.17",
|
||||
"author": "",
|
||||
"license": "ISC",
|
||||
"description": "",
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
|
||||
[project]
|
||||
name = "summary"
|
||||
version = "0.1.15"
|
||||
version = "0.1.17"
|
||||
dependencies = [
|
||||
"fastapi[standard]>=0.105.0",
|
||||
"uvicorn>=0.24.0",
|
||||
|
||||
@@ -1,10 +1,15 @@
|
||||
"""API routes related to application tasks."""
|
||||
|
||||
from typing import Optional
|
||||
|
||||
from celery.result import AsyncResult
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel
|
||||
|
||||
from summary.core.celery_worker import process_audio_transcribe_summarize
|
||||
from summary.core.celery_worker import (
|
||||
process_audio_transcribe_summarize,
|
||||
process_audio_transcribe_summarize_v2,
|
||||
)
|
||||
|
||||
|
||||
class TaskCreation(BaseModel):
|
||||
@@ -13,6 +18,7 @@ class TaskCreation(BaseModel):
|
||||
filename: str
|
||||
email: str
|
||||
sub: str
|
||||
version: Optional[int] = 2
|
||||
|
||||
|
||||
router = APIRouter(prefix="/tasks")
|
||||
@@ -21,9 +27,15 @@ router = APIRouter(prefix="/tasks")
|
||||
@router.post("/")
|
||||
async def create_task(request: TaskCreation):
|
||||
"""Create a task."""
|
||||
task = process_audio_transcribe_summarize.delay(
|
||||
request.filename, request.email, request.sub
|
||||
)
|
||||
if request.version == 1:
|
||||
task = process_audio_transcribe_summarize.delay(
|
||||
request.filename, request.email, request.sub
|
||||
)
|
||||
else:
|
||||
task = process_audio_transcribe_summarize_v2.delay(
|
||||
request.filename, request.email, request.sub
|
||||
)
|
||||
|
||||
return {"id": task.id, "message": "Task created"}
|
||||
|
||||
|
||||
|
||||
@@ -56,6 +56,35 @@ def create_retry_session():
|
||||
return session
|
||||
|
||||
|
||||
def format_segments(transcription_data):
|
||||
"""Format transcription segments from WhisperX into a readable conversation format.
|
||||
|
||||
Processes transcription data with segments containing speaker information and text,
|
||||
combining consecutive segments from the same speaker and formatting them as a
|
||||
conversation with speaker labels.
|
||||
"""
|
||||
formatted_output = ""
|
||||
if not transcription_data or not hasattr(transcription_data, 'segments'):
|
||||
if isinstance(transcription_data, dict) and 'segments' in transcription_data:
|
||||
segments = transcription_data['segments']
|
||||
else:
|
||||
return "Error: Invalid transcription data format"
|
||||
else:
|
||||
segments = transcription_data.segments
|
||||
|
||||
previous_speaker = None
|
||||
|
||||
for segment in segments:
|
||||
speaker = segment.get('speaker', 'UNKNOWN_SPEAKER')
|
||||
text = segment.get('text', '')
|
||||
if text:
|
||||
if speaker != previous_speaker:
|
||||
formatted_output += f"\n\n *{speaker}*: {text}"
|
||||
else:
|
||||
formatted_output += f" {text}"
|
||||
previous_speaker = speaker
|
||||
return formatted_output
|
||||
|
||||
def post_with_retries(url, data):
|
||||
"""Send POST request with automatic retries."""
|
||||
session = create_retry_session()
|
||||
@@ -140,3 +169,72 @@ def process_audio_transcribe_summarize(filename: str, email: str, sub: str):
|
||||
|
||||
logger.info("Webhook submitted successfully. Status: %s", response.status_code)
|
||||
logger.debug("Response body: %s", response.text)
|
||||
|
||||
|
||||
@celery.task(max_retries=settings.celery_max_retries)
|
||||
def process_audio_transcribe_summarize_v2(filename: str, email: str, sub: str):
|
||||
"""Process an audio file by transcribing it and generating a summary.
|
||||
|
||||
This Celery task performs the following operations:
|
||||
1. Retrieves the audio file from MinIO storage
|
||||
2. Transcribes the audio using WhisperX model
|
||||
3. Sends the results via webhook
|
||||
|
||||
"""
|
||||
logger.info("Notification received")
|
||||
logger.debug("filename: %s", filename)
|
||||
|
||||
minio_client = Minio(
|
||||
settings.aws_s3_endpoint_url,
|
||||
access_key=settings.aws_s3_access_key_id,
|
||||
secret_key=settings.aws_s3_secret_access_key,
|
||||
secure=settings.aws_s3_secure_access,
|
||||
)
|
||||
|
||||
logger.debug("Connection to the Minio bucket successful")
|
||||
|
||||
audio_file_stream = minio_client.get_object(
|
||||
settings.aws_storage_bucket_name, object_name=filename
|
||||
)
|
||||
|
||||
temp_file_path = save_audio_stream(audio_file_stream)
|
||||
logger.debug("Recording successfully downloaded, filepath: %s", temp_file_path)
|
||||
|
||||
logger.debug("Initiating OpenAI client")
|
||||
openai_client = openai.OpenAI(
|
||||
api_key=settings.openai_api_key, base_url=settings.openai_base_url
|
||||
)
|
||||
|
||||
try:
|
||||
logger.debug("Querying transcription …")
|
||||
with open(temp_file_path, "rb") as audio_file:
|
||||
transcription = openai_client.audio.transcriptions.create(
|
||||
model=settings.openai_asr_model, file=audio_file
|
||||
)
|
||||
|
||||
logger.debug("Transcription: \n %s", transcription)
|
||||
finally:
|
||||
if os.path.exists(temp_file_path):
|
||||
os.remove(temp_file_path)
|
||||
logger.debug("Temporary file removed: %s", temp_file_path)
|
||||
|
||||
formatted_transcription = format_segments(transcription)
|
||||
|
||||
data = {
|
||||
"title": "Transcription",
|
||||
"content": formatted_transcription,
|
||||
"email": email,
|
||||
"sub": sub,
|
||||
}
|
||||
|
||||
logger.debug("Submitting webhook to %s", settings.webhook_url)
|
||||
logger.debug("Request payload: %s", json.dumps(data, indent=2))
|
||||
|
||||
response = post_with_retries(settings.webhook_url, data)
|
||||
|
||||
logger.info("Webhook submitted successfully. Status: %s", response.status_code)
|
||||
logger.debug("Response body: %s", response.text)
|
||||
|
||||
# TODO - integrate summarize the transcript and create a new document.
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user