Compare commits

..

2 Commits

Author SHA1 Message Date
lebaudantoine 12e99f8f2a 💄(frontend) adjust font size based on Marianne recent changes
Marianne is bigger than OpenSans, adjust admin side panel style.
2025-03-24 13:38:40 +01:00
lebaudantoine 3358b23edd 🌐(frontend) add missing translation
The admin side panel was translated too quickly.
I forgot a title, fix it.
2025-03-24 13:38:38 +01:00
38 changed files with 2327 additions and 2816 deletions
+9 -6
View File
@@ -119,9 +119,12 @@ jobs:
runs-on: ubuntu-latest
if: github.event_name != 'pull_request'
steps:
- uses: numerique-gouv/action-argocd-webhook-notification@main
id: notify
with:
deployment_repo_path: "${{ secrets.DEPLOYMENT_REPO_URL }}"
argocd_webhook_secret: "${{ secrets.ARGOCD_PREPROD_WEBHOOK_SECRET }}"
argocd_url: "${{ vars.ARGOCD_PREPROD_WEBHOOK_URL }}"
-
name: Checkout repository
uses: actions/checkout@v4
-
name: Call argocd github webhook
run: |
data='{"ref": "'$GITHUB_REF'","repository": {"html_url":"'$GITHUB_SERVER_URL'/numerique-gouv/lasuite-deploiement"}}'
sig=$(echo -n ${data} | openssl dgst -sha1 -hmac "${{ secrets.ARGOCD_PREPROD_WEBHOOK_SECRET }}" | awk '{print "X-Hub-Signature: sha1="$2}')
curl -X POST -H 'X-GitHub-Event:push' -H "Content-Type: application/json" -H "${sig}" --data "${data}" ${{ vars.ARGOCD_PREPROD_WEBHOOK_URL }}
-21
View File
@@ -1,21 +0,0 @@
MIT License
Copyright (c) 2021-2023 DINUM/Etalab
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+1 -1
View File
@@ -1,4 +1,4 @@
FROM livekit/livekit-server:v1.8.0
FROM livekit/livekit-server:latest
# We inject the nip.io certificate manually because the livekit chart doesn't support volume mounting
COPY rootCA.pem /etc/ssl/certs/
+9 -106
View File
@@ -2,6 +2,7 @@
This document is a step-by-step guide that describes how to install Visio on a k8s cluster without AI features.
## Prerequisites
- k8s cluster with an nginx-ingress controller
@@ -42,11 +43,11 @@ It will expire on 23 March 2027 🗓
2. Create kind cluster with containerd registry config dir enabled
Creating cluster "visio" ...
✓ Ensuring node image (kindest/node:v1.27.3) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-visio"
You can now use your cluster with:
@@ -94,14 +95,13 @@ ingress-nginx-admission-create-jgnc9 0/1 Completed 0 2m44s
ingress-nginx-admission-patch-wrt47 0/1 Completed 0 2m44s
ingress-nginx-controller-57c548c4cd-9xwt6 1/1 Running 0 2m44s
```
When your k8s cluster is ready, you can start the deployment. This cluster is special because it uses the *.127.0.0.1.nip.io domain and mkcert certificates to have full HTTPS support and easy domain name management.
When your k8s cluster is ready, you can start the deployment. This cluster is special because it uses the \*.127.0.0.1.nip.io domain and mkcert certificates to have full HTTPS support and easy domain name management.
Please remember that \*.127.0.0.1.nip.io will always resolve to 127.0.0.1, except in the k8s cluster where we configure CoreDNS to answer with the ingress-nginx service IP.
Please remember that *.127.0.0.1.nip.io will always resolve to 127.0.0.1, except in the k8s cluster where we configure CoreDNS to answer with the ingress-nginx service IP.
## Preparation
### What will you use to authenticate your users ?
### What will you use to authenticate your users ?
Visio uses OIDC, so if you already have an OIDC provider, obtain the necessary information to use it. In the next step, we will see how to configure Django (and thus Visio) to use it. If you do not have a provider, we will show you how to deploy a local Keycloak instance (this is not a production deployment, just a demo).
@@ -153,7 +153,6 @@ keycloak-0 1/1 Running 0 26m
keycloak-postgresql-0 1/1 Running 0 26m
redis-master-0 1/1 Running 0 35s
```
When the redis is ready we can deploy livekit-server.
```
@@ -195,7 +194,6 @@ livekit-livekit-server-5c5fb87f7f-ct6x5 1/1 Running 0 15m
postgresql-0 1/1 Running 0 50s
redis-master-0 1/1 Running 0 19
```
From here important information you will need are :
```
@@ -233,98 +231,3 @@ meet-admin <none> meet.127.0.0.1.nip.io localhost 80, 44
```
You can use Visio on https://meet.127.0.0.1.nip.io. The provisioning user in keycloak is meet/meet.
## All options
These are the environmental options available on meet backend.
| Option | Description | default |
| ----------------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| DJANGO_ALLOWED_HOSTS | Host that are allowed | [] |
| DJANGO_SECRET_KEY | Secret key used | |
| DJANGO_SILENCED_SYSTEM_CHECKS | Silence system checks | [] |
| DJANGO_ALLOW_UNSECURE_USER_LISTING | | false |
| DB_ENGINE | Database engine used | django.db.backends.postgresql_psycopg2 |
| DB_NAME | name of the database | meet |
| DB_USER | user used to connect to database | dinum |
| DB_PASSWORD | password used to connect to the database | pass |
| DB_HOST | hostname of the database | localhost |
| DB_PORT | port to connect to database | 5432 |
| STORAGES_STATICFILES_BACKEND | Static file serving engine | whitenoise.storage.CompressedManifestStaticFilesStorage |
| AWS_S3_ENDPOINT_URL | S3 host endpoint | |
| AWS_S3_ACCESS_KEY_ID | s3 access key | |
| AWS_S3_SECRET_ACCESS_KEY | s3 secret key | |
| AWS_S3_REGION_NAME | s3 region | |
| AWS_STORAGE_BUCKET_NAME | s3 bucket name | |
| DJANGO_LANGUAGE_CODE | Default language | en-us |
| REDIS_URL | redis endpoint | redis://redis:6379/1 |
| REQUEST_ENTRY_THROTTLE_RATES | throttle rates | 150/minute |
| SPECTACULAR_SETTINGS_ENABLE_DJANGO_DEPLOY_CHECK | deploy check | False |
| FRONTEND_ANALYTICS | analytics information | {} |
| FRONTEND_SUPPORT | crisp frontend support | {} |
| FRONTEND_SILENCE_LIVEKIT_DEBUG | silence livekit debug | false |
| DJANGO_EMAIL_BACKEND | email backend library | django.core.mail.backends.smtp.EmailBackend |
| DJANGO_EMAIL_HOST | host of the email server | |
| DJANGO_EMAIL_HOST_USER | user to connect to the email server | |
| DJANGO_EMAIL_HOST_PASSWORD | password to connect tto the email server | |
| DJANGO_EMAIL_PORT | por tot connect to the email server | |
| DJANGO_EMAIL_USE_TLS | enable tls on email connection | false |
| DJANGO_EMAIL_USE_SSL | enable tls on email connection | false |
| DJANGO_EMAIL_FROM | email from account | from@example.com |
| DJANGO_CORS_ALLOW_ALL_ORIGINS | allow all cors origins | true |
| DJANGO_CORS_ALLOWED_ORIGINS | origins to allow in string | [] |
| DJANGO_CORS_ALLOWED_ORIGIN_REGEXES | origins to allow in regex | [] |
| SENTRY_DSN | sentry server | |
| DJANGO_CELERY_BROKER_URL | celery broker host | redis://redis:6379/0 |
| DJANGO_CELERY_BROKER_TRANSPORT_OPTIONS | celery broker options | {} |
| OIDC_CREATE_USER | create oidc user if not exists | false |
| OIDC_VERIFY_SSL | verify ssl for oidc | true |
| OIDC_FALLBACK_TO_EMAIL_FOR_IDENTIFICATION | fallback to email for identification | false |
| OIDC_RP_SIGN_ALGO | token verification algoritm used by oidc | RS256 |
| OIDC_RP_CLIENT_ID | oidc client | meet |
| OIDC_RP_CLIENT_SECRET | oidc client secret | |
| OIDC_OP_JWKS_ENDPOINT | oidc endpoint for JWKS | |
| OIDC_OP_AUTHORIZATION_ENDPOINT | oidc endpoint for authorization | |
| OIDC_OP_TOKEN_ENDPOINT | oidc endpoint for token | |
| OIDC_OP_USER_ENDPOINT | oidc endpoint for user | |
| OIDC_OP_LOGOUT_ENDPOINT | oidc endpoint for logout | |
| OIDC_AUTH_REQUEST_EXTRA_PARAMS | extra parameters for oidc request | |
| OIDC_RP_SCOPES | oidc scopes | openid email |
| LOGIN_REDIRECT_URL | login redirect url | |
| LOGIN_REDIRECT_URL_FAILURE | login redurect url for failure | |
| LOGOUT_REDIRECT_URL | url to redirect to on logout | |
| OIDC_USE_NONCE | use nonce for oidc | true |
| OIDC_REDIRECT_REQUIRE_HTTPS | require https for oidc | false |
| OIDC_REDIRECT_ALLOWED_HOSTS | allowed redirect hosts for oidc | [] |
| OIDC_STORE_ID_TOKEN | store oidc ID token | true |
| ALLOW_LOGOUT_GET_METHOD | allow logout through get method | true |
| OIDC_REDIRECT_FIELD_NAME | direct field for oidc | returnTo |
| OIDC_USERINFO_FULLNAME_FIELDS | full name claim from OIDC token | ["given_name", "usual_name"] |
| OIDC_USERINFO_SHORTNAME_FIELD | shortname claim from OIDC token | given_name |
| LIVEKIT_API_KEY | livekit api key | |
| LIVEKIT_API_SECRET | livekit api secret | |
| LIVEKIT_API_URL | livekit api url | |
| RESOURCE_DEFAULT_ACCESS_LEVEL | default resource access level for rooms | public |
| ALLOW_UNREGISTERED_ROOMS | Allow usage of unregistered rooms | true |
| RECORDING_ENABLE | record meeting option | false |
| RECORDING_OUTPUT_FOLDER | folder to store meetings | recordings |
| RECORDING_VERIFY_SSL | verify ssl for recording storage | true |
| RECORDING_WORKER_CLASSES | worker classes for recording | {"screen_recording": "core.recording.worker.services.VideoCompositeEgressService","transcript": "core.recording.worker.services.AudioCompositeEgressService"} |
| RECORDING_EVENT_PARSER_CLASS | storage event engine for recording | core.recording.event.parsers.MinioParser |
| RECORDING_ENABLE_STORAGE_EVENT_AUTH | enable storage event authorization | true |
| RECORDING_STORAGE_EVENT_ENABLE | enable recording storage events | false |
| RECORDING_STORAGE_EVENT_TOKEN | recording storage event token | |
| SUMMARY_SERVICE_ENDPOINT | summary service endpoint | |
| SUMMARY_SERVICE_API_TOKEN | api token voor summary service | |
| SIGNUP_NEW_USER_TO_MARKETING_EMAIL | signup users to marketing emails | false |
| MARKETING_SERVICE_CLASS | markering class | core.services.marketing.BrevoMarketingService |
| BREVO_API_KEY | breva api key for marketing emails | |
| BREVO_API_CONTACT_LIST_IDS | brevo api contact list ids | [] |
| DJANGO_BREVO_API_CONTACT_ATTRIBUTES | brevo contact attributes | {"VISIO_USER": True} |
| BREVO_API_TIMEOUT | brevo timeout | 1 |
| LOBBY_KEY_PREFIX | lobby prefix | room_lobby |
| LOBBY_WAITING_TIMEOUT | lobby waiting tumeout | 3 |
| LOBBY_DENIED_TIMEOUT | lobby deny timeout | 5 |
| LOBBY_ACCEPTED_TIMEOUT | lobby accept timeout | 21600 |
| LOBBY_NOTIFICATION_TYPE | lobby notification types | participantWaiting |
| LOBBY_COOKIE_NAME | lobby cookie name | lobbyParticipantId |
+1 -3
View File
@@ -121,9 +121,7 @@ class OIDCAuthenticationBackend(MozillaOIDCAuthenticationBackend):
contact_data = ContactData(
email=email, attributes={"VISIO_SOURCE": ["SIGNIN"]}
)
marketing_service.create_contact(
contact_data, timeout=settings.BREVO_API_TIMEOUT
)
marketing_service.create_contact(contact_data, timeout=1)
except (ContactCreationError, ImproperlyConfigured, ImportError):
pass
+7 -23
View File
@@ -12,12 +12,7 @@ from django.conf import settings
from django.core.cache import cache
from asgiref.sync import async_to_sync
from livekit.api import ( # pylint: disable=E0611
ListRoomsRequest,
LiveKitAPI,
SendDataRequest,
TwirpError,
)
from livekit.api import LiveKitAPI, SendDataRequest, TwirpError # pylint: disable=E0611
from core import models, utils
@@ -226,12 +221,6 @@ class LobbyService:
color=color,
)
try:
self.notify_participants(room_id=room_id)
except LobbyNotificationError:
# If room not created yet, there is no participants to notify
pass
cache_key = self._get_cache_key(room_id, participant_id)
cache.set(
cache_key,
@@ -239,6 +228,12 @@ class LobbyService:
timeout=settings.LOBBY_WAITING_TIMEOUT,
)
try:
self.notify_participants(room_id=room_id)
except LobbyNotificationError:
# If room not created yet, there is no participants to notify
pass
return participant
def _get_participant(
@@ -348,18 +343,7 @@ class LobbyService:
}
lkapi = LiveKitAPI(**settings.LIVEKIT_CONFIGURATION)
try:
room_response = await lkapi.room.list_rooms(
ListRoomsRequest(
names=[str(room_id)],
)
)
# Check if the room exists
if not room_response.rooms:
return
await lkapi.room.send_data(
SendDataRequest(
room=str(room_id),
@@ -776,43 +776,6 @@ def test_update_participant_status_success(mock_cache, lobby_service, participan
lobby_service._get_cache_key.assert_called_once_with(room.id, participant_id)
@mock.patch("core.services.lobby.LiveKitAPI")
def test_notify_participants_success_no_room(mock_livekit_api, lobby_service):
"""Test the notify_participants method when the LiveKit room doesn't exist yet."""
room = RoomFactory(access_level=RoomAccessLevel.RESTRICTED)
# Set up the mock LiveKitAPI and its behavior
mock_api_instance = mock.Mock()
mock_api_instance.room = mock.Mock()
mock_api_instance.room.send_data = mock.AsyncMock()
# Create a proper response object with an empty rooms list
class MockResponse:
"""LiveKit API response mock with empty rooms list."""
rooms = []
mock_api_instance.room.list_rooms = mock.AsyncMock(return_value=MockResponse())
mock_api_instance.aclose = mock.AsyncMock()
mock_livekit_api.return_value = mock_api_instance
# Act
lobby_service.notify_participants(room.id)
# Assert
# Verify the API was initialized with correct configuration
mock_livekit_api.assert_called_once_with(**settings.LIVEKIT_CONFIGURATION)
# Verify that the service checked for existing rooms
mock_api_instance.room.list_rooms.assert_called_once()
# Verify the send_data method was not called since no room exists
mock_api_instance.room.send_data.assert_not_called()
# Verify the connection was properly closed
mock_api_instance.aclose.assert_called_once()
@mock.patch("core.services.lobby.LiveKitAPI")
def test_notify_participants_success(mock_livekit_api, lobby_service):
"""Test successful participant notification."""
@@ -821,14 +784,6 @@ def test_notify_participants_success(mock_livekit_api, lobby_service):
mock_api_instance = mock.Mock()
mock_api_instance.room = mock.Mock()
mock_api_instance.room.send_data = mock.AsyncMock()
class MockResponse:
"""LiveKit API response mock with non-empty rooms list."""
rooms = ["room-1"]
mock_api_instance.room.list_rooms = mock.AsyncMock(return_value=MockResponse())
mock_api_instance.aclose = mock.AsyncMock()
mock_livekit_api.return_value = mock_api_instance
@@ -838,9 +793,6 @@ def test_notify_participants_success(mock_livekit_api, lobby_service):
# Verify the API was called correctly
mock_livekit_api.assert_called_once_with(**settings.LIVEKIT_CONFIGURATION)
# Verify that the service checked for existing rooms
mock_api_instance.room.list_rooms.assert_called_once()
# Verify the send_data method was called
mock_api_instance.room.send_data.assert_called_once()
send_data_request = mock_api_instance.room.send_data.call_args[0][0]
@@ -865,14 +817,6 @@ def test_notify_participants_error(mock_livekit_api, lobby_service):
mock_api_instance.room.send_data = mock.AsyncMock(
side_effect=TwirpError(msg="test error", code=123)
)
class MockResponse:
"""LiveKit API response mock with non-empty rooms list."""
rooms = ["room-1"]
mock_api_instance.room.list_rooms = mock.AsyncMock(return_value=MockResponse())
mock_api_instance.aclose = mock.AsyncMock()
mock_livekit_api.return_value = mock_api_instance
@@ -885,9 +829,6 @@ def test_notify_participants_error(mock_livekit_api, lobby_service):
# Verify the API was called correctly
mock_livekit_api.assert_called_once_with(**settings.LIVEKIT_CONFIGURATION)
# Verify that the service checked for existing rooms
mock_api_instance.room.list_rooms.assert_called_once()
# Verify send_data was called
mock_api_instance.room.send_data.assert_called_once()
-7
View File
@@ -19,7 +19,6 @@ from django.utils.translation import gettext_lazy as _
import sentry_sdk
from configurations import Configuration, values
from sentry_sdk.integrations.django import DjangoIntegration
from sentry_sdk.integrations.logging import ignore_logger
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -498,9 +497,6 @@ class Base(Configuration):
converter=lambda x: int(x), # pylint: disable=unnecessary-lambda
)
BREVO_API_CONTACT_ATTRIBUTES = values.DictValue({"VISIO_USER": True})
BREVO_API_TIMEOUT = values.PositiveIntegerValue(
1, environ_name="BREVO_API_TIMEOUT", environ_prefix=None
)
# Lobby configurations
LOBBY_KEY_PREFIX = values.Value(
@@ -579,9 +575,6 @@ class Base(Configuration):
scope = sentry_sdk.get_global_scope()
scope.set_tag("application", "backend")
# Ignore the logs added by the DockerflowMiddleware
ignore_logger("request.summary")
class Build(Base):
"""Settings used when the application is built.
+11 -11
View File
@@ -7,7 +7,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "meet"
version = "0.1.16"
version = "0.1.15"
authors = [{ "name" = "DINUM", "email" = "dev@mail.numerique.gouv.fr" }]
classifiers = [
"Development Status :: 5 - Production/Stable",
@@ -25,7 +25,7 @@ license = { file = "LICENSE" }
readme = "README.md"
requires-python = ">=3.10"
dependencies = [
"boto3==1.37.18",
"boto3==1.37.4",
"Brotli==1.1.0",
"brevo-python==1.1.2",
"celery[redis]==5.4.0",
@@ -48,16 +48,16 @@ dependencies = [
"june-analytics-python==2.3.0",
"markdown==3.7",
"nested-multipart-parser==1.5.0",
"psycopg[binary]==3.2.6",
"psycopg[binary]==3.2.5",
"PyJWT==2.10.1",
"python-frontmatter==1.1.0",
"requests==2.32.3",
"sentry-sdk==2.24.1",
"sentry-sdk==2.22.0",
"url-normalize==1.4.3",
"whitenoise==6.9.0",
"mozilla-django-oidc==4.0.1",
"livekit-api==0.8.2",
"aiohttp==3.11.14",
"aiohttp==3.11.13",
]
[project.urls]
@@ -72,18 +72,18 @@ dev = [
"drf-spectacular-sidecar==2025.3.1",
"freezegun==1.5.1",
"ipdb==0.13.13",
"ipython==9.0.2",
"pyfakefs==5.8.0",
"ipython==9.0.1",
"pyfakefs==5.7.4",
"pylint-django==2.6.1",
"pylint==3.3.6",
"pylint==3.3.4",
"pytest-cov==6.0.0",
"pytest-django==4.10.0",
"pytest==8.3.5",
"pytest-icdiff==0.9",
"pytest-xdist==3.6.1",
"responses==0.25.7",
"ruff==0.11.2",
"types-requests==2.32.0.20250306",
"responses==0.25.6",
"ruff==0.9.9",
"types-requests==2.32.0.20250301",
]
[tool.setuptools]
+2069 -1274
View File
File diff suppressed because it is too large Load Diff
+17 -17
View File
@@ -1,7 +1,7 @@
{
"name": "meet",
"private": true,
"version": "0.1.16",
"version": "0.1.15",
"type": "module",
"scripts": {
"dev": "panda codegen && vite",
@@ -16,45 +16,45 @@
"@livekit/components-react": "2.8.1",
"@livekit/components-styles": "1.1.4",
"@livekit/track-processors": "0.3.3",
"@pandacss/preset-panda": "0.53.3",
"@react-aria/toast": "3.0.1",
"@pandacss/preset-panda": "0.53.0",
"@react-aria/toast": "3.0.0-beta.19",
"@remixicon/react": "4.6.0",
"@tanstack/react-query": "5.69.0",
"@tanstack/react-query": "5.67.1",
"crisp-sdk-web": "1.0.25",
"hoofd": "1.7.3",
"i18next": "24.2.3",
"i18next": "24.2.2",
"i18next-browser-languagedetector": "8.0.4",
"i18next-parser": "9.3.0",
"i18next-resources-to-backend": "1.2.1",
"livekit-client": "2.9.8",
"posthog-js": "1.232.6",
"livekit-client": "2.9.5",
"posthog-js": "1.225.1",
"react": "18.3.1",
"react-aria-components": "1.7.1",
"react-aria-components": "1.6.0",
"react-dom": "18.3.1",
"react-i18next": "15.1.1",
"use-sound": "5.0.0",
"valtio": "2.1.4",
"valtio": "2.1.3",
"wouter": "3.6.0"
},
"devDependencies": {
"@pandacss/dev": "0.53.3",
"@tanstack/eslint-plugin-query": "5.68.0",
"@tanstack/react-query-devtools": "5.69.0",
"@types/node": "22.13.13",
"@pandacss/dev": "0.53.0",
"@tanstack/eslint-plugin-query": "5.66.1",
"@tanstack/react-query-devtools": "5.67.1",
"@types/node": "22.13.9",
"@types/react": "18.3.12",
"@types/react-dom": "18.3.1",
"@typescript-eslint/eslint-plugin": "8.28.0",
"@typescript-eslint/parser": "8.28.0",
"@typescript-eslint/eslint-plugin": "8.26.0",
"@typescript-eslint/parser": "8.26.0",
"@vitejs/plugin-react": "4.3.4",
"eslint": "8.57.0",
"eslint-config-prettier": "10.1.1",
"eslint-config-prettier": "10.0.2",
"eslint-plugin-jsx-a11y": "6.10.2",
"eslint-plugin-react-hooks": "5.2.0",
"eslint-plugin-react-refresh": "0.4.19",
"postcss": "8.5.3",
"prettier": "3.5.3",
"typescript": "5.8.2",
"vite": "6.2.3",
"vite": "6.2.0",
"vite-tsconfig-paths": "5.1.4"
}
}
+34 -1
View File
@@ -35,7 +35,40 @@ const config: Config = {
exclude: [],
jsxFramework: 'react',
outdir: 'src/styled-system',
globalFontface: {},
globalFontface: {
Marianne: [
{
src: 'url(/fonts/Marianne-Regular-subset.woff2) format("woff2")',
fontWeight: 400,
fontStyle: 'normal',
fontDisplay: 'swap',
},
{
path: 'url(/fonts/Marianne-Regular_Italic-subset.woff2) format("woff2")',
fontWeight: 400,
fontStyle: 'italic',
fontDisplay: 'swap',
},
{
path: 'url(/fonts/Marianne-Medium-subset.woff2) format("woff2")',
fontWeight: 500,
fontStyle: 'normal',
fontDisplay: 'swap',
},
{
path: 'url(/fonts/Marianne-Bold-subset.woff2) format("woff2")',
fontWeight: 700,
fontStyle: 'normal',
fontDisplay: 'swap',
},
{
path: 'url(/fonts/Marianne-ExtraBold-subset.woff2) format("woff2")',
fontWeight: 800,
fontStyle: 'normal',
fontDisplay: 'swap',
},
],
},
theme: {
...pandaPreset.theme,
// media queries are defined in em so that zooming with text-only mode triggers breakpoints
@@ -48,7 +48,7 @@ export const useUser = (
const isLoggedOut = isLoggedIn === false
return {
refetch: query.refetch,
...query,
user: isLoggedOut ? undefined : (query.data as ApiUser | undefined),
isLoggedIn,
logout,
@@ -18,9 +18,9 @@ const Heading = styled('h2', {
width: 'fit-content',
marginBottom: 0,
fontSize: '1.3rem',
fontWeight: '700',
fontWeight: '600',
marginTop: '0.75rem',
lineHeight: '1.7rem',
lineHeight: '2rem',
maxWidth: '23rem',
textAlign: 'center',
textWrap: 'balance',
@@ -120,7 +120,7 @@ const Separator = styled('div', {
const Heading = styled('h1', {
base: {
fontWeight: '700',
fontWeight: '600',
fontStyle: 'normal',
fontStretch: 'normal',
fontOpticalSizing: 'auto',
@@ -298,7 +298,7 @@ export const Join = ({
case ApiLobbyStatus.TIMEOUT:
return (
<VStack alignItems="center" textAlign="center">
<H lvl={1} margin={false} centered>
<H lvl={1} margin={false}>
{t('timeoutInvite.title')}
</H>
<Text as="p" variant="note">
@@ -310,7 +310,7 @@ export const Join = ({
case ApiLobbyStatus.DENIED:
return (
<VStack alignItems="center" textAlign="center">
<H lvl={1} margin={false} centered>
<H lvl={1} margin={false}>
{t('denied.title')}
</H>
<Text as="p" variant="note">
@@ -322,7 +322,7 @@ export const Join = ({
case ApiLobbyStatus.WAITING:
return (
<VStack alignItems="center" textAlign="center">
<H lvl={1} margin={false} centered>
<H lvl={1} margin={false}>
{t('waiting.title')}
</H>
<Text
@@ -346,7 +346,7 @@ export const Join = ({
}}
>
<VStack marginBottom={1}>
<H lvl={1} margin="sm" centered>
<H lvl={1} margin="sm">
{t('heading')}
</H>
<Field
@@ -75,7 +75,7 @@ export function FloatingReaction({
>
<span
className={css({
lineHeight: '57px',
lineHeight: '45px',
})}
>
{emoji}
@@ -89,10 +89,9 @@ export function FloatingReaction({
textAlign: 'center',
borderRadius: '20px',
paddingX: '0.5rem',
paddingBottom: '0.3125rem',
paddingTop: '0.15rem',
paddingY: '0.15rem',
boxShadow: '0 2px 4px rgba(0, 0, 0, 0.1)',
lineHeight: '16px',
lineHeight: '14px',
})}
>
{name}
@@ -17,9 +17,6 @@ import {
ProcessorType,
} from '.'
import { DrawingUtils } from './DrawingUtils'
import { DrawingUtils2 } from '@/features/rooms/livekit/components/blur/DrawingUtils2.ts'
const PROCESSING_WIDTH = 256
const PROCESSING_HEIGHT = 144
@@ -28,8 +25,6 @@ const BLUR_CANVAS_ID = 'background-blur-local'
const DEFAULT_BLUR = '10'
export class Wip {}
/**
* This implementation of video blurring is made to be run on CPU for browser that are
* not compatible with track-processor-js.
@@ -55,11 +50,8 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
imageSegmenterResult?: ImageSegmenterResult
// Canvas used for resizing video source and projecting mask.
workCanvas?: HTMLCanvasElement
// segmentationMaskCanvasCtx?: CanvasRenderingContext2D
drawingUtils: any
segmentationMaskCanvasCtx: any
segmentationMaskCanvas: any
segmentationMaskCanvas?: HTMLCanvasElement
segmentationMaskCanvasCtx?: CanvasRenderingContext2D
// Mask containg the inference result.
segmentationMask?: ImageData
@@ -98,11 +90,10 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
this._initVirtualBackgroundImage()
this._createMainCanvas()
this._createWorkCanvas()
this._createMaskCanvas()
const stream = this.outputCanvas!.captureStream()
const tracks = stream.getVideoTracks()
if (tracks.length == 0) {
throw new Error('No tracks found for processing')
}
@@ -168,16 +159,38 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
baseOptions: {
modelAssetPath:
'https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/float16/latest/selfie_segmenter_landscape.tflite',
// 'https://storage.googleapis.com/mediapipe-models/image_segmenter/deeplab_v3/float32/1/deeplab_v3.tflite',
delegate: 'GPU', // Use CPU for Firefox.
delegate: 'CPU', // Use CPU for Firefox.
},
runningMode: 'VIDEO',
outputCategoryMask: true,
outputConfidenceMasks: false,
canvas: this.workCanvas,
})
}
/**
* Resize the source video to the processing resolution.
*/
async sizeSource() {
this.segmentationMaskCanvasCtx?.drawImage(
this.videoElement!,
0,
0,
this.videoElement!.videoWidth,
this.videoElement!.videoHeight,
0,
0,
PROCESSING_WIDTH,
PROCESSING_HEIGHT
)
this.sourceImageData = this.segmentationMaskCanvasCtx?.getImageData(
0,
0,
PROCESSING_WIDTH,
PROCESSING_WIDTH
)
}
/**
* Run the segmentation.
*/
@@ -185,7 +198,7 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
const startTimeMs = performance.now()
return new Promise<void>((resolve) => {
this.imageSegmenter!.segmentForVideo(
this.videoElement,
this.sourceImageData!,
startTimeMs,
(result: ImageSegmenterResult) => {
this.imageSegmenterResult = result
@@ -199,26 +212,92 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
* TODO: future improvement with WebGL.
*/
async blur() {
// console.log('$$ blur')
const mask = this.imageSegmenterResult!.categoryMask!.getAsUint8Array()
for (let i = 0; i < mask.length; ++i) {
this.segmentationMask!.data[i * 4 + 3] = 255 - mask[i]
}
const legendColor = [[255, 197, 0, 255]]
this.segmentationMaskCanvasCtx!.putImageData(this.segmentationMask!, 0, 0)
this.drawingUtils.applyBackgroundBlur(
this.imageSegmenterResult.categoryMask,
this.videoElement,
3.0
this.outputCanvasCtx!.globalCompositeOperation = 'copy'
this.outputCanvasCtx!.filter = 'blur(8px)'
// Put opacity mask.
this.outputCanvasCtx!.drawImage(
this.segmentationMaskCanvas!,
0,
0,
PROCESSING_WIDTH,
PROCESSING_HEIGHT,
0,
0,
this.videoElement!.videoWidth,
this.videoElement!.videoHeight
)
// Draw clear body.
this.outputCanvasCtx!.globalCompositeOperation = 'source-in'
this.outputCanvasCtx!.filter = 'none'
this.outputCanvasCtx!.drawImage(this.videoElement!, 0, 0)
// Draw blurry background.
this.outputCanvasCtx!.globalCompositeOperation = 'destination-over'
this.outputCanvasCtx!.filter = `blur(${this.options.blurRadius ?? DEFAULT_BLUR}px)`
this.outputCanvasCtx!.drawImage(this.videoElement!, 0, 0)
}
/**
* TODO: future improvement with WebGL.
*/
async drawVirtualBackground() {
const mask = this.imageSegmenterResult!.categoryMask!.getAsUint8Array()
for (let i = 0; i < mask.length; ++i) {
this.segmentationMask!.data[i * 4 + 3] = 255 - mask[i]
}
this.segmentationMaskCanvasCtx!.putImageData(this.segmentationMask!, 0, 0)
this.outputCanvasCtx!.globalCompositeOperation = 'copy'
this.outputCanvasCtx!.filter = 'blur(8px)'
// Put opacity mask.
this.outputCanvasCtx!.drawImage(
this.segmentationMaskCanvas!,
0,
0,
PROCESSING_WIDTH,
PROCESSING_HEIGHT,
0,
0,
this.videoElement!.videoWidth,
this.videoElement!.videoHeight
)
// Draw clear body.
this.outputCanvasCtx!.globalCompositeOperation = 'source-in'
this.outputCanvasCtx!.filter = 'none'
this.outputCanvasCtx!.drawImage(this.videoElement!, 0, 0)
// Draw virtual background.
this.outputCanvasCtx!.globalCompositeOperation = 'destination-over'
this.outputCanvasCtx!.drawImage(
this.virtualBackgroundImage!,
0,
0,
this.outputCanvas!.width,
this.outputCanvas!.height
)
}
async process() {
await this.sizeSource()
await this.segment()
if (this.options.blurRadius) {
await this.blur()
} else {
await this.drawVirtualBackground()
}
// } else {
// await this.drawVirtualBackground()
// }
this.timerWorker!.postMessage({
id: SET_TIMEOUT,
timeMs: 1000 / 30,
@@ -239,33 +318,19 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
this.outputCanvasCtx = this.outputCanvas.getContext('2d')!
}
_createWorkCanvas() {
this.workCanvas = document.querySelector(`#workcanvas`)
if (!this.workCanvas) {
this.workCanvas = this._wip(
'workcanvas',
_createMaskCanvas() {
this.segmentationMaskCanvas = document.querySelector(
`#${SEGMENTATION_MASK_CANVAS_ID}`
) as HTMLCanvasElement
if (!this.segmentationMaskCanvas) {
this.segmentationMaskCanvas = this._createCanvas(
SEGMENTATION_MASK_CANVAS_ID,
PROCESSING_WIDTH,
PROCESSING_HEIGHT
)
}
this.drawingUtils = new DrawingUtils2(this.workCanvas.getContext('webgl2'))
}
_wip(id: string, width: number, height: number) {
const wip = document.querySelector(`#wip`)
const element = document.createElement('canvas')
element.setAttribute('id', id)
element.setAttribute('width', '' + width)
element.setAttribute('height', '' + height)
element.style.position = 'absolute'
element.style.top = '0'
element.style.left = '0'
element.style.maxWidth = '611px'
element.style.maxHeight = '305px'
wip.appendChild(element)
return element
this.segmentationMaskCanvasCtx =
this.segmentationMaskCanvas.getContext('2d')!
}
_createCanvas(id: string, width: number, height: number) {
@@ -1,158 +0,0 @@
import { MPMask } from '@mediapipe/tasks-vision'
import { CategoryMaskShaderContext } from './wip'
// Type definitions
type RGBAColor = [number, number, number, number]
type ImageSource =
| HTMLCanvasElement
| HTMLImageElement
| HTMLVideoElement
| ImageData
type CategoryToColorMap = Map<number, RGBAColor> | RGBAColor[]
export class DrawingUtils {
private categoryMaskShaderContext?: CategoryMaskShaderContext
private readonly contextWebGL?: WebGL2RenderingContext
/**
* Creates a new DrawingUtils class.
*
* @param gpuContext The WebGL canvas rendering context to render into. If
* your Task is using a GPU delegate, the context must be obtained from
* its canvas (provided via `setOptions({ canvas: .. })`).
*/
constructor(gpuContext: WebGL2RenderingContext)
/**
* Creates a new DrawingUtils class.
*
* @param cpuContext The 2D canvas rendering context to render into. If
* you are rendering GPU data you must also provide `gpuContext` to allow
* for data conversion.
* @param gpuContext A WebGL canvas that is used for GPU rendering and for
* converting GPU to CPU data. If your Task is using a GPU delegate, the
* context must be obtained from its canvas (provided via
* `setOptions({ canvas: .. })`).
*/
constructor(
cpuContext: CanvasRenderingContext2D | OffscreenCanvasRenderingContext2D,
gpuContext?: WebGL2RenderingContext
)
constructor(
cpuOrGpuGontext:
| CanvasRenderingContext2D
| OffscreenCanvasRenderingContext2D
| WebGL2RenderingContext,
gpuContext?: WebGL2RenderingContext
) {
if (
(typeof CanvasRenderingContext2D !== 'undefined' &&
cpuOrGpuGontext instanceof CanvasRenderingContext2D) ||
cpuOrGpuGontext instanceof OffscreenCanvasRenderingContext2D
) {
// this.context2d = cpuOrGpuGontext
this.contextWebGL = gpuContext
} else {
// If the first `if` statement is false, then the first argument must be a
// WebGL2RenderingContext, since CanvasRenderingContext2D can't be passed
// as the first argument. However, typescript isn't smart enough to infer
// this so we cast.
this.contextWebGL = cpuOrGpuGontext as WebGL2RenderingContext
}
}
private getWebGLRenderingContext(): WebGL2RenderingContext {
if (!this.contextWebGL) {
throw new Error(
'GPU rendering requested but WebGL2RenderingContext not provided.'
)
}
return this.contextWebGL
}
private getCategoryMaskShaderContext(): CategoryMaskShaderContext {
if (!this.categoryMaskShaderContext) {
this.categoryMaskShaderContext = new CategoryMaskShaderContext()
}
return this.categoryMaskShaderContext
}
/** Draws a category mask on a WebGL2RenderingContext2D. */
private drawCategoryMaskWebGL(
categoryTexture: WebGLTexture,
background: RGBAColor | ImageSource,
categoryToColorMap: Map<number, RGBAColor> | RGBAColor[]
): void {
const shaderContext = this.getCategoryMaskShaderContext()
const gl = this.getWebGLRenderingContext()
const backgroundImage = Array.isArray(background)
? new ImageData(new Uint8ClampedArray(background), 1, 1)
: background
shaderContext.run(gl, /* flipTexturesVertically= */ true, () => {
shaderContext.bindAndUploadTextures(
categoryTexture,
backgroundImage,
categoryToColorMap
)
gl.clearColor(0, 0, 0, 0)
gl.clear(gl.COLOR_BUFFER_BIT)
gl.drawArrays(gl.TRIANGLE_FAN, 0, 4)
shaderContext.unbindTextures()
})
}
/**
* Draws a category mask using the provided category-to-color mapping.
*
* @export
* @param mask A category mask that was returned from a segmentation task.
* @param categoryToColorMap A map that maps category indices to RGBA
* values. You must specify a map entry for each category.
* @param background A color or image to use as the background. Defaults to
* black.
*/
drawCategoryMask(
mask: MPMask,
categoryToColorMap: Map<number, RGBAColor>,
background?: RGBAColor | ImageSource
): void
/**
* Draws a category mask using the provided color array.
*
* @export
* @param mask A category mask that was returned from a segmentation task.
* @param categoryToColorMap An array that maps indices to RGBA values. The
* array's indices must correspond to the category indices of the model
* and an entry must be provided for each category.
* @param background A color or image to use as the background. Defaults to
* black.
*/
drawCategoryMask(
mask: MPMask,
categoryToColorMap: RGBAColor[],
background?: RGBAColor | ImageSource
): void
/** @export */
drawCategoryMask(
mask: MPMask,
categoryToColorMap: CategoryToColorMap,
background: RGBAColor | ImageSource = [0, 0, 0, 255]
): void {
this.drawCategoryMaskWebGL(
mask.getAsWebGLTexture(),
background,
categoryToColorMap
)
}
/**
* Frees all WebGL resources held by this class.
* @export
*/
close(): void {
this.categoryMaskShaderContext?.close()
this.categoryMaskShaderContext = undefined
}
}
@@ -1,120 +0,0 @@
import { MPMask } from '@mediapipe/tasks-vision'
import { BackgroundBlurShaderContext } from './wip2.ts'
// Type definitions
type ImageSource =
| HTMLCanvasElement
| HTMLImageElement
| HTMLVideoElement
| ImageData
export class DrawingUtils2 {
private backgroundBlurShaderContext?: BackgroundBlurShaderContext
private readonly contextWebGL?: WebGL2RenderingContext
/**
* Creates a new DrawingUtils class.
*
* @param gpuContext The WebGL canvas rendering context to render into. If
* your Task is using a GPU delegate, the context must be obtained from
* its canvas (provided via `setOptions({ canvas: .. })`).
*/
constructor(gpuContext: WebGL2RenderingContext)
/**
* Creates a new DrawingUtils class.
*
* @param cpuContext The 2D canvas rendering context to render into. If
* you are rendering GPU data you must also provide `gpuContext` to allow
* for data conversion.
* @param gpuContext A WebGL canvas that is used for GPU rendering and for
* converting GPU to CPU data. If your Task is using a GPU delegate, the
* context must be obtained from its canvas (provided via
* `setOptions({ canvas: .. })`).
*/
constructor(
cpuContext: CanvasRenderingContext2D | OffscreenCanvasRenderingContext2D,
gpuContext?: WebGL2RenderingContext
)
constructor(
cpuOrGpuContext:
| CanvasRenderingContext2D
| OffscreenCanvasRenderingContext2D
| WebGL2RenderingContext,
gpuContext?: WebGL2RenderingContext
) {
if (
(typeof CanvasRenderingContext2D !== 'undefined' &&
cpuOrGpuContext instanceof CanvasRenderingContext2D) ||
cpuOrGpuContext instanceof OffscreenCanvasRenderingContext2D
) {
this.contextWebGL = gpuContext
} else {
// If the first condition is false, then the first argument must be a
// WebGL2RenderingContext
this.contextWebGL = cpuOrGpuContext as WebGL2RenderingContext
}
}
private getWebGLRenderingContext(): WebGL2RenderingContext {
if (!this.contextWebGL) {
throw new Error(
'GPU rendering requested but WebGL2RenderingContext not provided.'
)
}
return this.contextWebGL
}
private getBackgroundBlurShaderContext(): BackgroundBlurShaderContext {
if (!this.backgroundBlurShaderContext) {
this.backgroundBlurShaderContext = new BackgroundBlurShaderContext()
}
return this.backgroundBlurShaderContext
}
/** Applies background blur effect using WebGL2. */
private applyBackgroundBlurWebGL(
maskTexture: WebGLTexture,
background: ImageSource,
blurAmount: number = 1.0
): void {
const shaderContext = this.getBackgroundBlurShaderContext()
const gl = this.getWebGLRenderingContext()
shaderContext.run(gl, /* flipTexturesVertically= */ true, () => {
shaderContext.bindAndUploadTextures(maskTexture, background, blurAmount)
gl.clearColor(0, 0, 0, 0)
gl.clear(gl.COLOR_BUFFER_BIT)
gl.drawArrays(gl.TRIANGLE_FAN, 0, 4)
shaderContext.unbindTextures()
})
}
/**
* Applies a background blur effect using the mask to preserve the foreground.
*
* @export
* @param mask A binary mask that was returned from a segmentation task (1 for foreground, 0 for background).
* @param background The input image to blur the background of.
* @param blurAmount Optional blur amount parameter (defaults to 1.0, higher values create stronger blur).
*/
applyBackgroundBlur(
mask: MPMask,
background: ImageSource,
blurAmount: number = 1.0
): void {
this.applyBackgroundBlurWebGL(
mask.getAsWebGLTexture(),
background,
blurAmount
)
}
/**
* Frees all WebGL resources held by this class.
* @export
*/
close(): void {
this.backgroundBlurShaderContext?.close()
this.backgroundBlurShaderContext = undefined
}
}
@@ -1,324 +0,0 @@
/**
* Copyright 2023 The MediaPipe Authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
const VERTEX_SHADER = `#version 300 es
in vec2 aVertex;
in vec2 aTex;
out vec2 vTex;
void main(void) {
gl_Position = vec4(aVertex, 0.0, 1.0);
vTex = aTex;
}`
const FRAGMENT_SHADER = `#version 300 es
precision mediump float;
in vec2 vTex;
uniform sampler2D inputTexture;
out vec4 fragColor;
void main() {
fragColor = texture(inputTexture, vTex);
}
`
/** Helper to assert that `value` is not null or undefined. */
export function assertExists<T>(value: T, msg: string): NonNullable<T> {
if (!value) {
throw new Error(`Unable to obtain required WebGL resource: ${msg}`)
}
return value
}
/**
* Utility class that encapsulates the buffers used by `MPImageShaderContext`.
* For internal use only.
*/
class MPImageShaderBuffers {
constructor(
private readonly gl: WebGL2RenderingContext,
private readonly vertexArrayObject: WebGLVertexArrayObject,
private readonly vertexBuffer: WebGLBuffer,
private readonly textureBuffer: WebGLBuffer
) {}
bind() {
this.gl.bindVertexArray(this.vertexArrayObject)
}
unbind() {
this.gl.bindVertexArray(null)
}
close() {
this.gl.deleteVertexArray(this.vertexArrayObject)
this.gl.deleteBuffer(this.vertexBuffer)
this.gl.deleteBuffer(this.textureBuffer)
}
}
/**
* A class that encapsulates the shaders used by an MPImage. Can be re-used
* across MPImages that use the same WebGL2Rendering context.
*
* For internal use only.
*/
export class MPImageShaderContext {
protected gl?: WebGL2RenderingContext
private framebuffer?: WebGLFramebuffer
protected program?: WebGLProgram
private vertexShader?: WebGLShader
private fragmentShader?: WebGLShader
private aVertex?: GLint
private aTex?: GLint
/**
* The shader buffers used for passthrough renders that don't modify the
* input texture.
*/
private shaderBuffersPassthrough?: MPImageShaderBuffers
/**
* The shader buffers used for passthrough renders that flip the input texture
* vertically before conversion to a different type. This is used to flip the
* texture to the expected orientation for drawing in the browser.
*/
private shaderBuffersFlipVertically?: MPImageShaderBuffers
protected getFragmentShader(): string {
return FRAGMENT_SHADER
}
protected getVertexShader(): string {
return VERTEX_SHADER
}
private compileShader(source: string, type: number): WebGLShader {
const gl = this.gl!
const shader = assertExists(
gl.createShader(type),
'Failed to create WebGL shader'
)
gl.shaderSource(shader, source)
gl.compileShader(shader)
if (!gl.getShaderParameter(shader, gl.COMPILE_STATUS)) {
const info = gl.getShaderInfoLog(shader)
throw new Error(`Could not compile WebGL shader: ${info}`)
}
gl.attachShader(this.program!, shader)
return shader
}
protected setupShaders(): void {
const gl = this.gl!
this.program = assertExists(
gl.createProgram()!,
'Failed to create WebGL program'
)
this.vertexShader = this.compileShader(
this.getVertexShader(),
gl.VERTEX_SHADER
)
this.fragmentShader = this.compileShader(
this.getFragmentShader(),
gl.FRAGMENT_SHADER
)
gl.linkProgram(this.program)
const linked = gl.getProgramParameter(this.program, gl.LINK_STATUS)
if (!linked) {
const info = gl.getProgramInfoLog(this.program)
throw new Error(`Error during program linking: ${info}`)
}
this.aVertex = gl.getAttribLocation(this.program, 'aVertex')
this.aTex = gl.getAttribLocation(this.program, 'aTex')
}
protected setupTextures(): void {}
protected configureUniforms(): void {}
private createBuffers(flipVertically: boolean): MPImageShaderBuffers {
const gl = this.gl!
const vertexArrayObject = assertExists(
gl.createVertexArray(),
'Failed to create vertex array'
)
gl.bindVertexArray(vertexArrayObject)
const vertexBuffer = assertExists(
gl.createBuffer(),
'Failed to create buffer'
)
gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer)
gl.enableVertexAttribArray(this.aVertex!)
gl.vertexAttribPointer(this.aVertex!, 2, gl.FLOAT, false, 0, 0)
gl.bufferData(
gl.ARRAY_BUFFER,
new Float32Array([-1, -1, -1, 1, 1, 1, 1, -1]),
gl.STATIC_DRAW
)
const textureBuffer = assertExists(
gl.createBuffer(),
'Failed to create buffer'
)
gl.bindBuffer(gl.ARRAY_BUFFER, textureBuffer)
gl.enableVertexAttribArray(this.aTex!)
gl.vertexAttribPointer(this.aTex!, 2, gl.FLOAT, false, 0, 0)
const bufferData = flipVertically
? [0, 1, 0, 0, 1, 0, 1, 1]
: [0, 0, 0, 1, 1, 1, 1, 0]
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(bufferData), gl.STATIC_DRAW)
gl.bindBuffer(gl.ARRAY_BUFFER, null)
gl.bindVertexArray(null)
return new MPImageShaderBuffers(
gl,
vertexArrayObject,
vertexBuffer,
textureBuffer
)
}
private getShaderBuffers(flipVertically: boolean): MPImageShaderBuffers {
if (flipVertically) {
if (!this.shaderBuffersFlipVertically) {
this.shaderBuffersFlipVertically = this.createBuffers(
/* flipVertically= */ true
)
}
return this.shaderBuffersFlipVertically
} else {
if (!this.shaderBuffersPassthrough) {
this.shaderBuffersPassthrough = this.createBuffers(
/* flipVertically= */ false
)
}
return this.shaderBuffersPassthrough
}
}
private maybeInitGL(gl: WebGL2RenderingContext): void {
if (!this.gl) {
this.gl = gl
} else if (gl !== this.gl) {
throw new Error('Cannot change GL context once initialized')
}
}
/** Runs the callback using the shader. */
run<T>(
gl: WebGL2RenderingContext,
flipVertically: boolean,
callback: () => T
): T {
this.maybeInitGL(gl)
if (!this.program) {
this.setupShaders()
this.setupTextures()
}
const shaderBuffers = this.getShaderBuffers(flipVertically)
gl.useProgram(this.program!)
shaderBuffers.bind()
this.configureUniforms()
const result = callback()
shaderBuffers.unbind()
return result
}
/**
* Creates and configures a texture.
*
* @param gl The rendering context.
* @param filter The setting to use for `gl.TEXTURE_MIN_FILTER` and
* `gl.TEXTURE_MAG_FILTER`. Defaults to `gl.LINEAR`.
* @param wrapping The setting to use for `gl.TEXTURE_WRAP_S` and
* `gl.TEXTURE_WRAP_T`. Defaults to `gl.CLAMP_TO_EDGE`.
*/
createTexture(
gl: WebGL2RenderingContext,
filter?: GLenum,
wrapping?: GLenum
): WebGLTexture {
this.maybeInitGL(gl)
const texture = assertExists(gl.createTexture(), 'Failed to create texture')
gl.bindTexture(gl.TEXTURE_2D, texture)
gl.texParameteri(
gl.TEXTURE_2D,
gl.TEXTURE_WRAP_S,
wrapping ?? gl.CLAMP_TO_EDGE
)
gl.texParameteri(
gl.TEXTURE_2D,
gl.TEXTURE_WRAP_T,
wrapping ?? gl.CLAMP_TO_EDGE
)
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, filter ?? gl.LINEAR)
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, filter ?? gl.LINEAR)
gl.bindTexture(gl.TEXTURE_2D, null)
return texture
}
/**
* Binds a framebuffer to the canvas. If the framebuffer does not yet exist,
* creates it first. Binds the provided texture to the framebuffer.
*/
bindFramebuffer(gl: WebGL2RenderingContext, texture: WebGLTexture): void {
this.maybeInitGL(gl)
if (!this.framebuffer) {
this.framebuffer = assertExists(
gl.createFramebuffer(),
'Failed to create framebuffe.'
)
}
gl.bindFramebuffer(gl.FRAMEBUFFER, this.framebuffer)
gl.framebufferTexture2D(
gl.FRAMEBUFFER,
gl.COLOR_ATTACHMENT0,
gl.TEXTURE_2D,
texture,
0
)
}
unbindFramebuffer(): void {
this.gl?.bindFramebuffer(this.gl.FRAMEBUFFER, null)
}
close() {
if (this.program) {
const gl = this.gl!
gl.deleteProgram(this.program)
gl.deleteShader(this.vertexShader!)
gl.deleteShader(this.fragmentShader!)
}
if (this.framebuffer) {
this.gl!.deleteFramebuffer(this.framebuffer)
}
if (this.shaderBuffersPassthrough) {
this.shaderBuffersPassthrough.close()
}
if (this.shaderBuffersFlipVertically) {
this.shaderBuffersFlipVertically.close()
}
}
}
@@ -36,25 +36,22 @@ export class BackgroundProcessorFactory {
type: ProcessorType,
opts: BackgroundOptions
): BackgroundProcessorInterface | undefined {
return new BackgroundCustomProcessor(opts)
// if (type === ProcessorType.BLUR) {
// if (ProcessorWrapper.isSupported) {
//
// }
// if (BackgroundCustomProcessor.isSupported) {
// return new BackgroundCustomProcessor(opts)
// }
// } else if (type === ProcessorType.VIRTUAL) {
// if (ProcessorWrapper.isSupported) {
// return new BackgroundVirtualTrackProcessorJsWrapper(opts)
// }
// if (BackgroundCustomProcessor.isSupported) {
// return new BackgroundCustomProcessor(opts)
// }
// }
// return undefined
if (type === ProcessorType.BLUR) {
if (ProcessorWrapper.isSupported) {
return new BackgroundBlurTrackProcessorJsWrapper(opts)
}
if (BackgroundCustomProcessor.isSupported) {
return new BackgroundCustomProcessor(opts)
}
} else if (type === ProcessorType.VIRTUAL) {
if (ProcessorWrapper.isSupported) {
return new BackgroundVirtualTrackProcessorJsWrapper(opts)
}
if (BackgroundCustomProcessor.isSupported) {
return new BackgroundCustomProcessor(opts)
}
}
return undefined
}
static deserializeProcessor(data?: ProcessorSerialized) {
@@ -1,217 +0,0 @@
/**
* Copyright 2023 The MediaPipe Authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { MPImageShaderContext, assertExists } from './image_shader_context'
import { ImageSource } from '@mediapipe/tasks-vision'
/**
* A fragment shader that maps categories to colors based on a background
* texture, a mask texture and a 256x1 "color mapping texture" that contains one
* color for each pixel.
*/
const FRAGMENT_SHADER = `
precision mediump float;
uniform sampler2D backgroundTexture;
uniform sampler2D maskTexture;
uniform sampler2D colorMappingTexture;
varying vec2 vTex;
void main() {
vec4 backgroundColor = texture2D(backgroundTexture, vTex);
float category = texture2D(maskTexture, vTex).r;
vec4 categoryColor = texture2D(colorMappingTexture, vec2(category, 0.0));
gl_FragColor = mix(backgroundColor, categoryColor, categoryColor.a);
}
`
/**
* A four channel color with values for red, green, blue and alpha
* respectively.
*/
export type RGBAColor = [number, number, number, number] | number[]
/**
* A category to color mapping that uses either a map or an array to assign
* category indexes to RGBA colors.
*/
export type CategoryToColorMap = Map<number, RGBAColor> | RGBAColor[]
/** Checks CategoryToColorMap maps for deep equality. */
function isEqualColorMap(
a: CategoryToColorMap,
b: CategoryToColorMap
): boolean {
if (a !== b) {
return false
}
const aEntries = a.entries()
const bEntries = b.entries()
for (const [aKey, aValue] of aEntries) {
const bNext = bEntries.next()
if (bNext.done) {
return false
}
const [bKey, bValue] = bNext.value
if (aKey !== bKey) {
return false
}
if (
aValue[0] !== bValue[0] ||
aValue[1] !== bValue[1] ||
aValue[2] !== bValue[2] ||
aValue[3] !== bValue[3]
) {
return false
}
}
return !!bEntries.next().done
}
/** A drawing util class for category masks. */
export class CategoryMaskShaderContext extends MPImageShaderContext {
backgroundTexture?: WebGLTexture
colorMappingTexture?: WebGLTexture
colorMappingTextureUniform?: WebGLUniformLocation
backgroundTextureUniform?: WebGLUniformLocation
maskTextureUniform?: WebGLUniformLocation
currentColorMap?: CategoryToColorMap
bindAndUploadTextures(
categoryMask: WebGLTexture,
background: ImageSource,
colorMap: Map<number, number[]> | number[][]
) {
const gl = this.gl!
// Bind category mask
gl.activeTexture(gl.TEXTURE0)
gl.bindTexture(gl.TEXTURE_2D, categoryMask)
// TODO: We should avoid uploading textures from CPU to GPU
// if the textures haven't changed. This can lead to drastic performance
// slowdowns (~50ms per frame). Users can reduce the penalty by passing a
// canvas object instead of ImageData/HTMLImageElement.
gl.activeTexture(gl.TEXTURE1)
gl.bindTexture(gl.TEXTURE_2D, this.backgroundTexture!)
gl.texImage2D(
gl.TEXTURE_2D,
0,
gl.RGBA,
gl.RGBA,
gl.UNSIGNED_BYTE,
background
)
// Bind color mapping texture if changed.
if (
!this.currentColorMap ||
!isEqualColorMap(this.currentColorMap, colorMap)
) {
this.currentColorMap = colorMap
const pixels = new Array(256 * 4).fill(0)
colorMap.forEach((rgba, index) => {
if (rgba.length !== 4) {
throw new Error(
`Color at index ${index} is not a four-channel value.`
)
}
pixels[index * 4] = rgba[0]
pixels[index * 4 + 1] = rgba[1]
pixels[index * 4 + 2] = rgba[2]
pixels[index * 4 + 3] = rgba[3]
})
gl.activeTexture(gl.TEXTURE2)
gl.bindTexture(gl.TEXTURE_2D, this.colorMappingTexture!)
gl.texImage2D(
gl.TEXTURE_2D,
0,
gl.RGBA,
256,
1,
0,
gl.RGBA,
gl.UNSIGNED_BYTE,
new Uint8Array(pixels)
)
} else {
gl.activeTexture(gl.TEXTURE2)
gl.bindTexture(gl.TEXTURE_2D, this.colorMappingTexture!)
}
}
unbindTextures() {
const gl = this.gl!
gl.activeTexture(gl.TEXTURE0)
gl.bindTexture(gl.TEXTURE_2D, null)
gl.activeTexture(gl.TEXTURE1)
gl.bindTexture(gl.TEXTURE_2D, null)
gl.activeTexture(gl.TEXTURE2)
gl.bindTexture(gl.TEXTURE_2D, null)
}
protected override getFragmentShader(): string {
return FRAGMENT_SHADER
}
protected override setupTextures(): void {
const gl = this.gl!
gl.activeTexture(gl.TEXTURE1)
this.backgroundTexture = this.createTexture(gl, gl.LINEAR)
// Use `gl.NEAREST` to prevent interpolating values in our category to
// color map.
gl.activeTexture(gl.TEXTURE2)
this.colorMappingTexture = this.createTexture(gl, gl.NEAREST)
}
protected override setupShaders(): void {
super.setupShaders()
const gl = this.gl!
this.backgroundTextureUniform = assertExists(
gl.getUniformLocation(this.program!, 'backgroundTexture'),
'Uniform location'
)
this.colorMappingTextureUniform = assertExists(
gl.getUniformLocation(this.program!, 'colorMappingTexture'),
'Uniform location'
)
this.maskTextureUniform = assertExists(
gl.getUniformLocation(this.program!, 'maskTexture'),
'Uniform location'
)
}
protected override configureUniforms(): void {
super.configureUniforms()
const gl = this.gl!
gl.uniform1i(this.maskTextureUniform!, 0)
gl.uniform1i(this.backgroundTextureUniform!, 1)
gl.uniform1i(this.colorMappingTextureUniform!, 2)
}
override close(): void {
if (this.backgroundTexture) {
this.gl!.deleteTexture(this.backgroundTexture)
}
if (this.colorMappingTexture) {
this.gl!.deleteTexture(this.colorMappingTexture)
}
super.close()
}
}
@@ -1,202 +0,0 @@
/**
* Background Blur Shader Implementation
*
* Implements a high-quality background blur while preserving
* foreground based on a mask. Uses a multi-pass blur approach
* for a smoother, more professional-looking effect.
*
* This implementation INVERTS the mask to blur what's outside the mask.
*/
import { MPImageShaderContext, assertExists } from './image_shader_context'
import { ImageSource } from '@mediapipe/tasks-vision'
/**
* Two-pass blur with smooth transition between foreground and background.
* Uses a larger kernel and multiple passes for a higher quality blur.
*
* This version inverts the mask interpretation - it blurs what's OUTSIDE the mask.
*/
const FRAGMENT_SHADER = `#version 300 es
precision mediump float;
uniform sampler2D backgroundTexture;
uniform sampler2D maskTexture;
uniform vec2 texelSize;
uniform float blurAmount;
in vec2 vTex;
out vec4 fragColor;
// Smoothing function for mask transitions
vec4 maskSmoothing() {
// Sample the mask at the current texel
float maskValue = texture(maskTexture, vTex).r;
// Apply smoothing to reduce aliasing at mask edges
float smoothedMask = smoothstep(0.2, 0.8, maskValue);
return vec4(smoothedMask);
}
// Compute Gaussian weight based on distance and blur amount
float gaussianWeight(float x, float y, float sigma) {
float sigmaSq = sigma * sigma;
float distanceSq = x * x + y * y;
// Gaussian function: (1/(2πσ²)) * e^(-((x²+y²)/(2σ²)))
return exp(-distanceSq / (2.0 * sigmaSq));
}
void main() {
vec4 centerColor = texture(backgroundTexture, vTex);
// Apply bilateral filtering on the mask before using it
vec4 personMask = maskSmoothing();
// Dynamic 5x5 Gaussian blur
vec4 blurredColor = vec4(0.0);
float totalWeight = 0.0;
// Compute effective sigma from blur amount (adjust as needed)
float sigma = max(0.1, blurAmount) + 50.0;
// Apply 5x5 kernel with dynamic weights
for (int y = -10; y <= 10; y++) {
for (int x = -10; x <= 10; x++) {
// Calculate Gaussian weight for this offset
float weight = gaussianWeight(float(x), float(y), sigma);
totalWeight += weight;
vec2 offset = vec2(float(x), float(y)) * texelSize;
vec2 sampleCoord = vTex + offset;
blurredColor += texture(backgroundTexture, sampleCoord) * weight;
}
}
// Normalize by total weight
blurredColor /= totalWeight;
// Blend original color with blurred background based on mask
fragColor = mix(centerColor, blurredColor, personMask.r);
}
`;
/** A drawing util class for high-quality background blur. */
export class BackgroundBlurShaderContext extends MPImageShaderContext {
backgroundTexture?: WebGLTexture
maskTextureUniform?: WebGLUniformLocation
backgroundTextureUniform?: WebGLUniformLocation
texelSizeUniform?: WebGLUniformLocation
blurAmountUniform?: WebGLUniformLocation
bindAndUploadTextures(
foregroundMask: WebGLTexture,
background: ImageSource,
blurAmount: number = 1.0
) {
const gl = this.gl!
// Bind foreground mask
gl.activeTexture(gl.TEXTURE0)
gl.bindTexture(gl.TEXTURE_2D, foregroundMask)
// Bind background texture
gl.activeTexture(gl.TEXTURE1)
gl.bindTexture(gl.TEXTURE_2D, this.backgroundTexture!)
gl.texImage2D(
gl.TEXTURE_2D,
0,
gl.RGBA,
gl.RGBA,
gl.UNSIGNED_BYTE,
background
)
// Set texel size (for proper blur scaling)
let width = 1;
let height = 1;
if (background instanceof ImageData) {
width = background.width;
height = background.height;
} else if (background instanceof HTMLCanvasElement) {
width = background.width;
height = background.height;
} else if (background instanceof HTMLImageElement) {
width = background.naturalWidth;
height = background.naturalHeight;
} else if (background instanceof HTMLVideoElement) {
width = background.videoWidth;
height = background.videoHeight;
}
// Ensure we have valid dimensions to prevent division by zero
width = Math.max(width, 1);
height = Math.max(height, 1);
gl.uniform2f(this.texelSizeUniform!, 1.0 / width, 1.0 / height)
// Set blur amount
if (this.blurAmountUniform) {
gl.uniform1f(this.blurAmountUniform, blurAmount)
}
}
unbindTextures() {
const gl = this.gl!
gl.activeTexture(gl.TEXTURE0)
gl.bindTexture(gl.TEXTURE_2D, null)
gl.activeTexture(gl.TEXTURE1)
gl.bindTexture(gl.TEXTURE_2D, null)
}
protected override getFragmentShader(): string {
return FRAGMENT_SHADER
}
protected override setupTextures(): void {
const gl = this.gl!
gl.activeTexture(gl.TEXTURE1)
this.backgroundTexture = this.createTexture(gl, gl.LINEAR)
}
protected override setupShaders(): void {
super.setupShaders()
const gl = this.gl!
this.backgroundTextureUniform = assertExists(
gl.getUniformLocation(this.program!, 'backgroundTexture'),
'Uniform location'
)
this.maskTextureUniform = assertExists(
gl.getUniformLocation(this.program!, 'maskTexture'),
'Uniform location'
)
this.texelSizeUniform = assertExists(
gl.getUniformLocation(this.program!, 'texelSize'),
'Uniform location'
)
this.blurAmountUniform = assertExists(
gl.getUniformLocation(this.program!, 'blurAmount'),
'Uniform location'
)
}
protected override configureUniforms(): void {
super.configureUniforms()
const gl = this.gl!
gl.uniform1i(this.maskTextureUniform!, 0)
gl.uniform1i(this.backgroundTextureUniform!, 1)
}
override close(): void {
if (this.backgroundTexture) {
this.gl!.deleteTexture(this.backgroundTexture)
}
super.close()
}
}
@@ -13,7 +13,6 @@ export const Effects = () => {
<div
className={css({
padding: '0 1.5rem',
overflowY: 'scroll',
})}
>
<EffectsConfiguration
@@ -170,23 +170,16 @@ export const EffectsConfiguration = ({
})}
>
{videoTrack && !videoTrack.isMuted ? (
<div
className={css({
position: 'relative',
})}
id="wip"
>
<video
ref={videoRef}
width="100%"
muted
style={{
transform: 'rotateY(180deg)',
minHeight: '175px',
borderRadius: '8px',
}}
/>
</div>
<video
ref={videoRef}
width="100%"
muted
style={{
transform: 'rotateY(180deg)',
minHeight: '175px',
borderRadius: '8px',
}}
/>
) : (
<div
style={{
@@ -58,14 +58,14 @@ export class SdkReverseClient {
* To be used in SDK scope.
*/
export function useEnsureAuth() {
const { isLoggedIn, refetch } = useUser({
const { isLoggedIn, ...other } = useUser({
fetchUserOptions: { attemptSilent: false },
})
const startSSO = () => {
return new Promise<void>((resolve) => {
SdkReverseClient.waitForAuthenticationAck().then(async () => {
await refetch()
await other.refetch()
resolve()
})
const params = `scrollbars=no,resizable=no,status=no,location=no,toolbar=no,menubar=no,
+1 -1
View File
@@ -106,7 +106,7 @@ const Marianne = () => {
className={css({
letterSpacing: '-.01em',
textTransform: 'uppercase',
fontWeight: '700',
fontWeight: '600',
fontFamily: 'Marianne',
fontSize: '1.25rem',
lineHeight: '1.75rem',
+1 -1
View File
@@ -44,7 +44,7 @@ const Marianne = () => {
className={css({
letterSpacing: '-.01em',
textTransform: 'uppercase',
fontWeight: '700',
fontWeight: '600',
fontFamily: 'Marianne',
fontSize: '0.7875rem !important',
})}
+1 -1
View File
@@ -44,7 +44,7 @@ const FieldWrapper = styled('div', {
const StyledLabel = styled(Label, {
base: {
display: 'block',
fontSize: '0.875rem',
fontSize: '0.75rem',
},
variants: {
center: {
-40
View File
@@ -30,46 +30,6 @@
font-display: swap;
}
@font-face {
font-family: 'Marianne';
src: url('/fonts/Marianne-Regular-subset.woff2') format('woff2');
font-weight: 400;
font-style: normal;
font-display: swap;
}
@font-face {
font-family: 'Marianne';
src: url('/fonts/Marianne-Regular_Italic-subset.woff2') format('woff2');
font-weight: 400;
font-style: italic;
font-display: swap;
}
@font-face {
font-family: 'Marianne';
src: url('/fonts/Marianne-Medium-subset.woff2') format('woff2');
font-weight: 500;
font-style: normal;
font-display: swap;
}
@font-face {
font-family: 'Marianne';
src: url('/fonts/Marianne-Bold-subset.woff2') format('woff2');
font-weight: 700;
font-style: normal;
font-display: swap;
}
@font-face {
font-family: 'Marianne';
src: url('/fonts/Marianne-ExtraBold-subset.woff2') format('woff2');
font-weight: 800;
font-style: normal;
font-display: swap;
}
/*
* to reduce CLS
* values taken from https://github.com/khempenius/font-fallbacks-dataset/blob/main/font-metric-overrides.csv#L2979
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "mail_mjml",
"version": "0.1.16",
"version": "0.1.15",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "mail_mjml",
"version": "0.1.16",
"version": "0.1.15",
"license": "MIT",
"dependencies": {
"@html-to/text-cli": "0.5.4",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "mail_mjml",
"version": "0.1.16",
"version": "0.1.15",
"description": "An util to generate html and text django's templates from mjml templates",
"type": "module",
"dependencies": {
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "sdk",
"version": "0.1.16",
"version": "0.1.15",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "sdk",
"version": "0.1.16",
"version": "0.1.15",
"license": "ISC",
"workspaces": [
"./library",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "sdk",
"version": "0.1.16",
"version": "0.1.15",
"author": "",
"license": "ISC",
"description": "",
+4 -4
View File
@@ -1,7 +1,7 @@
[project]
name = "summary"
version = "0.1.16"
version = "0.1.15"
dependencies = [
"fastapi[standard]>=0.105.0",
"uvicorn>=0.24.0",
@@ -10,14 +10,14 @@ dependencies = [
"celery==5.4.0",
"redis==5.2.1",
"minio==7.2.15",
"openai==1.68.2",
"openai==1.65.2",
"requests==2.32.3",
"sentry-sdk[fastapi, celery]==2.24.1",
"sentry-sdk[fastapi, celery]==2.22.0",
]
[project.optional-dependencies]
dev = [
"ruff==0.11.2",
"ruff==0.9.9",
]
[build-system]
+4 -16
View File
@@ -1,15 +1,10 @@
"""API routes related to application tasks."""
from typing import Optional
from celery.result import AsyncResult
from fastapi import APIRouter
from pydantic import BaseModel
from summary.core.celery_worker import (
process_audio_transcribe_summarize,
process_audio_transcribe_summarize_v2,
)
from summary.core.celery_worker import process_audio_transcribe_summarize
class TaskCreation(BaseModel):
@@ -18,7 +13,6 @@ class TaskCreation(BaseModel):
filename: str
email: str
sub: str
version: Optional[int] = 2
router = APIRouter(prefix="/tasks")
@@ -27,15 +21,9 @@ router = APIRouter(prefix="/tasks")
@router.post("/")
async def create_task(request: TaskCreation):
"""Create a task."""
if request.version == 1:
task = process_audio_transcribe_summarize.delay(
request.filename, request.email, request.sub
)
else:
task = process_audio_transcribe_summarize_v2.delay(
request.filename, request.email, request.sub
)
task = process_audio_transcribe_summarize.delay(
request.filename, request.email, request.sub
)
return {"id": task.id, "message": "Task created"}
-98
View File
@@ -56,35 +56,6 @@ def create_retry_session():
return session
def format_segments(transcription_data):
"""Format transcription segments from WhisperX into a readable conversation format.
Processes transcription data with segments containing speaker information and text,
combining consecutive segments from the same speaker and formatting them as a
conversation with speaker labels.
"""
formatted_output = ""
if not transcription_data or not hasattr(transcription_data, 'segments'):
if isinstance(transcription_data, dict) and 'segments' in transcription_data:
segments = transcription_data['segments']
else:
return "Error: Invalid transcription data format"
else:
segments = transcription_data.segments
previous_speaker = None
for segment in segments:
speaker = segment.get('speaker', 'UNKNOWN_SPEAKER')
text = segment.get('text', '')
if text:
if speaker != previous_speaker:
formatted_output += f"\n\n *{speaker}*: {text}"
else:
formatted_output += f" {text}"
previous_speaker = speaker
return formatted_output
def post_with_retries(url, data):
"""Send POST request with automatic retries."""
session = create_retry_session()
@@ -169,72 +140,3 @@ def process_audio_transcribe_summarize(filename: str, email: str, sub: str):
logger.info("Webhook submitted successfully. Status: %s", response.status_code)
logger.debug("Response body: %s", response.text)
@celery.task(max_retries=settings.celery_max_retries)
def process_audio_transcribe_summarize_v2(filename: str, email: str, sub: str):
"""Process an audio file by transcribing it and generating a summary.
This Celery task performs the following operations:
1. Retrieves the audio file from MinIO storage
2. Transcribes the audio using WhisperX model
3. Sends the results via webhook
"""
logger.info("Notification received")
logger.debug("filename: %s", filename)
minio_client = Minio(
settings.aws_s3_endpoint_url,
access_key=settings.aws_s3_access_key_id,
secret_key=settings.aws_s3_secret_access_key,
secure=settings.aws_s3_secure_access,
)
logger.debug("Connection to the Minio bucket successful")
audio_file_stream = minio_client.get_object(
settings.aws_storage_bucket_name, object_name=filename
)
temp_file_path = save_audio_stream(audio_file_stream)
logger.debug("Recording successfully downloaded, filepath: %s", temp_file_path)
logger.debug("Initiating OpenAI client")
openai_client = openai.OpenAI(
api_key=settings.openai_api_key, base_url=settings.openai_base_url
)
try:
logger.debug("Querying transcription …")
with open(temp_file_path, "rb") as audio_file:
transcription = openai_client.audio.transcriptions.create(
model=settings.openai_asr_model, file=audio_file
)
logger.debug("Transcription: \n %s", transcription)
finally:
if os.path.exists(temp_file_path):
os.remove(temp_file_path)
logger.debug("Temporary file removed: %s", temp_file_path)
formatted_transcription = format_segments(transcription)
data = {
"title": "Transcription",
"content": formatted_transcription,
"email": email,
"sub": sub,
}
logger.debug("Submitting webhook to %s", settings.webhook_url)
logger.debug("Request payload: %s", json.dumps(data, indent=2))
response = post_with_retries(settings.webhook_url, data)
logger.info("Webhook submitted successfully. Status: %s", response.status_code)
logger.debug("Response body: %s", response.text)
# TODO - integrate summarize the transcript and create a new document.