Compare commits

...

173 Commits

Author SHA1 Message Date
Quentin BEY 095bcaea1a 🔖(patch) bump release to 0.0.7
Fixed

- 🚑️(posthog) fix the posthog middleware for async mode #133
2025-10-28 19:01:52 +01:00
Quentin BEY fbe9e039cf 🚑️(posthog) fix the posthog middleware for async mode
The original Posthog middleware tries to get user from request
using the sync way, in both async and sync mode.
2025-10-28 18:19:34 +01:00
Quentin BEY 33a87c3959 ⚰️(settings) remove unused code from copy/paste
These lines where never meant to be used in this project.
2025-10-28 18:09:22 +01:00
Quentin BEY 18fc3390f3 🔖(patch) bump release to 0.0.6
Fixed

- 🚑️(stats) fix tracking id in upload event #130
2025-10-28 09:34:13 +01:00
Quentin BEY 1d54114a39 🚑️(posthog) use same distinct ID as set by frontend
The current behavior was duplicating users because the
frontend uses the user PK as distinct ID (which is
good) but the backend was using the email address.

=> burn the now useless middleware
=> use the user PK as distinct ID
2025-10-28 09:27:51 +01:00
Arnaud Robin 7d8b6fc07c 🚑️(stats) fix tracking id in upload event
Previously, upload event tracking used the user's email instead of
their user ID, preventing reliable association between uploads
and users.
2025-10-27 23:27:34 +01:00
Quentin BEY 2b96ba0597 🔧(django-lasuite) add new settings for back-channel logout
The latest version of the lib, allows to configure OIDC back-
channel logout which requires some specific settings.
While unused in our environment, this allows use by others.
2025-10-27 21:48:59 +01:00
Quentin BEY 34cf348f4c (pydantic-ai) update tests after last update
There is a new consistendy enforcement in the library, which
enforces each message to have a unique ID, therefor the UUID
mock fails (which was expected TBH).
2025-10-27 21:45:12 +01:00
Quentin BEY 0cce897c69 (django-lasuite) update tests after latest release
The library now does a `save` instead of an `update` which
triggers a unicity check.
2025-10-27 21:45:12 +01:00
renovate[bot] 7c8d8e9de7 ⬆️(dependencies) update python dependencies 2025-10-27 21:45:12 +01:00
Quentin BEY 14b920466b 🔖(patch) bump release to 0.0.5
Fixed

- 🚑️(drag-drop) fix the rejection display on Safari #127
2025-10-27 11:25:22 +01:00
Quentin BEY 42017a6180 🚑️(drag-drop) fix the rejection display on Safari
Safari does not fill the file information during drag
so we cannot check "accept" on the fly, we add a fallback.
2025-10-27 10:44:20 +01:00
qbey c89ce82a4a 🌐(i18n) update translated strings
Update translated files with new translations
2025-10-27 09:36:13 +01:00
Quentin BEY a738b6cfc3 🔖(patch) bump release to 0.0.4
Added

- 🌐(i18n) add dutch language #117

Changed

- ️(asgi) use `uvicorn` to serve backend #121

Fixed

- 🐛(front) fix mobile source
- 🐛(attachments) reject the whole drag&drop if unsupported formats #123
2025-10-27 09:27:32 +01:00
Quentin BEY 9c3f8a8541 🐛(attachments) reject the whole drag&drop if unsupported formats
In production, users can upload any file format because the
drag and drop feature does not check their type...
This first implementation is to prevent users to have a bad
experience on this.
2025-10-24 15:46:16 +02:00
Quentin BEY 09e885d7e7 ⚗️(LLM) allow to mock model call for conversation
This is switch to prevent real model call in a deployed stack.
This is usefull to test heavy load on the servers without
inference costs.
2025-10-23 18:59:14 +02:00
Quentin BEY 05a1844a0c ️(asgi) use uvicorn to serve backend
This is a naive first switch from sync to async.
This enables the backend to still answer to incomming requests
while streaming LLM results to the user.

For sure there is room for code cleaning and improvements, but
this provides a nice improvement out of the box.
2025-10-23 18:59:14 +02:00
elvoisin a82e0b4fa0 🐛(front) fix mobile source (#119)
fix show source mobile
2025-10-22 16:45:57 +02:00
Berry den Hartog 60b8338ae5 🌐(i18n) add dutch language
Dutch translations are up to date we can add the language in code.
2025-10-22 16:25:18 +02:00
renovate[bot] bbc8dad9da ⬆️(dependencies) update pylint to v3.3.9 2025-10-22 15:57:18 +02:00
qbey f9e446ec18 🌐(i18n) update translated strings
Update translated files with new translations

Fixes Dutch translation missing character.
2025-10-22 15:48:20 +02:00
qbey a013c69ba7 🌐(i18n) update translated strings
Update translated files with new translations
2025-10-22 13:31:36 +02:00
Quentin BEY 2a79655edb 🔖(patch) bump release to 0.0.3
Fixed

- 🚑️(web-search) fix missing argument in RAG backend #116
2025-10-21 23:54:59 +02:00
Quentin BEY ad4b5473aa 🚑️(web-search) fix missing argument in RAG backend
The wrong RAG backend (not the one used in production) was
updated in a previous commit...
2025-10-21 23:49:21 +02:00
qbey b8e6e32fed 🌐(i18n) update translated strings
Update translated files with new translations
2025-10-21 22:49:59 +02:00
Quentin BEY 0a5f71ef18 🔖(patch) bump release to 0.0.2
Added

- (front) add drag'n drop file
- (activation-codes) register users also on Brevo #98
- 📈(posthog) add `sub` field to tracking #95

Changed

- 🔧(front) change links feedback tchap + settings popup
- 🐛(front) code activation fix session end #93
- 💬(wording) error page wording #102
- ️(web-search) allow to override returned chunks #107
- 🐛(activation-codes) create contact in brevo before add to list #108
- ⚗️(summarization) add system prompt to handle tool #112
2025-10-21 22:45:32 +02:00
Quentin BEY 151914f55c ⚗️(summarization) add system prompt to handle tool
With these modifications, we expect the LLM to not sumarize the
summarization tool result.
2025-10-21 22:12:50 +02:00
elvoisin ab45eea736 🔧(front) update links feedback + popup settings (#111)
change tchap channel + change links in settings popup
2025-10-21 17:28:19 +02:00
elvoisin ef7c15507d (front) add drag'n drop file (#110)
add drag n drop in input chat on the entire screen
2025-10-21 16:04:07 +02:00
Quentin BEY ca0c3a0169 🐛(activation-codes) create contact in brevo before add to list
Brevo requires the contact to exist before being added to a list.
2025-10-21 14:18:25 +02:00
Quentin BEY 6b2327b0f1 ️(web-search) allow to override returned chunks
When using the document vector storage, we want to be able to
return more chunks.
2025-10-21 13:55:04 +02:00
Eléonore Voisin 9887d79bd5 💬(wording) error page wording
change error page wording 401 - 403
2025-10-21 13:21:11 +02:00
Arnaud Robin d1852d210c 📈(posthog) add sub field to tracking
This will allow to merge analytics from posthog and langfuse.
2025-10-21 13:12:59 +02:00
qbey 12a9488f30 🌐(i18n) update translated strings
Update translated files with new translations
2025-10-20 23:45:35 +02:00
Quentin BEY 15a84132b0 (activation-codes) register users also on Brevo
This provides a way to store the users on Brevo directly
for later product opening mail campaign.
2025-10-20 23:38:57 +02:00
elvoisin 8a52ae8608 🐛(front) code activation fix session end (#101)
add check auth in activation page
2025-10-20 18:45:56 +02:00
Quentin BEY fcd0e24a16 🔖(major) bump release to 0.0.1
This is the first release of the projet, yay

- 🎨(front) activation page footer
- 👷(front) change size small modal
- 🎨(front) retour ui global
- 👷(front) fix button scrollDown
- 💥(front) disable input when error occurred
- 👷(front) fix scroll
- 🐛(front) fix left panel status + fix scroll
- 🐛(llm) add is_active field and persist chat preference
- (frontend) add LLM selection in chat input #53
- 🎨(front) fix width chat container
- 🎨(front) fix width chat container #55
- 🐛(front) fix button search web on new conversation
- 🎨(front) improvement search input scroll
- (404) fix front 404 page
- (chat) add frontend feature flags #29
- 🎨(front) change list attachment in chat
- 🎨(front) move emplacement for attachment
- 🎨(ui) retour ui sources files
- (ui) fix retour global ui
- 🐛(fix) broken staging css
- 🎨(alpha) adjustment for alpha version
- (ui) delete flex message
- (front) add enabled/disabled conversation analysis
- 🎨(front) amelioration chat ux
- 🎨(front) global layout modification
- (front) global layout UI
- ♻️(chat) rewrite backend using Pydantic AI SDK #4
- 🗃️(chat) enforce messages stored JSON format #6
- 🐛(chat) UI messages must have a unique identifier #6
- (llm) allow configuration from JSON file #22
- 💥(agent) replace routing w/ tool calls #40
- 🧱(storage) upload the user documents into S3 #86

- 🎉(conversations) bootstrap backend & frontend #1
- (web-search) add RAG capability to do web search #7
- (chat) add document RAG on document uploaded by user #8
- (backend) allow use to stop conversation streaming #14
- 🐛(agent) add the current date in the system prompt #18
- (backend) add feature flags from posthog #13
- (user) allow to use conversation data for analytics #23
- (chat) enforce response in user language #24
- 📈(langfuse) add light instrumentation #26
- 🚑️(agent) allow Mistral w/ vLLM & tools #36
- (web-search) add Brave search tool #47
- (models) add mistral support & customization #51
- 🐛(web-search) add summarization to Brave results #58
- (langfuse) allow user to score messages from LLM #6
- (onboarding) add activation code logic for launch #62
- 💄(chat) add code highlighting for LLM responses #67
2025-10-20 00:00:21 +02:00
Quentin BEY 620575140e 🌐(project) add missing translation files
In Crowdin we added ru, uk and nl, which breaks the download
action.
2025-10-19 23:59:51 +02:00
Quentin BEY 1d55f51e15 🧱(storage) bump helm chart to v0.0.4
The new chart includes the "media" related service and ingress.
Even if they are not used yet...
2025-10-19 23:06:28 +02:00
Quentin BEY d91cb498e0 🩹(storage) add missing "media_auth" implementation
This provides a way for users to get access to the document
attached to a conversation. But for now the frontend does not
provide any link to this.
2025-10-19 22:58:55 +02:00
Quentin BEY 7cb21e694e 📈(langfuse) add user FQDN to trace metadata
This will allow use to improve according to user origin.
2025-10-19 17:42:22 +02:00
Quentin BEY 661f678363 💩(chat-input) manage S3 attachment upload
This is quite bad because it would be nice to start upload
before sending the first message.
Also, it would be nice to start parsing earlier (more work...)
And it would be nice to display progression to the user.
2025-10-19 17:24:31 +02:00
Quentin BEY f3781e0dde 🧱(storage) upload the user documents into S3
The user uploaded content (images/documents) are now directly
uploaded to S3, and a pre-signed URL is given to the LLM.

The code is a bit complicated because there are behaviors
from docs, other from drive, and in the end some parts are
not used.

This will need some cleanup.

Improvements:
 - Allow to upload PDF without vectorized storage (to let
   LLM manage them on the fly)
 - Allow to configure whether a model can support images/PDF
   or not
2025-10-19 16:34:45 +02:00
Quentin BEY c510adf832 🧱(storage) enable S3 media storage again
Simply restore the commits regarding the S3 management.
2025-10-18 21:42:29 +02:00
elvoisin 6641457cdd 🎨(front) activation page footer (#84)
fix footer in activation page + fix user message
2025-10-17 17:30:37 +02:00
github-actions[bot] b15aa901f8 🌐(i18n) update translated strings (#85)
Update translated files with new translations

Co-authored-by: elvoisin <95469923+elvoisin@users.noreply.github.com>
2025-10-17 16:47:42 +02:00
elvoisin 161cdfe17c 👷(front) change size small modal (#83)
change size modal + fix scroll first message
2025-10-17 09:45:59 +02:00
Quentin BEY c9b724f8da (activation-codes) allow code customization
The activation codes can be fully customized on creation and
not edited afterward.
2025-10-16 21:21:35 +02:00
Quentin BEY ef9bdf9b95 🐛(chat) fix the file format restriction for upload
Restore the `accept` attribute.
2025-10-16 20:45:02 +02:00
elvoisin daa623d2df 🎨(front) retour ui global (#80)
* 👷(front) fix scroll

fix scroll + fix background model selector

* 🎨(front) retour ui global

retour ui robin : scroll, searchbar, margin
2025-10-16 18:18:30 +02:00
Quentin BEY 14739ba2e6 🐛(llm) use Tex markers for formula rendering
The previous implementation was expecting dollar signs for
formulas, but mistral seems trained on \(...\) and \[...\]
format.

Let's hope it does not break anything.
2025-10-16 10:53:37 +02:00
elvoisin d242328c6d 👷(front) fix button scrollDown (#77)
* 👷(front) fix scroll

fix scroll + fix background model selector

* 👷(front) fix button scrollDown

fix sensitive button scrollDown
2025-10-15 10:48:46 +02:00
elvoisin aeba1ba869 💥(front) disable input when error occured (#76)
* 👷(front) fix scroll

fix scroll + fix background model selector

* 💥(front) disable input when error occured

add text error + disable input chat
2025-10-15 09:38:03 +02:00
elvoisin 81c1dcbf81 👷(front) fix scroll (#72)
fix scroll + fix background model selector
2025-10-15 09:21:38 +02:00
Quentin BEY 8dba8b08e3 🚑️(mistral) attempt to fix Unmarshaller error
Mistral model sometime returns
`{'type': 'reference', 'reference_ids': ['cuisineaz']}` which
does not match the Mistral SDK format for `ReferenceChunk`:
`reference_ids: List[int]``...

The Mistral documentation provides a way to return citation
(even if we don't want to use this systemp), so we expect
the model to behave properly with this change.
2025-10-14 22:25:18 +02:00
qbey fb92ac9f2b 🌐(i18n) update translated strings
Update translated files with new translations
2025-10-14 21:32:23 +02:00
Quentin BEY 859b2f312d ⚰️(RAG) remove old RAG_WEB_SEARCH_BACKEND setting
This settings has been removed when we switched to tool calls.
2025-10-14 21:18:17 +02:00
Quentin BEY 4b70641ac6 (pydantic) fix tests after dependencies update
`pydantic` and `pydantic-ai` has been upgraded, fix tests
accordingly
2025-10-14 16:49:46 +02:00
Quentin BEY c460f80cf8 📌(pylint) htmldate does not support lxml >=6 yet
https://github.com/adbar/htmldate/issues/185
2025-10-14 16:33:11 +02:00
renovate[bot] c4af7dcc0b ⬆️(dependencies) update python dependencies 2025-10-14 16:31:13 +02:00
Quentin BEY d12b959812 📌(pylint) pylint-django does not support pylint >=4 yet
https://github.com/pylint-dev/pylint-django/issues/467
2025-10-14 15:40:20 +02:00
Quentin BEY f014de69af 🚑️(mistral) fix the monkeypatch which ignores reference
The previous patch was a bit optimistic and happend to break
returned format.
The new patch will require update and monitor...
2025-10-14 15:27:15 +02:00
Quentin BEY 4b9a251219 💄(chat) add code highlighting for LLM responses
This adds a plugin to nicely display the code in the conversation.
The plugin is rehype-pretty-code, with the `github-dark-dimmed`
theme.

We may select another theme, and later also add light and dark
themes:
 - https://shiki.style/themes#themes

For later we may add instruction for the LLM to manage
transformers: https://shiki.style/packages/transformers

We also considered using https://sugar-high.vercel.app/ which
is quite light and powerful, we might switch later if pretty-code
is too heavy...

In both cases, the themes can be fully overriden if we want to
match the application colors ;)
2025-10-14 10:57:04 +02:00
elvoisin 146c9ff19f 🐛(front) fix left panel status + fix scroll (#65)
fix scroll + fix left panel status + fix show sources
2025-10-14 10:05:28 +02:00
Eléonore Voisin 7c6e3da2d9 (onboarding) add activation code front
add activation code form
2025-10-13 17:09:43 +02:00
Quentin BEY 605f730b16 💄(frontend) add account activation page
This displays an activation page when the setting
`ACTIVATION_REQUIRED` is enabled.
2025-10-13 17:09:43 +02:00
Quentin BEY 04cf9cbc34 (onboarding) allow users to register to beta opening
This simply store the user and their email to notify them
later about a wider opening of the beta version.
2025-10-13 17:09:43 +02:00
Quentin BEY 5d30f275cf (onboarding) add activation code logic for launch
This is a very specific feature, which might be only used for the
product launch.
It allows to restrict user who have access to the product by
asking them to enter an activation code.
2025-10-13 17:09:42 +02:00
BEY Quentin ba62bd82d5 (langfuse) allow user to score messages from LLM (#61)
* (langfuse) allow user to score messages from LLM

We store the trace ID in the message ID... this is not very clean
but it is the easiest way to consistently save the trace ID.

This adds the endpoint for the "thumb up"/"thumb down" calls.

* 💄(frontend) add the thumb up/down to score LLM messages

This enables the score button when a trace exists (ie the
user allows analytics and Langfuse is setup).

* (langfuse) allow user to score messages from LLM

allow user to score messages from LLM

---------

Co-authored-by: Eléonore Voisin <elevoisin@gmail.com>
Co-authored-by: elvoisin <95469923+elvoisin@users.noreply.github.com>
2025-10-10 11:27:22 +02:00
elvoisin 5ed92c286e 🐛(llm) add is_active field and persist chat preference (#63)
add is_active field for LLM models & add user preferences store
2025-10-10 10:40:49 +02:00
BEY Quentin feba5509d2 (frontend) add LLM selection in chat input (#53)
* (frontend) add LLM selection in chat input

This allows user to select which model to use, we currently plan
to use it only in staging.

* (frontend) add LLM selection in chat input

add LLM selection in chat input

---------

Co-authored-by: Eléonore Voisin <elevoisin@gmail.com>
Co-authored-by: elvoisin <95469923+elvoisin@users.noreply.github.com>
2025-10-08 17:16:22 +02:00
Quentin BEY 747ca8499d 🐛(web-search) support web search using document storage
This allows to not call an LLM for summary but instead use
a vectorized search to return only useful snippets.
2025-10-06 17:47:09 +02:00
Quentin BEY f2deeedbc4 🐛(document) fix collection context manager
Fix the URL build and the context manager (needs a test)
2025-10-06 17:34:59 +02:00
Quentin BEY 1cdccd3d24 🐛(web-search) add summarization to Brave results
Loaded pages can be too long for the context, so a naive
implementation is to summarize them.

It would be better to store results in a Vector space, but
also requires a call to a vectorization model.
2025-10-06 17:18:04 +02:00
Quentin BEY 79b5be6004 ♻️(rag) rewrite the document RAG search code
This will ease the use of other document storage for semantic
search.
2025-10-06 15:54:45 +02:00
elvoisin a6a5dec0ac 🎨(front) fix width chat container (#57)
refix width container
2025-10-04 14:36:36 +02:00
elvoisin 592b67ca4c 🌐(i18n) update translated strings
Update translated files with new translations
2025-10-03 15:45:36 +02:00
Quentin BEY 45c346e1e7 🚑️(mistral) fix wrongly commented line...
This monkeypatch is dirty, we may manage the assertion error at
the streaming level... Anyway, this lack a test.
2025-10-03 15:44:19 +02:00
github-actions[bot] fe7f7b2730 🌐(i18n) update translated strings (#54)
* 🌐(i18n) update translated strings

Update translated files with new translations

* 👷(i18n) add noChangeLog label to opened PR

The translation update commit does not need a changelog entry.

---------

Co-authored-by: qbey <5311687+qbey@users.noreply.github.com>
Co-authored-by: Quentin BEY <quentin.bey@mail.numerique.gouv.fr>
2025-10-03 15:04:17 +02:00
elvoisin aabb2a0991 🎨(front) fix width chat container (#55)
fix quicksearch + fix width container
2025-10-03 14:45:18 +02:00
Quentin BEY 48fb064594 ⬆️(django) bump version to 5.2.7
Trivy scan is complaining this fixes CVE-2025-59681.
2025-10-02 22:59:39 +02:00
Quentin BEY bc65d646e5 🎨(pydantic-ai) use the proper profile setting
The strict attribute for "tools" can now be removed with a specific
parameter for the model profile. Let's use it.
2025-10-02 22:59:39 +02:00
Quentin BEY 99c4b6ba29 (models) add mistral support & customization
This allows to connect to a Mistral model, on a Mistral
platform, self-hosted.
2025-10-02 16:52:38 +02:00
elvoisin c9c10ce19a 🐛(front) fix button search web on new conversation (#50)
* 🎨(front) improvement search input scroll

improvement search input scroll and margin

* 🐛(front) fix button search web on new conversation

fix button search web on new conversation'
2025-10-02 16:43:08 +02:00
elvoisin 1b9e814be7 🎨(front) improvement search input scroll (#49)
improvement search input scroll and margin
2025-10-02 11:50:10 +02:00
Quentin BEY 9644b8ccd3 🧑‍💻(admin) allow to update user consent
This might be removed later at the end of the alpha, because
we are not supposed to update this without the user knowing it.
2025-10-02 10:04:09 +02:00
Quentin BEY ac65ffe7f7 🐛(admin) do not load all users in conversation edition
This prevents the load of all users when editing a conversation.
2025-10-02 10:02:52 +02:00
Quentin BEY bb6ced4728 (web-search) add Brave search tool
This allows to use Brave as a web search engine in a tool.
2025-10-01 16:46:42 +02:00
Quentin BEY 02ac8af714 ⚗️(users) allow to enable analytics by default
This will be helpful to have analytics during the alpha
deployment.
2025-09-30 16:46:03 +02:00
Quentin BEY 8c706fd2d8 ⚗️(agent) improve force web search
Forcing a specific tool call is currently a bit complicated,
so we choose a more naive way.
2025-09-30 16:36:09 +02:00
Quentin BEY c9efa978e1 💥(agent) replace routing w/ tool calls
The first implementation was made because our models had not
tool calling capabilities, and the routing and context management
were quite bad.

Now we can use tools (yay) the whole code is switched to tool
calling.

This first implementation of summarize agent is quite bad, this
will be improved soon.
2025-09-30 15:28:56 +02:00
elvoisin d361d0177c (404) fix front 404 page (#44)
remove margin top on the header
2025-09-30 10:35:32 +02:00
BEY Quentin 934098a1c2 💩(chat) add frontend feature flags (#29)
* 💩(chat) add frontend feature flags

This introduce the use of feature flags:
 - flags can be globally enabled or disabled from the backend
 - when dynamic, it queries Posthog to get the value

* (chat) add frontend feature flags

disable buttons for feature flags

---------

Co-authored-by: Eléonore Voisin <elevoisin@gmail.com>
2025-09-29 10:42:10 +02:00
Quentin BEY 5c4b1ff388 📌(frontend) stay with Vercel SDK <5
While the frontend and the backend are not ready we
need to stay in the v4.x version
2025-09-26 14:18:45 +02:00
renovate[bot] 689f2ce672 ⬆️(dependencies) update python dependencies 2025-09-24 23:01:06 +02:00
elvoisin 2515e14942 🎨(front) move emplacement for attachment (#39)
move emplacement for attachement in chat
2025-09-24 16:23:49 +02:00
elvoisin 273332c370 🎨(front) move emplacement for attachment (#38)
move emplacement for attachement in chat
2025-09-23 16:25:15 +02:00
elvoisin 623a951ca2 🎨(ui) retour ui sources files (#37)
* (ui) fix retour global ui

correction retour ui

* 🎨(ui) retour ui sources files

fix retour ui source files

* 🐛(fix) fix front build

prettier - fix build front
2025-09-23 13:29:52 +02:00
Quentin BEY 9e4f0d3310 ♻️(agents) rewrite the awy to setup agents
Not sure this rewrite was mandatory, but it paves the way to a
rewrite of the "client" module.
2025-09-23 00:08:36 +02:00
Quentin BEY a560ea9e78 🔒️(frontend) update alpine packages in production image
Force an update of installed package in the image used for
the frontend in production.
2025-09-22 22:47:25 +02:00
Quentin BEY e04a050486 (agent) add fake streaming when the model does not support it
To have a better UX, we simulate a streamed return. The default
value is quite fast to not block the worker too long...
2025-09-22 22:47:25 +02:00
Quentin BEY 7e460b55a3 🚑️(agent) allow Mistral w/ vLLM & tools
To be able to call our Mistral model, few fixes are required:
- "strict" is nor allowed for tools, vLLM will refuse the schema
  validation
- response streaming w/o tools works, but not w/ tools...
  As we expect to switch from old-school "intent detection" to
  automatic tool use, we need to be able to disable streaming.

What we should do next:
- tidy code: the iteration loginc might be in the Agent class
- fake result streaming for end-user. I don't like that, but UX
  might prefer.
2025-09-22 22:47:25 +02:00
Quentin BEY dce69eb31c (llm) allow profile configuration for model
This allows to use a specific profile for some models.
This is useful in the case of a mistral model behind vLLM.

There is room for improvement to use the real default
and not be forced to pass OpenAI default "supports"...

```
"profile": {
  "json_schema_transformer":
    "chat.agents.json_schema_converters.MistralVllmJsonSchemaTransformer",
  "supports_json_schema_output": true,
  "supports_json_object_output": true
}
```
2025-09-22 22:47:21 +02:00
Quentin BEY 297af78936 ⚰️(vercel-sdk) remove code wrongly pushed
This was not supposed to be here.
2025-09-22 22:46:57 +02:00
Quentin BEY 7507f1a8db ⬆️(pydantic-ai) bump to the latest version
This might be helpful to include the latest fixes.
2025-09-22 22:46:50 +02:00
Quentin BEY 05dceec3e7 🐛(vercel-sdk) allow tool calls to have no arguments
Some tools might not need arguments because they rely only
on the run context, we can allow that.
2025-09-22 22:39:19 +02:00
Quentin BEY d486661f32 🐛(markitdown) fix converter data and detection
The data passed to the converter where not making it possible
to detect even plain text documents...
2025-09-22 22:39:19 +02:00
Quentin BEY 7bc38c7435 ♻️(chat) move agent to a dedicated module
This is a first step to tidy code and ease agent definition
readability.
2025-09-22 22:39:19 +02:00
elvoisin b4df44311a (ui) fix retour global ui (#35)
correction retour ui
2025-09-22 09:20:07 +02:00
elvoisin 9878498a34 🐛(fix) broken staging css (#34)
force fix staging css
2025-09-15 17:43:07 +02:00
elvoisin 517bc1a9d1 🐛(fix) broken staging css (#33)
force fix staging  css
2025-09-15 16:47:08 +02:00
Quentin BEY 81fd0d3400 🗃️(chat) fix conversation messages source type
This add the migration to update data after the fix introduced in
d579d9abd0
2025-09-15 15:03:51 +02:00
elvoisin 10f3ebf81f 🎨(front) add customisation for alpha version (#31)
change logo + home + add form to send feedback - wip
2025-09-15 14:50:38 +02:00
Quentin BEY d579d9abd0 🚑️(chat) source UI messages serializer typo
There was a typo in the source UI Message model used in chat
input serialization...
2025-09-08 22:35:33 +02:00
elvoisin 5bdd2f92f0 (ui) delete flex message (#28)
remove display flex on content chat
2025-09-08 10:40:31 +02:00
Quentin BEY 633bcf094d 🔒️(all) refactor Docker Hub login to use official GitHub actions
Replace custom Docker Hub authentication with standard, secure,
official GitHub actions for improved security and maintainability.

Uses officially supported actions that follow security best practices
and receive regular updates from GitHub.

Avoid unsecure handling of GitHub secrets.

Thanks to @lebaudantoine
2025-09-05 14:52:50 +02:00
Quentin BEY c491108f9e 🔧(ci) always run all git-lint steps
git-lint steps are independant and we would like to have all checks at
once. Using the `if: always()` instruction should ensure all steps
should be run event if the previous fails.

thank @lunika
2025-09-05 14:42:46 +02:00
Quentin BEY d582f85f59 ♻️(backend) prepare vercel AI SDK v4 -> v5
This introduce a new way to stream events and should ease
migration from v4 to V5 of the Vercel SDK.

WARNING: the v5 implementation is not ready yet, but we need
to merge this before making too much change to the codebase.
2025-09-05 13:55:56 +02:00
elvoisin 926ea2975f (front) add enabled/disabled conversation analysis (#27)
add enabled/disabled conversation analysis in settings modal
2025-09-05 12:45:22 +02:00
Quentin BEY ace27ab7fb 📈(langfuse) add light instrumentation
This is a first implementation of Langfuse to see user interactions.
Next step will be to allow prompt improvements.
2025-09-05 12:06:43 +02:00
Quentin BEY 9bbc3f160f 🔥(llm) remove mlflow intrumentation & deps
We don't use this instrumentation for now and plan to use something
else.
2025-09-05 12:06:43 +02:00
elvoisin 90d40eba6a 🎨(front) amelioration chat ux (#25)
fix appartion message + fix placeholder input
2025-09-05 12:03:26 +02:00
Quentin BEY 8b8b7bd7dd (chat) enforce response in user language
Ask the LLM to use the user's language to answer.
2025-09-04 13:24:02 +02:00
Quentin BEY b76760025f 🚑️(helm) fix llm_configuration file content
There is a typo in the helm chart...
2025-09-04 10:09:57 +02:00
Quentin BEY a267b0282b (user) allow to use conversation data for analytics
The frontend can store whether the user allows us to use
the conversation data to make analytics and help chat
improvement.
2025-09-03 23:27:57 +02:00
Quentin BEY c5b9576835 (llm) allow completion endpoint to select model
This will allow the frontend to select the model to use from
the configuration.
2025-09-03 22:50:45 +02:00
Quentin BEY 6e5220866a (llm) add models listing endpoint
This will allow the frontend to display several models available
for the user.
2025-09-03 13:32:56 +02:00
Quentin BEY 548679698f 🧱(helm) mount the LLM JSON configuration file
This allows to customize the LLM configuration for deployments.
2025-09-03 12:01:10 +02:00
Quentin BEY 5e2d77642d (llm) allow configuration from JSON file
This is the first step to allow several models to be available to
the users.

This is not a very good implementation, as it aims to keep the
ability to use django settings for configuration:

 - makes it easy to deploy only one model
 - allow changes like before in tests

Next step would be to remove the django settings and update
all tests etc.
2025-09-02 12:17:52 +02:00
elvoisin df3934367a 🎨(front) global layout modification (#21)
delete unused languages + modify scroll down + change source item + attachements
2025-09-02 11:39:26 +02:00
Quentin BEY 22769af0e5 🐛(posthog) add missing configuration
This should fix the issue of "$host" being unknown when checking
for feature flag.
2025-09-01 14:11:41 +02:00
Quentin BEY 59af95caf7 ⚰️(minio) remove code related to the S3 storage
This is related to 697b040a40
2025-08-27 23:55:26 +02:00
Quentin BEY b6fe71adb0 (backend) add feature flags from posthog
This adds posthog backend to know whether a feature is enabled
for a user.
More globally, if Posthog is not present feature flags can be
enabled/disabled globally.
2025-08-27 23:29:32 +02:00
Quentin BEY fd06d33d05 🐛(llm) disabling thinking configuration
While not fully managed, disable this configuration.
2025-08-27 17:22:52 +02:00
Quentin BEY e921aa9d94 🐛(prompt) slight change of routing prompt
Attempt to reduce when the routing LLM tries to make web search.
2025-08-27 17:22:52 +02:00
Quentin BEY f7e3ed19ba 🐛(agent) add the current date in the system prompt
We add the current date to the system prompt.
Warning: the timezone is not the user's one...
2025-08-27 17:22:52 +02:00
elvoisin 4e078d8bb1 🎨(front) global layout (#5)
* 🎨(front) global layout

modify left panel, header and chat

* 🎨(front) color dsfr v2

change color + left panel logic
2025-08-27 13:56:57 +02:00
Quentin BEY 2486db8287 ⚗️(github) run action on GitHub servers
This reverts commit 42ce2affe270f110f3312d3a3464de42638085d7.
2025-08-26 17:06:54 +02:00
Quentin BEY cc77378d39 ️(ci) use setup-python cache option
The setup-python action is able to cache the dependencies and reuse this
cache while the pyproject file has not changed. It is easy to setup,
the package manager used has to be declared in the cache settings.

Credits to lunika who made the change on
https://github.com/suitenumerique/drive
2025-08-26 17:06:54 +02:00
Quentin BEY fbed35f4c4 📝(helm) run readme-generator-for-helm
Re-generate the helm readme after modifications.
2025-08-26 17:06:44 +02:00
Quentin BEY 697b040a40 ⚰️(storage) remove S3 media storage
Unused for now and not clear if we will need it later.
2025-08-26 17:00:25 +02:00
Quentin BEY 86bcebadf8 📝(readme) small improvements
- remove the old handmade logo
- fix the dependencies
- add small introduction
2025-08-26 17:00:25 +02:00
Quentin BEY 0f45d2e896 (backend) fix test with user count
For some reason the test counting users are starting to fail.
I don't know if it's because of transactional test cases, but
a fix is to drop all users before running the test...

This would need further investigation.
2025-08-09 00:05:33 +02:00
Quentin BEY 475afcb454 📌(openai) pin version to 1.99.1
This needs to be unpinned when the openai is fixed.

=> https://github.com/openai/openai-python/issues/2525
=> https://github.com/pydantic/pydantic-ai/issues/2476
2025-08-08 21:43:36 +02:00
Quentin BEY 095735fb34 (backend) allow use to stop conversation streaming
This introduces a new endpoint to allow user to stop the message
stream.

We currently use uWSGI workers, which will not automatically stop the streaming
when the request is cancelled by the client. Therefore, we need to
explicitly stop the streaming by calling the /stop-steaming endpoint.
When (if) we switch to Gunicorn with Uvicorn workers, this will not be necessary
as the Uvicorn workers will automatically stop the streaming when the request
is cancelled. BUT, we will then need to handle the streaming cancellation when
the user is simply offline and still waiting for a response and the conversation
to be updated with the result. So this endpoint will still be useful to be able
to detect the cancellation of the streaming versus the user being offline.
2025-08-08 18:48:29 +02:00
Quentin BEY bd798816a2 🗃️(chat) rename message field and reset migrations
While the project is not deployed, I reset the migration history
and use a better name for the history messages.
2025-08-06 21:49:13 +02:00
Quentin BEY aafaaeed3e ⬇️(markitdown) enforce old 0.0.2 version
Our base docker image (alpine) does not support magika dependency
because there is no wheel for onnxruntime for alpine...
It's complex to build and a bit hackish.

Switching to a slim (debian) base image, is a solution but trivy
scan detects CVEs. We need to check the CVEs don't regard our stack
but it's bit of work also.

For the time being, I pin the markitdown version to an old one...
This is sad, since a lot of bugs has been fixed.
2025-08-06 15:31:11 +02:00
Quentin BEY a9883a1a31 💄(chat) restrict uploadable file format for RAG
This takes the allowed formats from the /config endpoint.
2025-08-06 11:49:25 +02:00
Quentin BEY 768aeec91f 🐛(config) restrict uploadable file types for RAG
This allows to let the frontend know whi file types can be
uploaded to do RAG.
By default it's the format accepted by markitdown + PDF.
2025-08-06 11:49:25 +02:00
Quentin BEY 88760022e0 💩(frontend) display a modal on specific completion error
We use the error mechanism to allow specific message display to user.
2025-08-06 11:49:25 +02:00
Quentin BEY 8cc6a78089 💩(frontend) allow chat to display document parsing process
We use the tool invocation system to display a specific message
to the use, to let them know we are parsing the attachments.
2025-08-06 11:49:25 +02:00
Quentin BEY 04f15a518a 💩(frontend) allow source to display a document
We simply check is the URL is ... an URL, if not it's a document.
2025-08-06 11:49:25 +02:00
Quentin BEY 2ef70d15ea (chat) add document RAG on document uploaded by user
This allow use to upload documents to ask question on them.
The summarize feature is willingly disabled for now, as it is not
the same process.
2025-08-06 11:49:25 +02:00
Quentin BEY 8358ca9490 🐛(rag-websearch) fix the Albert API search query
This was not properly tested and the code changed just before
last commit... boom.
2025-07-31 23:46:33 +02:00
Quentin BEY f1c8a09378 ✏️(docs) remove remaining references to docs
There was still outdate text from docs... Some are still remaining
but will be removed lated.
2025-07-31 15:15:50 +02:00
Quentin BEY 8c917b90c2 🔨(tilt) fix local k8s deployment
Removes the docs references and fix the helmfile extension to work
with newest version.
2025-07-31 15:15:50 +02:00
Quentin BEY 44b50d5085 👷(actions) increase front build timeout
While on self-hosted runners, the build can be very long...
2025-07-31 15:06:44 +02:00
Quentin BEY 30ca4a4066 (codespell) add configuration file and exceptions
This configuration file allow to easily run codespell locally.
The RAG mock result file is ignored because it contains french
text.
2025-07-31 11:52:51 +02:00
Quentin BEY 43e012d185 (rag) add test for conversation with websearch RAG
This add integration tests for chat when using RAG web search.
2025-07-31 11:52:51 +02:00
Quentin BEY 4b89c826c1 ️(pydantic) use stored history in agent requests
We were converting the whole message history from the frontend
on each call. We replace that by direct use of the history stored
in the proper format from the database.

This important to do so because the raw Pydantic messages may contain
more information than the ones displayed to the user (like when
doing RAG web search).
2025-07-31 11:52:51 +02:00
Quentin BEY e9b512400f 💩(frontend) add source display in chat
This is a bad implementation of source display, it only demonstrate
the use of the message source sent by the backend.

Needless to say the UI is really bad on this.

We could improve this by fetching icon and page title in the backend
and send it directly in the `providerMetadata` of the source.
See: LanguageModelV1Source
2025-07-31 11:52:51 +02:00
Quentin BEY f9d6934205 (web-search) add RAG capability to do web search
This introduces a first implementation of RAG web search using
the Albert API endpoint which retrieves chunks based on the user
prompt.

This also add a "routing" LLM call to detect whether the user wants
fresh information.
2025-07-31 09:57:43 +02:00
Quentin BEY a5ef813840 🐛(pyproject) fix the pydantic-ai dependency
The requirement was not in the proper place... Since I'm moving it
I also update the version :)
2025-07-30 15:10:25 +02:00
Quentin BEY 7d69eb2130 🐛(chat) UI messages must have a unique identifier
The new Pydantic implementation was not setting an ID for its message
storage, which make the frontend to not update properly the chat history
when selecting another conversation.
2025-07-28 16:45:25 +02:00
Quentin BEY f2707bb830 🗃️(chat) enforce messages stored JSON format
This enforces the models JSON format to `UIMessage`. It would be
nicer to decouple the Vercel format from the stored one, but it's also
more convenient for now, and will be quite easy to update later if
needed.
2025-07-28 16:45:25 +02:00
Quentin BEY b914214818 (e2e) fix tests after translations update
The english translation requires at least one translated string to
be considered as a supported language.

Remove tested languages which are not supported yet.
2025-07-28 10:58:18 +02:00
Quentin BEY e04f56059d ⚰️(chat) remove OpenAI Agent client
Since we now uses the Pydantic AI SDK we won't maintain this.
2025-07-25 17:46:42 +02:00
Quentin BEY c5af3cdcb1 ♻️(chat) rewrite backend using Pydantic AI SDK
We chose to use Pydantic AI instead of the OpenAI SDK.
This commit should not change the chat behavior.
2025-07-25 17:46:42 +02:00
qbey 0526c9bbef 🌐(i18n) update translated strings
Update translated files with new translations
2025-07-22 09:48:19 +02:00
Quentin BEY d23434aafe 💚(crowdin) fix download action's job
The self-hosted runner requires explicit yarn install.
2025-07-22 09:39:50 +02:00
Quentin BEY 8ec172583a 💚(crowdin) fix upload action's job
The self-hosted runner requires explicit yarn install.
2025-07-21 22:21:11 +02:00
Quentin BEY 53cfc89b9b ⚗️(github) run action on self-hosted runner
This commit will be removed whent the project goes public.
2025-07-21 18:10:37 +02:00
Quentin BEY fe7995a118 🎉(conversations) bootstrap backend & frontend
This is the first commit which provides all the first stack for a
working chat.

This is a first implementation with:
 - Vercel SDK for the frontend part
 - OpenAI Agent SDK for the backend

The stack can use a local LLM with docker ot a remote one.

This implementation is more a draft, but it provides the project
structure.

All tests are working even if we lack a lot of them.
2025-07-21 18:10:15 +02:00
598 changed files with 74396 additions and 17 deletions
+16
View File
@@ -0,0 +1,16 @@
[codespell]
skip =
./git/,
**/*.po,
**/*.pot,
**/*.json,
**/yarn.lock,
**/node_modules/**,
**/e2e/report/**,
*.tsbuildinfo,
check-filenames = true
ignore-words-list =
afterAll,
statics,
exclude-file =
./src/backend/chat/agent_rag/web_search/mocked.py,
+37
View File
@@ -0,0 +1,37 @@
# Python
__pycache__
*.pyc
**/__pycache__
**/*.pyc
venv
.venv
# System-specific files
.DS_Store
**/.DS_Store
# Docker
docker compose.*
env.d
# Docs
docs
*.md
*.log
# Development/test cache & configurations
data
.cache
.circleci
.git
.vscode
.iml
.idea
db.sqlite3
.mypy_cache
.pylint.d
.pytest_cache
# Frontend
node_modules
.next
+14
View File
@@ -0,0 +1,14 @@
DJANGO_CONFIGURATION=Test
DJANGO_SETTINGS_MODULE=conversations.settings
DJANGO_SECRET_KEY=ThisIsAnExampleKeyForTestPurposeOnly
OIDC_OP_JWKS_ENDPOINT=/endpoint-for-test-purpose-only
DB_HOST=localhost
DB_NAME=conversations
DB_USER=dinum
DB_PASSWORD=pass
DB_PORT=15432
STORAGES_STATICFILES_BACKEND=django.contrib.staticfiles.storage.StaticFilesStorage
AWS_S3_ENDPOINT_URL=http://localhost:9000
AWS_S3_ACCESS_KEY_ID=conversations
AWS_S3_SECRET_ACCESS_KEY=password
+23
View File
@@ -0,0 +1,23 @@
# Set the default behavior for all files
* text=auto eol=lf
# Binary files (should not be modified)
*.png binary
*.jpg binary
*.jpeg binary
*.gif binary
*.ico binary
*.mov binary
*.mp4 binary
*.mp3 binary
*.flv binary
*.fla binary
*.swf binary
*.gz binary
*.zip binary
*.7z binary
*.ttf binary
*.woff binary
*.woff2 binary
*.eot binary
*.pdf binary
+6
View File
@@ -0,0 +1,6 @@
<!---
Thanks for filing an issue 😄 ! Before you submit, please read the following:
Check the other issue templates if you are trying to submit a bug report, feature request, or question
Search open/closed issues before submitting since someone might have asked the same thing before!
-->
+28
View File
@@ -0,0 +1,28 @@
---
name: 🐛 Bug Report
about: If something is not working as expected 🤔.
labels: ["bug", "triage"]
---
## Bug Report
**Problematic behavior**
A clear and concise description of the behavior.
**Expected behavior/code**
A clear and concise description of what you expected to happen (or code).
**Steps to Reproduce**
1. Do this...
2. Then this...
3. And then the bug happens!
**Environment**
- Conversations version:
- Instance url:
**Possible Solution**
<!--- Only if you have suggestions on a fix for the bug -->
**Additional context/Screenshots**
Add any other context about the problem here. If applicable, add screenshots to help explain.
+23
View File
@@ -0,0 +1,23 @@
---
name: ✨ Feature Request
about: I have a suggestion (and may want to build it 💪)!
labels: ["feature", "triage"]
---
## Feature Request
**Is your feature request related to a problem or unsupported use case? Please describe.**
A clear and concise description of what the problem is. For example: I need to do some task and I have an issue...
**Describe the solution you'd like**
A clear and concise description of what you want to happen. Add any considered drawbacks.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Discovery, Documentation, Adoption, Migration Strategy**
If you can, explain how users will be able to use this and possibly write out some documentation (if applicable).
Maybe add a screenshot or design?
**Do you want to work on it through a Pull Request?**
<!-- Make sure to coordinate with us before you spend too much time working on an implementation! -->
@@ -0,0 +1,18 @@
---
name: 🤗 Support Question
about: If you have a question 💬, or something was not clear from the docs!
labels: ["support", "triage"]
---
## Support request
**Checks before filing**
Please make sure you have read our [main Readme](https://github.com/suitenumerique/conversations).
Also make sure it was not already answered in [an open or close issue](https://github.com/suitenumerique/conversations/issues?q=is%3Aissue%20state%3Aopen%20label%3Asupport).
If your question was not covered, and you feel like it should be, fire away! We'd love to improve our docs! 👌
**Topic**
What's the general area of your question: for example, docker setup, database schema, search functionality,...
**Question**
Try to be as specific as possible so we can help you as best we can. Please be patient 🙏
+22
View File
@@ -0,0 +1,22 @@
## Purpose
Describe the purpose of this pull request.
## Proposal
- [ ] item 1...
- [ ] item 2...
## External contributions
Thank you for your contribution! 🎉
Please ensure the following items are checked before submitting your pull request:
- [ ] I have read and followed the [contributing guidelines](https://github.com/suitenumerique/conversations/blob/main/CONTRIBUTING.md)
- [ ] I have read and agreed to the [Code of Conduct](https://github.com/suitenumerique/conversations/blob/main/CODE_OF_CONDUCT.md)
- [ ] I have signed off my commits with `git commit --signoff` (DCO compliance)
- [ ] I have signed my commits with my SSH or GPG key (`git commit -S`)
- [ ] My commit messages follow the required format: `<gitmoji>(type) title description`
- [ ] I have added a changelog entry under `## [Unreleased]` section (if noticeable change)
- [ ] I have added corresponding tests for new features or bug fixes (if applicable)
@@ -0,0 +1,151 @@
name: Frontend Workflow
on:
push:
branches:
- main
pull_request:
branches:
- "*"
jobs:
install-dependencies:
uses: ./.github/workflows/dependencies.yml
with:
node_version: '22.x'
with-front-dependencies-installation: true
test-front:
needs: install-dependencies
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: "22.x"
- name: Install yarn
run: npm install -g yarn
- name: Restore the frontend cache
uses: actions/cache@v4
with:
path: "src/frontend/**/node_modules"
key: front-node_modules-${{ hashFiles('src/frontend/**/yarn.lock') }}
fail-on-cache-miss: true
- name: Test App
run: cd src/frontend/ && yarn test
lint-front:
runs-on: ubuntu-latest
needs: install-dependencies
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: "22.x"
- name: Install yarn
run: npm install -g yarn
- name: Restore the frontend cache
uses: actions/cache@v4
with:
path: "src/frontend/**/node_modules"
key: front-node_modules-${{ hashFiles('src/frontend/**/yarn.lock') }}
fail-on-cache-miss: true
- name: Check linting
run: cd src/frontend/ && yarn lint
test-e2e-chromium:
runs-on: ubuntu-latest
needs: install-dependencies
timeout-minutes: 40
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: "22.x"
- name: Install yarn
run: npm install -g yarn
- name: Restore the frontend cache
uses: actions/cache@v4
with:
path: "src/frontend/**/node_modules"
key: front-node_modules-${{ hashFiles('src/frontend/**/yarn.lock') }}
fail-on-cache-miss: true
- name: Set e2e env variables
run: cat env.d/development/common.e2e.dist >> env.d/development/common.dist
- name: Install Playwright Browsers
run: cd src/frontend/apps/e2e && yarn install --frozen-lockfile && yarn install-playwright chromium
- name: Start Docker services
run: make bootstrap-e2e FLUSH_ARGS='--no-input'
- name: Run e2e tests
run: cd src/frontend/ && yarn e2e:test --project='chromium'
- uses: actions/upload-artifact@v4
if: always()
with:
name: playwright-chromium-report
path: src/frontend/apps/e2e/report/
retention-days: 7
test-e2e-other-browser:
runs-on: ubuntu-latest
needs: test-e2e-chromium
timeout-minutes: 40
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: "22.x"
- name: Install yarn
run: npm install -g yarn
- name: Restore the frontend cache
uses: actions/cache@v4
with:
path: "src/frontend/**/node_modules"
key: front-node_modules-${{ hashFiles('src/frontend/**/yarn.lock') }}
fail-on-cache-miss: true
- name: Set e2e env variables
run: cat env.d/development/common.e2e.dist >> env.d/development/common.dist
- name: Install Playwright Browsers
run: cd src/frontend/apps/e2e && yarn install --frozen-lockfile && yarn install-playwright firefox webkit chromium
- name: Start Docker services
run: make bootstrap-e2e FLUSH_ARGS='--no-input'
- name: Run e2e tests
run: cd src/frontend/ && yarn e2e:test --project=firefox --project=webkit
- uses: actions/upload-artifact@v4
if: always()
with:
name: playwright-other-report
path: src/frontend/apps/e2e/report/
retention-days: 7
+202
View File
@@ -0,0 +1,202 @@
name: Main Workflow
on:
push:
branches:
- main
pull_request:
branches:
- "*"
jobs:
install-dependencies:
uses: ./.github/workflows/dependencies.yml
with:
with-build_mails: true
lint-git:
runs-on: ubuntu-latest
if: github.event_name == 'pull_request' # Makes sense only for pull requests
steps:
- name: Checkout repository
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: show
run: git log
- name: Enforce absence of print statements in code
if: always()
run: |
! git diff origin/${{ github.event.pull_request.base.ref }}..HEAD -- . ':(exclude)**/conversations.yml' | grep "print("
- name: Check absence of fixup commits
if: always()
run: |
! git log | grep 'fixup!'
- name: Install gitlint
if: always()
run: pip install --user requests gitlint
- name: Lint commit messages added to main
if: always()
run: ~/.local/bin/gitlint --commits origin/${{ github.event.pull_request.base.ref }}..HEAD
check-changelog:
runs-on: ubuntu-latest
if: |
contains(github.event.pull_request.labels.*.name, 'noChangeLog') == false &&
github.event_name == 'pull_request'
steps:
- name: Checkout repository
uses: actions/checkout@v3
with:
fetch-depth: 50
- name: Check that the CHANGELOG has been modified in the current branch
run: git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.event.after }} | grep 'CHANGELOG.md'
lint-changelog:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v2
- name: Check CHANGELOG max line length
run: |
max_line_length=$(cat CHANGELOG.md | grep -Ev "^\[.*\]: https://github.com" | wc -L)
if [ $max_line_length -ge 80 ]; then
echo "ERROR: CHANGELOG has lines longer than 80 characters."
exit 1
fi
lint-spell-mistakes:
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
steps:
- name: Checkout repository
uses: actions/checkout@v2
- name: Install codespell
run: pip install --user codespell
- name: Check for typos
run: codespell
lint-back:
runs-on: ubuntu-latest
defaults:
run:
working-directory: src/backend
steps:
- name: Checkout repository
uses: actions/checkout@v2
- name: Install Python
uses: actions/setup-python@v5
with:
python-version: "3.13.3"
cache: "pip"
- name: Upgrade pip and setuptools
run: pip install --upgrade pip setuptools
- name: Install development dependencies
run: pip install --user .[dev]
- name: Check code formatting with ruff
run: ~/.local/bin/ruff format . --diff
- name: Lint code with ruff
run: ~/.local/bin/ruff check .
- name: Lint code with pylint
run: ~/.local/bin/pylint .
test-back:
runs-on: ubuntu-latest
needs: install-dependencies
defaults:
run:
working-directory: src/backend
services:
postgres:
image: postgres:16
env:
POSTGRES_DB: conversations
POSTGRES_USER: dinum
POSTGRES_PASSWORD: pass
ports:
- 5432:5432
# needed because the postgres container does not provide a healthcheck
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
env:
DJANGO_CONFIGURATION: Test
DJANGO_SETTINGS_MODULE: conversations.settings
DJANGO_SECRET_KEY: ThisIsAnExampleKeyForTestPurposeOnly
OIDC_OP_JWKS_ENDPOINT: /endpoint-for-test-purpose-only
DB_HOST: localhost
DB_NAME: conversations
DB_USER: dinum
DB_PASSWORD: pass
DB_PORT: 5432
STORAGES_STATICFILES_BACKEND: django.contrib.staticfiles.storage.StaticFilesStorage
AWS_S3_ENDPOINT_URL: http://localhost:9000
AWS_S3_ACCESS_KEY_ID: conversations
AWS_S3_SECRET_ACCESS_KEY: password
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Create writable /data
run: |
sudo mkdir -p /data/media && \
sudo mkdir -p /data/static
- name: Restore the mail templates
uses: actions/cache@v4
id: mail-templates
with:
path: "src/backend/core/templates/mail"
key: mail-templates-${{ hashFiles('src/mail/mjml') }}
fail-on-cache-miss: true
- name: Start MinIO
run: |
docker pull minio/minio
docker run -d --name minio \
-p 9000:9000 \
-e "MINIO_ACCESS_KEY=conversations" \
-e "MINIO_SECRET_KEY=password" \
-v /data/media:/data \
minio/minio server --console-address :9001 /data
# Tool to wait for a service to be ready
- name: Install Dockerize
run: |
curl -sSL https://github.com/jwilder/dockerize/releases/download/v0.8.0/dockerize-linux-amd64-v0.8.0.tar.gz | sudo tar -C /usr/local/bin -xzv
- name: Wait for MinIO to be ready
run: |
dockerize -wait tcp://localhost:9000 -timeout 10s
- name: Configure MinIO
run: |
MINIO=$(docker ps | grep minio/minio | sed -E 's/.*\s+([a-zA-Z0-9_-]+)$/\1/')
docker exec ${MINIO} sh -c \
"mc alias set conversations http://localhost:9000 conversations password && \
mc alias ls && \
mc mb conversations/conversations-media-storage && \
mc version enable conversations/conversations-media-storage"
- name: Install Python
uses: actions/setup-python@v5
with:
python-version: "3.13.3"
cache: "pip"
- name: Install development dependencies
run: pip install --user .[dev]
- name: Install gettext (required to compile messages) and MIME support
run: |
sudo apt-get update
sudo apt-get install -y gettext pandoc shared-mime-info
sudo wget https://svn.apache.org/repos/asf/httpd/httpd/trunk/docs/conf/mime.types -O /etc/mime.types
- name: Generate a MO file from strings extracted from the project
run: python manage.py compilemessages
- name: Run tests
run: ~/.local/bin/pytest -n 2
+83
View File
@@ -0,0 +1,83 @@
name: Download translations from Crowdin
on:
workflow_dispatch:
push:
branches:
- 'release/**'
jobs:
install-dependencies:
uses: ./.github/workflows/dependencies.yml
with:
node_version: '22.x'
with-front-dependencies-installation: true
synchronize-with-crowdin:
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Create empty source files
run: |
touch src/backend/locale/django.pot
mkdir -p src/frontend/packages/i18n/locales/conversations/
touch src/frontend/packages/i18n/locales/conversations/translations-crowdin.json
# crowdin workflow
- name: crowdin action
uses: crowdin/github-action@v2
with:
config: crowdin/config.yml
upload_sources: false
upload_translations: false
download_translations: true
create_pull_request: false
push_translations: false
push_sources: false
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# A numeric ID, found at https://crowdin.com/project/<projectName>/tools/api
CROWDIN_PROJECT_ID: ${{ secrets.CROWDIN_PROJECT_ID }}
# Visit https://crowdin.com/settings#api-key to create this token
CROWDIN_PERSONAL_TOKEN: ${{ secrets.CROWDIN_PERSONAL_TOKEN }}
CROWDIN_BASE_PATH: "../src/"
# frontend i18n
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: "22.x"
- name: Install yarn
run: npm install -g yarn
- name: Restore the frontend cache
uses: actions/cache@v4
with:
path: "src/frontend/**/node_modules"
key: front-node_modules-${{ hashFiles('src/frontend/**/yarn.lock') }}
fail-on-cache-miss: true
- name: generate translations files
working-directory: src/frontend
run: yarn i18n:deploy
# Create a new PR
- name: Create a new Pull Request with new translated strings
uses: peter-evans/create-pull-request@v7
with:
commit-message: |
🌐(i18n) update translated strings
Update translated files with new translations
title: 🌐(i18n) update translated strings
body: |
## Purpose
update translated strings
## Proposal
- [x] update translated strings
branch: i18n/update-translations
labels: i18n,noChangeLog
+83
View File
@@ -0,0 +1,83 @@
name: Update crowdin sources
on:
workflow_dispatch:
push:
branches:
- main
jobs:
install-dependencies:
uses: ./.github/workflows/dependencies.yml
with:
node_version: '22.x'
with-front-dependencies-installation: true
with-build_mails: true
synchronize-with-crowdin:
needs: install-dependencies
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
# Backend i18n
- name: Install Python
uses: actions/setup-python@v5
with:
python-version: "3.13.3"
cache: "pip"
- name: Upgrade pip and setuptools
run: pip install --upgrade pip setuptools
- name: Install development dependencies
run: pip install --user .
working-directory: src/backend
- name: Restore the mail templates
uses: actions/cache@v4
id: mail-templates
with:
path: "src/backend/core/templates/mail"
key: mail-templates-${{ hashFiles('src/mail/mjml') }}
fail-on-cache-miss: true
- name: Install gettext
run: |
sudo apt-get update
sudo apt-get install -y gettext pandoc
- name: generate pot files
working-directory: src/backend
run: |
DJANGO_CONFIGURATION=Build python manage.py makemessages -a --keep-pot
# frontend i18n
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: "22.x"
- name: Install yarn
run: npm install -g yarn
- name: Restore the frontend cache
uses: actions/cache@v4
with:
path: "src/frontend/**/node_modules"
key: front-node_modules-${{ hashFiles('src/frontend/**/yarn.lock') }}
fail-on-cache-miss: true
- name: generate source translation file
working-directory: src/frontend
run: yarn i18n:extract
# crowdin workflow
- name: crowdin action
uses: crowdin/github-action@v2
with:
config: crowdin/config.yml
upload_sources: true
upload_translations: false
download_translations: false
create_pull_request: false
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# A numeric ID, found at https://crowdin.com/project/<projectName>/tools/api
CROWDIN_PROJECT_ID: ${{ secrets.CROWDIN_PROJECT_ID }}
# Visit https://crowdin.com/settings#api-key to create this token
CROWDIN_PERSONAL_TOKEN: ${{ secrets.CROWDIN_PERSONAL_TOKEN }}
CROWDIN_BASE_PATH: "../src/"
+93
View File
@@ -0,0 +1,93 @@
name: Dependency reusable workflow
on:
workflow_call:
inputs:
node_version:
required: false
default: '22.x'
type: string
with-front-dependencies-installation:
type: boolean
default: false
with-build_mails:
type: boolean
default: false
jobs:
front-dependencies-installation:
if: ${{ inputs.with-front-dependencies-installation == true }}
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Restore the frontend cache
uses: actions/cache@v4
id: front-node_modules
with:
path: "src/frontend/**/node_modules"
key: front-node_modules-${{ hashFiles('src/frontend/**/yarn.lock') }}
- name: Setup Node.js
if: steps.front-node_modules.outputs.cache-hit != 'true'
uses: actions/setup-node@v4
with:
node-version: ${{ inputs.node_version }}
- name: Install yarn
if: steps.front-node_modules.outputs.cache-hit != 'true'
run: npm install -g yarn
- name: Install dependencies
if: steps.front-node_modules.outputs.cache-hit != 'true'
run: cd src/frontend/ && yarn install --frozen-lockfile
- name: Cache install frontend
if: steps.front-node_modules.outputs.cache-hit != 'true'
uses: actions/cache@v4
with:
path: "src/frontend/**/node_modules"
key: front-node_modules-${{ hashFiles('src/frontend/**/yarn.lock') }}
build-mails:
if: ${{ inputs.with-build_mails == true }}
runs-on: ubuntu-latest
defaults:
run:
working-directory: src/mail
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Restore the mail templates
uses: actions/cache@v4
id: mail-templates
with:
path: "src/backend/core/templates/mail"
key: mail-templates-${{ hashFiles('src/mail/mjml') }}
- name: Setup Node.js
if: steps.mail-templates.outputs.cache-hit != 'true'
uses: actions/setup-node@v4
with:
node-version: ${{ inputs.node_version }}
- name: Install yarn
if: steps.mail-templates.outputs.cache-hit != 'true'
run: npm install -g yarn
- name: Install node dependencies
if: steps.mail-templates.outputs.cache-hit != 'true'
run: yarn install --frozen-lockfile
- name: Build mails
if: steps.mail-templates.outputs.cache-hit != 'true'
run: yarn build
- name: Cache mail templates
if: steps.mail-templates.outputs.cache-hit != 'true'
uses: actions/cache@v4
with:
path: "src/backend/core/templates/mail"
key: mail-templates-${{ hashFiles('src/mail/mjml') }}
+112
View File
@@ -0,0 +1,112 @@
name: Docker Hub Workflow
run-name: Docker Hub Workflow
on:
workflow_dispatch:
push:
branches:
- 'main'
tags:
- 'v*'
pull_request:
branches:
- 'main'
- 'ci/trivy-fails'
env:
DOCKER_USER: 1001:127
jobs:
build-and-push-backend:
runs-on: ubuntu-latest
steps:
-
name: Checkout repository
uses: actions/checkout@v4
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
-
name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: lasuite/conversations-backend
-
name: Login to DockerHub
if: github.event_name != 'pull_request'
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKER_HUB_USER }}
password: ${{ secrets.DOCKER_HUB_PASSWORD }}
-
name: Run trivy scan
uses: numerique-gouv/action-trivy-cache@main
with:
docker-build-args: '--target backend-production -f Dockerfile'
docker-image-name: 'docker.io/lasuite/conversations-backend:${{ github.sha }}'
-
name: Build and push
uses: docker/build-push-action@v6
with:
context: .
target: backend-production
build-args: DOCKER_USER=${{ env.DOCKER_USER }}:-1000
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
build-and-push-frontend:
runs-on: ubuntu-latest
steps:
-
name: Checkout repository
uses: actions/checkout@v4
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
-
name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: lasuite/conversations-frontend
-
name: Login to DockerHub
if: github.event_name != 'pull_request'
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKER_HUB_USER }}
password: ${{ secrets.DOCKER_HUB_PASSWORD }}
-
name: Run trivy scan
uses: numerique-gouv/action-trivy-cache@main
with:
docker-build-args: '-f src/frontend/Dockerfile --target frontend-production'
docker-image-name: 'docker.io/lasuite/conversations-frontend:${{ github.sha }}'
-
name: Build and push
uses: docker/build-push-action@v6
with:
context: .
file: ./src/frontend/Dockerfile
target: frontend-production
build-args: |
DOCKER_USER=${{ env.DOCKER_USER }}:-1000
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
notify-argocd:
needs:
- build-and-push-frontend
- build-and-push-backend
runs-on: ubuntu-latest
if: github.event_name != 'pull_request'
steps:
- uses: numerique-gouv/action-argocd-webhook-notification@main
id: notify
with:
deployment_repo_path: "${{ secrets.DEPLOYMENT_REPO_URL }}"
argocd_webhook_secret: "${{ secrets.ARGOCD_PREPROD_WEBHOOK_SECRET }}"
argocd_url: "${{ vars.ARGOCD_PREPROD_WEBHOOK_URL }}"
+30
View File
@@ -0,0 +1,30 @@
name: Helmfile lint
run-name: Helmfile lint
on:
push:
pull_request:
branches:
- 'main'
jobs:
helmfile-lint:
runs-on: ubuntu-latest
container:
image: ghcr.io/helmfile/helmfile:v0.171.0
steps:
-
name: Checkout repository
uses: actions/checkout@v4
-
name: Helmfile lint
shell: bash
run: |
set -e
HELMFILE=src/helm/helmfile.yaml.gotmpl
environments=$(awk 'BEGIN {in_env=0} /^environments:/ {in_env=1; next} /^---/ {in_env=0} in_env && /^ [^ ]/ {gsub(/^ /,""); gsub(/:.*$/,""); print}' "$HELMFILE")
for env in $environments; do
echo "################### $env lint ###################"
helmfile -e $env -f $HELMFILE lint || exit 1
echo -e "\n"
done
+34
View File
@@ -0,0 +1,34 @@
name: Release Chart
run-name: Release Chart
on:
push:
paths:
- src/helm/conversations/**
jobs:
release:
# depending on default permission settings for your org (contents being read-only or read-write for workloads), you will have to add permissions
# see: https://docs.github.com/en/actions/security-guides/automatic-token-authentication#modifying-the-permissions-for-the-github_token
permissions:
contents: write
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Cleanup
run: rm -rf ./src/helm/extra
- name: Install Helm
uses: azure/setup-helm@v4
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
- name: Publish Helm charts
uses: numerique-gouv/helm-gh-pages@add-overwrite-option
with:
charts_dir: ./src/helm
token: ${{ secrets.GITHUB_TOKEN }}
+81
View File
@@ -0,0 +1,81 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
.DS_Store
.next/
# Translations # Translations
*.mo
*.pot
# Environments
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
env.d/development/*
!env.d/development/*.dist
env.d/terraform
# npm
node_modules
# Mails
src/backend/core/templates/mail/
# Swagger
**/swagger.json
# Logs
*.log
# Terraform
.terraform
*.tfstate
*.tfstate.backup
# Test & lint
.coverage
.pylint.d
.pytest_cache
db.sqlite3
.mypy_cache
# Site media
/data/
# IDEs
.idea/
.vscode/
*.iml
.devcontainer
# Docker compose override
compose.override.yml
+78
View File
@@ -0,0 +1,78 @@
# All these sections are optional, edit this file as you like.
[general]
# Ignore certain rules, you can reference them by their id or by their full name
# ignore=title-trailing-punctuation, T3
# verbosity should be a value between 1 and 3, the commandline -v flags take precedence over this
# verbosity = 2
# By default gitlint will ignore merge commits. Set to 'false' to disable.
# ignore-merge-commits=true
# By default gitlint will ignore fixup commits. Set to 'false' to disable.
# ignore-fixup-commits=true
# By default gitlint will ignore squash commits. Set to 'false' to disable.
# ignore-squash-commits=true
# Enable debug mode (prints more output). Disabled by default.
# debug=true
# Set the extra-path where gitlint will search for user defined rules
# See http://jorisroovers.github.io/gitlint/user_defined_rules for details
extra-path=gitlint/
# [title-max-length]
# line-length=80
[title-must-not-contain-word]
# Comma-separated list of words that should not occur in the title. Matching is case
# insensitive. It's fine if the keyword occurs as part of a larger word (so "WIPING"
# will not cause a violation, but "WIP: my title" will.
words=wip
#[title-match-regex]
# python like regex (https://docs.python.org/2/library/re.html) that the
# commit-msg title must be matched to.
# Note that the regex can contradict with other rules if not used correctly
# (e.g. title-must-not-contain-word).
#regex=
# [B1]
# B1 = body-max-line-length
# line-length=120
# [body-min-length]
# min-length=5
# [body-is-missing]
# Whether to ignore this rule on merge commits (which typically only have a title)
# default = True
# ignore-merge-commits=false
# [body-changed-file-mention]
# List of files that need to be explicitly mentioned in the body when they are changed
# This is useful for when developers often erroneously edit certain files or git submodules.
# By specifying this rule, developers can only change the file when they explicitly reference
# it in the commit message.
# files=gitlint/rules.py,README.md
# [author-valid-email]
# python like regex (https://docs.python.org/2/library/re.html) that the
# commit author email address should be matched to
# For example, use the following regex if you only want to allow email addresses from foo.com
# regex=[^@]+@foo.com
[ignore-by-title]
# Allow empty body & wrong title pattern only when bots (pyup/greenkeeper)
# upgrade dependencies
regex=^(⬆️.*|Update (.*) from (.*) to (.*)|(chore|fix)\(package\): update .*)$
ignore=B6,UC1
# [ignore-by-body]
# Ignore certain rules for commits of which the body has a line that matches a regex
# E.g. Match bodies that have a line that that contain "release"
# regex=(.*)release(.*)
#
# Ignore certain rules, you can reference them by their id or by their full name
# Use 'all' to ignore all rules
# ignore=T1,body-min-length
+125 -1
View File
@@ -8,5 +8,129 @@ and this project adheres to
## [Unreleased]
## [0.0.7] - 2025-10-28
[unreleased]: https://github.com/numerique-gouv/conversations/compare/HEAD...main
### Fixed
- 🚑️(posthog) fix the posthog middleware for async mode #133
## [0.0.6] - 2025-10-28
### Fixed
- 🚑️(stats) fix tracking id in upload event #130
## [0.0.5] - 2025-10-27
### Fixed
- 🚑️(drag-drop) fix the rejection display on Safari #127
## [0.0.4] - 2025-10-27
### Added
- 🌐(i18n) add dutch language #117
### Changed
- ⚡️(asgi) use `uvicorn` to serve backend #121
### Fixed
- 🐛(front) fix mobile source
- 🐛(attachments) reject the whole drag&drop if unsupported formats #123
## [0.0.3] - 2025-10-21
### Fixed
- 🚑️(web-search) fix missing argument in RAG backend #116
## [0.0.2] - 2025-10-21
### Added
- ✨(front) add drag'n drop file
- ✨(activation-codes) register users also on Brevo #98
- 📈(posthog) add `sub` field to tracking #95
### Changed
- 🔧(front) change links feedback tchap + settings popup
- 🐛(front) code activation fix session end #93
- 💬(wording) error page wording #102
- ⚡️(web-search) allow to override returned chunks #107
- 🐛(activation-codes) create contact in brevo before add to list #108
- ⚗️(summarization) add system prompt to handle tool #112
## [0.0.1] - 2025-10-19
### Changed
- 🎨(front) activation page footer
- 👷(front) change size small modal
- 🎨(front) retour ui global
- 👷(front) fix button scrollDown
- 💥(front) disable input when error occurred
- 👷(front) fix scroll
- 🐛(front) fix left panel status + fix scroll
- 🐛(llm) add is_active field and persist chat preference
- ✨(frontend) add LLM selection in chat input #53
- 🎨(front) fix width chat container
- 🎨(front) fix width chat container #55
- 🐛(front) fix button search web on new conversation
- 🎨(front) improvement search input scroll
- ✨(404) fix front 404 page
- ✅(chat) add frontend feature flags #29
- 🎨(front) change list attachment in chat
- 🎨(front) move emplacement for attachment
- 🎨(ui) retour ui sources files
- ✨(ui) fix retour global ui
- 🐛(fix) broken staging css
- 🎨(alpha) adjustment for alpha version
- ✨(ui) delete flex message
- ✅(front) add enabled/disabled conversation analysis
- 🎨(front) amelioration chat ux
- 🎨(front) global layout modification
- ✨(front) global layout UI
- ♻️(chat) rewrite backend using Pydantic AI SDK #4
- 🗃️(chat) enforce messages stored JSON format #6
- 🐛(chat) UI messages must have a unique identifier #6
- ✨(llm) allow configuration from JSON file #22
- 💥(agent) replace routing w/ tool calls #40
- 🧱(storage) upload the user documents into S3 #86
### Added
- 🎉(conversations) bootstrap backend & frontend #1
- ✨(web-search) add RAG capability to do web search #7
- ✨(chat) add document RAG on document uploaded by user #8
- ✨(backend) allow use to stop conversation streaming #14
- 🐛(agent) add the current date in the system prompt #18
- ✨(backend) add feature flags from posthog #13
- ✨(user) allow to use conversation data for analytics #23
- ✨(chat) enforce response in user language #24
- 📈(langfuse) add light instrumentation #26
- 🚑️(agent) allow Mistral w/ vLLM & tools #36
- ✨(web-search) add Brave search tool #47
- ✨(models) add mistral support & customization #51
- 🐛(web-search) add summarization to Brave results #58
- ✨(langfuse) allow user to score messages from LLM #6
- ✨(onboarding) add activation code logic for launch #62
- 💄(chat) add code highlighting for LLM responses #67
[unreleased]: https://github.com/suitenumerique/conversations/compare/v0.0.7...main
[0.0.7]: https://github.com/suitenumerique/conversations/releases/v0.0.7
[0.0.6]: https://github.com/suitenumerique/conversations/releases/v0.0.6
[0.0.5]: https://github.com/suitenumerique/conversations/releases/v0.0.5
[0.0.4]: https://github.com/suitenumerique/conversations/releases/v0.0.4
[0.0.3]: https://github.com/suitenumerique/conversations/releases/v0.0.3
[0.0.2]: https://github.com/suitenumerique/conversations/releases/v0.0.2
[0.0.1]: https://github.com/suitenumerique/conversations/releases/v0.0.1
+1 -9
View File
@@ -2,7 +2,7 @@
Thank you for taking the time to contribute! Please follow these guidelines to ensure a smooth and productive workflow. 🚀🚀🚀
To get started with the project, please refer to the [README.md](https://github.com/suitenumerique/conversations/blob/main/README.md) for detailed instructions on how to run Docs locally.
To get started with the project, please refer to the [README.md](https://github.com/suitenumerique/conversations/blob/main/README.md) for detailed instructions on how to run Conversations locally.
Contributors are required to sign off their commits with `git commit --signoff`: this confirms that they have read and accepted the [Developer's Certificate of Origin 1.1](https://developercertificate.org/). For security reasons we also require [signing your commits with your SSH or GPG key](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification) with `git commit -S`.
@@ -92,11 +92,3 @@ Make sure that all new features or fixes have corresponding tests. Run the test
If you need any help while contributing, feel free to open a discussion or ask for guidance in the issue tracker. We are more than happy to assist!
Thank you for your contributions! 👍
## Contribute to BlockNote
We use [BlockNote](https://www.blocknotejs.org/) for the text editing features of Docs.
If you find and issue with the editor you can [report it](https://github.com/TypeCellOS/BlockNote/issues) directly on their repository.
Please consider contributing to BlockNotejs, as a library, it's useful to many projects not just Docs.
The project is licended with Mozilla Public License Version 2.0 but be aware that [XL packages](https://github.com/TypeCellOS/BlockNote/blob/main/packages/xl-docx-exporter/LICENSE) are dual licenced with GNU AFFERO GENERAL PUBLIC LICENCE Version 3 and proprietary licence if you are [sponsor](https://www.blocknotejs.org/pricing).
+174
View File
@@ -0,0 +1,174 @@
# Django conversations
# ---- base image to inherit from ----
FROM python:3.13.3-alpine AS base
# Upgrade pip to its latest release to speed up dependencies installation
RUN python -m pip install --upgrade pip setuptools
# Upgrade system packages to install security updates
RUN apk update && \
apk upgrade
# ---- Back-end builder image ----
FROM base AS back-builder
WORKDIR /builder
# Install Rust and Cargo using Alpine's package manager
RUN apk add --no-cache \
build-base \
libffi-dev \
rust \
cargo
# Copy required python dependencies
COPY ./src/backend /builder
RUN mkdir /install && \
pip install --prefix=/install .
# ---- mails ----
FROM node:24 AS mail-builder
COPY ./src/mail /mail/app
WORKDIR /mail/app
RUN yarn install --frozen-lockfile && \
yarn build
# ---- static link collector ----
FROM base AS link-collector
ARG CONVERSATIONS_STATIC_ROOT=/data/static
# Install pango & rdfind
RUN apk add \
pango \
rdfind
# Copy installed python dependencies
COPY --from=back-builder /install /usr/local
# Copy conversations application (see .dockerignore)
COPY ./src/backend /app/
WORKDIR /app
# collectstatic
RUN DJANGO_CONFIGURATION=Build \
python manage.py collectstatic --noinput
# Replace duplicated file by a symlink to decrease the overall size of the
# final image
RUN rdfind -makesymlinks true -followsymlinks true -makeresultsfile false ${CONVERSATIONS_STATIC_ROOT}
# ---- Core application image ----
FROM base AS core
ENV PYTHONUNBUFFERED=1
# Install required system libs
RUN apk add \
cairo \
file \
font-noto \
font-noto-emoji \
gettext \
gdk-pixbuf \
libffi-dev \
pango \
shared-mime-info
RUN wget https://svn.apache.org/repos/asf/httpd/httpd/trunk/docs/conf/mime.types -O /etc/mime.types
# Copy entrypoint
COPY ./docker/files/usr/local/bin/entrypoint /usr/local/bin/entrypoint
# Give the "root" group the same permissions as the "root" user on /etc/passwd
# to allow a user belonging to the root group to add new users; typically the
# docker user (see entrypoint).
RUN chmod g=u /etc/passwd
# Copy installed python dependencies
COPY --from=back-builder /install /usr/local
# Copy conversations application (see .dockerignore)
COPY ./src/backend /app/
WORKDIR /app
# Generate compiled translation messages
RUN DJANGO_CONFIGURATION=Build \
python manage.py compilemessages
# We wrap commands run in this container by the following entrypoint that
# creates a user on-the-fly with the container user ID (see USER) and root group
# ID.
ENTRYPOINT [ "/usr/local/bin/entrypoint" ]
# ---- Development image ----
FROM core AS backend-development
# Switch back to the root user to install development dependencies
USER root:root
# Install psql
RUN apk add postgresql-client
# Uninstall conversations and re-install it in editable mode along with development
# dependencies
RUN pip uninstall -y conversations
RUN pip install -e .[dev]
# Restore the un-privileged user running the application
ARG DOCKER_USER
USER ${DOCKER_USER}
# Target database host (e.g. database engine following docker compose services
# name) & port
ENV DB_HOST=postgresql \
DB_PORT=5432
# Run django development server
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
# ---- Production image ----
FROM core AS backend-production
# Remove apk cache, we don't need it anymore
RUN rm -rf /var/cache/apk/*
ARG CONVERSATIONS_STATIC_ROOT=/data/static
# Gunicorn - not used by default but configuration file is provided
RUN mkdir -p /usr/local/etc/gunicorn
COPY docker/files/usr/local/etc/gunicorn/conversations.py /usr/local/etc/gunicorn/conversations.py
# Un-privileged user running the application
ARG DOCKER_USER
USER ${DOCKER_USER}
# Copy statics
COPY --from=link-collector ${CONVERSATIONS_STATIC_ROOT} ${CONVERSATIONS_STATIC_ROOT}
# Copy conversations mails
COPY --from=mail-builder /mail/backend/core/templates/mail /app/core/templates/mail
# The default command runs uvicorn ASGI server in conversations's main module
# WEB_CONCURRENCY: number of workers to run <=> --workers=4
ENV WEB_CONCURRENCY=4
CMD [\
"uvicorn",\
"--app-dir=/app",\
"--host=0.0.0.0",\
"--timeout-graceful-shutdown=300",\
"--limit-max-requests=20000",\
"conversations.asgi:application"\
]
# To run using gunicorn WSGI server use this instead:
#CMD ["gunicorn", "-c", "/usr/local/etc/gunicorn/conversations.py", "conversations.wsgi:application"]
+383
View File
@@ -0,0 +1,383 @@
# /!\ /!\ /!\ /!\ /!\ /!\ /!\ DISCLAIMER /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\
#
# This Makefile is only meant to be used for DEVELOPMENT purpose as we are
# changing the user id that will run in the container.
#
# PLEASE DO NOT USE IT FOR YOUR CI/PRODUCTION/WHATEVER...
#
# /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\ /!\
#
# Note to developers:
#
# While editing this file, please respect the following statements:
#
# 1. Every variable should be defined in the ad hoc VARIABLES section with a
# relevant subsection
# 2. Every new rule should be defined in the ad hoc RULES section with a
# relevant subsection depending on the targeted service
# 3. Rules should be sorted alphabetically within their section
# 4. When a rule has multiple dependencies, you should:
# - duplicate the rule name to add the help string (if required)
# - write one dependency per line to increase readability and diffs
# 5. .PHONY rule statement should be written after the corresponding rule
# ==============================================================================
# VARIABLES
BOLD := \033[1m
RESET := \033[0m
GREEN := \033[1;32m
# -- Database
DB_HOST = postgresql
DB_PORT = 5432
# -- Docker
# Get the current user ID to use for docker run and docker exec commands
DOCKER_UID = $(shell id -u)
DOCKER_GID = $(shell id -g)
DOCKER_USER = $(DOCKER_UID):$(DOCKER_GID)
COMPOSE = DOCKER_USER=$(DOCKER_USER) docker compose
COMPOSE_E2E = DOCKER_USER=$(DOCKER_USER) docker compose -f compose.yml -f compose-e2e.yml
COMPOSE_EXEC = $(COMPOSE) exec
COMPOSE_EXEC_APP = $(COMPOSE_EXEC) app-dev
COMPOSE_RUN = $(COMPOSE) run --rm
COMPOSE_RUN_APP = $(COMPOSE_RUN) app-dev
COMPOSE_RUN_CROWDIN = $(COMPOSE_RUN) crowdin crowdin
# -- Backend
MANAGE = $(COMPOSE_RUN_APP) python manage.py
MAIL_YARN = $(COMPOSE_RUN) -w /app/src/mail node yarn
# -- Frontend
PATH_FRONT = ./src/frontend
PATH_FRONT_CONVERSATIONS = $(PATH_FRONT)/apps/conversations
# ==============================================================================
# RULES
default: help
data/media:
@mkdir -p data/media
data/static:
@mkdir -p data/static
# -- Project
create-env-files: ## Copy the dist env files to env files
create-env-files: \
env.d/development/common \
env.d/development/crowdin \
env.d/development/postgresql \
env.d/development/kc_postgresql
.PHONY: create-env-files
pre-bootstrap: \
data/media \
data/static \
create-env-files
.PHONY: pre-bootstrap
post-bootstrap: \
migrate \
demo \
back-i18n-compile \
mails-install \
mails-build
.PHONY: post-bootstrap
bootstrap: ## Prepare Docker developmentimages for the project
bootstrap: \
pre-bootstrap \
build \
post-bootstrap \
run
.PHONY: bootstrap
bootstrap-e2e: ## Prepare Docker production images to be used for e2e tests
bootstrap-e2e: \
pre-bootstrap \
build-e2e \
post-bootstrap \
run-e2e
.PHONY: bootstrap-e2e
# -- Docker/compose
build: cache ?=
build: ## build the project containers
@$(MAKE) build-backend cache=$(cache)
@$(MAKE) build-frontend cache=$(cache)
.PHONY: build
build-backend: cache ?=
build-backend: ## build the app-dev container
@$(COMPOSE) build app-dev $(cache)
.PHONY: build-backend
build-frontend: cache ?=
build-frontend: ## build the frontend container
@$(COMPOSE) build frontend-development $(cache)
.PHONY: build-frontend
build-e2e: cache ?=
build-e2e: ## build the e2e container
@$(MAKE) build-backend cache=$(cache)
@$(COMPOSE_E2E) build frontend $(cache)
.PHONY: build-e2e
down: ## stop and remove containers, networks, images, and volumes
@$(COMPOSE_E2E) down
.PHONY: down
logs: ## display app-dev logs (follow mode)
@$(COMPOSE) logs -f app-dev
.PHONY: logs
run-backend: ## Start only the backend application and all needed services
@$(COMPOSE) up --force-recreate -d nginx app-dev
.PHONY: run-backend
run-frontend: ## Start only the frontend application
@$(COMPOSE) up --force-recreate -d frontend-development
.PHONY: run-frontend
run: ## start the wsgi (production) and development server
run:
@$(MAKE) run-backend
@$(MAKE) run-frontend
.PHONY: run
create-compose-with-models: ## override the docker-compose file with models
cp -n compose.with_model.override.yml compose.override.yml
.PHONY: create-compose-with-models
run-e2e: ## start the e2e server
run-e2e:
@$(MAKE) run-backend
@$(COMPOSE_E2E) up --force-recreate -d frontend
.PHONY: run-e2e
status: ## an alias for "docker compose ps"
@$(COMPOSE_E2E) ps
.PHONY: status
stop: ## stop the development server using Docker
@$(COMPOSE_E2E) stop
.PHONY: stop
# -- Backend
demo: ## flush db then create a demo for load testing purpose
@$(MAKE) resetdb
@$(MANAGE) create_demo
.PHONY: demo
# Nota bene: Black should come after isort just in case they don't agree...
lint: ## lint back-end python sources
lint: \
lint-ruff-format \
lint-ruff-check \
lint-pylint
.PHONY: lint
lint-ruff-format: ## format back-end python sources with ruff
@echo 'lint:ruff-format started…'
@$(COMPOSE_RUN_APP) ruff format .
.PHONY: lint-ruff-format
lint-ruff-check: ## lint back-end python sources with ruff
@echo 'lint:ruff-check started…'
@$(COMPOSE_RUN_APP) ruff check . --fix
.PHONY: lint-ruff-check
lint-pylint: ## lint back-end python sources with pylint only on changed files from main
@echo 'lint:pylint started…'
bin/pylint --diff-only=origin/main
.PHONY: lint-pylint
test: ## run project tests
@$(MAKE) test-back-parallel
.PHONY: test
test-back: ## run back-end tests
@args="$(filter-out $@,$(MAKECMDGOALS))" && \
bin/pytest $${args:-${1}}
.PHONY: test-back
test-back-parallel: ## run all back-end tests in parallel
@args="$(filter-out $@,$(MAKECMDGOALS))" && \
bin/pytest -n auto $${args:-${1}}
.PHONY: test-back-parallel
makemigrations: ## run django makemigrations for the conversations project.
@echo "$(BOLD)Running makemigrations$(RESET)"
@$(COMPOSE) up -d postgresql
@$(MANAGE) makemigrations
.PHONY: makemigrations
migrate: ## run django migrations for the conversations project.
@echo "$(BOLD)Running migrations$(RESET)"
@$(COMPOSE) up -d postgresql
@$(MANAGE) migrate
.PHONY: migrate
superuser: ## Create an admin superuser with password "admin"
@echo "$(BOLD)Creating a Django superuser$(RESET)"
@$(MANAGE) createsuperuser --email admin@example.com --password admin
.PHONY: superuser
back-i18n-compile: ## compile the gettext files
@$(MANAGE) compilemessages --ignore="venv/**/*"
.PHONY: back-i18n-compile
back-i18n-generate: ## create the .pot files used for i18n
@$(MANAGE) makemessages -a --keep-pot --all
.PHONY: back-i18n-generate
shell: ## connect to database shell
@$(MANAGE) shell #_plus
.PHONY: dbshell
# -- Database
dbshell: ## connect to database shell
docker compose exec app-dev python manage.py dbshell
.PHONY: dbshell
resetdb: FLUSH_ARGS ?=
resetdb: ## flush database and create a superuser "admin"
@echo "$(BOLD)Flush database$(RESET)"
@$(MANAGE) flush $(FLUSH_ARGS)
@${MAKE} superuser
.PHONY: resetdb
env.d/development/common:
cp -n env.d/development/common.dist env.d/development/common
env.d/development/postgresql:
cp -n env.d/development/postgresql.dist env.d/development/postgresql
env.d/development/kc_postgresql:
cp -n env.d/development/kc_postgresql.dist env.d/development/kc_postgresql
# -- Internationalization
env.d/development/crowdin:
cp -n env.d/development/crowdin.dist env.d/development/crowdin
crowdin-download: ## Download translated message from crowdin
@$(COMPOSE_RUN_CROWDIN) download -c crowdin/config.yml
.PHONY: crowdin-download
crowdin-download-sources: ## Download sources from Crowdin
@$(COMPOSE_RUN_CROWDIN) download sources -c crowdin/config.yml
.PHONY: crowdin-download-sources
crowdin-upload: ## Upload source translations to crowdin
@$(COMPOSE_RUN_CROWDIN) upload sources -c crowdin/config.yml
.PHONY: crowdin-upload
i18n-compile: ## compile all translations
i18n-compile: \
back-i18n-compile \
frontend-i18n-compile
.PHONY: i18n-compile
i18n-generate: ## create the .pot files and extract frontend messages
i18n-generate: \
back-i18n-generate \
frontend-i18n-generate
.PHONY: i18n-generate
i18n-download-and-compile: ## download all translated messages and compile them to be used by all applications
i18n-download-and-compile: \
crowdin-download \
i18n-compile
.PHONY: i18n-download-and-compile
i18n-generate-and-upload: ## generate source translations for all applications and upload them to Crowdin
i18n-generate-and-upload: \
i18n-generate \
crowdin-upload
.PHONY: i18n-generate-and-upload
# -- Mail generator
mails-build: ## Convert mjml files to html and text
@$(MAIL_YARN) build
.PHONY: mails-build
mails-build-html-to-plain-text: ## Convert html files to text
@$(MAIL_YARN) build-html-to-plain-text
.PHONY: mails-build-html-to-plain-text
mails-build-mjml-to-html: ## Convert mjml files to html and text
@$(MAIL_YARN) build-mjml-to-html
.PHONY: mails-build-mjml-to-html
mails-install: ## install the mail generator
@$(MAIL_YARN) install
.PHONY: mails-install
# -- Misc
clean: ## restore repository state as it was freshly cloned
git clean -idx
.PHONY: clean
help:
@echo "$(BOLD)conversations Makefile"
@echo "Please use 'make $(BOLD)target$(RESET)' where $(BOLD)target$(RESET) is one of:"
@grep -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(firstword $(MAKEFILE_LIST)) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "$(GREEN)%-30s$(RESET) %s\n", $$1, $$2}'
.PHONY: help
# Front
frontend-development-install: ## install the frontend locally
cd $(PATH_FRONT_CONVERSATIONS) && yarn
.PHONY: frontend-development-install
frontend-lint: ## run the frontend linter
cd $(PATH_FRONT) && yarn lint
.PHONY: frontend-lint
run-frontend-development: ## Run the frontend in development mode
#@$(COMPOSE) stop frontend frontend-development
cd $(PATH_FRONT_CONVERSATIONS) && yarn dev
.PHONY: run-frontend-development
frontend-i18n-extract: ## Extract the frontend translation inside a json to be used for crowdin
cd $(PATH_FRONT) && yarn i18n:extract
.PHONY: frontend-i18n-extract
frontend-i18n-generate: ## Generate the frontend json files used for crowdin
frontend-i18n-generate: \
crowdin-download-sources \
frontend-i18n-extract
.PHONY: frontend-i18n-generate
frontend-i18n-compile: ## Format the crowin json files used deploy to the apps
cd $(PATH_FRONT) && yarn i18n:deploy
.PHONY: frontend-i18n-compile
# -- K8S
build-k8s-cluster: ## build the kubernetes cluster using kind
./bin/start-kind.sh
.PHONY: build-k8s-cluster
start-tilt: ## start the kubernetes cluster using kind
tilt up -f ./bin/Tiltfile
.PHONY: build-k8s-cluster
bump-packages-version: VERSION_TYPE ?= minor
bump-packages-version: ## bump the version of the project - VERSION_TYPE can be "major", "minor", "patch"
cd ./src/mail && yarn version --no-git-tag-version --$(VERSION_TYPE)
cd ./src/frontend/ && yarn version --no-git-tag-version --$(VERSION_TYPE)
cd ./src/frontend/apps/e2e/ && yarn version --no-git-tag-version --$(VERSION_TYPE)
cd ./src/frontend/apps/conversations/ && yarn version --no-git-tag-version --$(VERSION_TYPE)
cd ./src/frontend/packages/eslint-config-conversations/ && yarn version --no-git-tag-version --$(VERSION_TYPE)
cd ./src/frontend/packages/i18n/ && yarn version --no-git-tag-version --$(VERSION_TYPE)
.PHONY: bump-packages-version
+21 -5
View File
@@ -16,6 +16,24 @@
</p>
**Warning:** This project is in active development and in a very early stage. Breaking changes may occur at any time.
## Yet another AI chatbot
Conversations is an open-source AI chatbot designed to be simple, secure and privacy-friendly.
Why another AI chatbot? Because we want to be able to fully control our data and the way we interact with AI.
We want to have a very friendly end-user interface and code, and we want to be able to easily customize the
chatbot to our needs.
We leverage open-source projects such as [Vercel&lsquo;s AI SDK](https://ai-sdk.dev/) and [Pydantic AI](https://ai.pydantic.dev)
and only assemble them in a way that makes sense for us and allows us to focus on the product.
This assistant's purpose is also to be integrated into the "La Suite numérique" ecosystem of tools for public services.
Any help to improve the project is very welcome!
### Self-host
🚀 Conversations is easy to install on your own servers
@@ -30,9 +48,9 @@ In the works: Docker Compose, soon YunoHost
You can test Conversations on your browser by visiting this => TBD
### Run Docs locally
### Run Conversations locally
> ⚠️ The methods described below for running Docs locally is **for testing purposes only**. It is based on building Docs using [Minio](https://min.io/) as an S3-compatible storage solution. Of course you can choose any S3-compatible storage solution.
> ⚠️ The methods described below for running Conversations locally is **for testing purposes only**.
**Prerequisite**
@@ -152,9 +170,7 @@ docs
### Stack
Conversations is built on top of [Django Rest Framework](https://www.django-rest-framework.org/), [Next.js](https://nextjs.org/), [Vercel&lsquo;s AI SDK](https://ai-sdk.dev/) and [OpenAI Agents SDK](https://github.com/openai/openai-agents-python). We thank the contributors of all these projects for their awesome work!
Conversations is built on top of [Django Rest Framework](https://www.django-rest-framework.org/), [Next.js](https://nextjs.org/), [Vercel&lsquo;s AI SDK](https://ai-sdk.dev/) and [Pydantic AI](https://ai.pydantic.dev). We thank the contributors of all these projects for their awesome work!
### Gov ❤️ open source
+2 -2
View File
@@ -6,7 +6,7 @@ Security is very important to us.
If you have any issue regarding security, please disclose the information responsibly submitting [this form](https://vdp.numerique.gouv.fr/p/Send-a-report?lang=en) and not by creating an issue on the repository. You can also email us at docs@numerique.gouv.fr
We appreciate your effort to make Docs more secure.
We appreciate your effort to make Conversations more secure.
## Vulnerability disclosure policy
@@ -20,4 +20,4 @@ and Exposures (CVE) identifier for the vulnerability.
3. Once this grace period has passed, we will publish the vulnerability.
By adhering to this security policy, we aim to address security concerns
effectively and responsibly in our open source software project.
effectively and responsibly in our open source software project.
+56
View File
@@ -0,0 +1,56 @@
load('ext://uibutton', 'cmd_button', 'bool_input', 'location')
load('ext://namespace', 'namespace_create', 'namespace_inject')
namespace_create('conversations')
docker_build(
'localhost:5001/conversations-backend:latest',
context='..',
dockerfile='../Dockerfile',
only=['./src/backend', './src/mail', './docker'],
target = 'backend-production',
live_update=[
sync('../src/backend', '/app'),
]
)
docker_build(
'localhost:5001/conversations-frontend:latest',
context='..',
dockerfile='../src/frontend/Dockerfile',
only=['./src/frontend', './docker', './.dockerignore'],
target = 'conversations',
live_update=[
sync('../src/frontend', '/home/frontend'),
]
)
k8s_resource('conversations-backend-migrate', resource_deps=['postgres-postgresql'])
k8s_resource('conversations-backend-createsuperuser', resource_deps=['conversations-backend-migrate'])
k8s_resource('conversations-backend', resource_deps=['conversations-backend-migrate'])
k8s_yaml(local('cd ../src/helm && helmfile -n conversations -e dev template .'))
migration = '''
set -eu
# get k8s pod name from tilt resource name
POD_NAME="$(tilt get kubernetesdiscovery conversations-backend -ojsonpath='{.status.pods[0].name}')"
kubectl -n conversations exec "$POD_NAME" -- python manage.py makemigrations
'''
cmd_button('Make migration',
argv=['sh', '-c', migration],
resource='conversations-backend',
icon_name='developer_board',
text='Run makemigration',
)
pod_migrate = '''
set -eu
# get k8s pod name from tilt resource name
POD_NAME="$(tilt get kubernetesdiscovery conversations-backend -ojsonpath='{.status.pods[0].name}')"
kubectl -n conversations exec "$POD_NAME" -- python manage.py migrate --no-input
'''
cmd_button('Migrate db',
argv=['sh', '-c', pod_migrate],
resource='conversations-backend',
icon_name='developer_board',
text='Run database migration',
)
+155
View File
@@ -0,0 +1,155 @@
#!/usr/bin/env bash
set -eo pipefail
REPO_DIR="$(cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd)"
UNSET_USER=0
TERRAFORM_DIRECTORY="./env.d/terraform"
COMPOSE_FILE="${REPO_DIR}/compose.yml"
# _set_user: set (or unset) default user id used to run docker commands
#
# usage: _set_user
#
# You can override default user ID (the current host user ID), by defining the
# USER_ID environment variable.
#
# To avoid running docker commands with a custom user, please set the
# $UNSET_USER environment variable to 1.
function _set_user() {
if [ $UNSET_USER -eq 1 ]; then
USER_ID=""
return
fi
# USER_ID = USER_ID or `id -u` if USER_ID is not set
USER_ID=${USER_ID:-$(id -u)}
echo "🙋(user) ID: ${USER_ID}"
}
# docker_compose: wrap docker compose command
#
# usage: docker_compose [options] [ARGS...]
#
# options: docker compose command options
# ARGS : docker compose command arguments
function _docker_compose() {
echo "🐳(compose) file: '${COMPOSE_FILE}'"
docker compose \
-f "${COMPOSE_FILE}" \
--project-directory "${REPO_DIR}" \
"$@"
}
# _dc_run: wrap docker compose run command
#
# usage: _dc_run [options] [ARGS...]
#
# options: docker compose run command options
# ARGS : docker compose run command arguments
function _dc_run() {
_set_user
user_args="--user=$USER_ID"
if [ -z $USER_ID ]; then
user_args=""
fi
_docker_compose run --rm $user_args "$@"
}
# _dc_exec: wrap docker compose exec command
#
# usage: _dc_exec [options] [ARGS...]
#
# options: docker compose exec command options
# ARGS : docker compose exec command arguments
function _dc_exec() {
_set_user
echo "🐳(compose) exec command: '\$@'"
user_args="--user=$USER_ID"
if [ -z $USER_ID ]; then
user_args=""
fi
_docker_compose exec $user_args "$@"
}
# _django_manage: wrap django's manage.py command with docker compose
#
# usage : _django_manage [ARGS...]
#
# ARGS : django's manage.py command arguments
function _django_manage() {
_dc_run "app-dev" python manage.py "$@"
}
# _set_openstack_project: select an OpenStack project from the openrc files defined in the
# terraform directory.
#
# usage: _set_openstack_project
#
# If necessary the script will prompt the user to choose a project from those available
function _set_openstack_project() {
declare prompt
declare -a projects
declare -i default=1
declare -i choice=0
declare -i n_projects
# List projects by looking in the "./env.d/terraform" directory
# and store them in an array
read -r -a projects <<< "$(
find "${TERRAFORM_DIRECTORY}" -maxdepth 1 -mindepth 1 -type d |
sed 's|'"${TERRAFORM_DIRECTORY}\/"'||' |
xargs
)"
nb_projects=${#projects[@]}
if [[ ${nb_projects} -le 0 ]]; then
echo "There are no OpenStack projects defined..." >&2
echo "To add one, create a subdirectory in \"${TERRAFORM_DIRECTORY}\" with the name" \
"of your project and copy your \"openrc.sh\" file into it." >&2
exit 10
fi
if [[ ${nb_projects} -gt 1 ]]; then
prompt="Select an OpenStack project to target:\\n"
for (( i=0; i<nb_projects; i++ )); do
prompt+="[$((i+1))] ${projects[$i]}"
if [[ $((i+1)) -eq ${default} ]]; then
prompt+=" (default)"
fi
prompt+="\\n"
done
prompt+="If your OpenStack project is not listed, add it to the \"env.d/terraform\" directory.\\n"
prompt+="Your choice: "
read -r -p "$(echo -e "${prompt}")" choice
if [[ ${choice} -gt nb_projects ]]; then
(>&2 echo "Invalid choice ${choice} (should be <= ${nb_projects})")
exit 11
fi
if [[ ${choice} -le 0 ]]; then
choice=${default}
fi
fi
project=${projects[$((choice-1))]}
# Check that the openrc.sh file exists for this project
if [ ! -f "${TERRAFORM_DIRECTORY}/${project}/openrc.sh" ]; then
(>&2 echo "Missing \"openrc.sh\" file in \"${TERRAFORM_DIRECTORY}/${project}\". Check documentation.")
exit 12
fi
echo "${project}"
}
Executable
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# shellcheck source=bin/_config.sh
source "$(dirname "${BASH_SOURCE[0]}")/_config.sh"
_docker_compose "$@"
+30
View File
@@ -0,0 +1,30 @@
#!/bin/bash
mkdir -p "$(dirname -- "${BASH_SOURCE[0]}")/../.git/hooks/"
PRE_COMMIT_FILE="$(dirname -- "${BASH_SOURCE[0]}")/../.git/hooks/pre-commit"
cat <<'EOF' >$PRE_COMMIT_FILE
#!/bin/bash
# directories containing potential secrets
DIRS="."
bold=$(tput bold)
normal=$(tput sgr0)
# allow to read user input, assigns stdin to keyboard
exec </dev/tty
for d in $DIRS; do
# find files containing secrets that should be encrypted
for f in $(find "${d}" -type f -regex ".*\.enc\..*"); do
if ! $(grep -q "unencrypted_suffix" $f); then
printf '\xF0\x9F\x92\xA5 '
echo "File $f has non encrypted secrets!"
exit 1
fi
done
done
EOF
chmod +x $PRE_COMMIT_FILE
Executable
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# shellcheck source=bin/_config.sh
source "$(dirname "${BASH_SOURCE[0]}")/_config.sh"
_django_manage "$@"
Executable
+38
View File
@@ -0,0 +1,38 @@
#!/usr/bin/env bash
# shellcheck source=bin/_config.sh
source "$(dirname "${BASH_SOURCE[0]}")/_config.sh"
declare diff_from
declare -a paths
declare -a args
# Parse options
for arg in "$@"
do
case $arg in
--diff-only=*)
diff_from="${arg#*=}"
shift
;;
-*)
args+=("$arg")
shift
;;
*)
paths+=("$arg")
shift
;;
esac
done
if [[ -n "${diff_from}" ]]; then
# Run pylint only on modified files located in src/backend
# (excluding deleted files and migration files)
# shellcheck disable=SC2207
paths=($(git diff "${diff_from}" --name-only --diff-filter=d -- src/backend ':!**/migrations/*.py' | grep -E '^src/backend/.*\.py$'))
fi
# Fix docker vs local path when project sources are mounted as a volume
read -ra paths <<< "$(echo "${paths[@]}" | sed "s|src/backend/||g")"
_dc_run app-dev pylint "${paths[@]}" "${args[@]}"
Executable
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env bash
source "$(dirname "${BASH_SOURCE[0]}")/_config.sh"
_dc_run \
-e DJANGO_CONFIGURATION=Test \
app-dev \
pytest "$@"
+2
View File
@@ -0,0 +1,2 @@
#!/bin/sh
curl https://raw.githubusercontent.com/numerique-gouv/tools/refs/heads/main/kind/create_cluster.sh | bash -s -- conversations
Executable
+25
View File
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
set -eo pipefail
source "$(dirname "${BASH_SOURCE[0]}")/_config.sh"
project=$(_set_openstack_project)
echo "Using \"${project}\" project..."
source "${TERRAFORM_DIRECTORY}/${project}/openrc.sh"
# Run Terraform commands in the Hashicorp docker container via docker compose
# shellcheck disable=SC2068
DOCKER_USER="$(id -u):$(id -g)" \
PROJECT="${project}" \
docker compose run --rm \
-e OS_AUTH_URL \
-e OS_IDENTITY_API_VERSION \
-e OS_USER_DOMAIN_NAME \
-e OS_PROJECT_DOMAIN_NAME \
-e OS_TENANT_ID \
-e OS_TENANT_NAME \
-e OS_USERNAME \
-e OS_PASSWORD \
-e OS_REGION_NAME \
terraform-state "$@"
Executable
+26
View File
@@ -0,0 +1,26 @@
#!/usr/bin/env bash
set -eo pipefail
source "$(dirname "${BASH_SOURCE[0]}")/_config.sh"
project=$(_set_openstack_project)
echo "Using \"${project}\" project..."
source "${TERRAFORM_DIRECTORY}/${project}/openrc.sh"
# Run Terraform commands in the Hashicorp docker container via docker compose
# shellcheck disable=SC2068
DOCKER_USER="$(id -u):$(id -g)" \
PROJECT="${project}" \
docker compose run --rm \
-e OS_AUTH_URL \
-e OS_IDENTITY_API_VERSION \
-e OS_USER_DOMAIN_NAME \
-e OS_PROJECT_DOMAIN_NAME \
-e OS_TENANT_ID \
-e OS_TENANT_NAME \
-e OS_USERNAME \
-e OS_PASSWORD \
-e OS_REGION_NAME \
-e TF_VAR_user_name \
terraform "$@"
+12
View File
@@ -0,0 +1,12 @@
#!/usr/bin/env bash
source "$(dirname "${BASH_SOURCE[0]}")/_config.sh"
_dc_run \
-e DJANGO_CONFIGURATION=Test \
app-dev \
python manage.py spectacular \
--api-version 'v1.0' \
--urlconf 'conversations.urls' \
--format openapi-json \
--file /app/core/tests/swagger/swagger.json
+13
View File
@@ -0,0 +1,13 @@
services:
frontend:
user: "${DOCKER_USER:-1000}"
build:
context: .
dockerfile: ./src/frontend/Dockerfile
target: frontend-production
args:
API_ORIGIN: "http://localhost:8071"
image: conversations:frontend-production
ports:
- "3000:3000"
+12
View File
@@ -0,0 +1,12 @@
name: conversations
services:
app-dev:
models:
llm:
endpoint_var: AI_BASE_URL
model_var: AI_MODEL
models:
llm:
model: ai/smollm2
+179
View File
@@ -0,0 +1,179 @@
name: conversations
services:
postgresql:
image: postgres:16
healthcheck:
test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
interval: 1s
timeout: 2s
retries: 300
env_file:
- env.d/development/postgresql
ports:
- "15432:5432"
redis:
image: redis:5
maildev:
image: maildev/maildev:latest
ports:
- "1081:1080"
minio:
user: ${DOCKER_USER:-1000}
image: minio/minio
environment:
- MINIO_ROOT_USER=conversations
- MINIO_ROOT_PASSWORD=password
ports:
- '9000:9000'
- '9001:9001'
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 1s
timeout: 20s
retries: 300
entrypoint: ""
command: minio server --console-address :9001 /data
volumes:
- ./data/media:/data
createbuckets:
image: minio/mc
depends_on:
minio:
condition: service_healthy
restart: true
entrypoint: >
sh -c "
/usr/bin/mc alias set conversations http://minio:9000 conversations password && \
/usr/bin/mc mb conversations/conversations-media-storage && \
/usr/bin/mc version enable conversations/conversations-media-storage && \
exit 0;"
app-dev:
build:
context: .
target: backend-development
args:
DOCKER_USER: ${DOCKER_USER:-1000}
user: ${DOCKER_USER:-1000}
image: conversations:backend-development
environment:
- PYLINTHOME=/app/.pylint.d
- DJANGO_CONFIGURATION=Development
env_file:
- env.d/development/common
- env.d/development/postgresql
extra_hosts:
- "host.docker.internal:host-gateway"
ports:
- "8071:8000"
volumes:
- ./src/backend:/app
- ./data/static:/data/static
depends_on:
postgresql:
condition: service_healthy
restart: true
maildev:
condition: service_started
redis:
condition: service_started
createbuckets:
condition: service_started
nginx:
image: nginx:1.25
ports:
- "8083:8083"
volumes:
- ./docker/files/etc/nginx/conf.d:/etc/nginx/conf.d:ro
depends_on:
app-dev:
condition: service_started
keycloak:
condition: service_healthy
restart: true
frontend-development:
user: "${DOCKER_USER:-1000}"
build:
context: .
dockerfile: ./src/frontend/Dockerfile
target: conversations-dev
args:
API_ORIGIN: "http://localhost:8071"
image: conversations:frontend-development
volumes:
- ./src/frontend:/home/frontend
- /home/frontend/node_modules
- /home/frontend/apps/conversations/node_modules
ports:
- "3000:3000"
crowdin:
image: crowdin/cli:3.16.0
volumes:
- ".:/app"
env_file:
- env.d/development/crowdin
user: "${DOCKER_USER:-1000}"
working_dir: /app
node:
image: node:22
user: "${DOCKER_USER:-1000}"
environment:
HOME: /tmp
volumes:
- ".:/app"
kc_postgresql:
image: postgres:17.5
healthcheck:
test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
interval: 1s
timeout: 2s
retries: 300
ports:
- "5433:5432"
env_file:
- env.d/development/kc_postgresql
keycloak:
image: quay.io/keycloak/keycloak:26.3
volumes:
- ./docker/auth/realm.json:/opt/keycloak/data/import/realm.json
command:
- start-dev
- --features=preview
- --import-realm
- --proxy=edge
- --hostname=http://localhost:8083
- --hostname-strict=false
- --health-enabled=true
- --metrics-enabled=true
- --http-enabled=true
healthcheck:
test: ['CMD-SHELL', '[ -f /tmp/HealthCheck.java ] || echo "public class HealthCheck { public static void main(String[] args) throws java.lang.Throwable { java.net.URI uri = java.net.URI.create(args[0]); System.exit(java.net.HttpURLConnection.HTTP_OK == ((java.net.HttpURLConnection)uri.toURL().openConnection()).getResponseCode() ? 0 : 1); } }" > /tmp/HealthCheck.java && java /tmp/HealthCheck.java http://localhost:9000/health/live']
start_period: 10s
interval: 1s
timeout: 2s
retries: 300
environment:
KC_BOOTSTRAP_ADMIN_USERNAME: admin
KC_BOOTSTRAP_ADMIN_PASSWORD: admin
KC_DB: postgres
KC_DB_URL_HOST: kc_postgresql
KC_DB_URL_DATABASE: keycloak
KC_DB_PASSWORD: pass
KC_DB_USERNAME: conversations
KC_DB_SCHEMA: public
PROXY_ADDRESS_FORWARDING: 'true'
depends_on:
kc_postgresql:
condition: service_healthy
restart: true
+29
View File
@@ -0,0 +1,29 @@
#
# Your crowdin's credentials
#
api_token_env: CROWDIN_PERSONAL_TOKEN
project_id_env: CROWDIN_PROJECT_ID
base_path_env: CROWDIN_BASE_PATH
#
# Choose file structure in crowdin
# e.g. true or false
#
preserve_hierarchy: true
#
# Files configuration
#
files: [
{
source : "/backend/locale/django.pot",
dest: "/backend-conversations.pot",
translation : "/backend/locale/%locale_with_underscore%/LC_MESSAGES/django.po"
},
{
source: "/frontend/packages/i18n/locales/conversations/translations-crowdin.json",
dest: "/frontend-conversations.json",
translation: "/frontend/packages/i18n/locales/conversations/%two_letters_code%/translations.json",
skip_untranslated_strings: true,
},
]
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,52 @@
server {
listen 8083;
server_name localhost;
charset utf-8;
# Proxy auth for media
location /media/ {
# Auth request configuration
auth_request /media-auth;
auth_request_set $authHeader $upstream_http_authorization;
auth_request_set $authDate $upstream_http_x_amz_date;
auth_request_set $authContentSha256 $upstream_http_x_amz_content_sha256;
# Pass specific headers from the auth response
proxy_set_header Authorization $authHeader;
proxy_set_header X-Amz-Date $authDate;
proxy_set_header X-Amz-Content-SHA256 $authContentSha256;
# Get resource from Minio
proxy_pass http://minio:9000/conversations-media-storage/;
proxy_set_header Host minio:9000;
add_header Content-Security-Policy "default-src 'none'" always;
}
location /media-auth {
proxy_pass http://app-dev:8000/api/v1.0/chats/media-auth/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Original-URL $request_uri;
# Prevent the body from being passed
proxy_pass_request_body off;
proxy_set_header Content-Length "";
proxy_set_header X-Original-Method $request_method;
}
location / {
proxy_pass http://keycloak:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# Increase proxy buffer size to allow keycloak to send large
# header responses when a user is created.
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
}
}
+35
View File
@@ -0,0 +1,35 @@
#!/bin/sh
#
# The container user (see USER in the Dockerfile) is an un-privileged user that
# does not exists and is not created during the build phase (see Dockerfile).
# Hence, we use this entrypoint to wrap commands that will be run in the
# container to create an entry for this user in the /etc/passwd file.
#
# The following environment variables may be passed to the container to
# customize running user account:
#
# * USER_NAME: container user name (default: default)
# * HOME : container user home directory (default: none)
#
# To pass environment variables, you can either use the -e option of the docker run command:
#
# docker run --rm -e USER_NAME=foo -e HOME='/home/foo' conversations:latest python manage.py migrate
#
# or define new variables in an environment file to use with docker or docker compose:
#
# # env.d/production
# USER_NAME=foo
# HOME=/home/foo
#
# docker run --rm --env-file env.d/production conversations:latest python manage.py migrate
#
echo "🐳(entrypoint) creating user running in the container..."
if ! whoami > /dev/null 2>&1; then
if [ -w /etc/passwd ]; then
echo "${USER_NAME:-default}:x:$(id -u):$(id -g):${USER_NAME:-default} user:${HOME}:/sbin/nologin" >> /etc/passwd
fi
fi
echo "🐳(entrypoint) running your command: ${*}"
exec "$@"
@@ -0,0 +1,16 @@
# Gunicorn-django settings
bind = ["0.0.0.0:8000"]
name = "conversations"
python_path = "/app"
# Run
graceful_timeout = 90
timeout = 90
workers = 3
# Logging
# Using '-' for the access log file makes gunicorn log accesses to stdout
accesslog = "-"
# Using '-' for the error log file makes gunicorn log errors to stderr
errorlog = "-"
loglevel = "info"
+14
View File
@@ -0,0 +1,14 @@
## Architecture
### Global system architecture
```mermaid
flowchart TD
User -- HTTP --> Front("Frontend (NextJS SPA)")
Front -- REST API --> Back("Backend (Django)")
Front -- OIDC --> Back -- OIDC ---> OIDC("Keycloak / ProConnect")
Back -- REST API --> Yserver
Back --> DB("Database (PostgreSQL)")
Back <--> Celery --> DB
Back ----> S3("Minio (S3)")
```
Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.4 MiB

After

Width:  |  Height:  |  Size: 39 KiB

+126
View File
@@ -0,0 +1,126 @@
# Conversations variables
Here we describe all environment variables that can be set for the conversations application.
## conversations-backend container
These are the environment variables you can set for the `conversations-backend` container.
| Option | Description | default |
|-------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| DJANGO_ALLOWED_HOSTS | allowed hosts | [] |
| DJANGO_SECRET_KEY | secret key | |
| DJANGO_SERVER_TO_SERVER_API_TOKENS | | [] |
| DB_ENGINE | engine to use for database connections | django.db.backends.postgresql_psycopg2 |
| DB_NAME | name of the database | conversations |
| DB_USER | user to authenticate with | dinum |
| DB_PASSWORD | password to authenticate with | pass |
| DB_HOST | host of the database | localhost |
| DB_PORT | port of the database | 5432 |
| MEDIA_BASE_URL | | |
| STORAGES_STATICFILES_BACKEND | | whitenoise.storage.CompressedManifestStaticFilesStorage |
| AWS_S3_ENDPOINT_URL | S3 endpoint | |
| AWS_S3_ACCESS_KEY_ID | access id for s3 endpoint | |
| AWS_S3_SECRET_ACCESS_KEY | access key for s3 endpoint | |
| AWS_S3_REGION_NAME | region name for s3 endpoint | |
| AWS_STORAGE_BUCKET_NAME | bucket name for s3 endpoint | conversations-media-storage |
| ATTACHMENT_MAX_SIZE | maximum size of document in bytes | 10485760 |
| LANGUAGE_CODE | default language | en-us |
| API_USERS_LIST_THROTTLE_RATE_SUSTAINED | throttle rate for api | 180/hour |
| API_USERS_LIST_THROTTLE_RATE_BURST | throttle rate for api on burst | 30/minute |
| SPECTACULAR_SETTINGS_ENABLE_DJANGO_DEPLOY_CHECK | | false |
| TRASHBIN_CUTOFF_DAYS | trashbin cutoff | 30 |
| DJANGO_EMAIL_BACKEND | email backend library | django.core.mail.backends.smtp.EmailBackend |
| DJANGO_EMAIL_BRAND_NAME | brand name for email | |
| DJANGO_EMAIL_HOST | host name of email | |
| DJANGO_EMAIL_HOST_USER | user to authenticate with on the email host | |
| DJANGO_EMAIL_HOST_PASSWORD | password to authenticate with on the email host | |
| DJANGO_EMAIL_LOGO_IMG | logo for the email | |
| DJANGO_EMAIL_PORT | port used to connect to email host | |
| DJANGO_EMAIL_USE_TLS | use tls for email host connection | false |
| DJANGO_EMAIL_USE_SSL | use sstl for email host connection | false |
| DJANGO_EMAIL_FROM | email address used as sender | from@example.com |
| DJANGO_CORS_ALLOW_ALL_ORIGINS | allow all CORS origins | false |
| DJANGO_CORS_ALLOWED_ORIGINS | list of origins allowed for CORS | [] |
| DJANGO_CORS_ALLOWED_ORIGIN_REGEXES | list of origins allowed for CORS using regulair expressions | [] |
| SENTRY_DSN | sentry host | |
| FRONTEND_CSS_URL | To add a external css file to the app | |
| FRONTEND_HOMEPAGE_FEATURE_ENABLED | frontend feature flag to display the homepage | false |
| FRONTEND_THEME | frontend theme to use | |
| POSTHOG_KEY | posthog key for analytics | |
| CRISP_WEBSITE_ID | crisp website id for support | |
| DJANGO_CELERY_BROKER_URL | celery broker url | redis://redis:6379/0 |
| DJANGO_CELERY_BROKER_TRANSPORT_OPTIONS | celery broker transport options | {} |
| SESSION_COOKIE_AGE | duration of the cookie session | 60*60*12 |
| OIDC_CREATE_USER | create used on OIDC | false |
| OIDC_RP_SIGN_ALGO | verification algorithm used OIDC tokens | RS256 |
| OIDC_RP_CLIENT_ID | client id used for OIDC | conversations |
| OIDC_RP_CLIENT_SECRET | client secret used for OIDC | |
| OIDC_OP_JWKS_ENDPOINT | JWKS endpoint for OIDC | |
| OIDC_OP_AUTHORIZATION_ENDPOINT | Authorization endpoint for OIDC | |
| OIDC_OP_TOKEN_ENDPOINT | Token endpoint for OIDC | |
| OIDC_OP_USER_ENDPOINT | User endpoint for OIDC | |
| OIDC_OP_LOGOUT_ENDPOINT | Logout endpoint for OIDC | |
| OIDC_AUTH_REQUEST_EXTRA_PARAMS | OIDC extra auth parameters | {} |
| OIDC_RP_SCOPES | scopes requested for OIDC | openid email |
| LOGIN_REDIRECT_URL | login redirect url | |
| LOGIN_REDIRECT_URL_FAILURE | login redirect url on failure | |
| LOGOUT_REDIRECT_URL | logout redirect url | |
| OIDC_USE_NONCE | use nonce for OIDC | true |
| OIDC_REDIRECT_REQUIRE_HTTPS | Require https for OIDC redirect url | false |
| OIDC_REDIRECT_ALLOWED_HOSTS | Allowed hosts for OIDC redirect url | [] |
| OIDC_STORE_ID_TOKEN | Store OIDC token | true |
| OIDC_FALLBACK_TO_EMAIL_FOR_IDENTIFICATION | faillback to email for identification | true |
| OIDC_ALLOW_DUPLICATE_EMAILS | Allow duplicate emails | false |
| USER_OIDC_ESSENTIAL_CLAIMS | essential claims in OIDC token | [] |
| OIDC_USERINFO_FULLNAME_FIELDS | OIDC token claims to create full name | ["first_name", "last_name"] |
| OIDC_USERINFO_SHORTNAME_FIELD | OIDC token claims to create shortname | first_name |
| ALLOW_LOGOUT_GET_METHOD | Allow get logout method | true |
| AI_API_KEY | AI key to be used for AI Base url | |
| AI_BASE_URL | OpenAI compatible AI base url | |
| AI_MODEL | AI Model to use | |
| AI_AGENT_INSTRUCTION | Base instruction for the AI agent | You are a helpful assistant |
| Y_PROVIDER_API_KEY | Y provider API key | |
| Y_PROVIDER_API_BASE_URL | Y Provider url | |
| CONVERSION_API_ENDPOINT | Conversion API endpoint | convert-markdown |
| CONVERSION_API_CONTENT_FIELD | Conversion api content field | content |
| CONVERSION_API_TIMEOUT | Conversion api timeout | 30 |
| CONVERSION_API_SECURE | Require secure conversion api | false |
| LOGGING_LEVEL_LOGGERS_ROOT | default logging level. options are "DEBUG", "INFO", "WARN", "ERROR", "CRITICAL" | INFO |
| LOGGING_LEVEL_LOGGERS_APP | application logging level. options are "DEBUG", "INFO", "WARN", "ERROR", "CRITICAL" | INFO |
| API_USERS_LIST_LIMIT | Limit on API users | 5 |
| DJANGO_CSRF_TRUSTED_ORIGINS | CSRF trusted origins | [] |
| REDIS_URL | cache url | redis://redis:6379/1 |
| CACHES_DEFAULT_TIMEOUT | cache default timeout | 30 |
| CACHES_KEY_PREFIX | The prefix used to every cache keys. | conversations |
| THEME_CUSTOMIZATION_FILE_PATH | full path to the file customizing the theme. An example is provided in src/backend/conversations/configuration/theme/default.json | BASE_DIR/conversations/configuration/theme/default.json |
| THEME_CUSTOMIZATION_CACHE_TIMEOUT | Cache duration for the customization settings | 86400 |
## conversations-frontend image
These are the environment variables you can set to build the `conversations-frontend` image.
Depending on how you are building the front-end application, this variable is used in different ways.
If you want to build the Docker image, this variable is used as an argument in the build command.
Example:
```
docker build -f src/frontend/Dockerfile --target frontend-production --build-arg API_ORIGIN=https://mybackend.example.com conversations-frontend:latest
```
If you want to build the front-end application using the yarn build command, you can edit the file `src/frontend/apps/conversations/.env` with the `NODE_ENV=production` environment variable and modify it. Alternatively, you can use the listed environment variables with the prefix `NEXT_PUBLIC_`.
Example:
```
cd src/frontend/apps/conversations
NODE_ENV=production NEXT_PUBLIC_API_ORIGIN=https://mybackend.example.com yarn build
```
| Option | Description | default |
|-------------------------------------------------|------------------------------------------------------------------------------------| ------------------------------------------------------- |
| API_ORIGIN | backend domain - it uses the current domain if not initialized | |
| PRODUCT_NAME | to change the default product name displayed in frontend | Conversations |
+134
View File
@@ -0,0 +1,134 @@
image:
repository: lasuite/conversations-backend
pullPolicy: Always
tag: "latest"
backend:
replicas: 1
envVars:
DJANGO_CSRF_TRUSTED_ORIGINS: https://conversations.127.0.0.1.nip.io
DJANGO_CONFIGURATION: Feature
DJANGO_ALLOWED_HOSTS: conversations.127.0.0.1.nip.io
DJANGO_SERVER_TO_SERVER_API_TOKENS: secret-api-key
DJANGO_SECRET_KEY: AgoodOrAbadKey
DJANGO_SETTINGS_MODULE: conversations.settings
DJANGO_SUPERUSER_PASSWORD: admin
DJANGO_EMAIL_BRAND_NAME: "La Suite Numérique"
DJANGO_EMAIL_HOST: "maildev"
DJANGO_EMAIL_LOGO_IMG: https://conversations.127.0.0.1.nip.io/assets/logo-suite-numerique.png
DJANGO_EMAIL_PORT: 1025
DJANGO_EMAIL_USE_SSL: False
LOGGING_LEVEL_HANDLERS_CONSOLE: ERROR
LOGGING_LEVEL_LOGGERS_ROOT: INFO
LOGGING_LEVEL_LOGGERS_APP: INFO
OIDC_OP_JWKS_ENDPOINT: https://keycloak.127.0.0.1.nip.io/realms/conversations/protocol/openid-connect/certs
OIDC_OP_AUTHORIZATION_ENDPOINT: https://keycloak.127.0.0.1.nip.io/realms/conversations/protocol/openid-connect/auth
OIDC_OP_TOKEN_ENDPOINT: https://keycloak.127.0.0.1.nip.io/realms/conversations/protocol/openid-connect/token
OIDC_OP_USER_ENDPOINT: https://keycloak.127.0.0.1.nip.io/realms/conversations/protocol/openid-connect/userinfo
OIDC_OP_LOGOUT_ENDPOINT: https://keycloak.127.0.0.1.nip.io/realms/conversations/protocol/openid-connect/session/end
OIDC_RP_CLIENT_ID: conversations
OIDC_RP_CLIENT_SECRET: ThisIsAnExampleKeyForDevPurposeOnly
OIDC_RP_SIGN_ALGO: RS256
OIDC_RP_SCOPES: "openid email"
OIDC_VERIFY_SSL: False
OIDC_USERINFO_SHORTNAME_FIELD: "given_name"
OIDC_USERINFO_FULLNAME_FIELDS: "given_name,usual_name"
OIDC_REDIRECT_ALLOWED_HOSTS: https://conversations.127.0.0.1.nip.io
OIDC_AUTH_REQUEST_EXTRA_PARAMS: "{'acr_values': 'eidas1'}"
LOGIN_REDIRECT_URL: https://conversations.127.0.0.1.nip.io
LOGIN_REDIRECT_URL_FAILURE: https://conversations.127.0.0.1.nip.io
LOGOUT_REDIRECT_URL: https://conversations.127.0.0.1.nip.io
POSTHOG_KEY: "{'id': 'posthog_key', 'host': 'https://product.conversations.127.0.0.1.nip.io'}"
DB_HOST: postgresql
DB_NAME: conversations
DB_USER: dinum
DB_PASSWORD: pass
DB_PORT: 5432
POSTGRES_DB: conversations
POSTGRES_USER: dinum
POSTGRES_PASSWORD: pass
REDIS_URL: redis://default:pass@redis-master:6379/1
AWS_S3_ENDPOINT_URL: http://minio.conversations.svc.cluster.local:9000
AWS_S3_ACCESS_KEY_ID: root
AWS_S3_SECRET_ACCESS_KEY: password
AWS_STORAGE_BUCKET_NAME: conversations-media-storage
STORAGES_STATICFILES_BACKEND: django.contrib.staticfiles.storage.StaticFilesStorage
migrate:
command:
- "/bin/sh"
- "-c"
- |
python manage.py migrate --no-input &&
python manage.py create_demo --force
restartPolicy: Never
command:
- "gunicorn"
- "-c"
- "/usr/local/etc/gunicorn/conversations.py"
- "conversations.wsgi:application"
- "--reload"
createsuperuser:
command:
- "/bin/sh"
- "-c"
- |
python manage.py createsuperuser --email admin@example.com --password admin
restartPolicy: Never
# Extra volume to manage our local custom CA and avoid to set ssl_verify: false
extraVolumeMounts:
- name: certs
mountPath: /usr/local/lib/python3.12/site-packages/certifi/cacert.pem
subPath: cacert.pem
# Extra volume to manage our local custom CA and avoid to set ssl_verify: false
extraVolumes:
- name: certs
configMap:
name: certifi
items:
- key: cacert.pem
path: cacert.pem
frontend:
envVars:
PORT: 8080
NEXT_PUBLIC_API_ORIGIN: https://conversations.127.0.0.1.nip.io
replicas: 1
image:
repository: lasuite/conversations-frontend
pullPolicy: Always
tag: "latest"
posthog:
ingress:
enabled: false
ingressAssets:
enabled: false
ingress:
enabled: true
host: conversations.127.0.0.1.nip.io
ingressAdmin:
enabled: true
host: conversations.127.0.0.1.nip.io
ingressMedia:
enabled: true
host: conversations.127.0.0.1.nip.io
annotations:
nginx.ingress.kubernetes.io/auth-url: https://conversations.127.0.0.1.nip.io/api/v1.0/chats/media-auth/
nginx.ingress.kubernetes.io/auth-response-headers: "Authorization, X-Amz-Date, X-Amz-Content-SHA256"
nginx.ingress.kubernetes.io/upstream-vhost: minio.conversations.svc.cluster.local:9000
nginx.ingress.kubernetes.io/rewrite-target: /conversations-media-storage/$1
serviceMedia:
host: minio.conversations.svc.cluster.local
port: 9000
File diff suppressed because it is too large Load Diff
+8
View File
@@ -0,0 +1,8 @@
auth:
rootUser: root
rootPassword: password
provisioning:
enabled: true
buckets:
- name: conversations-media-storage
versioning: true
+7
View File
@@ -0,0 +1,7 @@
auth:
username: dinum
password: pass
database: conversations
tls:
enabled: true
autoGenerated: true
+4
View File
@@ -0,0 +1,4 @@
auth:
password: pass
architecture: standalone
+228
View File
@@ -0,0 +1,228 @@
# Installation on a k8s cluster
This document is a step-by-step guide that describes how to install Conversations on a k8s cluster without AI features. It's a teaching document to learn how it works. It needs to be adapted for a production environment.
## Prerequisites
- k8s cluster with an nginx-ingress controller
- an OIDC provider (if you don't have one, we provide an example)
- a PostgreSQL server (if you don't have one, we provide an example)
- a Memcached server (if you don't have one, we provide an example)
- a S3 bucket (if you don't have one, we provide an example)
### Test cluster
If you do not have a test cluster, you can install everything on a local Kind cluster. In this case, the simplest way is to use our script **bin/start-kind.sh**.
To be able to use the script, you need to install:
- Docker (https://docs.docker.com/desktop/)
- Kind (https://kind.sigs.k8s.io/docs/user/quick-start/#installation)
- Mkcert (https://github.com/FiloSottile/mkcert#installation)
- Helm (https://helm.sh/docs/intro/quickstart/#install-helm)
```
./bin/start-kind.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4700 100 4700 0 0 92867 0 --:--:-- --:--:-- --:--:-- 94000
0. Create ca
The local CA is already installed in the system trust store! 👍
The local CA is already installed in the Firefox and/or Chrome/Chromium trust store! 👍
Created a new certificate valid for the following names 📜
- "127.0.0.1.nip.io"
- "*.127.0.0.1.nip.io"
Reminder: X.509 wildcards only go one level deep, so this won't match a.b.127.0.0.1.nip.io
The certificate is at "./127.0.0.1.nip.io+1.pem" and the key at "./127.0.0.1.nip.io+1-key.pem" ✅
It will expire on 24 March 2027 🗓
1. Create registry container unless it already exists
2. Create kind cluster with containerd registry config dir enabled
Creating cluster "suite" ...
✓ Ensuring node image (kindest/node:v1.27.3) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-suite"
You can now use your cluster with:
kubectl cluster-info --context kind-suite
Thanks for using kind! 😊
3. Add the registry config to the nodes
4. Connect the registry to the cluster network if not already connected
5. Document the local registry
configmap/local-registry-hosting created
Warning: resource configmaps/coredns is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
configmap/coredns configured
deployment.apps/coredns restarted
6. Install ingress-nginx
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
configmap/ingress-nginx-controller created
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
secret/mkcert created
deployment.apps/ingress-nginx-controller patched
7. Setup namespace
namespace/conversations created
Context "kind-suite" modified.
secret/mkcert created
$ kubectl -n ingress-nginx get po
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-t55ph 0/1 Completed 0 2m56s
ingress-nginx-admission-patch-94dvt 0/1 Completed 1 2m56s
ingress-nginx-controller-57c548c4cd-2rx47 1/1 Running 0 2m56s
```
When your k8s cluster is ready (the ingress nginx controller is up), you can start the deployment. This cluster is special because it uses the `*.127.0.0.1.nip.io` domain and mkcert certificates to have full HTTPS support and easy domain name management.
Please remember that `*.127.0.0.1.nip.io` will always resolve to `127.0.0.1`, except in the k8s cluster where we configure CoreDNS to answer with the ingress-nginx service IP.
## Preparation
### What do you use to authenticate your users?
Conversations uses OIDC, so if you already have an OIDC provider, obtain the necessary information to use it. In the next step, we will see how to configure Django (and thus Conversations) to use it. If you do not have a provider, we will show you how to deploy a local Keycloak instance (this is not a production deployment, just a demo).
```
$ kubectl create namespace conversations
$ kubectl config set-context --current --namespace=conversations
$ helm install keycloak oci://registry-1.docker.io/bitnamicharts/keycloak -f examples/keycloak.values.yaml
$ #wait until
$ kubectl get po
NAME READY STATUS RESTARTS AGE
keycloak-0 1/1 Running 0 6m48s
keycloak-postgresql-0 1/1 Running 0 6m48s
```
From here the important information you will need are:
```yaml
OIDC_OP_JWKS_ENDPOINT: https://keycloak.127.0.0.1.nip.io/realms/conversations/protocol/openid-connect/certs
OIDC_OP_AUTHORIZATION_ENDPOINT: https://keycloak.127.0.0.1.nip.io/realms/conversations/protocol/openid-connect/auth
OIDC_OP_TOKEN_ENDPOINT: https://keycloak.127.0.0.1.nip.io/realms/conversations/protocol/openid-connect/token
OIDC_OP_USER_ENDPOINT: https://keycloak.127.0.0.1.nip.io/realms/conversations/protocol/openid-connect/userinfo
OIDC_OP_LOGOUT_ENDPOINT: https://keycloak.127.0.0.1.nip.io/realms/conversations/protocol/openid-connect/session/end
OIDC_RP_CLIENT_ID: conversations
OIDC_RP_CLIENT_SECRET: ThisIsAnExampleKeyForDevPurposeOnly
OIDC_RP_SIGN_ALGO: RS256
OIDC_RP_SCOPES: "openid email"
```
You can find these values in **examples/keycloak.values.yaml**
### Find redis server connection values
Conversations needs a redis so we start by deploying one:
```
$ helm install redis oci://registry-1.docker.io/bitnamicharts/redis -f examples/redis.values.yaml
$ kubectl get po
NAME READY STATUS RESTARTS AGE
keycloak-0 1/1 Running 0 26m
keycloak-postgresql-0 1/1 Running 0 26m
redis-master-0 1/1 Running 0 35s
```
### Find postgresql connection values
Conversations uses a postgresql database as backend, so if you have a provider, obtain the necessary information to use it. If you don't, you can install a postgresql testing environment as follow:
```
$ helm install postgresql oci://registry-1.docker.io/bitnamicharts/postgresql -f examples/postgresql.values.yaml
$ kubectl get po
NAME READY STATUS RESTARTS AGE
keycloak-0 1/1 Running 0 28m
keycloak-postgresql-0 1/1 Running 0 28m
postgresql-0 1/1 Running 0 14m
redis-master-0 1/1 Running 0 42s
```
From here the important information you will need are:
```yaml
DB_HOST: postgres-postgresql
DB_NAME: conversations
DB_USER: dinum
DB_PASSWORD: pass
DB_PORT: 5432
POSTGRES_DB: conversations
POSTGRES_USER: dinum
POSTGRES_PASSWORD: pass
```
### Find s3 bucket connection values
Conversations uses an s3 bucket to store documents, so if you have a provider obtain the necessary information to use it. If you don't, you can install a local minio testing environment as follow:
```
$ helm install minio oci://registry-1.docker.io/bitnamicharts/minio -f examples/minio.values.yaml
$ kubectl get po
NAME READY STATUS RESTARTS AGE
keycloak-0 1/1 Running 0 38m
keycloak-postgresql-0 1/1 Running 0 38m
minio-84f5c66895-bbhsk 1/1 Running 0 42s
minio-provisioning-2b5sq 0/1 Completed 0 42s
postgresql-0 1/1 Running 0 24m
redis-master-0 1/1 Running 0 10m
```
## Deployment
Now you are ready to deploy Conversations without AI. AI requires more dependencies (OpenAI API). To deploy Conversations you need to provide all previous information to the helm chart.
```
$ helm repo add conversations https://suitenumerique.github.io/conversations/
$ helm repo update
$ helm install conversations conversations/conversations -f examples/conversations.values.yaml
$ kubectl get po
NAME READY STATUS RESTARTS AGE
conversations-conversations-backend-96558758d-xtkbp 0/1 Running 0 79s
conversations-conversations-backend-createsuperuser-r7ltc 0/1 Completed 0 79s
conversations-conversations-backend-migrate-c949s 0/1 Completed 0 79s
conversations-conversations-frontend-6749f644f7-p5s42 1/1 Running 0 79s
keycloak-0 1/1 Running 0 48m
keycloak-postgresql-0 1/1 Running 0 48m
minio-84f5c66895-bbhsk 1/1 Running 0 10m
minio-provisioning-2b5sq 0/1 Completed 0 10m
postgresql-0 1/1 Running 0 34m
redis-master-0 1/1 Running 0 20m
```
## Test your deployment
In order to test your deployment you have to log into your instance. If you exclusively use our examples you can do:
```
$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
conversations-conversations <none> conversations.127.0.0.1.nip.io localhost 80, 443 114s
conversations-conversations-admin <none> conversations.127.0.0.1.nip.io localhost 80, 443 114s
conversations-conversations-media <none> conversations.127.0.0.1.nip.io localhost 80, 443 114s
conversations-conversations-ws <none> conversations.127.0.0.1.nip.io localhost 80, 443 114s
keycloak <none> keycloak.127.0.0.1.nip.io localhost 80 49m
```
You can use Conversations at https://conversations.127.0.0.1.nip.io. The provisionning user in keycloak is conversations/conversations.
+66
View File
@@ -0,0 +1,66 @@
# Releasing a new version
Whenever we are cooking a new release (e.g. `4.18.1`) we should follow a standard procedure described below:
1. Create a new branch named: `release/4.18.1`.
2. Bump the release number for backend project, frontend projects, and Helm files:
- for backend, update the version number by hand in `pyproject.toml`,
- for each projects (`src/frontend`, `src/frontend/apps/*`, `src/frontend/packages/*`, `src/mail`), run `yarn version --new-version --no-git-tag-version 4.18.1` in their directory. This will update their `package.json` for you,
- for Helm, update Docker image tag in files located at `src/helm/env.d` for both `preprod` and `production` environments:
```yaml
image:
repository: lasuite/conversations-backend
pullPolicy: Always
tag: "v4.18.1" # Replace with your new version number, without forgetting the "v" prefix
...
frontend:
image:
repository: lasuite/conversations-frontend
pullPolicy: Always
tag: "v4.18.1"
```
The new images don't exist _yet_: they will be created automatically later in the process.
3. Update the project's `Changelog` following the [keepachangelog](https://keepachangelog.com/en/0.3.0/) recommendations
4. Commit your changes with the following format: the 🔖 release emoji, the type of release (patch/minor/patch) and the release version:
```text
🔖(minor) bump release to 4.18.0
```
5. Open a pull request, wait for an approval from your peers and merge it.
6. Checkout and pull changes from the `main` branch to ensure you have the latest updates.
7. Tag and push your commit:
```bash
git tag v4.18.1 && git push origin tag v4.18.1
```
Doing this triggers the CI and tells it to build the new Docker image versions that you targeted earlier in the Helm files.
8. Ensure the new [backend](https://hub.docker.com/r/lasuite/conversations-frontend/tags) and [frontend](https://hub.docker.com/r/lasuite/conversations-frontend/tags) image tags are on Docker Hub.
9. The release is now done!
# Deploying
> [!TIP]
> The `staging` platform is deployed automatically with every update of the `main` branch.
Making a new release doesn't publish it automatically in production.
Deployment is done by ArgoCD. ArgoCD checks for the `production` tag and automatically deploys the production platform with the targeted commit.
To publish, we mark the commit we want with the `production` tag. ArgoCD is then notified that the tag has changed. It then deploys the Docker image tags specified in the Helm files of the targeted commit.
To publish the release you just made:
```bash
git tag --force production v4.18.1
git push --force origin production
```
+105
View File
@@ -0,0 +1,105 @@
# La Suite Conversations System & Requirements (2025-06)
## 1. Quick-Reference Matrix (single VM / laptop)
| Scenario | RAM | vCPU | SSD | Notes |
|---------------------------|-------|------|---------|---------------------------|
| **Solo dev** | 8 GB | 4 | 15 GB | Hot-reload + one IDE |
| **Team QA** | 16 GB | 6 | 30 GB | Runs integration tests |
| **Prod ≤ 100 live users** | 32 GB | 8 + | 50 GB + | Scale linearly above this |
Memory is the first bottleneck; CPU matters only when Celery or the Next.js build is saturated.
> **Note:** Memory consumption varies by operating system. Windows tends to be more memory-hungry than Linux, so consider adding 10-20% extra RAM when running on Windows compared to Linux-based systems.
## 2. Development Environment Memory Requirements
| Service | Typical use | Rationale / source |
|-----------------------|-------------------------------|-----------------------------------------------------------------------------------------|
| PostgreSQL | **1 2 GB** | `shared_buffers` starting point ≈ 25% RAM ([postgresql.org][1]) |
| Keycloak | **≈ 1.3 GB** | 70% of limit for heap + ~300 MB non-heap ([keycloak.org][2]) |
| Redis | **≤ 256 MB** | Empty instance ≈ 3 MB; budget 256 MB to allow small datasets ([stackoverflow.com][3]) |
| MinIO | **2 GB (dev) / 32 GB (prod)** | Pre-allocates 12 GiB; docs recommend 32 GB per host for ≤ 100 Ti storage ([min.io][4]) |
| Django API (+ Celery) | **0.8 1.5 GB** | Empirical in-house metrics |
| Next.js frontend | **0.5 1 GB** | Dev build chain |
| Nginx | **< 100 MB** | Static reverse-proxy footprint |
[1]: https://www.postgresql.org/docs/9.1/runtime-config-resource.html "PostgreSQL: Documentation: 9.1: Resource Consumption"
[2]: https://www.keycloak.org/high-availability/concepts-memory-and-cpu-sizing "Concepts for sizing CPU and memory resources - Keycloak"
[3]: https://stackoverflow.com/questions/45233052/memory-footprint-for-redis-empty-instance "Memory footprint for Redis empty instance - Stack Overflow"
[4]: https://min.io/docs/minio/kubernetes/upstream/operations/checklists/hardware.html "Hardware Checklist — MinIO Object Storage for Kubernetes"
> **Rule of thumb:** add 2 GB for OS/overhead, then sum only the rows you actually run.
## 3. Production Environment Memory Requirements
Production deployments differ significantly from development environments. The table below shows typical memory usage for production services:
| Service | Typical use | Rationale / notes |
|---------------------------|-----------------------------|------------------------------------------------------------------------------------------|
| PostgreSQL | **2 8 GB** | Higher `shared_buffers` and connection pooling for concurrent users |
| OIDC Provider (optional) | **Variable** | Any OIDC-compatible provider (Keycloak, Auth0, Azure AD, etc.) - external or self-hosted |
| Redis | **256 MB 2 GB** | Session storage and caching; scales with active user sessions |
| Object Storage (optional) | **External or self-hosted** | Can use AWS S3, Azure Blob, Google Cloud Storage, or self-hosted MinIO |
| Django API (+ Celery) | **1 3 GB** | Production workloads with background tasks and higher concurrency |
| Static Files (Nginx) | **< 200 MB** | Serves Next.js build output and static assets; no development overhead |
| Nginx (Load Balancer) | **< 200 MB** | Reverse proxy, SSL termination, static file serving |
### Production Architecture Notes
- **Frontend**: Uses pre-built Next.js static assets served by Nginx (no Node.js runtime needed)
- **Authentication**: Any OIDC-compatible provider can be used instead of self-hosted Keycloak
- **Object Storage**: External services (S3, Azure Blob) or self-hosted solutions (MinIO) are both viable
- **Database**: Consider PostgreSQL clustering or managed database services for high availability
- **Scaling**: Horizontal scaling is recommended for Django API service
### Minimal Production Setup (Core Services Only)
| Service | Memory | Notes |
|----------------------------------|------------|----------------------------------------|
| PostgreSQL | **2 GB** | Core database |
| Django API (+ Celery) | **1.5 GB** | Backend services |
| Nginx | **100 MB** | Static files + reverse proxy |
| Redis | **256 MB** | Session storage |
| **Total (without auth/storage)** | **≈ 4 GB** | External OIDC + object storage assumed |
## 4. Recommended Software Versions
| Tool | Minimum |
|-------------------------|---------|
| Docker Engine / Desktop | 24.0 |
| Docker Compose | v2 |
| Git | 2.40 |
| **Node.js** | 22+ |
| **Python** | 3.13+ |
| GNU Make | 4.4 |
| Kind | 0.22 |
| Helm | 3.14 |
| kubectl | 1.29 |
| mkcert | 1.4 |
## 5. Ports (dev defaults)
| Port | Service |
|-----------|-----------------------|
| 3000 | Next.js |
| 8071 | Django |
| 8080 | Keycloak |
| 8083 | Nginx proxy |
| 9000/9001 | MinIO |
| 15432 | PostgreSQL (main) |
| 5433 | PostgreSQL (Keycloak) |
| 1081 | Maildev |
## 6. Sizing Guidelines
**RAM** start at 8 GB dev / 16 GB staging / 32 GB prod. Postgres and Keycloak are the first to OOM; scale them first.
> **OS considerations:** Windows systems typically require 10-20% more RAM than Linux due to higher OS overhead. Docker Desktop on Windows also uses additional memory compared to native Linux Docker.
**CPU** budget one vCPU per busy container until Celery or Next.js builds saturate.
**Disk** SSD; add 10 GB extra for the Docker layer cache.
**MinIO** for demos, mount a local folder instead of running MinIO to save 2 GB+ of RAM.
+64
View File
@@ -0,0 +1,64 @@
# Runtime Theming 🎨
### How to Use
To use this feature, simply set the `FRONTEND_CSS_URL` environment variable to the URL of your custom CSS file. For example:
```javascript
FRONTEND_CSS_URL=http://anything/custom-style.css
```
Once you've set this variable, our application will load your custom CSS file and apply the styles to our frontend application.
### Benefits
This feature provides several benefits, including:
* **Easy customization** 🔄: With this feature, you can easily customize the look and feel of our application without requiring any code changes.
* **Flexibility** 🌈: You can use any CSS styles you like to create a custom theme that meets your needs.
* **Runtime theming** ⏱️: This feature allows you to change the theme of our application at runtime, without requiring a restart or recompilation.
### Example Use Case
Let's say you want to change the background color of our application to a custom color. You can create a custom CSS file with the following contents:
```css
body {
background-color: #3498db;
}
```
Then, set the `FRONTEND_CSS_URL` environment variable to the URL of your custom CSS file. Once you've done this, our application will load your custom CSS file and apply the styles, changing the background color to the custom color you specified.
----
# **Footer Configuration** 📝
The footer is configurable from the theme customization file.
### Settings 🔧
```shellscript
THEME_CUSTOMIZATION_FILE_PATH=<path>
```
### Example of JSON
The json must follow some rules: https://github.com/suitenumerique/conversations/blob/main/src/helm/env.d/dev/configuration/theme/demo.json
`footer.default` is the fallback if the language is not supported.
# **Custom Translations** 📝
The translations can be partially overridden from the theme customization file.
### Settings 🔧
```shellscript
THEME_CUSTOMIZATION_FILE_PATH=<path>
```
### Example of JSON
The json must follow some rules: https://github.com/suitenumerique/conversations/blob/main/src/helm/env.d/dev/configuration/theme/demo.json
+192
View File
@@ -0,0 +1,192 @@
# Troubleshooting Guide
## Line Ending Issues on Windows (LF/CRLF)
### Problem Description
This project uses **LF (Line Feed: `\n`) line endings** exclusively. Windows users may encounter issues because:
- **Windows** defaults to CRLF (Carriage Return + Line Feed: `\r\n`) for line endings
- **This project** uses LF line endings for consistency across all platforms
- **Git** may automatically convert line endings, causing conflicts or build failures
### Common Symptoms
- Git shows files as modified even when no changes were made
- Error messages like "warning: LF will be replaced by CRLF"
- Build failures or linting errors due to line ending mismatches
### Solutions for Windows Users
#### Configure Git to Preserve LF (Recommended)
Configure Git to NOT convert line endings and preserve LF:
```bash
git config core.autocrlf false
git config core.eol lf
```
This tells Git to:
- Never convert line endings automatically
- Always use LF for line endings in working directory
#### Fix Existing Repository with Wrong Line Endings
If you already have CRLF line endings in your local repository, the **best approach** is to configure Git properly and clone the project again:
1. **Configure Git first**:
```bash
git config --global core.autocrlf false
git config --global core.eol lf
```
2. **Clone the project fresh** (recommended):
```bash
# Navigate to parent directory
cd ..
# Remove current repository (backup your changes first!)
rm -rf docs
# Clone again with correct line endings
git clone git@github.com:suitenumerique/conversations.git
```
**Alternative**: If you have uncommitted changes and cannot re-clone:
1. **Backup your changes**:
```bash
git add .
git commit -m "Save changes before fixing line endings"
```
2. **Remove all files from Git's index**:
```bash
git rm --cached -r .
```
3. **Reset Git configuration** (if not done globally):
```bash
git config core.autocrlf false
git config core.eol lf
```
4. **Re-add all files** (Git will use LF line endings):
```bash
git add .
```
5. **Commit the changes**:
```bash
git commit -m "✏️(project) Fix line endings to LF"
```
## Minio Permission Issues on Windows
### Problem Description
On Windows, you may encounter permission-related errors when running Minio in development mode with Docker Compose. This typically happens because:
- **Windows file permissions** don't map well to Unix-style user IDs used in Docker containers
- **Docker Desktop** may have issues with user mapping when using the `DOCKER_USER` environment variable
- **Minio container** fails to start or access volumes due to permission conflicts
### Common Symptoms
- Minio container fails to start with permission denied errors
- Error messages related to file system permissions in Minio logs
- Unable to create or access buckets in the development environment
- Docker Compose showing Minio service as unhealthy or exited
### Solution for Windows Users
If you encounter Minio permission issues on Windows, you can temporarily disable user mapping for the Minio service:
1. **Open the `compose.yml` file**
2. **Comment out the user directive** in the `minio` service section:
```yaml
minio:
# user: ${DOCKER_USER:-1000} # Comment this line on Windows if permission issues occur
image: minio/minio
environment:
- MINIO_ROOT_USER=conversations
- MINIO_ROOT_PASSWORD=password
# ... rest of the configuration
```
3. **Restart the services**:
```bash
make run
```
### Why This Works
- Commenting out the `user` directive allows the Minio container to run with its default user
- This bypasses Windows-specific permission mapping issues
- The container will have the necessary permissions to access and manage the mounted volumes
### Note
This is a **development-only workaround**. In production environments, proper user mapping and security considerations should be maintained according to your deployment requirements.
## Frontend File Watching Issues on Windows
### Problem Description
Windows users may experience issues with file watching in the frontend-development container. This typically happens because:
- **Docker on Windows** has known limitations with file change detection
- **Node.js file watchers** may not detect changes properly on Windows filesystem
- **Hot reloading** fails to trigger when files are modified
### Common Symptoms
- Changes to frontend code aren't detected automatically
- Hot module replacement doesn't work as expected
- Need to manually restart the frontend container after code changes
- Console shows no reaction when saving files
### Solution: Enable WATCHPACK_POLLING
Add the `WATCHPACK_POLLING=true` environment variable to the frontend-development service in your local environment:
1. **Modify the `compose.yml` file** by adding the environment variable to the frontend-development service:
```yaml
frontend-development:
user: "${DOCKER_USER:-1000}"
build:
context: .
dockerfile: ./src/frontend/Dockerfile
target: conversations-dev
args:
API_ORIGIN: "http://localhost:8071"
image: conversations:frontend-development
environment:
- WATCHPACK_POLLING=true # Add this line for Windows users
volumes:
- ./src/frontend:/home/frontend
- /home/frontend/node_modules
- /home/frontend/apps/conversations/node_modules
ports:
- "3000:3000"
```
2. **Restart your containers**:
```bash
make run
```
### Why This Works
- `WATCHPACK_POLLING=true` forces the file watcher to use polling instead of filesystem events
- Polling periodically checks for file changes rather than relying on OS-level file events
- This is more reliable on Windows but slightly increases CPU usage
- Changes to your frontend code should now be detected properly, enabling hot reloading
### Note
This setting is primarily needed for Windows users. Linux and macOS users typically don't need this setting as file watching works correctly by default on those platforms.
+55
View File
@@ -0,0 +1,55 @@
# Django
DJANGO_ALLOWED_HOSTS=*
DJANGO_SECRET_KEY=ThisIsAnExampleKeyForDevPurposeOnly
DJANGO_SETTINGS_MODULE=conversations.settings
DJANGO_SUPERUSER_PASSWORD=admin
# Logging
# Set to DEBUG level for dev only
LOGGING_LEVEL_HANDLERS_CONSOLE=INFO
LOGGING_LEVEL_LOGGERS_ROOT=INFO
LOGGING_LEVEL_LOGGERS_APP=INFO
# Python
PYTHONPATH=/app
# conversations settings
# Mail
DJANGO_EMAIL_BRAND_NAME="La Suite Numérique"
DJANGO_EMAIL_HOST="maildev"
DJANGO_EMAIL_LOGO_IMG="http://localhost:3000/assets/logo-suite-numerique.png"
DJANGO_EMAIL_PORT=1025
# Backend url
CONVERSATIONS_BASE_URL="http://localhost:8072"
# Media
STORAGES_STATICFILES_BACKEND=django.contrib.staticfiles.storage.StaticFilesStorage
AWS_S3_ENDPOINT_URL=http://minio:9000
AWS_S3_ACCESS_KEY_ID=conversations
AWS_S3_SECRET_ACCESS_KEY=password
MEDIA_BASE_URL=http://localhost:8083
# OIDC
OIDC_OP_JWKS_ENDPOINT=http://nginx:8083/realms/conversations/protocol/openid-connect/certs
OIDC_OP_AUTHORIZATION_ENDPOINT=http://localhost:8083/realms/conversations/protocol/openid-connect/auth
OIDC_OP_TOKEN_ENDPOINT=http://nginx:8083/realms/conversations/protocol/openid-connect/token
OIDC_OP_USER_ENDPOINT=http://nginx:8083/realms/conversations/protocol/openid-connect/userinfo
OIDC_RP_CLIENT_ID=conversations
OIDC_RP_CLIENT_SECRET=ThisIsAnExampleKeyForDevPurposeOnly
OIDC_RP_SIGN_ALGO=RS256
OIDC_RP_SCOPES="openid email"
LOGIN_REDIRECT_URL=http://localhost:3000
LOGIN_REDIRECT_URL_FAILURE=http://localhost:3000
LOGOUT_REDIRECT_URL=http://localhost:3000
OIDC_REDIRECT_ALLOWED_HOSTS=["http://localhost:8083", "http://localhost:3000"]
OIDC_AUTH_REQUEST_EXTRA_PARAMS={"acr_values": "eidas1"}
# AI
# AI_BASE_URL=https://openaiendpoint.com
AI_API_KEY=password
# AI_MODEL=llama
+8
View File
@@ -0,0 +1,8 @@
# For the CI job test-e2e
BURST_THROTTLE_RATES="200/minute"
DJANGO_SERVER_TO_SERVER_API_TOKENS=test-e2e
SUSTAINED_THROTTLE_RATES="200/hour"
# Features
FEATURE_FLAG_WEB_SEARCH=ENABLED
FEATURE_FLAG_DOCUMENT_UPLOAD=ENABLED
+3
View File
@@ -0,0 +1,3 @@
CROWDIN_PERSONAL_TOKEN=Your-Personal-Token
CROWDIN_PROJECT_ID=Your-Project-Id
CROWDIN_BASE_PATH=/app/src
+11
View File
@@ -0,0 +1,11 @@
# Postgresql db container configuration
POSTGRES_DB=keycloak
POSTGRES_USER=conversations
POSTGRES_PASSWORD=pass
# App database configuration
DB_HOST=kc_postgresql
DB_NAME=keycloak
DB_USER=conversations
DB_PASSWORD=pass
DB_PORT=5433
+11
View File
@@ -0,0 +1,11 @@
# Postgresql db container configuration
POSTGRES_DB=conversations
POSTGRES_USER=dinum
POSTGRES_PASSWORD=pass
# App database configuration
DB_HOST=postgresql
DB_NAME=conversations
DB_USER=dinum
DB_PASSWORD=pass
DB_PORT=5432
+37
View File
@@ -0,0 +1,37 @@
"""
Gitlint extra rule to validate that the message title is of the form
"<gitmoji>(<scope>) <subject>"
"""
from __future__ import unicode_literals
import re
import requests
from gitlint.rules import CommitMessageTitle, LineRule, RuleViolation
class GitmojiTitle(LineRule):
"""
This rule will enforce that each commit title is of the form "<gitmoji>(<scope>) <subject>"
where gitmoji is an emoji from the list defined in https://gitmoji.carloscuesta.me and
subject should be all lowercase
"""
id = "UC1"
name = "title-should-have-gitmoji-and-scope"
target = CommitMessageTitle
def validate(self, title, _commit):
"""
Download the list possible gitmojis from the project's github repository and check that
title contains one of them.
"""
gitmojis = requests.get(
"https://raw.githubusercontent.com/carloscuesta/gitmoji/master/packages/gitmojis/src/gitmojis.json"
).json()["gitmojis"]
emojis = [item["emoji"] for item in gitmojis]
pattern = r"^({:s})\(.*\)\s[a-zA-Z].*$".format("|".join(emojis))
if not re.search(pattern, title):
violation_msg = 'Title does not match regex "<gitmoji>(<scope>) <subject>"'
return [RuleViolation(self.id, violation_msg, title)]
+6
View File
@@ -0,0 +1,6 @@
{
"dependencies": {
"@ai-sdk/react": "^1.2.12",
"@ai-sdk/ui-utils": "^1.2.11"
}
}
+55
View File
@@ -0,0 +1,55 @@
{
"extends": ["github>numerique-gouv/renovate-configuration"],
"dependencyDashboard": true,
"labels": ["dependencies", "noChangeLog", "automated"],
"packageRules": [
{
"enabled": false,
"groupName": "ignored python dependencies",
"matchManagers": ["pep621"],
"matchPackageNames": []
},
{
"groupName": "allowed redis versions",
"matchManagers": ["pep621"],
"matchPackageNames": ["redis"],
"allowedVersions": "<6.0.0"
},
{
"groupName": "ignore recent markitdown versions",
"matchManagers": ["pep621"],
"matchPackageNames": ["markitdown"],
"allowedVersions": "==0.0.2"
},
{
"groupName": "ignore recent lxml versions not supported by htmldate==1.9.3",
"matchManagers": ["pep621"],
"matchPackageNames": ["lxml"],
"allowedVersions": "<6"
},
{
"groupName": "ignore recent pylint versions not supported by pylint-django",
"matchManagers": ["pep621"],
"matchPackageNames": ["pylint"],
"allowedVersions": "<4"
},
{
"enabled": false,
"groupName": "ignored js dependencies",
"matchManagers": ["npm"],
"matchPackageNames": [
"eslint",
"fetch-mock",
"node",
"node-fetch",
"workbox-webpack-plugin"
]
},
{
"groupName": "ignore Vercel SDK >= 5.0.0",
"matchManagers": ["npm"],
"matchPackageNames": ["@ai-sdk/react", "@ai-sdk/ui-utils"],
"allowedVersions": "^1.2.0"
}
]
}
+474
View File
@@ -0,0 +1,474 @@
[MASTER]
# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code
extension-pkg-whitelist=
# Add files or directories to the blacklist. They should be base names, not
# paths.
ignore=migrations
# Add files or directories matching the regex patterns to the blacklist. The
# regex matches against base names, not paths.
ignore-patterns=
# Python code to execute, usually for sys.path manipulation such as
# pygtk.require().
#init-hook=
# Use multiple processes to speed up Pylint. Specifying 0 will auto-detect the
# number of processors available to use.
jobs=0
# List of plugins (as comma separated values of python modules names) to load,
# usually to register additional checkers.
load-plugins=pylint_django,
pylint.extensions.no_self_use,
pylint_pydantic,
# Pickle collected data for later comparisons.
persistent=yes
# Specify a configuration file.
#rcfile=
# When enabled, pylint would attempt to guess common misconfiguration and emit
# user-friendly hints instead of false-positive error messages
suggestion-mode=yes
# Allow loading of arbitrary C extensions. Extensions are imported into the
# active Python interpreter and may run arbitrary code.
unsafe-load-any-extension=no
[MESSAGES CONTROL]
# Only show warnings with the listed confidence levels. Leave empty to show
# all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED
confidence=
# Disable the message, report, category or checker with the given id(s). You
# can either give multiple identifiers separated by comma (,) or put this
# option multiple times (only on the command line, not in the configuration
# file where it should appear only once).You can also use "--disable=all" to
# disable everything first and then reenable specific checks. For example, if
# you want to run only the similarities checker, you can use "--disable=all
# --enable=similarities". If you want to run only the classes checker, but have
# no Warning level messages displayed, use"--disable=all --enable=classes
# --disable=W"
disable=bad-inline-option,
deprecated-pragma,
django-not-configured,
file-ignored,
locally-disabled,
no-self-use,
raw-checker-failed,
suppressed-message,
useless-suppression
# Enable the message, report, category or checker with the given id(s). You can
# either give multiple identifier separated by comma (,) or put this option
# multiple time (only on the command line, not in the configuration file where
# it should appear only once). See also the "--disable" option for examples.
enable=c-extension-no-member
[REPORTS]
# Python expression which should return a note less than 10 (10 is the highest
# note). You have access to the variables errors warning, statement which
# respectively contain the number of errors / warnings messages and the total
# number of statements analyzed. This is used by the global evaluation report
# (RP0004).
evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10)
# Template used to display messages. This is a python new-style format string
# used to format the message information. See doc for all details
#msg-template=
# Set the output format. Available formats are text, parseable, colorized, json
# and msvs (visual studio).You can also give a reporter class, eg
# mypackage.mymodule.MyReporterClass.
output-format=text
# Tells whether to display a full report or only the messages
reports=no
# Activate the evaluation score.
score=yes
[REFACTORING]
# Maximum number of nested blocks for function / method body
max-nested-blocks=5
# Complete name of functions that never returns. When checking for
# inconsistent-return-statements if a never returning function is called then
# it will be considered as an explicit return statement and no message will be
# printed.
never-returning-functions=optparse.Values,sys.exit
[LOGGING]
# Logging modules to check that the string format arguments are in logging
# function parameter format
logging-modules=logging
[SPELLING]
# Limits count of emitted suggestions for spelling mistakes
max-spelling-suggestions=4
# Spelling dictionary name. Available dictionaries: none. To make it working
# install python-enchant package.
spelling-dict=
# List of comma separated words that should not be checked.
spelling-ignore-words=
# A path to a file that contains private dictionary; one word per line.
spelling-private-dict-file=
# Tells whether to store unknown words to indicated private dictionary in
# --spelling-private-dict-file option instead of raising a message.
spelling-store-unknown-words=no
[MISCELLANEOUS]
# List of note tags to take in consideration, separated by a comma.
notes=FIXME,
XXX,
TODO
[TYPECHECK]
# List of decorators that produce context managers, such as
# contextlib.contextmanager. Add to this list to register other decorators that
# produce valid context managers.
contextmanager-decorators=contextlib.contextmanager
# List of members which are set dynamically and missed by pylint inference
# system, and so shouldn't trigger E1101 when accessed. Python regular
# expressions are accepted.
generated-members=
# Tells whether missing members accessed in mixin class should be ignored. A
# mixin class is detected if its name ends with "mixin" (case insensitive).
ignore-mixin-members=yes
# This flag controls whether pylint should warn about no-member and similar
# checks whenever an opaque object is returned when inferring. The inference
# can return multiple potential results while evaluating a Python object, but
# some branches might not be evaluated, which results in partial inference. In
# that case, it might be useful to still emit no-member and other checks for
# the rest of the inferred objects.
ignore-on-opaque-inference=yes
# List of class names for which member attributes should not be checked (useful
# for classes with dynamically set attributes). This supports the use of
# qualified names.
ignored-classes=optparse.Values,thread._local,_thread._local,responses,
Template,Contact
# List of module names for which member attributes should not be checked
# (useful for modules/projects where namespaces are manipulated during runtime
# and thus existing member attributes cannot be deduced by static analysis. It
# supports qualified module names, as well as Unix pattern matching.
ignored-modules=
# Show a hint with possible names when a member name was not found. The aspect
# of finding the hint is based on edit distance.
missing-member-hint=yes
# The minimum edit distance a name should have in order to be considered a
# similar match for a missing member name.
missing-member-hint-distance=1
# The total number of similar names that should be taken in consideration when
# showing a hint for a missing member.
missing-member-max-choices=1
[VARIABLES]
# List of additional names supposed to be defined in builtins. Remember that
# you should avoid to define new builtins when possible.
additional-builtins=
# Tells whether unused global variables should be treated as a violation.
allow-global-unused-variables=yes
# List of strings which can identify a callback function by name. A callback
# name must start or end with one of those strings.
callbacks=cb_,
_cb
# A regular expression matching the name of dummy variables (i.e. expectedly
# not used).
dummy-variables-rgx=_+$|(_[a-zA-Z0-9_]*[a-zA-Z0-9]+?$)|dummy|^ignored_|^unused_
# Argument names that match this expression will be ignored. Default to name
# with leading underscore
ignored-argument-names=_.*|^ignored_|^unused_
# Tells whether we should check for unused import in __init__ files.
init-import=no
# List of qualified module names which can have objects that can redefine
# builtins.
redefining-builtins-modules=six.moves,past.builtins,future.builtins
[FORMAT]
# Expected format of line ending, e.g. empty (any line ending), LF or CRLF.
expected-line-ending-format=
# Regexp for a line that is allowed to be longer than the limit.
ignore-long-lines=^\s*(# )?<?https?://\S+>?$
# Number of spaces of indent required inside a hanging or continued line.
indent-after-paren=4
# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1
# tab).
indent-string=' '
# Maximum number of characters on a single line.
max-line-length=100
# Maximum number of lines in a module
max-module-lines=1000
# Allow the body of a class to be on the same line as the declaration if body
# contains single statement.
single-line-class-stmt=no
# Allow the body of an if to be on the same line as the test if there is no
# else.
single-line-if-stmt=no
[SIMILARITIES]
# Ignore comments when computing similarities.
ignore-comments=yes
# Ignore docstrings when computing similarities.
ignore-docstrings=yes
# Ignore imports when computing similarities.
ignore-imports=yes
# Minimum lines number of a similarity.
# First implementations of CMS wizards have common fields we do not want to factorize for now
min-similarity-lines=35
[BASIC]
# Naming style matching correct argument names
argument-naming-style=snake_case
# Regular expression matching correct argument names. Overrides argument-
# naming-style
#argument-rgx=
# Naming style matching correct attribute names
attr-naming-style=snake_case
# Regular expression matching correct attribute names. Overrides attr-naming-
# style
#attr-rgx=
# Bad variable names which should always be refused, separated by a comma
bad-names=foo,
bar,
baz,
toto,
tutu,
tata
# Naming style matching correct class attribute names
class-attribute-naming-style=any
# Regular expression matching correct class attribute names. Overrides class-
# attribute-naming-style
#class-attribute-rgx=
# Naming style matching correct class names
class-naming-style=PascalCase
# Regular expression matching correct class names. Overrides class-naming-style
#class-rgx=
# Naming style matching correct constant names
const-naming-style=UPPER_CASE
# Regular expression matching correct constant names. Overrides const-naming-
# style
const-rgx=(([A-Z_][A-Z0-9_]*)|(__.*__)|urlpatterns|logger)$
# Minimum line length for functions/classes that require docstrings, shorter
# ones are exempt.
docstring-min-length=-1
# Naming style matching correct function names
function-naming-style=snake_case
# Regular expression matching correct function names. Overrides function-
# naming-style
#function-rgx=
# Good variable names which should always be accepted, separated by a comma
good-names=i,
j,
k,
cm,
ex,
Run,
_
# Include a hint for the correct naming format with invalid-name
include-naming-hint=no
# Naming style matching correct inline iteration names
inlinevar-naming-style=any
# Regular expression matching correct inline iteration names. Overrides
# inlinevar-naming-style
#inlinevar-rgx=
# Naming style matching correct method names
method-naming-style=snake_case
# Regular expression matching correct method names. Overrides method-naming-
# style
method-rgx=([a-z_][a-z0-9_]{2,50}|setUp|set[Uu]pClass|tearDown|tear[Dd]ownClass|assert[A-Z]\w*|maxDiff|test_[a-z0-9_]+)$
# Naming style matching correct module names
module-naming-style=snake_case
# Regular expression matching correct module names. Overrides module-naming-
# style
#module-rgx=
# Colon-delimited sets of names that determine each other's naming style when
# the name regexes allow several styles.
name-group=
# Regular expression which should only match function or class names that do
# not require a docstring.
no-docstring-rgx=^_
# List of decorators that produce properties, such as abc.abstractproperty. Add
# to this list to register other decorators that produce valid properties.
property-classes=abc.abstractproperty
# Naming style matching correct variable names
variable-naming-style=snake_case
# Regular expression matching correct variable names. Overrides variable-
# naming-style
#variable-rgx=
[IMPORTS]
# Allow wildcard imports from modules that define __all__.
allow-wildcard-with-all=no
# Analyse import fallback blocks. This can be used to support both Python 2 and
# 3 compatible code, which means that the block might have code that exists
# only in one or another interpreter, leading to false positives when analysed.
analyse-fallback-blocks=no
# Deprecated modules which should not be used, separated by a comma
deprecated-modules=optparse,tkinter.tix
# Create a graph of external dependencies in the given file (report RP0402 must
# not be disabled)
ext-import-graph=
# Create a graph of every (i.e. internal and external) dependencies in the
# given file (report RP0402 must not be disabled)
import-graph=
# Create a graph of internal dependencies in the given file (report RP0402 must
# not be disabled)
int-import-graph=
# Force import order to recognize a module as part of the standard
# compatibility libraries.
known-standard-library=
# Force import order to recognize a module as part of a third party library.
known-third-party=enchant
[CLASSES]
# List of method names used to declare (i.e. assign) instance attributes.
defining-attr-methods=__init__,
__new__,
setUp
# List of member names, which should be excluded from the protected access
# warning.
exclude-protected=_asdict,
_fields,
_replace,
_source,
_make
# List of valid names for the first argument in a class method.
valid-classmethod-first-arg=cls
# List of valid names for the first argument in a metaclass class method.
valid-metaclass-classmethod-first-arg=mcs
[DESIGN]
# Maximum number of arguments for function / method
max-args=5
# Maximum number of attributes for a class (see R0902).
max-attributes=7
# Maximum number of boolean expressions in a if statement
max-bool-expr=5
# Maximum number of branch for function / method body
max-branches=12
# Maximum number of locals for function / method body
max-locals=20
# Maximum number of parents for a class (see R0901).
max-parents=10
# Maximum number of public methods for a class (see R0904).
max-public-methods=20
# Maximum number of return / yield for function / method body
max-returns=6
# Maximum number of statements in function / method body
max-statements=50
# Minimum number of public methods for a class (see R0903).
min-public-methods=0
[EXCEPTIONS]
# Exceptions that will emit a warning when being caught. Defaults to
# "Exception"
overgeneral-exceptions=builtins.Exception
+3
View File
@@ -0,0 +1,3 @@
include LICENSE
include README.md
recursive-include src/backend/conversations *.html *.png *.gif *.css *.ico *.jpg *.jpeg *.po *.mo *.eot *.svg *.ttf *.woff *.woff2
View File
+349
View File
@@ -0,0 +1,349 @@
"""Admin classes for activation codes application."""
from django.conf import settings
from django.contrib import admin
from django.utils.html import format_html, format_html_join
from django.utils.translation import gettext_lazy as _
from . import models
@admin.register(models.ActivationCode)
class ActivationCodeAdmin(admin.ModelAdmin):
"""Admin class for ActivationCode model"""
list_display = (
"code",
"usage_display",
"is_active",
"expires_at",
"created_at",
"description_short",
)
list_filter = (
"is_active",
"created_at",
"expires_at",
)
search_fields = (
"code",
"description",
)
readonly_fields = (
"id",
"current_uses",
"created_at",
"updated_at",
"usage_details",
)
fieldsets = (
(
None,
{
"fields": (
"id",
"code",
"description",
)
},
),
(
_("Configuration"),
{
"fields": (
"max_uses",
"current_uses",
"is_active",
"expires_at",
)
},
),
(
_("Usage details"),
{"fields": ("usage_details",)},
),
(
_("Timestamps"),
{
"fields": (
"created_at",
"updated_at",
)
},
),
)
actions = ["recompute_current_uses"]
def get_readonly_fields(self, request, obj=None):
"""Make `code` readonly when editing an existing ActivationCode.
When obj is None (creation form), `code` remains editable. When obj is
provided (editing), add `code` to readonly fields so it cannot be
changed after creation.
"""
# Start from the configured readonly_fields to preserve other read-only fields
ro_fields = list(self.readonly_fields)
if obj is not None:
ro_fields.append("code")
return tuple(ro_fields)
def usage_display(self, obj):
"""Display usage statistics."""
max_uses = obj.max_uses if obj.max_uses > 0 else ""
if obj.current_uses >= obj.max_uses and obj.max_uses > 0:
color = "red"
elif obj.current_uses > 0:
color = "orange"
else:
color = "green"
return format_html(
'<span style="color: {};">{} / {}</span>', color, obj.current_uses, max_uses
)
usage_display.short_description = _("Usage")
def description_short(self, obj):
"""Display truncated description."""
if obj.description:
return obj.description[:50] + "..." if len(obj.description) > 50 else obj.description
return "-"
description_short.short_description = _("Description")
def usage_details(self, obj):
"""Display detailed usage information."""
usages = obj.usages.select_related("user").all()
if not usages:
return _("No users have used this code yet")
table_head = format_html(
(
"<table style='width: 100%; border-collapse: collapse;'>"
"<tr style='background-color: #f0f0f0;'>"
"<th style='padding: 8px; text-align: left;'>{name}</th>"
"<th style='padding: 8px; text-align: left;'>{title}</th>"
"<th style='padding: 8px; text-align: left;'>{date}</th>"
"</tr>"
),
name=_("Name"),
title=_("Email"),
date=_("Date"),
)
rows = format_html_join(
"",
(
"<tr style='border-bottom: 1px solid #ddd;'>"
"<td style='padding: 8px;'>{name}</td>"
"<td style='padding: 8px;'>{email}</td>"
"<td style='padding: 8px;'>{created_at}</td>"
"</tr>"
),
(
{
"name": usage.user.full_name or "-",
"email": usage.user.email or "-",
"created_at": usage.created_at.strftime("%Y-%m-%d %H:%M"),
}
for usage in usages
),
)
return format_html("{table_head}{rows}</table>", table_head=table_head, rows=rows)
usage_details.short_description = _("Users who used this code")
@admin.action(description=_("Recompute current uses from related activations"))
def recompute_current_uses(self, request, queryset):
"""Recompute the current_uses field by counting related UserActivation objects."""
updated_count = 0
for activation_code in queryset:
actual_uses = activation_code.usages.count()
if activation_code.current_uses != actual_uses:
activation_code.current_uses = actual_uses
activation_code.save(update_fields=["current_uses", "updated_at"])
updated_count += 1
if updated_count == 0:
self.message_user(
request,
_("All selected activation codes already have correct usage counts."),
)
else:
self.message_user(
request,
_("Successfully recomputed usage counts for %(count)d activation code(s).")
% {"count": updated_count},
)
@admin.register(models.UserActivation)
class UserActivationAdmin(admin.ModelAdmin):
"""Admin class for UserActivation model"""
list_display = (
"user_display",
"user_email",
"activation_code",
"created_at",
)
list_filter = ("created_at",)
search_fields = (
"user__email",
"user__full_name",
"activation_code__code",
)
readonly_fields = (
"id",
"user",
"activation_code",
"created_at",
"updated_at",
)
fieldsets = (
(
None,
{
"fields": (
"id",
"user",
"activation_code",
)
},
),
(
_("Timestamps"),
{
"fields": (
"created_at",
"updated_at",
)
},
),
)
def user_display(self, obj):
"""Display user's full name."""
return obj.user.full_name or str(obj.user.id)
user_display.short_description = _("User")
def user_email(self, obj):
"""Display user's email."""
return obj.user.email or "-"
user_email.short_description = _("Email")
def has_add_permission(self, request):
"""Disable manual creation of user activations."""
return False
@admin.register(models.UserRegistrationRequest)
class UserRegistrationRequestAdmin(admin.ModelAdmin):
"""Admin class for UserRegistrationRequest model"""
list_display = (
"user_display",
"created_at",
"has_user_activation",
)
readonly_fields = (
"id",
"user",
"created_at",
"updated_at",
"user_activation",
)
search_fields = (
"user__email",
"user__full_name",
)
list_filter = ("created_at",)
actions = ["add_to_brevo_waiting_list", "remove_from_brevo_waiting_list"]
def user_display(self, obj):
"""Display user's full name."""
return obj.user.email or str(obj.user.pk)
user_display.short_description = _("User")
def has_user_activation(self, obj):
"""Indicate if the user has used an activation code."""
return obj.user_activation_id is not None
has_user_activation.boolean = True
has_user_activation.short_description = _("Has used activation code")
@admin.action(description=_("Add selected users to Brevo waiting list"))
def add_to_brevo_waiting_list(self, request, queryset):
"""Add selected users to Brevo waiting list."""
# pylint: disable=import-outside-toplevel
from core.brevo import add_user_to_brevo_list # noqa: PLC0415
registration_to_send = queryset.filter(
user_activation__isnull=True,
)
_total_emails = 0
for i in range(0, registration_to_send.count(), 150):
batch = registration_to_send[i : i + 150]
emails = [reg.user.email for reg in batch if reg.user.email]
if emails:
add_user_to_brevo_list(emails, settings.BREVO_WAITING_LIST_ID)
_total_emails += len(emails)
if _total_emails:
self.message_user(
request,
_("Added %(count)d user(s) to Brevo waiting list.") % {"count": _total_emails},
)
else:
self.message_user(
request,
_("No valid email address found in selected registrations."),
level="warning",
)
@admin.action(description=_("Remove selected users from Brevo waiting list"))
def remove_from_brevo_waiting_list(self, request, queryset):
"""Remove selected users from Brevo waiting list."""
# pylint: disable=import-outside-toplevel
from core.brevo import remove_user_from_brevo_list # noqa: PLC0415
registration_to_send = queryset.filter(
user_activation__isnull=False,
)
_total_emails = 0
for i in range(0, registration_to_send.count(), 150):
batch = registration_to_send[i : i + 150]
emails = [reg.user.email for reg in batch if reg.user.email]
if emails:
remove_user_from_brevo_list(emails, settings.BREVO_WAITING_LIST_ID)
_total_emails += len(emails)
if _total_emails:
self.message_user(
request,
_("Removed %(count)d user(s) from Brevo waiting list.") % {"count": _total_emails},
)
else:
self.message_user(
request,
_("No valid email address found in selected registrations."),
level="warning",
)
@@ -0,0 +1,9 @@
"""Exceptions for activation code handling."""
class InvalidCodeError(ValueError):
"""Raised when an activation code is invalid or cannot be used."""
class UserAlreadyActivatedError(ValueError):
"""Raised when a user tries to activate but is already activated."""
+29
View File
@@ -0,0 +1,29 @@
"""Factories for creating activation code and user activation instances for testing."""
from django.utils import timezone
import factory.django
from core.factories import UserFactory
from . import models
class ActivationCodeFactory(factory.django.DjangoModelFactory):
"""A factory to create activation codes for testing purposes."""
class Meta:
model = models.ActivationCode
code = factory.LazyAttribute(lambda x: models.generate_activation_code())
created_at = factory.LazyAttribute(lambda obj: timezone.now())
class UserActivationFactory(factory.django.DjangoModelFactory):
"""A factory to create user activations for testing purposes."""
class Meta:
model = models.UserActivation
user = factory.SubFactory(UserFactory)
activation_code = factory.SubFactory(ActivationCodeFactory)
@@ -0,0 +1,234 @@
# Generated by Django 5.2.7 on 2025-10-09 08:30
import uuid
import django.db.models.deletion
from django.conf import settings
from django.db import migrations, models
import activation_codes.models
class Migration(migrations.Migration):
initial = True
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.CreateModel(
name="ActivationCode",
fields=[
(
"id",
models.UUIDField(
default=uuid.uuid4,
editable=False,
help_text="primary key for the record as UUID",
primary_key=True,
serialize=False,
verbose_name="id",
),
),
(
"created_at",
models.DateTimeField(
auto_now_add=True,
help_text="date and time at which a record was created",
verbose_name="created on",
),
),
(
"updated_at",
models.DateTimeField(
auto_now=True,
help_text="date and time at which a record was last updated",
verbose_name="updated on",
),
),
(
"code",
models.CharField(
default=activation_codes.models.generate_activation_code,
help_text="The activation code that users will enter",
max_length=50,
unique=True,
validators=[
django.core.validators.RegexValidator(
message="Code must be alphanumeric and contain no spaces or special characters",
regex="^[A-Z0-9]+$",
)
],
verbose_name="activation code",
),
),
(
"max_uses",
models.PositiveIntegerField(
default=1,
help_text="Maximum number of times this code can be used. 0 means unlimited.",
verbose_name="maximum uses",
),
),
(
"current_uses",
models.PositiveIntegerField(
default=0,
editable=False,
help_text="Number of times this code has been used",
verbose_name="current uses",
),
),
(
"is_active",
models.BooleanField(
default=True,
help_text="Whether this code can still be used",
verbose_name="active",
),
),
(
"expires_at",
models.DateTimeField(
blank=True,
help_text="Date and time when this code expires",
null=True,
verbose_name="expires at",
),
),
(
"description",
models.TextField(
blank=True,
help_text="Internal description or notes about this code",
verbose_name="description",
),
),
],
options={
"verbose_name": "activation code",
"verbose_name_plural": "activation codes",
"db_table": "activation_code",
"ordering": ["-created_at"],
},
),
migrations.CreateModel(
name="UserActivation",
fields=[
(
"id",
models.UUIDField(
default=uuid.uuid4,
editable=False,
help_text="primary key for the record as UUID",
primary_key=True,
serialize=False,
verbose_name="id",
),
),
(
"created_at",
models.DateTimeField(
auto_now_add=True,
help_text="date and time at which a record was created",
verbose_name="created on",
),
),
(
"updated_at",
models.DateTimeField(
auto_now=True,
help_text="date and time at which a record was last updated",
verbose_name="updated on",
),
),
(
"activation_code",
models.ForeignKey(
help_text="The activation code that was used",
on_delete=django.db.models.deletion.PROTECT,
related_name="usages",
to="activation_codes.activationcode",
verbose_name="activation code",
),
),
(
"user",
models.OneToOneField(
help_text="The user who used the activation code",
on_delete=django.db.models.deletion.CASCADE,
related_name="activation",
to=settings.AUTH_USER_MODEL,
verbose_name="user",
),
),
],
options={
"verbose_name": "user activation",
"verbose_name_plural": "user activations",
"db_table": "user_activation",
"ordering": ["-created_at"],
},
),
migrations.CreateModel(
name="UserRegistrationRequest",
fields=[
(
"id",
models.UUIDField(
default=uuid.uuid4,
editable=False,
help_text="primary key for the record as UUID",
primary_key=True,
serialize=False,
verbose_name="id",
),
),
(
"created_at",
models.DateTimeField(
auto_now_add=True,
help_text="date and time at which a record was created",
verbose_name="created on",
),
),
(
"updated_at",
models.DateTimeField(
auto_now=True,
help_text="date and time at which a record was last updated",
verbose_name="updated on",
),
),
(
"user",
models.OneToOneField(
help_text="The user who made the registration request",
on_delete=django.db.models.deletion.CASCADE,
related_name="registration_request",
to=settings.AUTH_USER_MODEL,
verbose_name="user",
),
),
(
"user_activation",
models.OneToOneField(
blank=True,
help_text="Store if the user received an activation code and used it",
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="registration_request",
to="activation_codes.useractivation",
verbose_name="user activation",
),
),
],
options={
"verbose_name": "user registration request",
"verbose_name_plural": "user registration requests",
"db_table": "user_registration_request",
"ordering": ["-created_at"],
},
),
]
+226
View File
@@ -0,0 +1,226 @@
"""
Models for the activation codes application
"""
import logging
import secrets
import string
from django.conf import settings
from django.core.exceptions import ValidationError
from django.core.validators import RegexValidator
from django.db import IntegrityError, models, transaction
from django.utils import timezone
from django.utils.translation import gettext_lazy as _
from core.brevo import add_user_to_brevo_list, remove_user_from_brevo_list
from core.models import BaseModel, User
from activation_codes.exceptions import InvalidCodeError, UserAlreadyActivatedError
logger = logging.getLogger(__name__)
def generate_activation_code():
"""Generate a random 16-character activation code."""
alphabet = string.ascii_uppercase + string.digits
# Remove ambiguous characters
alphabet = alphabet.replace("O", "").replace("0", "").replace("I", "").replace("1", "")
return "".join(secrets.choice(alphabet) for _ in range(16))
class ActivationCode(BaseModel):
"""
Represents an activation code that can be used to activate user accounts.
"""
code = models.CharField(
verbose_name=_("activation code"),
help_text=_("The activation code that users will enter"),
max_length=50,
unique=True,
default=generate_activation_code,
validators=[
RegexValidator(
regex=r"^[A-Z0-9]+$",
message=_("Code must be alphanumeric and contain no spaces or special characters"),
)
],
)
max_uses = models.PositiveIntegerField(
verbose_name=_("maximum uses"),
help_text=_("Maximum number of times this code can be used. 0 means unlimited."),
default=1,
)
current_uses = models.PositiveIntegerField(
verbose_name=_("current uses"),
help_text=_("Number of times this code has been used"),
default=0,
editable=False,
)
is_active = models.BooleanField(
verbose_name=_("active"),
help_text=_("Whether this code can still be used"),
default=True,
)
expires_at = models.DateTimeField(
verbose_name=_("expires at"),
help_text=_("Date and time when this code expires"),
null=True,
blank=True,
)
description = models.TextField(
verbose_name=_("description"),
help_text=_("Internal description or notes about this code"),
blank=True,
)
class Meta:
db_table = "activation_code"
verbose_name = _("activation code")
verbose_name_plural = _("activation codes")
ordering = ["-created_at"]
def __str__(self):
"""Return string representation of the activation code."""
return f"{self.code} ({self.current_uses}/{self.max_uses if self.max_uses > 0 else ''})"
def is_valid(self):
"""Check if the code is still valid and can be used."""
if not self.is_active:
return False
if self.expires_at and self.expires_at < timezone.now():
return False
if self.max_uses > 0 and self.current_uses >= self.max_uses:
return False
return True
def can_be_used(self):
"""Alias for is_valid() for better readability."""
return self.is_valid()
def use(self, user):
"""
Mark this code as used by a user.
Args:
user: The User instance using this code
Returns:
UserActivation instance
Raises:
ValidationError: If the code cannot be used
"""
with transaction.atomic():
# Lock the activation code row to prevent concurrent overuse.
locked_code = ActivationCode.objects.select_for_update().get(pk=self.pk)
if not locked_code.is_valid():
raise InvalidCodeError(_("This activation code is no longer valid"))
# Create activation record; rely on DB uniqueness for concurrent duplicate attempts.
try:
activation = UserActivation.objects.create(user=user, activation_code=locked_code)
except (IntegrityError, ValidationError) as exc:
# User already has an activation in a concurrent or prior transaction.
raise UserAlreadyActivatedError(
_("You have already activated your account")
) from exc
existing_registration = bool(
UserRegistrationRequest.objects.filter(user=user).update(user_activation=activation)
)
if existing_registration:
transaction.on_commit(
lambda: remove_user_from_brevo_list(
[user.email], settings.BREVO_WAITING_LIST_ID
)
)
# Increment usage counter safely under the same lock.
locked_code.current_uses += 1
locked_code.save(update_fields=["current_uses", "updated_at"])
transaction.on_commit(
lambda: add_user_to_brevo_list([user.email], settings.BREVO_FOLLOWUP_LIST_ID)
)
if locked_code.max_uses > 0 and locked_code.current_uses >= locked_code.max_uses:
logger.warning("Activation code %s has reached its maximum uses", locked_code.code)
return activation
class UserActivation(BaseModel):
"""
Records with user used which activation code and when.
"""
user = models.OneToOneField(
User,
verbose_name=_("user"),
help_text=_("The user who used the activation code"),
on_delete=models.CASCADE,
related_name="activation",
)
activation_code = models.ForeignKey(
ActivationCode,
verbose_name=_("activation code"),
help_text=_("The activation code that was used"),
on_delete=models.PROTECT,
related_name="usages",
)
class Meta:
db_table = "user_activation"
verbose_name = _("user activation")
verbose_name_plural = _("user activations")
ordering = ["-created_at"]
def __str__(self):
"""Return string representation of the user activation."""
return f"{self.user} - {self.activation_code.code}"
class UserRegistrationRequest(BaseModel):
"""
Records of user registration requests.
"""
user = models.OneToOneField(
User,
verbose_name=_("user"),
help_text=_("The user who made the registration request"),
on_delete=models.CASCADE,
related_name="registration_request",
)
user_activation = models.OneToOneField(
UserActivation,
verbose_name=_("user activation"),
help_text=_("Store if the user received an activation code and used it"),
on_delete=models.SET_NULL,
related_name="registration_request",
null=True,
blank=True,
)
class Meta:
db_table = "user_registration_request"
verbose_name = _("user registration request")
verbose_name_plural = _("user registration requests")
ordering = ["-created_at"]
def __str__(self):
"""Return string representation of the user registration request."""
return f"Registration request by {self.user}"
@@ -0,0 +1,39 @@
"""Permission classes for activation codes."""
from django.conf import settings
from rest_framework import permissions
from . import models
class IsActivatedUser(permissions.BasePermission):
"""
Permission class that checks if user has activated their account.
This permission is only enforced if ACTIVATION_REQUIRED is True in settings.
Staff users and users without authentication requirement are always allowed.
"""
message = "activation-required" # Custom message to indicate activation is required to frontend
def has_permission(self, request, view):
"""Check if user has activated their account."""
# If activation is not required, allow access
if not settings.ACTIVATION_REQUIRED:
return True
# Staff users can always access
if request.user and request.user.is_staff:
return True
# Anonymous users are handled by other permission classes
if not request.user or not request.user.is_authenticated:
return True
# Check if user has an activation record
return models.UserActivation.objects.filter(user=request.user).exists()
def has_object_permission(self, request, view, obj):
"""Check object-level permission."""
return self.has_permission(request, view)
@@ -0,0 +1,50 @@
"""Serializers for the activation codes application."""
from django.utils.translation import gettext_lazy as _
from rest_framework import serializers
from . import models
class ActivationCodeValidationSerializer(serializers.Serializer): # pylint: disable=abstract-method
"""Serializer for validating an activation code."""
code = serializers.CharField(
max_length=50, required=True, help_text=_("The activation code to validate")
)
def validate_code(self, value):
"""Validate that the code exists and is valid."""
# Normalize the code (remove spaces, convert to uppercase)
return value.strip().upper().replace(" ", "").replace("-", "")
class UserActivationSerializer(serializers.ModelSerializer):
"""Serializer for user activation records."""
code = serializers.CharField(source="activation_code.code", read_only=True)
activated_at = serializers.DateTimeField(source="created_at", read_only=True)
class Meta:
model = models.UserActivation
fields = ["id", "code", "activated_at"]
read_only_fields = ["id", "code", "activated_at"]
class ActivationStatusSerializer(serializers.Serializer): # pylint: disable=abstract-method
"""Serializer for activation status response."""
is_activated = serializers.BooleanField(read_only=True)
activation = UserActivationSerializer(read_only=True, allow_null=True)
requires_activation = serializers.BooleanField(read_only=True)
class UserRegistrationRequestSerializer(serializers.ModelSerializer):
"""Serializer for registering a user for activation notifications."""
user = serializers.HiddenField(default=serializers.CurrentUserDefault())
class Meta:
model = models.UserRegistrationRequest
fields = ["user"]
@@ -0,0 +1,236 @@
"""Integration tests for activation_codes application."""
from datetime import timedelta
from django.utils import timezone
import pytest
from rest_framework import status
from core.factories import UserFactory
from activation_codes.factories import ActivationCodeFactory
from activation_codes.models import ActivationCode, UserActivation
@pytest.mark.django_db
def test_complete_activation_flow(api_client, settings):
"""Test complete user activation flow from registration to usage."""
settings.ACTIVATION_REQUIRED = True
# Create a user (simulating registration)
user = UserFactory(email="newuser@example.com", password="password123")
# Create an activation code (simulating admin creating codes)
activation_code = ActivationCode.objects.create(
code="WELCOME123456789", max_uses=10, description="Welcome batch for new users"
)
# User logs in
api_client.force_authenticate(user=user)
# Step 1: Check activation status (should not be activated)
response = api_client.get("/api/v1.0/activation/status/")
assert response.status_code == status.HTTP_200_OK
assert response.data["is_activated"] is False
assert response.data["requires_activation"] is True
# Step 2: User enters activation code
response = api_client.post("/api/v1.0/activation/validate/", {"code": "WELCOME123456789"})
assert response.status_code == status.HTTP_201_CREATED
assert "successfully activated" in response.data["detail"]
# Step 3: Check activation status again (should now be activated)
response = api_client.get("/api/v1.0/activation/status/")
assert response.status_code == status.HTTP_200_OK
assert response.data["is_activated"] is True
assert response.data["activation"]["code"] == "WELCOME123456789"
# Step 4: Verify in database
assert UserActivation.objects.filter(user=user).exists()
activation_code.refresh_from_db()
assert activation_code.current_uses == 1
@pytest.mark.django_db
def test_activation_not_required_flow(api_client, settings):
"""Test that when activation is not required, users can access without codes."""
settings.ACTIVATION_REQUIRED = False
user = UserFactory(email="freeuser@example.com", password="password123")
api_client.force_authenticate(user=user)
# Check status
response = api_client.get("/api/v1.0/activation/status/")
assert response.status_code == status.HTTP_200_OK
assert response.data["requires_activation"] is False
assert response.data["is_activated"] is False # Not activated but not required
@pytest.mark.django_db
def test_multiple_users_same_code(api_client, settings):
"""Test multiple users using the same multi-use code."""
settings.ACTIVATION_REQUIRED = True
# Create a multi-use code
code = ActivationCode.objects.create(
code="TEAMCODE12345678", max_uses=3, description="Team activation code"
)
# Create 3 users
users = []
for i in range(3):
user = UserFactory(email=f"teamuser{i}@example.com", password="password123")
users.append(user)
# Each user activates
for i, user in enumerate(users):
api_client.force_authenticate(user=user)
response = api_client.post("/api/v1.0/activation/validate/", {"code": "TEAMCODE12345678"})
assert response.status_code == status.HTTP_201_CREATED
code.refresh_from_db()
assert code.current_uses == i + 1
# Code should now be exhausted
code.refresh_from_db()
assert code.is_valid() is False
# Try with a 4th user (should fail)
user4 = UserFactory(email="teamuser4@example.com", password="password123")
api_client.force_authenticate(user=user4)
response = api_client.post("/api/v1.0/activation/validate/", {"code": "TEAMCODE12345678"})
assert response.status_code == status.HTTP_400_BAD_REQUEST
@pytest.mark.django_db
def test_code_expiration_scenario(api_client, settings):
"""Test code expiration over time."""
settings.ACTIVATION_REQUIRED = True
# Create a code that expires in 1 day
future_time = timezone.now() + timedelta(days=1)
_code = ActivationCode.objects.create(code="EXPIRES123456789", expires_at=future_time)
user = UserFactory(email="timeduser@example.com", password="password123")
api_client.force_authenticate(user=user)
# Should work now
response = api_client.post("/api/v1.0/activation/validate/", {"code": "EXPIRES123456789"})
assert response.status_code == status.HTTP_201_CREATED
@pytest.mark.django_db
def test_staff_user_bypass(api_client, settings):
"""Test that staff users bypass activation requirement."""
settings.ACTIVATION_REQUIRED = True
staff_user = UserFactory(email="staff@example.com", password="password123", is_staff=True)
api_client.force_authenticate(user=staff_user)
# Staff should be able to check status even without activation
response = api_client.get("/api/v1.0/activation/status/")
assert response.status_code == status.HTTP_200_OK
@pytest.mark.django_db
def test_user_cannot_activate_twice(api_client, settings):
"""Test that a user cannot activate their account twice."""
settings.ACTIVATION_REQUIRED = True
user = UserFactory(email="onceuser@example.com", password="password123")
_code1 = ActivationCodeFactory(code="FIRST12345678901")
_code2 = ActivationCodeFactory(code="SECOND1234567890")
api_client.force_authenticate(user=user)
# First activation
response = api_client.post("/api/v1.0/activation/validate/", {"code": "FIRST12345678901"})
assert response.status_code == status.HTTP_201_CREATED
# Try second activation
response = api_client.post("/api/v1.0/activation/validate/", {"code": "SECOND1234567890"})
assert response.status_code == status.HTTP_400_BAD_REQUEST
assert response.data == {"code": "account-already-activated"}
# Verify only one activation exists
assert UserActivation.objects.filter(user=user).count() == 1
@pytest.mark.django_db
def test_code_variations_normalized(api_client, settings):
"""Test that different code input formats are normalized correctly."""
settings.ACTIVATION_REQUIRED = True
code = ActivationCodeFactory(code="TESTCODE12345678")
test_cases = [
"testcode12345678", # lowercase
"TESTCODE12345678", # uppercase
"test code 1234 5678", # with spaces
"TEST-CODE-1234-5678", # with dashes
" test-code 1234-5678 ", # mixed with leading/trailing spaces
]
for i, code_variation in enumerate(test_cases):
user = UserFactory(email=f"varuser{i}@example.com", password="password123")
# Update code to allow multiple uses
code.max_uses = 0
code.save()
api_client.force_authenticate(user=user)
response = api_client.post("/api/v1.0/activation/validate/", {"code": code_variation})
assert response.status_code == status.HTTP_201_CREATED, (
f"Failed for variation: {code_variation}"
)
@pytest.mark.django_db
def test_inactive_code_cannot_be_used(api_client, settings):
"""Test that inactive codes cannot be used even if valid otherwise."""
settings.ACTIVATION_REQUIRED = True
_code = ActivationCodeFactory(
code="INACTIVE123VALID",
is_active=False,
max_uses=10,
expires_at=timezone.now() + timedelta(days=30),
)
user = UserFactory(email="blockeduser@example.com", password="password123")
api_client.force_authenticate(user=user)
response = api_client.post("/api/v1.0/activation/validate/", {"code": "INACTIVE123VALID"})
assert response.status_code == status.HTTP_400_BAD_REQUEST
@pytest.mark.django_db
def test_concurrent_activations_multi_use_code(api_client, settings):
"""Test that concurrent activations don't exceed max_uses."""
settings.ACTIVATION_REQUIRED = True
code = ActivationCodeFactory(code="CONCURRENT123456", max_uses=2)
# Create 3 users
users = [
UserFactory(email=f"concurrent{i}@example.com", password="password123") for i in range(3)
]
# First two should succeed
for i in range(2):
api_client.force_authenticate(user=users[i])
response = api_client.post("/api/v1.0/activation/validate/", {"code": "CONCURRENT123456"})
assert response.status_code == status.HTTP_201_CREATED
# Third should fail
api_client.force_authenticate(user=users[2])
response = api_client.post("/api/v1.0/activation/validate/", {"code": "CONCURRENT123456"})
assert response.status_code == status.HTTP_400_BAD_REQUEST
# Verify only 2 activations
code.refresh_from_db()
assert code.current_uses == 2
@@ -0,0 +1,328 @@
"""Tests for activation_codes models."""
import json
from datetime import timedelta
from django.core.exceptions import ValidationError
from django.db.models import ProtectedError
from django.utils import timezone
import pytest
import responses
from core.factories import UserFactory
from activation_codes.exceptions import InvalidCodeError, UserAlreadyActivatedError
from activation_codes.factories import ActivationCodeFactory, UserActivationFactory
from activation_codes.models import (
ActivationCode,
UserActivation,
UserRegistrationRequest,
generate_activation_code,
)
@pytest.mark.django_db
def test_generate_activation_code():
"""Test that generate_activation_code creates a valid code."""
code = generate_activation_code()
assert len(code) == 16
assert code.isupper()
assert all(c.isalnum() for c in code)
# Check that ambiguous characters are not present
assert "O" not in code
assert "0" not in code
assert "I" not in code
assert "1" not in code
@pytest.mark.django_db
def test_generate_activation_code_uniqueness():
"""Test that generated codes are unique."""
codes = [generate_activation_code() for _ in range(100)]
assert len(codes) == len(set(codes))
@pytest.mark.django_db
def test_activation_code_creation():
"""Test creating an activation code."""
activation_code = ActivationCodeFactory(code="TEST1234ABCD5678")
assert activation_code.code == "TEST1234ABCD5678"
assert activation_code.max_uses == 1
assert activation_code.current_uses == 0
assert activation_code.is_active is True
assert activation_code.expires_at is None
@pytest.mark.django_db
def test_activation_code_auto_generated_code():
"""Test that activation code is auto-generated if not provided."""
code = ActivationCodeFactory()
assert len(code.code) == 16
assert code.code.isupper()
@pytest.mark.django_db
def test_activation_code_str_representation():
"""Test string representation of activation code."""
activation_code = ActivationCodeFactory(code="TEST1234ABCD5678")
assert str(activation_code) == "TEST1234ABCD5678 (0/1)"
@pytest.mark.django_db
def test_activation_code_str_representation_unlimited():
"""Test string representation of unlimited activation code."""
unlimited_activation_code = ActivationCodeFactory(code="UNLIMITED123CODE", max_uses=0)
assert str(unlimited_activation_code) == "UNLIMITED123CODE (0/∞)"
@pytest.mark.django_db
def test_activation_code_is_valid_active():
"""Test that an active, non-expired code is valid."""
activation_code = ActivationCodeFactory()
assert activation_code.is_valid() is True
assert activation_code.can_be_used() is True
@pytest.mark.django_db
def test_activation_code_is_valid_inactive():
"""Test that an inactive code is not valid."""
inactive_activation_code = ActivationCodeFactory(is_active=False)
assert inactive_activation_code.is_valid() is False
assert inactive_activation_code.can_be_used() is False
@pytest.mark.django_db
def test_activation_code_is_valid_expired():
"""Test that an expired code is not valid."""
expired_activation_code = ActivationCodeFactory(
created_at=timezone.now() - timedelta(days=10),
expires_at=timezone.now() - timedelta(days=1),
)
assert expired_activation_code.is_valid() is False
assert expired_activation_code.can_be_used() is False
@pytest.mark.django_db
def test_activation_code_is_valid_max_uses_reached():
"""Test that a code with max uses reached is not valid."""
activation_code = ActivationCodeFactory(max_uses=1)
activation_code.current_uses = 1
activation_code.save()
assert activation_code.is_valid() is False
@pytest.mark.django_db
def test_activation_code_is_valid_unlimited_uses():
"""Test that unlimited code is always valid regardless of current uses."""
unlimited_activation_code = ActivationCodeFactory(max_uses=0)
unlimited_activation_code.current_uses = 100
unlimited_activation_code.save()
assert unlimited_activation_code.is_valid() is True
@pytest.mark.django_db
def test_activation_code_use_success():
"""Test successfully using an activation code."""
user = UserFactory()
activation_code = ActivationCodeFactory()
activation = activation_code.use(user)
assert isinstance(activation, UserActivation)
assert activation.user == user
assert activation.activation_code == activation_code
# Check that usage counter was incremented
activation_code.refresh_from_db()
assert activation_code.current_uses == 1
@pytest.mark.django_db
def test_activation_code_use_invalid_code():
"""Test using an invalid activation code raises error."""
inactive_activation_code = ActivationCodeFactory(is_active=False)
user = UserFactory()
with pytest.raises(InvalidCodeError):
inactive_activation_code.use(user)
@pytest.mark.django_db
def test_activation_code_use_already_activated():
"""Test using a code when user is already activated raises error."""
user = UserFactory()
activation_code = ActivationCodeFactory()
# First activation
activation_code.use(user)
# Try to activate again with a different code
another_code = ActivationCodeFactory(code="ANOTHER123456789")
with pytest.raises(UserAlreadyActivatedError):
another_code.use(user)
@pytest.mark.django_db
def test_activation_code_use_multi_use():
"""Test using a multi-use activation code."""
multi_use_activation_code = ActivationCodeFactory(max_uses=4)
users = [UserFactory(email=f"user{i}@example.com") for i in range(3)]
for i, user in enumerate(users):
activation = multi_use_activation_code.use(user)
assert activation.user == user
multi_use_activation_code.refresh_from_db()
assert multi_use_activation_code.current_uses == i + 1
# Code should still be valid
assert multi_use_activation_code.is_valid() is True
@pytest.mark.django_db
def test_activation_code_use_max_uses_exceeded():
"""Test that code cannot be used when max uses is reached."""
user = UserFactory()
activation_code = ActivationCodeFactory(max_uses=1)
# Use the code
activation_code.use(user)
# Try to use it again with another user
another_user = UserFactory(email="another@example.com")
with pytest.raises(InvalidCodeError):
activation_code.use(another_user)
@pytest.mark.django_db
def test_activation_code_expiration():
"""Test that code expires correctly."""
future_expiry = timezone.now() + timedelta(days=1)
code = ActivationCodeFactory(code="FUTURE123456789", expires_at=future_expiry)
assert code.is_valid() is True
# Manually set to past
code.expires_at = timezone.now() - timedelta(seconds=1)
code.save()
assert code.is_valid() is False
@pytest.mark.django_db
def test_user_activation_str_representation():
"""Test string representation of user activation."""
user_activation = UserActivationFactory(activation_code__code="TEST1234ABCD5678")
expected = f"{user_activation.user} - TEST1234ABCD5678"
assert str(user_activation) == expected
@pytest.mark.django_db
def test_user_activation_one_to_one_relationship():
"""Test that a user can only have one activation."""
user_activation = UserActivationFactory()
# Try to create another activation for the same user
with pytest.raises(ValidationError): # should be IntegrityError
UserActivationFactory(user=user_activation.user)
@pytest.mark.django_db
def test_activation_code_protect_on_delete():
"""Test that activation code is protected from deletion when used."""
user_activation = UserActivationFactory()
# Try to delete the activation code
with pytest.raises(ProtectedError):
user_activation.activation_code.delete()
@pytest.mark.django_db
def test_user_activation_cascade_on_user_delete():
"""Test that activation is deleted when user is deleted."""
activation = UserActivationFactory()
activation_id = activation.pk
activation.user.delete()
# Activation should be deleted
assert not UserActivation.objects.filter(id=activation_id).exists()
@pytest.mark.django_db
def test_activation_code_ordering():
"""Test that activation codes are ordered by created_at descending."""
code1 = ActivationCodeFactory(code="CODE1")
code2 = ActivationCodeFactory(code="CODE2")
code3 = ActivationCodeFactory(code="CODE3")
codes = list(ActivationCode.objects.all())
assert codes == [code3, code2, code1]
@pytest.mark.django_db
def test_user_activation_ordering():
"""Test that user activations are ordered by created_at descending."""
code1 = ActivationCodeFactory(code="CODE1", max_uses=3)
code2 = ActivationCodeFactory(code="CODE2", max_uses=3)
user1 = UserFactory(email="user1@example.com")
user2 = UserFactory(email="user2@example.com")
activation1 = UserActivationFactory(user=user1, activation_code=code1)
activation2 = UserActivationFactory(user=user2, activation_code=code2)
activations = list(UserActivation.objects.all())
assert activations == [activation2, activation1]
@responses.activate
@pytest.mark.django_db(transaction=True)
def test_activation_code_use_success_notify_brevo(settings):
"""Test successfully using an activation code and notify Brevo."""
settings.BREVO_API_KEY = "test_brevo_api_key"
settings.BREVO_WAITING_LIST_ID = "test_waiting_list_id"
settings.BREVO_FOLLOWUP_LIST_ID = "test_followup_list_name"
brevo_remove_mock = responses.post(
"https://api.brevo.com/v3/contacts/lists/test_waiting_list_id/contacts/remove",
json={"message": "Contacts added successfully"},
status=201,
)
brevo_create_contact = responses.post(
"https://api.brevo.com/v3/contacts",
status=200,
)
brevo_add_mock = responses.post(
"https://api.brevo.com/v3/contacts/lists/test_followup_list_name/contacts/add",
json={"message": "Contacts added successfully"},
status=201,
)
user = UserFactory()
registration = UserRegistrationRequest.objects.create(user=user)
activation_code = ActivationCodeFactory()
activation = activation_code.use(user)
registration.refresh_from_db()
assert registration.user_activation == activation
assert len(brevo_remove_mock.calls) == 1
assert brevo_remove_mock.calls[0].request.headers["api-key"] == "test_brevo_api_key"
assert json.loads(brevo_remove_mock.calls[0].request.body) == {"emails": [user.email]}
assert len(brevo_create_contact.calls) == 1
assert brevo_create_contact.calls[0].request.headers["api-key"] == "test_brevo_api_key"
assert json.loads(brevo_create_contact.calls[0].request.body) == {
"email": user.email,
"updateEnabled": True,
}
assert len(brevo_add_mock.calls) == 1
assert brevo_add_mock.calls[0].request.headers["api-key"] == "test_brevo_api_key"
assert json.loads(brevo_add_mock.calls[0].request.body) == {"emails": [user.email]}
@@ -0,0 +1,133 @@
"""Tests for activation_codes permissions."""
from django.test import RequestFactory
import pytest
from rest_framework.views import APIView
from core.factories import UserFactory
from activation_codes.factories import UserActivationFactory
from activation_codes.permissions import IsActivatedUser
@pytest.fixture(name="request_factory")
def request_factory_fixture():
"""Fixture to provide a request factory."""
return RequestFactory()
@pytest.fixture(name="view")
def view_fixture():
"""Fixture to provide a basic view instance."""
return APIView()
@pytest.mark.django_db
def test_is_activated_user_permission_activation_not_required(request_factory, view, settings):
"""Test that permission allows access when activation is not required."""
settings.ACTIVATION_REQUIRED = False
user = UserFactory()
request = request_factory.get("/")
request.user = user
permission = IsActivatedUser()
assert permission.has_permission(request, view) is True
@pytest.mark.django_db
def test_is_activated_user_permission_staff_user(request_factory, view, settings):
"""Test that staff users always have permission."""
settings.ACTIVATION_REQUIRED = True
staff_user = UserFactory(email="staff@example.com", password="password123", is_staff=True)
request = request_factory.get("/")
request.user = staff_user
permission = IsActivatedUser()
assert permission.has_permission(request, view) is True
@pytest.mark.django_db
def test_is_activated_user_permission_anonymous_user(request_factory, view, settings):
"""Test that anonymous users are allowed (handled by other permissions)."""
settings.ACTIVATION_REQUIRED = True
request = request_factory.get("/")
request.user = None
permission = IsActivatedUser()
assert permission.has_permission(request, view) is True
@pytest.mark.django_db
def test_is_activated_user_permission_activated_user(request_factory, view, settings):
"""Test that activated users have permission."""
settings.ACTIVATION_REQUIRED = True
# Activate the user
activation = UserActivationFactory()
request = request_factory.get("/")
request.user = activation.user
permission = IsActivatedUser()
assert permission.has_permission(request, view) is True
@pytest.mark.django_db
def test_is_activated_user_permission_not_activated_user(request_factory, view, settings):
"""Test that non-activated users do not have permission."""
settings.ACTIVATION_REQUIRED = True
user = UserFactory()
request = request_factory.get("/")
request.user = user
permission = IsActivatedUser()
assert permission.has_permission(request, view) is False
@pytest.mark.django_db
def test_is_activated_user_permission_custom_message():
"""Test that permission has custom message for frontend."""
permission = IsActivatedUser()
assert permission.message == "activation-required"
@pytest.mark.django_db
def test_is_activated_user_object_permission(request_factory, view, settings):
"""Test object-level permission delegates to has_permission."""
settings.ACTIVATION_REQUIRED = True
_user = UserFactory()
# Activate the user
activation = UserActivationFactory()
request = request_factory.get("/")
request.user = activation.user
permission = IsActivatedUser()
obj = object() # Any object
# Object permission should delegate to has_permission
assert permission.has_object_permission(request, view, obj) is True
@pytest.mark.django_db
def test_is_activated_user_object_permission_not_activated(request_factory, view, settings):
"""Test object-level permission when user is not activated."""
settings.ACTIVATION_REQUIRED = True
user = UserFactory()
request = request_factory.get("/")
request.user = user
permission = IsActivatedUser()
obj = object()
assert permission.has_object_permission(request, view, obj) is False
@@ -0,0 +1,150 @@
"""Tests for activation_codes serializers."""
import pytest
from activation_codes.factories import ActivationCodeFactory, UserActivationFactory
from activation_codes.serializers import (
ActivationCodeValidationSerializer,
ActivationStatusSerializer,
UserActivationSerializer,
)
@pytest.mark.django_db
def test_activation_code_validation_serializer_valid_code():
"""Test validating a valid activation code."""
# Create a valid activation code
_activation_code = ActivationCodeFactory(code="TEST1234ABCD5678")
serializer = ActivationCodeValidationSerializer(data={"code": "TEST1234ABCD5678"})
assert serializer.is_valid()
assert serializer.validated_data["code"] == "TEST1234ABCD5678"
@pytest.mark.django_db
def test_activation_code_validation_serializer_normalize_lowercase():
"""Test that code is normalized to uppercase."""
# Create a valid activation code
_activation_code = ActivationCodeFactory(code="TEST1234ABCD5678")
serializer = ActivationCodeValidationSerializer(data={"code": "test1234abcd5678"})
assert serializer.is_valid()
assert serializer.validated_data["code"] == "TEST1234ABCD5678"
@pytest.mark.django_db
def test_activation_code_validation_serializer_normalize_with_spaces():
"""Test that spaces are removed from code."""
# Create a valid activation code
_activation_code = ActivationCodeFactory(code="TEST1234ABCD5678")
serializer = ActivationCodeValidationSerializer(data={"code": "TEST 1234 ABCD 5678"})
assert serializer.is_valid()
assert serializer.validated_data["code"] == "TEST1234ABCD5678"
@pytest.mark.django_db
def test_activation_code_validation_serializer_normalize_with_dashes():
"""Test that dashes are removed from code."""
# Create a valid activation code
_activation_code = ActivationCodeFactory(code="TEST1234ABCD5678")
serializer = ActivationCodeValidationSerializer(data={"code": "TEST-1234-ABCD-5678"})
assert serializer.is_valid()
assert serializer.validated_data["code"] == "TEST1234ABCD5678"
@pytest.mark.django_db
def test_activation_code_validation_serializer_normalize_mixed():
"""Test that code with spaces, dashes and lowercase is normalized."""
# Create a valid activation code
_activation_code = ActivationCodeFactory(code="TEST1234ABCD5678")
serializer = ActivationCodeValidationSerializer(data={"code": " test-1234 abcd-5678 "})
assert serializer.is_valid()
assert serializer.validated_data["code"] == "TEST1234ABCD5678"
@pytest.mark.django_db
def test_activation_code_validation_serializer_missing_code():
"""Test that code field is required."""
serializer = ActivationCodeValidationSerializer(data={})
assert not serializer.is_valid()
assert "code" in serializer.errors
@pytest.mark.django_db
def test_user_activation_serializer():
"""Test serializing a user activation."""
activation = UserActivationFactory(activation_code__code="TEST1234ABCD5678")
serializer = UserActivationSerializer(activation)
data = serializer.data
assert "id" in data
assert data["code"] == "TEST1234ABCD5678"
assert "activated_at" in data
assert data["activated_at"] is not None
@pytest.mark.django_db
def test_user_activation_serializer_read_only_fields():
"""Test that all fields are read-only."""
activation = UserActivationFactory()
serializer = UserActivationSerializer(activation)
# All fields should be in read_only_fields
meta = serializer.Meta
assert set(meta.read_only_fields) == set(meta.fields)
@pytest.mark.django_db
def test_activation_status_serializer_activated():
"""Test serializing activation status for activated user."""
activation = UserActivationFactory(activation_code__code="TEST1234ABCD5678")
data = {"is_activated": True, "activation": activation, "requires_activation": True}
serializer = ActivationStatusSerializer(data)
serialized_data = serializer.data
assert serialized_data["is_activated"] is True
assert serialized_data["activation"] is not None
assert serialized_data["activation"]["code"] == "TEST1234ABCD5678"
assert serialized_data["requires_activation"] is True
@pytest.mark.django_db
def test_activation_status_serializer_not_activated():
"""Test serializing activation status for non-activated user."""
data = {"is_activated": False, "activation": None, "requires_activation": True}
serializer = ActivationStatusSerializer(data)
serialized_data = serializer.data
assert serialized_data["is_activated"] is False
assert serialized_data["activation"] is None
assert serialized_data["requires_activation"] is True
@pytest.mark.django_db
def test_activation_status_serializer_activation_not_required():
"""Test serializing activation status when activation is not required."""
data = {"is_activated": False, "activation": None, "requires_activation": False}
serializer = ActivationStatusSerializer(data)
serialized_data = serializer.data
assert serialized_data["is_activated"] is False
assert serialized_data["activation"] is None
assert serialized_data["requires_activation"] is False
@pytest.mark.django_db
def test_activation_status_serializer_all_fields_read_only():
"""Test that all fields in ActivationStatusSerializer are read-only."""
serializer = ActivationStatusSerializer()
for field_name, field in serializer.fields.items():
assert field.read_only is True, f"Field {field_name} should be read-only"
@@ -0,0 +1,411 @@
"""Tests for activation_codes viewsets."""
import json
from datetime import timedelta
from unittest.mock import patch
from django.utils import timezone
import pytest
import responses
from rest_framework import status
from core.factories import UserFactory
from activation_codes.factories import ActivationCodeFactory, UserActivationFactory
from activation_codes.models import ActivationCode, UserActivation, UserRegistrationRequest
@pytest.mark.django_db
def test_activation_status_unauthenticated(api_client):
"""Test that unauthenticated users cannot access status endpoint."""
response = api_client.get("/api/v1.0/activation/status/")
assert response.status_code == status.HTTP_401_UNAUTHORIZED
@pytest.mark.django_db
def test_activation_status_authenticated_not_activated(api_client, settings):
"""Test activation status for authenticated but not activated user."""
settings.ACTIVATION_REQUIRED = True
user = UserFactory()
api_client.force_authenticate(user=user)
response = api_client.get("/api/v1.0/activation/status/")
assert response.status_code == status.HTTP_200_OK
assert response.data["is_activated"] is False
assert response.data["activation"] is None
assert response.data["requires_activation"] is True
@pytest.mark.django_db
def test_activation_status_authenticated_activated(api_client, settings):
"""Test activation status for activated user."""
settings.ACTIVATION_REQUIRED = True
activation = UserActivationFactory(activation_code__code="TEST1234ABCD5678")
api_client.force_authenticate(user=activation.user)
response = api_client.get("/api/v1.0/activation/status/")
assert response.status_code == status.HTTP_200_OK
assert response.data["is_activated"] is True
assert response.data["activation"] is not None
assert response.data["activation"]["code"] == "TEST1234ABCD5678"
assert "activated_at" in response.data["activation"]
assert response.data["requires_activation"] is True
@pytest.mark.django_db
def test_activation_status_activation_not_required(api_client, settings):
"""Test activation status when activation is not required."""
settings.ACTIVATION_REQUIRED = False
user = UserFactory()
api_client.force_authenticate(user=user)
response = api_client.get("/api/v1.0/activation/status/")
assert response.status_code == status.HTTP_200_OK
assert response.data["requires_activation"] is False
@pytest.mark.django_db
def test_validate_code_unauthenticated(api_client):
"""Test that unauthenticated users cannot validate codes."""
response = api_client.post("/api/v1.0/activation/validate/", {"code": "TEST1234ABCD5678"})
assert response.status_code == status.HTTP_401_UNAUTHORIZED
@pytest.mark.django_db
def test_validate_code_success(api_client):
"""Test successfully validating and using an activation code."""
user = UserFactory()
activation_code = ActivationCode.objects.create(code="TEST1234ABCD5678")
api_client.force_authenticate(user=user)
with patch("activation_codes.viewsets.logger") as mock_logger:
response = api_client.post("/api/v1.0/activation/validate/", {"code": "TEST1234ABCD5678"})
assert response.status_code == status.HTTP_201_CREATED
assert "Your account has been successfully activated" in response.data["detail"]
assert "activation" in response.data
assert response.data["activation"]["code"] == "TEST1234ABCD5678"
# Verify user is now activated
assert UserActivation.objects.filter(user=user).exists()
# Verify activation code was used
activation_code.refresh_from_db()
assert activation_code.current_uses == 1
# Verify logging
mock_logger.info.assert_called_once()
@pytest.mark.django_db
def test_validate_code_with_spaces_and_lowercase(api_client):
"""Test validating code with spaces and lowercase."""
user = UserFactory()
ActivationCodeFactory(code="TEST1234ABCD5678")
api_client.force_authenticate(user=user)
response = api_client.post("/api/v1.0/activation/validate/", {"code": "test 1234 abcd 5678"})
assert response.status_code == status.HTTP_201_CREATED
assert UserActivation.objects.filter(user=user).exists()
@pytest.mark.django_db
def test_validate_code_already_activated(api_client):
"""Test validating code when user is already activated."""
# First activation
activation = UserActivationFactory()
api_client.force_authenticate(user=activation.user)
# Try to activate again with different code
_another_code = ActivationCodeFactory(code="ANOTHER123456789")
response = api_client.post("/api/v1.0/activation/validate/", {"code": "ANOTHER123456789"})
assert response.status_code == status.HTTP_400_BAD_REQUEST
assert response.data == {"code": "account-already-activated"}
@pytest.mark.django_db
def test_validate_code_nonexistent(api_client):
"""Test validating a non-existent code."""
user = UserFactory()
api_client.force_authenticate(user=user)
response = api_client.post("/api/v1.0/activation/validate/", {"code": "NONEXISTENT12345"})
assert response.status_code == status.HTTP_400_BAD_REQUEST
assert response.data == {"code": "invalid-code"}
@pytest.mark.django_db
def test_validate_code_invalid_serializer(api_client):
"""Test validating with invalid data."""
user = UserFactory()
api_client.force_authenticate(user=user)
response = api_client.post(
"/api/v1.0/activation/validate/",
{}, # Missing code
)
assert response.status_code == status.HTTP_400_BAD_REQUEST
assert "code" in response.data
@pytest.mark.django_db
def test_validate_code_inactive(api_client):
"""Test validating an inactive code."""
user = UserFactory()
ActivationCodeFactory(code="INACTIVE12345678", is_active=False)
api_client.force_authenticate(user=user)
response = api_client.post("/api/v1.0/activation/validate/", {"code": "INACTIVE12345678"})
assert response.status_code == status.HTTP_400_BAD_REQUEST
assert response.data == {"code": "invalid-code"}
@pytest.mark.django_db
def test_validate_code_expired(api_client):
"""Test validating an expired code."""
user = UserFactory()
ActivationCodeFactory(code="EXPIRED123456789", expires_at=timezone.now() - timedelta(days=1))
api_client.force_authenticate(user=user)
response = api_client.post("/api/v1.0/activation/validate/", {"code": "EXPIRED123456789"})
assert response.status_code == status.HTTP_400_BAD_REQUEST
@pytest.mark.django_db
def test_validate_code_max_uses_reached(api_client):
"""Test validating a code that has reached max uses."""
user = UserFactory()
ActivationCodeFactory(code="MAXUSED123456789", max_uses=1, current_uses=1)
api_client.force_authenticate(user=user)
response = api_client.post("/api/v1.0/activation/validate/", {"code": "MAXUSED123456789"})
assert response.status_code == status.HTTP_400_BAD_REQUEST
@pytest.mark.django_db
def test_validate_code_multi_use(api_client):
"""Test using a multi-use code with multiple users."""
multi_use_activation_code = ActivationCodeFactory(
code="MULTIUSE12345678",
max_uses=3,
)
users = []
for i in range(3):
user = UserFactory(email=f"user{i}@example.com")
users.append(user)
for i, user in enumerate(users):
api_client.force_authenticate(user=user)
response = api_client.post("/api/v1.0/activation/validate/", {"code": "MULTIUSE12345678"})
assert response.status_code == status.HTTP_201_CREATED
multi_use_activation_code.refresh_from_db()
assert multi_use_activation_code.current_uses == i + 1
@pytest.mark.django_db
def test_validate_code_unlimited_use(api_client):
"""Test using an unlimited code with multiple users."""
unlimited_activation_code = ActivationCodeFactory(
code="UNLIMITED123CODE",
max_uses=0, # Unlimited uses
)
for i in range(10):
user = UserFactory(email=f"user{i}@example.com")
api_client.force_authenticate(user=user)
response = api_client.post("/api/v1.0/activation/validate/", {"code": "UNLIMITED123CODE"})
assert response.status_code == status.HTTP_201_CREATED
# Code should still be valid
unlimited_activation_code.refresh_from_db()
assert unlimited_activation_code.is_valid() is True
@pytest.mark.django_db
def test_validate_code_logging_on_validation_error(api_client):
"""Test that validation errors are logged."""
user = UserFactory()
api_client.force_authenticate(user=user)
# Create a code that will cause validation error
code = ActivationCodeFactory(
code="WILLEXPIRE123456", expires_at=timezone.now() + timedelta(days=1)
)
# Make it expire
code.expires_at = timezone.now() - timedelta(seconds=1)
code.save()
with patch("activation_codes.viewsets.logger"):
response = api_client.post("/api/v1.0/activation/validate/", {"code": "WILLEXPIRE123456"})
assert response.status_code == status.HTTP_400_BAD_REQUEST
# Note: In this case the serializer will catch it first
# so the warning might not be called, but this tests the flow
@pytest.mark.django_db
def test_unauthenticated_register_email(api_client):
"""Test that unauthenticated users cannot register email."""
response = api_client.post(
"/api/v1.0/activation/register/",
{
"email": "test@example.com",
},
)
assert response.status_code == status.HTTP_401_UNAUTHORIZED
@pytest.mark.django_db
def test_register_email_success(api_client):
"""Test successfully registering an email."""
user = UserFactory()
api_client.force_authenticate(user=user)
response = api_client.post(
"/api/v1.0/activation/register/",
{},
)
assert response.status_code == status.HTTP_201_CREATED
assert response.data["code"] == "registration-successful"
registration = UserRegistrationRequest.objects.get(user=user)
assert registration.user == user
@pytest.mark.django_db
def test_register_already_created(api_client):
"""Test successfully registering an email."""
user = UserFactory()
_registration = UserRegistrationRequest.objects.create(
user=user,
)
api_client.force_authenticate(user=user)
response = api_client.post(
"/api/v1.0/activation/register/",
)
assert response.status_code == status.HTTP_200_OK
assert response.data == {"code": "registration-successful"}
assert UserRegistrationRequest.objects.filter(user=user).count() == 1
@pytest.mark.django_db
def test_validate_code_registered_user(api_client):
"""Test validating a code for a user with a pre-existing registration."""
user = UserFactory()
_registration = UserRegistrationRequest.objects.create(
user=user,
)
activation_code = ActivationCodeFactory(code="TEST1234ABCD5678")
api_client.force_authenticate(user=user)
response = api_client.post("/api/v1.0/activation/validate/", {"code": "TEST1234ABCD5678"})
assert response.status_code == status.HTTP_201_CREATED
_registration.refresh_from_db()
assert _registration.user_activation.activation_code == activation_code
@responses.activate
@pytest.mark.django_db
def test_register_email_success_brevo(api_client, settings):
"""Test successfully registering an email and notify Brevo."""
settings.BREVO_API_KEY = "test_brevo_api_key"
settings.BREVO_WAITING_LIST_ID = "test_waiting_list_id"
brevo_create_contact = responses.post(
"https://api.brevo.com/v3/contacts",
status=200,
)
brevo_mock = responses.post(
"https://api.brevo.com/v3/contacts/lists/test_waiting_list_id/contacts/add",
json={"message": "Contacts added successfully"},
status=201,
)
user = UserFactory()
api_client.force_authenticate(user=user)
response = api_client.post(
"/api/v1.0/activation/register/",
{},
)
assert response.status_code == status.HTTP_201_CREATED
assert response.data["code"] == "registration-successful"
registration = UserRegistrationRequest.objects.get(user=user)
assert registration.user == user
assert len(brevo_create_contact.calls) == 1
assert brevo_create_contact.calls[0].request.headers["api-key"] == "test_brevo_api_key"
assert json.loads(brevo_create_contact.calls[0].request.body) == {
"email": user.email,
"updateEnabled": True,
}
assert len(brevo_mock.calls) == 1
assert brevo_mock.calls[0].request.headers["api-key"] == "test_brevo_api_key"
assert json.loads(brevo_mock.calls[0].request.body) == {"emails": [user.email]}
# Register again to test idempotency
response = api_client.post(
"/api/v1.0/activation/register/",
{},
)
assert response.status_code == status.HTTP_200_OK
assert response.data["code"] == "registration-successful"
assert len(brevo_mock.calls) == 1 # No new call made
@responses.activate
@pytest.mark.django_db
def test_register_email_success_brevo_fails(api_client, settings):
"""Test successfully registering an email, even if Brevo fails."""
settings.BREVO_API_KEY = "test_brevo_api_key"
settings.BREVO_WAITING_LIST_ID = "test_waiting_list_id"
_brevo_create_contact = responses.post(
"https://api.brevo.com/v3/contacts",
status=200,
)
brevo_mock = responses.post(
"https://api.brevo.com/v3/contacts/lists/test_waiting_list_id/contacts/add",
status=400,
)
user = UserFactory()
api_client.force_authenticate(user=user)
response = api_client.post(
"/api/v1.0/activation/register/",
{},
)
assert response.status_code == status.HTTP_201_CREATED
assert response.data["code"] == "registration-successful"
registration = UserRegistrationRequest.objects.get(user=user)
assert registration.user == user
assert len(brevo_mock.calls) == 1
+153
View File
@@ -0,0 +1,153 @@
"""API ViewSets for activation codes."""
import logging
from django.conf import settings
from django.core.exceptions import ValidationError
from django.utils.translation import gettext_lazy as _
from rest_framework import status, viewsets
from rest_framework.decorators import action
from rest_framework.response import Response
from core.brevo import add_user_to_brevo_list
from core.permissions import IsAuthenticated
from . import models, serializers
from .exceptions import InvalidCodeError, UserAlreadyActivatedError
logger = logging.getLogger(__name__)
class ActivationViewSet(viewsets.GenericViewSet):
"""
ViewSet for handling user activation with codes.
Endpoints:
- GET /activation/status/ - Check if current user is activated
- POST /activation/validate/ - Validate and use an activation code
- POST /activation/register/ - Register an email to be notified later
"""
permission_classes = [IsAuthenticated]
serializer_class = serializers.ActivationCodeValidationSerializer
@action(detail=False, methods=["get"], url_path="status")
def status(self, request):
"""
Get the activation status of the current user.
Returns:
- is_activated: Whether the user has activated their account
- activation: Details of the activation (if exists)
- requires_activation: Whether activation is required by the system
"""
requires_activation = getattr(settings, "ACTIVATION_REQUIRED", False)
try:
activation = models.UserActivation.objects.select_related("activation_code").get(
user=request.user
)
is_activated = True
except models.UserActivation.DoesNotExist:
activation = None
is_activated = False
response_data = {
"is_activated": is_activated,
"activation": activation,
"requires_activation": requires_activation,
}
return Response(
serializers.ActivationStatusSerializer(response_data).data, status=status.HTTP_200_OK
)
@action(detail=False, methods=["post"], url_path="validate")
def validate_code(self, request):
"""
Validate an activation code and activate the user's account.
Request body:
- code: The activation code to validate
Returns:
- Success: Activation details
- Error: Validation error message
"""
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
code_value = serializer.validated_data["code"]
# Get the activation code
try:
activation_code = models.ActivationCode.objects.get(code=code_value)
except models.ActivationCode.DoesNotExist:
logger.info("Activation code %s does not exist", code_value)
return Response({"code": "invalid-code"}, status=status.HTTP_400_BAD_REQUEST)
# Use the code
try:
activation = activation_code.use(request.user)
except InvalidCodeError as exc:
logger.warning(exc)
return Response({"code": "invalid-code"}, status=status.HTTP_400_BAD_REQUEST)
except UserAlreadyActivatedError as exc:
logger.info(exc)
return Response(
{"code": "account-already-activated"}, status=status.HTTP_400_BAD_REQUEST
)
logger.info("User %s activated account with code %s", request.user.id, activation_code.code)
return Response(
{
"code": "activation-successful",
"detail": _("Your account has been successfully activated"),
"activation": serializers.UserActivationSerializer(activation).data,
},
status=status.HTTP_201_CREATED,
)
@action(detail=False, methods=["post"], url_path="register")
def register_email(self, request):
"""
Register an email to be notified when activation codes are available.
Request body:
- email: The email address to register
Returns:
- Success: Confirmation message
- Error: Validation error message
"""
serializer = serializers.UserRegistrationRequestSerializer(
data={},
context={"request": request},
)
serializer.is_valid(raise_exception=True)
# Create the registration
try:
serializer.save()
except ValidationError:
# user is already registered, it's OK
return Response(
{"code": "registration-successful"},
status=status.HTTP_200_OK,
)
add_user_to_brevo_list(
[serializer.validated_data["user"].email], settings.BREVO_WAITING_LIST_ID
)
logger.info(
"Registered email %s for activation notifications",
serializer.validated_data["user"].email,
)
return Response(
{"code": "registration-successful"},
status=status.HTTP_201_CREATED,
)
View File
+19
View File
@@ -0,0 +1,19 @@
"""Admin classes and registrations for chat application."""
from django.contrib import admin
from . import models
@admin.register(models.ChatConversation)
class ChatConversationAdmin(admin.ModelAdmin):
"""Admin class for the ChatConversation model"""
autocomplete_fields = ("owner",)
list_display = (
"id",
"title",
"created_at",
"updated_at",
)
@@ -0,0 +1,150 @@
"""Constants and schemas for the Albert RAG agent from Albert API codebase."""
from enum import Enum
from typing import Annotated, Any, Dict, List, Literal, Optional, Self
from pydantic import BaseModel, Field, StringConstraints, model_validator
# - app/schemas/chunks.py
class Chunk(BaseModel):
"""Model representing a chunk of text with metadata."""
object: Literal["chunk"] = "chunk"
id: int
metadata: Dict[str, Any]
content: str
class Chunks(BaseModel):
"""Model representing a list of chunks."""
object: Literal["list"] = "list"
data: List[Chunk]
# - app/schemas/usage.py
class CarbonFootprintUsageKWh(BaseModel):
"""Model representing the carbon footprint usage in kWh (kilowatt-hours)."""
min: Optional[float] = Field(default=None, description="Minimum carbon footprint in kWh.")
max: Optional[float] = Field(default=None, description="Maximum carbon footprint in kWh.")
class CarbonFootprintUsageKgCO2eq(BaseModel):
"""Model representing the carbon footprint usage in kgCO2eq (kilograms of CO2 equivalent)."""
min: Optional[float] = Field(
default=None, description="Minimum carbon footprint in kgCO2eq (global warming potential)."
)
max: Optional[float] = Field(
default=None, description="Maximum carbon footprint in kgCO2eq (global warming potential)."
)
class CarbonFootprintUsage(BaseModel):
"""Model representing the carbon footprint usage in kWh and kgCO2eq."""
kWh: CarbonFootprintUsageKWh = Field(default_factory=CarbonFootprintUsageKWh)
kgCO2eq: CarbonFootprintUsageKgCO2eq = Field(default_factory=CarbonFootprintUsageKgCO2eq)
class BaseUsage(BaseModel):
"""Base model for usage statistics in the Albert API."""
prompt_tokens: int = Field(
default=0, description="Number of prompt tokens (e.g. input tokens)."
)
completion_tokens: int = Field(
default=0, description="Number of completion tokens (e.g. output tokens)."
)
total_tokens: int = Field(
default=0, description="Total number of tokens (e.g. input and output tokens)."
)
cost: float = Field(default=0.0, description="Total cost of the request.")
carbon: CarbonFootprintUsage = Field(default_factory=CarbonFootprintUsage)
# - app/schemas/usage.py
class Detail(BaseModel):
"""Model representing a detail in the usage statistics."""
id: str
model: str
usage: BaseUsage = Field(default_factory=BaseUsage)
class Usage(BaseUsage):
"""Model representing the usage statistics for the Albert API."""
details: List[Detail] = []
class SearchMethod(str, Enum):
"""
Enum representing the search methods available (will be displayed in this order in playground).
"""
MULTIAGENT = "multiagent"
HYBRID = "hybrid"
SEMANTIC = "semantic"
LEXICAL = "lexical"
class SearchArgs(BaseModel):
"""Model representing the arguments for a search request in the Albert API."""
collections: List[Any] = Field(default=[], description="List of collections ID")
rff_k: int = Field(default=20, description="k constant in RFF algorithm")
k: int = Field(gt=0, default=4, description="Number of results to return")
method: SearchMethod = Field(default=SearchMethod.SEMANTIC)
score_threshold: Optional[float] = Field(
default=0.0,
ge=0.0,
le=1.0,
description=(
"Score of cosine similarity threshold for filtering results, "
"only available for semantic search method."
),
)
web_search: bool = Field(
default=False, description="Whether add internet search to the results."
)
web_search_k: int = Field(default=5, description="Number of results to return for web search.")
@model_validator(mode="after")
def score_threshold_filter(self) -> Self:
"""Validate the score threshold based on the search method."""
if self.score_threshold and self.method not in (
SearchMethod.SEMANTIC,
SearchMethod.MULTIAGENT,
):
raise ValueError(
"Score threshold is only available for semantic and multiagent search methods."
)
return self
class SearchRequest(SearchArgs):
"""Model representing a search request in the Albert API."""
prompt: Annotated[
str,
StringConstraints(strip_whitespace=True, min_length=1),
] = Field(description="Prompt related to the search")
class Search(BaseModel):
"""Model representing a search result in the Albert API."""
method: SearchMethod
score: float
chunk: Chunk
class Searches(BaseModel):
"""Model representing a list of search results in the Albert API."""
object: Literal["list"] = "list"
data: List[Search]
usage: Usage = Field(default_factory=Usage, description="Usage information for the request.")
+41
View File
@@ -0,0 +1,41 @@
"""Constants for RAG (Retrieval-Augmented Generation) results."""
from typing import List
from pydantic import BaseModel, Field
class RAGWebUsage(BaseModel):
"""
Model representing the usage statistics for web results in RAG (Retrieval-Augmented Generation).
"""
prompt_tokens: int = Field(default=0, description="Number of prompt tokens used.")
completion_tokens: int = Field(default=0, description="Number of completion tokens generated.")
class RAGWebResult(BaseModel):
"""Model representing a single web result in RAG (Retrieval-Augmented Generation)."""
url: str = Field(..., description="URL of the web result.")
content: str = Field(..., description="Content of the web result chunk.")
score: float = Field(
..., description="Relevance score of the web result, typically between 0 and 1."
)
class RAGWebResults(BaseModel):
"""Model representing a list of web results in RAG (Retrieval-Augmented Generation)."""
data: List[RAGWebResult]
usage: RAGWebUsage = Field(..., description="RAG usage statistics.")
def to_prompt(self) -> str:
"""Convert the web results to a prompt string."""
_format = " - From: {url}:\n content: {content}\n\n"
return (
"\n\n".join(
_format.format(url=result.url, content=result.content) for result in self.data
)
+ "\n\n"
)
@@ -0,0 +1,43 @@
"""Document Converter using MarkItDown"""
import os.path
from io import BytesIO
from markitdown import MarkItDown
class DocumentConverter:
"""Simple document converter that uses MarkItDown to convert documents to Markdown format."""
def __init__(self):
"""Initialize the DocumentConverter with MarkItDown."""
self.converter = MarkItDown()
def convert_raw( # pylint: disable=unused-argument
self,
*,
name: str,
content_type: str,
content: bytes,
) -> str:
"""
Convert a document to Markdown format.
The name, content_type, and content parameters comes from the user input
(vercel SDK Attachment, or BinaryContent).
Args:
name (str): The name of the document.
content_type (str): The MIME type of the document (e.g., "application/pdf").
content (bytes): The content of the document as bytes.
"""
return self._convert(BytesIO(content), file_extension=os.path.splitext(name)[1])
def _convert(self, document: BytesIO, file_extension: str) -> str:
"""
Convert the given document using the underlying DocumentConverter.
"""
conversion = self.converter.convert_stream(
document, file_extension=file_extension or ".txt"
)
document_markdown = conversion.text_content
return document_markdown
@@ -0,0 +1,192 @@
"""Implementation of the Albert API for RAG document search."""
import json
import logging
from io import BytesIO
from typing import Optional
from urllib.parse import urljoin
from django.conf import settings
import requests
from chat.agent_rag.albert_api_constants import Searches
from chat.agent_rag.constants import RAGWebResult, RAGWebResults, RAGWebUsage
from chat.agent_rag.document_converter.markitdown import DocumentConverter
from chat.agent_rag.document_rag_backends.base_rag_backend import BaseRagBackend
logger = logging.getLogger(__name__)
class AlbertRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-attributes
"""
This class is a placeholder for the Albert API implementation.
It is designed to be used with the RAG (Retrieval-Augmented Generation) document search system.
It provides methods to:
- Create a collection for the search operation.
- Parse documents and convert them to Markdown format:
+ Handle PDF parsing using the Albert API.
+ Use the DocumentConverter (markitdown) for other formats.
- Store parsed documents in the Albert collection.
- Perform a search operation using the Albert API.
"""
def __init__(self, collection_id: Optional[str] = None):
# Initialize any necessary parameters or configurations here
super().__init__(collection_id)
self._base_url = settings.ALBERT_API_URL
self._headers = {
"Authorization": f"Bearer {settings.ALBERT_API_KEY}",
}
self._collections_endpoint = urljoin(self._base_url, "/v1/collections")
self._documents_endpoint = urljoin(self._base_url, "/v1/documents")
self._pdf_parser_endpoint = urljoin(self._base_url, "/v1/parse-beta")
self._search_endpoint = urljoin(self._base_url, "/v1/search")
self._default_collection_description = "Temporary collection for RAG document search"
def create_collection(self, name: str, description: Optional[str] = None) -> str:
"""
Create a temporary collection for the search operation.
This method should handle the logic to create or retrieve an existing collection.
"""
response = requests.post(
self._collections_endpoint,
headers=self._headers,
json={
"name": name,
"description": description or self._default_collection_description,
"visibility": "private",
},
timeout=settings.ALBERT_API_TIMEOUT,
)
response.raise_for_status()
self.collection_id = str(response.json()["id"])
return self.collection_id
def delete_collection(self) -> None:
"""
Delete the current collection
"""
response = requests.delete(
urljoin(f"{self._collections_endpoint}/", self.collection_id),
headers=self._headers,
timeout=settings.ALBERT_API_TIMEOUT,
)
response.raise_for_status()
def parse_pdf_document(self, name: str, content_type: str, content: BytesIO) -> str:
"""
Parse the PDF document content and return the text content.
This method should handle the logic to convert the PDF into
a format suitable for the Albert API.
"""
response = requests.post(
self._pdf_parser_endpoint,
headers=self._headers,
files={
"file": (
name,
content,
content_type,
), # Use the name as the filename in the request
"output_format": (None, "markdown"), # Specify the output format as Markdown,
},
timeout=settings.ALBERT_API_PARSE_TIMEOUT,
)
response.raise_for_status()
return "\n\n".join(
document_page["content"] for document_page in response.json().get("data", [])
)
def parse_document(self, name: str, content_type: str, content: BytesIO):
"""
Parse the document and prepare it for the search operation.
This method should handle the logic to convert the document
into a format suitable for the Albert API.
Args:
name (str): The name of the document.
content_type (str): The MIME type of the document (e.g., "application/pdf").
content (BytesIO): The content of the document as a BytesIO stream.
Returns:
str: The document content in Markdown format.
"""
# Implement the parsing logic here
if content_type == "application/pdf":
# Handle PDF parsing
markdown_content = self.parse_pdf_document(
name=name, content_type=content_type, content=content
)
else:
markdown_content = DocumentConverter().convert_raw(
name=name, content_type=content_type, content=content
)
return markdown_content
def store_document(self, name: str, content: str) -> None:
"""
Store the document content in the Albert collection.
This method should handle the logic to send the document content to the Albert API.
Args:
name (str): The name of the document.
content (str): The content of the document in Markdown format.
"""
response = requests.post(
urljoin(self._base_url, self._documents_endpoint),
headers=self._headers,
files={
"file": (f"{name}.md", BytesIO(content.encode("utf-8")), "text/markdown"),
"collection": (None, int(self.collection_id)),
"metadata": (None, json.dumps({"document_name": name})), # undocumented API
},
timeout=settings.ALBERT_API_TIMEOUT,
)
logger.debug(response.json())
response.raise_for_status()
def search(self, query, results_count: int = 4) -> RAGWebResults:
"""
Perform a search using the Albert API based on the provided query.
Args:
query (str): The search query.
results_count (int): The number of results to return.
Returns:
RAGWebResults: The search results.
"""
response = requests.post(
urljoin(self._base_url, self._search_endpoint),
headers=self._headers,
json={
"collections": [int(self.collection_id)],
"prompt": query,
"score_threshold": 0.6,
"k": results_count, # Number of chunks to return from the search
},
timeout=settings.ALBERT_API_TIMEOUT,
)
response.raise_for_status()
searches = Searches(**response.json())
return RAGWebResults(
data=[
RAGWebResult(
url=result.chunk.metadata["document_name"],
content=result.chunk.content,
score=result.score,
)
for result in searches.data
],
usage=RAGWebUsage(
prompt_tokens=searches.usage.prompt_tokens,
completion_tokens=searches.usage.completion_tokens,
),
)
@@ -0,0 +1,94 @@
"""Implementation of the Albert API for RAG document search."""
import logging
from contextlib import contextmanager
from io import BytesIO
from typing import Optional
from chat.agent_rag.constants import RAGWebResults
logger = logging.getLogger(__name__)
class BaseRagBackend:
"""Base class for RAG backends."""
def __init__(self, collection_id: Optional[str] = None):
"""Backend settings."""
self.collection_id = collection_id
self._default_collection_description = "Temporary collection for RAG document search"
def create_collection(self, name: str, description: Optional[str] = None) -> str:
"""
Create a temporary collection for the search operation.
This method should handle the logic to create or retrieve an existing collection.
"""
raise NotImplementedError("Must be implemented in subclass.")
def parse_document(self, name: str, content_type: str, content: BytesIO):
"""
Parse the document and prepare it for the search operation.
This method should handle the logic to convert the document
into a format suitable for the Albert API.
Args:
name (str): The name of the document.
content_type (str): The MIME type of the document (e.g., "application/pdf").
content (BytesIO): The content of the document as a BytesIO stream.
Returns:
str: The document content in Markdown format.
"""
raise NotImplementedError("Must be implemented in subclass.")
def store_document(self, name: str, content: str) -> None:
"""
Store the document content in the Albert collection.
This method should handle the logic to send the document content to the Albert API.
Args:
name (str): The name of the document.
content (str): The content of the document in Markdown format.
"""
raise NotImplementedError("Must be implemented in subclass.")
def parse_and_store_document(self, name: str, content_type: str, content: BytesIO) -> str:
"""
Parse the document and store it in the Albert collection.
Args:
name (str): The name of the document.
content_type (str): The MIME type of the document (e.g., "application/pdf").
content (BytesIO): The content of the document as a BytesIO stream.
"""
if not self.collection_id:
raise RuntimeError("The RAG backend requires collection_id")
document_content = self.parse_document(name, content_type, content)
self.store_document(name, document_content)
return document_content
def delete_collection(self) -> None:
"""
Delete the collection.
This method should handle the logic to delete the collection from the backend.
"""
raise NotImplementedError("Must be implemented in subclass.")
def search(self, query, results_count: int = 4) -> RAGWebResults:
"""
Search the collection for the given query.
"""
raise NotImplementedError("Must be implemented in subclass.")
@classmethod
@contextmanager
def temporary_collection(cls, name: str, description: Optional[str] = None):
"""Context manager for RAG backend with temporary collections."""
backend = cls()
backend.create_collection(name=name, description=description)
try:
yield backend
finally:
backend.delete_collection()
@@ -0,0 +1,208 @@
"""Implementation of the Albert API for RAG document search."""
import json
import logging
from io import BytesIO
from urllib.parse import urljoin
from django.conf import settings
import requests
from chat.agent_rag.albert_api_constants import Searches
from chat.agent_rag.constants import RAGWebResult, RAGWebResults, RAGWebUsage
from chat.agent_rag.document_converter.markitdown import DocumentConverter
from chat.models import ChatConversation
logger = logging.getLogger(__name__)
class AlbertRagDocumentSearch:
"""
This class is a placeholder for the Albert API implementation.
It is designed to be used with the RAG (Retrieval-Augmented Generation) document search system.
It provides methods to:
- Create a collection for the search operation.
- Parse documents and convert them to Markdown format:
+ Handle PDF parsing using the Albert API.
+ Use the DocumentConverter (markitdown) for other formats.
- Store parsed documents in the Albert collection.
- Perform a search operation using the Albert API.
"""
def __init__(self, conversation: ChatConversation):
# Initialize any necessary parameters or configurations here
self._base_url = settings.ALBERT_API_URL
self._headers = {
"Authorization": f"Bearer {settings.ALBERT_API_KEY}",
}
self._collections_endpoint = urljoin(self._base_url, "/v1/collections")
self._documents_endpoint = urljoin(self._base_url, "/v1/documents")
self._pdf_parser_endpoint = urljoin(self._base_url, "/v1/parse-beta")
self._search_endpoint = urljoin(self._base_url, "/v1/search")
self.conversation = conversation
@property
def _albert_collection_id(self):
"""
Generate the collection name based on the conversation ID.
This is used to create or retrieve a collection for the search operation.
"""
return f"conversation-{self.conversation.pk}"
@property
def collection_id(self) -> int:
"""
Get the collection ID for the current conversation.
Might be created later by self._create_collection() if it does not exist.
"""
return int(self.conversation.collection_id) if self.conversation.collection_id else None
def _create_collection(self) -> bool:
"""
Create a temporary collection for the search operation.
This method should handle the logic to create or retrieve an existing collection.
"""
response = requests.post(
self._collections_endpoint,
headers=self._headers,
json={
"name": self._albert_collection_id,
"description": "Temporary collection for RAG document search",
"visibility": "private",
},
timeout=settings.ALBERT_API_TIMEOUT,
)
response.raise_for_status()
self.conversation.collection_id = str(response.json()["id"])
return True
def _parse_pdf_document(self, name: str, content_type: str, content: BytesIO) -> str:
"""
Parse the PDF document content and return the text content.
This method should handle the logic to convert the PDF into
a format suitable for the Albert API.
"""
response = requests.post(
self._pdf_parser_endpoint,
headers=self._headers,
files={
"file": (
name,
content,
content_type,
), # Use the name as the filename in the request
"output_format": (None, "markdown"), # Specify the output format as Markdown,
},
timeout=settings.ALBERT_API_PARSE_TIMEOUT,
)
response.raise_for_status()
return "\n\n".join(
document_page["content"] for document_page in response.json().get("data", [])
)
def parse_document(self, name: str, content_type: str, content: BytesIO):
"""
Parse the document and prepare it for the search operation.
This method should handle the logic to convert the document
into a format suitable for the Albert API.
Args:
name (str): The name of the document.
content_type (str): The MIME type of the document (e.g., "application/pdf").
content (BytesIO): The content of the document as a BytesIO stream.
Returns:
str: The document content in Markdown format.
"""
# Implement the parsing logic here
if content_type == "application/pdf":
# Handle PDF parsing
markdown_content = self._parse_pdf_document(
name=name, content_type=content_type, content=content
)
else:
markdown_content = DocumentConverter().convert_raw(
name=name, content_type=content_type, content=content
)
return markdown_content
def _store_document(self, name: str, content: str):
"""
Store the document content in the Albert collection.
This method should handle the logic to send the document content to the Albert API.
Args:
content (str): The content of the document in Markdown format.
"""
if not self.collection_id and not self._create_collection():
raise RuntimeError("Failed to create or retrieve the collection.")
response = requests.post(
urljoin(self._base_url, self._documents_endpoint),
headers=self._headers,
files={
"file": (f"{name}.md", BytesIO(content.encode("utf-8")), "text/markdown"),
"collection": (None, int(self.collection_id)),
"metadata": (None, json.dumps({"document_name": name})), # undocumented API
},
timeout=settings.ALBERT_API_TIMEOUT,
)
logger.debug(response.json())
response.raise_for_status()
def parse_and_store_document(self, name: str, content_type: str, content: BytesIO):
"""
Parse the document and store it in the Albert collection.
Args:
name (str): The name of the document.
content_type (str): The MIME type of the document (e.g., "application/pdf").
content (BytesIO): The content of the document as a BytesIO stream.
"""
document_content = self.parse_document(name, content_type, content)
self._store_document(name, document_content)
return document_content
def search(self, query, results_count: int = 4) -> RAGWebResults:
"""
Perform a search using the Albert API based on the provided query.
:param query: The search query string.
:param results_count: The number of results to return.
:return: Search results from the Albert API.
"""
response = requests.post(
urljoin(self._base_url, self._search_endpoint),
headers=self._headers,
json={
"collections": [self.collection_id],
"prompt": query,
"score_threshold": 0.6,
"k": results_count, # Number of chunks to return from the search
},
timeout=settings.ALBERT_API_TIMEOUT,
)
response.raise_for_status()
searches = Searches(**response.json())
return RAGWebResults(
data=[
RAGWebResult(
url=result.chunk.metadata["document_name"],
content=result.chunk.content,
score=result.score,
)
for result in searches.data
],
usage=RAGWebUsage(
prompt_tokens=searches.usage.prompt_tokens,
completion_tokens=searches.usage.completion_tokens,
),
)
@@ -0,0 +1,113 @@
"""
Albert API web search manager.
Instead of developing a full web search flow, we simply use the Albert API /v1/search endpoint.
See: https://albert.api.etalab.gouv.fr/documentation#tag/Search
Under the hood, on Albert side:
- It create a temporary collection
- It performs a web search using the query
- It returns the results as a list of URls
- It loads and parses the content of each URL (vectorization)
and stores the results in the temporary collection
- It makes a semanctic search on the temporary collection using the query
- It returns the results as a list of chunks with metadata
- It deletes the temporary collection
"""
import logging
from urllib.parse import urljoin
from django.conf import settings
import requests
from ..albert_api_constants import Searches, SearchRequest
from ..constants import RAGWebResult, RAGWebResults, RAGWebUsage
from .base import BaseWebSearchManager
logger = logging.getLogger(__name__)
class AlbertWebSearchManager(BaseWebSearchManager):
"""
A class to manage web search operations using the Albert API.
"""
def __init__(self):
"""Initializes with the base URL and endpoints for the Albert API."""
self._base_url = settings.ALBERT_API_URL
self._headers = {
"Authorization": f"Bearer {settings.ALBERT_API_KEY}",
"Content-Type": "application/json",
}
self._search_endpoint = urljoin(self._base_url, "/v1/search")
@staticmethod
def _clean_url(url: str) -> str:
"""
Clean the URL by removing the trailing '.html'.
We want it to fail when Albert fixes the bug that adds '.html' to the end of URLs.
Note: this is a bad workaround because when fixed it may break existing URLs.
Args:
url (str): The URL to clean.
Returns:
str: The cleaned URL.
"""
return url.rsplit(".html", 1)[0]
def web_search(self, query: str) -> RAGWebResults:
"""
Perform a web search using the Albert API.
Args:
query (str): The search query.
Returns:
Searches: A Searches object containing the search results.
Raises:
ValueError: If the query is empty.
requests.HTTPError: If the request to the Albert API fails.
requests.exceptions.JSONDecodeError: If the response body does not
contain valid json
"""
if not query.strip():
raise ValueError("Search query cannot be empty.")
search_request = SearchRequest(
prompt=query,
web_search=True, # Enable web search
web_search_k=settings.RAG_WEB_SEARCH_MAX_RESULTS, # Number of web search results
k=settings.RAG_WEB_SEARCH_CHUNK_NUMBER, # Number of chunks to return from the search
)
logger.debug("Albert API search request: %s", search_request.model_dump())
response = requests.post(
self._search_endpoint,
headers=self._headers,
json=search_request.model_dump(mode="json", exclude_unset=True),
timeout=settings.ALBERT_API_TIMEOUT,
)
response.raise_for_status()
searches = Searches(**response.json())
return RAGWebResults(
data=[
RAGWebResult(
url=self._clean_url(result.chunk.metadata["document_name"]),
content=result.chunk.content,
score=result.score,
)
for result in searches.data
],
usage=RAGWebUsage(
prompt_tokens=searches.usage.prompt_tokens, # pylint: disable=no-member
completion_tokens=searches.usage.completion_tokens, # pylint: disable=no-member
),
)
@@ -0,0 +1,24 @@
"""Base class for web search managers."""
from ..constants import RAGWebResults
class BaseWebSearchManager:
"""
A class to manage web search operations.
This is an abstract base class that should be implemented
for specific web search managers.
"""
def web_search(self, query: str) -> RAGWebResults:
"""
Perform a web search.
Args:
query (str): The search query.
Returns:
RAGWebResults: A Searches object containing the search results.
"""
raise NotImplementedError()
@@ -0,0 +1,68 @@
"""Mocked web search manager for testing purposes."""
# pylint: disable=line-too-long
import logging
from ..constants import RAGWebResult, RAGWebResults, RAGWebUsage
logger = logging.getLogger(__name__)
result_1 = {
"url": "https://www.lemonde.fr/sciences/article/2025/06/25/le-telescope-james-webb-decouvre-sa-premiere-exoplanete-identifiee-comme-une-petite-planete-froide_6615888_1650684.html",
"content": "+ [La beauté créatrice](https://la-beaute-creatrice.lemonde.fr/ \"La beauté créatrice\")\n\t+ [Perspectives](https://www.lemonde.fr/perspectives/ \"Perspectives\") \n\t+ [Gestion des cookies](#)\n* [Recherche](https://www.lemonde.fr/recherche/)\n\n Cet article vous est offert Pour lire gratuitement cet article réservé aux abonnés, connectez\\-vous [Se connecter](https://secure.lemonde.fr/sfuser/connexion?gift=true) Vous n'êtes pas inscrit sur Le Monde ? \n [Inscrivez\\-vous gratuitement](https://secure.lemonde.fr/sfuser/register?gift=true) * [Sciences Sciences](https://www.lemonde.fr/sciences/)\n* [Astronomie Astronomie](https://www.lemonde.fr/cosmos/)\n\n \n\nLe télescope James\\-Webb dcouvre sa première exoplanète, identifiée comme une petite planète froide\n====================================================================================================\n\n Lobservation a été faite grâce à une méthode prometteuse pour détecter des planètes dune taille similaire à celles du système solaire. Le Monde avec AFP \n\n Publié le 25 juin 2025 à 17h49, modifié le 26 juin 2025 à 03h24 Temps de Lecture 2 min. \n\n Lire plus tard Partager * Partager sur Messenger\n* Partager sur Facebook\n* Envoyer par e\\-mail\n* Partager sur Linkedin\n* Copier le lien\n\n <img src='https://img.lemde.fr/2025/06/25/0/0/866/867/664/0/75/0/1173df3_upload-1-g41rmpa9cg0l-902429.jpg' alt='Une image du disque protoplanétaire autour de l’étoile TWA 7, enregistrée à laide de linstrument Sphere du télescope basé au Chili, est présentée le 25\xa0juin 2025 avec une image superposée capturée par linstrument MIRI du télescope spatial James-Webb.' title='' width='664' height='443' /> ![Une image du disque protoplanétaire autour de l’étoile TWA 7, enregistrée à laide de linstrument Sphere du télescope basé au Chili, est présentée le 25\xa0juin 2025 avec une image superposée capturée par linstrument MIRI du télescope spatial James-Webb.](https://img.lemde.fr/2025/06/25/0/0/866/867/664/0/75/0/1173df3_upload-1-g41rmpa9cg0l-902429.jpg) \n\nUne image du disque protoplanétaire autour de l’étoile TWA 7, enregistrée à laide de linstrument Sphere du télescope basé au Chili, est présentée le 25\xa0juin 2025 avec une image superposée capturée par linstrument MIRI du télescope spatial James\\-Webb. WST/ESO/REUTERS \n\n Le télescope spatial James\\-Webb (JWST) a découvert sa première exoplanète dans lunivers proche. Une observation faite grâce à une méthode prometteuse pour détecter des planètes dune taille similaire à celles du système solaire.\n\n Depuis 2022, à 1,5\xa0million de kilomètres de la Terre, le JWST a aidé à caractériser plusieurs exoplanètes. *«\xa0Il a passé énormément de temps à observer des planètes qui nont jamais été imagées\xa0»*, explique lastrophysicienne Anne\\-Marie Lagrange, première autrice de l’étude sur le sujet, parue dans [*Nature*](https://www.nature.com/articles/s41550-024-02401-w?utm_source=chatgpt.com \"Nouvelle fenêtre\") mercredi 25\xa0juin.\n\n Lexercice est compliqué du fait que les exoplanètes *«\xa0sont très peu lumineuses parce quelles ne sont pas chaudes\xa0»*, mais aussi et surtout du fait qu\xa0on est aveuglé par la lumière de l’étoile autour de laquelle elles tournent\xa0»*, ajoute cette chercheuse du CNRS au Laboratoire dinstrumentation et de recherche en astrophysique de lObservatoire de Paris.\n\n La parade du James\\-Webb repose sur son coronographe, un instrument qui sinspire du phénomène de l’éclipse solaire en masquant l’étoile pour mieux révéler ce qui lentoure, et sur son spectrographe MIRI, capable dimager les astres les plus discrets grâce à une vision infrarouge. Ses utilisateurs ont pointé le télescope vers l’étoile TWA 7, située dans notre galaxie à une centaine dannées\\-lumière de la Terre, autrement dit sa très petite banlieue. La cible, initialement détectée par le télescope Hubble, était prometteuse à double titre.",
"score": 0.51039016,
}
result_2 = {
"url": "https://www.lemonde.fr/sciences/article/2025/06/25/le-telescope-james-webb-decouvre-sa-premiere-exoplanete-identifiee-comme-une-petite-planete-froide_6615888_1650684.html",
"content": "Le télescope James\\-Webb découvre sa première exoplanète, identifiée comme une petite planète froide",
"score": 0.49020067,
}
result_3 = {
"url": "https://www.lemonde.fr/sciences/article/2025/06/25/le-telescope-james-webb-decouvre-sa-premiere-exoplanete-identifiee-comme-une-petite-planete-froide_6615888_1650684.html",
"content": "---------------\n\n Dabord, parce quelle est jeune de seulement 6,4\xa0millions dannées et donc très susceptible de voir se former des corps planétaires dans le disque de matière la ceinturant. Ensuite, parce que le télescope voit ce disque protoplanétaire par le dessus. Son observation avec linstrument Sphere du Très Grand Télescope (VLT), situé au Chili, avait permis dy distinguer trois anneaux s’étageant sur une distance allant jusqu’à plus de cent fois celle séparant la Terre du Soleil.\n\n Et cest dans la partie dégarnie du deuxième anneau que linstrument du James\\-Webb a détecté une *«\xa0source\xa0»* lumineuse, baptisée TWA 7b. Ayant exclu que la découverte savère être un objet du système solaire ou une galaxie lointaine, les astronomes lont identifiée comme une petite planète froide, dune masse dix fois inférieure à celles imagées jusquici avec dautres instruments. Ils estiment sa masse comparable à celle de Saturne, une planète gazeuse qui ne *«\xa0pèse\xa0»* que le tiers de Jupiter, géante gazeuse et poids lourd de notre système solaire.\n\n Avec le James\\-Webb, *«\xa0on est tombé dun facteur dix en capacité de détection\xa0»*, explique Anne\\-Marie Lagrange, car les planètes les plus *«\xa0légères\xa0»* imagées jusquici depuis le sol pesaient à peu près trois fois la masse de Jupiter. *«\xa0La plupart des autres exoplanètes imagées sont ce quon appelle des super\\-Jupiter\xa0»*, ayant de huit à douze fois la masse de cette dernière.\n\n Lire aussi \\| Article réservé à nos abonnés [Potentielle présence de vie sur lexoplanète K2\\-18b\xa0: anatomie dun faux positif](https://www.lemonde.fr/sciences/article/2025/05/13/potentielle-presence-de-vie-sur-l-exoplanete-k2-18b-anatomie-d-un-faux-positif_6605670_1650684.html) Lire plus tard \n\nUn télescope prometteur attendu pour 2028",
"score": 0.48312354,
}
result_4 = {
"url": "https://www.lemonde.fr/sciences/article/2025/06/25/le-telescope-james-webb-decouvre-sa-premiere-exoplanete-identifiee-comme-une-petite-planete-froide_6615888_1650684.html",
"content": "-----------------------------------------\n\n La performance a dautant plus dintérêt que dans le bestiaire planétaire, les planètes rocheuses comme la Terre ou Mars ont des masses beaucoup plus faibles que les planètes gazeuses. Or ces exoplanètes rocheuses constituent une cible ultime des découvreurs de mondes potentiellement habitables.\n\n Le Monde Guides dachat [Aspirateurs robots Les meilleurs aspirateurs robots Lire](https://www.lemonde.fr/guides-d-achat/article/2021/10/25/les-meilleurs-aspirateurs-robots_6099813_5306571.html) Anne\\-Marie Lagrange souhaiterait désormais *«\xa0découvrir les planètes les plus légères et peut\\-être de trouver des Terres\xa0»*. Avant dajouter aussitt que si *«\xa0on veut comprendre comment les systèmes planétaires se forment, il ne suffit pas de voir les planètes très ou pas massives\xa0»*. Car il faut pouvoir détecter tous les types de planètes, afin de déterminer in fine si notre système solaire est unique ou pas.\n\n Les astronomes estiment que le JWST a le potentiel de détecter et dimager des planètes ayant une masse encore plus faible que TWA 7b. Mais il faudra de futurs instruments, comme le Télescope extrêmement large (ELT) attendu pour 2028, pour espérer saisir limage de mondes dune taille similaire au nôtre.\n\n Lire aussi \\| Article réservé à nos abonnés [La traque des axions, nouvelle coqueluche des physiciens](https://www.lemonde.fr/sciences/article/2025/06/09/la-traque-des-axions-nouvelle-coqueluche-des-physiciens_6611759_1650684.html) Lire plus tard Le Monde avec AFP \n\n Lespace des contributions est réservé aux abonnés. Abonnez\\-vous pour accéder à cet espace d’échange et contribuer à la discussion. [Sabonner](https://abo.lemonde.fr/?lmd_medium=BOUTONS_LMFR&lmd_campaign=CONTRIBUTION_ARTICLE) Contribuer\n\n [Réutiliser ce contenu](https://www.lemonde.fr/syndication/ \"Réutiliser ce contenu\") Édition du jour\n\n Daté du jeudi 31 juillet\n\n <img src='data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 276 201'%3E%3C/svg%3E' alt='' title='' width='276' height='201' /> [Lire le journal numérique](https://journal.lemonde.fr/) [<img src='' alt='' title='' width='' height='36' /> ![]() Culture générale Des leçons interactives par la rédaction pour tester vos connaissances. Découvrir](https://www.lemonde.fr/memorable/quiz-et-questions-de-culture-generale) [<img src='' alt='' title='' width='300' height='280' />](https://www.lemonde.fr/chaleur-humaine/article/2025/01/30/mesurez-votre-impact-environnement-avec-le-calculateur-d-empreinte-carbone-et-eau_6523433_6125299.html?lmd_medium=bizdev&lmd_campaign=services_partenaire_lmfr&lmd_creation=ademe)",
"score": 0.42882082,
}
result_5 = {
"url": "https://www.franceinfo.fr/economie/budget/",
"content": "Budget 2025 de la France \\- actualité et info en direct\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\ncontenu principal\n\n\n\n\n\n[<img src='https://www.franceinfo.fr/assets/components/headers/header/img/franceinfo-1d7b76a5.svg' alt='' title='' width='161' height='40' />\n\n<img src='data:image/svg+xml,%3Csvg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 161 40\"%3E%3C/svg%3E' alt='' title='' width='161' height='40' />\nAccueil France Info](/ \"Accueil France Info\")\n\n* [Rechercher une actualité\nRecherche](/recherche/)\n* [Direct TV\nTV](/en-direct/tv.html)\n* [Direct radio\nRadio](/en-direct/radio.html)\n* [Le live\nLive](/en-direct/)\n* Services\n\n\n\n\n\t+ [Newsletters\n\t (Nouvelle fenêtre)](https://www.francetelevisions.fr/abonnements/information)\n\t+ [Météo\n\t (Nouvelle fenêtre)](https://meteo.franceinfo.fr/)\n\t+ [Jeux\n\t (Nouvelle fenêtre)](https://jeux.franceinfo.fr)\n\t+ [Scolarité](/societe/education/scolarite/)\n* Mon\xa0espace\n\n\n\n\n\t+ [S'inscrire](javascript:void(0))\n\t+ [Se connecter](javascript:void(0))\n\t+ [À lire plus tard](/mon-compte/lire-plus-tard)\n\t+ [Mes commentaires](/mon-compte/interactions)\n\t+ [Mes newsletters](/mon-compte/newsletters)\n\t+ [Mes informations](javascript:void(0))\n\t+ [Se déconnecter](javascript:void(0))\n\n\n\n\n\n\n* [Accueil](/)\n* Menu\n* [Grands formats](/grands-formats/)\n* [Enquêtes](/enquetes-franceinfo/)\n* [Vrai ou faux](/vrai-ou-fake/)\n* [Droits de douane](/monde/usa/droits-de-douane/)\n* [Guerre dans la bande de Gaza](/monde/proche-orient/israel-palestine/)\n* [Loi Duplomb](/environnement/loi-duplomb/)\n\n\n\n\n\n\n\n<img src='https://www.franceinfo.fr/assets/components/headers/header/img/franceinfo-1d7b76a5.svg' alt='France Info' title='' width='129' height='32' />\n\n<img src='data:image/svg+xml,%3Csvg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 129 32\"%3E%3C/svg%3E' alt='France Info' title='' width='129' height='32' />\n\n\n\n\n\n\n\n\n\n\n\nRechercher sur franceinfo\n\n\n\n\n\n\nAnnuler la saisie\n\n\n\n\n* [Accueil](/)\n\n\n\n\n En ce moment\n \n\n* [Droits de douane](/monde/usa/droits-de-douane/)\n* [Guerre dans la bande de Gaza](/monde/proche-orient/israel-palestine/)\n* [Loi Duplomb](/environnement/loi-duplomb/)\n\n\n\n\n* [Grands formats](/grands-formats/)\n* [Enquêtes](/enquetes-franceinfo/)\n* [Vrai ou faux](/vrai-ou-fake/)\n\n\n\n\n Rubriques\n \n\n* [L'actu pour les jeunes](/l-actu-pour-les-jeunes/)\n* [Une info transparente](/une-information-transparente-franceinfo/)\n* Monde\n* Europe\n* Faits\\-divers\n* Politique\n* Société\n* Environnement\n* Sport\n* Culture\n* Éco / Conso\n* Santé\n* Sciences \\& technologies\n* Météo\nMétéo\n* Jeux\nJeux\n\n\n\n* Services\n\n\n\nRecevez l'essentiel de l'actualité et restez à jour avec nos newsletters\n\n\n[découvrir nos newsletters\n(Nouvelle fenêtre)](https://www.francetelevisions.fr/abonnements/information)\n\n\n\n Continuez votre exploration\n \n\n* [France 3 régions (nouvelle fenêtre)\nFrance 3 régions](https://france3-regions.franceinfo.fr/)\n* [Outremer la 1ère (nouvelle fenêtre)\nOutremer la 1ère](https://la1ere.franceinfo.fr/)\n* france TV\nfrance TV\n* radio france\nradio france\n\n\n\n\n\n\nServices\n\nServices\n* [Newsletters (nouvelle fenêtre)\nNewsletters](https://www.francetelevisions.fr/abonnements/information)\n* [Météo (nouvelle fenêtre)\nMétéo](https://meteo.franceinfo.fr/)\n* [Jeux (nouvelle fenêtre)\nJeux](https://jeux.franceinfo.fr)\n* [Scolarité](/societe/education/scolarite/)\n\n\n\n\n\n\nMonde\n\nMonde\n\n[voir toute l'actu Monde](/monde/)\n\n\n\n En ce moment\n \n\n* [Droits de douane](/monde/usa/droits-de-douane/)\n* [Guerre dans la bande de Gaza](/monde/proche-orient/israel-palestine/)\n* [Guerre en Ukraine](/monde/europe/manifestations-en-ukraine/)\n* [Affaire Jeffrey Epstein](/monde/usa/affaire-jeffrey-epstein/)\n* [Guerre au Proche\\-Orient](/monde/proche-orient/guerre/)\n* [Donald Trump \\- Président des États\\-Unis](/monde/usa/presidentielle/donald-trump/)\n* [Yémen](/monde/proche-orient/yemen/)\n* [Voir plus de sujets monde](/monde/)\n\n\n\n\n Rubriques",
"score": 0.6129482,
}
class MockedWebSearchManager:
"""
A class to manage web search operations.
This is an abstract base class that should be implemented
for specific web search managers.
"""
def web_search(self, query: str) -> RAGWebResults:
"""
Fake a web search.
Args:
query (str): The search query.
Returns:
RAGWebResults: A Searches object containing the search results.
"""
logger.info("Mocked web search called with query: %s", query)
return RAGWebResults(
data=[
RAGWebResult(**result_1),
RAGWebResult(**result_2),
RAGWebResult(**result_3),
RAGWebResult(**result_4),
RAGWebResult(**result_5),
],
usage=RAGWebUsage(
prompt_tokens=0, # Assuming no prompt tokens are provided in this mock
completion_tokens=0, # Assuming no completion tokens are provided in this mock
), # Assuming no usage data is provided in this mock
)

Some files were not shown because too many files have changed in this diff Show More