Compare commits

..

22 Commits

Author SHA1 Message Date
charles 4800ef442e 🚨(backend) fix tests
I am removing hard coded datetime.
2026-01-21 11:13:18 +01:00
charles e7fb73c53e 🚨(backend) various fixes
I fix linters and rabbit request change
2026-01-20 19:27:24 +01:00
charles 398f692dd1 (backend) handle deleting temporary collections
I handle deleting document in temporary collection
for web_search_brave_with_document_backend
2026-01-20 19:08:00 +01:00
charles aa898a8589 🚨(backend) various review fixes
I am doing various small review fixies
2026-01-20 19:08:00 +01:00
charles f88a80d93f 🚨(backend) various fixes
I am proofreading myself
2026-01-20 19:07:58 +01:00
charles 40d1d8cc24 ♻️(backend) refactor document parsers
I refactor document parsing by introducing
AlbertParser and BaseParser
2026-01-20 19:07:20 +01:00
charles 0abe12382a (backend) enhance Find API integration with user sub and tag
I enhance Find API integration with user
access control and configuration options
2026-01-20 19:07:20 +01:00
charles 713b34fdcd 🧪(backend) tests
I add tests to test the app.
2026-01-20 19:07:20 +01:00
charles b62fffc69d (backend) implement FindRagBackend
We want to be able to use Find api in rag tools.
I add a new rag backend class to do so.
2026-01-20 19:07:20 +01:00
Eléonore Voisin 3232da72c5 Revert "🐛(front) optimize chat"
This reverts commit 69bf2cab5d.
2026-01-20 19:07:01 +01:00
Eléonore Voisin 944d69aede 🐛(front) optimize chat
Simplified chat rendering
2026-01-20 19:06:41 +01:00
Quentin BEY 09b003856b 🔒️(node-packages) update fixed CVE packages
Trivy complains about some packages with fixed CVE,
we update them.
2026-01-19 16:09:07 +01:00
Quentin BEY 0b5317a773 🔒️(jaraco) enforce version to fix CVE
Vulnerability in jaraco.context caused security issue
in setuptools and python3. change python version to fix
see GHSA-58pv-8j8x-9vj2

The CVE is not actionable, anyway, we want to please
trivy.
2026-01-19 14:38:59 +01:00
Quentin BEY abf61a9556 🔥(chat) consider PDF documents as other kind of documents
We remove the specific management for PDF because it introduces:
 - limitation regarding the LLM we can use
 - bad behavior when uploading huge PDFs
 - more code complexity
while not providing really actionnable improvements.

This commit removes this, to keep a better control over this.
2026-01-19 14:04:32 +01:00
Laurent Paoletti 3e8c5c77d5 (chat) generate and edit conversation title
- Auto-generate title via LLM after reaching user message threshold
- Add title_set_by_user_at field to track user-customized titles
- Skip auto-generation when user has set a custom title
- Stream conversation_metadata event to frontend on title update
- Invalidate React Query cache to refresh conversation list

Signed-off-by: Laurent Paoletti <lp@providenz.fr>
2026-01-18 23:04:43 +01:00
Laurent Paoletti ddfc86a88f 🐛(back) stream tool responses to prevent too call timeouts
Implement sync/sync utilities that inject
keepalive messages at regular intervals during stream pauses,
preventing proxy timeouts on long-running operations like
document(s) summarization.

Keepalive messages maintain active connections while tools execute,
eliminating forced conversation restarts.

Signed-off-by: Laurent Paoletti <lp@providenz.fr>
2026-01-17 13:50:35 +01:00
qbey e7d76e4477 🌐(i18n) update translated strings
Update translated files with new translations
2026-01-16 12:13:55 +01:00
Quentin BEY fd3399dd66 🔖(patch) bump release to 0.0.11
Changed

- 📦️(front) update react

Fixed

- 🐛(e2e) fix test-e2e-chromium
- 🐛(back) fix system prompt compatibility with self-hosted models #200
- ⚰️(back) remove dead code and unused files

Removed

- 🔥(chat) remove thinking part from frontend #227
2026-01-16 12:03:05 +01:00
Quentin BEY 13c6499c66 🔥(chat) remove thinking part from frontend
We want to enable the OSS model but seems like it returns
thinking values twice and we don't manage it well...

So we disable the frontend while we still don't know
how to display the thinking stuff.
We could have also cleaned the backend while unused.
2026-01-16 11:43:05 +01:00
Berry den Hartog a0b31e1e61 🐛(front) fix link color in LeftPanelConversationItem component
fix link color component for default theme
2026-01-16 11:43:05 +01:00
Laurent Paoletti daf90cf110 ⚰️(back) remove dead code and unused files
Signed-off-by: Laurent Paoletti <lp@providenz.fr>
2026-01-16 11:43:05 +01:00
Laurent Paoletti 29f76fe040 🐛(back) fix system prompt compatibility with self-hosted models
Pydantic AI allows setting multiple static and dynamic system prompts
to define conversation context and rules. Previously, these were sent
to the model API as separate messages, which caused compatibility
issues with some self-hosted models (e.g., Gemma3/vLLM).

This commit switches from using `system_prompt` to `instruction` as
recommended in the Pydantic AI documentation, thus merging several
instructions into a single message.

Reference: https://ai.pydantic.dev/agents/#system-prompts
Signed-off-by: Laurent Paoletti <lp@providenz.fr>
2026-01-16 11:43:05 +01:00
64 changed files with 2511 additions and 1464 deletions
+3
View File
@@ -44,6 +44,9 @@ env.d/development/*
!env.d/development/*.dist
env.d/terraform
# Configuration
**/conversations/configuration/llm/dev.json
# npm
node_modules
+18 -1
View File
@@ -8,15 +8,31 @@ and this project adheres to
## [Unreleased]
### Added
- ✨(backend) add FindRagBackend
### Removed
- 🔥(chat) consider PDF documents as other kind of documents #234
## [0.0.11] - 2026-01-16
### Changed
- 📦️(front) update react
- ✨(chat) Generate and edit conversation title
### Fixed
- 🐛(e2e) fix test-e2e-chromium
- 🐛(back) fix system prompt compatibility with self-hosted models #200
- ⚰️(back) remove dead code and unused files
- 🐛(back) prevent tool call timeouts
### Removed
- 🔥(chat) remove thinking part from frontend #227
## [0.0.10] - 2025-12-15
@@ -172,7 +188,8 @@ and this project adheres to
- 💄(chat) add code highlighting for LLM responses #67
[unreleased]: https://github.com/suitenumerique/conversations/compare/v0.0.10...main
[unreleased]: https://github.com/suitenumerique/conversations/compare/v0.0.11...main
[0.0.11]: https://github.com/suitenumerique/conversations/releases/v0.0.11
[0.0.10]: https://github.com/suitenumerique/conversations/releases/v0.0.10
[0.0.9]: https://github.com/suitenumerique/conversations/releases/v0.0.9
[0.0.8]: https://github.com/suitenumerique/conversations/releases/v0.0.8
+1 -1
View File
@@ -4,7 +4,7 @@
FROM python:3.13.3-alpine AS base
# Upgrade pip to its latest release to speed up dependencies installation
RUN python -m pip install --upgrade pip setuptools
RUN python -m pip install --upgrade pip
# Upgrade system packages to install security updates
RUN apk update && \
+11
View File
@@ -71,6 +71,9 @@ services:
- "host.docker.internal:host-gateway"
ports:
- "8071:8000"
networks:
- default
- lasuite
volumes:
- ./src/backend:/app
- ./data/static:/data/static
@@ -89,6 +92,9 @@ services:
image: nginx:1.25
ports:
- "8083:8083"
networks:
- default
- lasuite
volumes:
- ./docker/files/etc/nginx/conf.d:/etc/nginx/conf.d:ro
depends_on:
@@ -177,3 +183,8 @@ services:
kc_postgresql:
condition: service_healthy
restart: true
networks:
lasuite:
name: lasuite-network
driver: bridge
+3
View File
@@ -95,6 +95,9 @@ These are the environment variables you can set for the `conversations-backend`
| CACHES_KEY_PREFIX | The prefix used to every cache keys. | conversations |
| THEME_CUSTOMIZATION_FILE_PATH | full path to the file customizing the theme. An example is provided in src/backend/conversations/configuration/theme/default.json | BASE_DIR/conversations/configuration/theme/default.json |
| THEME_CUSTOMIZATION_CACHE_TIMEOUT | Cache duration for the customization settings | 86400 |
| FIND_API_KEY | API key of Find | |
| FIND_API_URL | URL of Find | `https://app-find/api` |
| FIND_API_TIMEOUT | Find API timeout | 30 |
## conversations-frontend image
+3 -3
View File
@@ -244,9 +244,9 @@ For Mistral AI models using the Etalab platform:
{
"models": [
{
"hrid": "mistral-large",
"model_name": "mistral-large-latest",
"human_readable_name": "Mistral Large (Etalab)",
"hrid": "mistral-medium",
"model_name": "mistral-medium-2508",
"human_readable_name": "Mistral Medium (Etalab)",
"provider_name": "mistral-etalab",
"profile": null,
"settings": {
+1
View File
@@ -357,6 +357,7 @@ The RAG backend performs semantic search to find the most relevant content:
rag_results = document_store.search(
query,
results_count=settings.BRAVE_RAG_WEB_SEARCH_CHUNK_NUMBER,
**kwargs, # Additional search parameters like session with access_token
)
```
+2
View File
@@ -8,3 +8,5 @@ LLM_CONFIGURATION_FILE_PATH = /app/conversations/configuration/llm/default.e2e.j
# Features
FEATURE_FLAG_WEB_SEARCH=ENABLED
FEATURE_FLAG_DOCUMENT_UPLOAD=ENABLED
AUTO_TITLE_AFTER_USER_MESSAGES=3
@@ -0,0 +1,66 @@
"""Document parsers for RAG backends."""
import logging
from io import BytesIO
from urllib.parse import urljoin
from django.conf import settings
import requests
from chat.agent_rag.document_converter.markitdown import DocumentConverter
logger = logging.getLogger(__name__)
class BaseParser:
"""Base class for document parsers."""
def parse_document(self, name: str, content_type: str, content: BytesIO) -> str:
"""
Parse the document and prepare it for the search operation.
This method should handle the logic to convert the document
into a format suitable for storage.
Args:
name (str): The name of the document.
content_type (str): The MIME type of the document (e.g., "application/pdf").
content (BytesIO): The content of the document as a BytesIO stream.
Returns:
str: The document content in Markdown format.
"""
raise NotImplementedError("Must be implemented in subclass.")
class AlbertParser(BaseParser):
"""Document parser using Albert API for PDFs and DocumentConverter for other formats."""
endpoint = urljoin(settings.ALBERT_API_URL, "/v1/parse-beta")
def parse_pdf_document(self, name: str, content_type: str, content: bytes) -> str:
"""Parse PDF document using Albert API."""
response = requests.post(
self.endpoint,
headers={
"Authorization": f"Bearer {settings.ALBERT_API_KEY}",
},
files={
"file": (name, content, content_type),
"output_format": (None, "markdown"),
},
timeout=settings.ALBERT_API_PARSE_TIMEOUT,
)
response.raise_for_status()
return "\n\n".join(
document_page["content"] for document_page in response.json().get("data", [])
)
def parse_document(self, name: str, content_type: str, content: bytes) -> str:
"""Parse document based on content type."""
if content_type == "application/pdf":
return self.parse_pdf_document(name=name, content_type=content_type, content=content)
return DocumentConverter().convert_raw(
name=name, content_type=content_type, content=content
)
@@ -13,174 +13,11 @@ import requests
from chat.agent_rag.albert_api_constants import Searches
from chat.agent_rag.constants import RAGWebResult, RAGWebResults, RAGWebUsage
from chat.agent_rag.document_converter.markitdown import DocumentConverter
from chat.agent_rag.document_converter.parser import AlbertParser
from chat.agent_rag.document_rag_backends.base_rag_backend import BaseRagBackend
logger = logging.getLogger(__name__)
# Albert API token limit for document vectorization
# We use a conservative chunk size to stay well under the limit
ALBERT_MAX_TOKENS = 8192
ALBERT_CHUNK_SIZE_TOKENS = 5000 # More conservative chunk size with larger safety margin
# Approximate tokens: ~3 characters per token (more conservative estimate for Markdown/Excel)
# Markdown and Excel content often have more tokens per character due to formatting
ALBERT_CHUNK_SIZE_CHARS = ALBERT_CHUNK_SIZE_TOKENS * 3
def _estimate_tokens(content: str) -> int:
"""
Estimate the number of tokens in a text string.
Uses a conservative approximation: ~3 characters per token.
This is more conservative than 4 chars/token to account for:
- Markdown formatting (headers, lists, tables)
- Excel content with special characters
- Whitespace and punctuation
Args:
content (str): The text content to estimate.
Returns:
int: Estimated number of tokens.
"""
return len(content) // 3
def _chunk_content(content: str, max_chars: int = ALBERT_CHUNK_SIZE_CHARS) -> List[str]:
"""
Split content into chunks that fit within Albert's token limit.
Attempts to split at paragraph boundaries (double newlines) when possible,
otherwise splits at line boundaries, and finally at character boundaries.
Validates that each chunk is under the token limit after splitting.
Args:
content (str): The content to chunk.
max_chars (int): Maximum characters per chunk (default: ALBERT_CHUNK_SIZE_CHARS).
Returns:
list[str]: List of content chunks, each under the token limit.
"""
# First check if content fits in one chunk
estimated_tokens = _estimate_tokens(content)
if estimated_tokens <= ALBERT_CHUNK_SIZE_TOKENS:
return [content]
chunks = []
remaining = content
while len(remaining) > 0:
# Check if remaining content fits in one chunk
remaining_tokens = _estimate_tokens(remaining)
if remaining_tokens <= ALBERT_CHUNK_SIZE_TOKENS:
if remaining.strip():
chunks.append(remaining.strip())
break
# Need to split - find the best split point
# Start with max_chars but may need to reduce if token estimate is too high
search_limit = max_chars
# Try to find a split point that keeps us under token limit
# Reduce search limit if needed to ensure token limit is respected
while search_limit > 100: # Minimum chunk size
# Try to split at paragraph boundary (double newline)
split_pos = remaining.rfind("\n\n", 0, search_limit)
if split_pos == -1:
# Try to split at single newline
split_pos = remaining.rfind("\n", 0, search_limit)
if split_pos == -1:
# Force split at character boundary
split_pos = search_limit
# Validate that this chunk is under token limit
chunk_candidate = remaining[:split_pos].strip()
if chunk_candidate:
chunk_tokens = _estimate_tokens(chunk_candidate)
if chunk_tokens <= ALBERT_CHUNK_SIZE_TOKENS:
chunks.append(chunk_candidate)
remaining = remaining[split_pos:].lstrip()
break
# Chunk too large, reduce search limit and try again
search_limit = int(search_limit * 0.8) # Reduce by 20%
else:
# Fallback: force split at a safe size
# This should rarely happen, but ensures we don't get stuck
safe_size = min(max_chars, len(remaining))
chunk = remaining[:safe_size].strip()
if chunk:
chunks.append(chunk)
remaining = remaining[safe_size:].lstrip()
# Validate all chunks are under limit and split further if needed
validated_chunks = []
for chunk_item in chunks:
chunk_tokens = _estimate_tokens(chunk_item)
if chunk_tokens > ALBERT_MAX_TOKENS:
logger.warning(
"Chunk still exceeds token limit (%d tokens, max: %d), forcing split further",
chunk_tokens,
ALBERT_MAX_TOKENS,
)
# Force split this chunk further using a more conservative size
# Use a size that ensures we stay well under the token limit
# Target: ~5000 tokens max per chunk (conservative)
max_safe_chars = ALBERT_CHUNK_SIZE_TOKENS * 3 # 6000 * 3 = 18000 chars for ~5000 tokens
remaining_chunk = chunk_item
while len(remaining_chunk) > 0:
remaining_tokens = _estimate_tokens(remaining_chunk)
if remaining_tokens <= ALBERT_CHUNK_SIZE_TOKENS:
if remaining_chunk.strip():
validated_chunks.append(remaining_chunk.strip())
break
# Find a safe split point
split_pos = min(max_safe_chars, len(remaining_chunk))
# Try to split at a line boundary if possible
line_split = remaining_chunk.rfind("\n", 0, split_pos)
if line_split > max_safe_chars * 0.5: # Only use if it's not too small
split_pos = line_split
sub_chunk = remaining_chunk[:split_pos].strip()
if sub_chunk:
sub_tokens = _estimate_tokens(sub_chunk)
# Double-check this sub-chunk is safe
if sub_tokens > ALBERT_MAX_TOKENS:
# Still too large, use even smaller size
logger.warning(
"Sub-chunk still too large (%d tokens), using smaller split",
sub_tokens,
)
split_pos = ALBERT_CHUNK_SIZE_TOKENS * 2 # 12000 chars for ~3000 tokens
sub_chunk = remaining_chunk[:split_pos].strip()
validated_chunks.append(sub_chunk)
remaining_chunk = remaining_chunk[split_pos:].lstrip()
else:
validated_chunks.append(chunk_item)
# Final validation - ensure NO chunk exceeds the limit
final_chunks = []
for chunk in validated_chunks:
chunk_tokens = _estimate_tokens(chunk)
if chunk_tokens > ALBERT_MAX_TOKENS:
logger.error(
"CRITICAL: Chunk still exceeds limit after all splitting attempts: %d tokens",
chunk_tokens,
)
# Emergency split: use very conservative size
emergency_size = ALBERT_CHUNK_SIZE_TOKENS * 2 # 12000 chars
remaining = chunk
while len(remaining) > 0:
emergency_chunk = remaining[:emergency_size].strip()
if emergency_chunk:
final_chunks.append(emergency_chunk)
remaining = remaining[emergency_size:].lstrip()
else:
final_chunks.append(chunk)
return final_chunks
class AlbertRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-attributes
"""
@@ -189,9 +26,6 @@ class AlbertRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-att
It provides methods to:
- Create a collection for the search operation.
- Parse documents and convert them to Markdown format:
+ Handle PDF parsing using the Albert API.
+ Use the DocumentConverter (markitdown) for other formats.
- Store parsed documents in the Albert collection.
- Perform a search operation using the Albert API.
"""
@@ -209,10 +43,9 @@ class AlbertRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-att
}
self._collections_endpoint = urljoin(self._base_url, "/v1/collections")
self._documents_endpoint = urljoin(self._base_url, "/v1/documents")
self._pdf_parser_endpoint = urljoin(self._base_url, "/v1/parse-beta")
self._search_endpoint = urljoin(self._base_url, "/v1/search")
self._default_collection_description = "Temporary collection for RAG document search"
self.parser = AlbertParser()
def create_collection(self, name: str, description: Optional[str] = None) -> str:
"""
@@ -254,7 +87,7 @@ class AlbertRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-att
self.collection_id = str(response.json()["id"])
return self.collection_id
def delete_collection(self) -> None:
def delete_collection(self, **kwargs) -> None:
"""
Delete the current collection
"""
@@ -265,7 +98,7 @@ class AlbertRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-att
)
response.raise_for_status()
async def adelete_collection(self) -> None:
async def adelete_collection(self, **kwargs) -> None:
"""
Asynchronously delete the current collection
"""
@@ -277,101 +110,15 @@ class AlbertRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-att
)
response.raise_for_status()
def parse_pdf_document(self, name: str, content_type: str, content: BytesIO) -> str:
"""
Parse the PDF document content and return the text content.
This method should handle the logic to convert the PDF into
a format suitable for the Albert API.
"""
response = requests.post(
self._pdf_parser_endpoint,
headers=self._headers,
files={
"file": (
name,
content,
content_type,
), # Use the name as the filename in the request
"output_format": (None, "markdown"), # Specify the output format as Markdown,
},
timeout=settings.ALBERT_API_PARSE_TIMEOUT,
)
response.raise_for_status()
return "\n\n".join(
document_page["content"] for document_page in response.json().get("data", [])
)
def parse_document(self, name: str, content_type: str, content: BytesIO):
"""
Parse the document and prepare it for the search operation.
This method should handle the logic to convert the document
into a format suitable for the Albert API.
Args:
name (str): The name of the document.
content_type (str): The MIME type of the document (e.g., "application/pdf").
content (BytesIO): The content of the document as a BytesIO stream.
Returns:
str: The document content in Markdown format.
"""
# Implement the parsing logic here
if content_type == "application/pdf":
# Handle PDF parsing
markdown_content = self.parse_pdf_document(
name=name, content_type=content_type, content=content
)
else:
markdown_content = DocumentConverter().convert_raw(
name=name, content_type=content_type, content=content
)
return markdown_content
def store_document(self, name: str, content: str) -> None:
def store_document(self, name: str, content: str, **kwargs) -> None:
"""
Store the document content in the Albert collection.
This method should handle the logic to send the document content to the Albert API.
If the document is too large (exceeds Albert's token limit), it will be automatically
split into multiple chunks and stored as separate documents.
Args:
name (str): The name of the document.
content (str): The content of the document in Markdown format.
"""
# Check if content needs to be chunked
estimated_tokens = _estimate_tokens(content)
if estimated_tokens > ALBERT_MAX_TOKENS:
logger.info(
"Document '%s' is too large (%d estimated tokens, limit: %d). "
"Splitting into chunks.",
name,
estimated_tokens,
ALBERT_MAX_TOKENS,
)
chunks = _chunk_content(content)
logger.info("Split document '%s' into %d chunks", name, len(chunks))
# Store each chunk as a separate document
for i, chunk in enumerate(chunks, start=1):
chunk_name = f"{name}_part_{i}" if len(chunks) > 1 else name
self._store_single_document(chunk_name, chunk)
else:
# Document fits within limit, store as-is
self._store_single_document(name, content)
def _store_single_document(self, name: str, content: str) -> None:
"""
Store a single document chunk in the Albert collection.
Internal method that performs the actual API call to store one document.
Args:
name (str): The name of the document.
content (str): The content of the document in Markdown format.
**kwargs: Additional arguments.
"""
response = requests.post(
urljoin(self._base_url, self._documents_endpoint),
@@ -383,71 +130,18 @@ class AlbertRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-att
},
timeout=settings.ALBERT_API_TIMEOUT,
)
logger.debug("Stored document '%s': %s", name, response.json())
logger.debug(response.json())
response.raise_for_status()
async def astore_document(self, name: str, content: str) -> None:
async def astore_document(self, name: str, content: str, **kwargs) -> None:
"""
Store the document content in the Albert collection.
This method should handle the logic to send the document content to the Albert API.
If the document is too large (exceeds Albert's token limit), it will be automatically
split into multiple chunks and stored as separate documents.
Args:
name (str): The name of the document.
content (str): The content of the document in Markdown format.
"""
# Check if content needs to be chunked
estimated_tokens = _estimate_tokens(content)
if estimated_tokens > ALBERT_MAX_TOKENS:
logger.info(
"Document '%s' is too large (%d estimated tokens, limit: %d). "
"Splitting into chunks.",
name,
estimated_tokens,
ALBERT_MAX_TOKENS,
)
chunks = _chunk_content(content)
logger.info("Split document '%s' into %d chunks", name, len(chunks))
# Validate chunks before storing
for i, chunk in enumerate(chunks, start=1):
chunk_tokens = _estimate_tokens(chunk)
logger.debug(
"Chunk %d/%d: %d chars, ~%d tokens",
i,
len(chunks),
len(chunk),
chunk_tokens,
)
if chunk_tokens > ALBERT_MAX_TOKENS:
logger.error(
"Chunk %d/%d still exceeds token limit: %d tokens (max: %d)",
i,
len(chunks),
chunk_tokens,
ALBERT_MAX_TOKENS,
)
# Store each chunk as a separate document
for i, chunk in enumerate(chunks, start=1):
chunk_name = f"{name}_part_{i}" if len(chunks) > 1 else name
await self._astore_single_document(chunk_name, chunk)
else:
# Document fits within limit, store as-is
await self._astore_single_document(name, content)
async def _astore_single_document(self, name: str, content: str) -> None:
"""
Store a single document chunk in the Albert collection.
Internal method that performs the actual API call to store one document.
Args:
name (str): The name of the document.
content (str): The content of the document in Markdown format.
**kwargs: Additional arguments.
"""
async with httpx.AsyncClient(timeout=settings.ALBERT_API_TIMEOUT) as client:
response = await client.post(
@@ -462,16 +156,17 @@ class AlbertRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-att
},
timeout=settings.ALBERT_API_TIMEOUT,
)
logger.debug("Stored document '%s': %s", name, response.json())
logger.debug(response.json())
response.raise_for_status()
def search(self, query, results_count: int = 4) -> RAGWebResults:
def search(self, query: str, results_count: int = 4, **kwargs) -> RAGWebResults:
"""
Perform a search using the Albert API based on the provided query.
Args:
query (str): The search query.
results_count (int): The number of results to return.
**kwargs: Additional arguments.
Returns:
RAGWebResults: The search results.
@@ -508,13 +203,14 @@ class AlbertRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-att
),
)
async def asearch(self, query, results_count: int = 4) -> RAGWebResults:
async def asearch(self, query, results_count: int = 4, **kwargs) -> RAGWebResults:
"""
Perform an asynchronous search using the Albert API based on the provided query.
Args:
query (str): The search query.
results_count (int): The number of results to return.
**kwargs: Additional arguments.
Returns:
RAGWebResults: The search results.
@@ -1,6 +1,7 @@
"""Implementation of the Albert API for RAG document search."""
import logging
from abc import ABC, abstractmethod
from contextlib import asynccontextmanager, contextmanager
from io import BytesIO
from typing import List, Optional
@@ -8,11 +9,12 @@ from typing import List, Optional
from asgiref.sync import sync_to_async
from chat.agent_rag.constants import RAGWebResults
from chat.agent_rag.document_converter.parser import BaseParser
logger = logging.getLogger(__name__)
class BaseRagBackend:
class BaseRagBackend(ABC):
"""Base class for RAG backends."""
def __init__(
@@ -38,6 +40,7 @@ class BaseRagBackend:
self.collection_id = collection_id
self.read_only_collection_id = read_only_collection_id or []
self._default_collection_description = "Temporary collection for RAG document search"
self.parser: BaseParser = BaseParser()
def get_all_collection_ids(self) -> List[str]:
"""
@@ -53,13 +56,14 @@ class BaseRagBackend:
collection_ids = []
if self.collection_id:
collection_ids.append(int(self.collection_id))
collection_ids.append(self.collection_id)
if self.read_only_collection_id:
collection_ids.extend(
[int(collection_id) for collection_id in self.read_only_collection_id]
)
return collection_ids
@abstractmethod
def create_collection(self, name: str, description: Optional[str] = None) -> str:
"""
Create a temporary collection for the search operation.
@@ -88,9 +92,10 @@ class BaseRagBackend:
Returns:
str: The document content in Markdown format.
"""
raise NotImplementedError("Must be implemented in subclass.")
return self.parser.parse_document(name, content_type, content)
def store_document(self, name: str, content: str) -> None:
@abstractmethod
def store_document(self, name: str, content: str, **kwargs) -> None:
"""
Store the document content in the collection.
This method should handle the logic to send the document content to the API.
@@ -98,10 +103,11 @@ class BaseRagBackend:
Args:
name (str): The name of the document.
content (str): The content of the document in Markdown format.
**kwargs: Additional arguments. ex: "user_sub" for access control.
"""
raise NotImplementedError("Must be implemented in subclass.")
async def astore_document(self, name: str, content: str) -> None:
async def astore_document(self, name: str, content: str, **kwargs) -> None:
"""
Store the document content in the collection.
This method should handle the logic to send the document content to the API.
@@ -109,10 +115,13 @@ class BaseRagBackend:
Args:
name (str): The name of the document.
content (str): The content of the document in Markdown format.
**kwargs: Additional arguments. ex: "user_sub" for access control.
"""
return await sync_to_async(self.store_document)(name=name, content=content)
return await sync_to_async(self.store_document)(name=name, content=content, **kwargs)
def parse_and_store_document(self, name: str, content_type: str, content: BytesIO) -> str:
def parse_and_store_document(
self, name: str, content_type: str, content: BytesIO, **kwargs
) -> str:
"""
Parse the document and store it in the Albert collection.
@@ -120,39 +129,52 @@ class BaseRagBackend:
name (str): The name of the document.
content_type (str): The MIME type of the document (e.g., "application/pdf").
content (BytesIO): The content of the document as a BytesIO stream.
**kwargs: Additional arguments. ex: "user_sub" for access control.
"""
if not self.collection_id:
raise RuntimeError("The RAG backend requires collection_id")
document_content = self.parse_document(name, content_type, content)
self.store_document(name, document_content)
self.store_document(name, document_content, **kwargs)
return document_content
def delete_collection(self) -> None:
@abstractmethod
def delete_collection(self, **kwargs) -> None:
"""
Delete the collection.
This method should handle the logic to delete the collection from the backend.
"""
raise NotImplementedError("Must be implemented in subclass.")
async def adelete_collection(self) -> None:
async def adelete_collection(self, **kwargs) -> None:
"""
Delete the collection.
This method should handle the logic to delete the collection from the backend.
"""
return await sync_to_async(self.delete_collection)()
return await sync_to_async(self.delete_collection)(**kwargs)
def search(self, query, results_count: int = 4) -> RAGWebResults:
@abstractmethod
def search(self, query: str, results_count: int = 4, **kwargs) -> RAGWebResults:
"""
Search the collection for the given query.
Args:
query: The search query string.
results_count: Number of results to return.
**kwargs: Additional arguments. ex: 'session' for OIDC authentication.
"""
raise NotImplementedError("Must be implemented in subclass.")
async def asearch(self, query, results_count: int = 4) -> RAGWebResults:
async def asearch(self, query: str, results_count: int = 4, **kwargs) -> RAGWebResults:
"""
Search the collection for the given query.
Search the collection for the given query asynchronously.
Args:
query: The search query string.
results_count: Number of results to return.
**kwargs: Additional arguments. ex: 'session' for OIDC authentication.
"""
return await sync_to_async(self.search)(query=query, results_count=results_count)
return await sync_to_async(self.search)(query=query, results_count=results_count, **kwargs)
@classmethod
@contextmanager
@@ -168,7 +190,9 @@ class BaseRagBackend:
@classmethod
@asynccontextmanager
async def temporary_collection_async(cls, name: str, description: Optional[str] = None):
async def temporary_collection_async(
cls, name: str, description: Optional[str] = None, **kwargs
):
"""Context manager for RAG backend with temporary collections."""
backend = cls()
@@ -176,4 +200,4 @@ class BaseRagBackend:
try:
yield backend
finally:
await backend.adelete_collection()
await backend.adelete_collection(**kwargs)
@@ -0,0 +1,163 @@
"""Implementation of the Find API for RAG document search."""
import logging
import uuid
from typing import List, Optional
from urllib.parse import urljoin
from uuid import uuid4
from django.conf import settings
from django.utils import timezone
import requests
from chat.agent_rag.constants import RAGWebResult, RAGWebResults, RAGWebUsage
from chat.agent_rag.document_converter.parser import AlbertParser
from chat.agent_rag.document_rag_backends.base_rag_backend import BaseRagBackend
from utils.oidc import with_fresh_access_token
logger = logging.getLogger(__name__)
SUPPORTED_LANGUAGE_CODES = ["en", "fr", "de", "nl"]
class FindRagBackend(BaseRagBackend):
"""
This class is a placeholder for the Find API implementation.
It is designed to be used with the RAG (Retrieval-Augmented Generation) document search system.
It provides methods to:
- Store parsed documents in the Find index.
- Perform a search operation using the Find API.
"""
def __init__(
self,
collection_id: Optional[str] = None,
read_only_collection_id: Optional[List[str]] = None,
):
# Initialize any necessary parameters or configurations here
super().__init__(collection_id, read_only_collection_id)
self.api_key = settings.FIND_API_KEY
self.search_endpoint = "api/v1.0/documents/search/"
self.indexing_endpoint = "api/v1.0/documents/index/"
self.deleting_endpoint = "api/v1.0/documents/delete/"
self.parser = AlbertParser() # Find Rag relies on Albert parser
def create_collection(self, name: str, description: Optional[str] = None) -> str:
"""
init collection_id
"""
self.collection_id = self.collection_id or str(uuid.uuid4())
return self.collection_id
@with_fresh_access_token
def delete_collection(self, **kwargs) -> None:
"""
Delete the current collection
"""
response = requests.post(
urljoin(settings.FIND_API_URL, self.deleting_endpoint),
headers={"Authorization": f"Bearer {kwargs['session'].get('oidc_access_token')}"},
json={
"tags": [f"collection-{self.collection_id}"],
# "service": "conversations"
},
timeout=settings.FIND_API_TIMEOUT,
)
response.raise_for_status()
def store_document(self, name: str, content: str, **kwargs) -> None:
"""
index document in Find
Args:
name (str): The name of the document.
content (str): The content of the document in Markdown format.
user_sub (str): The user subject identifier for access control.
"""
logger.debug("index document '%s' in Find", name)
user_sub = kwargs.get("user_sub")
if not user_sub:
raise ValueError("user_sub is required to store document in FindRagBackend")
response = requests.post(
urljoin(settings.FIND_API_URL, self.indexing_endpoint),
headers={"Authorization": f"Bearer {self.api_key}"},
json={
"id": str(uuid4()),
"title": str(name) or "",
"depth": 0,
"path": str(name) or "",
"numchild": 0,
"content": content or "",
"created_at": timezone.now().isoformat(),
"updated_at": timezone.now().isoformat(),
"tags": [f"collection-{self.collection_id}"],
"size": len(content.encode("utf-8")),
"users": [user_sub],
"groups": [],
"reach": "authenticated",
"is_active": True,
},
timeout=settings.FIND_API_TIMEOUT,
)
response.raise_for_status()
@with_fresh_access_token
def search(self, query: str, results_count: int = 4, **kwargs) -> RAGWebResults:
"""
Perform a search using the Find API.
Uses the user's OIDC token from the request session.
Args:
query: The search query.
results_count: Number of results to return.
**kwargs: Additional arguments. Expected: 'session' containing OIDC tokens,
Returns:
RAGWebResults: The search results.
"""
logger.debug("search documents in Find with query '%s'", query)
response = requests.post(
urljoin(settings.FIND_API_URL, self.search_endpoint),
headers={"Authorization": f"Bearer {kwargs['session'].get('oidc_access_token')}"},
json={
"q": query or "*",
"tags": [
f"collection-{collection_id}" for collection_id in self.get_all_collection_ids()
],
"k": results_count,
},
timeout=settings.FIND_API_TIMEOUT,
)
response.raise_for_status()
return RAGWebResults(
data=[
RAGWebResult(
url=get_language_value(result["_source"], "title"),
content=get_language_value(result["_source"], "content"),
score=result["_score"],
)
for result in response.json()
],
usage=RAGWebUsage(
prompt_tokens=0,
completion_tokens=0,
),
)
def get_language_value(source, language_field):
"""
extract the value of the language field with the correct language_code extension.
"title" and "content" have extensions like "title.en" or "title.fr".
get_language_value will return the value regardless of the extension.
"""
for language_code in SUPPORTED_LANGUAGE_CODES:
if f"{language_field}.{language_code}" in source:
return source[f"{language_field}.{language_code}"]
raise ValueError(f"No '{language_field}' field with any supported language code in object")
+12 -15
View File
@@ -10,7 +10,6 @@ import httpx
from pydantic_ai import Agent
from pydantic_ai.models import get_user_agent
from pydantic_ai.profiles import ModelProfile
from pydantic_ai.toolsets import FunctionToolset
from chat.tools import get_pydantic_tools_by_name
@@ -174,20 +173,18 @@ class BaseAgent(Agent):
# and pydantic_ai.models.infer_model()
_model_instance = self.configuration.model_name
_system_prompt = self.configuration.system_prompt
_base_toolset = (
[
FunctionToolset(
tools=[
get_pydantic_tools_by_name(tool_name)
for tool_name in self.configuration.tools
]
)
]
if self.configuration.tools
else None
)
_system_prompt = self.get_system_prompt()
_tools = [get_pydantic_tools_by_name(tool_name) for tool_name in self.configuration.tools]
_tools = self.get_tools()
super().__init__(model=_model_instance, instructions=_system_prompt, tools=_tools, **kwargs)
def get_system_prompt(self) -> str | None:
"""Override this method to customize the system prompt."""
return self.configuration.system_prompt
def get_tools(self) -> list | None:
"""Override this method to customize tools."""
if not self.configuration.tools:
return []
return [get_pydantic_tools_by_name(tool_name) for tool_name in self.configuration.tools]
+22
View File
@@ -131,3 +131,25 @@ class ConversationAgent(BaseAgent):
if tool.name.startswith("web_search_"):
return tool.name
return None
@dataclasses.dataclass(init=False)
class TitleGenerationAgent(BaseAgent):
"""Agent that generates concise, descriptive titles for conversations."""
def __init__(self, **kwargs):
super().__init__(
model_hrid=settings.LLM_DEFAULT_MODEL_HRID,
output_type=str,
**kwargs,
)
def get_tools(self):
return []
def get_system_prompt(self):
return (
"You are a title generator. Your task is to create concise, descriptive titles "
"that accurately summarize conversation content and help the user quickly identify the "
"conversation.\n\n"
)
+90 -75
View File
@@ -52,10 +52,9 @@ from pydantic_ai.messages import (
)
from core.feature_flags.helpers import is_feature_enabled
from core.file_upload.utils import generate_retrieve_policy
from chat import models
from chat.agents.conversation import ConversationAgent
from chat.agents.conversation import ConversationAgent, TitleGenerationAgent
from chat.agents.local_media_url_processors import (
update_history_local_urls,
update_local_urls,
@@ -72,12 +71,11 @@ from chat.clients.pydantic_ui_message_converter import (
ui_message_to_user_content,
)
from chat.mcp_servers import get_mcp_servers
from chat.tools.data_analysis import add_data_analysis_tool
from chat.tools.document_generic_search_rag import add_document_rag_search_tool_from_setting
from chat.tools.document_search_rag import add_document_rag_search_tool
from chat.tools.document_summarize import document_summarize
from chat.vercel_ai_sdk.core import events_v4, events_v5
from chat.vercel_ai_sdk.encoder import EventEncoder
from chat.vercel_ai_sdk.encoder import CURRENT_EVENT_ENCODER_VERSION, EventEncoder
# Keep at the top of the file to avoid mocking issues
document_store_backend = import_string(settings.RAG_DOCUMENT_SEARCH_BACKEND)
@@ -93,6 +91,7 @@ class ContextDeps:
conversation: models.ChatConversation
user: User
session: Optional[Dict] = None
web_search_enabled: bool = False
@@ -107,7 +106,14 @@ def get_model_configuration(model_hrid: str):
class AIAgentService: # pylint: disable=too-many-instance-attributes
"""Service class for AI-related operations (Pydantic-AI edition)."""
def __init__(self, conversation: models.ChatConversation, user, model_hrid=None, language=None):
def __init__( # pylint: disable=too-many-arguments,too-many-positional-arguments
self,
conversation: models.ChatConversation,
user,
session=None,
model_hrid=None,
language=None,
):
"""
Initialize the AI agent service.
@@ -123,7 +129,7 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
self._langfuse_available = settings.LANGFUSE_ENABLED
self._store_analytics = self._langfuse_available and user.allow_conversation_analytics
self.event_encoder = EventEncoder("v4") # Always use v4 for now
self.event_encoder = EventEncoder(CURRENT_EVENT_ENCODER_VERSION) # We use v4 for now
self._support_streaming = True
if (streaming := get_model_configuration(self.model_hrid).supports_streaming) is not None:
@@ -137,6 +143,7 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
self._context_deps = ContextDeps(
conversation=conversation,
user=user,
session=session,
web_search_enabled=self._is_web_search_enabled,
)
@@ -152,7 +159,6 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
deps_type=ContextDeps,
)
add_document_rag_search_tool_from_setting(self.conversation_agent, self.user)
add_data_analysis_tool(self.conversation_agent)
@property
def _stop_cache_key(self):
@@ -280,6 +286,7 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
name=document.identifier,
content_type=document.media_type,
content=document_data,
user_sub=self.user.sub,
)
else:
# Remote URL
@@ -289,26 +296,10 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
name=document.identifier,
content_type=document.media_type,
content=document.data,
user_sub=self.user.sub,
)
# Don't convert tabular files (CSV, Excel) to Markdown - keep originals for data_analysis tool
# Tabular files are already text-based or can be used directly
is_tabular_file = (
document.media_type in [
"text/csv",
"application/csv",
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"application/vnd.ms-excel",
"application/excel",
]
or any(
document.identifier.lower().endswith(ext)
for ext in [".csv", ".xlsx", ".xls", ".xlsm", ".xlsb"]
)
)
# Only convert non-text files that are not tabular files
if not document.media_type.startswith("text/") and not is_tabular_file:
if not document.media_type.startswith("text/"):
md_attachment = await models.ChatConversationAttachment.objects.acreate(
conversation=self.conversation,
uploaded_by=self.user,
@@ -490,28 +481,19 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
_tool_is_streaming = False
_model_response_message_id = None
# Check for existing non-PDF documents in the conversation:
# - if no document at all: do nothing
# - if only PDFs: prepare document URLs for the agent
# - if other document types: add the RAG search tool
# to allow searching in all kinds of documents
has_not_pdf_docs = await (
# Check for existing documents (any non-image attachment for this conversation)
has_documents = await (
models.ChatConversationAttachment.objects.filter(
Q(conversion_from__isnull=True) | Q(conversion_from=""),
conversation=self.conversation,
)
.exclude(
Q(content_type__startswith="image/") | Q(content_type="application/pdf"),
)
.exclude(content_type__startswith="image/")
.aexists()
)
should_enable_rag = conversation_has_documents or has_documents
document_urls = []
if not conversation_has_documents and not has_not_pdf_docs:
# No documents to process
pass
elif has_not_pdf_docs:
if should_enable_rag:
add_document_rag_search_tool(self.conversation_agent)
@self.conversation_agent.instructions
@@ -542,37 +524,6 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
"""Wrap the document_summarize tool to provide context and add the tool."""
return await document_summarize(ctx, *args, **kwargs)
if not conversation_has_documents and not has_not_pdf_docs:
# No documents to process
pass
elif has_not_pdf_docs:
# Already handled above with RAG tool
pass
else:
conversation_documents = [
cd
async for cd in models.ChatConversationAttachment.objects.filter(
Q(conversion_from__isnull=True) | Q(conversion_from=""),
conversation=self.conversation,
)
.exclude(
content_type__startswith="image/",
)
.values_list("key", "content_type")
]
for doc_key, doc_content_type in conversation_documents:
if doc_content_type == "application/pdf":
_presigned_url = generate_retrieve_policy(doc_key)
document_urls.append(
DocumentUrl(
url=_presigned_url,
identifier=doc_key.split("/")[-1],
media_type="application/pdf",
)
)
image_key_mapping[_presigned_url] = f"/media-key/{doc_key}"
async with AsyncExitStack() as stack:
# MCP servers (if any) can be initialized here
mcp_servers = [await stack.enter_async_context(mcp) for mcp in get_mcp_servers()]
@@ -586,7 +537,7 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
history.append(ModelResponse(parts=[TextPart(content="ok")], kind="response"))
async with self.conversation_agent.iter(
[user_prompt] + input_images + document_urls,
[user_prompt] + input_images,
message_history=history, # history will pass through agent's history_processors
deps=self._context_deps,
toolsets=mcp_servers,
@@ -747,8 +698,8 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
await self._agent_stop_streaming(force_cache_check=True)
# Persist conversation
await sync_to_async(self._update_conversation)(
# Prepare conversation update (save deferred until after potential title generation)
await sync_to_async(self._prepare_update_conversation)(
final_output=run.result.new_messages(),
usage=usage,
final_output_from_tool=_final_output_from_tool,
@@ -757,6 +708,35 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
image_key_mapping=image_key_mapping or None,
)
generated_title = None
# Auto-generate title after N user messages if not manually set
user_messages_count = sum(1 for msg in self.conversation.messages if msg.role == "user")
should_generate_title = (
user_messages_count == settings.AUTO_TITLE_AFTER_USER_MESSAGES
and not self.conversation.title_set_by_user_at
)
if should_generate_title:
if generated_title := await self._generate_title():
self.conversation.title = generated_title
# Persist conversation (including any generated title)
await sync_to_async(self.conversation.save)()
# Notify frontend about the title update
if generated_title:
yield events_v4.DataPart(
data=[
{
"type": "conversation_metadata",
"conversationId": str(self.conversation.pk),
"title": generated_title,
}
]
)
if self._langfuse_available:
langfuse.update_current_trace(
output=run.result.output if self._store_analytics else "REDACTED"
@@ -770,7 +750,7 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
),
)
def _update_conversation( # noqa: PLR0913
def _prepare_update_conversation( # noqa: PLR0913
self,
*,
final_output: List[ModelRequest | ModelMessage],
@@ -837,4 +817,39 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
ModelMessagesTypeAdapter.dump_json(final_output).decode("utf-8")
)
self.conversation.save()
async def _generate_title(self) -> str | None:
"""Generate a title for the conversation using LLM based on first messages."""
# Build context from messages
# Note: We intentionally use only msg.content for title generation.
# Parts containing tool invocations or reasoning are excluded as they
# don't contribute to a meaningful context here
context = "\n".join(
f"{msg.role}: {(msg.content or '')[:300]}" # Limit content length per message
for msg in self.conversation.messages
if msg.content
)
language = self.language or settings.LANGUAGE_CODE
prompt = (
"Generate a concise title (3-5 words, max 100 characters) for this conversation.\n\n"
"Requirements:\n"
"- Capture the main topic or user intent\n"
"- The title must be a simple string, no markdown\n"
"- Help the user quickly identify the conversation\n"
f"- Match the language of the user messages (default: {language})\n"
"- Avoid the word 'summary' unless explicitly requested\n\n"
"Output: Title text only, no quotes, labels, or explanation.\n\n"
f"Conversation:\n{context}"
)
try:
agent = TitleGenerationAgent()
result = await agent.run(prompt)
title = (result.output or "").strip()[:100] # Enforce max length (conversation.title)
logger.info("Generated title for conversation %s: %s", self.conversation.pk, title)
return title if title else None
except Exception as exc: # pylint: disable=broad-except #noqa: BLE001
logger.warning(
"Failed to generate title for conversation %s: %s", self.conversation.pk, exc
)
return None
+171
View File
@@ -0,0 +1,171 @@
"""Helpers to prevent proxy timeouts during long-running stream operations.
This module provides utilities to wrap synchronous and asynchronous iterators
with keepalive messages. When a stream pauses for longer than the specified
interval, keepalive messages are injected to prevent proxy/gateway
timeouts while waiting for the stream data.
"""
import asyncio
import logging
import queue
import threading
import time
from typing import AsyncIterator, Iterator
from django.conf import settings
from .vercel_ai_sdk.core.events_v4 import DataPart as V4DataPart
from .vercel_ai_sdk.core.events_v5 import DataPart as V5DataPart
from .vercel_ai_sdk.encoder import (
CURRENT_EVENT_ENCODER_VERSION,
EventEncoder,
EventEncoderVersion,
)
logger = logging.getLogger(__name__)
def get_keepalive_message() -> str:
"""Generate a keepalive message based on encoder/SDK version."""
if CURRENT_EVENT_ENCODER_VERSION == EventEncoderVersion.V4:
event = V4DataPart(data=[{"status": "WAITING"}])
else:
event = V5DataPart(data={"status": "WAITING"})
encoder = EventEncoder(CURRENT_EVENT_ENCODER_VERSION)
return encoder.encode(event)
async def stream_with_keepalive_async(
stream: AsyncIterator[str],
) -> AsyncIterator[str]:
"""Wrap an async iterator to emit keepalive during long pauses.
Args:
stream: The async iterator to wrap
Yields:
Items from the original stream, plus keepalive messages during pauses
Raises:
Any exception raised by the original stream
"""
q: asyncio.Queue = asyncio.Queue()
finished = asyncio.Event()
keepalive_message = get_keepalive_message()
async def producer():
"""Background task that consumes the original stream into a queue."""
try:
async for stream_item in stream:
await q.put(stream_item)
except Exception as exc: # pylint: disable=broad-except #noqa: BLE001
# Pass exceptions through the queue so the consumer can re-raise them.
# This ensures errors aren't silently swallowed.
await q.put(exc)
finally:
finished.set()
await q.put(None) # Sentinel to signal completion
producer_task = asyncio.create_task(producer())
try:
while True:
try:
item = await asyncio.wait_for(q.get(), timeout=settings.KEEPALIVE_INTERVAL)
if item is None:
break
if isinstance(item, Exception):
raise item
yield item
except asyncio.TimeoutError:
# No data received within interval
if finished.is_set():
# Producer is done, queue is empty (else we would not have timed out)
break
logger.debug("Send keepalive")
yield keepalive_message
finally:
# Cleanup
producer_task.cancel()
try:
await producer_task
except asyncio.CancelledError:
pass
def get_current_time() -> float:
"""Get current monotonic time, avoiding freezegun interferences.
Returns time.monotonic() which:
- Is NOT affected by freezegun's @freeze_time decorator (unlike time.time())
- Prevents issues where frozen time in main thread differs from real time in
spawned threads, causing incorrect keepalive interval computation
- Is the best clock for measuring time intervals
Wrapped in a function to ease mocking in tests.
Returns:
float: Monotonic time in seconds since an arbitrary reference point
"""
return time.monotonic()
def stream_with_keepalive_sync(stream: Iterator[str]) -> Iterator[str]:
"""Wraps a synchronous stream with keepalive messages."""
q: queue.Queue = queue.Queue()
stream_done = threading.Event()
keepalive_message = get_keepalive_message()
# Mutable container so threads can read/write shared timestamp
last_yield_time = [get_current_time()]
def consume_stream():
"""Read from source stream and forward chunks to queue."""
try:
for chunk in stream:
if stream_done.is_set():
return # early exit
q.put(chunk, timeout=1) # Arbitrary timeout prevents blocking forever
# pylint: disable=broad-exception-caught
except Exception as e:
logger.exception("Error in stream consumption")
q.put(e)
finally:
stream_done.set()
def send_keepalives():
"""Inject keepalive messages when idle too long.
Uses get_current_time() (time.monotonic) instead of time.time()
to avoid issues with freezegun in tests.
"""
while not stream_done.is_set():
# Sleep before checking to give main loop time to process and update timestamp
time.sleep(0.5) # let main loop process first, empiric value
if get_current_time() - last_yield_time[0] >= settings.KEEPALIVE_INTERVAL:
try:
q.put(keepalive_message, timeout=0.1)
except queue.Full:
pass
for target in (consume_stream, send_keepalives):
threading.Thread(target=target, daemon=True).start()
try:
# Continue while stream is active or queue has still items
while not stream_done.is_set() or not q.empty():
try:
item = q.get(timeout=1) # short timeout, avoid blocking and stay responsive
except queue.Empty:
continue
# Re-raise from consume_stream
if isinstance(item, Exception):
raise item
yield item
last_yield_time[0] = get_current_time()
finally:
# Signal threads to stop
stream_done.set()
@@ -0,0 +1,21 @@
# Generated by Django 5.2.9 on 2025-12-30 09:44
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("chat", "0004_chatconversationattachment_and_more"),
]
operations = [
migrations.AddField(
model_name="chatconversation",
name="title_set_by_user_at",
field=models.DateTimeField(
blank=True,
help_text="Timestamp when the user manually set the title. If set, prevent automatic title generation.",
null=True,
),
),
]
+6 -1
View File
@@ -44,7 +44,12 @@ class ChatConversation(BaseModel):
null=True,
help_text="Title of the chat conversation",
)
title_set_by_user_at = models.DateTimeField(
blank=True,
null=True,
help_text="Timestamp when the user manually set the title. If set, prevent automatic "
"title generation.",
)
ui_messages = models.JSONField(
default=list,
blank=True,
+7
View File
@@ -4,6 +4,7 @@ from typing import Optional
from urllib.parse import quote
from django.conf import settings
from django.utils import timezone
from django_pydantic_field.rest_framework import SchemaField # pylint: disable=no-name-in-module
from drf_spectacular.utils import extend_schema_field
@@ -27,6 +28,12 @@ class ChatConversationSerializer(serializers.ModelSerializer):
fields = ["id", "title", "created_at", "updated_at", "messages", "owner"]
read_only_fields = ["id", "created_at", "updated_at", "messages"]
def update(self, instance, validated_data):
# If title is being changed, mark it as user-set
if "title" in validated_data and validated_data["title"] != instance.title:
instance.title_set_by_user_at = timezone.now()
return super().update(instance, validated_data)
class ChatConversationInputSerializer(serializers.Serializer):
"""
@@ -0,0 +1,66 @@
"""Test cases for the TitleGenerationAgent class."""
# pylint: disable=protected-access
import pytest
from pydantic_ai.models.openai import OpenAIChatModel
from chat.agents.conversation import TitleGenerationAgent
@pytest.fixture(autouse=True)
def base_settings(settings):
"""Set up base settings for the tests."""
settings.AI_BASE_URL = "https://api.llm.com/v1/"
settings.AI_API_KEY = "test-key"
settings.AI_MODEL = "model-XYZ"
settings.AI_AGENT_INSTRUCTIONS = "You are a helpful assistant"
settings.AI_AGENT_TOOLS = []
settings.LLM_DEFAULT_MODEL_HRID = "default-model"
def test_title_generation_agent_uses_default_model_hrid(settings):
"""Test that TitleGenerationAgent uses LLM_DEFAULT_MODEL_HRID from settings."""
settings.AI_MODEL = "custom-llm-model"
settings.AI_BASE_URL = "https://custom.api.com/v1/"
settings.AI_API_KEY = "custom-key"
settings.LLM_DEFAULT_MODEL_HRID = "default-model"
agent = TitleGenerationAgent()
assert isinstance(agent._model, OpenAIChatModel)
assert settings.LLM_CONFIGURATIONS["default-model"].model_name == "custom-llm-model"
assert agent._model.model_name == "custom-llm-model"
def test_title_generation_agent_model_configuration():
"""Test that the agent model is properly configured."""
agent = TitleGenerationAgent()
assert isinstance(agent._model, OpenAIChatModel)
assert agent._model.model_name == "model-XYZ"
assert str(agent._model.client.base_url) == "https://api.llm.com/v1/"
assert agent._model.client.api_key == "test-key"
def test_title_generation_agent_has_no_tools():
"""Test that TitleGenerationAgent has no tools configured."""
agent = TitleGenerationAgent()
assert agent._function_toolset.tools == {}
assert not agent.get_tools()
def test_title_generation_agent_instructions():
"""Test that the agent instructions contain the system prompt."""
agent = TitleGenerationAgent()
# The agent should have the title generation system prompt as instructions
instructions = agent._instructions
assert len(instructions) == 1
expected = (
"You are a title generator. Your task is to create concise, descriptive titles "
"that accurately summarize conversation content and help the user quickly identify the "
"conversation.\n\n"
)
assert instructions[0] == expected
@@ -38,9 +38,6 @@ def brave_settings(settings):
settings.BRAVE_SEARCH_EXTRA_SNIPPETS = True
settings.BRAVE_SUMMARIZATION_ENABLED = False
settings.BRAVE_CACHE_TTL = 3600
settings.RAG_DOCUMENT_SEARCH_BACKEND = (
"chat.agent_rag.document_rag_backends.albert_rag_backend.AlbertRagBackend"
)
settings.BRAVE_RAG_WEB_SEARCH_CHUNK_NUMBER = 5
@@ -0,0 +1,17 @@
"""Common test fixtures for chat views tests."""
from unittest import mock
import pytest
@pytest.fixture(autouse=True)
def mock_process_request():
"""
Mock process_request to bypass OIDC authentication in tests.
"""
with mock.patch(
"lasuite.oidc_login.decorators.RefreshOIDCAccessToken.process_request"
) as mocked_process_request:
mocked_process_request.return_value = None
yield mocked_process_request
@@ -1,5 +1,6 @@
"""Common test fixtures for chat conversation endpoint tests."""
import asyncio
import json
from django.utils import timezone
@@ -10,15 +11,9 @@ import respx
from freezegun import freeze_time
@pytest.fixture(name="mock_openai_stream")
@freeze_time("2025-07-25T10:36:35.297675Z")
def fixture_mock_openai_stream():
"""
Fixture to mock the OpenAI stream response.
See https://platform.openai.com/docs/api-reference/chat-streaming/streaming
"""
openai_stream = (
def _create_openai_stream_data():
"""Helper to create OpenAI stream data."""
return (
"data: "
+ json.dumps(
{
@@ -59,12 +54,111 @@ def fixture_mock_openai_stream():
"data: [DONE]\n\n"
)
def _create_mock_openai_route(with_delays: bool = False, delay_seconds: float = 1.0):
"""Create a mock OpenAI stream route with optional delays."""
openai_stream = _create_openai_stream_data()
async def mock_stream():
for line in openai_stream.splitlines(keepends=True):
lines = openai_stream.splitlines(keepends=True)
for i, line in enumerate(lines):
yield line.encode()
if with_delays and i == 1:
# Delay after second line to trigger keepalive during streaming
await asyncio.sleep(delay_seconds)
return respx.post("https://www.external-ai-service.com/chat/completions").mock(
return_value=httpx.Response(200, stream=mock_stream())
)
@pytest.fixture(name="mock_openai_stream")
@freeze_time("2025-07-25T10:36:35.297675Z")
def fixture_mock_openai_stream():
"""
Fixture to mock the OpenAI stream response (no delays).
See https://platform.openai.com/docs/api-reference/chat-streaming/streaming
"""
return _create_mock_openai_route(with_delays=False)
@pytest.fixture(name="mock_openai_stream_slow")
def fixture_mock_openai_stream_slow():
"""
Fixture to mock the OpenAI stream response with delays to trigger keepalives.
No @freeze_time decorator because asyncio.sleep() needs real time to work properly.
"""
return _create_mock_openai_route(with_delays=True, delay_seconds=1.0)
@pytest.fixture(name="mock_openai_stream_with_title_generation")
@freeze_time("2025-07-25T10:36:35.297675Z")
def fixture_mock_openai_stream_with_title_generation():
"""
Fixture to mock the OpenAI stream response.
This fixture handles two different types of API calls made during a single request:
1. **Conversation (streaming)**: The main chat uses `stream=True` to get real-time
token-by-token responses. The API returns chunked data like:
`data: {"choices": [{"delta": {"content": "Hello"}}]}`
2. **Title generation (non-streaming)**: After the conversation, the backend calls
the API again with `stream=False` to generate a title. This returns a standard
JSON response with the complete message.
The `handle_request` function inspects each incoming request's body to determine
which type of response to return:
- `{"stream": true, ...}` → SSE streaming response
- `{"stream": false, ...}` → JSON response with generated title
Each call gets a new generator instance (avoiding generator exhaustion)
"""
def create_stream_response():
"""Create a fresh streaming response for each call."""
openai_stream = _create_openai_stream_data()
async def mock_stream():
for line in openai_stream.splitlines(keepends=True):
yield line.encode()
return httpx.Response(200, stream=mock_stream())
def create_non_stream_response():
"""Create a non-streaming response for title generation."""
return httpx.Response(
200,
json={
"id": "chatcmpl-title",
"object": "chat.completion",
"created": int(timezone.make_naive(timezone.now()).timestamp()),
"model": "test-model",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "GENERATED TITLE",
},
"finish_reason": "stop",
}
],
"usage": {"prompt_tokens": 50, "completion_tokens": 5, "total_tokens": 55},
},
)
def handle_request(request):
"""Route to streaming or non-streaming response based on request."""
body = json.loads(request.content)
if body.get("stream", False):
return create_stream_response()
return create_non_stream_response()
route = respx.post("https://www.external-ai-service.com/chat/completions").mock(
return_value=httpx.Response(200, stream=mock_stream())
side_effect=handle_request
)
return route
@@ -3,6 +3,7 @@
import json
import logging
from unittest.mock import ANY, patch
from django.utils import timezone
@@ -221,6 +222,133 @@ def test_post_conversation_data_protocol(api_client, mock_openai_stream):
]
@freeze_time("2025-07-25T10:36:35.297675Z")
@respx.mock
@patch("chat.keepalive.get_current_time")
def test_post_conversation_data_protocol_triggers_keepalives(
mock_time, api_client, mock_openai_stream
):
"""Test streaming response contains keepalive messages"""
chat_conversation = ChatConversationFactory(owner__language="en-us")
mock_time.side_effect = [float(i * 60) for i in range(10)]
url = f"/api/v1.0/chats/{chat_conversation.pk}/conversation/?protocol=data"
data = {
"messages": [
{
"id": "yuPoOuBkKA4FnKvk",
"role": "user",
"parts": [{"text": "Hello", "type": "text"}],
"content": "Hello",
"createdAt": "2025-07-03T15:22:17.105Z",
}
]
}
api_client.force_login(chat_conversation.owner)
response = api_client.post(url, data, format="json")
assert response.status_code == status.HTTP_200_OK
assert response.get("Content-Type") == "text/event-stream"
assert response.get("x-vercel-ai-data-stream") == "v1"
assert response.streaming
# Wait for the streaming content to be fully received
response_content = b"".join(response.streaming_content).decode("utf-8")
# Replace UUIDs with placeholders for assertion
response_content = replace_uuids_with_placeholder(response_content)
assert response_content == (
'0:"Hello"\n'
'0:" there"\n'
'f:{"messageId":"<mocked_uuid>"}\n'
'd:{"finishReason":"stop","usage":{"promptTokens":0,"completionTokens":0}}\n'
'2:[{"status": "WAITING"}]\n'
)
assert mock_openai_stream.called
chat_conversation.refresh_from_db()
assert chat_conversation.ui_messages == [
{
"content": "Hello",
"createdAt": "2025-07-03T15:22:17.105Z",
"id": "yuPoOuBkKA4FnKvk",
"parts": [{"text": "Hello", "type": "text"}],
"role": "user",
}
]
assert len(chat_conversation.messages) == 2
assert chat_conversation.messages[0].id == IsUUID(4)
assert chat_conversation.messages[0] == UIMessage(
id=chat_conversation.messages[0].id, # don't test the message ID here
createdAt=timezone.now(), # Mocked timestamp
content="Hello",
reasoning=None,
experimental_attachments=None,
role="user",
annotations=None,
toolInvocations=None,
parts=[TextUIPart(type="text", text="Hello")],
)
assert chat_conversation.messages[1].id == IsUUID(4)
assert chat_conversation.messages[1] == UIMessage(
id=chat_conversation.messages[1].id, # don't test the message ID here
createdAt=timezone.now(), # Mocked timestamp
content="Hello there",
reasoning=None,
experimental_attachments=None,
role="assistant",
annotations=None,
toolInvocations=None,
parts=[TextUIPart(type="text", text="Hello there")],
)
_run_id = chat_conversation.pydantic_messages[0]["run_id"]
assert chat_conversation.pydantic_messages == [
{
"instructions": (
"You are a helpful test assistant :)\n\n"
"Today is Friday 25/07/2025.\n\n"
"Answer in english."
),
"kind": "request",
"parts": [
{
"content": ["Hello"],
"part_kind": "user-prompt",
"timestamp": "2025-07-25T10:36:35.297675Z",
},
],
"run_id": _run_id,
},
{
"finish_reason": "stop",
"kind": "response",
"model_name": "test-model",
"parts": [{"content": "Hello there", "id": None, "part_kind": "text"}],
"provider_details": {"finish_reason": "stop"},
"provider_name": "openai",
"provider_response_id": "chatcmpl-1234567890",
"timestamp": "2025-07-25T10:36:35.297675Z",
"usage": {
"cache_audio_read_tokens": 0,
"cache_read_tokens": 0,
"cache_write_tokens": 0,
"details": {},
"input_audio_tokens": 0,
"input_tokens": 0,
"output_audio_tokens": 0,
"output_tokens": 0,
},
"run_id": _run_id,
},
]
@freeze_time("2025-07-25T10:36:35.297675Z")
@respx.mock
def test_post_conversation_text_protocol(api_client, mock_openai_stream):
@@ -1344,3 +1472,143 @@ async def test_post_conversation_async(api_client, mock_openai_stream, monkeypat
"run_id": _run_id,
},
]
@freeze_time("2025-07-25T10:36:35.297675Z", tick=True)
@respx.mock
@pytest.mark.asyncio
async def test_post_conversation_async_triggers_keepalive(
api_client, mock_openai_stream_slow, monkeypatch, caplog, settings
):
"""Test posting messages to a conversation using the 'data' protocol."""
monkeypatch.setenv("PYTHON_SERVER_MODE", "async")
settings.KEEPALIVE_INTERVAL = 1 # s
chat_conversation = await sync_to_async(ChatConversationFactory)(owner__language="en-us")
url = f"/api/v1.0/chats/{chat_conversation.pk}/conversation/?protocol=data"
data = {
"messages": [
{
"id": "yuPoOuBkKA4FnKvk",
"role": "user",
"parts": [{"text": "Hello", "type": "text"}],
"content": "Hello",
"createdAt": "2025-07-03T15:22:17.105Z",
}
]
}
await api_client.aforce_login(chat_conversation.owner)
caplog.clear()
caplog.set_level(level=logging.DEBUG, logger="chat.views")
response = await sync_to_async(api_client.post)(url, data, format="json") # client is sync
assert response.status_code == status.HTTP_200_OK
assert response.get("Content-Type") == "text/event-stream"
assert response.get("x-vercel-ai-data-stream") == "v1"
assert response.streaming
assert "Using ASYNC streaming for chat conversation" in caplog.text
# Wait for the streaming content to be fully received => async iterator -> list
# This fails it the streaming is not an async generator
response_content = b"".join([content async for content in response.streaming_content]).decode(
"utf-8"
)
# Replace UUIDs with placeholders for assertion
response_content = replace_uuids_with_placeholder(response_content)
assert response_content == (
'0:"Hello"\n'
'2:[{"status": "WAITING"}]\n'
'0:" there"\n'
'f:{"messageId":"<mocked_uuid>"}\n'
'd:{"finishReason":"stop","usage":{"promptTokens":0,"completionTokens":0}}\n'
)
assert mock_openai_stream_slow.called
await chat_conversation.arefresh_from_db()
assert chat_conversation.ui_messages == [
{
"content": "Hello",
"createdAt": "2025-07-03T15:22:17.105Z",
"id": "yuPoOuBkKA4FnKvk",
"parts": [{"text": "Hello", "type": "text"}],
"role": "user",
}
]
assert len(chat_conversation.messages) == 2
assert chat_conversation.messages[0].id == IsUUID(4)
assert chat_conversation.messages[0] == UIMessage(
id=chat_conversation.messages[0].id, # don't test the message ID here
createdAt=chat_conversation.messages[0].createdAt, # Mocked timestamp
content="Hello",
reasoning=None,
experimental_attachments=None,
role="user",
annotations=None,
toolInvocations=None,
parts=[TextUIPart(type="text", text="Hello")],
)
assert chat_conversation.messages[1].id == IsUUID(4)
assert chat_conversation.messages[1] == UIMessage(
id=chat_conversation.messages[1].id, # don't test the message ID here
createdAt=chat_conversation.messages[1].createdAt, # Mocked timestamp
content="Hello there",
reasoning=None,
experimental_attachments=None,
role="assistant",
annotations=None,
toolInvocations=None,
parts=[TextUIPart(type="text", text="Hello there")],
)
_run_id = chat_conversation.pydantic_messages[0]["run_id"]
# using ANY because time is not frozen in this api mock
assert chat_conversation.pydantic_messages == [
{
"instructions": (
"You are a helpful test assistant :)\n\n"
"Today is Friday 25/07/2025.\n\nAnswer in english."
),
"kind": "request",
"parts": [
{
"content": ["Hello"],
"part_kind": "user-prompt",
"timestamp": ANY,
},
],
"run_id": _run_id,
},
{
"finish_reason": "stop",
"kind": "response",
"model_name": "test-model",
"parts": [{"content": "Hello there", "id": None, "part_kind": "text"}],
"provider_details": {"finish_reason": "stop"},
"provider_name": "openai",
"provider_response_id": "chatcmpl-1234567890",
"timestamp": ANY,
"usage": {
"cache_audio_read_tokens": 0,
"cache_read_tokens": 0,
"cache_write_tokens": 0,
"details": {},
"input_audio_tokens": 0,
"input_tokens": 0,
"output_audio_tokens": 0,
"output_tokens": 0,
},
"run_id": _run_id,
},
]
@@ -8,6 +8,7 @@ import logging
from io import BytesIO
from unittest import mock
from django.contrib.sessions.backends.cache import SessionStore
from django.utils import formats, timezone
import httpx
@@ -41,28 +42,49 @@ from chat.tests.utils import replace_uuids_with_placeholder
pytestmark = pytest.mark.django_db(transaction=True)
@pytest.fixture(autouse=True)
def ai_settings(settings):
@pytest.fixture(
autouse=True,
params=[
"chat.agent_rag.document_rag_backends.find_rag_backend.FindRagBackend",
"chat.agent_rag.document_rag_backends.albert_rag_backend.AlbertRagBackend",
],
)
def ai_settings(request, settings):
"""Fixture to set AI service URLs for testing."""
settings.AI_BASE_URL = "https://www.external-ai-service.com/"
settings.AI_API_KEY = "test-api-key"
settings.AI_MODEL = "test-model"
settings.AI_AGENT_INSTRUCTIONS = "You are a helpful test assistant :)"
# Enable Albert API for document search
settings.RAG_DOCUMENT_SEARCH_BACKEND = (
"chat.agent_rag.document_rag_backends.albert_rag_backend.AlbertRagBackend"
)
settings.ALBERT_API_URL = "https://albert.api.etalab.gouv.fr"
settings.ALBERT_API_KEY = "albert-api-key"
# enable on rag document search tool
settings.RAG_DOCUMENT_SEARCH_BACKEND = request.param
settings.RAG_WEB_SEARCH_PROMPT_UPDATE = (
"Based on the following document contents:\n\n{search_results}\n\n"
"Please answer the user's question: {user_prompt}"
)
settings.AI_BASE_URL = "https://www.external-ai-service.com/"
settings.AI_API_KEY = "test-api-key"
settings.AI_MODEL = "test-model"
settings.AI_AGENT_INSTRUCTIONS = "You are a helpful test assistant :)"
# Albert API settings
settings.ALBERT_API_URL = "https://albert.api.etalab.gouv.fr"
settings.ALBERT_API_KEY = "albert-api-key"
# Find API settings
settings.FIND_API_URL = "https://find.api.example.com"
settings.FIND_API_KEY = "find-api-key"
return settings
@pytest.fixture(autouse=True)
def mock_refresh_access_token():
"""Mock refresh_access_token to bypass token refresh in tests."""
with mock.patch("utils.oidc.refresh_access_token") as mocked_refresh_access_token:
session = SessionStore()
session["oidc_access_token"] = "mocked-access-token"
mocked_refresh_access_token.return_value = session
yield mocked_refresh_access_token
@pytest.fixture(name="sample_pdf_content")
def fixture_sample_pdf_content():
"""Create a dummy PDF content as BytesIO."""
@@ -81,10 +103,18 @@ def fixture_sample_pdf_content():
return BytesIO(pdf_data)
@pytest.fixture(name="mock_albert_api")
def fixture_mock_albert_api():
@pytest.fixture(name="mock_document_api")
def fixture_mock_document_api():
"""Fixture to mock the Albert API endpoints."""
# Mock collection creation
document_name = "sample.pdf"
document_content = "This is the content of the PDF."
prompt_tokens = 10
completion_tokens = 20
search_method = "semantic"
search_score = 0.9
responses.post(
"https://albert.api.etalab.gouv.fr/v1/collections",
json={"id": "123", "name": "test-collection"},
@@ -101,7 +131,7 @@ def fixture_mock_albert_api():
"metadata": {"document_name": "sample.pdf"},
}
],
"usage": {"prompt_tokens": 10, "completion_tokens": 20},
"usage": {"prompt_tokens": prompt_tokens, "completion_tokens": completion_tokens},
},
status=status.HTTP_200_OK,
)
@@ -119,20 +149,42 @@ def fixture_mock_albert_api():
json={
"data": [
{
"method": "semantic",
"method": search_method,
"chunk": {
"id": 123,
"content": "This is the content of the PDF.",
"metadata": {"document_name": "sample.pdf"},
"content": document_content,
"metadata": {"document_name": document_name},
},
"score": 0.9,
"score": search_score,
}
],
"usage": {"prompt_tokens": 10, "completion_tokens": 20},
"usage": {"prompt_tokens": prompt_tokens, "completion_tokens": completion_tokens},
},
status=status.HTTP_200_OK,
)
# Mock document indexing (Find API)
responses.post(
"https://find.api.example.com/api/v1.0/documents/index/",
json={"id": "456", "status": "indexed"},
status=status.HTTP_200_OK,
)
# Mock document search (Find API)
responses.post(
"https://find.api.example.com/api/v1.0/documents/search/",
json=[
{
"_source": {
"title.fr": document_name,
"content.fr": document_content,
},
"_score": search_score,
}
],
status=status.HTTP_200_OK,
)
@pytest.fixture(name="mock_summarization_agent")
def fixture_mock_summarization_agent():
@@ -219,7 +271,7 @@ def fixture_mock_openai_stream():
def test_post_conversation_with_document_upload(
# pylint: disable=too-many-arguments,too-many-positional-arguments
api_client,
mock_albert_api, # pylint: disable=unused-argument
mock_document_api, # pylint: disable=unused-argument
sample_pdf_content,
today_promt_date,
mock_ai_agent_service,
@@ -548,7 +600,7 @@ def test_post_conversation_with_document_upload_feature_disabled(
@freeze_time()
def test_post_conversation_with_document_upload_summarize( # pylint: disable=too-many-arguments,too-many-positional-arguments # noqa: PLR0913
api_client,
mock_albert_api, # pylint: disable=unused-argument
mock_document_api, # pylint: disable=unused-argument
sample_pdf_content,
today_promt_date,
mock_ai_agent_service,
@@ -37,11 +37,19 @@ from chat.tests.utils import replace_uuids_with_placeholder
pytestmark = pytest.mark.django_db(transaction=True)
@pytest.fixture(autouse=True)
def ai_settings(settings):
@pytest.fixture(
autouse=True,
params=[
"chat.agent_rag.document_rag_backends.find_rag_backend.FindRagBackend",
"chat.agent_rag.document_rag_backends.albert_rag_backend.AlbertRagBackend",
],
)
def ai_settings(request, settings):
"""Fixture to set AI service URLs for testing."""
settings.RAG_DOCUMENT_SEARCH_BACKEND = request.param
settings.AI_BASE_URL = "https://www.external-ai-service.com/"
settings.AI_API_KEY = "test-api-key"
settings.FIND_API_KEY = "find-api-key"
settings.AI_MODEL = "test-model"
settings.AI_AGENT_INSTRUCTIONS = "You are a helpful test assistant :)"
return settings
@@ -85,6 +93,10 @@ def test_post_conversation_with_local_pdf_document_url(
json={"id": "document_id", "object": "document"},
status=200,
)
responses.post(
"https://app-find/api/v1.0/documents/index/",
status=200,
)
chat_conversation = ChatConversationFactory(owner__language="en-us")
api_client.force_authenticate(user=chat_conversation.owner)
@@ -120,29 +132,28 @@ def test_post_conversation_with_local_pdf_document_url(
)
async def agent_model(messages: list[ModelMessage], _info: AgentInfo):
presigned_url = messages[0].parts[0].content[1].url
assert presigned_url.startswith("http://localhost:9000/conversations-media-storage/")
assert presigned_url.find("X-Amz-Signature=") != -1
assert presigned_url.find("X-Amz-Date=") != -1
assert presigned_url.find("X-Amz-Expires=") != -1
assert messages == [
ModelRequest(
parts=[
UserPromptPart(
content=[
"What is in this document?",
DocumentUrl(
url=presigned_url, # presigned URL for this conversation
media_type="application/pdf",
identifier="sample.pdf",
),
],
timestamp=timezone.now(),
),
UserPromptPart(content=["What is in this document?"], timestamp=timezone.now())
],
instructions=f"You are a helpful test assistant :)\n\n{today_promt_date}"
"\n\nAnswer in english.",
instructions=(
"You are a helpful test assistant :)\n\n"
f"{today_promt_date}\n\n"
"Answer in english.\n\n"
"Use document_search_rag ONLY to retrieve specific passages from attached "
"documents. Do NOT use it to summarize; for summaries, call the summarize "
"tool instead.\n\nWhen you receive a result from the summarization tool, "
"you MUST return it directly to the user without any modification, "
"paraphrasing, or additional summarization.The tool already produces "
"optimized summaries that should be presented verbatim.You may translate "
"the summary if required, but you MUST preserve all the information from "
"the original summary.You may add a follow-up question after the summary "
"if needed.\n\n"
"[Internal context] User documents are attached to this conversation. "
"Do not request re-upload of documents; consider them already available "
"via the internal store."
),
run_id=messages[0].run_id,
)
]
@@ -186,9 +197,7 @@ def test_post_conversation_with_local_pdf_document_url(
createdAt=timezone.now(),
content="What is in this document?",
reasoning=None,
experimental_attachments=[
Attachment(name="sample.pdf", contentType="application/pdf", url=document_url)
],
experimental_attachments=None, # We should fix this, but for now document appears in source
role="user",
annotations=None,
toolInvocations=None,
@@ -220,20 +229,29 @@ def test_post_conversation_with_local_pdf_document_url(
{
"instructions": "You are a helpful test assistant :)\n\n"
f"{today_promt_date}\n\n"
"Answer in english.",
"Answer in english.\n"
"\n"
"Use document_search_rag ONLY to retrieve specific passages "
"from attached documents. Do NOT use it to summarize; for "
"summaries, call the summarize tool instead.\n"
"\n"
"When you receive a result from the summarization tool, you "
"MUST return it directly to the user without any "
"modification, paraphrasing, or additional summarization.The "
"tool already produces optimized summaries that should be "
"presented verbatim.You may translate the summary if "
"required, but you MUST preserve all the information from "
"the original summary.You may add a follow-up question after "
"the summary if needed.\n"
"\n"
"[Internal context] User documents are attached to this "
"conversation. Do not request re-upload of documents; "
"consider them already available via the internal store.",
"kind": "request",
"parts": [
{
"content": [
"What is in this document?",
{
"force_download": False,
"identifier": "sample.pdf",
"kind": "document-url",
"media_type": "application/pdf",
"url": document_url,
"vendor_metadata": None,
},
],
"part_kind": "user-prompt",
"timestamp": timestamp,
@@ -795,6 +813,10 @@ def test_post_conversation_with_local_not_pdf_document_url(
json={"id": "document_id", "object": "document"},
status=200,
)
responses.post(
"https://app-find/api/v1.0/documents/index/",
status=200,
)
chat_conversation = ChatConversationFactory(owner__language="en-us")
api_client.force_authenticate(user=chat_conversation.owner)
@@ -2,6 +2,7 @@
# pylint: disable=too-many-lines
import json
from unittest.mock import patch
from django.utils import timezone
@@ -11,6 +12,7 @@ from dirty_equals import IsUUID
from freezegun import freeze_time
from rest_framework import status
from chat.agents.conversation import TitleGenerationAgent
from chat.ai_sdk_types import (
Attachment,
TextUIPart,
@@ -35,6 +37,7 @@ def ai_settings(settings):
settings.AI_MODEL = "test-model"
settings.AI_AGENT_INSTRUCTIONS = "You are a helpful test assistant :)"
settings.AUTO_TITLE_AFTER_USER_MESSAGES = None # disable auto title generation
return settings
@@ -1573,3 +1576,307 @@ def test_post_conversation_add_image_to_conversation_with_tool_history(
toolInvocations=None,
parts=[TextUIPart(type="text", text="I see a cat in the picture.")],
)
@freeze_time("2025-07-25T10:36:35.297675Z")
@respx.mock
@patch("chat.clients.pydantic_ai.TitleGenerationAgent", wraps=TitleGenerationAgent)
def test_post_conversation_triggers_automatic_title_generation_after_first_message(
mock_title_agent, api_client, mock_openai_stream_with_title_generation, settings
):
"""
Test that posting the first user message triggers automatic title generation.
AUTO_TITLE_AFTER_USER_MESSAGES = 1
The conversation is a new one. Posting the first message
should trigger title generation via the TitleGenerationAgent.
"""
# Configure the title generation threshold
settings.AUTO_TITLE_AFTER_USER_MESSAGES = 1
conversation = ChatConversationFactory()
url = f"/api/v1.0/chats/{conversation.pk}/conversation/?protocol=data"
data = {
"messages": [
{
"id": "third-user-msg",
"role": "user",
"parts": [{"text": "Can you explain backpropagation?", "type": "text"}],
"content": "Can you explain backpropagation?",
"createdAt": "2025-07-25T10:36:00.000Z",
}
]
}
api_client.force_login(conversation.owner)
conversation.title = "initial title"
conversation.save()
assert not conversation.title_set_by_user_at
response = api_client.post(url, data, format="json")
assert response.status_code == status.HTTP_200_OK
assert response.get("Content-Type") == "text/event-stream"
assert response.streaming
# Wait for the streaming content to be fully received
response_content = b"".join(response.streaming_content).decode("utf-8")
# Verify the conversation_metadata event is in the stream
assert '"type": "conversation_metadata"' in response_content
# Refresh and verify title was updated
conversation.refresh_from_db()
assert conversation.title == "GENERATED TITLE"
# title_set_by_user_at should remain None since it was auto-generated
assert not conversation.title_set_by_user_at
assert mock_openai_stream_with_title_generation.called
assert mock_openai_stream_with_title_generation.call_count == 2
# Verify TitleGenerationAgent was called
mock_title_agent.assert_called_once()
@freeze_time("2025-07-25T10:36:35.297675Z")
@respx.mock
def test_post_conversation_triggers_automatic_title_generation_at_threshold(
api_client, mock_openai_stream_with_title_generation, settings, history_conversation
):
"""
Test that posting the 3rd user message triggers automatic title generation.
AUTO_TITLE_AFTER_USER_MESSAGES = 3
The history_conversation fixture has 2 user messages. Posting a 3rd message
should trigger title generation via the TitleGenerationAgent.
"""
# Configure the title generation threshold
settings.AUTO_TITLE_AFTER_USER_MESSAGES = 3
url = f"/api/v1.0/chats/{history_conversation.pk}/conversation/?protocol=data"
data = {
"messages": [
{
"id": "third-user-msg",
"role": "user",
"parts": [{"text": "Can you explain backpropagation?", "type": "text"}],
"content": "Can you explain backpropagation?",
"createdAt": "2025-07-25T10:36:00.000Z",
}
]
}
api_client.force_login(history_conversation.owner)
history_conversation.title = "initial title"
history_conversation.save()
assert not history_conversation.title_set_by_user_at
response = api_client.post(url, data, format="json")
assert response.status_code == status.HTTP_200_OK
assert response.get("Content-Type") == "text/event-stream"
assert response.streaming
# Wait for the streaming content to be fully received
response_content = b"".join(response.streaming_content).decode("utf-8")
# Verify the conversation_metadata event is in the stream
assert '"type": "conversation_metadata"' in response_content
# Refresh and verify title was updated
history_conversation.refresh_from_db()
assert history_conversation.title == "GENERATED TITLE"
# title_set_by_user_at should remain None since it was auto-generated
assert not history_conversation.title_set_by_user_at
assert mock_openai_stream_with_title_generation.called
assert mock_openai_stream_with_title_generation.call_count == 2
@freeze_time("2025-07-25T10:36:35.297675Z")
@respx.mock
def test_post_conversation_does_not_regenerate_title_when_user_set(
api_client, mock_openai_stream_with_title_generation, settings, history_conversation
):
"""
Test that title is NOT regenerated if the user has manually set a title.
"""
settings.AUTO_TITLE_AFTER_USER_MESSAGES = 3
# Simulate user having set a custom title
history_conversation.title = "My Custom Title"
history_conversation.title_set_by_user_at = timezone.now()
history_conversation.save()
url = f"/api/v1.0/chats/{history_conversation.pk}/conversation/?protocol=data"
data = {
"messages": [
{
"id": "third-user-msg",
"role": "user",
"parts": [{"text": "Can you explain backpropagation?", "type": "text"}],
"content": "Can you explain backpropagation?",
"createdAt": "2025-07-25T10:36:00.000Z",
}
]
}
api_client.force_login(history_conversation.owner)
response = api_client.post(url, data, format="json")
assert response.status_code == status.HTTP_200_OK
# Consume the stream
response_content = b"".join(response.streaming_content).decode("utf-8")
# conversation_metadata should NOT be in the stream since title wasn't generated
assert "conversation_metadata" not in response_content
# Refresh and verify title was NOT changed
history_conversation.refresh_from_db()
assert history_conversation.title == "My Custom Title"
assert history_conversation.title_set_by_user_at
assert mock_openai_stream_with_title_generation.called
assert mock_openai_stream_with_title_generation.call_count == 1
@freeze_time("2025-07-25T10:36:35.297675Z")
@respx.mock
def test_post_conversation_does_not_generate_title_before_threshold(
api_client, mock_openai_stream_with_title_generation, settings
):
"""
Test that title is NOT generated before reaching the message threshold.
"""
settings.AUTO_TITLE_AFTER_USER_MESSAGES = 3
# Create a conversation with only 1 user message
history_timestamp = timezone.now().replace(year=2025, month=6, day=15, hour=10, minute=30)
conversation = ChatConversationFactory(title="initial title")
conversation.messages = [
UIMessage(
id="prev-user-msg-1",
createdAt=history_timestamp,
content="Hello!",
reasoning=None,
experimental_attachments=None,
role="user",
annotations=None,
toolInvocations=None,
parts=[TextUIPart(type="text", text="Hello!")],
),
UIMessage(
id="prev-assistant-msg-1",
createdAt=history_timestamp.replace(minute=31),
content="Hi there! How can I help you?",
reasoning=None,
experimental_attachments=None,
role="assistant",
annotations=None,
toolInvocations=None,
parts=[TextUIPart(type="text", text="Hi there! How can I help you?")],
),
]
conversation.save()
url = f"/api/v1.0/chats/{conversation.pk}/conversation/?protocol=data"
data = {
"messages": [
{
"id": "second-user-msg",
"role": "user",
"parts": [{"text": "What's machine learning?", "type": "text"}],
"content": "What's machine learning?",
"createdAt": "2025-07-25T10:36:00.000Z",
}
]
}
api_client.force_login(conversation.owner)
response = api_client.post(url, data, format="json")
assert response.status_code == status.HTTP_200_OK
# Consume the stream
response_content = b"".join(response.streaming_content).decode("utf-8")
# conversation_metadata should NOT be in the stream (only 2 user messages)
assert "conversation_metadata" not in response_content
# Refresh and verify title was not updated
conversation.refresh_from_db()
assert conversation.title == "initial title"
assert not conversation.title_set_by_user_at
assert mock_openai_stream_with_title_generation.call_count == 1
@freeze_time("2025-07-25T10:36:35.297675Z")
@respx.mock
def test_post_conversation_does_not_generate_title_after_threshold(
api_client, mock_openai_stream_with_title_generation, settings, history_conversation
):
"""
Test that posting the 3rd user message does not trigger automatic title generation.
AUTO_TITLE_AFTER_USER_MESSAGES = 2
The history_conversation fixture has 2 user messages. Posting a 3rd message
should not trigger title generation.
"""
# Configure the title generation threshold
settings.AUTO_TITLE_AFTER_USER_MESSAGES = 2
url = f"/api/v1.0/chats/{history_conversation.pk}/conversation/?protocol=data"
data = {
"messages": [
{
"id": "third-user-msg",
"role": "user",
"parts": [{"text": "Can you explain backpropagation?", "type": "text"}],
"content": "Can you explain backpropagation?",
"createdAt": "2025-07-25T10:36:00.000Z",
}
]
}
api_client.force_login(history_conversation.owner)
history_conversation.title = "initial title"
history_conversation.save()
assert not history_conversation.title_set_by_user_at
response = api_client.post(url, data, format="json")
assert response.status_code == status.HTTP_200_OK
assert response.get("Content-Type") == "text/event-stream"
assert response.streaming
# Wait for the streaming content to be fully received
response_content = b"".join(response.streaming_content).decode("utf-8")
# Verify the conversation_metadata event is not in the stream
assert "conversation_metadata" not in response_content
# Refresh and verify title was NOT updated (past threshold)
history_conversation.refresh_from_db()
# title not updated
assert history_conversation.title == "initial title"
# title_set_by_user_at should remain None since it was auto-generated
assert not history_conversation.title_set_by_user_at
assert mock_openai_stream_with_title_generation.call_count == 1
@@ -28,6 +28,7 @@ def test_create_conversation(api_client):
conversation = ChatConversation.objects.get(id=response.data["id"])
assert conversation.owner == user
assert conversation.title == "New Conversation"
assert not conversation.title_set_by_user_at
def test_create_conversation_other_owner(api_client):
@@ -2,6 +2,7 @@
import pytest
from rest_framework import status
from rest_framework.exceptions import ErrorDetail
from core.factories import UserFactory
@@ -26,6 +27,34 @@ def test_update_conversation(api_client):
# Verify in database
conversation = ChatConversation.objects.get(id=chat_conversation.pk)
assert conversation.title == "Updated Title"
assert conversation.title_set_by_user_at
def test_update_conversation_limit_title_length(api_client):
"""Test that updating a conversation with a title exceeding 100 characters fails validation."""
chat_conversation = ChatConversationFactory(title="Initial title")
url = f"/api/v1.0/chats/{chat_conversation.pk}/"
# Create a 101-character title to exceed the 100-character maximum limit
new_title = "X" * 101
data = {"title": new_title}
api_client.force_login(chat_conversation.owner)
response = api_client.put(url, data, format="json")
assert response.status_code == status.HTTP_400_BAD_REQUEST
assert response.data == {
"title": [
ErrorDetail(
string="Ensure this field has no more than 100 characters.", code="max_length"
)
]
}
# Verify in database (title should remain unchanged)
conversation = ChatConversation.objects.get(id=chat_conversation.pk)
assert conversation.title == "Initial title"
assert not conversation.title_set_by_user_at
def test_update_conversation_anonymous(api_client):
-873
View File
@@ -1,873 +0,0 @@
"""Data analysis tool for tabular files (CSV, Excel)."""
import base64
import functools
import json
import logging
import uuid
from io import BytesIO
from typing import Any, Dict
import matplotlib
import numpy as np
matplotlib.use("Agg") # Use non-interactive backend
import matplotlib.pyplot as plt
import pandas as pd
from django.conf import settings
from django.core.files.storage import default_storage
from django.db.models import Q
from asgiref.sync import sync_to_async
from pydantic_ai import Agent, RunContext
from pydantic_ai.exceptions import ModelRetry
from pydantic_ai.messages import ToolReturn
from core.file_upload.enums import AttachmentStatus
from core.file_upload.utils import generate_retrieve_policy
from chat.agents.base import BaseAgent, prepare_custom_model
from chat.models import ChatConversationAttachment
from chat.tools.exceptions import ModelCannotRetry
from chat.tools.utils import last_model_retry_soft_fail
logger = logging.getLogger(__name__)
# MIME types for tabular files
TABULAR_MIME_TYPES = [
"text/csv",
"application/csv",
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"application/vnd.ms-excel",
"application/excel",
]
@sync_to_async
def read_tabular_file(attachment) -> bytes:
"""Read tabular file content asynchronously."""
with default_storage.open(attachment.key, "rb") as f:
return f.read()
def detect_csv_separator(file_data: bytes) -> str:
"""
Detect the CSV separator by analyzing the first few lines.
Returns the most likely separator: ',', ';', or '\t'
"""
# Read first 10KB to analyze
sample = file_data[:10240].decode("utf-8", errors="ignore")
lines = sample.split("\n")[:10] # First 10 lines
if not lines:
return "," # Default to comma
# Count occurrences of each separator in the first few lines
comma_count = sum(line.count(",") for line in lines)
semicolon_count = sum(line.count(";") for line in lines)
tab_count = sum(line.count("\t") for line in lines)
# Return the separator with the highest count
if tab_count > comma_count and tab_count > semicolon_count:
return "\t"
elif semicolon_count > comma_count:
return ";"
else:
return "," # Default to comma
def _convert_to_serializable(obj: Any) -> Any:
"""
Convert pandas/numpy types to Python native types for JSON serialization.
Handles:
- pandas DataFrame/Series
- numpy scalars (int64, float64, etc.)
- numpy arrays
- pandas Timestamp
- Other non-serializable types
Args:
obj: The object to convert.
Returns:
A JSON-serializable version of the object.
"""
# Handle pandas DataFrame
if isinstance(obj, pd.DataFrame):
# Limit number of rows to avoid huge responses
if len(obj) > 1000:
obj = obj.head(1000)
logger.warning("Result truncated to 1000 rows")
return obj.to_dict(orient="records")
# Handle pandas Series
if isinstance(obj, pd.Series):
# Convert Series to dict, handling index
result_dict = obj.to_dict()
# Convert any numpy/pandas types in the values
return {str(k): _convert_to_serializable(v) for k, v in result_dict.items()}
# Handle numpy scalars
if isinstance(obj, (np.integer, np.floating)):
return obj.item() # Convert to Python native int/float
# Handle numpy arrays
if isinstance(obj, np.ndarray):
return obj.tolist()
# Handle pandas Timestamp
if isinstance(obj, pd.Timestamp):
return obj.isoformat()
# Handle lists and tuples - recursively convert elements
if isinstance(obj, (list, tuple)):
return [_convert_to_serializable(item) for item in obj]
# Handle dicts - recursively convert values
if isinstance(obj, dict):
return {str(k): _convert_to_serializable(v) for k, v in obj.items()}
# Handle None, bool, int, float, str - these are already serializable
if obj is None or isinstance(obj, (bool, int, float, str)):
return obj
# Fallback: try to convert to string
try:
return str(obj)
except Exception:
logger.warning("Could not serialize object of type %s, returning None", type(obj))
return None
def _is_valid_excel_file(file_data: bytes, file_name: str) -> bool:
"""
Check if the file data appears to be a valid Excel file.
XLSX files are ZIP archives, so they should start with ZIP signature (PK\x03\x04).
XLS files have a different signature.
"""
if not file_data:
return False
file_lower = file_name.lower()
# Check for XLSX (ZIP-based) signature
if file_lower.endswith((".xlsx", ".xlsm", ".xlsb")):
# XLSX files are ZIP archives, should start with PK\x03\x04
return file_data[:4] == b"PK\x03\x04"
# Check for XLS (OLE2) signature
if file_lower.endswith(".xls"):
# XLS files are OLE2 compound documents, should start with specific signature
# Common signatures: 0xD0CF11E0 (OLE2) or 0x504B0304 (sometimes saved as ZIP)
return (
file_data[:4] == b"\xd0\xcf\x11\xe0" # OLE2 signature
or file_data[:4] == b"PK\x03\x04" # Sometimes XLS are actually ZIP
)
return False
@sync_to_async
def load_dataframe(file_data: bytes, content_type: str, file_name: str) -> Dict[str, pd.DataFrame]:
"""
Load tabular file into pandas DataFrames.
Returns a dictionary mapping sheet/table names to DataFrames.
For CSV files, uses 'default' as the key.
For Excel files, uses sheet names as keys.
"""
try:
# Handle CSV files - also accept text/plain if file extension is .csv
if content_type in ["text/csv", "application/csv"] or (
content_type == "text/plain" and file_name.lower().endswith(".csv")
):
# Detect the separator
separator = detect_csv_separator(file_data)
logger.debug("Detected CSV separator: %r", separator)
# Read CSV with detected separator
df = pd.read_csv(
BytesIO(file_data),
sep=separator,
on_bad_lines="skip", # Skip problematic lines
engine="python", # More flexible parser
encoding="utf-8",
)
if df.empty:
raise ValueError("CSV file appears to be empty or could not be parsed")
return {"default": df}
elif content_type in [
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"application/vnd.ms-excel",
"application/excel",
] or file_name.lower().endswith((".xlsx", ".xls", ".xlsm", ".xlsb")):
# Validate Excel file format before attempting to read
if not _is_valid_excel_file(file_data, file_name):
logger.warning(
"File '%s' does not appear to be a valid Excel file. "
"File size: %d bytes, First bytes: %r",
file_name,
len(file_data),
file_data[:20] if len(file_data) >= 20 else file_data,
)
raise ValueError(
f"File '{file_name}' does not appear to be a valid Excel file. "
"It may be corrupted or in an unsupported format."
)
file_lower = file_name.lower()
dataframes = {}
# Try different engines based on file extension
if file_lower.endswith(".xls"):
# Old Excel format - try xlrd engine
try:
logger.debug("Attempting to read .xls file with xlrd engine")
excel_file = pd.ExcelFile(BytesIO(file_data), engine="xlrd")
dataframes = {
sheet_name: excel_file.parse(sheet_name)
for sheet_name in excel_file.sheet_names
}
except Exception as xlrd_error:
logger.warning("Failed to read .xls with xlrd: %s", xlrd_error)
# Fallback: try openpyxl (sometimes .xls files are actually .xlsx)
try:
logger.debug("Trying openpyxl as fallback for .xls file")
excel_file = pd.ExcelFile(BytesIO(file_data), engine="openpyxl")
dataframes = {
sheet_name: excel_file.parse(sheet_name)
for sheet_name in excel_file.sheet_names
}
except Exception as openpyxl_error:
logger.error("Failed to read .xls with both engines: %s", openpyxl_error)
raise ValueError(
f"Failed to read Excel file '{file_name}': "
f"xlrd error: {xlrd_error}, openpyxl error: {openpyxl_error}"
) from openpyxl_error
else:
# XLSX format - try openpyxl first
try:
logger.debug("Attempting to read Excel file with openpyxl engine")
excel_file = pd.ExcelFile(BytesIO(file_data), engine="openpyxl")
dataframes = {
sheet_name: excel_file.parse(sheet_name)
for sheet_name in excel_file.sheet_names
}
except Exception as openpyxl_error:
logger.warning("Failed to read with openpyxl: %s", openpyxl_error)
# Try calamine engine if available (faster and more robust)
try:
logger.debug("Trying calamine engine as fallback")
excel_file = pd.ExcelFile(BytesIO(file_data), engine="calamine")
dataframes = {
sheet_name: excel_file.parse(sheet_name)
for sheet_name in excel_file.sheet_names
}
except ImportError:
logger.debug("calamine engine not available")
raise ValueError(
f"Failed to read Excel file '{file_name}' with openpyxl: {openpyxl_error}. "
"The file may be corrupted or in an unsupported format."
) from openpyxl_error
except Exception as calamine_error:
logger.error("Failed to read with calamine: %s", calamine_error)
raise ValueError(
f"Failed to read Excel file '{file_name}': "
f"openpyxl error: {openpyxl_error}, calamine error: {calamine_error}"
) from calamine_error
if not dataframes:
raise ValueError(f"Excel file '{file_name}' contains no readable sheets")
logger.info(
"Successfully loaded Excel file '%s' with %d sheet(s): %s",
file_name,
len(dataframes),
list(dataframes.keys()),
)
return dataframes
else:
raise ValueError(f"Unsupported content type: {content_type}")
except Exception as e:
logger.error("Error loading tabular file: %s", e, exc_info=True)
raise ModelCannotRetry(f"Failed to load file: {str(e)}") from e
def generate_metadata(dataframes: Dict[str, pd.DataFrame], file_name: str) -> Dict[str, Any]:
"""
Generate metadata about the tabular file.
Returns:
Dictionary containing:
- sheets: List of sheet/table names
- schemas: Dictionary mapping sheet names to their schemas
- row_counts: Dictionary mapping sheet names to row counts
- column_info: Dictionary mapping sheet names to column information
"""
metadata = {
"file_name": file_name,
"sheets": list(dataframes.keys()),
"schemas": {},
"row_counts": {},
"column_info": {},
}
for sheet_name, df in dataframes.items():
# Schema: column names and types
metadata["schemas"][sheet_name] = {
col: str(dtype) for col, dtype in df.dtypes.items()
}
# Row count
metadata["row_counts"][sheet_name] = len(df)
# Column info: name, type, sample values, null counts
metadata["column_info"][sheet_name] = {}
for col in df.columns:
col_info = {
"type": str(df[col].dtype),
"null_count": int(df[col].isna().sum()),
"unique_count": int(df[col].nunique()),
}
# Add sample values (non-null)
sample_values = df[col].dropna().head(5).tolist()
if sample_values:
col_info["sample_values"] = [str(v) for v in sample_values]
metadata["column_info"][sheet_name][col] = col_info
return metadata
async def generate_query(
user_query: str, metadata: Dict[str, Any], query_agent: BaseAgent, ctx: RunContext
) -> str:
"""
Use an LLM agent to generate a pandas query from user query and file metadata.
"""
metadata_str = json.dumps(metadata, indent=2)
prompt = f"""You are a data analysis assistant. Given a user query and file metadata, generate a Python pandas query to answer the question.
File metadata:
{metadata_str}
User query: {user_query}
Generate a Python code snippet that:
1. Uses pandas operations (filter, groupby, aggregate, etc.)
2. Works with the dataframes loaded in memory (available as 'dataframes' dict)
3. Assigns the final result to a variable named 'result'
4. Handles the specific sheet/table if multiple sheets exist
5. ALWAYS handles NaN/NA values in boolean conditions using .notna() or .fillna() before filtering
6. If the user asks for a plot/graph/chart, create it using matplotlib and save to 'plot_image' variable as base64
IMPORTANT RULES:
- The code MUST assign the final result to a variable named 'result'
- When filtering with conditions, ALWAYS check for NaN first: df[df['col'].notna() & (df['col'] > value)]
- Use .dropna() if you need to remove rows with missing values
- Use .fillna() if you need to replace missing values
- If plotting: use plt (already imported), create the plot, convert to base64:
```python
plt.figure(figsize=(10, 6))
# ... your plot code ...
buf = BytesIO()
plt.savefig(buf, format='png')
buf.seek(0)
plot_image = base64.b64encode(buf.getvalue()).decode('utf-8')
plt.close()
```
NOTE: Do NOT use import statements - plt, base64, BytesIO are already available.
Return ONLY the Python code, without markdown formatting or explanations. The code should be executable and use variables:
- 'dataframes': dict mapping sheet names to DataFrames
- Sheet names available: {', '.join(metadata['sheets'])}
Example format (without plot):
df = dataframes['default']
df = df[df['column'].notna()] # Remove NaN values first
result = df[df['column'] > 100].groupby('category').sum()
Example format (with plot):
df = dataframes['default']
plt.figure(figsize=(10, 6))
plt.plot(df.index, df['close'])
plt.xlabel('Index')
plt.ylabel('Close')
plt.title('Close vs Index')
buf = BytesIO()
plt.savefig(buf, format='png')
buf.seek(0)
plot_image = base64.b64encode(buf.getvalue()).decode('utf-8')
plt.close()
result = "Plot generated successfully. The plot image has been saved and is available in the tool response."
IMPORTANT:
- Do NOT use import statements in the code. All necessary modules (pd, plt, np, base64, BytesIO) are already available. Do NOT use anything else than these modules.
- When returning the result text, mention that a plot was generated and will be available in the response, but do NOT include the URL in the text - the system will handle displaying it.
Generate the query code:"""
try:
response = await query_agent.run(prompt, usage=ctx.usage)
query_code = response.output.strip()
# Extract code from markdown code blocks if present
if "```python" in query_code:
query_code = query_code.split("```python")[1].split("```")[0].strip()
elif "```" in query_code:
query_code = query_code.split("```")[1].split("```")[0].strip()
return query_code
except Exception as e:
logger.error("Error generating query: %s", e, exc_info=True)
raise ModelRetry("Failed to generate query. Please try rephrasing your question.") from e
@sync_to_async
def execute_query(query_code: str, dataframes: Dict[str, pd.DataFrame]) -> Any:
"""
Execute the generated pandas query safely.
Note: Uses exec() in a restricted environment. The query code is generated
by an LLM based on file metadata, so it should be relatively safe, but
we restrict the available builtins and globals.
"""
try:
# Pre-process dataframes to handle common issues
processed_dataframes = {}
for name, df in dataframes.items():
# Make a copy to avoid modifying original
df_processed = df.copy()
# Replace common NaN representations
df_processed = df_processed.replace(["", " ", "nan", "NaN", "None", "null"], pd.NA)
processed_dataframes[name] = df_processed
# Create a safe execution environment
safe_globals = {
"pd": pd,
"plt": plt,
"np": np,
"base64": base64,
"BytesIO": BytesIO,
"dataframes": processed_dataframes,
"__builtins__": {
"len": len,
"str": str,
"int": int,
"float": float,
"bool": bool,
"list": list,
"dict": dict,
"set": set,
"tuple": tuple,
"range": range,
"sum": sum,
"max": max,
"min": min,
"abs": abs,
"round": round,
},
}
# Clean up query code - remove any import statements that might cause issues
# Split by lines and filter out import statements
lines = query_code.split("\n")
cleaned_lines = [
line
for line in lines
if not line.strip().startswith("import ") and not line.strip().startswith("from ")
]
query_code = "\n".join(cleaned_lines)
# Execute the query in a restricted namespace
local_vars = {}
exec(query_code, safe_globals, local_vars) # noqa: S102
# Get the result - check if 'result' variable exists, otherwise try 'df'
if "result" in local_vars:
result = local_vars["result"]
elif "df" in local_vars:
result = local_vars["df"]
else:
# If no explicit result variable, get the last expression
# This is a fallback - ideally the LLM should assign to 'result'
raise ValueError("Query must assign result to 'result' variable")
# Check if a plot was generated
plot_image = None
if "plot_image" in local_vars:
plot_image = local_vars["plot_image"]
logger.info("Plot image generated")
# Convert result to a serializable format
result = _convert_to_serializable(result)
return {"result": result, "plot_image": plot_image}
except Exception as e:
logger.error("Error executing query: %s", e, exc_info=True)
# Provide more helpful error message
error_msg = str(e)
if "NaN" in error_msg or "NA" in error_msg:
error_msg = (
f"{error_msg}. "
"The query may need to handle missing values (NaN/NA) using .notna() or .dropna() before filtering."
)
raise ModelCannotRetry(f"Failed to execute query: {error_msg}") from e
@last_model_retry_soft_fail
async def data_analysis(ctx: RunContext, query: str) -> ToolReturn:
"""
Analyze tabular data files (CSV, Excel) based on user query.
Can also generate plots/graphs/charts.
This tool:
1. Loads the tabular file(s) from attachments
2. Generates metadata about the file structure
3. Uses an LLM to generate a pandas query based on user query
4. Executes the query and returns results
Args:
ctx (RunContext): The run context containing the conversation.
query (str): The user's data analysis question.
Returns:
ToolReturn: Contains the analysis results and metadata.
"""
try:
# Find tabular files in attachments
# First, get all attachments for debugging
all_attachments = await sync_to_async(list)(
ctx.deps.conversation.attachments.all()
)
logger.info(
"All attachments in conversation: %s",
[
{
"file_name": a.file_name,
"content_type": a.content_type,
"upload_state": a.upload_state,
"conversion_from": a.conversion_from,
}
for a in all_attachments
],
)
# Find tabular files - exclude converted files (they have conversion_from set)
# First try by content_type
tabular_attachments_by_type = await sync_to_async(list)(
ctx.deps.conversation.attachments.filter(
content_type__in=TABULAR_MIME_TYPES,
upload_state=AttachmentStatus.READY,
)
.filter(
Q(conversion_from__isnull=True) | Q(conversion_from="")
)
)
# If no files found by content_type, try by file extension as fallback
# (some systems detect CSV as text/plain instead of text/csv)
if not tabular_attachments_by_type:
csv_extensions = [".csv", ".xlsx", ".xls"]
all_ready_attachments = await sync_to_async(list)(
ctx.deps.conversation.attachments.filter(
upload_state=AttachmentStatus.READY,
)
.filter(
Q(conversion_from__isnull=True) | Q(conversion_from="")
)
)
tabular_attachments = [
att
for att in all_ready_attachments
if any(att.file_name.lower().endswith(ext) for ext in csv_extensions)
# Exclude Markdown files (converted files have .md extension or content_type text/markdown)
and not att.file_name.lower().endswith(".md")
and att.content_type != "text/markdown"
]
if tabular_attachments:
logger.info(
"Found %d tabular file(s) by extension fallback (content_type was not recognized): %s",
len(tabular_attachments),
[f"{a.file_name} ({a.content_type})" for a in tabular_attachments],
)
else:
tabular_attachments = tabular_attachments_by_type
# If still no files found, check if there are converted files that might have originals
# This handles the case where an Excel file was converted to Markdown for RAG
if not tabular_attachments:
# Look for converted files with tabular extensions
csv_extensions = [".csv", ".xlsx", ".xls"]
converted_attachments = await sync_to_async(list)(
ctx.deps.conversation.attachments.filter(
upload_state=AttachmentStatus.READY,
)
.exclude(
Q(conversion_from__isnull=True) | Q(conversion_from="")
)
)
# For each converted file, try to find the original
for converted_att in converted_attachments:
if any(converted_att.file_name.lower().endswith(ext) for ext in csv_extensions):
# Try to find the original file using conversion_from key
original_key = converted_att.conversion_from
if original_key:
original_attachment = await sync_to_async(
ctx.deps.conversation.attachments.filter(
key=original_key,
upload_state=AttachmentStatus.READY,
).first
)()
if original_attachment:
logger.info(
"Found original file '%s' for converted file '%s'",
original_attachment.file_name,
converted_att.file_name,
)
tabular_attachments.append(original_attachment)
break
logger.info(
"Found %d tabular attachment(s): %s",
len(tabular_attachments),
[f"{a.file_name} ({a.content_type})" for a in tabular_attachments],
)
if not tabular_attachments:
raise ModelCannotRetry(
"No tabular files (CSV or Excel) found in the conversation. "
"Please upload a CSV or Excel file first. "
"Note: If you uploaded an Excel file that was converted to Markdown for RAG, "
"the original file must still be available."
)
# Use the first tabular file
attachment = tabular_attachments[0]
logger.info("Analyzing file: %s (type: %s)", attachment.file_name, attachment.content_type)
# Load file data
file_data = await read_tabular_file(attachment)
# Validate that this is actually a valid Excel/CSV file (not a converted Markdown file)
# Check if it's an Excel file that should have ZIP signature
if attachment.file_name.lower().endswith((".xlsx", ".xls", ".xlsm", ".xlsb")):
if not _is_valid_excel_file(file_data, attachment.file_name):
logger.warning(
"File '%s' does not appear to be a valid Excel file. "
"It may be a converted Markdown file. Searching for original...",
attachment.file_name,
)
# Try to find the original file
# Look for an attachment with the same name but without conversion_from
original_attachment = await sync_to_async(
ctx.deps.conversation.attachments.filter(
file_name=attachment.file_name,
upload_state=AttachmentStatus.READY,
)
.filter(
Q(conversion_from__isnull=True) | Q(conversion_from="")
)
.exclude(pk=attachment.pk)
.first
)()
if original_attachment:
logger.info(
"Found original file '%s' (key: %s), using it instead",
original_attachment.file_name,
original_attachment.key,
)
attachment = original_attachment
file_data = await read_tabular_file(attachment)
elif hasattr(attachment, 'conversion_from') and attachment.conversion_from:
# Try to find by key if this file has a conversion_from
original_attachment = await sync_to_async(
ctx.deps.conversation.attachments.filter(
key=attachment.conversion_from,
upload_state=AttachmentStatus.READY,
).first
)()
if original_attachment:
logger.info(
"Found original file via conversion_from: '%s'",
original_attachment.file_name,
)
attachment = original_attachment
file_data = await read_tabular_file(attachment)
else:
raise ModelCannotRetry(
f"File '{attachment.file_name}' appears to be a converted Markdown file, "
"not the original Excel file. The original file is not available. "
"Please re-upload the original Excel file."
)
else:
raise ModelCannotRetry(
f"File '{attachment.file_name}' does not appear to be a valid Excel file. "
"It may be corrupted or in an unsupported format."
)
# Load into pandas DataFrames
dataframes = await load_dataframe(file_data, attachment.content_type, attachment.file_name)
# Generate metadata
metadata = generate_metadata(dataframes, attachment.file_name)
logger.debug("File metadata: %s", json.dumps(metadata, indent=2))
# Generate query using LLM
# NOTE:
# We intentionally create a "bare" Agent instance here instead of using BaseAgent
# with tools enabled. Using BaseAgent would attach all configured tools (including
# this data_analysis tool itself), which can cause the model to try to call tools
# while we're already inside a tool execution, leading to nested tool calls and
# failures like "Failed to generate query. Please try rephrasing your question.".
#
# Here we reuse the same model configuration as BaseAgent but WITHOUT any tools,
# so this internal call is purely text-to-text.
llm_config = settings.LLM_CONFIGURATIONS[settings.LLM_DEFAULT_MODEL_HRID]
if llm_config.is_custom:
model_instance = prepare_custom_model(llm_config)
else:
# Rely on pydantic-ai's built-in model registry / name inference
model_instance = llm_config.model_name
# Use the same keyword as when using BaseAgent, which forwards to Agent.
# On the current pydantic_ai version, the correct kwarg is `output_type`,
# not `result_type` (passing `result_type` raises a UserError).
query_agent = Agent(model=model_instance, output_type=str)
query_code = await generate_query(query, metadata, query_agent, ctx)
logger.debug("Generated query: %s", query_code)
# Execute query
try:
execution_result = await execute_query(query_code, dataframes)
result = execution_result.get("result")
plot_image_base64 = execution_result.get("plot_image")
except Exception as e:
logger.error("Query execution failed: %s", e, exc_info=True)
raise ModelRetry(
f"Failed to execute the generated query: {str(e)}. "
"Please try rephrasing your question."
) from e
# Format result for return
return_value = {
"query": query,
"query_code": query_code,
"result": result,
"metadata": metadata,
}
# Save plot image to storage if generated
plot_url = None
plot_attachment = None
if plot_image_base64:
try:
# Decode base64 image
plot_image_data = base64.b64decode(plot_image_base64)
# Generate a unique filename for the plot
plot_filename = f"plot_{uuid.uuid4().hex[:8]}.png"
plot_key = f"{ctx.deps.conversation.pk}/plots/{plot_filename}"
# Save to storage
await sync_to_async(default_storage.save)(
plot_key, BytesIO(plot_image_data)
)
# Create a permanent attachment record in the database
plot_attachment = await sync_to_async(ChatConversationAttachment.objects.create)(
conversation=ctx.deps.conversation,
uploaded_by=ctx.deps.user,
key=plot_key,
file_name=plot_filename,
content_type="image/png",
upload_state=AttachmentStatus.READY,
size=len(plot_image_data),
)
# Generate presigned URL for immediate access (valid for 1 hour)
plot_url = await sync_to_async(generate_retrieve_policy)(plot_key)
logger.info(
"Plot image saved to storage and database: %s (presigned URL: %s)",
plot_key,
plot_url[:50] + "..."
)
except Exception as e:
logger.error("Failed to save plot image: %s", e, exc_info=True)
# Continue without plot URL if save fails
if plot_url:
# Include both local and presigned URLs
return_value["plot_url"] = plot_url # Presigned URL for direct access
return_value["plot_local_url"] = f"/media-key/{plot_key}" # Local URL for reference
# Include attachment ID for reference
if plot_attachment:
return_value["plot_attachment_id"] = str(plot_attachment.pk)
return ToolReturn(
return_value=return_value,
metadata={"file_name": attachment.file_name, "content_type": attachment.content_type},
)
except (ModelCannotRetry, ModelRetry):
# Re-raise these as-is
raise
except Exception as exc:
# Unexpected error - stop and inform user
logger.exception("Unexpected error in data_analysis: %s", exc)
raise ModelCannotRetry(
f"An unexpected error occurred during data analysis: {type(exc).__name__}. "
"You must explain this to the user and not try to answer based on your knowledge."
) from exc
def add_data_analysis_tool(agent: Agent) -> None:
"""Add the data analysis tool to an existing agent."""
@agent.tool(retries=2)
@functools.wraps(data_analysis)
async def data_analysis_tool(ctx: RunContext, query: str) -> ToolReturn:
"""
Analyze tabular data files (CSV, Excel) based on user query.
This tool loads tabular files, generates metadata about their structure,
uses an LLM to generate a pandas query based on the user's question,
executes the query, and returns the results.
Use this tool when the user asks questions about data in CSV or Excel files,
such as:
- "What is the average sales by region?"
- "Show me the top 10 products by revenue"
- "How many records are in this file?"
- "Filter data where column X is greater than Y"
Args:
ctx (RunContext): The run context containing the conversation.
query (str): The user's data analysis question.
"""
# Import here to avoid circular import
from chat.tools.data_analysis import data_analysis as _data_analysis
return await _data_analysis(ctx, query)
@agent.instructions
def data_analysis_instructions() -> str:
"""Dynamic system prompt function to add data analysis instructions."""
return (
"When the user asks questions about data in CSV or Excel files, "
"use the data_analysis tool to analyze the data and answer their question. "
"The tool will handle loading the file, generating queries, and executing them. "
"When a plot is generated, the tool returns a 'plot_url' in the result. "
"Use this presigned URL directly in markdown image syntax: ![Description](plot_url). "
"Do NOT use local URLs like /media-key/... - always use the presigned URL from plot_url. "
"Present the results clearly to the user."
)
@@ -26,7 +26,7 @@ def add_document_rag_search_tool(agent: Agent) -> None:
document_store = document_store_backend(ctx.deps.conversation.collection_id)
rag_results = document_store.search(query)
rag_results = document_store.search(query, session=ctx.deps.session)
ctx.usage += RunUsage(
input_tokens=rag_results.usage.prompt_tokens,
+12 -5
View File
@@ -127,7 +127,7 @@ async def _extract_and_summarize_snippets_async(query: str, url: str) -> List[st
return []
async def _fetch_and_store_async(url: str, document_store) -> None:
async def _fetch_and_store_async(url: str, document_store, **kwargs) -> None:
"""Fetch, extract and store text content from the URL in the document store."""
try:
@@ -136,7 +136,7 @@ async def _fetch_and_store_async(url: str, document_store) -> None:
logger.debug("Fetched document: %s", document)
if document:
await document_store.astore_document(url, document)
await document_store.astore_document(url, document, **kwargs)
except DocumentFetchError as e:
logger.warning("Failed to fetch and store %s: %s", url, e)
# Continue with other documents
@@ -307,19 +307,26 @@ async def web_search_brave_with_document_backend(ctx: RunContext, query: str) ->
temp_collection_name = f"tmp-{uuid.uuid4()}"
try:
async with document_store_backend.temporary_collection_async(
temp_collection_name
temp_collection_name, session=ctx.deps.session
) as document_store:
# Fetch and store all documents concurrently
tasks = [
_fetch_and_store_async(result["url"], document_store)
_fetch_and_store_async(
result["url"],
document_store,
user_sub=ctx.deps.user.sub,
session=ctx.deps.session,
)
for result in raw_search_results
]
await asyncio.gather(*tasks, return_exceptions=True)
# Perform RAG search
rag_results = await document_store.asearch(
query,
query=query,
results_count=settings.BRAVE_RAG_WEB_SEARCH_CHUNK_NUMBER,
session=ctx.deps.session,
user_sub=ctx.deps.user.sub,
)
logger.info("RAG search returned: %s", rag_results)
@@ -2,6 +2,6 @@
This module contains the EventEncoder class.
"""
from .encoder import EventEncoder
from .encoder import CURRENT_EVENT_ENCODER_VERSION, EventEncoder, EventEncoderVersion
__all__ = ["EventEncoder"]
__all__ = ["EventEncoder", "CURRENT_EVENT_ENCODER_VERSION", "EventEncoderVersion"]
@@ -1,6 +1,7 @@
"""Event Encoder for Vercel AI SDK"""
from typing import Literal, Union
from enum import Enum
from typing import Union
from ..core.events_v4 import BaseEvent as V4BaseEvent
from ..core.events_v4 import TextPart
@@ -8,16 +9,26 @@ from ..core.events_v5 import BaseEvent as V5BaseEvent
from ..core.events_v5 import TextDeltaEvent
class EventEncoderVersion(str, Enum):
"""Enumeration of supported event encoder versions."""
V4 = "v4"
V5 = "v5"
CURRENT_EVENT_ENCODER_VERSION = EventEncoderVersion.V4 # used encoder version
class EventEncoder:
"""
Encodes events for the Vercel AI SDK based on the specified version.
"""
def __init__(self, version: Literal["v4", "v5"] = None):
def __init__(self, version: EventEncoderVersion):
"""
Initializes the EventEncoder with the specified version.
"""
if version not in ["v4", "v5"]:
if version not in [EventEncoderVersion.V4, EventEncoderVersion.V5]:
raise ValueError("Unsupported version. Supported versions are 'v4' and 'v5'.")
self.version = version
@@ -28,7 +39,7 @@ class EventEncoder:
"""
return "text/event-stream"
def encode(self, event: Union[V5BaseEvent, V5BaseEvent]) -> str | None:
def encode(self, event: Union[V4BaseEvent, V5BaseEvent]) -> str | None:
"""
Encodes an event based on the version.
@@ -38,15 +49,15 @@ class EventEncoder:
str | None: The encoded event as a string,
or None if the event type is not adapted to the SDK version.
"""
if self.version == "v4" and isinstance(event, V4BaseEvent):
if self.version == EventEncoderVersion.V4 and isinstance(event, V4BaseEvent):
return self._encode_v4_streaming(event)
if self.version == "v5" and isinstance(event, V5BaseEvent):
if self.version == EventEncoderVersion.V5 and isinstance(event, V5BaseEvent):
return self._encode_sse(event)
return None
def encode_text(self, event: Union[V5BaseEvent, V5BaseEvent]) -> str | None:
def encode_text(self, event: Union[V4BaseEvent, V5BaseEvent]) -> str | None:
"""
Encodes an event based on the version.
@@ -56,10 +67,10 @@ class EventEncoder:
str | None: The encoded event as a string,
or None if the event type is not adapted to the SDK version.
"""
if self.version == "v4" and isinstance(event, TextPart):
if self.version == EventEncoderVersion.V4 and isinstance(event, TextPart):
return event.text
if self.version == "v5" and isinstance(event, TextDeltaEvent):
if self.version == EventEncoderVersion.V5 and isinstance(event, TextDeltaEvent):
return event.delta
return None
@@ -70,7 +81,7 @@ class EventEncoder:
"""
return f"{event.type}:{event.model_dump_json(by_alias=True, exclude={'type'})}\n"
def _encode_sse(self, event: Union[V5BaseEvent, V5BaseEvent]) -> str:
def _encode_sse(self, event: Union[V4BaseEvent, V5BaseEvent]) -> str:
"""
Encodes an event into an SSE string.
"""
+12 -9
View File
@@ -7,11 +7,13 @@ from uuid import uuid4
from django.conf import settings
from django.core.files.storage import default_storage
from django.http import Http404, StreamingHttpResponse
from django.utils.decorators import method_decorator
import langfuse
import magic
import posthog
from lasuite.malware_detection import malware_detection
from lasuite.oidc_login.decorators import refresh_oidc_access_token
from rest_framework import decorators, filters, mixins, permissions, status, viewsets
from rest_framework.exceptions import MethodNotAllowed, PermissionDenied, ValidationError
from rest_framework.response import Response
@@ -26,6 +28,7 @@ from core.filters import remove_accents
from activation_codes.permissions import IsActivatedUser
from chat import models, serializers
from chat.clients.pydantic_ai import AIAgentService
from chat.keepalive import stream_with_keepalive_async, stream_with_keepalive_sync
from chat.serializers import ChatConversationRequestSerializer
logger = logging.getLogger(__name__)
@@ -122,6 +125,7 @@ class ChatViewSet( # pylint: disable=too-many-ancestors, abstract-method
self.permission_classes = []
return super().get_permissions()
@method_decorator(refresh_oidc_access_token)
@decorators.action(
methods=["post"],
detail=True,
@@ -173,6 +177,7 @@ class ChatViewSet( # pylint: disable=too-many-ancestors, abstract-method
ai_service = AIAgentService(
conversation=conversation,
user=self.request.user,
session=request.session,
model_hrid=model_hrid,
language=(
self.request.user.language
@@ -188,29 +193,28 @@ class ChatViewSet( # pylint: disable=too-many-ancestors, abstract-method
if is_async_mode:
logger.debug("Using ASYNC streaming for chat conversation.")
if protocol == "data":
streaming_content = ai_service.stream_data_async(
base_stream = ai_service.stream_data_async(
messages, force_web_search=force_web_search
)
else: # Default to 'text' protocol
streaming_content = ai_service.stream_text_async(
base_stream = ai_service.stream_text_async(
messages, force_web_search=force_web_search
)
streaming_content = stream_with_keepalive_async(base_stream)
else:
logger.debug("Using SYNC streaming for chat conversation.")
if protocol == "data":
streaming_content = ai_service.stream_data(
messages, force_web_search=force_web_search
)
base_stream = ai_service.stream_data(messages, force_web_search=force_web_search)
else: # Default to 'text' protocol
streaming_content = ai_service.stream_text(
messages, force_web_search=force_web_search
)
base_stream = ai_service.stream_text(messages, force_web_search=force_web_search)
streaming_content = stream_with_keepalive_sync(base_stream)
response = StreamingHttpResponse(
streaming_content,
content_type="text/event-stream",
headers={
"x-vercel-ai-data-stream": "v1", # This header is used for Vercel AI streaming,
"X-Accel-Buffering": "no", # Prevent nginx buffering
},
)
return response
@@ -371,7 +375,6 @@ class ChatConversationAttachmentViewSet(
owner=self.request.user,
).exists():
raise Http404
file_name = serializer.validated_data["file_name"]
extension = file_name.rpartition(".")[-1] if "." in file_name else None
+1 -1
View File
@@ -22,7 +22,7 @@ def no_http_requests(monkeypatch):
Credits: https://blog.jerrycodes.com/no-http-requests/
"""
allowed_hosts = {"localhost", "minio", "minio:9000"}
allowed_hosts = {"localhost", "127.0.0.1", "minio", "minio:9000"}
original_urlopen = HTTPConnectionPool.urlopen
def urlopen_mock(self, method, url, *args, **kwargs):
+28 -1
View File
@@ -841,6 +841,23 @@ USER QUESTION:
environ_prefix=None,
)
# Find
FIND_API_KEY = values.Value(
None,
environ_name="FIND_API_KEY",
environ_prefix=None,
)
FIND_API_URL = values.Value(
"https://app-find/api",
environ_name="FIND_API_URL",
environ_prefix=None,
)
FIND_API_TIMEOUT = values.PositiveIntegerValue(
default=30, # seconds
environ_name="FIND_API_TIMEOUT",
environ_prefix=None,
)
# Logging
# We want to make it easy to log to console but by default we log production
# to Sentry and don't want to log to console.
@@ -911,7 +928,9 @@ USER QUESTION:
LANGFUSE_MEDIA_UPLOAD_ENABLED = values.BooleanValue(
default=False, environ_name="LANGFUSE_MEDIA_UPLOAD_ENABLED", environ_prefix=None
)
AUTO_TITLE_AFTER_USER_MESSAGES = values.PositiveIntegerValue(
default=None, environ_name="AUTO_TITLE_AFTER_USER_MESSAGES", environ_prefix=None
)
# WARNING: Testing purpose only. Do not use in production.
WARNING_MOCK_CONVERSATION_AGENT = values.BooleanValue(
default=False,
@@ -919,6 +938,12 @@ USER QUESTION:
environ_prefix=None,
)
# Default keepalive interval: 55s (safely below typical 60s proxy timeouts)
# Prevents connection drops during long stream pauses while providing 5s safety margin.
KEEPALIVE_INTERVAL = values.PositiveIntegerValue(
default=55, environ_name="KEEPALIVE_INTERVAL", environ_prefix=None
)
# pylint: disable=invalid-name
@property
def ENVIRONMENT(self):
@@ -1131,6 +1156,8 @@ class Test(Base):
POSTHOG_KEY = None
AUTO_TITLE_AFTER_USER_MESSAGES = None
def __init__(self):
# pylint: disable=invalid-name
self.INSTALLED_APPS += ["drf_spectacular_sidecar"]
@@ -3,7 +3,7 @@ msgstr ""
"Project-Id-Version: la-suite-conversations\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2025-10-20 21:48+0000\n"
"PO-Revision-Date: 2025-12-15 13:49\n"
"PO-Revision-Date: 2026-01-16 11:04\n"
"Last-Translator: \n"
"Language-Team: German\n"
"Language: de_DE\n"
@@ -3,7 +3,7 @@ msgstr ""
"Project-Id-Version: la-suite-conversations\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2025-10-20 21:48+0000\n"
"PO-Revision-Date: 2025-12-15 13:49\n"
"PO-Revision-Date: 2026-01-16 11:04\n"
"Last-Translator: \n"
"Language-Team: English\n"
"Language: en_US\n"
@@ -3,7 +3,7 @@ msgstr ""
"Project-Id-Version: la-suite-conversations\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2025-10-20 21:48+0000\n"
"PO-Revision-Date: 2025-12-15 13:49\n"
"PO-Revision-Date: 2026-01-16 11:04\n"
"Last-Translator: \n"
"Language-Team: French\n"
"Language: fr_FR\n"
@@ -3,7 +3,7 @@ msgstr ""
"Project-Id-Version: la-suite-conversations\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2025-10-20 21:48+0000\n"
"PO-Revision-Date: 2025-12-15 13:49\n"
"PO-Revision-Date: 2026-01-16 11:04\n"
"Last-Translator: \n"
"Language-Team: Dutch\n"
"Language: nl_NL\n"
@@ -3,7 +3,7 @@ msgstr ""
"Project-Id-Version: la-suite-conversations\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2025-10-20 21:48+0000\n"
"PO-Revision-Date: 2025-12-15 13:49\n"
"PO-Revision-Date: 2026-01-16 11:04\n"
"Last-Translator: \n"
"Language-Team: Russian\n"
"Language: ru_RU\n"
@@ -3,7 +3,7 @@ msgstr ""
"Project-Id-Version: la-suite-conversations\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2025-10-20 21:48+0000\n"
"PO-Revision-Date: 2025-12-15 13:49\n"
"PO-Revision-Date: 2026-01-16 11:04\n"
"Last-Translator: \n"
"Language-Team: Ukrainian\n"
"Language: uk_UA\n"
+1 -4
View File
@@ -46,6 +46,7 @@ dependencies = [
"easy_thumbnails==2.10.1",
"factory_boy==3.3.3",
"gunicorn==23.0.0",
"jaraco.context>=6.1.0",
"jsonschema==4.25.1",
"langfuse==3.10.0",
"lxml==5.4.0",
@@ -53,10 +54,6 @@ dependencies = [
"markitdown==0.0.2",
"mozilla-django-oidc==4.0.1",
"nested-multipart-parser==1.6.0",
"matplotlib==3.9.2",
"numpy==2.1.3",
"openpyxl==3.1.5",
"pandas==2.2.3",
"posthog==7.0.0",
"pydantic==2.12.4",
"pydantic-ai-slim[openai,mistral,mcp,evals,logfire]==1.17.0",
+54
View File
@@ -0,0 +1,54 @@
"""Utility functions for OIDC token management."""
from functools import wraps
from django.conf import settings
import requests
from lasuite.oidc_login.backends import get_oidc_refresh_token, store_tokens
from rest_framework.exceptions import AuthenticationFailed
def refresh_access_token(session):
"""Refresh the OIDC access token using the refresh token."""
refresh_token = get_oidc_refresh_token(session)
if not refresh_token:
raise AuthenticationFailed({"error": "Refresh token is missing from session"})
response = requests.post(
settings.OIDC_OP_TOKEN_ENDPOINT,
data={
"grant_type": "refresh_token",
"client_id": settings.OIDC_RP_CLIENT_ID,
"client_secret": settings.OIDC_RP_CLIENT_SECRET,
"refresh_token": refresh_token,
},
timeout=5,
)
response.raise_for_status()
token_info = response.json()
store_tokens(
session,
access_token=token_info.get("access_token"),
id_token=None,
refresh_token=token_info.get("refresh_token"),
)
return session
def with_fresh_access_token(func):
"""
Decorator to handle OIDC token refresh and extraction.
Expects 'session' in kwargs and update it with the fresh token.
"""
@wraps(func)
def wrapper(*args, **kwargs):
session = kwargs.pop("session", None)
if session is None:
raise AuthenticationFailed({"error": "Session is required but not provided"})
refreshed_session = refresh_access_token(session)
return func(*args, session=refreshed_session, **kwargs)
return wrapper
+2 -1
View File
@@ -1,6 +1,6 @@
{
"name": "app-conversations",
"version": "0.0.10",
"version": "0.0.11",
"private": true,
"scripts": {
"dev": "next dev",
@@ -9,6 +9,7 @@
"build-theme": "cunningham -g css,ts -o src/cunningham --utility-classes && yarn prettier && yarn stylelint --fix",
"start": "npx -y serve@latest out",
"lint": "tsc --noEmit && next lint",
"lint:fix": "tsc --noEmit && next lint --fix",
"prettier": "prettier --write .",
"stylelint": "stylelint \"**/*.css\"",
"test": "jest",
@@ -1,6 +1,9 @@
import { UseChatOptions, useChat as useAiSdkChat } from '@ai-sdk/react';
import { useQueryClient } from '@tanstack/react-query';
import { useEffect } from 'react';
import { fetchAPI } from '@/api';
import { KEY_LIST_CONVERSATION } from '@/features/chat/api/useConversations';
import { useChatPreferencesStore } from '@/features/chat/stores/useChatPreferencesStore';
const fetchAPIAdapter = (input: RequestInfo | URL, init?: RequestInit) => {
@@ -36,10 +39,46 @@ const fetchAPIAdapter = (input: RequestInfo | URL, init?: RequestInit) => {
return fetchAPI(url, init);
};
interface ConversationMetadataEvent {
type: 'conversation_metadata';
conversationId: string;
title: string;
}
// Type guard to check if an item is a ConversationMetadataEvent
function isConversationMetadataEvent(
item: unknown,
): item is ConversationMetadataEvent {
return (
typeof item === 'object' &&
item !== null &&
'type' in item &&
item.type === 'conversation_metadata' &&
'conversationId' in item &&
typeof item.conversationId === 'string' &&
'title' in item &&
typeof item.title === 'string'
);
}
export function useChat(options: Omit<UseChatOptions, 'fetch'>) {
return useAiSdkChat({
const queryClient = useQueryClient();
const result = useAiSdkChat({
...options,
maxSteps: 3,
fetch: fetchAPIAdapter,
});
useEffect(() => {
if (result.data && Array.isArray(result.data)) {
for (const item of result.data) {
if (isConversationMetadataEvent(item)) {
void queryClient.invalidateQueries({
queryKey: [KEY_LIST_CONVERSATION],
});
}
}
}
}, [result.data, queryClient]);
return result;
}
@@ -0,0 +1,62 @@
import {
UseMutationOptions,
useMutation,
useQueryClient,
} from '@tanstack/react-query';
import { APIError, errorCauses, fetchAPI } from '@/api';
import { KEY_LIST_CONVERSATION } from './useConversations';
interface RenameConversationProps {
conversationId: string;
title: string;
}
export const renameConversation = async ({
conversationId,
title,
}: RenameConversationProps): Promise<void> => {
const response = await fetchAPI(`chats/${conversationId}/`, {
method: 'PUT',
body: JSON.stringify({
title,
}),
});
if (!response.ok) {
throw new APIError(
'Failed to rename the conversation',
await errorCauses(response),
);
}
};
type UseRenameConversationOptions = UseMutationOptions<
void,
APIError,
RenameConversationProps
>;
export const useRenameConversation = (
options?: UseRenameConversationOptions,
) => {
const queryClient = useQueryClient();
return useMutation<void, APIError, RenameConversationProps>({
mutationFn: renameConversation,
...options,
onSuccess: (data, variables, context) => {
void queryClient.invalidateQueries({
queryKey: [KEY_LIST_CONVERSATION],
});
if (options?.onSuccess) {
void options.onSuccess(data, variables, context);
}
},
onError: (error, variables, context) => {
if (options?.onError) {
void options.onError(error, variables, context);
}
},
});
};
@@ -1,9 +1,4 @@
import {
Message,
ReasoningUIPart,
SourceUIPart,
ToolInvocationUIPart,
} from '@ai-sdk/ui-utils';
import { Message, SourceUIPart, ToolInvocationUIPart } from '@ai-sdk/ui-utils';
import { Modal, ModalSize } from '@openfun/cunningham-react';
import 'katex/dist/katex.min.css'; // `rehype-katex` does not import the CSS for you
import { useRouter } from 'next/router';
@@ -816,30 +811,12 @@ export const Chat = ({
</Box>
)}
{message.parts
?.filter(
(part) =>
part.type === 'reasoning' ||
part.type === 'tool-invocation',
)
?.filter((part) => part.type === 'tool-invocation')
.map(
(
part: ReasoningUIPart | ToolInvocationUIPart,
partIndex: number,
) =>
part.type === 'reasoning' ? (
<Box
key={`reasoning-${partIndex}`}
$background="var(--c--theme--colors--greyscale-100)"
$color="var(--c--theme--colors--greyscale-500)"
$padding={{ all: 'sm' }}
$radius="md"
$css="font-size: 0.9em;"
>
{part.reasoning}
</Box>
) : part.type === 'tool-invocation' &&
isCurrentlyStreaming &&
isLastAssistantMessageInConversation ? (
(part: ToolInvocationUIPart, partIndex: number) =>
part.type === 'tool-invocation' &&
isCurrentlyStreaming &&
isLastAssistantMessageInConversation ? (
<ToolInvocationItem
key={`tool-invocation-${partIndex}`}
toolInvocation={part.toolInvocation}
@@ -1,4 +1,4 @@
import { Button as _Button, useModal } from '@openfun/cunningham-react';
import { useModal } from '@openfun/cunningham-react';
import { useTranslation } from 'react-i18next';
import { css } from 'styled-components';
@@ -6,6 +6,7 @@ import { DropdownMenu, DropdownMenuOption, Icon } from '@/components';
import { ChatConversation } from '@/features/chat/types';
import { ModalRemoveConversation } from './ModalRemoveConversation';
import { ModalRenameConversation } from './ModalRenameConversation';
interface ConversationItemActionsProps {
conversation: ChatConversation;
@@ -17,8 +18,16 @@ export const ConversationItemActions = ({
const { t } = useTranslation();
const deleteModal = useModal();
const renameModal = useModal();
const options: DropdownMenuOption[] = [
{
label: t('Rename chat'),
icon: 'edit',
callback: () => renameModal.open(),
disabled: false,
testId: `conversation-item-actions-rename-${conversation.id}`,
},
{
label: t('Delete chat'),
icon: 'delete',
@@ -71,6 +80,12 @@ export const ConversationItemActions = ({
conversation={conversation}
/>
)}
{renameModal.isOpen && (
<ModalRenameConversation
onClose={renameModal.onClose}
conversation={conversation}
/>
)}
</>
);
};
@@ -58,7 +58,7 @@ export const LeftPanelConversationItem = ({
>
<StyledLink
href={`/chat/${conversation.id}/`}
$css="overflow: auto; flex-grow: 1;"
$css="overflow: auto; flex-grow: 1; color: var(--c--theme--colors--greyscale-900);"
onClick={handleLinkClick}
>
<SimpleConversationItem showAccesses conversation={conversation} />
@@ -46,7 +46,7 @@ export const ModalRemoveConversation = ({
<>
<Button
aria-label={t('Close the modal')}
color="secondary"
color="tertiary"
fullWidth
onClick={() => onClose()}
>
@@ -79,7 +79,10 @@ export const ModalRemoveConversation = ({
</Text>
}
>
<Box className="--converstions--modal-remove-chat">
<Box
className="--conversations--modal-remove-chat"
data-testid="delete-chat-confirm"
>
<Text $size="sm" $variation="600">
{t('Are you sure you want to delete this conversation ?')}
</Text>
@@ -0,0 +1,108 @@
import { Button, Input, Modal, ModalSize } from '@openfun/cunningham-react';
import { useState } from 'react';
import { useTranslation } from 'react-i18next';
import { Box, Text, useToast } from '@/components';
import { useRenameConversation } from '@/features/chat/api/useRenameConversation';
import { ChatConversation } from '@/features/chat/types';
interface ModalRenameConversationProps {
onClose: () => void;
conversation: ChatConversation;
}
export const ModalRenameConversation = ({
onClose,
conversation,
}: ModalRenameConversationProps) => {
const { showToast } = useToast();
const { t } = useTranslation();
const { mutate: renameConversation } = useRenameConversation({
onSuccess: () => {
showToast(
'success',
t('The conversation has been renamed.'),
undefined,
4000,
);
onClose();
},
onError: (error) => {
const errorMessage =
error.cause?.[0] ||
error.message ||
t('An error occurred while renaming the conversation');
showToast('error', errorMessage, undefined, 4000);
},
});
const [newName, setNewName] = useState(conversation.title ?? '');
const handleSubmit = (e: React.FormEvent) => {
e.preventDefault();
const trimmedNewName = newName.trim();
if (trimmedNewName) {
renameConversation({
conversationId: conversation.id,
title: trimmedNewName,
});
}
};
return (
<Modal
isOpen
closeOnClickOutside
onClose={() => onClose()}
aria-label={t('Content modal to rename a conversation')}
rightActions={
<>
<Button
aria-label={t('Close the modal')}
color="tertiary"
onClick={() => onClose()}
>
{t('Cancel')}
</Button>
<Button
aria-label={t('Rename chat')}
color="primary"
type="submit"
form="rename-chat-form"
>
{t('Rename')}
</Button>
</>
}
size={ModalSize.SMALL}
title={
<Text
$size="h6"
as="h6"
$margin={{ all: '0' }}
$align="flex-start"
$variation="1000"
>
{t('Rename chat')}
</Text>
}
>
<Box className="--conversations--modal-rename-chat">
<form
onSubmit={handleSubmit}
id="rename-chat-form"
data-testid="rename-chat-form"
className="mt-s"
>
<Input
type="text"
label={t('New name')}
maxLength={100}
value={newName}
onChange={(e: React.ChangeEvent<HTMLInputElement>) => {
setNewName(e.target.value);
}}
/>
</form>
</Box>
</Modal>
);
};
@@ -0,0 +1,255 @@
import { CunninghamProvider } from '@openfun/cunningham-react';
import { render, screen, waitFor } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { ToastProvider } from '@/components';
import { ChatConversation } from '@/features/chat/types';
import { ConversationItemActions } from '../ConversationItemActions';
const mockPush = jest.fn();
let mockPathname = '/';
jest.mock('next/router', () => ({
useRouter: () => ({
push: mockPush,
pathname: mockPathname,
route: '/',
query: {},
asPath: '/',
}),
}));
jest.mock('next/navigation', () => ({
usePathname: () => mockPathname,
}));
jest.mock('react-i18next', () => ({
useTranslation: () => ({
t: (key: string, options?: Record<string, string>) => {
if (options) {
return Object.entries(options).reduce(
(acc, [k, v]) => acc.replace(`{{${k}}}`, v),
key,
);
}
return key;
},
}),
}));
jest.mock('i18next', () => ({
t: (key: string) => key,
}));
jest.mock('@/features/chat/api/useRenameConversation', () => ({
useRenameConversation: () => ({
mutate: jest.fn(),
}),
}));
jest.mock('@/features/chat/api/useRemoveConversation', () => ({
useRemoveConversation: () => ({
mutate: jest.fn(),
}),
}));
const renderWithProviders = (ui: React.ReactNode) => {
return render(
<CunninghamProvider>
<ToastProvider>{ui}</ToastProvider>
</CunninghamProvider>,
);
};
describe('ConversationItemActions', () => {
const mockConversation: ChatConversation = {
id: 'conv-123',
title: 'Original Title',
messages: [],
created_at: new Date().toISOString(),
updated_at: new Date().toISOString(),
};
beforeEach(() => {
jest.clearAllMocks();
mockPathname = '/';
});
const renderComponent = (conversation = mockConversation) => {
return renderWithProviders(
<ConversationItemActions conversation={conversation} />,
);
};
it('renders the actions button', () => {
renderComponent();
expect(
screen.getByTestId(
`conversation-item-actions-button-${mockConversation.id}`,
),
).toBeInTheDocument();
});
it('renders dropdown menu with correct aria-label', () => {
renderComponent();
expect(
screen.getByLabelText(
`Actions list for conversation ${mockConversation.title}`,
),
).toBeInTheDocument();
});
it('renders dropdown menu with fallback title when conversation has no title', () => {
const untitledConversation = { ...mockConversation, title: '' };
renderComponent(untitledConversation);
expect(
screen.getByLabelText(
`Actions list for conversation Untitled conversation`,
),
).toBeInTheDocument();
});
it('opens dropdown menu when clicking the actions button', async () => {
const user = userEvent.setup();
renderComponent();
const actionsButton = screen.getByLabelText(
`Actions list for conversation ${mockConversation.title}`,
);
await user.click(actionsButton);
expect(
screen.getByTestId(
`conversation-item-actions-rename-${mockConversation.id}`,
),
).toBeInTheDocument();
expect(
screen.getByTestId(
`conversation-item-actions-remove-${mockConversation.id}`,
),
).toBeInTheDocument();
});
it('displays rename and delete options in the dropdown', async () => {
const user = userEvent.setup();
renderComponent();
const actionsButton = screen.getByLabelText(
`Actions list for conversation ${mockConversation.title}`,
);
await user.click(actionsButton);
expect(screen.getByText('Rename chat')).toBeInTheDocument();
expect(screen.getByText('Delete chat')).toBeInTheDocument();
});
it('opens rename modal when clicking rename option', async () => {
const user = userEvent.setup();
renderComponent();
const actionsButton = screen.getByLabelText(
`Actions list for conversation ${mockConversation.title}`,
);
await user.click(actionsButton);
const renameOption = screen.getByTestId(
`conversation-item-actions-rename-${mockConversation.id}`,
);
await user.click(renameOption);
// Modal should be open
await waitFor(() => {
expect(screen.getByRole('dialog')).toBeInTheDocument();
});
expect(screen.getByRole('textbox')).toHaveValue(mockConversation.title);
expect(screen.getByTestId('rename-chat-form')).toBeInTheDocument();
});
it('opens delete modal when clicking delete option', async () => {
const user = userEvent.setup();
renderComponent();
const actionsButton = screen.getByLabelText(
`Actions list for conversation ${mockConversation.title}`,
);
await user.click(actionsButton);
const deleteOption = screen.getByTestId(
`conversation-item-actions-remove-${mockConversation.id}`,
);
await user.click(deleteOption);
await waitFor(() => {
expect(screen.getByRole('dialog')).toBeInTheDocument();
});
expect(screen.getByTestId('delete-chat-confirm')).toBeInTheDocument();
});
it('does not render modals initially', () => {
renderComponent();
expect(screen.queryByRole('dialog')).not.toBeInTheDocument();
expect(screen.queryByTestId('delete-chat-confirm')).not.toBeInTheDocument();
expect(screen.queryByTestId('rename-chat-form')).not.toBeInTheDocument();
});
it('closes rename modal when onClose is called', async () => {
const user = userEvent.setup();
renderComponent();
// Open dropdown and click rename
const actionsButton = screen.getByLabelText(
`Actions list for conversation ${mockConversation.title}`,
);
await user.click(actionsButton);
await user.click(
screen.getByTestId(
`conversation-item-actions-rename-${mockConversation.id}`,
),
);
// Modal should be open
await waitFor(() => {
expect(screen.getByRole('dialog')).toBeInTheDocument();
});
// Close the modal
await user.click(screen.getByText('Cancel'));
await waitFor(() => {
expect(screen.queryByRole('dialog')).not.toBeInTheDocument();
});
});
it('closes delete modal when onClose is called', async () => {
const user = userEvent.setup();
renderComponent();
// Open dropdown and click delete
const actionsButton = screen.getByLabelText(
`Actions list for conversation ${mockConversation.title}`,
);
await user.click(actionsButton);
await user.click(
screen.getByTestId(
`conversation-item-actions-remove-${mockConversation.id}`,
),
);
// Modal should be open
await waitFor(() => {
expect(screen.getByRole('dialog')).toBeInTheDocument();
});
// Close the modal
await user.click(screen.getByText('Cancel'));
await waitFor(() => {
expect(screen.queryByRole('dialog')).not.toBeInTheDocument();
});
});
});
@@ -0,0 +1,280 @@
import { CunninghamProvider } from '@openfun/cunningham-react';
import { render, screen, waitFor } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { useToast } from '@/components';
import { useRenameConversation } from '@/features/chat/api/useRenameConversation';
import { ChatConversation } from '@/features/chat/types';
import { ModalRenameConversation } from '../ModalRenameConversation';
jest.mock('@/components', () => ({
...jest.requireActual('@/components'),
useToast: jest.fn(),
}));
jest.mock('@/features/chat/api/useRenameConversation');
jest.mock('react-i18next', () => ({
useTranslation: () => ({
t: (key: string, options?: Record<string, string>) => {
if (options) {
return Object.entries(options).reduce(
(acc, [k, v]) => acc.replace(`{{${k}}}`, v),
key,
);
}
return key;
},
}),
}));
jest.mock('i18next', () => ({
t: (key: string) => key,
}));
const renderWithProviders = (component: React.ReactNode) => {
return render(<CunninghamProvider>{component}</CunninghamProvider>);
};
describe('ModalRenameConversation', () => {
const mockOnClose = jest.fn();
const mockShowToast = jest.fn();
const mockRenameConversation = jest.fn();
const mockConversation: ChatConversation = {
id: 'conv-123',
title: 'Original Title',
messages: [],
created_at: new Date().toISOString(),
updated_at: new Date().toISOString(),
} as ChatConversation;
beforeEach(() => {
jest.clearAllMocks();
(useToast as jest.Mock).mockReturnValue({
showToast: mockShowToast,
});
(useRenameConversation as jest.Mock).mockReturnValue({
mutate: mockRenameConversation,
});
});
it('renders the modal with correct title and initial value', () => {
renderWithProviders(
<ModalRenameConversation
onClose={mockOnClose}
conversation={mockConversation}
/>,
);
expect(screen.getByText('Rename chat')).toBeInTheDocument();
expect(screen.getByRole('textbox')).toHaveValue('Original Title');
expect(screen.getByText('Cancel')).toBeInTheDocument();
expect(screen.getByText('Rename')).toBeInTheDocument();
});
it('updates input value when user types', async () => {
const user = userEvent.setup();
renderWithProviders(
<ModalRenameConversation
onClose={mockOnClose}
conversation={mockConversation}
/>,
);
const input = screen.getByRole('textbox');
await user.clear(input);
await user.type(input, 'New Title');
expect(input).toHaveValue('New Title');
});
it('closes modal when Cancel button is clicked', async () => {
const user = userEvent.setup();
renderWithProviders(
<ModalRenameConversation
onClose={mockOnClose}
conversation={mockConversation}
/>,
);
await user.click(screen.getByText('Cancel'));
expect(mockOnClose).toHaveBeenCalledTimes(1);
});
it('submits form with new name and shows success toast', async () => {
const user = userEvent.setup();
let onSuccessCallback: (() => void) | undefined;
(useRenameConversation as jest.Mock).mockImplementation(({ onSuccess }) => {
onSuccessCallback = onSuccess;
return { mutate: mockRenameConversation };
});
renderWithProviders(
<ModalRenameConversation
onClose={mockOnClose}
conversation={mockConversation}
/>,
);
const input = screen.getByRole('textbox');
await user.clear(input);
await user.type(input, 'Updated Title');
await user.click(screen.getByText('Rename'));
expect(mockRenameConversation).toHaveBeenCalledWith({
conversationId: 'conv-123',
title: 'Updated Title',
});
onSuccessCallback?.();
await waitFor(() => {
expect(mockShowToast).toHaveBeenCalledWith(
'success',
'The conversation has been renamed.',
undefined,
4000,
);
});
expect(mockOnClose).toHaveBeenCalled();
});
it('does not submit form when new name is empty or whitespace', async () => {
const user = userEvent.setup();
renderWithProviders(
<ModalRenameConversation
onClose={mockOnClose}
conversation={mockConversation}
/>,
);
const input = screen.getByRole('textbox');
await user.clear(input);
await user.type(input, ' ');
await user.click(screen.getByText('Rename'));
expect(mockRenameConversation).not.toHaveBeenCalled();
});
it('shows error toast when rename fails with cause', async () => {
const user = userEvent.setup();
let onErrorCallback: ((error: any) => void) | undefined;
(useRenameConversation as jest.Mock).mockImplementation(({ onError }) => {
onErrorCallback = onError;
return { mutate: mockRenameConversation };
});
renderWithProviders(
<ModalRenameConversation
onClose={mockOnClose}
conversation={mockConversation}
/>,
);
const input = screen.getByRole('textbox');
await user.clear(input);
await user.type(input, 'New Title');
await user.click(screen.getByText('Rename'));
const error = {
cause: ['Specific error from API'],
message: 'Generic error',
};
onErrorCallback?.(error);
await waitFor(() => {
expect(mockShowToast).toHaveBeenCalledWith(
'error',
'Specific error from API',
undefined,
4000,
);
});
});
it('shows error toast with message when no cause is provided', async () => {
const user = userEvent.setup();
let onErrorCallback: ((error: any) => void) | undefined;
(useRenameConversation as jest.Mock).mockImplementation(({ onError }) => {
onErrorCallback = onError;
return { mutate: mockRenameConversation };
});
renderWithProviders(
<ModalRenameConversation
onClose={mockOnClose}
conversation={mockConversation}
/>,
);
const input = screen.getByRole('textbox');
await user.clear(input);
await user.type(input, 'New Title');
await user.click(screen.getByText('Rename'));
const error = {
message: 'Network error',
};
onErrorCallback?.(error);
await waitFor(() => {
expect(mockShowToast).toHaveBeenCalledWith(
'error',
'Network error',
undefined,
4000,
);
});
});
it('shows default error message when error has no cause or message', async () => {
const user = userEvent.setup();
let onErrorCallback: ((error: any) => void) | undefined;
(useRenameConversation as jest.Mock).mockImplementation(({ onError }) => {
onErrorCallback = onError;
return { mutate: mockRenameConversation };
});
renderWithProviders(
<ModalRenameConversation
onClose={mockOnClose}
conversation={mockConversation}
/>,
);
const input = screen.getByRole('textbox');
await user.clear(input);
await user.type(input, 'New Title');
await user.click(screen.getByText('Rename'));
const error = {};
onErrorCallback?.(error);
await waitFor(() => {
expect(mockShowToast).toHaveBeenCalledWith(
'error',
'An error occurred while renaming the conversation',
undefined,
4000,
);
});
});
it('enforces maxLength of 100 characters on input', () => {
renderWithProviders(
<ModalRenameConversation
onClose={mockOnClose}
conversation={mockConversation}
/>,
);
const input = screen.getByRole('textbox');
expect(input).toHaveAttribute('maxLength', '100');
});
});
@@ -88,6 +88,7 @@
"Quick search input": "Saisie de recherche rapide",
"Remove attachment": "Supprimer la pièce jointe",
"Research on the web": "Rechercher sur le web",
"Retry": "Réessayer",
"Search": "Rechercher",
"Search for a chat": "Rechercher un chat",
"Search results": "Résultats de la recherche",
@@ -100,7 +101,7 @@
"Simple chat icon": "Icône de chat simple",
"Something bad happens, please retry.": "Une erreur inattendue s'est produite, veuillez réessayer.",
"Sorry, an error occurred. Please try again.": "Désolé, une erreur s'est produite. Veuillez réessayer.",
"Start a new conversation.": "Commencer une nouvelle conversation.",
"Start a new conversation": "Commencer une nouvelle conversation",
"Start conversation": "Entamer la conversation",
"Stop": "Stop",
"Summarizing...": "Résumé en cours...",
@@ -220,6 +221,7 @@
"Quick search input": "Snelle zoekinvoer",
"Remove attachment": "Bijlage verwijderen",
"Research on the web": "Onderzoek op het internet",
"Retry": "Opnieuw proberen",
"Search": "Zoek",
"Search for a chat": "Zoek naar een chat",
"Search results": "Zoekresultaten",
@@ -232,7 +234,7 @@
"Simple chat icon": "Eenvoudig chatpictogram",
"Something bad happens, please retry.": "Er is iets misgegaan. Probeer het opnieuw.",
"Sorry, an error occurred. Please try again.": "Sorry, er is een fout opgetreden. Probeer het opnieuw.",
"Start a new conversation.": "Begin een nieuw gesprek.",
"Start a new conversation": "Begin een nieuw gesprek",
"Start conversation": "Begin een gesprek",
"Stop": "Stop",
"Summarizing...": "Samenvatten...",
@@ -352,6 +354,7 @@
"Quick search input": "Быстрый поиск",
"Remove attachment": "Удалить вложение",
"Research on the web": "Исследование в Интернете",
"Retry": "Повторить",
"Search": "Поиск",
"Search for a chat": "Поиск беседы",
"Search results": "Результаты поиска",
@@ -364,7 +367,7 @@
"Simple chat icon": "Простой значок чата",
"Something bad happens, please retry.": "Что-то пошло не так, повторите попытку.",
"Sorry, an error occurred. Please try again.": "Извините, произошла ошибка. Пожалуйста, попробуйте ещё раз.",
"Start a new conversation.": "Начать новую беседу.",
"Start a new conversation": "Начать новую беседу",
"Start conversation": "Начать беседу",
"Stop": "Остановить",
"Summarizing...": "Обобщение...",
@@ -484,6 +487,7 @@
"Quick search input": "Швидкий пошук",
"Remove attachment": "Видалити вкладення",
"Research on the web": "Дослідження в Інтернеті",
"Retry": "Повторити",
"Search": "Пошук",
"Search for a chat": "Пошук розмови",
"Search results": "Результати пошуку",
@@ -496,7 +500,7 @@
"Simple chat icon": "Проста піктограма розмови",
"Something bad happens, please retry.": "Сталася помилка, спробуйте ще раз.",
"Sorry, an error occurred. Please try again.": "Вибачте, виникла помилка. Спробуйте ще раз.",
"Start a new conversation.": "Розпочати нову розмову.",
"Start a new conversation": "Розпочати нову розмову",
"Start conversation": "Почати розмову",
"Stop": "Зупинити",
"Summarizing...": "Узагальнення...",
@@ -52,7 +52,7 @@ test.describe('Chat page', () => {
const messageContent = page.getByTestId('assistant-message-content');
await expect(messageContent).toBeVisible();
await expect(messageContent).not.toBeEmpty();
await expect(messageContent).not.toBeEmpty();
// Check history
const chatHistoryLink = page
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "app-e2e",
"version": "0.0.10",
"version": "0.0.11",
"private": true,
"scripts": {
"lint": "eslint . --ext .ts",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "conversations",
"version": "0.0.10",
"version": "0.0.11",
"private": true,
"workspaces": {
"packages": [
@@ -1,6 +1,6 @@
{
"name": "eslint-config-conversations",
"version": "0.0.10",
"version": "0.0.11",
"license": "MIT",
"scripts": {
"lint": "eslint --ext .js ."
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "packages-i18n",
"version": "0.0.10",
"version": "0.0.11",
"private": true,
"scripts": {
"extract-translation": "yarn extract-translation:conversations",
+3 -3
View File
@@ -11355,9 +11355,9 @@ posthog-js@1.249.3:
web-vitals "^4.2.4"
preact@^10.19.3:
version "10.26.6"
resolved "https://registry.npmjs.org/preact/-/preact-10.26.6.tgz"
integrity sha512-5SRRBinwpwkaD+OqlBDeITlRgvd8I8QlxHJw9AxSdMNV6O+LodN9nUyYGpSF7sadHjs6RzeFShMexC6DbtWr9g==
version "10.24.0"
resolved "https://registry.npmjs.org/preact/-/preact-10.24.0.tgz"
integrity sha512-aK8Cf+jkfyuZ0ZZRG9FbYqwmEiGQ4y/PUO4SuTWoyWL244nZZh7bd5h2APd4rSNDYTBNghg1L+5iJN3Skxtbsw==
prelude-ls@^1.2.1:
version "1.2.1"
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "mail_mjml",
"version": "0.0.10",
"version": "0.0.11",
"description": "An util to generate html and text django's templates from mjml templates",
"type": "module",
"dependencies": {
+4 -4
View File
@@ -399,10 +399,10 @@ glob-parent@~5.1.2:
dependencies:
is-glob "^4.0.1"
glob@^10.3.10, glob@^10.3.3:
version "10.4.5"
resolved "https://registry.yarnpkg.com/glob/-/glob-10.4.5.tgz#f4d9f0b90ffdbab09c9d77f5f29b4262517b0956"
integrity sha512-7Bv8RF0k6xjo7d4A/PxYLbUCfb6c+Vpd2/mB2yRDlew7Jb5hEXiCD9ibfO7wpk8i4sevK6DFny9h7EYbM3/sHg==
glob@^10.5.0:
version "10.5.0"
resolved "https://registry.yarnpkg.com/glob/-/glob-10.5.0.tgz#8ec0355919cd3338c28428a23d4f24ecc5fe738c"
integrity sha512-DfXN8DfhJ7NH3Oe7cFmu3NCu1wKbkReJ8TorzSAFbSKrlNaQSKfIzqYqVY8zlbs2NLBbWpRiU52GX2PbaBVNkg==
dependencies:
foreground-child "^3.1.0"
jackspeak "^3.1.2"