Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| fc58b43b9b | |||
| 0807f51a97 |
@@ -200,22 +200,3 @@ jobs:
|
||||
|
||||
- name: Run tests
|
||||
run: ~/.local/bin/pytest -n 2
|
||||
|
||||
security-trivy-critical:
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Run Trivy analysis for critical vulnerabilities
|
||||
# We use main branch while we might still iterate on the action
|
||||
uses: numerique-gouv/action-trivy-cache/security-trivy-critical@main
|
||||
|
||||
security-trivy:
|
||||
permissions:
|
||||
contents: read
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Run Trivy analysis for vulnerabilities
|
||||
# We use main branch while we might still iterate on the action
|
||||
uses: numerique-gouv/action-trivy-cache/security-trivy@main
|
||||
|
||||
+10
-21
@@ -8,35 +8,18 @@ and this project adheres to
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Changed
|
||||
|
||||
- 📦️(front) update react
|
||||
|
||||
### Fixed
|
||||
|
||||
- 🐛(e2e) fix test-e2e-chromium
|
||||
- 🐛(back) fix system prompt compatibility with self-hosted models #200
|
||||
- ⚰️(back) remove dead code and unused files
|
||||
|
||||
## [0.0.10] - 2025-12-15
|
||||
|
||||
### Added
|
||||
|
||||
- ✨(front) add retry button
|
||||
|
||||
### Fixed
|
||||
|
||||
- 🐛(front) fix long user messages
|
||||
- 🐛(front) fix "Maximum update depth exceeded" error in Chat component
|
||||
- 🐛(front) fix parsing documents display
|
||||
- 🐛(front) fix opacity input in error
|
||||
- 🐛(front) resolve React hydration errors
|
||||
- 🚑️(user) allow longer short names #182
|
||||
|
||||
## [0.0.9] - 2025-11-17
|
||||
|
||||
### Added
|
||||
|
||||
- ✨(front) add code copy button
|
||||
- ✨(RAG) add generic collection RAG tools #159
|
||||
|
||||
@@ -44,6 +27,7 @@ and this project adheres to
|
||||
|
||||
- 🔊(langfuse) enable tracing with redacted content #162
|
||||
|
||||
|
||||
## [0.0.8] - 2025-11-10
|
||||
|
||||
### Fixed
|
||||
@@ -58,24 +42,28 @@ and this project adheres to
|
||||
|
||||
- 🔥(posthog) remove posthog middleware for async mode fix #146
|
||||
|
||||
|
||||
## [0.0.7] - 2025-10-28
|
||||
|
||||
### Fixed
|
||||
|
||||
- 🚑️(posthog) fix the posthog middleware for async mode #133
|
||||
|
||||
|
||||
## [0.0.6] - 2025-10-28
|
||||
|
||||
### Fixed
|
||||
|
||||
- 🚑️(stats) fix tracking id in upload event #130
|
||||
|
||||
|
||||
## [0.0.5] - 2025-10-27
|
||||
|
||||
### Fixed
|
||||
|
||||
- 🚑️(drag-drop) fix the rejection display on Safari #127
|
||||
|
||||
|
||||
## [0.0.4] - 2025-10-27
|
||||
|
||||
### Added
|
||||
@@ -92,12 +80,14 @@ and this project adheres to
|
||||
- 🐛(front) fix mobile source
|
||||
- 🐛(attachments) reject the whole drag&drop if unsupported formats #123
|
||||
|
||||
|
||||
## [0.0.3] - 2025-10-21
|
||||
|
||||
### Fixed
|
||||
|
||||
- 🚑️(web-search) fix missing argument in RAG backend #116
|
||||
|
||||
|
||||
## [0.0.2] - 2025-10-21
|
||||
|
||||
### Added
|
||||
@@ -107,7 +97,6 @@ and this project adheres to
|
||||
- 📈(posthog) add `sub` field to tracking #95
|
||||
|
||||
### Changed
|
||||
|
||||
- 🔧(front) change links feedback tchap + settings popup
|
||||
- 🐛(front) code activation fix session end #93
|
||||
- 💬(wording) error page wording #102
|
||||
@@ -115,6 +104,7 @@ and this project adheres to
|
||||
- 🐛(activation-codes) create contact in brevo before add to list #108
|
||||
- ⚗️(summarization) add system prompt to handle tool #112
|
||||
|
||||
|
||||
## [0.0.1] - 2025-10-19
|
||||
|
||||
### Changed
|
||||
@@ -137,7 +127,7 @@ and this project adheres to
|
||||
- 🎨(front) change list attachment in chat
|
||||
- 🎨(front) move emplacement for attachment
|
||||
- 🎨(ui) retour ui sources files
|
||||
- ✨(ui) fix retour global ui
|
||||
- ✨(ui) fix retour global ui
|
||||
- 🐛(fix) broken staging css
|
||||
- 🎨(alpha) adjustment for alpha version
|
||||
- ✨(ui) delete flex message
|
||||
@@ -172,8 +162,7 @@ and this project adheres to
|
||||
- 💄(chat) add code highlighting for LLM responses #67
|
||||
|
||||
|
||||
[unreleased]: https://github.com/suitenumerique/conversations/compare/v0.0.10...main
|
||||
[0.0.10]: https://github.com/suitenumerique/conversations/releases/v0.0.10
|
||||
[unreleased]: https://github.com/suitenumerique/conversations/compare/v0.0.9...main
|
||||
[0.0.9]: https://github.com/suitenumerique/conversations/releases/v0.0.9
|
||||
[0.0.8]: https://github.com/suitenumerique/conversations/releases/v0.0.8
|
||||
[0.0.7]: https://github.com/suitenumerique/conversations/releases/v0.0.7
|
||||
|
||||
+11
@@ -71,6 +71,9 @@ services:
|
||||
- "host.docker.internal:host-gateway"
|
||||
ports:
|
||||
- "8071:8000"
|
||||
networks:
|
||||
- default
|
||||
- lasuite
|
||||
volumes:
|
||||
- ./src/backend:/app
|
||||
- ./data/static:/data/static
|
||||
@@ -89,6 +92,9 @@ services:
|
||||
image: nginx:1.25
|
||||
ports:
|
||||
- "8083:8083"
|
||||
networks:
|
||||
- default
|
||||
- lasuite
|
||||
volumes:
|
||||
- ./docker/files/etc/nginx/conf.d:/etc/nginx/conf.d:ro
|
||||
depends_on:
|
||||
@@ -177,3 +183,8 @@ services:
|
||||
kc_postgresql:
|
||||
condition: service_healthy
|
||||
restart: true
|
||||
|
||||
networks:
|
||||
lasuite:
|
||||
name: lasuite-network
|
||||
driver: bridge
|
||||
|
||||
@@ -244,8 +244,8 @@ For Mistral AI models using the Etalab platform:
|
||||
{
|
||||
"models": [
|
||||
{
|
||||
"hrid": "mistral-large",
|
||||
"model_name": "mistral-large-latest",
|
||||
"hrid": "mistral-medium",
|
||||
"model_name": "mistral-medium-2508",
|
||||
"human_readable_name": "Mistral Large (Etalab)",
|
||||
"provider_name": "mistral-etalab",
|
||||
"profile": null,
|
||||
|
||||
@@ -357,6 +357,7 @@ The RAG backend performs semantic search to find the most relevant content:
|
||||
rag_results = document_store.search(
|
||||
query,
|
||||
results_count=settings.BRAVE_RAG_WEB_SEARCH_CHUNK_NUMBER,
|
||||
**kwargs,
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"dependencies": {
|
||||
"@ai-sdk/react": "^1.2.12",
|
||||
"@ai-sdk/ui-utils": "^1.2.11"
|
||||
}
|
||||
}
|
||||
@@ -18,169 +18,6 @@ from chat.agent_rag.document_rag_backends.base_rag_backend import BaseRagBackend
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Albert API token limit for document vectorization
|
||||
# We use a conservative chunk size to stay well under the limit
|
||||
ALBERT_MAX_TOKENS = 8192
|
||||
ALBERT_CHUNK_SIZE_TOKENS = 5000 # More conservative chunk size with larger safety margin
|
||||
# Approximate tokens: ~3 characters per token (more conservative estimate for Markdown/Excel)
|
||||
# Markdown and Excel content often have more tokens per character due to formatting
|
||||
ALBERT_CHUNK_SIZE_CHARS = ALBERT_CHUNK_SIZE_TOKENS * 3
|
||||
|
||||
|
||||
def _estimate_tokens(content: str) -> int:
|
||||
"""
|
||||
Estimate the number of tokens in a text string.
|
||||
|
||||
Uses a conservative approximation: ~3 characters per token.
|
||||
This is more conservative than 4 chars/token to account for:
|
||||
- Markdown formatting (headers, lists, tables)
|
||||
- Excel content with special characters
|
||||
- Whitespace and punctuation
|
||||
|
||||
Args:
|
||||
content (str): The text content to estimate.
|
||||
|
||||
Returns:
|
||||
int: Estimated number of tokens.
|
||||
"""
|
||||
return len(content) // 3
|
||||
|
||||
|
||||
def _chunk_content(content: str, max_chars: int = ALBERT_CHUNK_SIZE_CHARS) -> List[str]:
|
||||
"""
|
||||
Split content into chunks that fit within Albert's token limit.
|
||||
|
||||
Attempts to split at paragraph boundaries (double newlines) when possible,
|
||||
otherwise splits at line boundaries, and finally at character boundaries.
|
||||
Validates that each chunk is under the token limit after splitting.
|
||||
|
||||
Args:
|
||||
content (str): The content to chunk.
|
||||
max_chars (int): Maximum characters per chunk (default: ALBERT_CHUNK_SIZE_CHARS).
|
||||
|
||||
Returns:
|
||||
list[str]: List of content chunks, each under the token limit.
|
||||
"""
|
||||
# First check if content fits in one chunk
|
||||
estimated_tokens = _estimate_tokens(content)
|
||||
if estimated_tokens <= ALBERT_CHUNK_SIZE_TOKENS:
|
||||
return [content]
|
||||
|
||||
chunks = []
|
||||
remaining = content
|
||||
|
||||
while len(remaining) > 0:
|
||||
# Check if remaining content fits in one chunk
|
||||
remaining_tokens = _estimate_tokens(remaining)
|
||||
if remaining_tokens <= ALBERT_CHUNK_SIZE_TOKENS:
|
||||
if remaining.strip():
|
||||
chunks.append(remaining.strip())
|
||||
break
|
||||
|
||||
# Need to split - find the best split point
|
||||
# Start with max_chars but may need to reduce if token estimate is too high
|
||||
search_limit = max_chars
|
||||
|
||||
# Try to find a split point that keeps us under token limit
|
||||
# Reduce search limit if needed to ensure token limit is respected
|
||||
while search_limit > 100: # Minimum chunk size
|
||||
# Try to split at paragraph boundary (double newline)
|
||||
split_pos = remaining.rfind("\n\n", 0, search_limit)
|
||||
if split_pos == -1:
|
||||
# Try to split at single newline
|
||||
split_pos = remaining.rfind("\n", 0, search_limit)
|
||||
if split_pos == -1:
|
||||
# Force split at character boundary
|
||||
split_pos = search_limit
|
||||
|
||||
# Validate that this chunk is under token limit
|
||||
chunk_candidate = remaining[:split_pos].strip()
|
||||
if chunk_candidate:
|
||||
chunk_tokens = _estimate_tokens(chunk_candidate)
|
||||
if chunk_tokens <= ALBERT_CHUNK_SIZE_TOKENS:
|
||||
chunks.append(chunk_candidate)
|
||||
remaining = remaining[split_pos:].lstrip()
|
||||
break
|
||||
|
||||
# Chunk too large, reduce search limit and try again
|
||||
search_limit = int(search_limit * 0.8) # Reduce by 20%
|
||||
else:
|
||||
# Fallback: force split at a safe size
|
||||
# This should rarely happen, but ensures we don't get stuck
|
||||
safe_size = min(max_chars, len(remaining))
|
||||
chunk = remaining[:safe_size].strip()
|
||||
if chunk:
|
||||
chunks.append(chunk)
|
||||
remaining = remaining[safe_size:].lstrip()
|
||||
|
||||
# Validate all chunks are under limit and split further if needed
|
||||
validated_chunks = []
|
||||
for chunk_item in chunks:
|
||||
chunk_tokens = _estimate_tokens(chunk_item)
|
||||
if chunk_tokens > ALBERT_MAX_TOKENS:
|
||||
logger.warning(
|
||||
"Chunk still exceeds token limit (%d tokens, max: %d), forcing split further",
|
||||
chunk_tokens,
|
||||
ALBERT_MAX_TOKENS,
|
||||
)
|
||||
# Force split this chunk further using a more conservative size
|
||||
# Use a size that ensures we stay well under the token limit
|
||||
# Target: ~5000 tokens max per chunk (conservative)
|
||||
max_safe_chars = ALBERT_CHUNK_SIZE_TOKENS * 3 # 6000 * 3 = 18000 chars for ~5000 tokens
|
||||
remaining_chunk = chunk_item
|
||||
while len(remaining_chunk) > 0:
|
||||
remaining_tokens = _estimate_tokens(remaining_chunk)
|
||||
if remaining_tokens <= ALBERT_CHUNK_SIZE_TOKENS:
|
||||
if remaining_chunk.strip():
|
||||
validated_chunks.append(remaining_chunk.strip())
|
||||
break
|
||||
|
||||
# Find a safe split point
|
||||
split_pos = min(max_safe_chars, len(remaining_chunk))
|
||||
# Try to split at a line boundary if possible
|
||||
line_split = remaining_chunk.rfind("\n", 0, split_pos)
|
||||
if line_split > max_safe_chars * 0.5: # Only use if it's not too small
|
||||
split_pos = line_split
|
||||
|
||||
sub_chunk = remaining_chunk[:split_pos].strip()
|
||||
if sub_chunk:
|
||||
sub_tokens = _estimate_tokens(sub_chunk)
|
||||
# Double-check this sub-chunk is safe
|
||||
if sub_tokens > ALBERT_MAX_TOKENS:
|
||||
# Still too large, use even smaller size
|
||||
logger.warning(
|
||||
"Sub-chunk still too large (%d tokens), using smaller split",
|
||||
sub_tokens,
|
||||
)
|
||||
split_pos = ALBERT_CHUNK_SIZE_TOKENS * 2 # 12000 chars for ~3000 tokens
|
||||
sub_chunk = remaining_chunk[:split_pos].strip()
|
||||
validated_chunks.append(sub_chunk)
|
||||
remaining_chunk = remaining_chunk[split_pos:].lstrip()
|
||||
else:
|
||||
validated_chunks.append(chunk_item)
|
||||
|
||||
# Final validation - ensure NO chunk exceeds the limit
|
||||
final_chunks = []
|
||||
for chunk in validated_chunks:
|
||||
chunk_tokens = _estimate_tokens(chunk)
|
||||
if chunk_tokens > ALBERT_MAX_TOKENS:
|
||||
logger.error(
|
||||
"CRITICAL: Chunk still exceeds limit after all splitting attempts: %d tokens",
|
||||
chunk_tokens,
|
||||
)
|
||||
# Emergency split: use very conservative size
|
||||
emergency_size = ALBERT_CHUNK_SIZE_TOKENS * 2 # 12000 chars
|
||||
remaining = chunk
|
||||
while len(remaining) > 0:
|
||||
emergency_chunk = remaining[:emergency_size].strip()
|
||||
if emergency_chunk:
|
||||
final_chunks.append(emergency_chunk)
|
||||
remaining = remaining[emergency_size:].lstrip()
|
||||
else:
|
||||
final_chunks.append(chunk)
|
||||
|
||||
return final_chunks
|
||||
|
||||
|
||||
class AlbertRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-attributes
|
||||
"""
|
||||
@@ -333,42 +170,7 @@ class AlbertRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-att
|
||||
"""
|
||||
Store the document content in the Albert collection.
|
||||
This method should handle the logic to send the document content to the Albert API.
|
||||
|
||||
If the document is too large (exceeds Albert's token limit), it will be automatically
|
||||
split into multiple chunks and stored as separate documents.
|
||||
|
||||
Args:
|
||||
name (str): The name of the document.
|
||||
content (str): The content of the document in Markdown format.
|
||||
"""
|
||||
# Check if content needs to be chunked
|
||||
estimated_tokens = _estimate_tokens(content)
|
||||
|
||||
if estimated_tokens > ALBERT_MAX_TOKENS:
|
||||
logger.info(
|
||||
"Document '%s' is too large (%d estimated tokens, limit: %d). "
|
||||
"Splitting into chunks.",
|
||||
name,
|
||||
estimated_tokens,
|
||||
ALBERT_MAX_TOKENS,
|
||||
)
|
||||
chunks = _chunk_content(content)
|
||||
logger.info("Split document '%s' into %d chunks", name, len(chunks))
|
||||
|
||||
# Store each chunk as a separate document
|
||||
for i, chunk in enumerate(chunks, start=1):
|
||||
chunk_name = f"{name}_part_{i}" if len(chunks) > 1 else name
|
||||
self._store_single_document(chunk_name, chunk)
|
||||
else:
|
||||
# Document fits within limit, store as-is
|
||||
self._store_single_document(name, content)
|
||||
|
||||
def _store_single_document(self, name: str, content: str) -> None:
|
||||
"""
|
||||
Store a single document chunk in the Albert collection.
|
||||
|
||||
Internal method that performs the actual API call to store one document.
|
||||
|
||||
Args:
|
||||
name (str): The name of the document.
|
||||
content (str): The content of the document in Markdown format.
|
||||
@@ -383,68 +185,14 @@ class AlbertRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-att
|
||||
},
|
||||
timeout=settings.ALBERT_API_TIMEOUT,
|
||||
)
|
||||
logger.debug("Stored document '%s': %s", name, response.json())
|
||||
logger.debug(response.json())
|
||||
response.raise_for_status()
|
||||
|
||||
async def astore_document(self, name: str, content: str) -> None:
|
||||
"""
|
||||
Store the document content in the Albert collection.
|
||||
This method should handle the logic to send the document content to the Albert API.
|
||||
|
||||
If the document is too large (exceeds Albert's token limit), it will be automatically
|
||||
split into multiple chunks and stored as separate documents.
|
||||
|
||||
Args:
|
||||
name (str): The name of the document.
|
||||
content (str): The content of the document in Markdown format.
|
||||
"""
|
||||
# Check if content needs to be chunked
|
||||
estimated_tokens = _estimate_tokens(content)
|
||||
|
||||
if estimated_tokens > ALBERT_MAX_TOKENS:
|
||||
logger.info(
|
||||
"Document '%s' is too large (%d estimated tokens, limit: %d). "
|
||||
"Splitting into chunks.",
|
||||
name,
|
||||
estimated_tokens,
|
||||
ALBERT_MAX_TOKENS,
|
||||
)
|
||||
chunks = _chunk_content(content)
|
||||
logger.info("Split document '%s' into %d chunks", name, len(chunks))
|
||||
|
||||
# Validate chunks before storing
|
||||
for i, chunk in enumerate(chunks, start=1):
|
||||
chunk_tokens = _estimate_tokens(chunk)
|
||||
logger.debug(
|
||||
"Chunk %d/%d: %d chars, ~%d tokens",
|
||||
i,
|
||||
len(chunks),
|
||||
len(chunk),
|
||||
chunk_tokens,
|
||||
)
|
||||
if chunk_tokens > ALBERT_MAX_TOKENS:
|
||||
logger.error(
|
||||
"Chunk %d/%d still exceeds token limit: %d tokens (max: %d)",
|
||||
i,
|
||||
len(chunks),
|
||||
chunk_tokens,
|
||||
ALBERT_MAX_TOKENS,
|
||||
)
|
||||
|
||||
# Store each chunk as a separate document
|
||||
for i, chunk in enumerate(chunks, start=1):
|
||||
chunk_name = f"{name}_part_{i}" if len(chunks) > 1 else name
|
||||
await self._astore_single_document(chunk_name, chunk)
|
||||
else:
|
||||
# Document fits within limit, store as-is
|
||||
await self._astore_single_document(name, content)
|
||||
|
||||
async def _astore_single_document(self, name: str, content: str) -> None:
|
||||
"""
|
||||
Store a single document chunk in the Albert collection.
|
||||
|
||||
Internal method that performs the actual API call to store one document.
|
||||
|
||||
Args:
|
||||
name (str): The name of the document.
|
||||
content (str): The content of the document in Markdown format.
|
||||
@@ -462,16 +210,17 @@ class AlbertRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-att
|
||||
},
|
||||
timeout=settings.ALBERT_API_TIMEOUT,
|
||||
)
|
||||
logger.debug("Stored document '%s': %s", name, response.json())
|
||||
logger.debug(response.json())
|
||||
response.raise_for_status()
|
||||
|
||||
def search(self, query, results_count: int = 4) -> RAGWebResults:
|
||||
def search(self, query: str, results_count: int = 4, **kwargs) -> RAGWebResults:
|
||||
"""
|
||||
Perform a search using the Albert API based on the provided query.
|
||||
|
||||
Args:
|
||||
query (str): The search query.
|
||||
results_count (int): The number of results to return.
|
||||
**kwargs: Additional arguments.
|
||||
|
||||
Returns:
|
||||
RAGWebResults: The search results.
|
||||
@@ -508,7 +257,7 @@ class AlbertRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-att
|
||||
),
|
||||
)
|
||||
|
||||
async def asearch(self, query, results_count: int = 4) -> RAGWebResults:
|
||||
async def asearch(self, query, results_count: int = 4, **kwargs) -> RAGWebResults:
|
||||
"""
|
||||
Perform an asynchronous search using the Albert API based on the provided query.
|
||||
|
||||
|
||||
@@ -142,17 +142,27 @@ class BaseRagBackend:
|
||||
"""
|
||||
return await sync_to_async(self.delete_collection)()
|
||||
|
||||
def search(self, query, results_count: int = 4) -> RAGWebResults:
|
||||
def search(self, query: str, results_count: int = 4, **kwargs) -> RAGWebResults:
|
||||
"""
|
||||
Search the collection for the given query.
|
||||
|
||||
Args:
|
||||
query: The search query string.
|
||||
results_count: Number of results to return.
|
||||
**kwargs: Additional arguments. Expected: 'session' for OIDC authentication.
|
||||
"""
|
||||
raise NotImplementedError("Must be implemented in subclass.")
|
||||
|
||||
async def asearch(self, query, results_count: int = 4) -> RAGWebResults:
|
||||
async def asearch(self, query: str, results_count: int = 4, **kwargs) -> RAGWebResults:
|
||||
"""
|
||||
Search the collection for the given query.
|
||||
Search the collection for the given query asynchronously.
|
||||
|
||||
Args:
|
||||
query: The search query string.
|
||||
results_count: Number of results to return.
|
||||
**kwargs: Additional arguments. Expected: 'session' for OIDC authentication.
|
||||
"""
|
||||
return await sync_to_async(self.search)(query=query, results_count=results_count)
|
||||
return await sync_to_async(self.search)(query=query, results_count=results_count, **kwargs)
|
||||
|
||||
@classmethod
|
||||
@contextmanager
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
"""Implementation of the Find API for RAG document search."""
|
||||
|
||||
import logging
|
||||
from io import BytesIO
|
||||
from multiprocessing.context import AuthenticationError
|
||||
from typing import List, Optional
|
||||
from urllib.parse import urljoin
|
||||
from uuid import uuid4
|
||||
|
||||
from django.conf import settings
|
||||
from django.core.exceptions import ImproperlyConfigured
|
||||
from django.utils import timezone
|
||||
|
||||
import requests
|
||||
|
||||
from chat.agent_rag.constants import RAGWebResult, RAGWebResults, RAGWebUsage
|
||||
from chat.agent_rag.document_converter.markitdown import DocumentConverter
|
||||
from chat.agent_rag.document_rag_backends.base_rag_backend import BaseRagBackend
|
||||
from utils.oicd import refresh_access_token
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class FindRagBackend(BaseRagBackend): # pylint: disable=too-many-instance-attributes
|
||||
"""
|
||||
This class is a placeholder for the Find API implementation.
|
||||
It is designed to be used with the RAG (Retrieval-Augmented Generation) document search system.
|
||||
|
||||
It provides methods to:
|
||||
- Parse documents and convert them to Markdown format:
|
||||
+ Handle PDF parsing using the Albert API.
|
||||
+ Use the DocumentConverter (markitdown) for other formats.
|
||||
- Store parsed documents in the Find index.
|
||||
- Perform a search operation using the Find API.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
collection_id: Optional[str] = None,
|
||||
read_only_collection_id: Optional[List[str]] = None,
|
||||
):
|
||||
# Initialize any necessary parameters or configurations here
|
||||
super().__init__(collection_id, read_only_collection_id)
|
||||
self._pdf_parser_endpoint = urljoin(settings.ALBERT_API_URL, "/v1/parse-beta")
|
||||
self.api_key = settings.FIND_API_KEY
|
||||
self.search_endpoint = "api/v1.0/documents/search/"
|
||||
self.indexing_endpoint = "api/v1.0/documents/index/"
|
||||
|
||||
if not self.api_key:
|
||||
raise ImproperlyConfigured("FIND_API_KEY must be set in Django settings.")
|
||||
|
||||
def create_collection(self, name: str, description: Optional[str] = None) -> str:
|
||||
"""
|
||||
init collection_id
|
||||
"""
|
||||
self.collection_id = self.collection_id or 1
|
||||
return self.collection_id
|
||||
|
||||
# TODO
|
||||
def delete_collection(self) -> None:
|
||||
"""
|
||||
Deletion not available
|
||||
"""
|
||||
logger.warning("deletion of collections is not yet supported in FindRagBackend")
|
||||
|
||||
# TODO: factor with albert api
|
||||
def parse_pdf_document(self, name: str, content_type: str, content: BytesIO) -> str:
|
||||
"""
|
||||
Parse the PDF document content and return the text content.
|
||||
This method should handle the logic to convert the PDF into
|
||||
a format suitable for the Albert API.
|
||||
"""
|
||||
response = requests.post(
|
||||
self._pdf_parser_endpoint,
|
||||
headers={
|
||||
"Authorization": f"Bearer {settings.ALBERT_API_KEY}",
|
||||
},
|
||||
files={
|
||||
"file": (
|
||||
name,
|
||||
content,
|
||||
content_type,
|
||||
), # Use the name as the filename in the request
|
||||
"output_format": (None, "markdown"), # Specify the output format as Markdown,
|
||||
},
|
||||
timeout=settings.ALBERT_API_PARSE_TIMEOUT,
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
return "\n\n".join(
|
||||
document_page["content"] for document_page in response.json().get("data", [])
|
||||
)
|
||||
|
||||
# TODO: factor with albert api
|
||||
def parse_document(self, name: str, content_type: str, content: BytesIO):
|
||||
"""
|
||||
Parse the document and prepare it for the search operation.
|
||||
This method should handle the logic to convert the document
|
||||
into a format suitable for the Find API.
|
||||
|
||||
Args:
|
||||
name (str): The name of the document.
|
||||
content_type (str): The MIME type of the document (e.g., "application/pdf").
|
||||
content (BytesIO): The content of the document as a BytesIO stream.
|
||||
|
||||
Returns:
|
||||
str: The document content in Markdown format.
|
||||
"""
|
||||
# Implement the parsing logic here
|
||||
if content_type == "application/pdf":
|
||||
# Handle PDF parsing
|
||||
markdown_content = self.parse_pdf_document(
|
||||
name=name, content_type=content_type, content=content
|
||||
)
|
||||
else:
|
||||
markdown_content = DocumentConverter().convert_raw(
|
||||
name=name, content_type=content_type, content=content
|
||||
)
|
||||
|
||||
return markdown_content
|
||||
|
||||
def store_document(self, name: str, content: str) -> None:
|
||||
"""
|
||||
index document in Find
|
||||
|
||||
Args:
|
||||
name (str): The name of the document.
|
||||
content (str): The content of the document in Markdown format.
|
||||
"""
|
||||
logger.debug("index document '%s' in Find", name)
|
||||
response = requests.post(
|
||||
urljoin(settings.FIND_API_URL, self.indexing_endpoint),
|
||||
headers={"Authorization": f"Bearer {self.api_key}"},
|
||||
json={
|
||||
"id": str(uuid4()),
|
||||
"title": str(name) or "",
|
||||
"depth": 0,
|
||||
"path": str(name) or "",
|
||||
"numchild": 0,
|
||||
"content": content or "",
|
||||
"created_at": timezone.now().isoformat(),
|
||||
"updated_at": timezone.now().isoformat(),
|
||||
"tags": [f"collection-{self.collection_id}"],
|
||||
"size": len(content.encode("utf-8")),
|
||||
"users": [], # TODO
|
||||
"groups": [],
|
||||
"reach": "public",
|
||||
"is_active": True,
|
||||
},
|
||||
timeout=settings.ALBERT_API_TIMEOUT,
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
def search(self, query: str, results_count: int = 4, **kwargs) -> RAGWebResults:
|
||||
"""
|
||||
Perform a search using the Find API.
|
||||
Uses the user's OIDC token from the request session.
|
||||
|
||||
Args:
|
||||
query: The search query.
|
||||
results_count: Number of results to return.
|
||||
**kwargs: Additional arguments. Expected: 'session' containing OIDC tokens.
|
||||
|
||||
Returns:
|
||||
RAGWebResults: The search results.
|
||||
"""
|
||||
logger.debug("search documents in Find with query '%s'", query)
|
||||
# TODO: factor session auth in a decorator
|
||||
session = refresh_access_token(kwargs.get("session"))
|
||||
oidc_access_token = session.get("oidc_access_token")
|
||||
if not oidc_access_token:
|
||||
raise AuthenticationError({"error": "Not authenticated"})
|
||||
collection_ids = self.get_all_collection_ids()
|
||||
|
||||
response = requests.post(
|
||||
urljoin(settings.FIND_API_URL, self.search_endpoint),
|
||||
headers={"Authorization": f"Bearer {oidc_access_token}"},
|
||||
json={
|
||||
"q": query,
|
||||
"tags": [f"collection:{collection_id}" for collection_id in collection_ids],
|
||||
"k": 10,
|
||||
},
|
||||
timeout=settings.ALBERT_API_TIMEOUT,
|
||||
)
|
||||
logger.debug(response.json())
|
||||
response.raise_for_status()
|
||||
|
||||
return RAGWebResults(
|
||||
data=[
|
||||
RAGWebResult(
|
||||
url=result["_source"]["title.fr"],
|
||||
content=result["_source"]["content.fr"],
|
||||
score=result["_score"],
|
||||
)
|
||||
for result in response.json()
|
||||
],
|
||||
usage=RAGWebUsage(
|
||||
prompt_tokens=0,
|
||||
completion_tokens=0,
|
||||
),
|
||||
)
|
||||
@@ -190,4 +190,6 @@ class BaseAgent(Agent):
|
||||
|
||||
_tools = [get_pydantic_tools_by_name(tool_name) for tool_name in self.configuration.tools]
|
||||
|
||||
super().__init__(model=_model_instance, instructions=_system_prompt, tools=_tools, **kwargs)
|
||||
super().__init__(
|
||||
model=_model_instance, system_prompt=_system_prompt, tools=_tools, **kwargs
|
||||
)
|
||||
|
||||
@@ -16,6 +16,7 @@ from .base import BaseAgent
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
MOCKED_RESPONSE = """
|
||||
# **Ode to the AI Assistant** 🤖✨
|
||||
|
||||
@@ -101,10 +102,10 @@ class ConversationAgent(BaseAgent):
|
||||
if settings.WARNING_MOCK_CONVERSATION_AGENT:
|
||||
self._model = FunctionModel(stream_function=mocked_agent_model)
|
||||
|
||||
@self.instructions
|
||||
@self.system_prompt
|
||||
def add_the_date() -> str:
|
||||
"""
|
||||
Dynamic instruction function to add the current date.
|
||||
Dynamic system prompt function to add the current date.
|
||||
|
||||
Warning: this will always use the date in the server timezone,
|
||||
not the user's timezone...
|
||||
@@ -112,9 +113,9 @@ class ConversationAgent(BaseAgent):
|
||||
_formatted_date = formats.date_format(timezone.now(), "l d/m/Y", use_l10n=False)
|
||||
return f"Today is {_formatted_date}."
|
||||
|
||||
@self.instructions
|
||||
@self.system_prompt
|
||||
def enforce_response_language() -> str:
|
||||
"""Dynamic instruction function to set the expected language to use."""
|
||||
"""Dynamic system prompt function to set the expected language to use."""
|
||||
return f"Answer in {get_language_name(language).lower()}." if language else ""
|
||||
|
||||
def get_web_search_tool_name(self) -> str | None:
|
||||
|
||||
@@ -72,16 +72,12 @@ from chat.clients.pydantic_ui_message_converter import (
|
||||
ui_message_to_user_content,
|
||||
)
|
||||
from chat.mcp_servers import get_mcp_servers
|
||||
from chat.tools.data_analysis import add_data_analysis_tool
|
||||
from chat.tools.document_generic_search_rag import add_document_rag_search_tool_from_setting
|
||||
from chat.tools.document_search_rag import add_document_rag_search_tool
|
||||
from chat.tools.document_summarize import document_summarize
|
||||
from chat.vercel_ai_sdk.core import events_v4, events_v5
|
||||
from chat.vercel_ai_sdk.encoder import EventEncoder
|
||||
|
||||
# Keep at the top of the file to avoid mocking issues
|
||||
document_store_backend = import_string(settings.RAG_DOCUMENT_SEARCH_BACKEND)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
User = get_user_model()
|
||||
@@ -93,6 +89,7 @@ class ContextDeps:
|
||||
|
||||
conversation: models.ChatConversation
|
||||
user: User
|
||||
session: Optional[Dict] = None
|
||||
web_search_enabled: bool = False
|
||||
|
||||
|
||||
@@ -107,7 +104,14 @@ def get_model_configuration(model_hrid: str):
|
||||
class AIAgentService: # pylint: disable=too-many-instance-attributes
|
||||
"""Service class for AI-related operations (Pydantic-AI edition)."""
|
||||
|
||||
def __init__(self, conversation: models.ChatConversation, user, model_hrid=None, language=None):
|
||||
def __init__( # noqa: PLR0913 # pylint: disable=too-many-arguments,too-many-positional-arguments
|
||||
self,
|
||||
conversation: models.ChatConversation,
|
||||
user,
|
||||
session=None,
|
||||
model_hrid=None,
|
||||
language=None,
|
||||
):
|
||||
"""
|
||||
Initialize the AI agent service.
|
||||
|
||||
@@ -137,6 +141,7 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
|
||||
self._context_deps = ContextDeps(
|
||||
conversation=conversation,
|
||||
user=user,
|
||||
session=session,
|
||||
web_search_enabled=self._is_web_search_enabled,
|
||||
)
|
||||
|
||||
@@ -152,7 +157,6 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
|
||||
deps_type=ContextDeps,
|
||||
)
|
||||
add_document_rag_search_tool_from_setting(self.conversation_agent, self.user)
|
||||
add_data_analysis_tool(self.conversation_agent)
|
||||
|
||||
@property
|
||||
def _stop_cache_key(self):
|
||||
@@ -241,7 +245,6 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
|
||||
Parse and store input documents in the conversation's document store.
|
||||
"""
|
||||
# Early external document URL rejection
|
||||
|
||||
if any(
|
||||
not document.url.startswith("/media-key/")
|
||||
for document in documents
|
||||
@@ -255,6 +258,8 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
|
||||
):
|
||||
raise ValueError("Document URL does not belong to the conversation.")
|
||||
|
||||
document_store_backend = import_string(settings.RAG_DOCUMENT_SEARCH_BACKEND)
|
||||
|
||||
document_store = document_store_backend(self.conversation.collection_id)
|
||||
if not document_store.collection_id:
|
||||
# Create a new collection for the conversation
|
||||
@@ -291,24 +296,7 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
|
||||
content=document.data,
|
||||
)
|
||||
|
||||
# Don't convert tabular files (CSV, Excel) to Markdown - keep originals for data_analysis tool
|
||||
# Tabular files are already text-based or can be used directly
|
||||
is_tabular_file = (
|
||||
document.media_type in [
|
||||
"text/csv",
|
||||
"application/csv",
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
"application/vnd.ms-excel",
|
||||
"application/excel",
|
||||
]
|
||||
or any(
|
||||
document.identifier.lower().endswith(ext)
|
||||
for ext in [".csv", ".xlsx", ".xls", ".xlsm", ".xlsb"]
|
||||
)
|
||||
)
|
||||
|
||||
# Only convert non-text files that are not tabular files
|
||||
if not document.media_type.startswith("text/") and not is_tabular_file:
|
||||
if not document.media_type.startswith("text/"):
|
||||
md_attachment = await models.ChatConversationAttachment.objects.acreate(
|
||||
conversation=self.conversation,
|
||||
uploaded_by=self.user,
|
||||
@@ -441,7 +429,6 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
try:
|
||||
await self.parse_input_documents(input_documents)
|
||||
except Exception as exc: # pylint: disable=broad-except
|
||||
@@ -479,7 +466,7 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
|
||||
|
||||
if force_web_search:
|
||||
|
||||
@self.conversation_agent.instructions
|
||||
@self.conversation_agent.system_prompt
|
||||
def force_web_search_prompt() -> str:
|
||||
"""Dynamic system prompt function to force web search."""
|
||||
return (
|
||||
@@ -506,7 +493,6 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
|
||||
.aexists()
|
||||
)
|
||||
|
||||
|
||||
document_urls = []
|
||||
if not conversation_has_documents and not has_not_pdf_docs:
|
||||
# No documents to process
|
||||
@@ -528,7 +514,7 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
|
||||
)
|
||||
|
||||
# Inform the model (system-level) that documents are attached and available
|
||||
@self.conversation_agent.instructions
|
||||
@self.conversation_agent.system_prompt
|
||||
def attached_documents_note() -> str:
|
||||
return (
|
||||
"[Internal context] User documents are attached to this conversation. "
|
||||
@@ -541,13 +527,6 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
|
||||
async def summarize(ctx: RunContext, *args, **kwargs) -> ToolReturn:
|
||||
"""Wrap the document_summarize tool to provide context and add the tool."""
|
||||
return await document_summarize(ctx, *args, **kwargs)
|
||||
|
||||
if not conversation_has_documents and not has_not_pdf_docs:
|
||||
# No documents to process
|
||||
pass
|
||||
elif has_not_pdf_docs:
|
||||
# Already handled above with RAG tool
|
||||
pass
|
||||
else:
|
||||
conversation_documents = [
|
||||
cd
|
||||
@@ -761,6 +740,7 @@ class AIAgentService: # pylint: disable=too-many-instance-attributes
|
||||
langfuse.update_current_trace(
|
||||
output=run.result.output if self._store_analytics else "REDACTED"
|
||||
)
|
||||
|
||||
# Vercel finish message
|
||||
yield events_v4.FinishMessagePart(
|
||||
finish_reason=events_v4.FinishReason.STOP,
|
||||
|
||||
@@ -27,14 +27,9 @@ def test_build_pydantic_agent_success_no_tools():
|
||||
"""Test successful agent creation without tools."""
|
||||
agent = ConversationAgent(model_hrid="default-model")
|
||||
assert isinstance(agent, Agent)
|
||||
assert agent._system_prompts == ()
|
||||
|
||||
instructions = agent._instructions
|
||||
assert len(instructions) == 3
|
||||
assert instructions[0] == "You are a helpful assistant"
|
||||
assert instructions[1].__name__ == "add_the_date"
|
||||
assert instructions[2].__name__ == "enforce_response_language"
|
||||
|
||||
assert agent._system_prompts == ("You are a helpful assistant",)
|
||||
assert agent._instructions == []
|
||||
assert isinstance(agent.model, OpenAIChatModel)
|
||||
assert agent.model.model_name == "model-123"
|
||||
assert str(agent.model.client.base_url) == "https://api.llm.com/v1/"
|
||||
@@ -42,7 +37,6 @@ def test_build_pydantic_agent_success_no_tools():
|
||||
assert agent._function_toolset.tools == {}
|
||||
|
||||
|
||||
@freeze_time("2025-07-25T10:36:35.297675Z")
|
||||
def test_build_pydantic_agent_with_tools(settings):
|
||||
"""Test successful agent creation with tools."""
|
||||
settings.AI_AGENT_TOOLS = ["get_current_weather"]
|
||||
@@ -50,14 +44,8 @@ def test_build_pydantic_agent_with_tools(settings):
|
||||
agent = ConversationAgent(model_hrid="default-model")
|
||||
assert isinstance(agent, Agent)
|
||||
|
||||
instructions = agent._instructions
|
||||
assert len(instructions) == 3
|
||||
assert instructions[0] == "You are a helpful assistant"
|
||||
assert instructions[1].__name__ == "add_the_date"
|
||||
assert instructions[1]() == "Today is Friday 25/07/2025."
|
||||
assert instructions[2].__name__ == "enforce_response_language"
|
||||
assert instructions[2]() == ""
|
||||
|
||||
assert agent._system_prompts == ("You are a helpful assistant",)
|
||||
assert agent._instructions == []
|
||||
assert isinstance(agent.model, OpenAIChatModel)
|
||||
assert agent.model.model_name == "model-123"
|
||||
assert str(agent.model.client.base_url) == "https://api.llm.com/v1/"
|
||||
@@ -68,23 +56,21 @@ def test_build_pydantic_agent_with_tools(settings):
|
||||
@freeze_time("2025-07-25T10:36:35.297675Z")
|
||||
def test_add_dynamic_system_prompt():
|
||||
"""
|
||||
Ensure add_the_date and enforce_response_language instructions are registered
|
||||
Ensure add_the_date and enforce_response_language system prompt are registered
|
||||
and returns proper values.
|
||||
"""
|
||||
agent = ConversationAgent(model_hrid="default-model")
|
||||
|
||||
assert len(agent._system_prompt_functions) == 0
|
||||
assert len(agent._system_prompt_functions) == 2
|
||||
|
||||
instructions = agent._instructions
|
||||
assert len(instructions) == 3
|
||||
assert instructions[0] == "You are a helpful assistant"
|
||||
assert instructions[1].__name__ == "add_the_date"
|
||||
assert instructions[1]() == "Today is Friday 25/07/2025."
|
||||
assert instructions[2].__name__ == "enforce_response_language"
|
||||
assert instructions[2]() == ""
|
||||
assert agent._system_prompt_functions[0].function.__name__ == "add_the_date"
|
||||
assert agent._system_prompt_functions[0].function() == "Today is Friday 25/07/2025."
|
||||
|
||||
assert agent._system_prompt_functions[1].function.__name__ == "enforce_response_language"
|
||||
assert agent._system_prompt_functions[1].function() == ""
|
||||
|
||||
agent = ConversationAgent(model_hrid="default-model", language="fr-fr")
|
||||
assert agent._instructions[2]() == "Answer in french."
|
||||
assert agent._system_prompt_functions[1].function() == "Answer in french."
|
||||
|
||||
|
||||
def test_agent_get_web_search_tool_name(settings):
|
||||
|
||||
@@ -38,9 +38,6 @@ def brave_settings(settings):
|
||||
settings.BRAVE_SEARCH_EXTRA_SNIPPETS = True
|
||||
settings.BRAVE_SUMMARIZATION_ENABLED = False
|
||||
settings.BRAVE_CACHE_TTL = 3600
|
||||
settings.RAG_DOCUMENT_SEARCH_BACKEND = (
|
||||
"chat.agent_rag.document_rag_backends.albert_rag_backend.AlbertRagBackend"
|
||||
)
|
||||
settings.BRAVE_RAG_WEB_SEARCH_CHUNK_NUMBER = 5
|
||||
|
||||
|
||||
|
||||
@@ -130,16 +130,6 @@ def test_post_conversation_data_protocol(api_client, mock_openai_stream):
|
||||
|
||||
assert mock_openai_stream.called
|
||||
|
||||
# ensure instructions are merged as a system prompt
|
||||
last_request_payload = json.loads(respx.calls.last.request.content)
|
||||
assert last_request_payload["messages"][0] == {
|
||||
"content": (
|
||||
"You are a helpful test assistant :)\n\nToday is Friday 25/07/2025.\n\n"
|
||||
"Answer in english."
|
||||
),
|
||||
"role": "system",
|
||||
}
|
||||
|
||||
chat_conversation.refresh_from_db()
|
||||
assert chat_conversation.ui_messages == [
|
||||
{
|
||||
@@ -180,15 +170,29 @@ def test_post_conversation_data_protocol(api_client, mock_openai_stream):
|
||||
)
|
||||
|
||||
_run_id = chat_conversation.pydantic_messages[0]["run_id"]
|
||||
|
||||
assert chat_conversation.pydantic_messages == [
|
||||
{
|
||||
"instructions": (
|
||||
"You are a helpful test assistant :)\n\n"
|
||||
"Today is Friday 25/07/2025.\n\nAnswer in english."
|
||||
),
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": "Today is Friday 25/07/2025.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": "Answer in english.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": ["Hello"],
|
||||
"part_kind": "user-prompt",
|
||||
@@ -251,15 +255,6 @@ def test_post_conversation_text_protocol(api_client, mock_openai_stream):
|
||||
assert response_content == "Hello there"
|
||||
|
||||
assert mock_openai_stream.called
|
||||
# ensure instructions are merged as a system prompt
|
||||
last_request_payload = json.loads(respx.calls.last.request.content)
|
||||
assert last_request_payload["messages"][0] == {
|
||||
"content": (
|
||||
"You are a helpful test assistant :)\n\nToday is Friday 25/07/2025.\n\n"
|
||||
"Answer in english."
|
||||
),
|
||||
"role": "system",
|
||||
}
|
||||
|
||||
chat_conversation.refresh_from_db()
|
||||
assert chat_conversation.ui_messages == [
|
||||
@@ -301,15 +296,29 @@ def test_post_conversation_text_protocol(api_client, mock_openai_stream):
|
||||
)
|
||||
|
||||
_run_id = chat_conversation.pydantic_messages[0]["run_id"]
|
||||
|
||||
assert chat_conversation.pydantic_messages == [
|
||||
{
|
||||
"instructions": (
|
||||
"You are a helpful test assistant :)\n\n"
|
||||
"Today is Friday 25/07/2025.\n\nAnswer in english."
|
||||
),
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": "Today is Friday 25/07/2025.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": "Answer in english.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": ["Hello"],
|
||||
"part_kind": "user-prompt",
|
||||
@@ -400,12 +409,11 @@ def test_post_conversation_with_image(api_client, mock_openai_stream_image):
|
||||
# Check the exact structure expected by the AI service
|
||||
assert body["messages"] == [
|
||||
{
|
||||
"content": (
|
||||
"You are a helpful test assistant :)\n\nToday is Friday 25/07/2025."
|
||||
"\n\nAnswer in english."
|
||||
),
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"role": "system",
|
||||
},
|
||||
{"content": "Today is Friday 25/07/2025.", "role": "system"},
|
||||
{"content": "Answer in english.", "role": "system"},
|
||||
{
|
||||
"content": [
|
||||
{"text": "Hello, what do you see on this picture?", "type": "text"},
|
||||
@@ -490,12 +498,27 @@ def test_post_conversation_with_image(api_client, mock_openai_stream_image):
|
||||
_run_id = chat_conversation.pydantic_messages[0]["run_id"]
|
||||
assert chat_conversation.pydantic_messages == [
|
||||
{
|
||||
"instructions": (
|
||||
"You are a helpful test assistant :)\n\n"
|
||||
"Today is Friday 25/07/2025.\n\nAnswer in english."
|
||||
),
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": "Today is Friday 25/07/2025.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": "Answer in english.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": [
|
||||
"Hello, what do you see on this picture?",
|
||||
@@ -593,12 +616,11 @@ def test_post_conversation_tool_call(api_client, mock_openai_stream_tool, settin
|
||||
|
||||
assert body["messages"] == [
|
||||
{
|
||||
"content": (
|
||||
"You are a helpful test assistant :)\n\n"
|
||||
"Today is Friday 25/07/2025.\n\nAnswer in english."
|
||||
),
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"role": "system",
|
||||
},
|
||||
{"content": "Today is Friday 25/07/2025.", "role": "system"},
|
||||
{"content": "Answer in english.", "role": "system"},
|
||||
{"content": [{"text": "Weather in Paris?", "type": "text"}], "role": "user"},
|
||||
]
|
||||
|
||||
@@ -656,12 +678,27 @@ def test_post_conversation_tool_call(api_client, mock_openai_stream_tool, settin
|
||||
_run_id = chat_conversation.pydantic_messages[0]["run_id"]
|
||||
assert chat_conversation.pydantic_messages == [
|
||||
{
|
||||
"instructions": (
|
||||
"You are a helpful test assistant :)\n\n"
|
||||
"Today is Friday 25/07/2025.\n\nAnswer in english."
|
||||
),
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": "Today is Friday 25/07/2025.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": "Answer in english.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": ["Weather in Paris?"],
|
||||
"part_kind": "user-prompt",
|
||||
@@ -700,10 +737,7 @@ def test_post_conversation_tool_call(api_client, mock_openai_stream_tool, settin
|
||||
"run_id": _run_id,
|
||||
},
|
||||
{
|
||||
"instructions": (
|
||||
"You are a helpful test assistant :)\n\n"
|
||||
"Today is Friday 25/07/2025.\n\nAnswer in english."
|
||||
),
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
@@ -795,12 +829,11 @@ def test_post_conversation_tool_call_fails(api_client, mock_openai_stream_tool,
|
||||
|
||||
assert body["messages"] == [
|
||||
{
|
||||
"content": (
|
||||
"You are a helpful test assistant :)\n\n"
|
||||
"Today is Friday 25/07/2025.\n\nAnswer in french."
|
||||
),
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"role": "system",
|
||||
},
|
||||
{"content": "Today is Friday 25/07/2025.", "role": "system"},
|
||||
{"content": "Answer in french.", "role": "system"},
|
||||
{"content": [{"text": "Weather in Paris?", "type": "text"}], "role": "user"},
|
||||
]
|
||||
|
||||
@@ -858,12 +891,27 @@ def test_post_conversation_tool_call_fails(api_client, mock_openai_stream_tool,
|
||||
_run_id = chat_conversation.pydantic_messages[0]["run_id"]
|
||||
assert chat_conversation.pydantic_messages == [
|
||||
{
|
||||
"instructions": (
|
||||
"You are a helpful test assistant :)\n\n"
|
||||
"Today is Friday 25/07/2025.\n\nAnswer in french."
|
||||
),
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": "Today is Friday 25/07/2025.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": "Answer in french.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": ["Weather in Paris?"],
|
||||
"part_kind": "user-prompt",
|
||||
@@ -902,10 +950,7 @@ def test_post_conversation_tool_call_fails(api_client, mock_openai_stream_tool,
|
||||
"run_id": _run_id,
|
||||
},
|
||||
{
|
||||
"instructions": (
|
||||
"You are a helpful test assistant :)\n\n"
|
||||
"Today is Friday 25/07/2025.\n\nAnswer in french."
|
||||
),
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
@@ -1169,11 +1214,27 @@ def test_post_conversation_data_protocol_no_stream(
|
||||
_run_id = chat_conversation.pydantic_messages[0]["run_id"]
|
||||
assert chat_conversation.pydantic_messages == [
|
||||
{
|
||||
"instructions": (
|
||||
"You are an amazing assistant.\n\nToday is Friday 25/07/2025.\n\nAnswer in english."
|
||||
),
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are an amazing assistant.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": "Today is Friday 25/07/2025.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": "Answer in english.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": ["Why the sky is blue?"],
|
||||
"part_kind": "user-prompt",
|
||||
@@ -1308,12 +1369,27 @@ async def test_post_conversation_async(api_client, mock_openai_stream, monkeypat
|
||||
_run_id = chat_conversation.pydantic_messages[0]["run_id"]
|
||||
assert chat_conversation.pydantic_messages == [
|
||||
{
|
||||
"instructions": (
|
||||
"You are a helpful test assistant :)\n\n"
|
||||
"Today is Friday 25/07/2025.\n\nAnswer in english."
|
||||
),
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": "Today is Friday 25/07/2025.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": "Answer in english.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-07-25T10:36:35.297675Z",
|
||||
},
|
||||
{
|
||||
"content": ["Hello"],
|
||||
"part_kind": "user-prompt",
|
||||
|
||||
+181
-81
@@ -7,6 +7,7 @@ import json
|
||||
import logging
|
||||
from io import BytesIO
|
||||
from unittest import mock
|
||||
from unittest.mock import Mock
|
||||
|
||||
from django.utils import formats, timezone
|
||||
|
||||
@@ -41,28 +42,59 @@ from chat.tests.utils import replace_uuids_with_placeholder
|
||||
pytestmark = pytest.mark.django_db(transaction=True)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def ai_settings(settings):
|
||||
@pytest.fixture(
|
||||
autouse=True,
|
||||
params=[
|
||||
"chat.agent_rag.document_rag_backends.find_rag_backend.FindRagBackend",
|
||||
"chat.agent_rag.document_rag_backends.albert_rag_backend.AlbertRagBackend",
|
||||
],
|
||||
)
|
||||
def ai_settings(request, settings):
|
||||
"""Fixture to set AI service URLs for testing."""
|
||||
settings.AI_BASE_URL = "https://www.external-ai-service.com/"
|
||||
settings.AI_API_KEY = "test-api-key"
|
||||
settings.AI_MODEL = "test-model"
|
||||
settings.AI_AGENT_INSTRUCTIONS = "You are a helpful test assistant :)"
|
||||
|
||||
# Enable Albert API for document search
|
||||
settings.RAG_DOCUMENT_SEARCH_BACKEND = (
|
||||
"chat.agent_rag.document_rag_backends.albert_rag_backend.AlbertRagBackend"
|
||||
)
|
||||
settings.ALBERT_API_URL = "https://albert.api.etalab.gouv.fr"
|
||||
settings.ALBERT_API_KEY = "albert-api-key"
|
||||
# enable on rag document search tool
|
||||
settings.RAG_DOCUMENT_SEARCH_BACKEND = request.param
|
||||
settings.RAG_WEB_SEARCH_PROMPT_UPDATE = (
|
||||
"Based on the following document contents:\n\n{search_results}\n\n"
|
||||
"Please answer the user's question: {user_prompt}"
|
||||
)
|
||||
|
||||
settings.AI_BASE_URL = "https://www.external-ai-service.com/"
|
||||
settings.AI_API_KEY = "test-api-key"
|
||||
settings.AI_MODEL = "test-model"
|
||||
settings.AI_AGENT_INSTRUCTIONS = "You are a helpful test assistant :)"
|
||||
|
||||
# Albert API settings
|
||||
settings.ALBERT_API_URL = "https://albert.api.etalab.gouv.fr"
|
||||
settings.ALBERT_API_KEY = "albert-api-key"
|
||||
|
||||
# Find API settings
|
||||
settings.FIND_API_URL = "https://find.api.example.com"
|
||||
settings.FIND_API_KEY = "find-api-key"
|
||||
|
||||
return settings
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def mock_process_request():
|
||||
"""Mock process_request to bypass authentication in tests."""
|
||||
with mock.patch(
|
||||
"lasuite.oidc_login.decorators.RefreshOIDCAccessToken.process_request"
|
||||
) as mocked_process_request:
|
||||
mocked_process_request.return_value = None
|
||||
yield mocked_process_request
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def mock_refresh_access_token():
|
||||
"""Mock refresh_access_token to bypass token refresh in tests."""
|
||||
with mock.patch("utils.oicd.refresh_access_token") as mocked_refresh_access_token:
|
||||
mock_session = Mock(spec=httpx.Client)
|
||||
mock_session.access_token = "mocked-access-token"
|
||||
mocked_refresh_access_token.return_value = mock_session
|
||||
yield mocked_refresh_access_token
|
||||
|
||||
|
||||
@pytest.fixture(name="sample_pdf_content")
|
||||
def fixture_sample_pdf_content():
|
||||
"""Create a dummy PDF content as BytesIO."""
|
||||
@@ -134,6 +166,32 @@ def fixture_mock_albert_api():
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(name="mock_find_api")
|
||||
def fixture_mock_find_api():
|
||||
"""Fixture to mock the Find API endpoints."""
|
||||
# Mock document indexing (Find API)
|
||||
responses.post(
|
||||
"https://find.api.example.com/api/v1.0/documents/index/",
|
||||
json={"id": "456", "status": "indexed"},
|
||||
status=status.HTTP_200_OK,
|
||||
)
|
||||
|
||||
# Mock document search (Find API)
|
||||
responses.post(
|
||||
"https://find.api.example.com/api/v1.0/documents/search/",
|
||||
json=[
|
||||
{
|
||||
"_source": {
|
||||
"title.fr": "sample.pdf",
|
||||
"content.fr": "This is the content of the PDF.",
|
||||
},
|
||||
"_score": 0.9,
|
||||
}
|
||||
],
|
||||
status=status.HTTP_200_OK,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(name="mock_summarization_agent")
|
||||
def fixture_mock_summarization_agent():
|
||||
"""Mock the SummarizationAgent to return a fixed summary."""
|
||||
@@ -216,10 +274,10 @@ def fixture_mock_openai_stream():
|
||||
@responses.activate
|
||||
@respx.mock
|
||||
@freeze_time()
|
||||
def test_post_conversation_with_document_upload(
|
||||
# pylint: disable=too-many-arguments,too-many-positional-arguments
|
||||
def test_post_conversation_with_document_upload( # noqa: PLR0913 pylint: disable=too-many-arguments,too-many-positional-arguments
|
||||
api_client,
|
||||
mock_albert_api, # pylint: disable=unused-argument
|
||||
mock_find_api, # pylint: disable=unused-argument
|
||||
sample_pdf_content,
|
||||
today_promt_date,
|
||||
mock_ai_agent_service,
|
||||
@@ -354,25 +412,53 @@ def test_post_conversation_with_document_upload(
|
||||
assert len(chat_conversation.pydantic_messages) == 4
|
||||
|
||||
_run_id = chat_conversation.pydantic_messages[0]["run_id"]
|
||||
|
||||
assert chat_conversation.pydantic_messages[0] == {
|
||||
"instructions": "You are a helpful test assistant :)\n\n"
|
||||
f"{today_promt_date}\n\n"
|
||||
"Answer in english.\n\n"
|
||||
"Use document_search_rag ONLY to retrieve specific passages from "
|
||||
"attached documents. Do NOT use it to summarize; for summaries, "
|
||||
"call the summarize tool instead.\n\nWhen you receive a result from the "
|
||||
"summarization tool, you MUST return it directly to the user without "
|
||||
"any modification, paraphrasing, or additional summarization."
|
||||
"The tool already produces optimized summaries that should be "
|
||||
"presented verbatim.You may translate the summary if required, "
|
||||
"but you MUST preserve all the information from the original summary."
|
||||
"You may add a follow-up question after the summary if needed.\n\n"
|
||||
"[Internal context] User documents are attached to this conversation. "
|
||||
"Do not request re-upload of documents; consider them already "
|
||||
"available via the internal store.",
|
||||
"instructions": "When you receive a result from the summarization tool, you "
|
||||
"MUST return it directly to the user without any "
|
||||
"modification, paraphrasing, or additional summarization.The "
|
||||
"tool already produces optimized summaries that should be "
|
||||
"presented verbatim.You may translate the summary if "
|
||||
"required, but you MUST preserve all the information from the "
|
||||
"original summary.You may add a follow-up question after the "
|
||||
"summary if needed.",
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timezone_now,
|
||||
},
|
||||
{
|
||||
"content": today_promt_date,
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timezone_now,
|
||||
},
|
||||
{
|
||||
"content": "Answer in english.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timezone_now,
|
||||
},
|
||||
{
|
||||
"content": "Use document_search_rag ONLY to retrieve specific "
|
||||
"passages from attached documents. Do NOT use it to "
|
||||
"summarize; for summaries, call the summarize tool "
|
||||
"instead.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timezone_now,
|
||||
},
|
||||
{
|
||||
"content": "[Internal context] User documents are attached to this "
|
||||
"conversation. Do not request re-upload of documents; "
|
||||
"consider them already available via the internal "
|
||||
"store.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timezone_now,
|
||||
},
|
||||
{
|
||||
"content": ["What does the document say?"],
|
||||
"part_kind": "user-prompt",
|
||||
@@ -412,21 +498,14 @@ def test_post_conversation_with_document_upload(
|
||||
}
|
||||
assert chat_conversation.pydantic_messages[2] == {
|
||||
"instructions": (
|
||||
"You are a helpful test assistant :)\n\n"
|
||||
f"{today_promt_date}\n\n"
|
||||
"Answer in english.\n\n"
|
||||
"Use document_search_rag ONLY to retrieve specific passages from "
|
||||
"attached documents. Do NOT use it to summarize; for summaries, "
|
||||
"call the summarize tool instead.\n\nWhen you receive a result from the "
|
||||
"summarization tool, you MUST return it directly to the user without "
|
||||
"any modification, paraphrasing, or additional summarization."
|
||||
"The tool already produces optimized summaries that should be "
|
||||
"presented verbatim.You may translate the summary if required, "
|
||||
"but you MUST preserve all the information from the original summary."
|
||||
"You may add a follow-up question after the summary if needed.\n\n"
|
||||
"[Internal context] User documents are attached to this conversation. "
|
||||
"Do not request re-upload of documents; consider them already "
|
||||
"available via the internal store."
|
||||
"When you receive a result from the summarization tool, you MUST "
|
||||
"return it directly to the user without any modification, "
|
||||
"paraphrasing, or additional summarization."
|
||||
"The tool already produces optimized summaries that should "
|
||||
"be presented verbatim."
|
||||
"You may translate the summary if required, but you MUST preserve "
|
||||
"all the information from the original summary."
|
||||
"You may add a follow-up question after the summary if needed."
|
||||
),
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
@@ -479,8 +558,7 @@ def test_post_conversation_with_document_upload(
|
||||
@responses.activate
|
||||
@respx.mock
|
||||
@freeze_time("2025-07-25T10:36:35.297675Z")
|
||||
def test_post_conversation_with_document_upload_feature_disabled(
|
||||
# pylint: disable=too-many-arguments,too-many-positional-arguments
|
||||
def test_post_conversation_with_document_upload_feature_disabled( # pylint: disable=too-many-arguments,too-many-positional-arguments
|
||||
api_client,
|
||||
caplog,
|
||||
mock_openai_stream, # pylint: disable=unused-argument
|
||||
@@ -533,12 +611,14 @@ def test_post_conversation_with_document_upload_feature_disabled(
|
||||
|
||||
# Replace UUIDs with placeholders for assertion
|
||||
response_content = replace_uuids_with_placeholder(response_content)
|
||||
|
||||
assert response_content == (
|
||||
'0:"From the document, I can see that "\n'
|
||||
"0:\"it says 'Hello PDF'.\"\n"
|
||||
'f:{"messageId":"<mocked_uuid>"}\n'
|
||||
'd:{"finishReason":"stop","usage":{"promptTokens":150,"completionTokens":25}}\n'
|
||||
)
|
||||
|
||||
# This behavior must be improved in the future to inform the user properly
|
||||
assert "Document upload feature is disabled, ignoring input documents." in caplog.text
|
||||
|
||||
@@ -549,6 +629,7 @@ def test_post_conversation_with_document_upload_feature_disabled(
|
||||
def test_post_conversation_with_document_upload_summarize( # pylint: disable=too-many-arguments,too-many-positional-arguments # noqa: PLR0913
|
||||
api_client,
|
||||
mock_albert_api, # pylint: disable=unused-argument
|
||||
mock_find_api, # pylint: disable=unused-argument
|
||||
sample_pdf_content,
|
||||
today_promt_date,
|
||||
mock_ai_agent_service,
|
||||
@@ -561,7 +642,6 @@ def test_post_conversation_with_document_upload_summarize( # pylint: disable=to
|
||||
api_client.force_authenticate(user=chat_conversation.owner)
|
||||
|
||||
pdf_base64 = base64.b64encode(sample_pdf_content.read()).decode("utf-8")
|
||||
|
||||
message = UIMessage(
|
||||
id="1",
|
||||
role="user",
|
||||
@@ -623,7 +703,7 @@ def test_post_conversation_with_document_upload_summarize( # pylint: disable=to
|
||||
'document discusses various topics."}\n'
|
||||
'0:"The document discusses various topics."\n'
|
||||
'f:{"messageId":"<mocked_uuid>"}\n'
|
||||
'd:{"finishReason":"stop","usage":{"promptTokens":287,"completionTokens":19}}\n'
|
||||
'd:{"finishReason":"stop","usage":{"promptTokens":317,"completionTokens":19}}\n'
|
||||
)
|
||||
|
||||
# Check that the conversation was updated
|
||||
@@ -685,25 +765,52 @@ def test_post_conversation_with_document_upload_summarize( # pylint: disable=to
|
||||
|
||||
_run_id = chat_conversation.pydantic_messages[0]["run_id"]
|
||||
assert chat_conversation.pydantic_messages[0] == {
|
||||
"instructions": (
|
||||
"You are a helpful test assistant :)\n\n"
|
||||
f"{today_promt_date}\n\n"
|
||||
"Answer in english.\n\n"
|
||||
"Use document_search_rag ONLY to retrieve specific passages from "
|
||||
"attached documents. Do NOT use it to summarize; for summaries, "
|
||||
"call the summarize tool instead.\n\nWhen you receive a result from the "
|
||||
"summarization tool, you MUST return it directly to the user without "
|
||||
"any modification, paraphrasing, or additional summarization."
|
||||
"The tool already produces optimized summaries that should be "
|
||||
"presented verbatim.You may translate the summary if required, "
|
||||
"but you MUST preserve all the information from the original summary."
|
||||
"You may add a follow-up question after the summary if needed.\n\n"
|
||||
"[Internal context] User documents are attached to this conversation. "
|
||||
"Do not request re-upload of documents; consider them already "
|
||||
"available via the internal store."
|
||||
),
|
||||
"instructions": "When you receive a result from the summarization tool, you "
|
||||
"MUST return it directly to the user without any "
|
||||
"modification, paraphrasing, or additional summarization.The "
|
||||
"tool already produces optimized summaries that should be "
|
||||
"presented verbatim.You may translate the summary if "
|
||||
"required, but you MUST preserve all the information from the "
|
||||
"original summary.You may add a follow-up question after the "
|
||||
"summary if needed.",
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timezone_now,
|
||||
},
|
||||
{
|
||||
"content": today_promt_date,
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timezone_now,
|
||||
},
|
||||
{
|
||||
"content": "Answer in english.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timezone_now,
|
||||
},
|
||||
{
|
||||
"content": "Use document_search_rag ONLY to retrieve specific "
|
||||
"passages from attached documents. Do NOT use it to "
|
||||
"summarize; for summaries, call the summarize tool "
|
||||
"instead.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timezone_now,
|
||||
},
|
||||
{
|
||||
"content": "[Internal context] User documents are attached to this "
|
||||
"conversation. Do not request re-upload of documents; "
|
||||
"consider them already available via the internal "
|
||||
"store.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timezone_now,
|
||||
},
|
||||
{
|
||||
"content": ["Make a summary of this document."],
|
||||
"part_kind": "user-prompt",
|
||||
@@ -743,21 +850,14 @@ def test_post_conversation_with_document_upload_summarize( # pylint: disable=to
|
||||
}
|
||||
assert chat_conversation.pydantic_messages[2] == {
|
||||
"instructions": (
|
||||
"You are a helpful test assistant :)\n\n"
|
||||
f"{today_promt_date}\n\n"
|
||||
"Answer in english.\n\n"
|
||||
"Use document_search_rag ONLY to retrieve specific passages from "
|
||||
"attached documents. Do NOT use it to summarize; for summaries, "
|
||||
"call the summarize tool instead.\n\nWhen you receive a result from the "
|
||||
"summarization tool, you MUST return it directly to the user without "
|
||||
"any modification, paraphrasing, or additional summarization."
|
||||
"The tool already produces optimized summaries that should be "
|
||||
"presented verbatim.You may translate the summary if required, "
|
||||
"but you MUST preserve all the information from the original summary."
|
||||
"You may add a follow-up question after the summary if needed.\n\n"
|
||||
"[Internal context] User documents are attached to this conversation. "
|
||||
"Do not request re-upload of documents; consider them already "
|
||||
"available via the internal store."
|
||||
"When you receive a result from the summarization tool, you MUST "
|
||||
"return it directly to the user without any modification, "
|
||||
"paraphrasing, or additional summarization."
|
||||
"The tool already produces optimized summaries that should "
|
||||
"be presented verbatim."
|
||||
"You may translate the summary if required, but you MUST preserve "
|
||||
"all the information from the original summary."
|
||||
"You may add a follow-up question after the summary if needed."
|
||||
),
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
|
||||
+154
-60
@@ -17,6 +17,7 @@ from pydantic_ai.messages import (
|
||||
DocumentUrl,
|
||||
ModelMessage,
|
||||
ModelResponse,
|
||||
SystemPromptPart,
|
||||
TextPart,
|
||||
UserPromptPart,
|
||||
)
|
||||
@@ -60,8 +61,7 @@ def fixture_sample_document_content():
|
||||
|
||||
@responses.activate
|
||||
@freeze_time()
|
||||
def test_post_conversation_with_local_pdf_document_url(
|
||||
# pylint: disable=too-many-arguments,too-many-positional-arguments
|
||||
def test_post_conversation_with_local_pdf_document_url( # pylint: disable=too-many-arguments,too-many-positional-arguments
|
||||
api_client,
|
||||
sample_document_content,
|
||||
today_promt_date,
|
||||
@@ -120,7 +120,7 @@ def test_post_conversation_with_local_pdf_document_url(
|
||||
)
|
||||
|
||||
async def agent_model(messages: list[ModelMessage], _info: AgentInfo):
|
||||
presigned_url = messages[0].parts[0].content[1].url
|
||||
presigned_url = messages[0].parts[3].content[1].url
|
||||
assert presigned_url.startswith("http://localhost:9000/conversations-media-storage/")
|
||||
assert presigned_url.find("X-Amz-Signature=") != -1
|
||||
assert presigned_url.find("X-Amz-Date=") != -1
|
||||
@@ -129,6 +129,11 @@ def test_post_conversation_with_local_pdf_document_url(
|
||||
assert messages == [
|
||||
ModelRequest(
|
||||
parts=[
|
||||
SystemPromptPart(
|
||||
content="You are a helpful test assistant :)", timestamp=timezone.now()
|
||||
),
|
||||
SystemPromptPart(content=today_promt_date, timestamp=timezone.now()),
|
||||
SystemPromptPart(content="Answer in english.", timestamp=timezone.now()),
|
||||
UserPromptPart(
|
||||
content=[
|
||||
"What is in this document?",
|
||||
@@ -141,8 +146,6 @@ def test_post_conversation_with_local_pdf_document_url(
|
||||
timestamp=timezone.now(),
|
||||
),
|
||||
],
|
||||
instructions=f"You are a helpful test assistant :)\n\n{today_promt_date}"
|
||||
"\n\nAnswer in english.",
|
||||
run_id=messages[0].run_id,
|
||||
)
|
||||
]
|
||||
@@ -218,11 +221,27 @@ def test_post_conversation_with_local_pdf_document_url(
|
||||
_run_id = chat_conversation.pydantic_messages[0]["run_id"]
|
||||
assert chat_conversation.pydantic_messages == [
|
||||
{
|
||||
"instructions": "You are a helpful test assistant :)\n\n"
|
||||
f"{today_promt_date}\n\n"
|
||||
"Answer in english.",
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timestamp,
|
||||
},
|
||||
{
|
||||
"content": today_promt_date,
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timestamp,
|
||||
},
|
||||
{
|
||||
"content": "Answer in english.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timestamp,
|
||||
},
|
||||
{
|
||||
"content": [
|
||||
"What is in this document?",
|
||||
@@ -410,6 +429,7 @@ def test_post_conversation_with_remote_document_url(
|
||||
@freeze_time("2025-10-18T20:48:20.286204Z")
|
||||
def test_post_conversation_with_local_document_url_in_history( # pylint: disable=too-many-arguments,too-many-positional-arguments
|
||||
api_client,
|
||||
today_promt_date,
|
||||
mock_ai_agent_service,
|
||||
):
|
||||
"""
|
||||
@@ -417,8 +437,6 @@ def test_post_conversation_with_local_document_url_in_history( # pylint: disabl
|
||||
"""
|
||||
chat_conversation_pk = "0be55da5-8eb7-4dad-aa0f-fea454bd5809"
|
||||
document_url = f"/media-key/{chat_conversation_pk}/sample.pdf"
|
||||
formatted_date = formats.date_format(timezone.now(), "l d/m/Y", use_l10n=False)
|
||||
|
||||
chat_conversation = ChatConversationFactory(
|
||||
pk=chat_conversation_pk,
|
||||
owner__language="en-us",
|
||||
@@ -454,11 +472,27 @@ def test_post_conversation_with_local_document_url_in_history( # pylint: disabl
|
||||
],
|
||||
pydantic_messages=[
|
||||
{
|
||||
"instructions": "You are a helpful test assistant :)\n\n"
|
||||
f"Today is {formatted_date}.\n\n"
|
||||
"Answer in english.",
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-10-18T20:48:20.286204Z",
|
||||
},
|
||||
{
|
||||
"content": today_promt_date,
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-10-18T20:48:20.286204Z",
|
||||
},
|
||||
{
|
||||
"content": "Answer in english.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-10-18T20:48:20.286204Z",
|
||||
},
|
||||
{
|
||||
"content": [
|
||||
"What is in this document?",
|
||||
@@ -521,7 +555,7 @@ def test_post_conversation_with_local_document_url_in_history( # pylint: disabl
|
||||
)
|
||||
|
||||
async def agent_model(messages: list[ModelMessage], _info: AgentInfo):
|
||||
presigned_url = messages[0].parts[0].content[1].url
|
||||
presigned_url = messages[0].parts[3].content[1].url
|
||||
assert presigned_url.startswith("http://localhost:9000/conversations-media-storage/")
|
||||
assert presigned_url.find("X-Amz-Signature=") != -1
|
||||
assert presigned_url.find("X-Amz-Date=") != -1
|
||||
@@ -530,6 +564,18 @@ def test_post_conversation_with_local_document_url_in_history( # pylint: disabl
|
||||
assert messages == [
|
||||
ModelRequest(
|
||||
parts=[
|
||||
SystemPromptPart(
|
||||
content="You are a helpful test assistant :)",
|
||||
timestamp=timezone.now(),
|
||||
),
|
||||
SystemPromptPart(
|
||||
content=today_promt_date,
|
||||
timestamp=timezone.now(),
|
||||
),
|
||||
SystemPromptPart(
|
||||
content="Answer in english.",
|
||||
timestamp=timezone.now(),
|
||||
),
|
||||
UserPromptPart(
|
||||
content=[
|
||||
"What is in this document?",
|
||||
@@ -542,9 +588,6 @@ def test_post_conversation_with_local_document_url_in_history( # pylint: disabl
|
||||
timestamp=timezone.now(),
|
||||
),
|
||||
],
|
||||
instructions="You are a helpful test assistant :)\n\n"
|
||||
"Today is Saturday 18/10/2025.\n\n"
|
||||
"Answer in english.",
|
||||
run_id=messages[0].run_id,
|
||||
),
|
||||
ModelResponse(
|
||||
@@ -563,9 +606,6 @@ def test_post_conversation_with_local_document_url_in_history( # pylint: disabl
|
||||
timestamp=timezone.now(),
|
||||
)
|
||||
],
|
||||
instructions="You are a helpful test assistant :)\n\n"
|
||||
"Today is Saturday 18/10/2025.\n\n"
|
||||
"Answer in english.",
|
||||
run_id=messages[2].run_id,
|
||||
),
|
||||
]
|
||||
@@ -665,11 +705,27 @@ def test_post_conversation_with_local_document_url_in_history( # pylint: disabl
|
||||
_run_id = chat_conversation.pydantic_messages[2]["run_id"]
|
||||
assert chat_conversation.pydantic_messages == [
|
||||
{
|
||||
"instructions": "You are a helpful test assistant :)\n\n"
|
||||
"Today is Saturday 18/10/2025.\n\n"
|
||||
"Answer in english.",
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-10-18T20:48:20.286204Z",
|
||||
},
|
||||
{
|
||||
"content": today_promt_date,
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-10-18T20:48:20.286204Z",
|
||||
},
|
||||
{
|
||||
"content": "Answer in english.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-10-18T20:48:20.286204Z",
|
||||
},
|
||||
{
|
||||
"content": [
|
||||
"What is in this document?",
|
||||
@@ -716,9 +772,7 @@ def test_post_conversation_with_local_document_url_in_history( # pylint: disabl
|
||||
# no run_id here
|
||||
},
|
||||
{
|
||||
"instructions": "You are a helpful test assistant :)\n\n"
|
||||
"Today is Saturday 18/10/2025.\n\n"
|
||||
"Answer in english.",
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
@@ -769,8 +823,7 @@ def test_post_conversation_with_local_document_url_in_history( # pylint: disabl
|
||||
("data.csv", "text/csv"),
|
||||
],
|
||||
)
|
||||
def test_post_conversation_with_local_not_pdf_document_url(
|
||||
# pylint: disable=too-many-arguments,too-many-positional-arguments
|
||||
def test_post_conversation_with_local_not_pdf_document_url( # pylint: disable=too-many-arguments,too-many-positional-arguments
|
||||
api_client,
|
||||
today_promt_date,
|
||||
mock_ai_agent_service,
|
||||
@@ -833,6 +886,27 @@ def test_post_conversation_with_local_not_pdf_document_url(
|
||||
assert messages == [
|
||||
ModelRequest(
|
||||
parts=[
|
||||
SystemPromptPart(
|
||||
content="You are a helpful test assistant :)", timestamp=timezone.now()
|
||||
),
|
||||
SystemPromptPart(content=today_promt_date, timestamp=timezone.now()),
|
||||
SystemPromptPart(content="Answer in english.", timestamp=timezone.now()),
|
||||
SystemPromptPart(
|
||||
content=(
|
||||
"Use document_search_rag ONLY to retrieve specific passages from "
|
||||
"attached documents. Do NOT use it to summarize; for summaries, "
|
||||
"call the summarize tool instead."
|
||||
),
|
||||
timestamp=timezone.now(),
|
||||
),
|
||||
SystemPromptPart(
|
||||
content=(
|
||||
"[Internal context] User documents are attached to this conversation. "
|
||||
"Do not request re-upload of documents; consider them already "
|
||||
"available via the internal store."
|
||||
),
|
||||
timestamp=timezone.now(),
|
||||
),
|
||||
UserPromptPart(
|
||||
content=[
|
||||
"What is in this document?",
|
||||
@@ -842,22 +916,14 @@ def test_post_conversation_with_local_not_pdf_document_url(
|
||||
),
|
||||
],
|
||||
instructions=(
|
||||
"You are a helpful test assistant :)\n\n"
|
||||
f"{today_promt_date}\n\n"
|
||||
"Answer in english.\n\n"
|
||||
"Use document_search_rag ONLY to retrieve specific passages from "
|
||||
"attached documents. Do NOT use it to summarize; for summaries, "
|
||||
"call the summarize tool instead.\n\nWhen you receive a result "
|
||||
"from the summarization tool, you MUST return it directly to "
|
||||
"the user without any modification, paraphrasing, or additional "
|
||||
"summarization.The tool already produces optimized summaries "
|
||||
"that should be presented verbatim.You may translate the summary "
|
||||
"if required, but you MUST preserve all the information from the "
|
||||
"original summary.You may add a follow-up question after the "
|
||||
"summary if needed.\n\n"
|
||||
"[Internal context] User documents are attached to this conversation. "
|
||||
"Do not request re-upload of documents; "
|
||||
"consider them already available via the internal store."
|
||||
"When you receive a result from the summarization tool, you MUST "
|
||||
"return it directly to the user without any modification, "
|
||||
"paraphrasing, or additional summarization."
|
||||
"The tool already produces optimized summaries that should "
|
||||
"be presented verbatim."
|
||||
"You may translate the summary if required, but you MUST preserve "
|
||||
"all the information from the original summary."
|
||||
"You may add a follow-up question after the summary if needed."
|
||||
),
|
||||
run_id=messages[0].run_id,
|
||||
)
|
||||
@@ -933,25 +999,53 @@ def test_post_conversation_with_local_not_pdf_document_url(
|
||||
assert chat_conversation.pydantic_messages == [
|
||||
{
|
||||
"instructions": (
|
||||
"You are a helpful test assistant :)\n\n"
|
||||
f"{today_promt_date}\n\n"
|
||||
"Answer in english.\n\n"
|
||||
"Use document_search_rag ONLY to retrieve specific passages from "
|
||||
"attached documents. Do NOT use it to summarize; for summaries, "
|
||||
"call the summarize tool instead.\n\nWhen you receive a result "
|
||||
"from the summarization tool, you MUST return it directly to "
|
||||
"the user without any modification, paraphrasing, or additional "
|
||||
"summarization.The tool already produces optimized summaries "
|
||||
"that should be presented verbatim.You may translate the summary "
|
||||
"if required, but you MUST preserve all the information from the "
|
||||
"original summary.You may add a follow-up question after the "
|
||||
"summary if needed.\n\n"
|
||||
"[Internal context] User documents are attached to this conversation. "
|
||||
"Do not request re-upload of documents; "
|
||||
"consider them already available via the internal store."
|
||||
"When you receive a result from the summarization tool, you MUST "
|
||||
"return it directly to the user without any modification, "
|
||||
"paraphrasing, or additional summarization."
|
||||
"The tool already produces optimized summaries that should "
|
||||
"be presented verbatim."
|
||||
"You may translate the summary if required, but you MUST preserve "
|
||||
"all the information from the original summary."
|
||||
"You may add a follow-up question after the summary if needed."
|
||||
),
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timestamp,
|
||||
},
|
||||
{
|
||||
"content": today_promt_date,
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timestamp,
|
||||
},
|
||||
{
|
||||
"content": "Answer in english.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timestamp,
|
||||
},
|
||||
{
|
||||
"content": "Use document_search_rag ONLY to retrieve specific "
|
||||
"passages from attached documents. Do NOT use it to "
|
||||
"summarize; for summaries, call the summarize tool "
|
||||
"instead.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timestamp,
|
||||
},
|
||||
{
|
||||
"content": "[Internal context] User documents are attached to "
|
||||
"this conversation. Do not request re-upload of "
|
||||
"documents; consider them already available via the "
|
||||
"internal store.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": timestamp,
|
||||
},
|
||||
{
|
||||
"content": [
|
||||
"What is in this document?",
|
||||
|
||||
@@ -919,7 +919,7 @@ def history_conversation_with_tool_fixture():
|
||||
history_timestamp = timezone.now().replace(year=2025, month=6, day=15, hour=10, minute=30)
|
||||
|
||||
# Create a conversation with pre-existing messages including a tool invocation
|
||||
conversation = ChatConversationFactory(owner__language="nl-nl")
|
||||
conversation = ChatConversationFactory()
|
||||
|
||||
# Add previous user and assistant messages with tool invocation
|
||||
conversation.messages = [
|
||||
@@ -1377,9 +1377,7 @@ def test_post_conversation_with_existing_tool_history(
|
||||
|
||||
# Verify the new tool call request is included
|
||||
assert history_conversation_with_tool.pydantic_messages[8] == {
|
||||
"instructions": "You are a helpful test assistant :)\n\n"
|
||||
"Today is Friday 25/07/2025.\n\n"
|
||||
"Answer in dutch.",
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
@@ -1422,9 +1420,7 @@ def test_post_conversation_with_existing_tool_history(
|
||||
}
|
||||
|
||||
assert history_conversation_with_tool.pydantic_messages[10] == {
|
||||
"instructions": "You are a helpful test assistant :)\n\n"
|
||||
"Today is Friday 25/07/2025.\n\n"
|
||||
"Answer in dutch.",
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
|
||||
+94
-24
@@ -2,7 +2,7 @@
|
||||
|
||||
import uuid
|
||||
|
||||
from django.utils import formats, timezone
|
||||
from django.utils import timezone
|
||||
|
||||
import pytest
|
||||
from dirty_equals import IsUUID
|
||||
@@ -12,6 +12,7 @@ from pydantic_ai.messages import (
|
||||
ImageUrl,
|
||||
ModelMessage,
|
||||
ModelResponse,
|
||||
SystemPromptPart,
|
||||
TextPart,
|
||||
UserPromptPart,
|
||||
)
|
||||
@@ -86,15 +87,22 @@ def test_post_conversation_with_local_image_url(
|
||||
)
|
||||
|
||||
async def agent_model(messages: list[ModelMessage], _info: AgentInfo):
|
||||
presigned_url = messages[0].parts[0].content[1].url
|
||||
# assert presigned_url.startswith("http://localhost:9000/conversations-media-storage/")
|
||||
presigned_url = messages[0].parts[3].content[1].url
|
||||
assert presigned_url.startswith("http://localhost:9000/conversations-media-storage/")
|
||||
assert presigned_url.find("X-Amz-Signature=") != -1
|
||||
assert presigned_url.find("X-Amz-Date=") != -1
|
||||
assert presigned_url.find("X-Amz-Expires=") != -1
|
||||
formatted_date = formats.date_format(timezone.now(), "l d/m/Y", use_l10n=False)
|
||||
|
||||
assert messages == [
|
||||
ModelRequest(
|
||||
parts=[
|
||||
SystemPromptPart(
|
||||
content="You are a helpful test assistant :)", timestamp=timezone.now()
|
||||
),
|
||||
SystemPromptPart(
|
||||
content="Today is Saturday 18/10/2025.", timestamp=timezone.now()
|
||||
),
|
||||
SystemPromptPart(content="Answer in english.", timestamp=timezone.now()),
|
||||
UserPromptPart(
|
||||
content=[
|
||||
"What is in this image?",
|
||||
@@ -107,8 +115,6 @@ def test_post_conversation_with_local_image_url(
|
||||
timestamp=timezone.now(),
|
||||
),
|
||||
],
|
||||
instructions="You are a helpful test assistant :)\n\nToday is "
|
||||
f"{formatted_date}.\n\nAnswer in english.",
|
||||
run_id=messages[0].run_id,
|
||||
)
|
||||
]
|
||||
@@ -178,10 +184,27 @@ def test_post_conversation_with_local_image_url(
|
||||
_run_id = chat_conversation.pydantic_messages[0]["run_id"]
|
||||
assert chat_conversation.pydantic_messages == [
|
||||
{
|
||||
"instructions": "You are a helpful test assistant :)\n\n"
|
||||
"Today is Saturday 18/10/2025.\n\nAnswer in english.",
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-10-18T20:48:20.286204Z",
|
||||
},
|
||||
{
|
||||
"content": "Today is Saturday 18/10/2025.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-10-18T20:48:20.286204Z",
|
||||
},
|
||||
{
|
||||
"content": "Answer in english.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-10-18T20:48:20.286204Z",
|
||||
},
|
||||
{
|
||||
"content": [
|
||||
"What is in this image?",
|
||||
@@ -263,6 +286,11 @@ def test_post_conversation_with_local_image_wrong_url(
|
||||
assert messages == [
|
||||
ModelRequest(
|
||||
parts=[
|
||||
SystemPromptPart(
|
||||
content="You are a helpful test assistant :)", timestamp=timezone.now()
|
||||
),
|
||||
SystemPromptPart(content=today_promt_date, timestamp=timezone.now()),
|
||||
SystemPromptPart(content="Answer in english.", timestamp=timezone.now()),
|
||||
UserPromptPart(
|
||||
content=[
|
||||
"What is in this image?",
|
||||
@@ -275,8 +303,6 @@ def test_post_conversation_with_local_image_wrong_url(
|
||||
timestamp=timezone.now(),
|
||||
),
|
||||
],
|
||||
instructions=f"You are a helpful test assistant :)\n\n{today_promt_date}"
|
||||
"\n\nAnswer in english.",
|
||||
run_id=messages[0].run_id,
|
||||
)
|
||||
]
|
||||
@@ -348,6 +374,11 @@ def test_post_conversation_with_remote_image_url(
|
||||
assert messages == [
|
||||
ModelRequest(
|
||||
parts=[
|
||||
SystemPromptPart(
|
||||
content="You are a helpful test assistant :)", timestamp=timezone.now()
|
||||
),
|
||||
SystemPromptPart(content=today_promt_date, timestamp=timezone.now()),
|
||||
SystemPromptPart(content="Answer in english.", timestamp=timezone.now()),
|
||||
UserPromptPart(
|
||||
content=[
|
||||
"What is in this image?",
|
||||
@@ -360,8 +391,6 @@ def test_post_conversation_with_remote_image_url(
|
||||
timestamp=timezone.now(),
|
||||
),
|
||||
],
|
||||
instructions="You are a helpful test assistant :)\n\n"
|
||||
f"{today_promt_date}\n\nAnswer in english.",
|
||||
run_id=messages[0].run_id,
|
||||
)
|
||||
]
|
||||
@@ -475,10 +504,27 @@ def test_post_conversation_with_local_image_url_in_history(
|
||||
],
|
||||
pydantic_messages=[
|
||||
{
|
||||
"instructions": f"You are a helpful test assistant :)\n\n{today_promt_date}"
|
||||
"\n\nAnswer in english.",
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-10-18T20:48:20.286204Z",
|
||||
},
|
||||
{
|
||||
"content": today_promt_date,
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-10-18T20:48:20.286204Z",
|
||||
},
|
||||
{
|
||||
"content": "Answer in english.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-10-18T20:48:20.286204Z",
|
||||
},
|
||||
{
|
||||
"content": [
|
||||
"What is in this image?",
|
||||
@@ -541,7 +587,7 @@ def test_post_conversation_with_local_image_url_in_history(
|
||||
)
|
||||
|
||||
async def agent_model(messages: list[ModelMessage], _info: AgentInfo):
|
||||
presigned_url = messages[0].parts[0].content[1].url
|
||||
presigned_url = messages[0].parts[3].content[1].url
|
||||
assert presigned_url.startswith("http://localhost:9000/conversations-media-storage/")
|
||||
assert presigned_url.find("X-Amz-Signature=") != -1
|
||||
assert presigned_url.find("X-Amz-Date=") != -1
|
||||
@@ -550,6 +596,18 @@ def test_post_conversation_with_local_image_url_in_history(
|
||||
assert messages == [
|
||||
ModelRequest(
|
||||
parts=[
|
||||
SystemPromptPart(
|
||||
content="You are a helpful test assistant :)",
|
||||
timestamp=timezone.now(),
|
||||
),
|
||||
SystemPromptPart(
|
||||
content=today_promt_date,
|
||||
timestamp=timezone.now(),
|
||||
),
|
||||
SystemPromptPart(
|
||||
content="Answer in english.",
|
||||
timestamp=timezone.now(),
|
||||
),
|
||||
UserPromptPart(
|
||||
content=[
|
||||
"What is in this image?",
|
||||
@@ -561,9 +619,7 @@ def test_post_conversation_with_local_image_url_in_history(
|
||||
],
|
||||
timestamp=timezone.now(),
|
||||
),
|
||||
],
|
||||
instructions="You are a helpful test assistant :)\n\n"
|
||||
f"{today_promt_date}\n\nAnswer in english.",
|
||||
]
|
||||
),
|
||||
ModelResponse(
|
||||
parts=[TextPart(content="This is an image of a single pixel.")],
|
||||
@@ -581,8 +637,6 @@ def test_post_conversation_with_local_image_url_in_history(
|
||||
)
|
||||
],
|
||||
run_id=messages[2].run_id,
|
||||
instructions="You are a helpful test assistant :)\n\n"
|
||||
"Today is Saturday 18/10/2025.\n\nAnswer in english.",
|
||||
),
|
||||
]
|
||||
yield "This is an image of square, very small and nice."
|
||||
@@ -681,10 +735,27 @@ def test_post_conversation_with_local_image_url_in_history(
|
||||
_run_id = chat_conversation.pydantic_messages[2]["run_id"]
|
||||
assert chat_conversation.pydantic_messages == [
|
||||
{
|
||||
"instructions": f"You are a helpful test assistant :)\n\n{today_promt_date}"
|
||||
"\n\nAnswer in english.",
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
"content": "You are a helpful test assistant :)",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-10-18T20:48:20.286204Z",
|
||||
},
|
||||
{
|
||||
"content": today_promt_date,
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-10-18T20:48:20.286204Z",
|
||||
},
|
||||
{
|
||||
"content": "Answer in english.",
|
||||
"dynamic_ref": None,
|
||||
"part_kind": "system-prompt",
|
||||
"timestamp": "2025-10-18T20:48:20.286204Z",
|
||||
},
|
||||
{
|
||||
"content": [
|
||||
"What is in this image?",
|
||||
@@ -725,8 +796,7 @@ def test_post_conversation_with_local_image_url_in_history(
|
||||
},
|
||||
},
|
||||
{
|
||||
"instructions": "You are a helpful test assistant :)\n\nToday is Saturday 18/10/2025."
|
||||
"\n\nAnswer in english.",
|
||||
"instructions": None,
|
||||
"kind": "request",
|
||||
"parts": [
|
||||
{
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
"""Test the post_stop_streaming view."""
|
||||
"""Test the post_stop_steaming view."""
|
||||
|
||||
from unittest.mock import patch
|
||||
|
||||
@@ -1,873 +0,0 @@
|
||||
"""Data analysis tool for tabular files (CSV, Excel)."""
|
||||
|
||||
import base64
|
||||
import functools
|
||||
import json
|
||||
import logging
|
||||
import uuid
|
||||
from io import BytesIO
|
||||
from typing import Any, Dict
|
||||
|
||||
import matplotlib
|
||||
import numpy as np
|
||||
|
||||
matplotlib.use("Agg") # Use non-interactive backend
|
||||
import matplotlib.pyplot as plt
|
||||
import pandas as pd
|
||||
|
||||
from django.conf import settings
|
||||
from django.core.files.storage import default_storage
|
||||
from django.db.models import Q
|
||||
from asgiref.sync import sync_to_async
|
||||
from pydantic_ai import Agent, RunContext
|
||||
from pydantic_ai.exceptions import ModelRetry
|
||||
from pydantic_ai.messages import ToolReturn
|
||||
|
||||
from core.file_upload.enums import AttachmentStatus
|
||||
from core.file_upload.utils import generate_retrieve_policy
|
||||
|
||||
from chat.agents.base import BaseAgent, prepare_custom_model
|
||||
from chat.models import ChatConversationAttachment
|
||||
from chat.tools.exceptions import ModelCannotRetry
|
||||
from chat.tools.utils import last_model_retry_soft_fail
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# MIME types for tabular files
|
||||
TABULAR_MIME_TYPES = [
|
||||
"text/csv",
|
||||
"application/csv",
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
"application/vnd.ms-excel",
|
||||
"application/excel",
|
||||
]
|
||||
|
||||
|
||||
@sync_to_async
|
||||
def read_tabular_file(attachment) -> bytes:
|
||||
"""Read tabular file content asynchronously."""
|
||||
with default_storage.open(attachment.key, "rb") as f:
|
||||
return f.read()
|
||||
|
||||
|
||||
def detect_csv_separator(file_data: bytes) -> str:
|
||||
"""
|
||||
Detect the CSV separator by analyzing the first few lines.
|
||||
|
||||
Returns the most likely separator: ',', ';', or '\t'
|
||||
"""
|
||||
# Read first 10KB to analyze
|
||||
sample = file_data[:10240].decode("utf-8", errors="ignore")
|
||||
lines = sample.split("\n")[:10] # First 10 lines
|
||||
|
||||
if not lines:
|
||||
return "," # Default to comma
|
||||
|
||||
# Count occurrences of each separator in the first few lines
|
||||
comma_count = sum(line.count(",") for line in lines)
|
||||
semicolon_count = sum(line.count(";") for line in lines)
|
||||
tab_count = sum(line.count("\t") for line in lines)
|
||||
|
||||
# Return the separator with the highest count
|
||||
if tab_count > comma_count and tab_count > semicolon_count:
|
||||
return "\t"
|
||||
elif semicolon_count > comma_count:
|
||||
return ";"
|
||||
else:
|
||||
return "," # Default to comma
|
||||
|
||||
|
||||
def _convert_to_serializable(obj: Any) -> Any:
|
||||
"""
|
||||
Convert pandas/numpy types to Python native types for JSON serialization.
|
||||
|
||||
Handles:
|
||||
- pandas DataFrame/Series
|
||||
- numpy scalars (int64, float64, etc.)
|
||||
- numpy arrays
|
||||
- pandas Timestamp
|
||||
- Other non-serializable types
|
||||
|
||||
Args:
|
||||
obj: The object to convert.
|
||||
|
||||
Returns:
|
||||
A JSON-serializable version of the object.
|
||||
"""
|
||||
|
||||
# Handle pandas DataFrame
|
||||
if isinstance(obj, pd.DataFrame):
|
||||
# Limit number of rows to avoid huge responses
|
||||
if len(obj) > 1000:
|
||||
obj = obj.head(1000)
|
||||
logger.warning("Result truncated to 1000 rows")
|
||||
return obj.to_dict(orient="records")
|
||||
|
||||
# Handle pandas Series
|
||||
if isinstance(obj, pd.Series):
|
||||
# Convert Series to dict, handling index
|
||||
result_dict = obj.to_dict()
|
||||
# Convert any numpy/pandas types in the values
|
||||
return {str(k): _convert_to_serializable(v) for k, v in result_dict.items()}
|
||||
|
||||
# Handle numpy scalars
|
||||
if isinstance(obj, (np.integer, np.floating)):
|
||||
return obj.item() # Convert to Python native int/float
|
||||
|
||||
# Handle numpy arrays
|
||||
if isinstance(obj, np.ndarray):
|
||||
return obj.tolist()
|
||||
|
||||
# Handle pandas Timestamp
|
||||
if isinstance(obj, pd.Timestamp):
|
||||
return obj.isoformat()
|
||||
|
||||
# Handle lists and tuples - recursively convert elements
|
||||
if isinstance(obj, (list, tuple)):
|
||||
return [_convert_to_serializable(item) for item in obj]
|
||||
|
||||
# Handle dicts - recursively convert values
|
||||
if isinstance(obj, dict):
|
||||
return {str(k): _convert_to_serializable(v) for k, v in obj.items()}
|
||||
|
||||
# Handle None, bool, int, float, str - these are already serializable
|
||||
if obj is None or isinstance(obj, (bool, int, float, str)):
|
||||
return obj
|
||||
|
||||
# Fallback: try to convert to string
|
||||
try:
|
||||
return str(obj)
|
||||
except Exception:
|
||||
logger.warning("Could not serialize object of type %s, returning None", type(obj))
|
||||
return None
|
||||
|
||||
|
||||
def _is_valid_excel_file(file_data: bytes, file_name: str) -> bool:
|
||||
"""
|
||||
Check if the file data appears to be a valid Excel file.
|
||||
|
||||
XLSX files are ZIP archives, so they should start with ZIP signature (PK\x03\x04).
|
||||
XLS files have a different signature.
|
||||
"""
|
||||
if not file_data:
|
||||
return False
|
||||
|
||||
file_lower = file_name.lower()
|
||||
|
||||
# Check for XLSX (ZIP-based) signature
|
||||
if file_lower.endswith((".xlsx", ".xlsm", ".xlsb")):
|
||||
# XLSX files are ZIP archives, should start with PK\x03\x04
|
||||
return file_data[:4] == b"PK\x03\x04"
|
||||
|
||||
# Check for XLS (OLE2) signature
|
||||
if file_lower.endswith(".xls"):
|
||||
# XLS files are OLE2 compound documents, should start with specific signature
|
||||
# Common signatures: 0xD0CF11E0 (OLE2) or 0x504B0304 (sometimes saved as ZIP)
|
||||
return (
|
||||
file_data[:4] == b"\xd0\xcf\x11\xe0" # OLE2 signature
|
||||
or file_data[:4] == b"PK\x03\x04" # Sometimes XLS are actually ZIP
|
||||
)
|
||||
|
||||
return False
|
||||
|
||||
|
||||
@sync_to_async
|
||||
def load_dataframe(file_data: bytes, content_type: str, file_name: str) -> Dict[str, pd.DataFrame]:
|
||||
"""
|
||||
Load tabular file into pandas DataFrames.
|
||||
|
||||
Returns a dictionary mapping sheet/table names to DataFrames.
|
||||
For CSV files, uses 'default' as the key.
|
||||
For Excel files, uses sheet names as keys.
|
||||
"""
|
||||
try:
|
||||
# Handle CSV files - also accept text/plain if file extension is .csv
|
||||
if content_type in ["text/csv", "application/csv"] or (
|
||||
content_type == "text/plain" and file_name.lower().endswith(".csv")
|
||||
):
|
||||
# Detect the separator
|
||||
separator = detect_csv_separator(file_data)
|
||||
logger.debug("Detected CSV separator: %r", separator)
|
||||
|
||||
# Read CSV with detected separator
|
||||
df = pd.read_csv(
|
||||
BytesIO(file_data),
|
||||
sep=separator,
|
||||
on_bad_lines="skip", # Skip problematic lines
|
||||
engine="python", # More flexible parser
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
if df.empty:
|
||||
raise ValueError("CSV file appears to be empty or could not be parsed")
|
||||
|
||||
return {"default": df}
|
||||
elif content_type in [
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
"application/vnd.ms-excel",
|
||||
"application/excel",
|
||||
] or file_name.lower().endswith((".xlsx", ".xls", ".xlsm", ".xlsb")):
|
||||
# Validate Excel file format before attempting to read
|
||||
if not _is_valid_excel_file(file_data, file_name):
|
||||
logger.warning(
|
||||
"File '%s' does not appear to be a valid Excel file. "
|
||||
"File size: %d bytes, First bytes: %r",
|
||||
file_name,
|
||||
len(file_data),
|
||||
file_data[:20] if len(file_data) >= 20 else file_data,
|
||||
)
|
||||
raise ValueError(
|
||||
f"File '{file_name}' does not appear to be a valid Excel file. "
|
||||
"It may be corrupted or in an unsupported format."
|
||||
)
|
||||
|
||||
file_lower = file_name.lower()
|
||||
dataframes = {}
|
||||
|
||||
# Try different engines based on file extension
|
||||
if file_lower.endswith(".xls"):
|
||||
# Old Excel format - try xlrd engine
|
||||
try:
|
||||
logger.debug("Attempting to read .xls file with xlrd engine")
|
||||
excel_file = pd.ExcelFile(BytesIO(file_data), engine="xlrd")
|
||||
dataframes = {
|
||||
sheet_name: excel_file.parse(sheet_name)
|
||||
for sheet_name in excel_file.sheet_names
|
||||
}
|
||||
except Exception as xlrd_error:
|
||||
logger.warning("Failed to read .xls with xlrd: %s", xlrd_error)
|
||||
# Fallback: try openpyxl (sometimes .xls files are actually .xlsx)
|
||||
try:
|
||||
logger.debug("Trying openpyxl as fallback for .xls file")
|
||||
excel_file = pd.ExcelFile(BytesIO(file_data), engine="openpyxl")
|
||||
dataframes = {
|
||||
sheet_name: excel_file.parse(sheet_name)
|
||||
for sheet_name in excel_file.sheet_names
|
||||
}
|
||||
except Exception as openpyxl_error:
|
||||
logger.error("Failed to read .xls with both engines: %s", openpyxl_error)
|
||||
raise ValueError(
|
||||
f"Failed to read Excel file '{file_name}': "
|
||||
f"xlrd error: {xlrd_error}, openpyxl error: {openpyxl_error}"
|
||||
) from openpyxl_error
|
||||
else:
|
||||
# XLSX format - try openpyxl first
|
||||
try:
|
||||
logger.debug("Attempting to read Excel file with openpyxl engine")
|
||||
excel_file = pd.ExcelFile(BytesIO(file_data), engine="openpyxl")
|
||||
dataframes = {
|
||||
sheet_name: excel_file.parse(sheet_name)
|
||||
for sheet_name in excel_file.sheet_names
|
||||
}
|
||||
except Exception as openpyxl_error:
|
||||
logger.warning("Failed to read with openpyxl: %s", openpyxl_error)
|
||||
# Try calamine engine if available (faster and more robust)
|
||||
try:
|
||||
logger.debug("Trying calamine engine as fallback")
|
||||
excel_file = pd.ExcelFile(BytesIO(file_data), engine="calamine")
|
||||
dataframes = {
|
||||
sheet_name: excel_file.parse(sheet_name)
|
||||
for sheet_name in excel_file.sheet_names
|
||||
}
|
||||
except ImportError:
|
||||
logger.debug("calamine engine not available")
|
||||
raise ValueError(
|
||||
f"Failed to read Excel file '{file_name}' with openpyxl: {openpyxl_error}. "
|
||||
"The file may be corrupted or in an unsupported format."
|
||||
) from openpyxl_error
|
||||
except Exception as calamine_error:
|
||||
logger.error("Failed to read with calamine: %s", calamine_error)
|
||||
raise ValueError(
|
||||
f"Failed to read Excel file '{file_name}': "
|
||||
f"openpyxl error: {openpyxl_error}, calamine error: {calamine_error}"
|
||||
) from calamine_error
|
||||
|
||||
if not dataframes:
|
||||
raise ValueError(f"Excel file '{file_name}' contains no readable sheets")
|
||||
|
||||
logger.info(
|
||||
"Successfully loaded Excel file '%s' with %d sheet(s): %s",
|
||||
file_name,
|
||||
len(dataframes),
|
||||
list(dataframes.keys()),
|
||||
)
|
||||
return dataframes
|
||||
else:
|
||||
raise ValueError(f"Unsupported content type: {content_type}")
|
||||
except Exception as e:
|
||||
logger.error("Error loading tabular file: %s", e, exc_info=True)
|
||||
raise ModelCannotRetry(f"Failed to load file: {str(e)}") from e
|
||||
|
||||
|
||||
def generate_metadata(dataframes: Dict[str, pd.DataFrame], file_name: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Generate metadata about the tabular file.
|
||||
|
||||
Returns:
|
||||
Dictionary containing:
|
||||
- sheets: List of sheet/table names
|
||||
- schemas: Dictionary mapping sheet names to their schemas
|
||||
- row_counts: Dictionary mapping sheet names to row counts
|
||||
- column_info: Dictionary mapping sheet names to column information
|
||||
"""
|
||||
metadata = {
|
||||
"file_name": file_name,
|
||||
"sheets": list(dataframes.keys()),
|
||||
"schemas": {},
|
||||
"row_counts": {},
|
||||
"column_info": {},
|
||||
}
|
||||
|
||||
for sheet_name, df in dataframes.items():
|
||||
# Schema: column names and types
|
||||
metadata["schemas"][sheet_name] = {
|
||||
col: str(dtype) for col, dtype in df.dtypes.items()
|
||||
}
|
||||
|
||||
# Row count
|
||||
metadata["row_counts"][sheet_name] = len(df)
|
||||
|
||||
# Column info: name, type, sample values, null counts
|
||||
metadata["column_info"][sheet_name] = {}
|
||||
for col in df.columns:
|
||||
col_info = {
|
||||
"type": str(df[col].dtype),
|
||||
"null_count": int(df[col].isna().sum()),
|
||||
"unique_count": int(df[col].nunique()),
|
||||
}
|
||||
# Add sample values (non-null)
|
||||
sample_values = df[col].dropna().head(5).tolist()
|
||||
if sample_values:
|
||||
col_info["sample_values"] = [str(v) for v in sample_values]
|
||||
metadata["column_info"][sheet_name][col] = col_info
|
||||
|
||||
return metadata
|
||||
|
||||
|
||||
async def generate_query(
|
||||
user_query: str, metadata: Dict[str, Any], query_agent: BaseAgent, ctx: RunContext
|
||||
) -> str:
|
||||
"""
|
||||
Use an LLM agent to generate a pandas query from user query and file metadata.
|
||||
"""
|
||||
metadata_str = json.dumps(metadata, indent=2)
|
||||
|
||||
prompt = f"""You are a data analysis assistant. Given a user query and file metadata, generate a Python pandas query to answer the question.
|
||||
|
||||
File metadata:
|
||||
{metadata_str}
|
||||
|
||||
User query: {user_query}
|
||||
|
||||
Generate a Python code snippet that:
|
||||
1. Uses pandas operations (filter, groupby, aggregate, etc.)
|
||||
2. Works with the dataframes loaded in memory (available as 'dataframes' dict)
|
||||
3. Assigns the final result to a variable named 'result'
|
||||
4. Handles the specific sheet/table if multiple sheets exist
|
||||
5. ALWAYS handles NaN/NA values in boolean conditions using .notna() or .fillna() before filtering
|
||||
6. If the user asks for a plot/graph/chart, create it using matplotlib and save to 'plot_image' variable as base64
|
||||
|
||||
IMPORTANT RULES:
|
||||
- The code MUST assign the final result to a variable named 'result'
|
||||
- When filtering with conditions, ALWAYS check for NaN first: df[df['col'].notna() & (df['col'] > value)]
|
||||
- Use .dropna() if you need to remove rows with missing values
|
||||
- Use .fillna() if you need to replace missing values
|
||||
- If plotting: use plt (already imported), create the plot, convert to base64:
|
||||
```python
|
||||
plt.figure(figsize=(10, 6))
|
||||
# ... your plot code ...
|
||||
buf = BytesIO()
|
||||
plt.savefig(buf, format='png')
|
||||
buf.seek(0)
|
||||
plot_image = base64.b64encode(buf.getvalue()).decode('utf-8')
|
||||
plt.close()
|
||||
```
|
||||
NOTE: Do NOT use import statements - plt, base64, BytesIO are already available.
|
||||
|
||||
Return ONLY the Python code, without markdown formatting or explanations. The code should be executable and use variables:
|
||||
- 'dataframes': dict mapping sheet names to DataFrames
|
||||
- Sheet names available: {', '.join(metadata['sheets'])}
|
||||
|
||||
Example format (without plot):
|
||||
df = dataframes['default']
|
||||
df = df[df['column'].notna()] # Remove NaN values first
|
||||
result = df[df['column'] > 100].groupby('category').sum()
|
||||
|
||||
Example format (with plot):
|
||||
df = dataframes['default']
|
||||
plt.figure(figsize=(10, 6))
|
||||
plt.plot(df.index, df['close'])
|
||||
plt.xlabel('Index')
|
||||
plt.ylabel('Close')
|
||||
plt.title('Close vs Index')
|
||||
buf = BytesIO()
|
||||
plt.savefig(buf, format='png')
|
||||
buf.seek(0)
|
||||
plot_image = base64.b64encode(buf.getvalue()).decode('utf-8')
|
||||
plt.close()
|
||||
result = "Plot generated successfully. The plot image has been saved and is available in the tool response."
|
||||
|
||||
IMPORTANT:
|
||||
- Do NOT use import statements in the code. All necessary modules (pd, plt, np, base64, BytesIO) are already available. Do NOT use anything else than these modules.
|
||||
- When returning the result text, mention that a plot was generated and will be available in the response, but do NOT include the URL in the text - the system will handle displaying it.
|
||||
|
||||
Generate the query code:"""
|
||||
|
||||
try:
|
||||
response = await query_agent.run(prompt, usage=ctx.usage)
|
||||
query_code = response.output.strip()
|
||||
|
||||
# Extract code from markdown code blocks if present
|
||||
if "```python" in query_code:
|
||||
query_code = query_code.split("```python")[1].split("```")[0].strip()
|
||||
elif "```" in query_code:
|
||||
query_code = query_code.split("```")[1].split("```")[0].strip()
|
||||
|
||||
return query_code
|
||||
except Exception as e:
|
||||
logger.error("Error generating query: %s", e, exc_info=True)
|
||||
raise ModelRetry("Failed to generate query. Please try rephrasing your question.") from e
|
||||
|
||||
|
||||
@sync_to_async
|
||||
def execute_query(query_code: str, dataframes: Dict[str, pd.DataFrame]) -> Any:
|
||||
"""
|
||||
Execute the generated pandas query safely.
|
||||
|
||||
Note: Uses exec() in a restricted environment. The query code is generated
|
||||
by an LLM based on file metadata, so it should be relatively safe, but
|
||||
we restrict the available builtins and globals.
|
||||
"""
|
||||
try:
|
||||
# Pre-process dataframes to handle common issues
|
||||
processed_dataframes = {}
|
||||
for name, df in dataframes.items():
|
||||
# Make a copy to avoid modifying original
|
||||
df_processed = df.copy()
|
||||
# Replace common NaN representations
|
||||
df_processed = df_processed.replace(["", " ", "nan", "NaN", "None", "null"], pd.NA)
|
||||
processed_dataframes[name] = df_processed
|
||||
|
||||
# Create a safe execution environment
|
||||
safe_globals = {
|
||||
"pd": pd,
|
||||
"plt": plt,
|
||||
"np": np,
|
||||
"base64": base64,
|
||||
"BytesIO": BytesIO,
|
||||
"dataframes": processed_dataframes,
|
||||
"__builtins__": {
|
||||
"len": len,
|
||||
"str": str,
|
||||
"int": int,
|
||||
"float": float,
|
||||
"bool": bool,
|
||||
"list": list,
|
||||
"dict": dict,
|
||||
"set": set,
|
||||
"tuple": tuple,
|
||||
"range": range,
|
||||
"sum": sum,
|
||||
"max": max,
|
||||
"min": min,
|
||||
"abs": abs,
|
||||
"round": round,
|
||||
},
|
||||
}
|
||||
|
||||
# Clean up query code - remove any import statements that might cause issues
|
||||
# Split by lines and filter out import statements
|
||||
lines = query_code.split("\n")
|
||||
cleaned_lines = [
|
||||
line
|
||||
for line in lines
|
||||
if not line.strip().startswith("import ") and not line.strip().startswith("from ")
|
||||
]
|
||||
query_code = "\n".join(cleaned_lines)
|
||||
|
||||
# Execute the query in a restricted namespace
|
||||
local_vars = {}
|
||||
exec(query_code, safe_globals, local_vars) # noqa: S102
|
||||
|
||||
# Get the result - check if 'result' variable exists, otherwise try 'df'
|
||||
if "result" in local_vars:
|
||||
result = local_vars["result"]
|
||||
elif "df" in local_vars:
|
||||
result = local_vars["df"]
|
||||
else:
|
||||
# If no explicit result variable, get the last expression
|
||||
# This is a fallback - ideally the LLM should assign to 'result'
|
||||
raise ValueError("Query must assign result to 'result' variable")
|
||||
|
||||
# Check if a plot was generated
|
||||
plot_image = None
|
||||
if "plot_image" in local_vars:
|
||||
plot_image = local_vars["plot_image"]
|
||||
logger.info("Plot image generated")
|
||||
|
||||
# Convert result to a serializable format
|
||||
result = _convert_to_serializable(result)
|
||||
|
||||
return {"result": result, "plot_image": plot_image}
|
||||
except Exception as e:
|
||||
logger.error("Error executing query: %s", e, exc_info=True)
|
||||
# Provide more helpful error message
|
||||
error_msg = str(e)
|
||||
if "NaN" in error_msg or "NA" in error_msg:
|
||||
error_msg = (
|
||||
f"{error_msg}. "
|
||||
"The query may need to handle missing values (NaN/NA) using .notna() or .dropna() before filtering."
|
||||
)
|
||||
raise ModelCannotRetry(f"Failed to execute query: {error_msg}") from e
|
||||
|
||||
|
||||
@last_model_retry_soft_fail
|
||||
async def data_analysis(ctx: RunContext, query: str) -> ToolReturn:
|
||||
"""
|
||||
Analyze tabular data files (CSV, Excel) based on user query.
|
||||
Can also generate plots/graphs/charts.
|
||||
|
||||
This tool:
|
||||
1. Loads the tabular file(s) from attachments
|
||||
2. Generates metadata about the file structure
|
||||
3. Uses an LLM to generate a pandas query based on user query
|
||||
4. Executes the query and returns results
|
||||
|
||||
Args:
|
||||
ctx (RunContext): The run context containing the conversation.
|
||||
query (str): The user's data analysis question.
|
||||
|
||||
Returns:
|
||||
ToolReturn: Contains the analysis results and metadata.
|
||||
"""
|
||||
try:
|
||||
# Find tabular files in attachments
|
||||
# First, get all attachments for debugging
|
||||
all_attachments = await sync_to_async(list)(
|
||||
ctx.deps.conversation.attachments.all()
|
||||
)
|
||||
logger.info(
|
||||
"All attachments in conversation: %s",
|
||||
[
|
||||
{
|
||||
"file_name": a.file_name,
|
||||
"content_type": a.content_type,
|
||||
"upload_state": a.upload_state,
|
||||
"conversion_from": a.conversion_from,
|
||||
}
|
||||
for a in all_attachments
|
||||
],
|
||||
)
|
||||
|
||||
# Find tabular files - exclude converted files (they have conversion_from set)
|
||||
# First try by content_type
|
||||
tabular_attachments_by_type = await sync_to_async(list)(
|
||||
ctx.deps.conversation.attachments.filter(
|
||||
content_type__in=TABULAR_MIME_TYPES,
|
||||
upload_state=AttachmentStatus.READY,
|
||||
)
|
||||
.filter(
|
||||
Q(conversion_from__isnull=True) | Q(conversion_from="")
|
||||
)
|
||||
)
|
||||
|
||||
# If no files found by content_type, try by file extension as fallback
|
||||
# (some systems detect CSV as text/plain instead of text/csv)
|
||||
if not tabular_attachments_by_type:
|
||||
csv_extensions = [".csv", ".xlsx", ".xls"]
|
||||
all_ready_attachments = await sync_to_async(list)(
|
||||
ctx.deps.conversation.attachments.filter(
|
||||
upload_state=AttachmentStatus.READY,
|
||||
)
|
||||
.filter(
|
||||
Q(conversion_from__isnull=True) | Q(conversion_from="")
|
||||
)
|
||||
)
|
||||
tabular_attachments = [
|
||||
att
|
||||
for att in all_ready_attachments
|
||||
if any(att.file_name.lower().endswith(ext) for ext in csv_extensions)
|
||||
# Exclude Markdown files (converted files have .md extension or content_type text/markdown)
|
||||
and not att.file_name.lower().endswith(".md")
|
||||
and att.content_type != "text/markdown"
|
||||
]
|
||||
if tabular_attachments:
|
||||
logger.info(
|
||||
"Found %d tabular file(s) by extension fallback (content_type was not recognized): %s",
|
||||
len(tabular_attachments),
|
||||
[f"{a.file_name} ({a.content_type})" for a in tabular_attachments],
|
||||
)
|
||||
else:
|
||||
tabular_attachments = tabular_attachments_by_type
|
||||
|
||||
# If still no files found, check if there are converted files that might have originals
|
||||
# This handles the case where an Excel file was converted to Markdown for RAG
|
||||
if not tabular_attachments:
|
||||
# Look for converted files with tabular extensions
|
||||
csv_extensions = [".csv", ".xlsx", ".xls"]
|
||||
converted_attachments = await sync_to_async(list)(
|
||||
ctx.deps.conversation.attachments.filter(
|
||||
upload_state=AttachmentStatus.READY,
|
||||
)
|
||||
.exclude(
|
||||
Q(conversion_from__isnull=True) | Q(conversion_from="")
|
||||
)
|
||||
)
|
||||
|
||||
# For each converted file, try to find the original
|
||||
for converted_att in converted_attachments:
|
||||
if any(converted_att.file_name.lower().endswith(ext) for ext in csv_extensions):
|
||||
# Try to find the original file using conversion_from key
|
||||
original_key = converted_att.conversion_from
|
||||
if original_key:
|
||||
original_attachment = await sync_to_async(
|
||||
ctx.deps.conversation.attachments.filter(
|
||||
key=original_key,
|
||||
upload_state=AttachmentStatus.READY,
|
||||
).first
|
||||
)()
|
||||
if original_attachment:
|
||||
logger.info(
|
||||
"Found original file '%s' for converted file '%s'",
|
||||
original_attachment.file_name,
|
||||
converted_att.file_name,
|
||||
)
|
||||
tabular_attachments.append(original_attachment)
|
||||
break
|
||||
|
||||
logger.info(
|
||||
"Found %d tabular attachment(s): %s",
|
||||
len(tabular_attachments),
|
||||
[f"{a.file_name} ({a.content_type})" for a in tabular_attachments],
|
||||
)
|
||||
|
||||
if not tabular_attachments:
|
||||
raise ModelCannotRetry(
|
||||
"No tabular files (CSV or Excel) found in the conversation. "
|
||||
"Please upload a CSV or Excel file first. "
|
||||
"Note: If you uploaded an Excel file that was converted to Markdown for RAG, "
|
||||
"the original file must still be available."
|
||||
)
|
||||
|
||||
# Use the first tabular file
|
||||
attachment = tabular_attachments[0]
|
||||
logger.info("Analyzing file: %s (type: %s)", attachment.file_name, attachment.content_type)
|
||||
|
||||
# Load file data
|
||||
file_data = await read_tabular_file(attachment)
|
||||
|
||||
# Validate that this is actually a valid Excel/CSV file (not a converted Markdown file)
|
||||
# Check if it's an Excel file that should have ZIP signature
|
||||
if attachment.file_name.lower().endswith((".xlsx", ".xls", ".xlsm", ".xlsb")):
|
||||
if not _is_valid_excel_file(file_data, attachment.file_name):
|
||||
logger.warning(
|
||||
"File '%s' does not appear to be a valid Excel file. "
|
||||
"It may be a converted Markdown file. Searching for original...",
|
||||
attachment.file_name,
|
||||
)
|
||||
# Try to find the original file
|
||||
# Look for an attachment with the same name but without conversion_from
|
||||
original_attachment = await sync_to_async(
|
||||
ctx.deps.conversation.attachments.filter(
|
||||
file_name=attachment.file_name,
|
||||
upload_state=AttachmentStatus.READY,
|
||||
)
|
||||
.filter(
|
||||
Q(conversion_from__isnull=True) | Q(conversion_from="")
|
||||
)
|
||||
.exclude(pk=attachment.pk)
|
||||
.first
|
||||
)()
|
||||
|
||||
if original_attachment:
|
||||
logger.info(
|
||||
"Found original file '%s' (key: %s), using it instead",
|
||||
original_attachment.file_name,
|
||||
original_attachment.key,
|
||||
)
|
||||
attachment = original_attachment
|
||||
file_data = await read_tabular_file(attachment)
|
||||
elif hasattr(attachment, 'conversion_from') and attachment.conversion_from:
|
||||
# Try to find by key if this file has a conversion_from
|
||||
original_attachment = await sync_to_async(
|
||||
ctx.deps.conversation.attachments.filter(
|
||||
key=attachment.conversion_from,
|
||||
upload_state=AttachmentStatus.READY,
|
||||
).first
|
||||
)()
|
||||
if original_attachment:
|
||||
logger.info(
|
||||
"Found original file via conversion_from: '%s'",
|
||||
original_attachment.file_name,
|
||||
)
|
||||
attachment = original_attachment
|
||||
file_data = await read_tabular_file(attachment)
|
||||
else:
|
||||
raise ModelCannotRetry(
|
||||
f"File '{attachment.file_name}' appears to be a converted Markdown file, "
|
||||
"not the original Excel file. The original file is not available. "
|
||||
"Please re-upload the original Excel file."
|
||||
)
|
||||
else:
|
||||
raise ModelCannotRetry(
|
||||
f"File '{attachment.file_name}' does not appear to be a valid Excel file. "
|
||||
"It may be corrupted or in an unsupported format."
|
||||
)
|
||||
|
||||
# Load into pandas DataFrames
|
||||
dataframes = await load_dataframe(file_data, attachment.content_type, attachment.file_name)
|
||||
|
||||
# Generate metadata
|
||||
metadata = generate_metadata(dataframes, attachment.file_name)
|
||||
logger.debug("File metadata: %s", json.dumps(metadata, indent=2))
|
||||
|
||||
# Generate query using LLM
|
||||
# NOTE:
|
||||
# We intentionally create a "bare" Agent instance here instead of using BaseAgent
|
||||
# with tools enabled. Using BaseAgent would attach all configured tools (including
|
||||
# this data_analysis tool itself), which can cause the model to try to call tools
|
||||
# while we're already inside a tool execution, leading to nested tool calls and
|
||||
# failures like "Failed to generate query. Please try rephrasing your question.".
|
||||
#
|
||||
# Here we reuse the same model configuration as BaseAgent but WITHOUT any tools,
|
||||
# so this internal call is purely text-to-text.
|
||||
llm_config = settings.LLM_CONFIGURATIONS[settings.LLM_DEFAULT_MODEL_HRID]
|
||||
if llm_config.is_custom:
|
||||
model_instance = prepare_custom_model(llm_config)
|
||||
else:
|
||||
# Rely on pydantic-ai's built-in model registry / name inference
|
||||
model_instance = llm_config.model_name
|
||||
|
||||
# Use the same keyword as when using BaseAgent, which forwards to Agent.
|
||||
# On the current pydantic_ai version, the correct kwarg is `output_type`,
|
||||
# not `result_type` (passing `result_type` raises a UserError).
|
||||
query_agent = Agent(model=model_instance, output_type=str)
|
||||
query_code = await generate_query(query, metadata, query_agent, ctx)
|
||||
logger.debug("Generated query: %s", query_code)
|
||||
|
||||
# Execute query
|
||||
try:
|
||||
execution_result = await execute_query(query_code, dataframes)
|
||||
result = execution_result.get("result")
|
||||
plot_image_base64 = execution_result.get("plot_image")
|
||||
except Exception as e:
|
||||
logger.error("Query execution failed: %s", e, exc_info=True)
|
||||
raise ModelRetry(
|
||||
f"Failed to execute the generated query: {str(e)}. "
|
||||
"Please try rephrasing your question."
|
||||
) from e
|
||||
|
||||
# Format result for return
|
||||
return_value = {
|
||||
"query": query,
|
||||
"query_code": query_code,
|
||||
"result": result,
|
||||
"metadata": metadata,
|
||||
}
|
||||
|
||||
# Save plot image to storage if generated
|
||||
plot_url = None
|
||||
plot_attachment = None
|
||||
if plot_image_base64:
|
||||
try:
|
||||
# Decode base64 image
|
||||
plot_image_data = base64.b64decode(plot_image_base64)
|
||||
|
||||
# Generate a unique filename for the plot
|
||||
plot_filename = f"plot_{uuid.uuid4().hex[:8]}.png"
|
||||
plot_key = f"{ctx.deps.conversation.pk}/plots/{plot_filename}"
|
||||
|
||||
# Save to storage
|
||||
await sync_to_async(default_storage.save)(
|
||||
plot_key, BytesIO(plot_image_data)
|
||||
)
|
||||
|
||||
# Create a permanent attachment record in the database
|
||||
plot_attachment = await sync_to_async(ChatConversationAttachment.objects.create)(
|
||||
conversation=ctx.deps.conversation,
|
||||
uploaded_by=ctx.deps.user,
|
||||
key=plot_key,
|
||||
file_name=plot_filename,
|
||||
content_type="image/png",
|
||||
upload_state=AttachmentStatus.READY,
|
||||
size=len(plot_image_data),
|
||||
)
|
||||
|
||||
# Generate presigned URL for immediate access (valid for 1 hour)
|
||||
plot_url = await sync_to_async(generate_retrieve_policy)(plot_key)
|
||||
logger.info(
|
||||
"Plot image saved to storage and database: %s (presigned URL: %s)",
|
||||
plot_key,
|
||||
plot_url[:50] + "..."
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error("Failed to save plot image: %s", e, exc_info=True)
|
||||
# Continue without plot URL if save fails
|
||||
|
||||
if plot_url:
|
||||
# Include both local and presigned URLs
|
||||
return_value["plot_url"] = plot_url # Presigned URL for direct access
|
||||
return_value["plot_local_url"] = f"/media-key/{plot_key}" # Local URL for reference
|
||||
# Include attachment ID for reference
|
||||
if plot_attachment:
|
||||
return_value["plot_attachment_id"] = str(plot_attachment.pk)
|
||||
|
||||
return ToolReturn(
|
||||
return_value=return_value,
|
||||
metadata={"file_name": attachment.file_name, "content_type": attachment.content_type},
|
||||
)
|
||||
|
||||
except (ModelCannotRetry, ModelRetry):
|
||||
# Re-raise these as-is
|
||||
raise
|
||||
except Exception as exc:
|
||||
# Unexpected error - stop and inform user
|
||||
logger.exception("Unexpected error in data_analysis: %s", exc)
|
||||
raise ModelCannotRetry(
|
||||
f"An unexpected error occurred during data analysis: {type(exc).__name__}. "
|
||||
"You must explain this to the user and not try to answer based on your knowledge."
|
||||
) from exc
|
||||
|
||||
|
||||
def add_data_analysis_tool(agent: Agent) -> None:
|
||||
"""Add the data analysis tool to an existing agent."""
|
||||
|
||||
@agent.tool(retries=2)
|
||||
@functools.wraps(data_analysis)
|
||||
async def data_analysis_tool(ctx: RunContext, query: str) -> ToolReturn:
|
||||
"""
|
||||
Analyze tabular data files (CSV, Excel) based on user query.
|
||||
|
||||
This tool loads tabular files, generates metadata about their structure,
|
||||
uses an LLM to generate a pandas query based on the user's question,
|
||||
executes the query, and returns the results.
|
||||
|
||||
Use this tool when the user asks questions about data in CSV or Excel files,
|
||||
such as:
|
||||
- "What is the average sales by region?"
|
||||
- "Show me the top 10 products by revenue"
|
||||
- "How many records are in this file?"
|
||||
- "Filter data where column X is greater than Y"
|
||||
|
||||
Args:
|
||||
ctx (RunContext): The run context containing the conversation.
|
||||
query (str): The user's data analysis question.
|
||||
"""
|
||||
# Import here to avoid circular import
|
||||
from chat.tools.data_analysis import data_analysis as _data_analysis
|
||||
|
||||
return await _data_analysis(ctx, query)
|
||||
|
||||
@agent.instructions
|
||||
def data_analysis_instructions() -> str:
|
||||
"""Dynamic system prompt function to add data analysis instructions."""
|
||||
return (
|
||||
"When the user asks questions about data in CSV or Excel files, "
|
||||
"use the data_analysis tool to analyze the data and answer their question. "
|
||||
"The tool will handle loading the file, generating queries, and executing them. "
|
||||
"When a plot is generated, the tool returns a 'plot_url' in the result. "
|
||||
"Use this presigned URL directly in markdown image syntax: . "
|
||||
"Do NOT use local URLs like /media-key/... - always use the presigned URL from plot_url. "
|
||||
"Present the results clearly to the user."
|
||||
)
|
||||
|
||||
@@ -26,7 +26,7 @@ def add_document_rag_search_tool(agent: Agent) -> None:
|
||||
|
||||
document_store = document_store_backend(ctx.deps.conversation.collection_id)
|
||||
|
||||
rag_results = document_store.search(query)
|
||||
rag_results = document_store.search(query, session=ctx.deps.session)
|
||||
|
||||
ctx.usage += RunUsage(
|
||||
input_tokens=rag_results.usage.prompt_tokens,
|
||||
@@ -39,7 +39,7 @@ def add_document_rag_search_tool(agent: Agent) -> None:
|
||||
metadata={"sources": {result.url for result in rag_results.data}},
|
||||
)
|
||||
|
||||
@agent.instructions
|
||||
@agent.system_prompt
|
||||
def document_rag_instructions() -> str:
|
||||
"""Dynamic system prompt function to add RAG instructions if any."""
|
||||
return (
|
||||
|
||||
@@ -3,6 +3,17 @@
|
||||
from pydantic_ai import ModelRetry
|
||||
|
||||
|
||||
class ModelRetryLast(ModelRetry):
|
||||
"""
|
||||
Same as ModelRetry but also holds the last retry message to return when all attempts failed.
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, last_retry_message: str):
|
||||
"""Initialize ModelRetryLast with message and last retry message."""
|
||||
self.last_retry_message = last_retry_message
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class ModelCannotRetry(ModelRetry):
|
||||
"""
|
||||
Exception to raise when a tool function cannot be retried.
|
||||
|
||||
@@ -7,11 +7,13 @@ from uuid import uuid4
|
||||
from django.conf import settings
|
||||
from django.core.files.storage import default_storage
|
||||
from django.http import Http404, StreamingHttpResponse
|
||||
from django.utils.decorators import method_decorator
|
||||
|
||||
import langfuse
|
||||
import magic
|
||||
import posthog
|
||||
from lasuite.malware_detection import malware_detection
|
||||
from lasuite.oidc_login.decorators import refresh_oidc_access_token
|
||||
from rest_framework import decorators, filters, mixins, permissions, status, viewsets
|
||||
from rest_framework.exceptions import MethodNotAllowed, PermissionDenied, ValidationError
|
||||
from rest_framework.response import Response
|
||||
@@ -122,6 +124,7 @@ class ChatViewSet( # pylint: disable=too-many-ancestors, abstract-method
|
||||
self.permission_classes = []
|
||||
return super().get_permissions()
|
||||
|
||||
@method_decorator(refresh_oidc_access_token)
|
||||
@decorators.action(
|
||||
methods=["post"],
|
||||
detail=True,
|
||||
@@ -173,6 +176,7 @@ class ChatViewSet( # pylint: disable=too-many-ancestors, abstract-method
|
||||
ai_service = AIAgentService(
|
||||
conversation=conversation,
|
||||
user=self.request.user,
|
||||
session=request.session,
|
||||
model_hrid=model_hrid,
|
||||
language=(
|
||||
self.request.user.language
|
||||
@@ -221,7 +225,7 @@ class ChatViewSet( # pylint: disable=too-many-ancestors, abstract-method
|
||||
url_path="stop-streaming",
|
||||
url_name="stop-streaming",
|
||||
)
|
||||
def post_stop_streaming(self, request, pk): # pylint: disable=unused-argument
|
||||
def post_stop_steaming(self, request, pk): # pylint: disable=unused-argument
|
||||
"""Handle POST requests to stop streaming the chat conversation.
|
||||
|
||||
This action will put a poison pill in the redis cache to stop any ongoing streaming.
|
||||
|
||||
@@ -22,7 +22,7 @@ def no_http_requests(monkeypatch):
|
||||
Credits: https://blog.jerrycodes.com/no-http-requests/
|
||||
"""
|
||||
|
||||
allowed_hosts = {"localhost", "minio", "minio:9000"}
|
||||
allowed_hosts = {"localhost", "127.0.0.1", "minio", "minio:9000"}
|
||||
original_urlopen = HTTPConnectionPool.urlopen
|
||||
|
||||
def urlopen_mock(self, method, url, *args, **kwargs):
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
{
|
||||
"models": [
|
||||
{
|
||||
"hrid": "default-model",
|
||||
"model_name": "settings.AI_MODEL",
|
||||
"human_readable_name": "Default Model",
|
||||
"provider_name": "default-provider",
|
||||
"profile": null,
|
||||
"settings": {},
|
||||
"is_active": true,
|
||||
"icon": [
|
||||
"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAMAAABF0y+mAAAAn1BMVEUALosAKoovTZjw8vb////+9/jlPUniAAz",
|
||||
"iABUAGIWbpsTwq7HhAAAAI4dle7DrdX4AJohRaaboXWj7+/zn6On5//9NZaT29vfoWmVHYKDoUl/k5OUAIYddc6vpbHYCM47Y3+v53+LiFCUA",
|
||||
"HIWnsckYPJHi6PL77O7jJjW3wdf1w8jre4QgQ5TZ2txwg7Pr3+I8WZ6OnsTuoamClL7tlZ5xz5y8AAAAzUlEQVR4AZ3RRQKDQBBEUSTu7h5c4",
|
||||
"vc/W6Yp3KG2Dz4ynDdeEBvOmq12xx2E1u0B+4NOEocj4DgNJ1PgLAvni8WyBq5Yc71ubFJx23C2q4P7dRYejg1xzvCUgvz5guz11k7gXYKF/1",
|
||||
"8oyiYuvHAYeVkhXCzolVStHcGDjiQzNmMQxsMI5rEJRdQSPZvbpE2E8aY6gC6Z+2Hg4dFA0Yb4YedNL/v4Fk8WJuwiGhrChJNXI210rnib9Fs",
|
||||
"JlXRUC/HwTscPIXf/iklq/tjb/gHAdxkCUjAg2QAAAABJRU5ErkJggg=="
|
||||
],
|
||||
"system_prompt": "settings.AI_AGENT_INSTRUCTIONS",
|
||||
"tools": "settings.AI_AGENT_TOOLS"
|
||||
},
|
||||
{
|
||||
"hrid": "default-summarization-model",
|
||||
"model_name": "settings.AI_MODEL",
|
||||
"human_readable_name": "Default Summarization Model",
|
||||
"provider_name": "default-provider",
|
||||
"profile": null,
|
||||
"settings": {},
|
||||
"is_active": true,
|
||||
"icon": null,
|
||||
"system_prompt": "settings.SUMMARIZATION_SYSTEM_PROMPT",
|
||||
"tools": []
|
||||
}
|
||||
],
|
||||
"providers": [
|
||||
{
|
||||
"hrid": "default-provider",
|
||||
"base_url": "settings.AI_BASE_URL",
|
||||
"api_key": "settings.AI_API_KEY",
|
||||
"kind": "mistral"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -713,7 +713,7 @@ class Base(BraveSettings, Configuration):
|
||||
environ_prefix=None,
|
||||
)
|
||||
RAG_DOCUMENT_SEARCH_BACKEND = values.Value(
|
||||
"chat.agent_rag.document_rag_backends.albert_rag_backend.AlbertRagBackend",
|
||||
"chat.agent_rag.document_rag_backends.find_rag_backend.FindRagBackend",
|
||||
environ_name="RAG_DOCUMENT_SEARCH_BACKEND",
|
||||
environ_prefix=None,
|
||||
)
|
||||
@@ -841,6 +841,16 @@ USER QUESTION:
|
||||
environ_prefix=None,
|
||||
)
|
||||
|
||||
# Find
|
||||
FIND_API_KEY = values.Value(
|
||||
environ_name="FIND_API_KEY",
|
||||
environ_prefix=None,
|
||||
)
|
||||
FIND_API_URL = values.Value(
|
||||
environ_name="FIND_API_URL",
|
||||
environ_prefix=None,
|
||||
)
|
||||
|
||||
# Logging
|
||||
# We want to make it easy to log to console but by default we log production
|
||||
# to Sentry and don't want to log to console.
|
||||
|
||||
@@ -1,9 +1,12 @@
|
||||
"""Conversations core API endpoints"""
|
||||
|
||||
from django.conf import settings
|
||||
from django.core.exceptions import ValidationError
|
||||
|
||||
from rest_framework import exceptions as drf_exceptions
|
||||
from rest_framework import views as drf_views
|
||||
from rest_framework.decorators import api_view
|
||||
from rest_framework.response import Response
|
||||
|
||||
|
||||
def exception_handler(exc, context):
|
||||
@@ -25,3 +28,14 @@ def exception_handler(exc, context):
|
||||
exc = drf_exceptions.ValidationError(detail=detail)
|
||||
|
||||
return drf_views.exception_handler(exc, context)
|
||||
|
||||
|
||||
# pylint: disable=unused-argument
|
||||
@api_view(["GET"])
|
||||
def get_frontend_configuration(request):
|
||||
"""Returns the frontend configuration dict as configured in settings."""
|
||||
frontend_configuration = {
|
||||
"LANGUAGE_CODE": settings.LANGUAGE_CODE,
|
||||
}
|
||||
frontend_configuration.update(settings.FRONTEND_CONFIGURATION)
|
||||
return Response(frontend_configuration)
|
||||
|
||||
@@ -20,3 +20,23 @@ class UserSerializer(serializers.ModelSerializer):
|
||||
"sub",
|
||||
]
|
||||
read_only_fields = ["id", "email", "full_name", "short_name", "sub"]
|
||||
|
||||
|
||||
class UserLightSerializer(UserSerializer):
|
||||
"""Serialize users with limited fields."""
|
||||
|
||||
id = serializers.SerializerMethodField(read_only=True)
|
||||
email = serializers.SerializerMethodField(read_only=True)
|
||||
|
||||
def get_id(self, _user):
|
||||
"""Return always None. Here to have the same fields than in UserSerializer."""
|
||||
return None
|
||||
|
||||
def get_email(self, _user):
|
||||
"""Return always None. Here to have the same fields than in UserSerializer."""
|
||||
return None
|
||||
|
||||
class Meta:
|
||||
model = models.User
|
||||
fields = ["id", "email", "full_name", "short_name"]
|
||||
read_only_fields = ["id", "email", "full_name", "short_name"]
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
"""Custom authentication classes for the Conversations core app"""
|
||||
|
||||
from django.conf import settings
|
||||
|
||||
from rest_framework.authentication import BaseAuthentication
|
||||
from rest_framework.exceptions import AuthenticationFailed
|
||||
|
||||
|
||||
class ServerToServerAuthentication(BaseAuthentication):
|
||||
"""
|
||||
Custom authentication class for server-to-server requests.
|
||||
Validates the presence and correctness of the Authorization header.
|
||||
"""
|
||||
|
||||
AUTH_HEADER = "Authorization"
|
||||
TOKEN_TYPE = "Bearer" # noqa S105
|
||||
|
||||
def authenticate(self, request):
|
||||
"""
|
||||
Authenticate the server-to-server request by validating the Authorization header.
|
||||
|
||||
This method checks if the Authorization header is present in the request, ensures it
|
||||
contains a valid token with the correct format, and verifies the token against the
|
||||
list of allowed server-to-server tokens. If the header is missing, improperly formatted,
|
||||
or contains an invalid token, an AuthenticationFailed exception is raised.
|
||||
|
||||
Returns:
|
||||
None: If authentication is successful
|
||||
(no user is authenticated for server-to-server requests).
|
||||
|
||||
Raises:
|
||||
AuthenticationFailed: If the Authorization header is missing, malformed,
|
||||
or contains an invalid token.
|
||||
"""
|
||||
auth_header = request.headers.get(self.AUTH_HEADER)
|
||||
if not auth_header:
|
||||
raise AuthenticationFailed("Authorization header is missing.")
|
||||
|
||||
# Validate token format and existence
|
||||
auth_parts = auth_header.split(" ")
|
||||
if len(auth_parts) != 2 or auth_parts[0] != self.TOKEN_TYPE:
|
||||
raise AuthenticationFailed("Invalid authorization header.")
|
||||
|
||||
token = auth_parts[1]
|
||||
if token not in settings.SERVER_TO_SERVER_API_TOKENS:
|
||||
raise AuthenticationFailed("Invalid server-to-server token.")
|
||||
|
||||
# Authentication is successful, but no user is authenticated
|
||||
|
||||
def authenticate_header(self, request):
|
||||
"""Return the WWW-Authenticate header value."""
|
||||
return f"{self.TOKEN_TYPE} realm='Create document server to server'"
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
"""A JSONField for DRF to handle serialization/deserialization."""
|
||||
|
||||
import json
|
||||
|
||||
from rest_framework import serializers
|
||||
|
||||
|
||||
class JSONField(serializers.Field):
|
||||
"""
|
||||
A custom field for handling JSON data.
|
||||
"""
|
||||
|
||||
def to_representation(self, value):
|
||||
"""
|
||||
Convert the JSON string to a Python dictionary for serialization.
|
||||
"""
|
||||
return value
|
||||
|
||||
def to_internal_value(self, data):
|
||||
"""
|
||||
Convert the Python dictionary to a JSON string for deserialization.
|
||||
"""
|
||||
if data is None:
|
||||
return None
|
||||
return json.dumps(data)
|
||||
@@ -2,9 +2,31 @@
|
||||
|
||||
import unicodedata
|
||||
|
||||
import django_filters
|
||||
|
||||
|
||||
def remove_accents(value):
|
||||
"""Remove accents from a string (vélo -> velo)."""
|
||||
return "".join(
|
||||
c for c in unicodedata.normalize("NFD", value) if unicodedata.category(c) != "Mn"
|
||||
)
|
||||
|
||||
|
||||
class AccentInsensitiveCharFilter(django_filters.CharFilter):
|
||||
"""
|
||||
A custom CharFilter that filters on the accent-insensitive value searched.
|
||||
"""
|
||||
|
||||
def filter(self, qs, value):
|
||||
"""
|
||||
Apply the filter to the queryset using the unaccented version of the field.
|
||||
|
||||
Args:
|
||||
qs: The queryset to filter.
|
||||
value: The value to search for in the unaccented field.
|
||||
Returns:
|
||||
A filtered queryset.
|
||||
"""
|
||||
if value:
|
||||
value = remove_accents(value)
|
||||
return super().filter(qs, value)
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<title>Generate Document</title>
|
||||
</head>
|
||||
<body>
|
||||
<h2>Generate Document</h2>
|
||||
<form method="post" enctype="multipart/form-data">
|
||||
{% csrf_token %}
|
||||
{{ form.as_p }}
|
||||
<button type="submit">Generate PDF</button>
|
||||
</form>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,58 @@
|
||||
"""Custom template tags for the core application of People."""
|
||||
|
||||
import base64
|
||||
|
||||
from django import template
|
||||
from django.contrib.staticfiles import finders
|
||||
|
||||
from PIL import ImageFile as PillowImageFile
|
||||
|
||||
register = template.Library()
|
||||
|
||||
|
||||
def image_to_base64(file_or_path, close=False):
|
||||
"""
|
||||
Return the src string of the base64 encoding of an image represented by its path
|
||||
or file opened or not.
|
||||
|
||||
Inspired by Django's "get_image_dimensions"
|
||||
"""
|
||||
pil_parser = PillowImageFile.Parser()
|
||||
if hasattr(file_or_path, "read"):
|
||||
file = file_or_path
|
||||
if file.closed and hasattr(file, "open"):
|
||||
file_or_path.open()
|
||||
file_pos = file.tell()
|
||||
file.seek(0)
|
||||
else:
|
||||
try:
|
||||
# pylint: disable=consider-using-with
|
||||
file = open(file_or_path, "rb")
|
||||
except OSError:
|
||||
return ""
|
||||
close = True
|
||||
|
||||
try:
|
||||
image_data = file.read()
|
||||
if not image_data:
|
||||
return ""
|
||||
pil_parser.feed(image_data)
|
||||
if pil_parser.image:
|
||||
mime_type = pil_parser.image.get_format_mimetype()
|
||||
encoded_string = base64.b64encode(image_data)
|
||||
return f"data:{mime_type:s};base64, {encoded_string.decode('utf-8'):s}"
|
||||
return ""
|
||||
finally:
|
||||
if close:
|
||||
file.close()
|
||||
else:
|
||||
file.seek(file_pos)
|
||||
|
||||
|
||||
@register.simple_tag
|
||||
def base64_static(path):
|
||||
"""Return a static file into a base64."""
|
||||
full_path = finders.find(path)
|
||||
if full_path:
|
||||
return image_to_base64(full_path, True)
|
||||
return ""
|
||||
@@ -3,7 +3,7 @@ msgstr ""
|
||||
"Project-Id-Version: la-suite-conversations\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2025-10-20 21:48+0000\n"
|
||||
"PO-Revision-Date: 2025-12-15 13:49\n"
|
||||
"PO-Revision-Date: 2025-11-17 16:37\n"
|
||||
"Last-Translator: \n"
|
||||
"Language-Team: German\n"
|
||||
"Language: de_DE\n"
|
||||
|
||||
@@ -3,7 +3,7 @@ msgstr ""
|
||||
"Project-Id-Version: la-suite-conversations\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2025-10-20 21:48+0000\n"
|
||||
"PO-Revision-Date: 2025-12-15 13:49\n"
|
||||
"PO-Revision-Date: 2025-11-17 16:37\n"
|
||||
"Last-Translator: \n"
|
||||
"Language-Team: English\n"
|
||||
"Language: en_US\n"
|
||||
|
||||
@@ -3,7 +3,7 @@ msgstr ""
|
||||
"Project-Id-Version: la-suite-conversations\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2025-10-20 21:48+0000\n"
|
||||
"PO-Revision-Date: 2025-12-15 13:49\n"
|
||||
"PO-Revision-Date: 2025-11-17 16:37\n"
|
||||
"Last-Translator: \n"
|
||||
"Language-Team: French\n"
|
||||
"Language: fr_FR\n"
|
||||
|
||||
@@ -3,7 +3,7 @@ msgstr ""
|
||||
"Project-Id-Version: la-suite-conversations\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2025-10-20 21:48+0000\n"
|
||||
"PO-Revision-Date: 2025-12-15 13:49\n"
|
||||
"PO-Revision-Date: 2025-11-17 16:37\n"
|
||||
"Last-Translator: \n"
|
||||
"Language-Team: Dutch\n"
|
||||
"Language: nl_NL\n"
|
||||
|
||||
@@ -3,7 +3,7 @@ msgstr ""
|
||||
"Project-Id-Version: la-suite-conversations\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2025-10-20 21:48+0000\n"
|
||||
"PO-Revision-Date: 2025-12-15 13:49\n"
|
||||
"PO-Revision-Date: 2025-11-17 16:37\n"
|
||||
"Last-Translator: \n"
|
||||
"Language-Team: Russian\n"
|
||||
"Language: ru_RU\n"
|
||||
|
||||
@@ -3,7 +3,7 @@ msgstr ""
|
||||
"Project-Id-Version: la-suite-conversations\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2025-10-20 21:48+0000\n"
|
||||
"PO-Revision-Date: 2025-12-15 13:49\n"
|
||||
"PO-Revision-Date: 2025-11-17 16:37\n"
|
||||
"Last-Translator: \n"
|
||||
"Language-Team: Ukrainian\n"
|
||||
"Language: uk_UA\n"
|
||||
|
||||
@@ -7,7 +7,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "conversations"
|
||||
version = "0.0.10"
|
||||
version = "0.0.9"
|
||||
authors = [{ "name" = "DINUM", "email" = "dev@mail.numerique.gouv.fr" }]
|
||||
classifiers = [
|
||||
"Development Status :: 5 - Production/Stable",
|
||||
@@ -39,7 +39,7 @@ dependencies = [
|
||||
"django-redis==6.0.0",
|
||||
"django-storages[s3]==1.14.6",
|
||||
"django-timezone-field>=5.1",
|
||||
"django==5.2.9",
|
||||
"django==5.2.8",
|
||||
"djangorestframework==3.16.1",
|
||||
"drf_spectacular==0.29.0",
|
||||
"dockerflow==2024.4.2",
|
||||
@@ -53,10 +53,6 @@ dependencies = [
|
||||
"markitdown==0.0.2",
|
||||
"mozilla-django-oidc==4.0.1",
|
||||
"nested-multipart-parser==1.6.0",
|
||||
"matplotlib==3.9.2",
|
||||
"numpy==2.1.3",
|
||||
"openpyxl==3.1.5",
|
||||
"pandas==2.2.3",
|
||||
"posthog==7.0.0",
|
||||
"pydantic==2.12.4",
|
||||
"pydantic-ai-slim[openai,mistral,mcp,evals,logfire]==1.17.0",
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
"""Utility functions for OIDC token management."""
|
||||
|
||||
from django.conf import settings
|
||||
|
||||
import requests
|
||||
from lasuite.oidc_login.backends import get_oidc_refresh_token, store_tokens
|
||||
|
||||
|
||||
def refresh_access_token(session):
|
||||
"""Refresh the OIDC access token using the refresh token."""
|
||||
response = requests.post(
|
||||
settings.OIDC_OP_TOKEN_ENDPOINT,
|
||||
data={
|
||||
"grant_type": "refresh_token",
|
||||
"client_id": settings.OIDC_RP_CLIENT_ID,
|
||||
"client_secret": settings.OIDC_RP_CLIENT_SECRET,
|
||||
"refresh_token": get_oidc_refresh_token(session),
|
||||
},
|
||||
timeout=5,
|
||||
)
|
||||
response.raise_for_status()
|
||||
token_info = response.json()
|
||||
|
||||
store_tokens(
|
||||
session,
|
||||
access_token=token_info.get("access_token"),
|
||||
id_token=None,
|
||||
refresh_token=token_info.get("refresh_token"),
|
||||
)
|
||||
return session
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "app-conversations",
|
||||
"version": "0.0.10",
|
||||
"version": "0.0.9",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"dev": "next dev",
|
||||
@@ -39,11 +39,11 @@
|
||||
"lottie-react": "^2.4.1",
|
||||
"luxon": "3.6.1",
|
||||
"micromark-extension-llm-math": "3.1.1-20250610",
|
||||
"next": "15.3.8",
|
||||
"next": "15.3.3",
|
||||
"posthog-js": "1.249.3",
|
||||
"react": "19.2.1",
|
||||
"react": "*",
|
||||
"react-aria-components": "1.9.0",
|
||||
"react-dom": "19.2.1",
|
||||
"react-dom": "18.3.1",
|
||||
"react-i18next": "15.5.2",
|
||||
"react-intersection-observer": "9.16.0",
|
||||
"react-markdown": "10.1.0",
|
||||
|
||||
@@ -55,7 +55,7 @@ export const AttachmentList = ({
|
||||
>
|
||||
<Box
|
||||
$background="var(--c--theme--colors--greyscale-050)"
|
||||
$width="200px"
|
||||
$minWidth="200px"
|
||||
$direction="row"
|
||||
$gap="8px"
|
||||
$align="center"
|
||||
|
||||
@@ -104,14 +104,19 @@ export const Chat = ({
|
||||
|
||||
if (modelToSelect) {
|
||||
setSelectedModel(modelToSelect);
|
||||
setSelectedModelHrid(modelToSelect.hrid);
|
||||
}
|
||||
}
|
||||
}, [llmConfig, selectedModel, selectedModelHrid, setSelectedModelHrid]);
|
||||
}, [llmConfig, selectedModel, selectedModelHrid]);
|
||||
|
||||
// Update store when model selection changes
|
||||
useEffect(() => {
|
||||
if (selectedModel?.hrid !== selectedModelHrid) {
|
||||
setSelectedModelHrid(selectedModel?.hrid || null);
|
||||
}
|
||||
}, [selectedModel, selectedModelHrid, setSelectedModelHrid]);
|
||||
|
||||
const handleModelSelect = (model: LLMModel) => {
|
||||
setSelectedModel(model);
|
||||
setSelectedModelHrid(model.hrid);
|
||||
};
|
||||
|
||||
const router = useRouter();
|
||||
@@ -291,8 +296,7 @@ export const Chat = ({
|
||||
});
|
||||
}
|
||||
});
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [messages]);
|
||||
}, [messages, prefetchMetadata]);
|
||||
|
||||
const openSources = (messageId: string) => {
|
||||
if (isSourceOpen === messageId) {
|
||||
@@ -335,23 +339,16 @@ export const Chat = ({
|
||||
|
||||
const availableHeight = containerHeight - userMessageHeight - 38;
|
||||
|
||||
setStreamingMessageHeight((prev) => {
|
||||
if (prev === null || Math.abs(prev - availableHeight) > 10) {
|
||||
return availableHeight;
|
||||
}
|
||||
return prev;
|
||||
});
|
||||
if (streamingMessageHeight !== availableHeight) {
|
||||
setStreamingMessageHeight(availableHeight);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}, [messages]);
|
||||
}, [messages, streamingMessageHeight]);
|
||||
|
||||
// Détecter l'arrivée d'un nouveau message user et retirer la hauteur de l'ancien
|
||||
useEffect(() => {
|
||||
if (status === 'streaming') {
|
||||
return;
|
||||
}
|
||||
|
||||
const userMessages = messages.filter((msg) => msg.role === 'user');
|
||||
const lastUserMessage = userMessages[userMessages.length - 1];
|
||||
|
||||
@@ -364,14 +361,14 @@ export const Chat = ({
|
||||
}
|
||||
lastUserMessageIdRef.current = lastUserMessage.id;
|
||||
}
|
||||
}, [messages, status]);
|
||||
}, [messages]);
|
||||
|
||||
// Calculer la hauteur pendant submitted/streaming
|
||||
useEffect(() => {
|
||||
if (status === 'submitted' || status === 'streaming') {
|
||||
calculateStreamingHeight();
|
||||
}
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [status]);
|
||||
}, [status, calculateStreamingHeight]);
|
||||
|
||||
// Scroller vers la question au moment du submit
|
||||
useEffect(() => {
|
||||
@@ -391,8 +388,7 @@ export const Chat = ({
|
||||
|
||||
messageElement?.scrollIntoView({ block: 'start', behavior: 'smooth' });
|
||||
});
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [status]);
|
||||
}, [status, messages]);
|
||||
|
||||
// Synchronize conversationId state with prop when it changes (e.g., after navigation)
|
||||
useEffect(() => {
|
||||
@@ -404,8 +400,7 @@ export const Chat = ({
|
||||
} as ChangeEvent<HTMLTextAreaElement>);
|
||||
setHasInitialized(false); // Réinitialiser pour permettre le scroll au prochain chargement
|
||||
}
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [initialConversationId, conversationId]);
|
||||
}, [initialConversationId, conversationId, handleInputChange]);
|
||||
|
||||
// On mount, if there is pending input/files, initialize state and set flag
|
||||
useEffect(() => {
|
||||
@@ -709,11 +704,6 @@ export const Chat = ({
|
||||
{message.content && (
|
||||
<Box
|
||||
className="mainContent-chat"
|
||||
data-testid={
|
||||
message.role === 'assistant'
|
||||
? 'assistant-message-content'
|
||||
: undefined
|
||||
}
|
||||
$padding={{ all: 'xxs' }}
|
||||
>
|
||||
<p className="sr-only">
|
||||
@@ -721,53 +711,42 @@ export const Chat = ({
|
||||
? t('You said: ')
|
||||
: t('Assistant IA replied: ')}
|
||||
</p>
|
||||
{message.role === 'user' ? (
|
||||
<Text
|
||||
as="p"
|
||||
$css="white-space: pre-wrap; display: block;"
|
||||
$theme="greyscale"
|
||||
$variation="850"
|
||||
>
|
||||
{message.content}
|
||||
</Text>
|
||||
) : (
|
||||
<MarkdownHooks
|
||||
remarkPlugins={[remarkGfm, remarkMath]}
|
||||
rehypePlugins={[
|
||||
[
|
||||
rehypePrettyCode,
|
||||
{
|
||||
theme: 'github-dark-dimmed',
|
||||
},
|
||||
],
|
||||
rehypeKatex,
|
||||
]}
|
||||
components={{
|
||||
// Custom components for Markdown rendering
|
||||
// eslint-disable-next-line @typescript-eslint/no-unused-vars
|
||||
p: ({ node, ...props }) => (
|
||||
<Text
|
||||
as="p"
|
||||
$css="display: block"
|
||||
$theme="greyscale"
|
||||
$variation="850"
|
||||
{...props}
|
||||
/>
|
||||
),
|
||||
a: ({ children, ...props }) => (
|
||||
<a target="_blank" {...props}>
|
||||
{children}
|
||||
</a>
|
||||
),
|
||||
// eslint-disable-next-line @typescript-eslint/no-unused-vars
|
||||
pre: ({ node, children, ...props }) => (
|
||||
<CodeBlock {...props}>{children}</CodeBlock>
|
||||
),
|
||||
}}
|
||||
>
|
||||
{message.content}
|
||||
</MarkdownHooks>
|
||||
)}
|
||||
<MarkdownHooks
|
||||
remarkPlugins={[remarkGfm, remarkMath]}
|
||||
rehypePlugins={[
|
||||
[
|
||||
rehypePrettyCode,
|
||||
{
|
||||
theme: 'github-dark-dimmed',
|
||||
},
|
||||
],
|
||||
rehypeKatex,
|
||||
]}
|
||||
components={{
|
||||
// Custom components for Markdown rendering
|
||||
// eslint-disable-next-line @typescript-eslint/no-unused-vars
|
||||
p: ({ node, ...props }) => (
|
||||
<Text
|
||||
as="p"
|
||||
$css="display: block"
|
||||
$theme="greyscale"
|
||||
$variation="850"
|
||||
{...props}
|
||||
/>
|
||||
),
|
||||
a: ({ children, ...props }) => (
|
||||
<a target="_blank" {...props}>
|
||||
{children}
|
||||
</a>
|
||||
),
|
||||
// eslint-disable-next-line @typescript-eslint/no-unused-vars
|
||||
pre: ({ node, children, ...props }) => (
|
||||
<CodeBlock {...props}>{children}</CodeBlock>
|
||||
),
|
||||
}}
|
||||
>
|
||||
{message.content}
|
||||
</MarkdownHooks>
|
||||
</Box>
|
||||
)}
|
||||
|
||||
@@ -1000,7 +979,7 @@ export const Chat = ({
|
||||
</Text>
|
||||
</Box>
|
||||
) : null}
|
||||
{status === 'error' && !isUploadingFiles && (
|
||||
{status === 'error' && (
|
||||
<ChatError
|
||||
hasLastSubmission={!!lastSubmissionRef.current}
|
||||
onRetry={handleRetry}
|
||||
@@ -1012,7 +991,6 @@ export const Chat = ({
|
||||
position: relative;
|
||||
bottom: ${isMobile ? '8px' : '20px'};
|
||||
margin: auto;
|
||||
background-color: white;
|
||||
z-index: 1000;
|
||||
`}
|
||||
$gap="6px"
|
||||
|
||||
@@ -61,15 +61,11 @@ export const ChatError = ({ hasLastSubmission, onRetry }: ChatErrorProps) => {
|
||||
) : (
|
||||
<Button
|
||||
size="small"
|
||||
color="tertiary"
|
||||
style={{
|
||||
color: 'var(--c--theme--colors--greyscale-550)',
|
||||
borderColor: 'var(--c--theme--colors--greyscale-300)',
|
||||
}}
|
||||
color="primary"
|
||||
onClick={() => {
|
||||
void router.push('/');
|
||||
}}
|
||||
icon={<Icon iconName="add" $color="greyscale" />}
|
||||
icon={<Icon iconName="add" $color="white" />}
|
||||
>
|
||||
{t('Start a new conversation')}
|
||||
</Button>
|
||||
|
||||
@@ -300,6 +300,7 @@ export const InputChat = ({
|
||||
$css={`
|
||||
display: block;
|
||||
position: relative;
|
||||
opacity: ${status === 'error' ? '0.5' : '1'};
|
||||
margin: auto;
|
||||
width: 100%;
|
||||
padding: ${isDesktop ? '0' : '0 10px'};
|
||||
@@ -307,22 +308,26 @@ export const InputChat = ({
|
||||
`}
|
||||
>
|
||||
{/* Bouton de scroll vers le bas */}
|
||||
{messagesLength > 1 && containerRef && onScrollToBottom && (
|
||||
<Box
|
||||
$css={`
|
||||
{messagesLength > 1 &&
|
||||
status !== 'streaming' &&
|
||||
status !== 'submitted' &&
|
||||
containerRef &&
|
||||
onScrollToBottom && (
|
||||
<Box
|
||||
$css={`
|
||||
position: relative;
|
||||
height: 0;
|
||||
width: 100%;
|
||||
margin: auto;
|
||||
max-width: 750px;
|
||||
`}
|
||||
>
|
||||
<ScrollDown
|
||||
onClick={onScrollToBottom}
|
||||
containerRef={containerRef}
|
||||
/>
|
||||
</Box>
|
||||
)}
|
||||
>
|
||||
<ScrollDown
|
||||
onClick={onScrollToBottom}
|
||||
containerRef={containerRef}
|
||||
/>
|
||||
</Box>
|
||||
)}
|
||||
{/* Message de bienvenue */}
|
||||
{messagesLength === 0 && (
|
||||
<Box
|
||||
@@ -418,7 +423,6 @@ export const InputChat = ({
|
||||
fontSize: '1rem',
|
||||
border: 'none',
|
||||
resize: 'none',
|
||||
opacity: status === 'error' ? '0.5' : '1',
|
||||
fontFamily: 'inherit',
|
||||
minHeight: '64px',
|
||||
maxHeight: '200px',
|
||||
@@ -569,9 +573,6 @@ export const InputChat = ({
|
||||
$gap="sm"
|
||||
$padding={{ bottom: 'base' }}
|
||||
$align="space-between"
|
||||
$css={`
|
||||
opacity: ${status === 'error' ? '0.5' : '1'};
|
||||
`}
|
||||
>
|
||||
<Box
|
||||
$flex="1"
|
||||
|
||||
@@ -40,7 +40,7 @@ export const ToolInvocationItem: React.FC<ToolInvocationItemProps> = ({
|
||||
$color="var(--c--theme--colors--greyscale-500)"
|
||||
$padding={{ all: 'sm' }}
|
||||
$radius="8px"
|
||||
$css="font-size: 0.9em; width: 100%; white-space: pre-wrap; word-wrap: break-word;"
|
||||
$css="font-size: 0.9em;"
|
||||
>
|
||||
{toolInvocation.state === 'result' ? (
|
||||
<Text>{`Parsing done: ${documentIdentifiers.join(', ')}`}</Text>
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { useCallback, useRef, useState } from 'react';
|
||||
import { useState } from 'react';
|
||||
|
||||
interface SourceMetadata {
|
||||
title: string | null;
|
||||
@@ -9,124 +9,109 @@ interface SourceMetadata {
|
||||
|
||||
// Cache global pour éviter de refetch les mêmes URLs
|
||||
const metadataCache = new Map<string, SourceMetadata>();
|
||||
const fetchingUrls = new Set<string>();
|
||||
|
||||
export const useSourceMetadataCache = () => {
|
||||
const [, forceUpdate] = useState({});
|
||||
const updateCountRef = useRef(0);
|
||||
const [cache, setCache] =
|
||||
useState<Map<string, SourceMetadata>>(metadataCache);
|
||||
|
||||
const triggerUpdate = useCallback(() => {
|
||||
updateCountRef.current++;
|
||||
if (updateCountRef.current % 5 === 0) {
|
||||
forceUpdate({});
|
||||
const prefetchMetadata = async (url: string) => {
|
||||
// Si déjà en cache, ne rien faire
|
||||
if (metadataCache.has(url)) {
|
||||
return;
|
||||
}
|
||||
}, []);
|
||||
|
||||
const prefetchMetadata = useCallback(
|
||||
async (url: string) => {
|
||||
if (metadataCache.has(url) || fetchingUrls.has(url)) {
|
||||
return;
|
||||
}
|
||||
|
||||
fetchingUrls.add(url);
|
||||
|
||||
metadataCache.set(url, {
|
||||
title: null,
|
||||
favicon: null,
|
||||
loading: true,
|
||||
error: false,
|
||||
});
|
||||
triggerUpdate();
|
||||
|
||||
try {
|
||||
if (!url.startsWith('http')) {
|
||||
metadataCache.set(url, {
|
||||
title: url,
|
||||
favicon: '📄',
|
||||
loading: false,
|
||||
error: false,
|
||||
});
|
||||
fetchingUrls.delete(url);
|
||||
triggerUpdate();
|
||||
return;
|
||||
}
|
||||
|
||||
const parser = new DOMParser();
|
||||
|
||||
let response;
|
||||
try {
|
||||
response = await fetch(url, {
|
||||
mode: 'cors',
|
||||
headers: {
|
||||
'User-Agent': 'Mozilla/5.0 (compatible; ChatBot/1.0)',
|
||||
},
|
||||
});
|
||||
} catch {
|
||||
// Si CORS échoue, utiliser juste le hostname
|
||||
metadataCache.set(url, {
|
||||
title: new URL(url).hostname,
|
||||
favicon: null,
|
||||
loading: false,
|
||||
error: false,
|
||||
});
|
||||
fetchingUrls.delete(url);
|
||||
triggerUpdate();
|
||||
return;
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`HTTP ${response.status}`);
|
||||
}
|
||||
|
||||
const html = await response.text();
|
||||
const doc = parser.parseFromString(html, 'text/html');
|
||||
|
||||
// Récupérer le titre
|
||||
const pageTitle =
|
||||
doc.querySelector('title')?.textContent || new URL(url).hostname;
|
||||
|
||||
// Récupérer le favicon
|
||||
let faviconUrl =
|
||||
doc.querySelector('link[rel="icon"]')?.getAttribute('href') ||
|
||||
doc.querySelector('link[rel="shortcut icon"]')?.getAttribute('href');
|
||||
|
||||
if (!faviconUrl) {
|
||||
const urlObj = new URL(url);
|
||||
faviconUrl = `${urlObj.origin}/favicon.ico`;
|
||||
}
|
||||
|
||||
// Convertir les URLs relatives en absolues
|
||||
if (faviconUrl && !faviconUrl.startsWith('http')) {
|
||||
const urlObj = new URL(url);
|
||||
faviconUrl = new URL(faviconUrl, urlObj.origin).href;
|
||||
}
|
||||
// Marquer comme en cours de chargement
|
||||
metadataCache.set(url, {
|
||||
title: null,
|
||||
favicon: null,
|
||||
loading: true,
|
||||
error: false,
|
||||
});
|
||||
setCache(new Map(metadataCache));
|
||||
|
||||
try {
|
||||
if (!url.startsWith('http')) {
|
||||
metadataCache.set(url, {
|
||||
title: pageTitle,
|
||||
favicon: faviconUrl || null,
|
||||
title: url,
|
||||
favicon: '📄',
|
||||
loading: false,
|
||||
error: false,
|
||||
});
|
||||
fetchingUrls.delete(url);
|
||||
triggerUpdate();
|
||||
} catch (err) {
|
||||
console.log('Error fetching metadata for:', url, err);
|
||||
setCache(new Map(metadataCache));
|
||||
return;
|
||||
}
|
||||
|
||||
const parser = new DOMParser();
|
||||
|
||||
let response;
|
||||
try {
|
||||
response = await fetch(url, {
|
||||
mode: 'cors',
|
||||
headers: {
|
||||
'User-Agent': 'Mozilla/5.0 (compatible; ChatBot/1.0)',
|
||||
},
|
||||
});
|
||||
} catch {
|
||||
// Si CORS échoue, utiliser juste le hostname
|
||||
metadataCache.set(url, {
|
||||
title: new URL(url).hostname,
|
||||
favicon: null,
|
||||
loading: false,
|
||||
error: true,
|
||||
error: false,
|
||||
});
|
||||
fetchingUrls.delete(url);
|
||||
triggerUpdate();
|
||||
setCache(new Map(metadataCache));
|
||||
return;
|
||||
}
|
||||
},
|
||||
[triggerUpdate],
|
||||
);
|
||||
|
||||
const getMetadata = useCallback((url: string): SourceMetadata | undefined => {
|
||||
return metadataCache.get(url);
|
||||
}, []);
|
||||
if (!response.ok) {
|
||||
throw new Error(`HTTP ${response.status}`);
|
||||
}
|
||||
|
||||
return { prefetchMetadata, getMetadata };
|
||||
const html = await response.text();
|
||||
const doc = parser.parseFromString(html, 'text/html');
|
||||
|
||||
// Récupérer le titre
|
||||
const pageTitle =
|
||||
doc.querySelector('title')?.textContent || new URL(url).hostname;
|
||||
|
||||
// Récupérer le favicon
|
||||
let faviconUrl =
|
||||
doc.querySelector('link[rel="icon"]')?.getAttribute('href') ||
|
||||
doc.querySelector('link[rel="shortcut icon"]')?.getAttribute('href');
|
||||
|
||||
if (!faviconUrl) {
|
||||
const urlObj = new URL(url);
|
||||
faviconUrl = `${urlObj.origin}/favicon.ico`;
|
||||
}
|
||||
|
||||
// Convertir les URLs relatives en absolues
|
||||
if (faviconUrl && !faviconUrl.startsWith('http')) {
|
||||
const urlObj = new URL(url);
|
||||
faviconUrl = new URL(faviconUrl, urlObj.origin).href;
|
||||
}
|
||||
|
||||
metadataCache.set(url, {
|
||||
title: pageTitle,
|
||||
favicon: faviconUrl || null,
|
||||
loading: false,
|
||||
error: false,
|
||||
});
|
||||
setCache(new Map(metadataCache));
|
||||
} catch (err) {
|
||||
console.log('Error fetching metadata for:', url, err);
|
||||
metadataCache.set(url, {
|
||||
title: new URL(url).hostname,
|
||||
favicon: null,
|
||||
loading: false,
|
||||
error: true,
|
||||
});
|
||||
setCache(new Map(metadataCache));
|
||||
}
|
||||
};
|
||||
|
||||
const getMetadata = (url: string): SourceMetadata | undefined => {
|
||||
return cache.get(url);
|
||||
};
|
||||
|
||||
return { prefetchMetadata, getMetadata, cache };
|
||||
};
|
||||
|
||||
@@ -60,7 +60,7 @@
|
||||
"Give feedback": "Faire un retour",
|
||||
"History": "Historique",
|
||||
"Home": "Accueil",
|
||||
"If enabled, this allows us to analyse your exchanges to improve the Assistant. If disabled, all conversations remain confidential and are not used in any way. ": "Si cette option est activée, cela nous permet d'analyser vos conversations afin d'améliorer l'Assistant. Si elle est désactivée, toutes les conversations restent confidentielles et ne sont utilisées d'aucune manière. ",
|
||||
"If enabled, this allows us to analyse your exchanges to improve the Assistant. If disabled, all conversations remain confidential and are not used in any way. ": "Si cette option est activée, cela nous permet d'analyser vos conversations afin d'améliorer l'Assistant. Si elle est désactivée, toutes les conversations restent confidentielles et ne sont utilisées d'aucune manière ",
|
||||
"Illustration": "Image",
|
||||
"Image 401": "Image 401",
|
||||
"Image 403": "Image 403",
|
||||
|
||||
@@ -47,12 +47,16 @@ test.describe('Chat page', () => {
|
||||
|
||||
await page.keyboard.press('Enter');
|
||||
|
||||
// Wait for the response to appear
|
||||
await page
|
||||
.getByRole('button', { name: 'See more' })
|
||||
.waitFor({ timeout: 10000 });
|
||||
|
||||
const copyButton = page.getByRole('button', { name: 'Copy' });
|
||||
await expect(copyButton).toBeVisible();
|
||||
|
||||
const messageContent = page.getByTestId('assistant-message-content');
|
||||
const messageContent = page.getByText('Lorem ipsum dolor sit amet');
|
||||
await expect(messageContent).toBeVisible();
|
||||
await expect(messageContent).not.toBeEmpty();
|
||||
|
||||
// Check history
|
||||
const chatHistoryLink = page
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "app-e2e",
|
||||
"version": "0.0.10",
|
||||
"version": "0.0.9",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"lint": "eslint . --ext .ts",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "conversations",
|
||||
"version": "0.0.10",
|
||||
"version": "0.0.9",
|
||||
"private": true,
|
||||
"workspaces": {
|
||||
"packages": [
|
||||
@@ -32,8 +32,8 @@
|
||||
"@typescript-eslint/eslint-plugin": "8.33.1",
|
||||
"@typescript-eslint/parser": "8.33.1",
|
||||
"eslint": "8.57.0",
|
||||
"react": "19.2.1",
|
||||
"react-dom": "19.2.1",
|
||||
"react": "19.1.0",
|
||||
"react-dom": "19.1.0",
|
||||
"typescript": "5.8.3"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "eslint-config-conversations",
|
||||
"version": "0.0.10",
|
||||
"version": "0.0.9",
|
||||
"license": "MIT",
|
||||
"scripts": {
|
||||
"lint": "eslint --ext .js ."
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "packages-i18n",
|
||||
"version": "0.0.10",
|
||||
"version": "0.0.9",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"extract-translation": "yarn extract-translation:conversations",
|
||||
|
||||
+62
-62
@@ -1971,10 +1971,10 @@
|
||||
"@emnapi/runtime" "^1.4.3"
|
||||
"@tybys/wasm-util" "^0.9.0"
|
||||
|
||||
"@next/env@15.3.8":
|
||||
version "15.3.8"
|
||||
resolved "https://registry.yarnpkg.com/@next/env/-/env-15.3.8.tgz#02326c38d315d72f2ab8b2bcc9a5c81ec5482873"
|
||||
integrity sha512-SAfHg0g91MQVMPioeFeDjE+8UPF3j3BvHjs8ZKJAUz1BG7eMPvfCKOAgNWJ6s1MLNeP6O2InKQRTNblxPWuq+Q==
|
||||
"@next/env@15.3.3":
|
||||
version "15.3.3"
|
||||
resolved "https://registry.npmjs.org/@next/env/-/env-15.3.3.tgz"
|
||||
integrity sha512-OdiMrzCl2Xi0VTjiQQUK0Xh7bJHnOuET2s+3V+Y40WJBAXrJeGA3f+I8MZJ/YQ3mVGi5XGR1L66oFlgqXhQ4Vw==
|
||||
|
||||
"@next/eslint-plugin-next@15.3.3":
|
||||
version "15.3.3"
|
||||
@@ -1983,45 +1983,45 @@
|
||||
dependencies:
|
||||
fast-glob "3.3.1"
|
||||
|
||||
"@next/swc-darwin-arm64@15.3.5":
|
||||
version "15.3.5"
|
||||
resolved "https://registry.yarnpkg.com/@next/swc-darwin-arm64/-/swc-darwin-arm64-15.3.5.tgz#75606cb72e1659a23f15195dba760dc01b186c5d"
|
||||
integrity sha512-lM/8tilIsqBq+2nq9kbTW19vfwFve0NR7MxfkuSUbRSgXlMQoJYg+31+++XwKVSXk4uT23G2eF/7BRIKdn8t8w==
|
||||
"@next/swc-darwin-arm64@15.3.3":
|
||||
version "15.3.3"
|
||||
resolved "https://registry.npmjs.org/@next/swc-darwin-arm64/-/swc-darwin-arm64-15.3.3.tgz"
|
||||
integrity sha512-WRJERLuH+O3oYB4yZNVahSVFmtxRNjNF1I1c34tYMoJb0Pve+7/RaLAJJizyYiFhjYNGHRAE1Ri2Fd23zgDqhg==
|
||||
|
||||
"@next/swc-darwin-x64@15.3.5":
|
||||
version "15.3.5"
|
||||
resolved "https://registry.yarnpkg.com/@next/swc-darwin-x64/-/swc-darwin-x64-15.3.5.tgz#78ffad7ef26685e5b8150891b467a4ecef94e179"
|
||||
integrity sha512-WhwegPQJ5IfoUNZUVsI9TRAlKpjGVK0tpJTL6KeiC4cux9774NYE9Wu/iCfIkL/5J8rPAkqZpG7n+EfiAfidXA==
|
||||
"@next/swc-darwin-x64@15.3.3":
|
||||
version "15.3.3"
|
||||
resolved "https://registry.yarnpkg.com/@next/swc-darwin-x64/-/swc-darwin-x64-15.3.3.tgz#71588bad245180ffd1af1e1f894477287e739eb0"
|
||||
integrity sha512-XHdzH/yBc55lu78k/XwtuFR/ZXUTcflpRXcsu0nKmF45U96jt1tsOZhVrn5YH+paw66zOANpOnFQ9i6/j+UYvw==
|
||||
|
||||
"@next/swc-linux-arm64-gnu@15.3.5":
|
||||
version "15.3.5"
|
||||
resolved "https://registry.yarnpkg.com/@next/swc-linux-arm64-gnu/-/swc-linux-arm64-gnu-15.3.5.tgz#d9a405ceec729d62033dbdc48f8c331c544f09fd"
|
||||
integrity sha512-LVD6uMOZ7XePg3KWYdGuzuvVboxujGjbcuP2jsPAN3MnLdLoZUXKRc6ixxfs03RH7qBdEHCZjyLP/jBdCJVRJQ==
|
||||
"@next/swc-linux-arm64-gnu@15.3.3":
|
||||
version "15.3.3"
|
||||
resolved "https://registry.yarnpkg.com/@next/swc-linux-arm64-gnu/-/swc-linux-arm64-gnu-15.3.3.tgz#66a15f749c14f04a89f8c7e21c7a8d343fc34e6e"
|
||||
integrity sha512-VZ3sYL2LXB8znNGcjhocikEkag/8xiLgnvQts41tq6i+wql63SMS1Q6N8RVXHw5pEUjiof+II3HkDd7GFcgkzw==
|
||||
|
||||
"@next/swc-linux-arm64-musl@15.3.5":
|
||||
version "15.3.5"
|
||||
resolved "https://registry.yarnpkg.com/@next/swc-linux-arm64-musl/-/swc-linux-arm64-musl-15.3.5.tgz#65f19ad3ecd2881381ec2a149afba261ba180dde"
|
||||
integrity sha512-k8aVScYZ++BnS2P69ClK7v4nOu702jcF9AIHKu6llhHEtBSmM2zkPGl9yoqbSU/657IIIb0QHpdxEr0iW9z53A==
|
||||
"@next/swc-linux-arm64-musl@15.3.3":
|
||||
version "15.3.3"
|
||||
resolved "https://registry.yarnpkg.com/@next/swc-linux-arm64-musl/-/swc-linux-arm64-musl-15.3.3.tgz#14bd66213f7f33d6909574750bcb05037221a2ac"
|
||||
integrity sha512-h6Y1fLU4RWAp1HPNJWDYBQ+e3G7sLckyBXhmH9ajn8l/RSMnhbuPBV/fXmy3muMcVwoJdHL+UtzRzs0nXOf9SA==
|
||||
|
||||
"@next/swc-linux-x64-gnu@15.3.5":
|
||||
version "15.3.5"
|
||||
resolved "https://registry.yarnpkg.com/@next/swc-linux-x64-gnu/-/swc-linux-x64-gnu-15.3.5.tgz#cd7f7e002212360b99f7e791a2d2fedb352f2374"
|
||||
integrity sha512-2xYU0DI9DGN/bAHzVwADid22ba5d/xrbrQlr2U+/Q5WkFUzeL0TDR963BdrtLS/4bMmKZGptLeg6282H/S2i8A==
|
||||
"@next/swc-linux-x64-gnu@15.3.3":
|
||||
version "15.3.3"
|
||||
resolved "https://registry.yarnpkg.com/@next/swc-linux-x64-gnu/-/swc-linux-x64-gnu-15.3.3.tgz#4a19434545e5e752d9a3ed71f9b34982725f6293"
|
||||
integrity sha512-jJ8HRiF3N8Zw6hGlytCj5BiHyG/K+fnTKVDEKvUCyiQ/0r5tgwO7OgaRiOjjRoIx2vwLR+Rz8hQoPrnmFbJdfw==
|
||||
|
||||
"@next/swc-linux-x64-musl@15.3.5":
|
||||
version "15.3.5"
|
||||
resolved "https://registry.yarnpkg.com/@next/swc-linux-x64-musl/-/swc-linux-x64-musl-15.3.5.tgz#302c9e4ace935c963d45fce9584754a19295c452"
|
||||
integrity sha512-TRYIqAGf1KCbuAB0gjhdn5Ytd8fV+wJSM2Nh2is/xEqR8PZHxfQuaiNhoF50XfY90sNpaRMaGhF6E+qjV1b9Tg==
|
||||
"@next/swc-linux-x64-musl@15.3.3":
|
||||
version "15.3.3"
|
||||
resolved "https://registry.yarnpkg.com/@next/swc-linux-x64-musl/-/swc-linux-x64-musl-15.3.3.tgz#41ab140dd0a04ab7291adbec5836c1ce251a588c"
|
||||
integrity sha512-HrUcTr4N+RgiiGn3jjeT6Oo208UT/7BuTr7K0mdKRBtTbT4v9zJqCDKO97DUqqoBK1qyzP1RwvrWTvU6EPh/Cw==
|
||||
|
||||
"@next/swc-win32-arm64-msvc@15.3.5":
|
||||
version "15.3.5"
|
||||
resolved "https://registry.yarnpkg.com/@next/swc-win32-arm64-msvc/-/swc-win32-arm64-msvc-15.3.5.tgz#5bbe1434afa2360634d45fc7860a038d11e4e296"
|
||||
integrity sha512-h04/7iMEUSMY6fDGCvdanKqlO1qYvzNxntZlCzfE8i5P0uqzVQWQquU1TIhlz0VqGQGXLrFDuTJVONpqGqjGKQ==
|
||||
"@next/swc-win32-arm64-msvc@15.3.3":
|
||||
version "15.3.3"
|
||||
resolved "https://registry.yarnpkg.com/@next/swc-win32-arm64-msvc/-/swc-win32-arm64-msvc-15.3.3.tgz#fcd1d7e0007b7b73d1acdbf0ad6d91f7aa2deb15"
|
||||
integrity sha512-SxorONgi6K7ZUysMtRF3mIeHC5aA3IQLmKFQzU0OuhuUYwpOBc1ypaLJLP5Bf3M9k53KUUUj4vTPwzGvl/NwlQ==
|
||||
|
||||
"@next/swc-win32-x64-msvc@15.3.5":
|
||||
version "15.3.5"
|
||||
resolved "https://registry.yarnpkg.com/@next/swc-win32-x64-msvc/-/swc-win32-x64-msvc-15.3.5.tgz#9629b2eac3159c70f3449cecc2a29bfd4bcb2d5a"
|
||||
integrity sha512-5fhH6fccXxnX2KhllnGhkYMndhOiLOLEiVGYjP2nizqeGWkN10sA9taATlXwake2E2XMvYZjjz0Uj7T0y+z1yw==
|
||||
"@next/swc-win32-x64-msvc@15.3.3":
|
||||
version "15.3.3"
|
||||
resolved "https://registry.yarnpkg.com/@next/swc-win32-x64-msvc/-/swc-win32-x64-msvc-15.3.3.tgz#c0e33e069d7922dd0546cac77a0247ad81d4a1aa"
|
||||
integrity sha512-4QZG6F8enl9/S2+yIiOiju0iCTFd93d8VC1q9LZS4p/Xuk81W2QDjCFeoogmrWWkAD59z8ZxepBQap2dKS5ruw==
|
||||
|
||||
"@nodelib/fs.scandir@2.1.5":
|
||||
version "2.1.5"
|
||||
@@ -10838,12 +10838,12 @@ neo-async@^2.6.2:
|
||||
resolved "https://registry.npmjs.org/neo-async/-/neo-async-2.6.2.tgz"
|
||||
integrity sha512-Yd3UES5mWCSqR+qNT93S3UoYUkqAZ9lLg8a7g9rimsWmYGK8cVToA4/sF3RrshdyV3sAGMXVUmpMYOw+dLpOuw==
|
||||
|
||||
next@15.3.8:
|
||||
version "15.3.8"
|
||||
resolved "https://registry.yarnpkg.com/next/-/next-15.3.8.tgz#c7df2fa890c66fa3042e85437e3c1e8e6bd38b26"
|
||||
integrity sha512-L+4c5Hlr84fuaNADZbB9+ceRX9/CzwxJ+obXIGHupboB/Q1OLbSUapFs4bO8hnS/E6zV/JDX7sG1QpKVR2bguA==
|
||||
next@15.3.3:
|
||||
version "15.3.3"
|
||||
resolved "https://registry.npmjs.org/next/-/next-15.3.3.tgz"
|
||||
integrity sha512-JqNj29hHNmCLtNvd090SyRbXJiivQ+58XjCcrC50Crb5g5u2zi7Y2YivbsEfzk6AtVI80akdOQbaMZwWB1Hthw==
|
||||
dependencies:
|
||||
"@next/env" "15.3.8"
|
||||
"@next/env" "15.3.3"
|
||||
"@swc/counter" "0.1.3"
|
||||
"@swc/helpers" "0.5.15"
|
||||
busboy "1.6.0"
|
||||
@@ -10851,14 +10851,14 @@ next@15.3.8:
|
||||
postcss "8.4.31"
|
||||
styled-jsx "5.1.6"
|
||||
optionalDependencies:
|
||||
"@next/swc-darwin-arm64" "15.3.5"
|
||||
"@next/swc-darwin-x64" "15.3.5"
|
||||
"@next/swc-linux-arm64-gnu" "15.3.5"
|
||||
"@next/swc-linux-arm64-musl" "15.3.5"
|
||||
"@next/swc-linux-x64-gnu" "15.3.5"
|
||||
"@next/swc-linux-x64-musl" "15.3.5"
|
||||
"@next/swc-win32-arm64-msvc" "15.3.5"
|
||||
"@next/swc-win32-x64-msvc" "15.3.5"
|
||||
"@next/swc-darwin-arm64" "15.3.3"
|
||||
"@next/swc-darwin-x64" "15.3.3"
|
||||
"@next/swc-linux-arm64-gnu" "15.3.3"
|
||||
"@next/swc-linux-arm64-musl" "15.3.3"
|
||||
"@next/swc-linux-x64-gnu" "15.3.3"
|
||||
"@next/swc-linux-x64-musl" "15.3.3"
|
||||
"@next/swc-win32-arm64-msvc" "15.3.3"
|
||||
"@next/swc-win32-x64-msvc" "15.3.3"
|
||||
sharp "^0.34.1"
|
||||
|
||||
no-case@^3.0.4:
|
||||
@@ -11779,12 +11779,12 @@ react-dnd@^14.0.3:
|
||||
fast-deep-equal "^3.1.3"
|
||||
hoist-non-react-statics "^3.3.2"
|
||||
|
||||
react-dom@19.0.0, react-dom@19.2.1:
|
||||
version "19.2.1"
|
||||
resolved "https://registry.yarnpkg.com/react-dom/-/react-dom-19.2.1.tgz#ce3527560bda4f997e47d10dab754825b3061f59"
|
||||
integrity sha512-ibrK8llX2a4eOskq1mXKu/TGZj9qzomO+sNfO98M6d9zIPOEhlBkMkBUBLd1vgS0gQsLDBzA+8jJBVXDnfHmJg==
|
||||
react-dom@18.3.1, react-dom@19.0.0, react-dom@19.1.0:
|
||||
version "19.1.0"
|
||||
resolved "https://registry.yarnpkg.com/react-dom/-/react-dom-19.1.0.tgz#133558deca37fa1d682708df8904b25186793623"
|
||||
integrity sha512-Xs1hdnE+DyKgeHJeJznQmYMIBG3TKIHJJT95Q58nHLSrElKlGQqDTR2HQ9fx5CN/Gk6Vh/kupBTDLU11/nDk/g==
|
||||
dependencies:
|
||||
scheduler "^0.27.0"
|
||||
scheduler "^0.26.0"
|
||||
|
||||
react-i18next@15.5.2:
|
||||
version "15.5.2"
|
||||
@@ -12006,10 +12006,10 @@ react-window@^1.8.11:
|
||||
"@babel/runtime" "^7.0.0"
|
||||
memoize-one ">=3.1.1 <6"
|
||||
|
||||
react@19.0.0, react@19.2.1:
|
||||
version "19.2.1"
|
||||
resolved "https://registry.yarnpkg.com/react/-/react-19.2.1.tgz#8600fa205e58e2e807f6ef431c9f6492591a2700"
|
||||
integrity sha512-DGrYcCWK7tvYMnWh79yrPHt+vdx9tY+1gPZa7nJQtO/p8bLTDaHp4dzwEhQB7pZ4Xe3ok4XKuEPrVuc+wlpkmw==
|
||||
react@*, react@19.0.0, react@19.1.0:
|
||||
version "19.1.0"
|
||||
resolved "https://registry.yarnpkg.com/react/-/react-19.1.0.tgz#926864b6c48da7627f004795d6cce50e90793b75"
|
||||
integrity sha512-FS+XFBNvn3GTAWq26joslQgWNoFu08F4kl0J4CgdNKADkdSGXQyTCnKteIAJy96Br6YbpEU1LSzV5dYtjMkMDg==
|
||||
|
||||
readable-stream@^3.4.0:
|
||||
version "3.6.2"
|
||||
@@ -12444,10 +12444,10 @@ scheduler@0.25.0-rc-603e6108-20241029:
|
||||
resolved "https://registry.npmjs.org/scheduler/-/scheduler-0.25.0-rc-603e6108-20241029.tgz"
|
||||
integrity sha512-pFwF6H1XrSdYYNLfOcGlM28/j8CGLu8IvdrxqhjWULe2bPcKiKW4CV+OWqR/9fT52mywx65l7ysNkjLKBda7eA==
|
||||
|
||||
scheduler@^0.27.0:
|
||||
version "0.27.0"
|
||||
resolved "https://registry.yarnpkg.com/scheduler/-/scheduler-0.27.0.tgz#0c4ef82d67d1e5c1e359e8fc76d3a87f045fe5bd"
|
||||
integrity sha512-eNv+WrVbKu1f3vbYJT/xtiF5syA5HPIMtf9IgY/nKg0sWqzAUEvqY/xm7OcZc/qafLx/iO9FgOmeSAp4v5ti/Q==
|
||||
scheduler@^0.26.0:
|
||||
version "0.26.0"
|
||||
resolved "https://registry.yarnpkg.com/scheduler/-/scheduler-0.26.0.tgz#4ce8a8c2a2095f13ea11bf9a445be50c555d6337"
|
||||
integrity sha512-NlHwttCI/l5gCPR3D1nNXtWABUmBwvZpEQiD4IXSbIDq8BzLIK/7Ir5gTFSGZDUu37K5cMNp0hFtzO38sC7gWA==
|
||||
|
||||
schema-utils@^4.3.0, schema-utils@^4.3.2:
|
||||
version "4.3.2"
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "mail_mjml",
|
||||
"version": "0.0.10",
|
||||
"version": "0.0.9",
|
||||
"description": "An util to generate html and text django's templates from mjml templates",
|
||||
"type": "module",
|
||||
"dependencies": {
|
||||
|
||||
Reference in New Issue
Block a user