Compare commits

..

3 Commits

Author SHA1 Message Date
Anthony LC 0fb51a2453 fixup! 🐛(backend) race condition create doc 2025-02-24 10:01:57 +01:00
Anthony LC 042034a9c9 fixup! 🐛(backend) race condition create doc 2025-02-24 09:50:13 +01:00
Anthony LC f28081a72a 🐛(backend) race condition create doc
When 2 docs are created almost at the same time,
the second one will fail because the first one.
We get a unicity error on the path key already
used ("impress_document_path_key").
To fix this issue, we will lock the table the
time to create the document, the next query will
wait for the lock to be released.
2025-02-17 09:31:50 +01:00
17 changed files with 215 additions and 268 deletions
+24 -8
View File
@@ -6,18 +6,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0),
and this project adheres to
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## Changed
- 🛂(frontend) Restore version visibility #629
- 📝(doc) minor README.md formatting and wording enhancements
- ♻️Stop setting a default title on doc creation #634
- 📝(readme) remove front-end local run instructions local.md #651
## Fixed
- ♻️(frontend) improve table pdf rendering
- 🐛(backend) race condition create doc #633
## [2.2.0] - 2025-02-10
@@ -38,6 +38,7 @@ and this project adheres to
- 🐛(frontend) fix cursor breakline #609
- 🐛(frontend) fix style pdf export #609
## [2.1.0] - 2025-01-29
## Added
@@ -53,7 +54,7 @@ and this project adheres to
## Changed
- 💄(frontend) add abilities on doc row #581
- 💄(frontend) improve DocsGridItem responsive padding #582
- 💄(frontend) improve DocsGridItem responsive padding #582
- 🔧(backend) Bump maximum page size to 200 #516
- 📝(doc) Improve Read me #558
@@ -65,6 +66,7 @@ and this project adheres to
- 🔥(backend) remove "content" field from list serializer # 516
## [2.0.1] - 2025-01-17
## Fixed
@@ -119,11 +121,12 @@ and this project adheres to
- ⚡️(e2e) reduce flakiness on e2e tests #511
## Fixed
## Fixed
- 🐛(frontend) update doc editor height #481
- 💄(frontend) add doc search #485
## [1.9.0] - 2024-12-11
## Added
@@ -145,18 +148,21 @@ and this project adheres to
- 🐛(frontend) Fix hidden menu on Firefox #468
- 🐛(backend) fix sanitize problem IA #490
## [1.8.2] - 2024-11-28
## Changed
- ♻️(SW) change strategy html caching #460
## [1.8.1] - 2024-11-27
## Fixed
- 🐛(frontend) link not clickable and flickering firefox #457
## [1.8.0] - 2024-11-25
## Added
@@ -185,6 +191,7 @@ and this project adheres to
- 🐛(frontend) users have view access when revoked #387
- 🐛(frontend) fix placeholder editable when double clicks #454
## [1.7.0] - 2024-10-24
## Added
@@ -212,6 +219,7 @@ and this project adheres to
- 🔥(helm) remove infra related codes #366
## [1.6.0] - 2024-10-17
## Added
@@ -234,6 +242,7 @@ and this project adheres to
- 🐛(backend) fix nginx docker container #340
- 🐛(frontend) fix copy paste firefox #353
## [1.5.1] - 2024-10-10
## Fixed
@@ -268,6 +277,7 @@ and this project adheres to
- 🔧(backend) fix configuration to avoid different ssl warning #297
- 🐛(frontend) fix editor break line not working #302
## [1.4.0] - 2024-09-17
## Added
@@ -288,6 +298,7 @@ and this project adheres to
- 🐛(backend) Fix forcing ID when creating a document via API endpoint #234
- 🐛 Rebuild frontend dev container from makefile #248
## [1.3.0] - 2024-09-05
## Added
@@ -312,6 +323,7 @@ and this project adheres to
- 🔥(frontend) remove saving modal #213
## [1.2.1] - 2024-08-23
## Changed
@@ -319,6 +331,7 @@ and this project adheres to
- ♻️ Change ordering docs datagrid #195
- 🔥(helm) use scaleway email #194
## [1.2.0] - 2024-08-22
## Added
@@ -342,14 +355,14 @@ and this project adheres to
- ⚡️(CI) only e2e chrome mandatory #177
## Removed
- 🔥(helm) remove htaccess #181
## [1.1.0] - 2024-07-15
## Added
- 🤡(demo) generate dummy documents on dev users #120
- 🤡(demo) generate dummy documents on dev users #120
- ✨(frontend) create side modal component #134
- ✨(frontend) Doc grid actions (update / delete) #136
- ✨(frontend) Doc editor header information #137
@@ -360,11 +373,12 @@ and this project adheres to
- ♻️(frontend) create a doc from a modal #132
- ♻️(frontend) manage members from the share modal #140
## [1.0.0] - 2024-07-02
## Added
- 🛂(frontend) Manage the document's right (#75)
- 🛂(frontend) Manage the document's right (#75)
- ✨(frontend) Update document (#68)
- ✨(frontend) Remove document (#68)
- 🐳(docker) dockerize dev frontend (#63)
@@ -398,6 +412,7 @@ and this project adheres to
- 💚(CI) Remove trigger workflow on push tags on CI (#68)
- 🔥(frontend) Remove coming soon page (#121)
## [0.1.0] - 2024-05-24
## Added
@@ -405,6 +420,7 @@ and this project adheres to
- ✨(frontend) Coming Soon page (#67)
- 🚀 Impress, project to manage your documents easily and collaboratively.
[unreleased]: https://github.com/numerique-gouv/impress/compare/v2.2.0...main
[v2.2.0]: https://github.com/numerique-gouv/impress/releases/v2.2.0
[v2.1.0]: https://github.com/numerique-gouv/impress/releases/v2.1.0
+21 -1
View File
@@ -40,7 +40,7 @@ Docs is a collaborative text editor designed to address common challenges in kno
* 📚 Built-in wiki functionality to turn your team's collaborative work into organized knowledge `ETA 02/2025`
### Self-host
* 🚀 Easy to install, scalable and secure alternative to Notion and Outline.
* 🚀 Easy to install, scalable and secure alternative to Notion, Outline or Confluence
## Getting started 🔧
@@ -100,6 +100,26 @@ password: impress
$ make run
```
⚠️ For the frontend developer, it is often better to run the frontend in development mode locally.
To do so, install the frontend dependencies with the following command:
```shellscript
$ make frontend-development-install
```
And run the frontend locally in development mode with the following command:
```shellscript
$ make run-frontend-development
```
To start all the services, except the frontend container, you can use the following command:
```shellscript
$ make run-backend
```
**Adding content**
You can create a basic demo site by running:
+3 -1
View File
@@ -7,6 +7,7 @@ UNSET_USER=0
TERRAFORM_DIRECTORY="./env.d/terraform"
COMPOSE_FILE="${REPO_DIR}/docker-compose.yml"
COMPOSE_PROJECT="docs"
# _set_user: set (or unset) default user id used to run docker commands
@@ -39,8 +40,9 @@ function _set_user() {
# ARGS : docker compose command arguments
function _docker_compose() {
echo "🐳(compose) file: '${COMPOSE_FILE}'"
echo "🐳(compose) project: '${COMPOSE_PROJECT}' file: '${COMPOSE_FILE}'"
docker compose \
-p "${COMPOSE_PROJECT}" \
-f "${COMPOSE_FILE}" \
--project-directory "${REPO_DIR}" \
"$@"
-2
View File
@@ -1,5 +1,3 @@
name: docs
services:
postgresql:
image: postgres:16
-92
View File
@@ -1,92 +0,0 @@
# Run Docs locally
> ⚠️ Running Docs locally using the methods described below is for testing purposes only. It is based on building Docs using Minio as the S3 storage solution: if you want to use Minio for production deployment of Docs, you will need to comply with Minio's AGPL-3.0 licence.
**Prerequisite**
Make sure you have a recent version of Docker and [Docker Compose](https://docs.docker.com/compose/install) installed on your laptop:
```shellscript
$ docker -v
Docker version 20.10.2, build 2291f61
$ docker compose version
Docker Compose version v2.32.4
```
> ⚠️ You may need to run the following commands with sudo but this can be avoided by adding your user to the `docker` group.
**Project bootstrap**
The easiest way to start working on the project is to use GNU Make:
```shellscript
$ make bootstrap FLUSH_ARGS='--no-input'
```
This command builds the `app` container, installs dependencies, performs database migrations and compile translations. It's a good idea to use this command each time you are pulling code from the project repository to avoid dependency-related or migration-related issues.
Your Docker services should now be up and running 🎉
You can access to the project by going to <http://localhost:3000>.
You will be prompted to log in, the default credentials are:
```
username: impress
password: impress
```
📝 Note that if you need to run them afterwards, you can use the eponym Make rule:
```shellscript
$ make run
```
**Adding content**
You can create a basic demo site by running:
```shellscript
$ make demo
```
Finally, you can check all available Make rules using:
```shellscript
$ make help
```
**Django admin**
You can access the Django admin site at
<http://localhost:8071/admin>.
You first need to create a superuser account:
```shellscript
$ make superuser
```
## Front-end dev instructions
⚠️ For the frontend developer, it is often better to run the frontend in development mode locally.
To do so, install the frontend dependencies with the following command:
```shellscript
$ make frontend-development-install
```
And run the frontend locally in development mode with the following command:
```shellscript
$ make run-frontend-development
```
To start all the services, except the frontend container, you can use the following command:
```shellscript
$ make run-backend
```
+19 -2
View File
@@ -12,8 +12,8 @@ from django.contrib.postgres.fields import ArrayField
from django.contrib.postgres.search import TrigramSimilarity
from django.core.exceptions import ValidationError
from django.core.files.storage import default_storage
from django.db import connection, transaction
from django.db import models as db
from django.db import transaction
from django.db.models.expressions import RawSQL
from django.db.models.functions import Left, Length
from django.http import Http404
@@ -573,6 +573,12 @@ class DocumentViewSet(
@transaction.atomic
def perform_create(self, serializer):
"""Set the current user as creator and owner of the newly created object."""
with connection.cursor() as cursor:
cursor.execute(
f'LOCK TABLE "{models.Document._meta.db_table}" ' # noqa: SLF001
"IN SHARE ROW EXCLUSIVE MODE;"
)
obj = models.Document.add_root(
creator=self.request.user,
**serializer.validated_data,
@@ -631,10 +637,17 @@ class DocumentViewSet(
permission_classes=[],
url_path="create-for-owner",
)
@transaction.atomic
def create_for_owner(self, request):
"""
Create a document on behalf of a specified owner (pre-existing user or invited).
"""
with connection.cursor() as cursor:
cursor.execute(
f'LOCK TABLE "{models.Document._meta.db_table}" ' # noqa: SLF001
"IN SHARE ROW EXCLUSIVE MODE;"
)
# Deserialize and validate the data
serializer = serializers.ServerCreateDocumentSerializer(data=request.data)
if not serializer.is_valid():
@@ -741,7 +754,11 @@ class DocumentViewSet(
serializer.is_valid(raise_exception=True)
with transaction.atomic():
child_document = document.add_child(
locked_parent = models.Document.objects.select_for_update().get(
pk=document.pk
)
child_document = locked_parent.add_child(
creator=request.user,
**serializer.validated_data,
)
@@ -2,6 +2,7 @@
Tests for Documents API endpoint in impress's core app: create
"""
from concurrent.futures import ThreadPoolExecutor
from uuid import uuid4
import pytest
@@ -249,3 +250,41 @@ def test_api_documents_children_create_force_id_existing():
assert response.json() == {
"id": ["A document with this ID already exists. You cannot override it."]
}
@pytest.mark.django_db(transaction=True)
def test_api_documents_create_document_children_race_condition():
"""
It should be possible to create several documents at the same time
without causing any race conditions or data integrity issues.
"""
user = factories.UserFactory()
client = APIClient()
client.force_login(user)
document = factories.DocumentFactory()
factories.UserDocumentAccessFactory(user=user, document=document, role="owner")
def create_document():
return client.post(
f"/api/v1.0/documents/{document.id}/children/",
{
"title": "my child",
},
)
with ThreadPoolExecutor(max_workers=2) as executor:
future1 = executor.submit(create_document)
future2 = executor.submit(create_document)
response1 = future1.result()
response2 = future2.result()
assert response1.status_code == 201
assert response2.status_code == 201
document.refresh_from_db()
assert document.numchild == 2
@@ -2,6 +2,7 @@
Tests for Documents API endpoint in impress's core app: create
"""
from concurrent.futures import ThreadPoolExecutor
from uuid import uuid4
import pytest
@@ -51,6 +52,36 @@ def test_api_documents_create_authenticated_success():
assert document.accesses.filter(role="owner", user=user).exists()
@pytest.mark.django_db(transaction=True)
def test_api_documents_create_document_race_condition():
"""
It should be possible to create several documents at the same time
without causing any race conditions or data integrity issues.
"""
def create_document(title):
user = factories.UserFactory()
client = APIClient()
client.force_login(user)
return client.post(
"/api/v1.0/documents/",
{
"title": title,
},
format="json",
)
with ThreadPoolExecutor(max_workers=2) as executor:
future1 = executor.submit(create_document, "my document 1")
future2 = executor.submit(create_document, "my document 2")
response1 = future1.result()
response2 = future2.result()
assert response1.status_code == 201
assert response2.status_code == 201
def test_api_documents_create_authenticated_title_null():
"""It should be possible to create several documents with a null title."""
user = factories.UserFactory()
@@ -4,6 +4,7 @@ Tests for Documents API endpoint in impress's core app: create
# pylint: disable=W0621
from concurrent.futures import ThreadPoolExecutor
from unittest.mock import patch
from django.core import mail
@@ -425,6 +426,36 @@ def test_api_documents_create_for_owner_new_user_no_sub_no_fallback_allow_duplic
assert document.creator == user
@pytest.mark.django_db(transaction=True)
def test_api_documents_create_document_race_condition():
"""
It should be possible to create several documents at the same time
without causing any race conditions or data integrity issues.
"""
def create_document(title):
user = factories.UserFactory()
client = APIClient()
client.force_login(user)
return client.post(
"/api/v1.0/documents/",
{
"title": title,
},
format="json",
)
with ThreadPoolExecutor(max_workers=2) as executor:
future1 = executor.submit(create_document, "my document 1")
future2 = executor.submit(create_document, "my document 2")
response1 = future1.result()
response2 = future2.result()
assert response1.status_code == 201
assert response2.status_code == 201
@patch.object(ServerCreateDocumentSerializer, "_send_email_notification")
@override_settings(SERVER_TO_SERVER_API_TOKENS=["DummyToken"], LANGUAGE_CODE="de-de")
def test_api_documents_create_for_owner_with_default_language(
@@ -1,46 +1,27 @@
import { FullConfig, FullProject, chromium, expect } from '@playwright/test';
import { test as setup } from '@playwright/test';
import { keyCloakSignIn } from './common';
const saveStorageState = async (
browserConfig: FullProject<unknown, unknown>,
) => {
const browserName = browserConfig?.name || 'chromium';
setup('authenticate-chromium', async ({ page }) => {
await page.goto('/');
await keyCloakSignIn(page, 'chromium');
await page
.context()
.storageState({ path: `playwright/.auth/user-chromium.json` });
});
const { storageState, ...useConfig } = browserConfig?.use;
const browser = await chromium.launch();
const context = await browser.newContext(useConfig);
const page = await context.newPage();
setup('authenticate-webkit', async ({ page }) => {
await page.goto('/');
await keyCloakSignIn(page, 'webkit');
await page
.context()
.storageState({ path: `playwright/.auth/user-webkit.json` });
});
await page.goto('/', { waitUntil: 'networkidle' });
await page.content();
await expect(page.getByText('Docs').first()).toBeVisible();
await keyCloakSignIn(page, browserName);
await expect(
page.locator('header').first().getByRole('button', {
name: 'Logout',
}),
).toBeVisible();
await page.context().storageState({
path: storageState as string,
});
await browser.close();
};
async function globalSetup(config: FullConfig) {
/* eslint-disable @typescript-eslint/no-non-null-assertion */
const chromeConfig = config.projects.find((p) => p.name === 'chromium')!;
const firefoxConfig = config.projects.find((p) => p.name === 'firefox')!;
const webkitConfig = config.projects.find((p) => p.name === 'webkit')!;
/* eslint-enable @typescript-eslint/no-non-null-assertion */
await saveStorageState(chromeConfig);
await saveStorageState(webkitConfig);
await saveStorageState(firefoxConfig);
}
export default globalSetup;
setup('authenticate-firefox', async ({ page }) => {
await page.goto('/');
await keyCloakSignIn(page, 'firefox');
await page
.context()
.storageState({ path: `playwright/.auth/user-firefox.json` });
});
+5 -1
View File
@@ -38,9 +38,10 @@ export default defineConfig({
timeout: 120 * 1000,
reuseExistingServer: true,
},
globalSetup: require.resolve('./__tests__/app-impress/auth.setup'),
/* Configure projects for major browsers */
projects: [
{ name: 'setup', testMatch: /.*\.setup\.ts/ },
{
name: 'chromium',
use: {
@@ -52,6 +53,7 @@ export default defineConfig({
permissions: ['clipboard-read', 'clipboard-write'],
},
},
dependencies: ['setup'],
},
{
name: 'webkit',
@@ -61,6 +63,7 @@ export default defineConfig({
timezoneId: 'Europe/Paris',
storageState: 'playwright/.auth/user-webkit.json',
},
dependencies: ['setup'],
},
{
name: 'firefox',
@@ -76,6 +79,7 @@ export default defineConfig({
},
},
},
dependencies: ['setup'],
},
],
});
-1
View File
@@ -15,7 +15,6 @@
"test:watch": "jest --watch"
},
"dependencies": {
"@ag-media/react-pdf-table": "2.0.1",
"@blocknote/core": "0.23.2",
"@blocknote/mantine": "0.23.2",
"@blocknote/react": "0.23.2",
@@ -15,7 +15,7 @@ import {
VariantType,
useToastProvider,
} from '@openfun/cunningham-react';
import { Text as PDFText, pdf } from '@react-pdf/renderer';
import { pdf } from '@react-pdf/renderer';
import { useMemo, useState } from 'react';
import { useTranslation } from 'react-i18next';
import { css } from 'styled-components';
@@ -27,8 +27,6 @@ import { Doc } from '@/features/docs/doc-management';
import { TemplatesOrdering, useTemplates } from '../api/useTemplates';
import { downloadFile, exportResolveFileUrl } from '../utils';
import { Table } from './blocks/Table';
enum DocDownloadFormat {
PDF = 'pdf',
DOCX = 'docx',
@@ -115,8 +113,7 @@ export const ModalExport = ({ onClose, doc }: ModalExportProps) => {
? 1.5
: 1.17;
return (
<PDFText
key={block.id}
<Text
style={{
fontSize: fontSizeEM * FONT_SIZE * PIXELS_PER_POINT,
fontWeight: 700,
@@ -125,7 +122,7 @@ export const ModalExport = ({ onClose, doc }: ModalExportProps) => {
}}
>
{exporter.transformInlineContent(block.content)}
</PDFText>
</Text>
);
},
paragraph: (block, exporter) => {
@@ -149,14 +146,11 @@ export const ModalExport = ({ onClose, doc }: ModalExportProps) => {
}
}
return (
<PDFText key={block.id}>
<Text key={block.id}>
{exporter.transformInlineContent(block.content)}
</PDFText>
</Text>
);
},
table: (block, transformer) => {
return <Table data={block.content} transformer={transformer} />;
},
},
},
{
@@ -1,76 +0,0 @@
import { TD, TH, TR, Table as TablePDF } from '@ag-media/react-pdf-table';
import {
DefaultBlockSchema,
Exporter,
InlineContentSchema,
StyleSchema,
TableContent,
} from '@blocknote/core';
import { View } from '@react-pdf/renderer';
import { ReactNode } from 'react';
export const Table = (props: {
data: TableContent<InlineContentSchema>;
transformer: Exporter<
DefaultBlockSchema,
InlineContentSchema,
StyleSchema,
unknown,
unknown,
unknown,
unknown
>;
}) => {
return (
<TablePDF>
{props.data.rows.map((row, index) => {
if (index === 0) {
return (
<TH key={index}>
{row.cells.map((cell, index) => {
// Make empty cells are rendered.
if (cell.length === 0) {
cell.push({
styles: {},
text: ' ',
type: 'text',
});
}
return (
<TD key={index}>
{props.transformer.transformInlineContent(cell)}
</TD>
);
})}
</TH>
);
}
return (
<TR key={index}>
{row.cells.map((cell, index) => {
// Make empty cells are rendered.
if (cell.length === 0) {
cell.push({
styles: {},
text: ' ',
type: 'text',
});
}
return (
<TD key={index}>
<View>
{
props.transformer.transformInlineContent(
cell,
) as ReactNode
}
</View>
</TD>
);
})}
</TR>
);
})}
</TablePDF>
);
};
@@ -36,9 +36,8 @@ export const ModalSelectVersion = ({
const { t } = useTranslation();
const [selectedVersionId, setSelectedVersionId] =
useState<Versions['version_id']>();
const canRestore = doc.abilities.partial_update;
const restoreModal = useModal();
const restoreModal = useModal();
return (
<>
<Modal
@@ -128,23 +127,21 @@ export const ModalSelectVersion = ({
selectedVersionId={selectedVersionId}
/>
</Box>
{canRestore && (
<Box
$padding="xs"
$css={css`
border-top: 1px solid var(--c--theme--colors--greyscale-200);
`}
<Box
$padding="xs"
$css={css`
border-top: 1px solid var(--c--theme--colors--greyscale-200);
`}
>
<Button
fullWidth
disabled={!selectedVersionId}
onClick={restoreModal.open}
color="primary"
>
<Button
fullWidth
disabled={!selectedVersionId}
onClick={restoreModal.open}
color="primary"
>
{t('Restore')}
</Button>
</Box>
)}
{t('Restore')}
</Button>
</Box>
</Box>
</Box>
</Modal>
@@ -1,9 +0,0 @@
/* eslint-disable @typescript-eslint/no-explicit-any */
// This file is used to declare the types for the @ag-media/react-pdf-table package
// This library does not export its types in the "exports" field in the package.json.
declare module '@ag-media/react-pdf-table' {
export const TD: React.FC<any>;
export const TH: React.FC<any>;
export const TR: React.FC<any>;
export const Table: React.FC<any>;
}
-5
View File
@@ -7,11 +7,6 @@
resolved "https://registry.yarnpkg.com/@adobe/css-tools/-/css-tools-4.4.1.tgz#2447a230bfe072c1659e6815129c03cf170710e3"
integrity sha512-12WGKBQzjUAI4ayyF4IAtfw2QR/IDoqk6jTddXDhtYTJF9ASmoE1zst7cVtP0aL/F1jUJL5r+JxKXKEgHNbEUQ==
"@ag-media/react-pdf-table@2.0.1":
version "2.0.1"
resolved "https://registry.yarnpkg.com/@ag-media/react-pdf-table/-/react-pdf-table-2.0.1.tgz#609e51992faed54bcf379a376442997c6bac53cc"
integrity sha512-UMNdGYAfuI6L1wLRziYmwcp/8I2JgbwX+PY7bHXGb2+P6MwgFJH8W71qZO1bxfxrmVUTP8YblQwl1PkXG2m6Rg==
"@ampproject/remapping@^2.2.0":
version "2.3.0"
resolved "https://registry.yarnpkg.com/@ampproject/remapping/-/remapping-2.3.0.tgz#ed441b6fa600072520ce18b43d2c8cc8caecc7f4"