Compare commits

..

39 Commits

Author SHA1 Message Date
StillKnotKnown 0689dcc6c8 fix(gitlab): URL-encode namespace path in permission check
The namespace path from project info may contain special characters
that need URL encoding. Use encode_project_path to ensure proper
encoding when querying the /namespaces/ endpoint.

This addresses review feedback about URL encoding consistency.
2026-01-27 23:32:58 +02:00
StillKnotKnown ccfe1361a4 fix(tests): use safe path for nonexistent directory tests
The path /nonexistent may exist on some systems (causing PermissionError).
Use /tmp/nonexistent_test_path_xyz123_that_does_not_exist instead which
is guaranteed not to exist.

Fixes 5 test failures:
- test_ci_discovery.py::TestEdgeCases::test_nonexistent_directory
- test_discovery.py::TestEdgeCases::test_nonexistent_directory
- test_security_scanner.py::TestEdgeCases::test_nonexistent_directory
- test_service_orchestrator.py::TestEdgeCases::test_nonexistent_directory
- test_validation_strategy.py::TestEdgeCases::test_nonexistent_directory
2026-01-27 23:24:18 +02:00
StillKnotKnown d7b515a94d fix(tests): fix missing imports in test_github_bot_detection.py
- Add MagicMock import (was causing NameError)
- Add timedelta import from datetime
- Fix datetime.now() calls by importing datetime module inline

This fixes the test-python CI failures caused by missing imports.
2026-01-27 23:20:28 +02:00
StillKnotKnown 3ed405bd68 fix(tests): update mock setup in remaining permissions tests 2026-01-27 23:06:03 +02:00
StillKnotKnown ff71b8d089 fix(tests): fix GitLabRole usage and mock method in permissions tests 2026-01-27 23:04:08 +02:00
StillKnotKnown 7f3ca14fc3 fix(tests): correct expected exception message in test_retry_exhausted 2026-01-27 22:52:51 +02:00
StillKnotKnown d1c8de3c24 fix(tests): update GitLab client tests to use new API
- Updated test fixtures to use GitLabConfig dataclass
- Changed all GitLabClient instantiations to use project_dir and config
- Replaced requests.exceptions with urllib.error for timeout/connection tests
- Updated async tests to patch _fetch_async instead of sync methods
- Fixed retry tests to use urllib.request.urlopen mocking
- Updated method names (get_pipeline_status, get_file_contents)
- Removed test for removed list_projects method
- Fixed blind exception assertions with specific exception types

Fixes 42 errors and 68 test failures related to API changes.
2026-01-27 22:51:06 +02:00
StillKnotKnown 4f0bfec2fb fix(codeql): remove unused imports and variables
- Remove unused imports (datetime, timezone, Path, AsyncMock, MagicMock, RequestException) from test_glab_client.py
- Remove unused imports (AsyncMock, MagicMock) from test_gitlab_webhook_operations.py
- Remove unused imports (Path) from test_gitlab_types.py
- Remove unused imports (Path, AsyncMock, MagicMock) from test_gitlab_triage_engine.py
- Remove unused imports (datetime, timedelta) from tests/test_github_bot_detection.py
- Remove unused 'author' variable in runner.py triage function

These changes resolve CodeQL alerts about unused code while maintaining
all necessary functionality. BotDetector parameter warnings are false
positives - the parameters are correct for each module (GitHub vs GitLab).
2026-01-27 22:19:05 +02:00
StillKnotKnown fdd1bbfe1d ci: trigger CodeQL re-run after fixes
All reported issues from PR review have been addressed:
- Sentry: async methods get_mr_pipeline_async, get_mr_notes_async, get_pipeline_jobs_async exist
- Previous commits fixed all HIGH and MEDIUM blocking issues
- Need fresh CodeQL scan to verify fixes
2026-01-27 22:12:00 +02:00
StillKnotKnown 74a21dbf1d fix(gitlab): address Sentry bug reports for PRData and commit order
Bug 1 (MEDIUM): Add missing fields to fallback PRData dataclass
- Add is_draft: bool = False
- Add mergeable: bool = True
- Add raw_data: dict[str, Any] | None = None
- Prevents TypeError when GitHub modules are unavailable

Bug 2 (HIGH): Fix GitLab commits API order assumption
- GitLab API returns commits in newest-first order, not oldest-first
- Changed commits[-1] to commits[0] in two locations (lines 237, 965)
- Fixes follow-up review tracking to use correct head commit

Fixes: Sentry bug reports
2026-01-27 22:12:00 +02:00
StillKnotKnown c11ebf3606 Merge branch 'develop' into feat-gitlab-full-integration 2026-01-27 22:00:44 +02:00
StillKnotKnown daf0c28c6c Merge branch 'develop' into feat-gitlab-full-integration 2026-01-26 14:26:28 +02:00
StillKnotKnown 8b374707b6 Merge branch 'develop' into feat-gitlab-full-integration 2026-01-26 14:16:42 +02:00
StillKnotKnown dd71c5131f Merge branch 'develop' into feat-gitlab-full-integration 2026-01-26 13:52:26 +02:00
StillKnotKnown 1831b37072 fix(gitlab): address all HIGH and MEDIUM blocking issues from follow-up review
HIGH severity fixes:
- Fix save_batch() static method calls (NEW-003)
  * Lines 193, 210: Use batcher instance instead of class
  * Create batcher instance in exception handler with required params

- Fix ReviewPass.pass_number AttributeError (PROMPT-001)
  * Use review_pass.value instead of non-existent pass_number attribute
  * Fixes runtime error in multi-pass review functionality

MEDIUM severity fixes:
- Add /namespaces/ to VALID_ENDPOINT_PATTERNS (NEW-007/NEW-008)
  * Allows permission checks to use namespace endpoint
  * Fixes ValueError in get_user_role()

- Fix unencoded project paths in API calls (URL-001)
  * Import encode_project_path from glab_client
  * URL-encode all project paths in API calls
  * Fixes 404 errors for namespaced projects

- Add exact username verification (6041ace9b285)
  * Verify returned member username matches query exactly
  * Prevents privilege escalation from fuzzy matching
  * Return NONE if no exact match found
2026-01-26 13:43:42 +02:00
StillKnotKnown 20f049ee17 Merge branch 'develop' into feat-gitlab-full-integration 2026-01-26 13:43:15 +02:00
StillKnotKnown 4efa7ea14f Merge branch 'develop' into feat-gitlab-full-integration 2026-01-26 13:24:17 +02:00
StillKnotKnown 71717c5469 Merge branch 'develop' into feat-gitlab-full-integration 2026-01-26 11:16:42 +02:00
StillKnotKnown 8cf89bb4b8 fix(gitlab): add missing instance_url parameter to MR E2E tests
Fix GitLabConfig initialization in MR E2E tests by adding the
required instance_url parameter.
2026-01-26 11:10:46 +02:00
StillKnotKnown 63891aa816 fix(gitlab): address test import issues and protocol compatibility
This commit fixes multiple test failures and import issues in the GitLab
integration:

1. Fix test fixture imports:
   - Change imports from 'tests.fixtures.gitlab' to '__tests__.fixtures.gitlab'
   - Add missing mock_mr_commits() function to fixtures

2. Fix mock_mr_data() to handle author overrides properly:
   - Use deepcopy to avoid modifying SAMPLE_MR_DATA
   - Handle string author values by updating username field

3. Fix GitLabProvider protocol compatibility:
   - Add fallback protocol type definitions when GitHub runners unavailable
   - Include all required fields for PRData, IssueData, LabelData
   - Add missing fields to ReviewData, PRFilters, IssueFilters

4. Fix GitHub orchestrator import issues:
   - Add try/except blocks for each fallback import
   - Prevent cascading import failures

5. Fix timezone-naive datetime comparisons:
   - Use datetime.now(timezone.utc) in cleanup_stale_mrs()

6. Update test imports:
   - Use local ProviderType enum instead of GitHub protocol
   - Fix ReviewData usage to match protocol signature
2026-01-26 11:05:12 +02:00
StillKnotKnown 125cafcb8d fix(gitlab): address follow-up review findings
HIGH severity fixes:
- Fix GitlabIssueBatcher initialization with missing required parameters
  * Add project and project_dir parameters to batcher instantiation
- Fix instance method save_batch now properly called on batcher instance (already fixed)

MEDIUM severity fixes:
- Fix timezone-naive datetime comparisons in bot_detection.py
  * Use datetime.now(timezone.utc) instead of datetime.now()
  * Import timezone from datetime module
  * Fix mark_reviewed to use timezone-aware timestamps
- Fix test function indentation in test_gitlab_context_gatherer.py
  * Indent test_gather_handles_missing_ci to be inside TestGatherIntegration class
  * Test uses self parameter, so must be instance method
2026-01-26 08:15:07 +02:00
StillKnotKnown 342508ef73 Merge branch 'develop' into feat-gitlab-full-integration 2026-01-26 08:01:58 +02:00
StillKnotKnown 9fd1a125f0 test: remove unused Path import from webhook operations test 2026-01-22 18:50:24 +02:00
StillKnotKnown 611b0fe94a fix(gitlab): address Sentry bug reports
HIGH severity:
- Add get_project_members_async method to glab_client.py
- Fix get_user_role to use await with async method instead of sync
  * Was awaiting non-async get_project_members which would cause TypeError

MEDIUM severity:
- Fix diff hunk regex to handle single-line changes (followup_reviewer.py)
  * Old regex required line counts: r"-(\d+),?(\d+) \+(\d+),?(\d+) @@"
  * New regex handles optional counts: r"@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@"
  * Single-line changes like "@@ -40 +40 @@" are now parsed correctly
  * Defaults to count of 1 when line count is omitted
2026-01-22 18:41:05 +02:00
StillKnotKnown 43da32a8d3 Merge branch 'develop' into feat-gitlab-full-integration 2026-01-22 18:16:46 +02:00
StillKnotKnown d7eaadd87a docs(batch_processor): document circular import avoidance pattern in _report_progress 2026-01-22 18:13:08 +02:00
StillKnotKnown 751e5d6683 fix(gitlab): address CodeQL and review findings
CRITICAL fixes:
- Fix save_batch called as class method instead of instance method (batch_processor.py)
  * Create GitlabIssueBatcher instance before calling save_batch()

MAJOR fixes:
- Parse --labels into list before calling list_issues (runner.py)
  * CLI args.labels is comma-separated string, but list_issues expects list[str]
- Fix review-pass prompt to return fallback instead of empty string (prompt_manager.py)
  * Return main MR review prompt when pass-specific prompt doesn't exist

MEDIUM fixes:
- Fix rc pattern matching to not match filenames like 'src' (context_gatherer.py)
  * Use ".rc" pattern instead of "rc" to require leading dot

Verified already fixed:
- Missing async methods (get_mr_pipeline_async, get_mr_notes_async, get_pipeline_jobs_async)
- safe_print shadowing (local redefinition removed in previous commit)
- ReviewCategory enum (DOCS vs DOCUMENTATION mismatch already corrected)
2026-01-22 17:51:55 +02:00
StillKnotKnown cd514908c1 fix(gitlab): address more Auto Claude PR review findings (10/15)
- Add CANCELLED transition to MR_CREATED state
- Add WAITING_APPROVAL to active_states() method

Fixes findings: 4af0fa53, 52a48e8c

Signed-off-by: StillKnotKnown <stillknotknown@users.noreply.github.com>
2026-01-22 17:13:56 +02:00
StillKnotKnown 7066424d09 fix(gitlab): address Auto Claude PR review findings (7/15)
- Add file locking to BotDetectionState.load() to prevent race conditions
- Replace duplicated io_utils.py with re-export wrapper from core.io_utils
- Remove duplicate safe_print function that shadows imported version
- Remove empty conditional branches with pass statements
- Fix endswith('rc') pattern to use '.rc' to avoid matching 'src'
- Replace print() statements with proper logger.warning() calls
- Add assertions to test_log_permission_denial test
- Rename PermissionError to GitLabPermissionError to avoid shadowing built-in

Fixes findings: e8caaf98, 7d5960bc, 9fe5f71d, 14c4ff3d, 13adc2f3, 416908b8, cfce0a95, 54539d9c

Signed-off-by: StillKnotKnown <stillknotknown@users.noreply.github.com>
2026-01-22 17:11:02 +02:00
StillKnotKnown eda03802fa fix(frontend): add provider-detection module and update ClaudeUsageSnapshot type
- Add src/shared/utils/provider-detection.ts with provider detection utilities
- Update ClaudeUsageSnapshot type to include missing fields from develop:
  - sessionResetTimestamp, weeklyResetTimestamp for ISO timestamps
  - usageWindows for window label metadata
  - sessionUsageValue, sessionUsageLimit for raw session quota data
  - weeklyUsageValue, weeklyUsageLimit for raw weekly quota data

Resolves merge conflict from develop branch where usage-monitor.ts
was using these types but they weren't defined.

Signed-off-by: StillKnotKnown <stillknotknown@users.noreply.github.com>
2026-01-22 12:06:55 +02:00
StillKnotKnown ff7d3539f8 fix(frontend): resolve merge conflict in usage-monitor.ts
The merge from develop brought in changes to usage-monitor.ts that
were not properly merged. The file had 405 lines instead of 1229 lines.

This commit properly resolves the merge by taking the complete file
from the develop branch (3b87e24d).

Signed-off-by: StillKnotKnown <stillknotknown@users.noreply.github.com>
2026-01-22 11:46:52 +02:00
StillKnotKnown 10fdc32617 fix(gitlab): add missing io_utils module and fix followup reviewer tests
- Add io_utils.py module with safe_print function for subprocess output
- Fix ReviewCategory.DOCUMENTATION -> DOCS in followup_reviewer.py
- Fix doc pattern matching to include '+ """' variant
- Fix test assertion to use lowercase 'todo' instead of uppercase
- Fix test diff format to include proper unified diff headers

Signed-off-by: StillKnotKnown <stillknotknown@users.noreply.github.com>
2026-01-21 21:48:39 +02:00
StillKnotKnown c0bdc1c3bd fix(test): update test_check_encoding expectation to match actual behavior
The encoding checker now detects 3 issues instead of 2 in the
test_multiple_issues_in_single_file test:
- 2 open() calls (one in comment, one actual)
- 1 write_text() call

Signed-off-by: StillKnotKnown <stillknotknown@users.noreply.github.com>
2026-01-21 20:45:59 +02:00
StillKnotKnown e74328e644 fix(tests): address CodeRabbit review feedback
- Fix unnecessary __import__(datetime) calls in test_gitlab_bot_detection.py
  The datetime module is already imported, so use datetime.timedelta directly

- Fix incorrect fixture import path in test_gitlab_provider.py
  Change from tests.fixtures.gitlab to __tests__.fixtures.gitlab

These changes address CodeRabbit review comments on PR #1413.
2026-01-21 20:30:20 +02:00
StillKnotKnown 0f1aa01814 fix(gitlab): add missing async methods for pipeline and notes
- Add get_mr_pipeline() to get the latest pipeline for an MR
- Add get_mr_pipeline_async() - async version of get_mr_pipeline
- Add get_mr_notes_async() - async version of get_mr_notes
- Add get_pipeline_jobs_async() - async version of get_pipeline_jobs

These methods were being called by context_gatherer.py and ci_checker.py
but were not implemented, which would cause AttributeError at runtime.

Fixes: https://github.com/AndyMik90/Auto-Claude/pull/1413#issuecomment-XXXX
2026-01-21 19:26:36 +02:00
StillKnotKnown a9689d0369 feat(gitlab): comprehensive integration with API extensions, services, and tests
This commit completes the GitLab integration implementation with full
feature parity to GitHub, including:

Core API Extensions (glab_client.py):
- Merge Request CRUD: create_mr, list_mrs, update_mr
- Branch Management: list_branches, get_branch, create_branch, delete_branch, compare_branches
- File Operations: get_file_contents, create_file, update_file, delete_file
- Webhook Management: list_webhooks, get_webhook, create_webhook, update_webhook, delete_webhook
- Enhanced _fetch method with params support and retry logic

New Service Modules:
- autofix_processor.py: Auto-fix workflow with permission verification
- batch_issues.py: Issue batching with Claude AI similarity detection
- permissions.py: GitLab permission checker for automation triggers
- services/triage_engine.py: AI-driven issue triage and categorization
- services/followup_reviewer.py: Automated MR review for follow-up commits
- services/context_gatherer.py: Enhanced context gathering with monorepo support
- services/prompt_manager.py: System prompt management
- services/response_parsers.py: Parse Claude API responses
- services/batch_processor.py: Batch issue processing

Enhanced Modules:
- models.py: AutoFixState, GitLabRunnerConfig, and other data models
- bot_detection.py: Bot comment detection with import fixes
- utils/file_lock.py: Added encoding parameter to atomic_write
- utils/rate_limiter.py: Updated for GitLab (not GitHub)
- types.py: TypedDict type definitions for API responses

Test Files (12 new test files, 122 tests):
- test_gitlab_client_extensions.py: MR CRUD, branch, file, webhook tests
- test_gitlab_client_errors.py: Retry logic, rate limiting, error handling
- test_gitlab_branch_operations.py: Branch management tests
- test_gitlab_file_operations.py: File operations tests
- test_gitlab_webhook_operations.py: Webhook management tests
- test_gitlab_autofix_processor.py: Auto-fix workflow tests
- test_gitlab_batch_issues.py: Issue batching tests
- test_gitlab_triage_engine.py: Triage engine tests
- test_gitlab_followup_reviewer.py: Follow-up reviewer tests
- test_gitlab_context_gatherer.py: Context gathering tests
- test_gitlab_permissions.py: Permission checker tests
- test_gitlab_types.py: TypedDict type tests

All tests passing (122/122).
2026-01-21 19:13:01 +02:00
StillKnotKnown e5c8f435ff fix: fix encoding issues in file operations and encoding checker
- Fix check_encoding.py to handle nested parentheses in open() calls
- Add encoding="utf-8" to file operations in GitLab test files
- Add encoding to bot_detection.py and file_lock.py
- Fix trailing whitespace and end-of-file issues (auto-fixed)
2026-01-21 17:07:49 +02:00
StillKnotKnown 8f52024956 feat(gitlab): add params support to _fetch for query parameters
- Add params parameter to GitLabClient._fetch() method
- Update runners/__init__.py to comment out missing roadmap_runner import
- Update .gitignore to exclude gitlab-integration-tests directory
2026-01-21 17:01:56 +02:00
StillKnotKnown b9b2d237e2 feat(gitlab): comprehensive integration alignment with GitHub
Implement full GitLab integration parity with GitHub, including multi-pass
MR review, bot detection, CI/CD checking, file locking, rate limiting, and
comprehensive test suite.

Implementation (7 new files, ~2,600 lines):
- providers/gitlab_provider.py: GitProvider protocol implementation
- services/context_gatherer.py: MR context gathering with AI bot comment detection
- services/ci_checker.py: GitLab CI/CD pipeline status checking
- bot_detection.py: Bot detection with cooling-off period
- utils/file_lock.py: Concurrent-safe file operations
- utils/rate_limiter.py: Token bucket rate limiting

Enhanced files (5 files):
- glab_client.py: Added async methods and new API endpoints
- models.py: Added 6-pass review, evidence-based findings, structural issues
- orchestrator.py: Integrated bot detection, CI checking, multi-pass review
- runner.py: Added triage, auto-fix, batch-issues CLI commands
- services/__init__.py: Export new services

Test suite (8 new files, ~2,400 lines):
- test_gitlab_provider.py: GitProvider protocol tests
- test_gitlab_bot_detection.py: Bot detection tests
- test_gitlab_ci_checker.py: CI checker tests
- test_gitlab_mr_review.py: MR review models tests
- test_gitlab_mr_e2e.py: End-to-end review lifecycle tests
- test_gitlab_file_lock.py: File locking tests
- test_gitlab_rate_limiter.py: Rate limiter tests
- test_glab_client.py: Client timeout/retry tests
- fixtures/gitlab.py: Test fixtures

Features implemented:
- Phase 1: GitProvider protocol for GitLab
- Phase 2: 6-pass MR review (quick_scan, security, quality, deep_analysis, structural, ai_comment_triage)
- Phase 3: Bot detection with cooling-off period
- Phase 4: CI/CD pipeline integration
- Phase 5: File locking for concurrent safety
- Phase 6: Token bucket rate limiting
- Phase 7: Enhanced orchestrator with all integrations
- Phase 8: Comprehensive test suite
- Phase 9: CLI commands (triage, auto-fix, batch-issues)
- Phase 10: Frontend parity (IPC handlers already complete)

All code follows Python best practices with type hints, error handling,
logging, and cross-platform compatibility.
2026-01-21 17:01:55 +02:00
2006 changed files with 209668 additions and 144630 deletions
+8 -4
View File
@@ -42,14 +42,18 @@ reviews:
# Path-specific review instructions
path_instructions:
- path: "apps/desktop/**/*.{ts,tsx}"
- path: "apps/backend/**/*.py"
instructions: |
Focus on Python best practices, type hints, and async patterns.
Check for proper error handling and security considerations.
Verify compatibility with Python 3.12+.
- path: "apps/frontend/**/*.{ts,tsx}"
instructions: |
Review React patterns and TypeScript type safety.
Check for proper state management and component composition.
Verify Vercel AI SDK v6 usage patterns and tool definitions.
- path: "apps/desktop/**/*.test.{ts,tsx}"
- path: "tests/**"
instructions: |
Ensure tests are comprehensive and follow Vitest conventions.
Ensure tests are comprehensive and follow pytest conventions.
Check for proper mocking and test isolation.
chat:
+24 -24
View File
@@ -41,7 +41,7 @@ runs:
shell: bash
# Run npm ci from root to properly handle workspace dependencies.
# With npm workspaces, the lock file is at root and dependencies are hoisted there.
# Running npm ci in apps/desktop would fail to populate node_modules correctly.
# Running npm ci in apps/frontend would fail to populate node_modules correctly.
run: |
if [ "${{ inputs.ignore-scripts }}" == "true" ]; then
npm ci --ignore-scripts
@@ -51,12 +51,12 @@ runs:
- name: Link node_modules for electron-builder
shell: bash
# electron-builder expects node_modules in apps/desktop for native module rebuilding.
# electron-builder expects node_modules in apps/frontend for native module rebuilding.
# With npm workspaces, packages are hoisted to root. Create a link so electron-builder
# can find the modules during packaging and code signing.
# Uses symlink on Unix, directory junction on Windows (works without admin privileges).
#
# IMPORTANT: npm workspaces may create a partial node_modules in apps/desktop for
# IMPORTANT: npm workspaces may create a partial node_modules in apps/frontend for
# packages that couldn't be hoisted. We must remove it and create a proper link to root.
run: |
# Verify npm ci succeeded
@@ -65,42 +65,42 @@ runs:
exit 1
fi
# Remove any existing node_modules in apps/desktop
# Remove any existing node_modules in apps/frontend
# This handles: partial directories from npm workspaces, AND broken symlinks
if [ -e "apps/desktop/node_modules" ] || [ -L "apps/desktop/node_modules" ]; then
if [ -e "apps/frontend/node_modules" ] || [ -L "apps/frontend/node_modules" ]; then
# Check if it's a valid symlink pointing to root node_modules
if [ -L "apps/desktop/node_modules" ]; then
target=$(readlink apps/desktop/node_modules 2>/dev/null || echo "")
if [ "$target" = "../../node_modules" ] && [ -d "apps/desktop/node_modules" ]; then
echo "Correct symlink already exists: apps/desktop/node_modules -> ../../node_modules"
if [ -L "apps/frontend/node_modules" ]; then
target=$(readlink apps/frontend/node_modules 2>/dev/null || echo "")
if [ "$target" = "../../node_modules" ] && [ -d "apps/frontend/node_modules" ]; then
echo "Correct symlink already exists: apps/frontend/node_modules -> ../../node_modules"
else
echo "Removing incorrect/broken symlink (was: $target)..."
rm -f "apps/desktop/node_modules"
rm -f "apps/frontend/node_modules"
fi
else
echo "Removing partial node_modules directory created by npm workspaces..."
rm -rf "apps/desktop/node_modules"
rm -rf "apps/frontend/node_modules"
fi
fi
# Create link if it doesn't exist or was removed
if [ ! -L "apps/desktop/node_modules" ]; then
if [ ! -L "apps/frontend/node_modules" ]; then
if [ "$RUNNER_OS" == "Windows" ]; then
# Use directory junction on Windows (works without admin privileges)
# Use PowerShell's New-Item -ItemType Junction for reliable path handling
abs_target=$(cygpath -w "$(pwd)/node_modules")
link_path=$(cygpath -w "$(pwd)/apps/desktop/node_modules")
link_path=$(cygpath -w "$(pwd)/apps/frontend/node_modules")
powershell -Command "New-Item -ItemType Junction -Path '$link_path' -Target '$abs_target'" > /dev/null
if [ $? -eq 0 ]; then
echo "Created junction: apps/desktop/node_modules -> $abs_target"
echo "Created junction: apps/frontend/node_modules -> $abs_target"
else
echo "::error::Failed to create directory junction on Windows"
exit 1
fi
else
# Use symlink on Unix (macOS/Linux)
if ln -s ../../node_modules apps/desktop/node_modules; then
echo "Created symlink: apps/desktop/node_modules -> ../../node_modules"
if ln -s ../../node_modules apps/frontend/node_modules; then
echo "Created symlink: apps/frontend/node_modules -> ../../node_modules"
else
echo "::error::Failed to create symlink"
exit 1
@@ -111,16 +111,16 @@ runs:
# Final verification - the link must exist and resolve correctly
# Note: On Windows, junctions don't show as symlinks (-L), so we check if the directory exists
# and can be listed. On Unix, we also verify it's a symlink.
if [ "$RUNNER_OS" != "Windows" ] && [ ! -L "apps/desktop/node_modules" ]; then
echo "::error::apps/desktop/node_modules symlink was not created"
if [ "$RUNNER_OS" != "Windows" ] && [ ! -L "apps/frontend/node_modules" ]; then
echo "::error::apps/frontend/node_modules symlink was not created"
exit 1
fi
# Verify the link resolves to a valid directory with content
if ! ls apps/desktop/node_modules/electron >/dev/null 2>&1; then
echo "::error::apps/desktop/node_modules does not resolve correctly (electron not found)"
ls -la apps/desktop/ || true
ls apps/desktop/node_modules 2>&1 | head -5 || true
if ! ls apps/frontend/node_modules/electron >/dev/null 2>&1; then
echo "::error::apps/frontend/node_modules does not resolve correctly (electron not found)"
ls -la apps/frontend/ || true
ls apps/frontend/node_modules 2>&1 | head -5 || true
exit 1
fi
count=$(ls apps/desktop/node_modules 2>/dev/null | wc -l)
echo "Verified: apps/desktop/node_modules resolves correctly ($count entries)"
count=$(ls apps/frontend/node_modules 2>/dev/null | wc -l)
echo "Verified: apps/frontend/node_modules resolves correctly ($count entries)"
@@ -0,0 +1,52 @@
name: 'Setup Python Backend'
description: 'Set up Python with uv package manager and cached dependencies for the backend'
inputs:
python-version:
description: 'Python version to use'
required: false
default: '3.12'
install-test-deps:
description: 'Whether to install test dependencies'
required: false
default: 'false'
outputs:
cache-hit:
description: 'Whether cache was hit'
value: ${{ steps.cache.outputs.cache-hit }}
runs:
using: 'composite'
steps:
- name: Set up Python ${{ inputs.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ inputs.python-version }}
- name: Install uv package manager
uses: astral-sh/setup-uv@v4
with:
version: "latest"
- name: Cache uv dependencies
id: cache
uses: actions/cache@v4
with:
path: |
~/.cache/uv
~/AppData/Local/uv/cache
~/Library/Caches/uv
key: uv-${{ runner.os }}-${{ runner.arch }}-${{ inputs.python-version }}-${{ hashFiles('apps/backend/requirements.txt', 'tests/requirements-test.txt') }}
restore-keys: |
uv-${{ runner.os }}-${{ runner.arch }}-${{ inputs.python-version }}-
- name: Install dependencies
working-directory: apps/backend
shell: bash
run: |
uv venv
uv pip install -r requirements.txt
if [ "${{ inputs.install-test-deps }}" == "true" ]; then
uv pip install -r ../../tests/requirements-test.txt
fi
@@ -14,7 +14,7 @@ inputs:
dmg-path:
description: 'Path to the dist directory containing the DMG file'
required: false
default: 'apps/desktop/dist'
default: 'apps/frontend/dist'
outputs:
notarization-id:
+13 -1
View File
@@ -1,8 +1,20 @@
version: 2
updates:
# Python dependencies
- package-ecosystem: pip
directory: /apps/backend
schedule:
interval: weekly
open-pull-requests-limit: 5
labels:
- dependencies
- python
commit-message:
prefix: "chore(deps)"
# npm dependencies
- package-ecosystem: npm
directory: /apps/desktop
directory: /apps/frontend
schedule:
interval: weekly
open-pull-requests-limit: 5
+112 -28
View File
@@ -74,11 +74,35 @@ jobs:
# Use tag for real releases, develop branch for dry runs
ref: ${{ github.event.inputs.dry_run == 'true' && 'develop' || format('v{0}', needs.create-tag.outputs.version) }}
- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Setup Node.js and install dependencies
uses: ./.github/actions/setup-node-frontend
- name: Install Rust toolchain (for building native Python packages)
uses: dtolnay/rust-toolchain@stable
- name: Cache pip wheel cache (for compiled packages like real_ladybug)
uses: actions/cache@v5
with:
path: ~/Library/Caches/pip
key: pip-wheel-${{ runner.os }}-x64-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
pip-wheel-${{ runner.os }}-x64-
- name: Cache bundled Python
uses: actions/cache@v5
with:
path: apps/frontend/python-runtime
key: python-bundle-${{ runner.os }}-x64-3.12.8-rust-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
python-bundle-${{ runner.os }}-x64-3.12.8-rust-
- name: Build application
run: cd apps/desktop && npm run build
run: cd apps/frontend && npm run build
env:
SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
SENTRY_TRACES_SAMPLE_RATE: ${{ secrets.SENTRY_TRACES_SAMPLE_RATE }}
@@ -87,7 +111,7 @@ jobs:
- name: Package macOS (Intel)
run: |
VERSION="${{ needs.create-tag.outputs.version }}"
cd apps/desktop && npm run package:mac -- --x64 --config.extraMetadata.version="$VERSION"
cd apps/frontend && npm run package:mac -- --x64 --config.extraMetadata.version="$VERSION"
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
CSC_LINK: ${{ secrets.MAC_CERTIFICATE }}
@@ -109,9 +133,9 @@ jobs:
with:
name: macos-intel-builds
path: |
apps/desktop/dist/*.dmg
apps/desktop/dist/*.zip
apps/desktop/dist/*.yml
apps/frontend/dist/*.dmg
apps/frontend/dist/*.zip
apps/frontend/dist/*.yml
# Apple Silicon build on ARM64 runner for native compilation
build-macos-arm64:
@@ -126,11 +150,32 @@ jobs:
# Use tag for real releases, develop branch for dry runs
ref: ${{ github.event.inputs.dry_run == 'true' && 'develop' || format('v{0}', needs.create-tag.outputs.version) }}
- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Setup Node.js and install dependencies
uses: ./.github/actions/setup-node-frontend
- name: Cache pip wheel cache
uses: actions/cache@v5
with:
path: ~/Library/Caches/pip
key: pip-wheel-${{ runner.os }}-arm64-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
pip-wheel-${{ runner.os }}-arm64-
- name: Cache bundled Python
uses: actions/cache@v5
with:
path: apps/frontend/python-runtime
key: python-bundle-${{ runner.os }}-arm64-3.12.8-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
python-bundle-${{ runner.os }}-arm64-3.12.8-
- name: Build application
run: cd apps/desktop && npm run build
run: cd apps/frontend && npm run build
env:
SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
SENTRY_TRACES_SAMPLE_RATE: ${{ secrets.SENTRY_TRACES_SAMPLE_RATE }}
@@ -139,7 +184,7 @@ jobs:
- name: Package macOS (Apple Silicon)
run: |
VERSION="${{ needs.create-tag.outputs.version }}"
cd apps/desktop && npm run package:mac -- --arm64 --config.extraMetadata.version="$VERSION"
cd apps/frontend && npm run package:mac -- --arm64 --config.extraMetadata.version="$VERSION"
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
CSC_LINK: ${{ secrets.MAC_CERTIFICATE }}
@@ -161,9 +206,9 @@ jobs:
with:
name: macos-arm64-builds
path: |
apps/desktop/dist/*.dmg
apps/desktop/dist/*.zip
apps/desktop/dist/*.yml
apps/frontend/dist/*.dmg
apps/frontend/dist/*.zip
apps/frontend/dist/*.yml
build-windows:
needs: create-tag
@@ -180,11 +225,32 @@ jobs:
# Use tag for real releases, develop branch for dry runs
ref: ${{ github.event.inputs.dry_run == 'true' && 'develop' || format('v{0}', needs.create-tag.outputs.version) }}
- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Setup Node.js and install dependencies
uses: ./.github/actions/setup-node-frontend
- name: Cache pip wheel cache
uses: actions/cache@v5
with:
path: ~\AppData\Local\pip\Cache
key: pip-wheel-${{ runner.os }}-x64-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
pip-wheel-${{ runner.os }}-x64-
- name: Cache bundled Python
uses: actions/cache@v5
with:
path: apps/frontend/python-runtime
key: python-bundle-${{ runner.os }}-x64-3.12.8-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
python-bundle-${{ runner.os }}-x64-3.12.8-
- name: Build application
run: cd apps/desktop && npm run build
run: cd apps/frontend && npm run build
env:
SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
SENTRY_TRACES_SAMPLE_RATE: ${{ secrets.SENTRY_TRACES_SAMPLE_RATE }}
@@ -194,7 +260,7 @@ jobs:
shell: bash
run: |
VERSION="${{ needs.create-tag.outputs.version }}"
cd apps/desktop && npm run package:win -- --config.extraMetadata.version="$VERSION"
cd apps/frontend && npm run package:win -- --config.extraMetadata.version="$VERSION"
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Disable electron-builder's built-in signing (we use Azure Trusted Signing instead)
@@ -218,7 +284,7 @@ jobs:
endpoint: https://neu.codesigning.azure.net/
trusted-signing-account-name: ${{ secrets.AZURE_SIGNING_ACCOUNT }}
certificate-profile-name: ${{ secrets.AZURE_CERTIFICATE_PROFILE }}
files-folder: apps/desktop/dist
files-folder: apps/frontend/dist
files-folder-filter: exe
file-digest: SHA256
timestamp-rfc3161: http://timestamp.acs.microsoft.com
@@ -228,7 +294,7 @@ jobs:
if: env.AZURE_CLIENT_ID != ''
shell: pwsh
run: |
cd apps/desktop/dist
cd apps/frontend/dist
$exeFile = Get-ChildItem -Filter "*.exe" | Select-Object -First 1
if ($exeFile) {
Write-Host "Verifying signature on $($exeFile.Name)..."
@@ -252,7 +318,7 @@ jobs:
shell: pwsh
run: |
$ErrorActionPreference = "Stop"
cd apps/desktop/dist
cd apps/frontend/dist
# Find the installer exe (electron-builder names it with "Setup" or just the app name)
# electron-builder produces one installer exe per build
@@ -319,8 +385,8 @@ jobs:
with:
name: windows-builds
path: |
apps/desktop/dist/*.exe
apps/desktop/dist/*.yml
apps/frontend/dist/*.exe
apps/frontend/dist/*.yml
build-linux:
needs: create-tag
@@ -331,20 +397,41 @@ jobs:
# Use tag for real releases, develop branch for dry runs
ref: ${{ github.event.inputs.dry_run == 'true' && 'develop' || format('v{0}', needs.create-tag.outputs.version) }}
- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Setup Node.js and install dependencies
uses: ./.github/actions/setup-node-frontend
- name: Setup Flatpak and verification tools
- name: Setup Flatpak
run: |
set -e
sudo apt-get update
sudo apt-get install -y flatpak flatpak-builder squashfs-tools
sudo apt-get install -y flatpak flatpak-builder
flatpak remote-add --user --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
flatpak install -y --user flathub org.freedesktop.Platform//25.08 org.freedesktop.Sdk//25.08
flatpak install -y --user flathub org.electronjs.Electron2.BaseApp//25.08
- name: Cache pip wheel cache
uses: actions/cache@v5
with:
path: ~/.cache/pip
key: pip-wheel-${{ runner.os }}-x64-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
pip-wheel-${{ runner.os }}-x64-
- name: Cache bundled Python
uses: actions/cache@v5
with:
path: apps/frontend/python-runtime
key: python-bundle-${{ runner.os }}-x64-3.12.8-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
python-bundle-${{ runner.os }}-x64-3.12.8-
- name: Build application
run: cd apps/desktop && npm run build
run: cd apps/frontend && npm run build
env:
SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
SENTRY_TRACES_SAMPLE_RATE: ${{ secrets.SENTRY_TRACES_SAMPLE_RATE }}
@@ -353,25 +440,22 @@ jobs:
- name: Package Linux
run: |
VERSION="${{ needs.create-tag.outputs.version }}"
cd apps/desktop && npm run package:linux -- --config.extraMetadata.version="$VERSION"
cd apps/frontend && npm run package:linux -- --config.extraMetadata.version="$VERSION"
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
SENTRY_TRACES_SAMPLE_RATE: ${{ secrets.SENTRY_TRACES_SAMPLE_RATE }}
SENTRY_PROFILES_SAMPLE_RATE: ${{ secrets.SENTRY_PROFILES_SAMPLE_RATE }}
- name: Verify Linux packages
run: cd apps/desktop && npm run verify:linux
- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: linux-builds
path: |
apps/desktop/dist/*.AppImage
apps/desktop/dist/*.deb
apps/desktop/dist/*.flatpak
apps/desktop/dist/*.yml
apps/frontend/dist/*.AppImage
apps/frontend/dist/*.deb
apps/frontend/dist/*.flatpak
apps/frontend/dist/*.yml
# Finalize macOS notarization (runs in parallel with Windows/Linux builds)
finalize-notarization:
+5 -5
View File
@@ -38,7 +38,7 @@ jobs:
uses: microsoft/setup-msbuild@v2
- name: Install node-pty and rebuild for Electron
working-directory: apps/desktop
working-directory: apps/frontend
shell: pwsh
run: |
# Install only node-pty
@@ -52,7 +52,7 @@ jobs:
npx @electron/rebuild --version $env:ELECTRON_VERSION --module-dir node_modules/node-pty --arch ${{ matrix.arch }}
- name: Package prebuilt binaries
working-directory: apps/desktop
working-directory: apps/frontend
shell: pwsh
run: |
$electronAbi = (npx electron-abi $env:ELECTRON_VERSION)
@@ -78,7 +78,7 @@ jobs:
Get-ChildItem $prebuildDir
- name: Create archive
working-directory: apps/desktop
working-directory: apps/frontend
shell: pwsh
run: |
$electronAbi = (npx electron-abi $env:ELECTRON_VERSION)
@@ -93,14 +93,14 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: node-pty-win32-${{ matrix.arch }}
path: apps/desktop/node-pty-*.zip
path: apps/frontend/node-pty-*.zip
retention-days: 90
- name: Upload to release
if: github.event_name == 'release'
uses: softprops/action-gh-release@v1
with:
files: apps/desktop/node-pty-*.zip
files: apps/frontend/node-pty-*.zip
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+80 -34
View File
@@ -3,7 +3,8 @@
# Tests on all target platforms (Linux, Windows, macOS) to catch
# platform-specific bugs before they merge. ALL platforms must pass.
#
# Optimized: Frontend-only matrix, path filters to skip on docs-only changes.
# Optimized: Reduced matrix (4 jobs vs 6), merged integration tests,
# coverage on Linux only, path filters to skip on docs-only changes.
name: CI
@@ -12,7 +13,10 @@ on:
branches: [main, develop]
paths:
- 'apps/**'
- 'tests/**'
- 'package*.json'
- 'requirements*.txt'
- 'pyproject.toml'
- 'tsconfig*.json'
- 'biome.jsonc'
- '.github/workflows/ci.yml'
@@ -21,7 +25,10 @@ on:
branches: [main, develop]
paths:
- 'apps/**'
- 'tests/**'
- 'package*.json'
- 'requirements*.txt'
- 'pyproject.toml'
- 'tsconfig*.json'
- 'biome.jsonc'
- '.github/workflows/ci.yml'
@@ -34,9 +41,72 @@ concurrency:
permissions:
contents: read
actions: read
pull-requests: write
jobs:
# --------------------------------------------------------------------------
# Python Backend Tests - Optimized Matrix (4 jobs instead of 6)
# --------------------------------------------------------------------------
test-python:
name: test-python (${{ matrix.python-version }}, ${{ matrix.os }})
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
# 3.12 on all OS for cross-platform coverage
# 3.13 on Linux only for compatibility check (saves 2 jobs)
include:
- os: ubuntu-latest
python-version: '3.12'
- os: ubuntu-latest
python-version: '3.13'
- os: windows-latest
python-version: '3.12'
- os: macos-latest
python-version: '3.12'
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup Python backend
uses: ./.github/actions/setup-python-backend
with:
python-version: ${{ matrix.python-version }}
install-test-deps: 'true'
- name: Run all tests (including platform-specific)
working-directory: apps/backend
shell: bash
env:
PYTHONPATH: ${{ github.workspace }}/apps/backend
run: |
if [ "$RUNNER_OS" == "Windows" ]; then
source .venv/Scripts/activate
else
source .venv/bin/activate
fi
pytest ../../tests/ -v --tb=short -x
- name: Run coverage (Linux + Python 3.12 only)
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12'
working-directory: apps/backend
shell: bash
env:
PYTHONPATH: ${{ github.workspace }}/apps/backend
run: |
source .venv/bin/activate
pytest ../../tests/ -v --cov=. --cov-report=xml --cov-report=term-missing --cov-fail-under=10
- name: Upload coverage to Codecov
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12'
uses: codecov/codecov-action@v4
with:
file: ./apps/backend/coverage.xml
fail_ci_if_error: false
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
# --------------------------------------------------------------------------
# Frontend Tests - All Platforms
# --------------------------------------------------------------------------
@@ -59,41 +129,15 @@ jobs:
ignore-scripts: 'true'
- name: Run TypeScript type check
working-directory: apps/desktop
working-directory: apps/frontend
run: npm run typecheck
- name: Run unit tests with coverage
if: matrix.os == 'ubuntu-latest'
working-directory: apps/desktop
run: npm run test:coverage
- name: Run unit tests
if: matrix.os != 'ubuntu-latest'
working-directory: apps/desktop
run: npm run test:unit
- name: Run integration tests
working-directory: apps/desktop
run: npm run test:integration
- name: Upload coverage report
if: matrix.os == 'ubuntu-latest' && always()
uses: actions/upload-artifact@v4
with:
name: coverage-report
path: apps/desktop/coverage/
retention-days: 14
- name: Coverage PR comment
if: matrix.os == 'ubuntu-latest' && github.event_name == 'pull_request'
uses: davelosert/vitest-coverage-report-action@v2
with:
working-directory: apps/desktop
json-summary-path: coverage/coverage-summary.json
json-final-path: coverage/coverage-final.json
working-directory: apps/frontend
run: npm run test
- name: Build application
working-directory: apps/desktop
working-directory: apps/frontend
run: npm run build
# --------------------------------------------------------------------------
@@ -102,16 +146,18 @@ jobs:
ci-complete:
name: CI Complete
runs-on: ubuntu-latest
needs: [test-frontend]
needs: [test-python, test-frontend]
if: always()
steps:
- name: Check all CI jobs passed
run: |
echo "CI Job Results:"
echo " test-python: ${{ needs.test-python.result }}"
echo " test-frontend: ${{ needs.test-frontend.result }}"
echo ""
if [[ "${{ needs.test-frontend.result }}" != "success" ]]; then
if [[ "${{ needs.test-python.result }}" != "success" ]] || \
[[ "${{ needs.test-frontend.result }}" != "success" ]]; then
echo "❌ One or more CI jobs failed"
exit 1
fi
-1
View File
@@ -3,7 +3,6 @@ name: Discord Release Notification
on:
release:
types: [published]
workflow_dispatch:
jobs:
discord-notification:
-62
View File
@@ -1,62 +0,0 @@
# E2E Tests
#
# Runs Playwright E2E tests for the Electron desktop app on Linux.
# Ubuntu-only since Electron E2E is platform-agnostic (Chromium renderer).
# Non-blocking initially — separate from ci-complete gate while stabilizing.
name: E2E
on:
push:
branches: [main, develop]
paths:
- 'apps/**'
- '.github/workflows/e2e.yml'
pull_request:
branches: [main, develop]
paths:
- 'apps/**'
- '.github/workflows/e2e.yml'
concurrency:
group: e2e-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
e2e:
name: E2E Tests
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup Node.js frontend
uses: ./.github/actions/setup-node-frontend
- name: Install Playwright browsers
working-directory: apps/desktop
run: npx playwright install --with-deps chromium
- name: Build application
working-directory: apps/desktop
run: npm run build
- name: Run E2E tests
working-directory: apps/desktop
continue-on-error: true # Non-blocking while stabilizing — pre-existing __dirname ESM issue
run: xvfb-run --auto-servernum npm run test:e2e
- name: Upload E2E report
if: failure()
uses: actions/upload-artifact@v4
with:
name: e2e-report
path: |
apps/desktop/e2e/playwright-report/
apps/desktop/e2e/test-results/
retention-days: 14
+36 -7
View File
@@ -4,23 +4,50 @@ on:
push:
branches: [main, develop]
paths:
- 'apps/desktop/**'
- 'apps/**'
- 'tests/**'
- '.github/workflows/lint.yml'
- '.github/actions/**'
- 'apps/desktop/biome.jsonc'
- 'apps/frontend/biome.jsonc'
- '.pre-commit-config.yaml'
pull_request:
branches: [main, develop]
paths:
- 'apps/desktop/**'
- 'apps/**'
- 'tests/**'
- '.github/workflows/lint.yml'
- '.github/actions/**'
- 'apps/desktop/biome.jsonc'
- 'apps/frontend/biome.jsonc'
- '.pre-commit-config.yaml'
concurrency:
group: lint-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
# Python linting (Ruff) - already fast, no changes needed
python:
name: Python (Ruff)
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: '3.12'
# Pin ruff version to match .pre-commit-config.yaml
- name: Install ruff
run: pip install ruff==0.14.10
- name: Run ruff check
run: ruff check apps/backend/ --output-format=github
- name: Run ruff format check
run: ruff format apps/backend/ --check --diff
# TypeScript/JavaScript linting (Biome) - 15-25x faster than ESLint
typescript:
name: TypeScript (Biome)
@@ -36,7 +63,7 @@ jobs:
version: 2.3.11
- name: Run Biome
working-directory: apps/desktop
working-directory: apps/frontend
# biome ci fails on errors by default; warnings are reported but don't block
# Use --error-on-warnings when ready to enforce all rules
run: biome ci .
@@ -47,13 +74,15 @@ jobs:
lint-complete:
name: Lint Complete
runs-on: ubuntu-latest
needs: [typescript]
needs: [python, typescript]
if: always()
steps:
- name: Check lint results
run: |
if [[ "${{ needs.typescript.result }}" != "success" ]]; then
if [[ "${{ needs.python.result }}" != "success" ]] || \
[[ "${{ needs.typescript.result }}" != "success" ]]; then
echo "❌ Linting failed"
echo " Python: ${{ needs.python.result }}"
echo " TypeScript: ${{ needs.typescript.result }}"
exit 1
fi
+9 -5
View File
@@ -56,14 +56,15 @@ jobs:
// Area detection paths
AREA_PATHS: Object.freeze({
frontend: 'apps/desktop/',
frontend: 'apps/frontend/',
backend: 'apps/backend/',
ci: '.github/'
}),
// Label definitions
LABELS: Object.freeze({
SIZE: ['size/XS', 'size/S', 'size/M', 'size/L', 'size/XL'],
AREA: ['area/frontend', 'area/ci']
AREA: ['area/frontend', 'area/backend', 'area/fullstack', 'area/ci']
}),
// Pagination
@@ -116,15 +117,16 @@ jobs:
/**
* Detect areas affected by file changes
* @param {Array} files - List of changed files
* @returns {{frontend: boolean, ci: boolean}}
* @returns {{frontend: boolean, backend: boolean, ci: boolean}}
*/
function detectAreas(files) {
const areas = { frontend: false, ci: false };
const areas = { frontend: false, backend: false, ci: false };
const { AREA_PATHS } = CONFIG;
for (const file of files) {
const path = file.filename || '';
if (path.startsWith(AREA_PATHS.frontend)) areas.frontend = true;
if (path.startsWith(AREA_PATHS.backend)) areas.backend = true;
if (path.startsWith(AREA_PATHS.ci)) areas.ci = true;
}
@@ -133,11 +135,13 @@ jobs:
/**
* Determine area label based on detected areas
* @param {{frontend: boolean, ci: boolean}} areas
* @param {{frontend: boolean, backend: boolean, ci: boolean}} areas
* @returns {string|null} Area label or null
*/
function determineAreaLabel(areas) {
if (areas.frontend && areas.backend) return 'area/fullstack';
if (areas.frontend) return 'area/frontend';
if (areas.backend) return 'area/backend';
if (areas.ci) return 'area/ci';
return null;
}
+2 -2
View File
@@ -10,7 +10,7 @@ on:
push:
branches: [main]
paths:
- 'apps/desktop/package.json'
- 'apps/frontend/package.json'
- 'package.json'
workflow_dispatch:
inputs:
@@ -50,7 +50,7 @@ jobs:
- name: Get package version
id: package
run: |
VERSION=$(node -p "require('./apps/desktop/package.json').version")
VERSION=$(node -p "require('./apps/frontend/package.json').version")
echo "version=$VERSION" >> $GITHUB_OUTPUT
echo "Package version: $VERSION"
+93 -6
View File
@@ -1,19 +1,24 @@
name: Quality Security
# CodeQL runs on all PRs, pushes to main, and weekly schedule
# Note: CodeQL takes 20-30 min
# Note: CodeQL takes 20-30 min per language (40-60 min total)
# Bandit is fast (5-10 min)
on:
push:
branches: [main]
paths:
- 'apps/desktop/**'
- 'apps/**'
- 'tests/**'
- 'pyproject.toml'
- 'package.json'
- '.github/workflows/quality-security.yml'
pull_request:
branches: [main, develop]
paths:
- 'apps/desktop/**'
- 'apps/**'
- 'tests/**'
- 'pyproject.toml'
- 'package.json'
- '.github/workflows/quality-security.yml'
schedule:
@@ -36,7 +41,7 @@ jobs:
strategy:
fail-fast: false
matrix:
language: [javascript-typescript]
language: [python, javascript-typescript]
steps:
- name: Checkout
uses: actions/checkout@v4
@@ -55,13 +60,91 @@ jobs:
with:
category: "/language:${{ matrix.language }}"
# Bandit runs on all PRs - it's fast (5-10 min)
python-security:
name: Python Security (Bandit)
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: '3.12'
- name: Install Bandit
run: pip install bandit
- name: Run Bandit security scan
id: bandit
run: |
echo "::group::Running Bandit security scan"
bandit -r apps/backend/ -ll -ii -f json -o bandit-report.json || BANDIT_EXIT=$?
if [ "${BANDIT_EXIT:-0}" -gt 1 ]; then
echo "::error::Bandit scan failed with exit code $BANDIT_EXIT"
exit 1
fi
echo "::endgroup::"
- name: Analyze Bandit results
uses: actions/github-script@v8
with:
script: |
const fs = require('fs');
if (!fs.existsSync('bandit-report.json')) {
core.setFailed('Bandit report not found - scan may have failed');
return;
}
const report = JSON.parse(fs.readFileSync('bandit-report.json', 'utf8'));
const results = report.results || [];
const high = results.filter(r => r.issue_severity === 'HIGH');
const medium = results.filter(r => r.issue_severity === 'MEDIUM');
const low = results.filter(r => r.issue_severity === 'LOW');
console.log(`::group::Bandit Security Scan Results`);
console.log(`Found ${results.length} issues:`);
console.log(` HIGH: ${high.length}`);
console.log(` MEDIUM: ${medium.length}`);
console.log(` LOW: ${low.length}`);
console.log('::endgroup::');
let summary = `## Python Security Scan (Bandit)\n\n`;
summary += `| Severity | Count |\n`;
summary += `|----------|-------|\n`;
summary += `| High | ${high.length} |\n`;
summary += `| Medium | ${medium.length} |\n`;
summary += `| Low | ${low.length} |\n\n`;
if (high.length > 0) {
summary += `### High Severity Issues\n\n`;
for (const issue of high) {
summary += `- **${issue.filename}:${issue.line_number}**\n`;
summary += ` - ${issue.issue_text}\n`;
summary += ` - Test: \`${issue.test_id}\` (${issue.test_name})\n\n`;
}
}
core.summary.addRaw(summary);
await core.summary.write();
if (high.length > 0) {
core.setFailed(`Found ${high.length} high severity security issue(s)`);
} else {
console.log('No high severity security issues found');
}
# --------------------------------------------------------------------------
# Gate Job - Single check for branch protection
# --------------------------------------------------------------------------
security-summary:
name: Security Summary
runs-on: ubuntu-latest
needs: [codeql]
needs: [codeql, python-security]
if: always()
timeout-minutes: 5
steps:
@@ -70,15 +153,19 @@ jobs:
with:
script: |
const codeql = '${{ needs.codeql.result }}';
const bandit = '${{ needs.python-security.result }}';
console.log('Security Check Results:');
console.log(` CodeQL: ${codeql}`);
console.log(` Bandit: ${bandit}`);
// Only 'failure' is a real failure; 'skipped' is acceptable (e.g., path filters, PR skipping CodeQL)
const acceptable = ['success', 'skipped'];
const codeqlOk = acceptable.includes(codeql);
const banditOk = acceptable.includes(bandit);
const allPassed = codeqlOk && banditOk;
if (codeqlOk) {
if (allPassed) {
console.log('\n✅ All security checks passed');
core.summary.addRaw('## ✅ Security Checks Passed\n\nAll security scans completed successfully.');
} else {
+119 -35
View File
@@ -29,18 +29,42 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Setup Node.js and install dependencies
uses: ./.github/actions/setup-node-frontend
- name: Install Rust toolchain (for building native Python packages)
uses: dtolnay/rust-toolchain@stable
- name: Cache pip wheel cache (for compiled packages like real_ladybug)
uses: actions/cache@v5
with:
path: ~/Library/Caches/pip
key: pip-wheel-${{ runner.os }}-x64-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
pip-wheel-${{ runner.os }}-x64-
- name: Cache bundled Python
uses: actions/cache@v5
with:
path: apps/frontend/python-runtime
key: python-bundle-${{ runner.os }}-x64-3.12.8-rust-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
python-bundle-${{ runner.os }}-x64-3.12.8-rust-
- name: Build application
run: cd apps/desktop && npm run build
run: cd apps/frontend && npm run build
env:
SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
SENTRY_TRACES_SAMPLE_RATE: ${{ secrets.SENTRY_TRACES_SAMPLE_RATE }}
SENTRY_PROFILES_SAMPLE_RATE: ${{ secrets.SENTRY_PROFILES_SAMPLE_RATE }}
- name: Package macOS (Intel)
run: cd apps/desktop && npm run package:mac -- --x64
run: cd apps/frontend && npm run package:mac -- --x64
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
CSC_LINK: ${{ secrets.MAC_CERTIFICATE }}
@@ -62,10 +86,10 @@ jobs:
with:
name: macos-intel-builds
path: |
apps/desktop/dist/*.dmg
apps/desktop/dist/*.zip
apps/desktop/dist/*.yml
apps/desktop/dist/*.blockmap
apps/frontend/dist/*.dmg
apps/frontend/dist/*.zip
apps/frontend/dist/*.yml
apps/frontend/dist/*.blockmap
# Apple Silicon build on ARM64 runner for native compilation
build-macos-arm64:
@@ -76,18 +100,39 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Setup Node.js and install dependencies
uses: ./.github/actions/setup-node-frontend
- name: Cache pip wheel cache
uses: actions/cache@v5
with:
path: ~/Library/Caches/pip
key: pip-wheel-${{ runner.os }}-arm64-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
pip-wheel-${{ runner.os }}-arm64-
- name: Cache bundled Python
uses: actions/cache@v5
with:
path: apps/frontend/python-runtime
key: python-bundle-${{ runner.os }}-arm64-3.12.8-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
python-bundle-${{ runner.os }}-arm64-3.12.8-
- name: Build application
run: cd apps/desktop && npm run build
run: cd apps/frontend && npm run build
env:
SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
SENTRY_TRACES_SAMPLE_RATE: ${{ secrets.SENTRY_TRACES_SAMPLE_RATE }}
SENTRY_PROFILES_SAMPLE_RATE: ${{ secrets.SENTRY_PROFILES_SAMPLE_RATE }}
- name: Package macOS (Apple Silicon)
run: cd apps/desktop && npm run package:mac -- --arm64
run: cd apps/frontend && npm run package:mac -- --arm64
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
CSC_LINK: ${{ secrets.MAC_CERTIFICATE }}
@@ -109,10 +154,10 @@ jobs:
with:
name: macos-arm64-builds
path: |
apps/desktop/dist/*.dmg
apps/desktop/dist/*.zip
apps/desktop/dist/*.yml
apps/desktop/dist/*.blockmap
apps/frontend/dist/*.dmg
apps/frontend/dist/*.zip
apps/frontend/dist/*.yml
apps/frontend/dist/*.blockmap
build-windows:
runs-on: windows-latest
@@ -125,18 +170,39 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Setup Node.js and install dependencies
uses: ./.github/actions/setup-node-frontend
- name: Cache pip wheel cache
uses: actions/cache@v5
with:
path: ~\AppData\Local\pip\Cache
key: pip-wheel-${{ runner.os }}-x64-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
pip-wheel-${{ runner.os }}-x64-
- name: Cache bundled Python
uses: actions/cache@v5
with:
path: apps/frontend/python-runtime
key: python-bundle-${{ runner.os }}-x64-3.12.8-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
python-bundle-${{ runner.os }}-x64-3.12.8-
- name: Build application
run: cd apps/desktop && npm run build
run: cd apps/frontend && npm run build
env:
SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
SENTRY_TRACES_SAMPLE_RATE: ${{ secrets.SENTRY_TRACES_SAMPLE_RATE }}
SENTRY_PROFILES_SAMPLE_RATE: ${{ secrets.SENTRY_PROFILES_SAMPLE_RATE }}
- name: Package Windows
run: cd apps/desktop && npm run package:win
run: cd apps/frontend && npm run package:win
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Disable electron-builder's built-in signing (we use Azure Trusted Signing instead)
@@ -160,7 +226,7 @@ jobs:
endpoint: https://neu.codesigning.azure.net/
trusted-signing-account-name: ${{ secrets.AZURE_SIGNING_ACCOUNT }}
certificate-profile-name: ${{ secrets.AZURE_CERTIFICATE_PROFILE }}
files-folder: apps/desktop/dist
files-folder: apps/frontend/dist
files-folder-filter: exe
file-digest: SHA256
timestamp-rfc3161: http://timestamp.acs.microsoft.com
@@ -170,7 +236,7 @@ jobs:
if: env.AZURE_CLIENT_ID != ''
shell: pwsh
run: |
cd apps/desktop/dist
cd apps/frontend/dist
$exeFile = Get-ChildItem -Filter "*.exe" | Select-Object -First 1
if ($exeFile) {
Write-Host "Verifying signature on $($exeFile.Name)..."
@@ -194,7 +260,7 @@ jobs:
shell: pwsh
run: |
$ErrorActionPreference = "Stop"
cd apps/desktop/dist
cd apps/frontend/dist
# Find the installer exe (electron-builder names it with "Setup" or just the app name)
# electron-builder produces one installer exe per build
@@ -261,54 +327,72 @@ jobs:
with:
name: windows-builds
path: |
apps/desktop/dist/*.exe
apps/desktop/dist/*.yml
apps/desktop/dist/*.blockmap
apps/frontend/dist/*.exe
apps/frontend/dist/*.yml
apps/frontend/dist/*.blockmap
build-linux:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Setup Node.js and install dependencies
uses: ./.github/actions/setup-node-frontend
- name: Setup Flatpak and verification tools
- name: Setup Flatpak
run: |
sudo apt-get update
sudo apt-get install -y flatpak flatpak-builder squashfs-tools
sudo apt-get install -y flatpak flatpak-builder
flatpak remote-add --user --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
flatpak install -y --user flathub org.freedesktop.Platform//25.08 org.freedesktop.Sdk//25.08
flatpak install -y --user flathub org.electronjs.Electron2.BaseApp//25.08
- name: Cache pip wheel cache
uses: actions/cache@v5
with:
path: ~/.cache/pip
key: pip-wheel-${{ runner.os }}-x64-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
pip-wheel-${{ runner.os }}-x64-
- name: Cache bundled Python
uses: actions/cache@v5
with:
path: apps/frontend/python-runtime
key: python-bundle-${{ runner.os }}-x64-3.12.8-${{ hashFiles('apps/backend/requirements.txt') }}
restore-keys: |
python-bundle-${{ runner.os }}-x64-3.12.8-
- name: Build application
run: cd apps/desktop && npm run build
run: cd apps/frontend && npm run build
env:
SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
SENTRY_TRACES_SAMPLE_RATE: ${{ secrets.SENTRY_TRACES_SAMPLE_RATE }}
SENTRY_PROFILES_SAMPLE_RATE: ${{ secrets.SENTRY_PROFILES_SAMPLE_RATE }}
- name: Package Linux
run: cd apps/desktop && npm run package:linux
run: cd apps/frontend && npm run package:linux
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
SENTRY_TRACES_SAMPLE_RATE: ${{ secrets.SENTRY_TRACES_SAMPLE_RATE }}
SENTRY_PROFILES_SAMPLE_RATE: ${{ secrets.SENTRY_PROFILES_SAMPLE_RATE }}
- name: Verify Linux packages
run: cd apps/desktop && npm run verify:linux
- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: linux-builds
path: |
apps/desktop/dist/*.AppImage
apps/desktop/dist/*.deb
apps/desktop/dist/*.flatpak
apps/desktop/dist/*.yml
apps/desktop/dist/*.blockmap
apps/frontend/dist/*.AppImage
apps/frontend/dist/*.deb
apps/frontend/dist/*.flatpak
apps/frontend/dist/*.yml
apps/frontend/dist/*.blockmap
# Finalize macOS notarization (runs in parallel with Windows/Linux builds)
finalize-notarization:
@@ -533,7 +617,7 @@ jobs:
draft: false
prerelease: ${{ contains(github.ref, 'beta') || contains(github.ref, 'alpha') }}
env:
GITHUB_TOKEN: ${{ secrets.PAT_TOKEN || secrets.GITHUB_TOKEN }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Update README with new version after successful release
update-readme:
@@ -572,9 +656,9 @@ jobs:
IS_PRERELEASE="${{ steps.version.outputs.is_prerelease }}"
if [ "$IS_PRERELEASE" = "true" ]; then
node scripts/update-readme.mjs "$VERSION" --prerelease
python3 scripts/update-readme.py "$VERSION" --prerelease
else
node scripts/update-readme.mjs "$VERSION"
python3 scripts/update-readme.py "$VERSION"
fi
echo "--- Verifying update ---"
+47 -11
View File
@@ -7,7 +7,6 @@
Thumbs.db
ehthumbs.db
Desktop.ini
nul
# ===========================
# Security - Environment & Secrets
@@ -66,10 +65,52 @@ lerna-debug.log*
.update-metadata.json
# ===========================
# Node.js (apps/desktop)
# Python (apps/backend)
# ===========================
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
eggs/
.eggs/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# Virtual environments
.venv/
venv/
ENV/
env/
.conda/
# Testing
.pytest_cache/
.coverage
htmlcov/
.tox/
.nox/
coverage.xml
*.cover
*.py,cover
.hypothesis/
# Type checking
.mypy_cache/
.dmypy.json
dmypy.json
.pytype/
.pyre/
# ===========================
# Node.js (apps/frontend)
# ===========================
node_modules
apps/desktop/node_modules
apps/frontend/node_modules
.npm
.yarn/
.pnp.*
@@ -78,6 +119,7 @@ apps/desktop/node_modules
dist/
out/
*.tsbuildinfo
apps/frontend/python-runtime/
# Cache
.cache/
@@ -89,8 +131,8 @@ out/
# ===========================
# Electron
# ===========================
apps/desktop/dist/
apps/desktop/out/
apps/frontend/dist/
apps/frontend/out/
*.asar
*.blockmap
*.snap
@@ -110,12 +152,6 @@ test-results/
playwright-report/
playwright/.cache/
# ===========================
# Python
# ===========================
__pycache__/
*.pyc
# ===========================
# Misc
# ===========================
+154 -39
View File
@@ -1,36 +1,52 @@
#!/bin/sh
# =============================================================================
# GIT WORKTREE ENVIRONMENT CLEANUP
# GIT WORKTREE CONTEXT HANDLING
# =============================================================================
# Git automatically sets GIT_DIR (and CWD to the working tree root) before
# running hooks -- even in worktrees. We do NOT need to manually parse .git
# files or export GIT_DIR/GIT_WORK_TREE.
# When running in a worktree, we need to preserve git context to prevent HEAD
# corruption. However, we must also CLEAR these variables when NOT in a worktree
# to prevent cross-worktree contamination (files leaking between worktrees).
#
# However, external tools (IDEs, agents, parent shells) may leave stale
# GIT_DIR/GIT_WORK_TREE values in the environment. If these point to a
# different repo or worktree, git commands in this hook would target the
# wrong repository. Unsetting them lets git re-resolve the correct values
# from the working directory.
# The bug: If GIT_DIR/GIT_WORK_TREE are set from a previous worktree session
# and this hook runs in the main repo (where .git is a directory, not a file),
# git commands will target the wrong repository, causing files to appear as
# untracked in the wrong location.
#
# Fix: Explicitly unset these variables when NOT in a worktree context.
# =============================================================================
unset GIT_DIR
unset GIT_WORK_TREE
if [ -f ".git" ]; then
# We're in a worktree (.git is a file pointing to the actual git dir)
# Use -n with /p to only print lines that match the gitdir: prefix, head -1 for safety
WORKTREE_GIT_DIR=$(sed -n 's/^gitdir: //p' .git | head -1)
if [ -n "$WORKTREE_GIT_DIR" ] && [ -d "$WORKTREE_GIT_DIR" ]; then
export GIT_DIR="$WORKTREE_GIT_DIR"
export GIT_WORK_TREE="$(pwd)"
else
# .git file exists but is malformed or points to non-existent directory
# CRITICAL: Clear any inherited GIT_DIR/GIT_WORK_TREE to prevent cross-worktree leakage
unset GIT_DIR
unset GIT_WORK_TREE
fi
else
# We're in the main repo (.git is a directory)
# CRITICAL: Clear any inherited GIT_DIR/GIT_WORK_TREE to prevent cross-worktree leakage
unset GIT_DIR
unset GIT_WORK_TREE
fi
# =============================================================================
# SAFETY CHECK: Detect and fix corrupted core.worktree configuration
# =============================================================================
# core.worktree lives in the SHARED .git/config (not per-worktree). If any
# process accidentally writes it (e.g., running `git init` with a leaked
# GIT_WORK_TREE), ALL repos and worktrees see the wrong working tree root,
# causing files from one worktree to "leak" into others.
#
# This check runs from both main repo and worktree contexts since the config
# is shared and corruption can happen from either.
CORE_WORKTREE=$(git config --get core.worktree 2>/dev/null || true)
if [ -n "$CORE_WORKTREE" ]; then
echo "Warning: Detected corrupted core.worktree setting ('$CORE_WORKTREE'), removing it..."
if ! git config --unset core.worktree 2>/dev/null; then
echo "Warning: Failed to unset core.worktree. Manual intervention may be needed."
# If core.worktree is set in the main repo's config (pointing to a worktree),
# this indicates previous corruption. Fix it automatically.
if [ ! -f ".git" ]; then
CORE_WORKTREE=$(git config --get core.worktree 2>/dev/null || true)
if [ -n "$CORE_WORKTREE" ]; then
echo "Warning: Detected corrupted core.worktree setting, removing it..."
if ! git config --unset core.worktree 2>/dev/null; then
echo "Warning: Failed to unset core.worktree. Manual intervention may be needed."
fi
fi
fi
@@ -48,18 +64,26 @@ if git diff --cached --name-only | grep -q "^package.json$"; then
VERSION=$(node -p "require('./package.json').version")
if [ -n "$VERSION" ]; then
# Sync to apps/desktop/package.json
if [ -f "apps/desktop/package.json" ]; then
# Sync to apps/frontend/package.json
if [ -f "apps/frontend/package.json" ]; then
node -e "
const fs = require('fs');
const pkg = require('./apps/desktop/package.json');
const pkg = require('./apps/frontend/package.json');
if (pkg.version !== '$VERSION') {
pkg.version = '$VERSION';
fs.writeFileSync('./apps/desktop/package.json', JSON.stringify(pkg, null, 2) + '\n');
console.log(' Updated apps/desktop/package.json to $VERSION');
fs.writeFileSync('./apps/frontend/package.json', JSON.stringify(pkg, null, 2) + '\n');
console.log(' Updated apps/frontend/package.json to $VERSION');
}
"
git add apps/desktop/package.json
git add apps/frontend/package.json
fi
# Sync to apps/backend/__init__.py
if [ -f "apps/backend/__init__.py" ]; then
sed -i.bak "s/__version__ = \"[^\"]*\"/__version__ = \"$VERSION\"/" apps/backend/__init__.py
rm -f apps/backend/__init__.py.bak
git add apps/backend/__init__.py
echo " Updated apps/backend/__init__.py to $VERSION"
fi
# Sync to README.md - section-aware updates (stable vs beta)
@@ -111,14 +135,97 @@ if git diff --cached --name-only | grep -q "^package.json$"; then
fi
fi
# =============================================================================
# DESKTOP APP CHECKS (TypeScript/React)
# BACKEND CHECKS (Python) - Run first, before frontend
# =============================================================================
# Check if there are staged files in apps/desktop
if git diff --cached --name-only | grep -q "^apps/desktop/"; then
echo "Desktop app changes detected, running checks..."
# Check if there are staged Python files in apps/backend
if git diff --cached --name-only | grep -q "^apps/backend/.*\.py$"; then
echo "Python changes detected, running backend checks..."
# Determine ruff command (venv or global)
RUFF=""
if [ -f "apps/backend/.venv/bin/ruff" ]; then
RUFF="apps/backend/.venv/bin/ruff"
elif [ -f "apps/backend/.venv/Scripts/ruff.exe" ]; then
RUFF="apps/backend/.venv/Scripts/ruff.exe"
elif command -v ruff >/dev/null 2>&1; then
RUFF="ruff"
fi
if [ -n "$RUFF" ]; then
# Get only staged Python files in apps/backend (process only what's being committed)
STAGED_PY_FILES=$(git diff --cached --name-only --diff-filter=ACM | grep "^apps/backend/.*\.py$" || true)
if [ -n "$STAGED_PY_FILES" ]; then
# Run ruff linting (auto-fix) only on staged files
echo "Running ruff lint on staged files..."
echo "$STAGED_PY_FILES" | xargs $RUFF check --fix
if [ $? -ne 0 ]; then
echo "Ruff lint failed. Please fix Python linting errors before committing."
exit 1
fi
# Run ruff format (auto-fix) only on staged files
echo "Running ruff format on staged files..."
echo "$STAGED_PY_FILES" | xargs $RUFF format
# Re-stage only the files that were originally staged (in case ruff modified them)
echo "$STAGED_PY_FILES" | xargs git add
fi
else
echo "Warning: ruff not found, skipping Python linting. Install with: uv pip install ruff"
fi
# Run pytest (skip slow/integration tests and Windows-incompatible tests for pre-commit speed)
# Use subshell to isolate directory changes and prevent worktree corruption
echo "Running Python tests..."
(
cd apps/backend
# Tests to skip: graphiti (external deps), merge_file_tracker/service_orchestrator/worktree/workspace (Windows path/git issues)
# Also skip tests that require optional dependencies (pydantic structured outputs)
IGNORE_TESTS="--ignore=../../tests/test_graphiti.py --ignore=../../tests/test_merge_file_tracker.py --ignore=../../tests/test_service_orchestrator.py --ignore=../../tests/test_worktree.py --ignore=../../tests/test_workspace.py --ignore=../../tests/test_finding_validation.py --ignore=../../tests/test_sdk_structured_output.py --ignore=../../tests/test_structured_outputs.py"
# Determine Python executable from venv
VENV_PYTHON=""
if [ -f ".venv/bin/python" ]; then
VENV_PYTHON=".venv/bin/python"
elif [ -f ".venv/Scripts/python.exe" ]; then
VENV_PYTHON=".venv/Scripts/python.exe"
fi
if [ -n "$VENV_PYTHON" ]; then
# Check if pytest is installed in venv
if $VENV_PYTHON -c "import pytest" 2>/dev/null; then
PYTHONPATH=. $VENV_PYTHON -m pytest ../../tests/ -v --tb=short -x -m "not slow and not integration" $IGNORE_TESTS
else
echo "Warning: pytest not installed in venv. Installing test dependencies..."
$VENV_PYTHON -m pip install -q -r ../../tests/requirements-test.txt
PYTHONPATH=. $VENV_PYTHON -m pytest ../../tests/ -v --tb=short -x -m "not slow and not integration" $IGNORE_TESTS
fi
elif [ -d ".venv" ]; then
echo "Warning: venv exists but Python not found in it, using system Python"
PYTHONPATH=. python -m pytest ../../tests/ -v --tb=short -x -m "not slow and not integration" $IGNORE_TESTS
else
echo "Warning: No .venv found in apps/backend, using system Python"
PYTHONPATH=. python -m pytest ../../tests/ -v --tb=short -x -m "not slow and not integration" $IGNORE_TESTS
fi
)
if [ $? -ne 0 ]; then
echo "Python tests failed. Please fix failing tests before committing."
exit 1
fi
echo "Backend checks passed!"
fi
# =============================================================================
# FRONTEND CHECKS (TypeScript/React)
# =============================================================================
# Check if there are staged files in apps/frontend
if git diff --cached --name-only | grep -q "^apps/frontend/"; then
echo "Frontend changes detected, running frontend checks..."
# Detect if we're in a worktree and check if dependencies are available
IS_WORKTREE=false
@@ -132,11 +239,11 @@ if git diff --cached --name-only | grep -q "^apps/desktop/"; then
# Check if node_modules has actual dependencies by looking for a known package
# @lydell/node-pty is required for terminal code and is a common source of TypeScript errors
# It may be in root node_modules (hoisted) or apps/desktop/node_modules
# It may be in root node_modules (hoisted) or apps/frontend/node_modules
# Note: -d follows symlinks automatically, so this works for both real dirs and symlinks
# We check for the full package path (@lydell/node-pty) rather than just the namespace
# for precise detection - ensures the actual dependency is installed, not just any @lydell package
if [ ! -d "node_modules/@lydell/node-pty" ] && [ ! -d "apps/desktop/node_modules/@lydell/node-pty" ]; then
if [ ! -d "node_modules/@lydell/node-pty" ] && [ ! -d "apps/frontend/node_modules/@lydell/node-pty" ]; then
DEPS_AVAILABLE=false
fi
@@ -158,7 +265,7 @@ if git diff --cached --name-only | grep -q "^apps/desktop/"; then
# Dependencies available - run full frontend checks
# Use subshell to isolate directory changes and prevent worktree corruption
(
cd apps/desktop
cd apps/frontend
# Run lint-staged (handles staged .ts/.tsx files)
npm exec lint-staged
@@ -167,14 +274,22 @@ if git diff --cached --name-only | grep -q "^apps/desktop/"; then
exit 1
fi
# Run TypeScript type check (incremental: only rechecks changed files after first run)
# Run TypeScript type check
echo "Running type check..."
NODE_OPTIONS="--max-old-space-size=2048" npm run typecheck
npm run typecheck
if [ $? -ne 0 ]; then
echo "Type check failed. Please fix TypeScript errors before committing."
exit 1
fi
# Run linting
echo "Running lint..."
npm run lint
if [ $? -ne 0 ]; then
echo "Lint failed. Run 'npm run lint:fix' to auto-fix issues."
exit 1
fi
# Check for vulnerabilities (only critical severity)
# Note: Using critical level because electron-builder has a known high-severity
# tar vulnerability (CVE-2026-23745) that cannot be fixed until electron-builder
+75 -11
View File
@@ -18,17 +18,20 @@ repos:
VERSION=$(node -p "require('./package.json').version")
if [ -n "$VERSION" ]; then
# Sync to apps/desktop/package.json
# Sync to apps/frontend/package.json
node -e "
const fs = require('fs');
const p = require('./apps/desktop/package.json');
const p = require('./apps/frontend/package.json');
const v = process.argv[1];
if (p.version !== v) {
p.version = v;
fs.writeFileSync('./apps/desktop/package.json', JSON.stringify(p, null, 2) + '\n');
fs.writeFileSync('./apps/frontend/package.json', JSON.stringify(p, null, 2) + '\n');
}
" "$VERSION"
# Sync to apps/backend/__init__.py
sed -i.bak "s/__version__ = \"[^\"]*\"/__version__ = \"$VERSION\"/" apps/backend/__init__.py && rm -f apps/backend/__init__.py.bak
# Sync to README.md - section-aware updates (stable vs beta)
ESCAPED_VERSION=$(echo "$VERSION" | sed 's/-/--/g')
@@ -67,13 +70,74 @@ repos:
rm -f README.md.bak
# Stage changes
git add apps/desktop/package.json README.md 2>/dev/null || true
git add apps/frontend/package.json apps/backend/__init__.py README.md 2>/dev/null || true
fi
language: system
files: ^package\.json$
pass_filenames: false
# Frontend linting (apps/desktop/) - Biome is 15-25x faster than ESLint
# Python encoding check - prevent regression of UTF-8 encoding fixes (PR #782)
- repo: local
hooks:
- id: check-file-encoding
name: Check file encoding parameters
entry: python scripts/check_encoding.py
language: system
types: [python]
files: ^apps/backend/
description: Ensures all file operations specify encoding="utf-8"
# Python linting (apps/backend/)
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.14.10
hooks:
- id: ruff
args: [--fix]
files: ^apps/backend/
- id: ruff-format
files: ^apps/backend/
# Python tests (apps/backend/) - skip slow/integration tests for pre-commit speed
# Tests to skip: graphiti (external deps), merge_file_tracker/service_orchestrator/worktree/workspace (Windows path/git issues)
# NOTE: Skip this hook in worktrees (where .git is a file, not a directory)
- repo: local
hooks:
- id: pytest
name: Python Tests
entry: bash
args:
- -c
- |
# Skip in worktrees - .git is a file pointing to main repo, not a directory
# This prevents path resolution issues with ../../tests/ in worktree context
if [ -f ".git" ]; then
echo "Skipping pytest in worktree (path resolution would fail)"
exit 0
fi
cd apps/backend
if [ -f ".venv/bin/pytest" ]; then
PYTEST_CMD=".venv/bin/pytest"
elif [ -f ".venv/Scripts/pytest.exe" ]; then
PYTEST_CMD=".venv/Scripts/pytest.exe"
else
PYTEST_CMD="python -m pytest"
fi
PYTHONPATH=. $PYTEST_CMD \
../../tests/ \
-v \
--tb=short \
-x \
-m "not slow and not integration" \
--ignore=../../tests/test_graphiti.py \
--ignore=../../tests/test_merge_file_tracker.py \
--ignore=../../tests/test_service_orchestrator.py \
--ignore=../../tests/test_worktree.py \
--ignore=../../tests/test_workspace.py
language: system
files: ^(apps/backend/.*\.py$|tests/.*\.py$)
pass_filenames: false
# Frontend linting (apps/frontend/) - Biome is 15-25x faster than ESLint
# NOTE: These hooks check for worktree context to avoid npm/node_modules issues
- repo: local
hooks:
@@ -84,13 +148,13 @@ repos:
- -c
- |
# Skip in worktrees if node_modules doesn't exist (Biome not installed)
if [ -f ".git" ] && [ ! -d "apps/desktop/node_modules" ]; then
if [ -f ".git" ] && [ ! -d "apps/frontend/node_modules" ]; then
echo "Skipping Biome in worktree (node_modules not found)"
exit 0
fi
cd apps/desktop && npx biome check --write --no-errors-on-unmatched .
cd apps/frontend && npx biome check --write --no-errors-on-unmatched .
language: system
files: ^apps/desktop/.*\.(ts|tsx|js|jsx|json)$
files: ^apps/frontend/.*\.(ts|tsx|js|jsx|json)$
pass_filenames: false
- id: typecheck
@@ -100,13 +164,13 @@ repos:
- -c
- |
# Skip in worktrees if node_modules doesn't exist (dependencies not installed)
if [ -f ".git" ] && [ ! -d "apps/desktop/node_modules" ]; then
if [ -f ".git" ] && [ ! -d "apps/frontend/node_modules" ]; then
echo "Skipping TypeScript check in worktree (node_modules not found)"
exit 0
fi
cd apps/desktop && npm run typecheck
cd apps/frontend && npm run typecheck
language: system
files: ^apps/desktop/.*\.(ts|tsx)$
files: ^apps/frontend/.*\.(ts|tsx)$
pass_filenames: false
# General checks
+10 -243
View File
@@ -1,234 +1,3 @@
## 2.7.6 - Stability & Feature Enhancements
### ✨ New Features
- **Multi-profile account management** — Unified profile swapping with automatic token refresh and rate limit recovery for both OAuth and API-compatible providers
- **Enhanced terminal experience** — Customizable terminal fonts with OS-specific defaults, Claude Code CLI settings injection, and improved worktree integration
- **Advanced roadmap management** — Expand/collapse functionality for phase features and real-time sync with task lifecycle
- **Queue System v2** — Smart task prioritization with auto-promotion and intelligent rate limit recovery
- **GitHub integration enhancements** — AI-powered PR template generation, user-friendly API error handling, and improved review visibility
- **UI/UX improvements** — Spell check support for text inputs, collapsible sidebar toggle, task screenshot capture, expandable task descriptions, and bulk worktree operations
- **Evidence-based PR validation** — Advanced review system with trigger-driven exploration and enhanced recovery mechanisms
### 🛠️ Improvements
- **Performance optimizations** — Async parallel worktree listing prevents UI freezes and improves responsiveness
- **Robustness enhancements** — Atomic file writes, better error detection in AI responses, and improved OOM/orphaned agent management for overnight builds
- **Terminal stability** — Fixed GPU context exhaustion from large pastes, SIGABRT crashes on macOS shutdown, and session restoration on app restart
- **Build & packaging** — XState bundling for packaged apps, aligned Linux package builds, and improved auto-updater for beta releases and DMG installations
- **Diagnostic improvements** — Sentry instrumentation for Python subprocesses and better error tracking across the system
### 🐛 Bug Fixes
- **Terminal & PTY** — Fixed paste size limits, race conditions, rendering issues, text alignment, worktree crashes, and terminal content resizing on expansion
- **PR review system** — Resolved error visibility in bundled apps, improved structured output validation with three-tier recovery, preserved findings during crashes, and fixed UTC timestamp detection for comment tracking
- **Planning & task execution** — Fixed handling of empty/greenfield projects, atomic writes to prevent 0-byte file corruption, planning phase crashes, and implementation plan file watching
- **Authentication & profiles** — Resolved OAuth token revocation loops, API profile mode support without OAuth requirement, subscription type preservation during token refresh, and Linux credential file detection
- **Windows/cross-platform** — Complete System32 executable path fixes for where.exe and taskkill.exe, Windows credential normalization, and proper shell detection for Windows terminals
- **Agent management** — Fixed infinite retry loops for tool concurrency errors, auth error detection, and title generator production path resolution
- **UI/UX fixes** — Resolved Insights scroll-to-blank-space issues, infinite re-render loops in terminal font settings, kanban board scaling collisions, ideation stuck states, and panel constraint errors during terminal exit
- **Worktree & Git** — Improved branch pattern validation, removed auto-commit on deletion, support for detached HEAD state during PR creation, and better merge conflict resolution with progress tracking
- **Integrations** — Fixed Ollama infinite subprocess spawning, Graphiti import paths, OpenRouter API URL suffix, and GitLab authentication bugs
- **Settings & configuration** — Corrected .auto-claude path discovery timeout, z.AI China preset URL, log order sorting, and onboarding completion state persistence
### 📚 Documentation
- Added Awesome Claude Code badge to README
- Added instructions for resetting PR review state in CLAUDE.md
---
## What's Changed
- fix: handle unknown SDK message types (rate_limit_event) to prevent session crashes by @AndyMik90 in 4a75ea9f9
- fix: PR review error visibility and gh CLI resolution in bundled apps by @AndyMik90 in 732fc1cd3
- fix: handle empty/greenfield projects in spec creation (#1426) (#1841) by @Andy in 819f98d9f
- fix: clear terminalEventSeen on task restart to prevent stuck-after-planning (#1828) (#1840) by @Andy in 28a620079
- fix: watch worktree path for implementation_plan.json changes (#1805) (#1842) by @Andy in fb3a3fbda
- fix: resolve Claude CLI not found on Windows - PATH, prompt size, cwd (#1661) (#1843) by @Andy in 76d1d3b03
- fix: handle planning phase crash and resume recovery (#1562) (#1844) by @Andy in 3cb05781f
- fix: show dismissed PR review findings in UI instead of silently dropping them (#1852) by @Andy in d98ff7d19
- fix: preserve file/line info in PR review extraction recovery (#1857) by @Andy in 635b53eea
- docs: add Awesome Claude Code badge to README (#1838) by @Andy in 2e4b5ac65
- test: achieve 100% test coverage for backend CLI commands (#1772) by @StillKnotKnown in 385f04414
- fix: cap terminal paste size to 1MB to prevent GPU context exhaustion by @AndyMik90 in 7b0f3a2c0
- fix: prevent OOM, orphaned agents, and unbounded growth during overnight builds (#1813) by @Andy in 4091d1d4b
- docs: add instructions for resetting PR review state in CLAUDE.md by @AndyMik90 in ecb615802
- auto-claude: 217-investigate-symlink-issues-in-work-tree-creation-f (#1808) by @Andy in ae13ce14c
- auto-claude: 218-enable-claude-code-features-in-worktree-terminals (#1809) by @Andy in e3b219288
- auto-claude: 219-investigate-and-fix-authentication-subscription-sy (#1810) by @Andy in 6204d5fc2
- feat(roadmap): add expand/collapse functionality for phase features (#1796) by @Burak in f735f0b49
- auto-claude: 216-display-ongoing-pr-review-logs-in-progress (#1807) by @Andy in a4870fa0c
- fix(pr-review): reduce structured output failures and preserve findings in recovery (#1806) by @Andy in f1b8cd3a7
- fix(sentry): enable Sentry for Python subprocesses and add diagnostic instrumentation (#1804) by @Andy in 4d4234378
- fix(pr-review): add three-tier recovery for structured output validation failure (#1797) by @Andy in d1fbccde3
- test: improve backend agent test coverage to 94% (#1779) by @StillKnotKnown in ed93df698
- fix(github): use UTC timestamps for reviewed_at to fix comment detection (#1795) by @Andy in 8872d33e3
- feat: add user-friendly GitHub API error handling (#1790) by @StillKnotKnown in 8ece0009e
- fix(roadmap): sync roadmap features with task lifecycle (#1791) by @Andy in 115576e85
- fix(github): resolve PR review hanging in bundled app (#1793) by @Andy in 3791b37bb
- feat(profiles): implement unified profile swapping across OAuth and API accounts (#1794) by @StillKnotKnown in 282387356
- test: improve backend memory system test coverage to 100% (#1780) by @StillKnotKnown in 4f1b7b2a9
- fix(ideation): guard against non-string properties in IdeaCard badges by @AndyMik90 in 5e78d748e
- fix(updater): convert HTML release notes to markdown before rendering by @AndyMik90 in aa5fc7f95
- fix(pr-review): simplify structured output schema to reduce validation failures (#1787) by @Andy in cd8914700
- fix(qa): enforce visual verification for UI changes and inject startup commands (#1784) by @Andy in f149a7fbd
- fix(plan-files): use atomic writes to prevent 0-byte corruption (#1785) by @Andy in c2245b812
- fix(terminal): make worktree dropdown scrollable and show all items by @AndyMik90 in 950da45e4
- auto-claude: subtask-1-1 - Add adaptive thinking badge to thinking level label (#1782) by @Andy in 25acf2826
- auto-claude: subtask-1-1 - Add overflow-hidden and break-words to subtask cards by @AndyMik90 in 39aa08872
- refactor(app-updater): disable automatic downloads and allow intentional downgrades by @AndyMik90 in 8de8039db
- fix(auth): detect auth errors in AI response text and prevent retry loops (#1776) by @Andy in f4788e4af
- test: achieve 100% coverage for backend core workspace module (#1774) by @StillKnotKnown in 3f95765cf
- fix(title-generator): add production path resolution for backend source (#1778) by @Andy in 923880f5b
- fix(fast-mode): use setting_sources instead of env var for CLI fast mode (#1771) by @Andy in 390ba6a58
- fix(windows): complete System32 executable path fixes for where.exe and taskkill.exe (#1715) by @VDT-91 in aa7f56e5d
- fix(worktree): remove auto-commit on deletion and add uncommitted changes warning by @AndyMik90 in cec8e65ee
- Smart PR Status Polling System (#1766) by @Andy in 48d5f7a32
- feat: simplify thinking system and remove opus-1m model variant (#1760) by @Andy in bb7e18937
- auto-claude: 203-fix-pr-review-ui-update-issue (#1732) by @Andy in 7589f8e4f
- auto-claude: subtask-2-1 - Create isAPIProfileAuthenticated() function to val (#1745) by @Andy in 57e38a692
- auto-claude: 202-fix-kanban-board-scaling-collisions (#1731) by @Andy in d09ebb850
- auto-claude: 204-fix-pr-review-ui-not-updating-without-manual-navig (#1734) by @Andy in 087091cef
- auto-claude: 203-fix-ui-not-updating-during-pr-review-operations (#1733) by @Andy in f085c08bd
- auto-claude: 205-fix-insights-chat-only-shows-last-task-suggestion- (#1735) by @Andy in f121f9cdd
- auto-claude: 197-roadmap-generation-stuck-at-50-file-locking-race-c (#1746) by @Andy in f41f15e59
- auto-claude: 193-fix-update-context7-mcp-tool-name-from-get-library (#1744) by @Andy in bdff9141a
- auto-claude: 192-changelog-generation-multiple-critical-bugs-tasks- (#1725) by @Andy in 8c9a504df
- auto-claude: 194-bug-rate-limit-during-task-execution-causes-subtas (#1726) by @Andy in 8a7443d24
- auto-claude: 201-bug-pr-review-logs-and-analysis (#1730) by @Andy in e0d53adb4
- auto-claude: 196-fix-worktrees-dialog-auto-close-race-condition-and (#1727) by @Andy in 323b0d3be
- auto-claude: 199-bug-logs-disappear-after-restart (#1728) by @Andy in d639f6ef8
- auto-claude: 198-critical-oauth-token-revocation-causes-infinite-40 (#1747) by @Andy in 4438c0b10
- Fix Panel Constraints Error During Terminal Exit (#1757) by @Andy in 32bf353da
- auto-claude: 190-bug-context-page-crash-multiple-root-causes-when-v (#1724) by @Andy in 2db36982f
- feat: add search/filter to WorktreeSelector dropdown (#1754) by @Andy in 09f059ca3
- fix(terminal): push worktree branch to remote with tracking on creation (#1753) by @Andy in b5de0d9ff
- auto-claude: 189-subtask-execution-stuck-in-infinite-retry-loop-whe (#1723) by @Andy in 445da186c
- auto-claude: 188-terminal-claude-sessions-require-manual-click-to-r (#1743) by @Andy in f8499e965
- auto-claude: 200-bug-changelog-and-release-generation (#1729) by @Andy in 826583b82
- fix(terminal): use each terminal's cwd for invoke Claude all button (#1756) by @Andy in ac4fe4f42
- feat(terminal): read Claude Code CLI settings and inject env vars into PTY sessions (#1750) by @Andy in 152e54093
- fix: correct .auto-claude path mismatch causing discovery phase timeout (#1748) by @VDT-91 in 2c2a8a754
- fix: remove incorrect /v1 suffix from OpenRouter API URL (#1749) by @StillKnotKnown in 7e799ee57
- fix: prevent terminal worktree crash with race condition fixes (#1586) (#1658) by @VDT-91 in 216b58bcf
- fix: correct log order sorting and add configurable log order setting (#1720) by @Burak in 2e2b82365
- fix(ollama): stop infinite subprocess spawning from useEffect re-render loop (#1716) by @Quentin Veys in acb131b72
- fix(graphiti): migrate graphiti_memory imports to canonical paths (#1714) by @Quentin Veys in df528f065
- fix: improve auto-updater for beta releases and DMG installs (#1681) by @Andy in ff91a1af0
- feat: unified operation registry for intelligent auth/rate limit recovery (#1698) by @Andy in 6d0222fa9
- fix: Prevent stale worktree data from overriding correct task status (#1710) by @Burak in fe08c644c
- feat: add subscriptionType and rateLimitTier to ClaudeProfile (#1688) by @Andy in a5e3cc9a2
- auto-claude: subtask-1-1 - Add useTaskStore import and update task state after successful PR creation (#1683) by @Andy in 4587162e4
- auto-claude: 182-implement-pagination-and-filtering-for-github-pr-l (#1654) by @Andy in b4e6b2fe4
- auto-claude: 181-add-expand-button-for-long-task-descriptions (#1653) by @Andy in d9cd300fe
- fix(terminal): resolve text alignment issues on expand/minimize (#1650) by @VDT-91 in f5a7e26d9
- fix(windows): use full path to where.exe for reliable executable lookup (#1659) by @VDT-91 in 5f63daa3c
- fix: resolve ideation stuck at 3/6 types bug (#1660) by @VDT-91 in e6e8da17c
- Clarify Local and Origin Branch Distinction (#1652) by @Andy in 9317148b6
- auto-claude: 186-set-default-dark-mode-on-startup (#1656) by @Andy in 473020621
- auto-claude: subtask-1-1 - Add min-h-0 to enable scrolling in Roadmap tabs (#1655) by @Andy in ae703be9f
- fix: XState status lifecycle & cross-project contamination fixes (#1647) by @kaigler in 5293fb399
- refactor(frontend): complete XState task state machine migration (#1338) (#1575) by @kaigler in e2f9abadb
- Merge conflict resolution progress bar and log viewer (#1620) by @Andy in d16be3077
- fix: align Linux package builds (AppImage/deb/Flatpak) with target-specific extraResources (#1623) by @StillKnotKnown in bad1a9b2c
- Fix/gitlab bugs (#1519 and #1521) (#1544) by @bu5hm4nn in cd423c65c
- feat(kanban): add bulk task delete and worktree cleanup improvements (#1588) by @kaigler in 02ed91c91
- fix: add worktree isolation warning to prevent agent escape (#1528) by @kaigler in fe5cc582b
- feat(ui): add spell check support for text inputs (#1304) by @kaigler in 8f02a5129
- fix(windows): complete Windows credential fixes with path normalization (#1585) by @kaigler in 1e1997167
- AI-Powered GitHub PR Template Generation (#1618) by @Andy in 900dd4360
- Fix pty.node SIGABRT crash on macOS shutdown (#1619) by @Andy in f355e09d7
- fix(merge): use git merge for diverged branches with progress tracking (#1605) by @Andy in bde2ca4b2
- Surface Billing/Credit Exhaustion Errors to UI (Issue #1580) (#1617) by @Andy in 7bf12e856
- auto-claude: subtask-1-1 - Change $teamId type from ID! to String! in the team query (#1627) by @Andy in 54d0cd2f4
- fix(auth): support API profile mode without OAuth requirement (#1616) by @StillKnotKnown in f8cc63af4
- fix: agent retry loop for tool concurrency errors (#1546) [v3] (#1606) by @Michael Ludlow in 0aea4fb5e
- fix(queue): enforce max parallel tasks and auto-refresh UI (#1594) by @Andy in 4070a4c29
- Persist Kanban column collapse state per project via main process (#1579) by @Andy in a1114664e
- feat(pr-review): evidence-based validation and trigger-driven exploration (#1593) by @Andy in bfc232825
- fix(ui): smart auto-scroll for Insights streaming responses (#1591) by @kaigler in eee97e7ea
- fix(changelog): validate Claude CLI exists before generation (#1305) by @kaigler in c1f24c07f
- auto-claude: subtask-1-1 - Add min-w-0 class to subtask title row flex container (#1578) by @Andy in 286591c02
- auto-claude: subtask-1-1 - Remove Popover wrapper and related functionality from ClaudeCodeStatusBadge (#1566) by @Andy in 8d18cc81a
- fix(claude-profile): preserve subscriptionType and rateLimitTier during token refresh (#1556) by @Andy in 52e426a48
- auto-claude: subtask-1-1 - Update cancelReview callback to handle both success and failure cases (#1551) by @Andy in d8f00fe5a
- fix(backend): prioritize git remote detection over env var for repo (#1555) by @Andy in 9b07ed464
- fix(backend): handle detached HEAD state when pushing branch for PR creation (#1560) by @Andy in 2b72694d0
- fix: add explicit UTF-8 encoding across all Electron main process I/O (#1554) by @Andy in 4243530e9
- fix(backend): pass OAuth token to Python subprocess for authentication by @AndyMik90 in 6f1002dd7
- perf(frontend): async parallel worktree listing to prevent UI freezes (#1553) by @Andy in 399a7e736
- auto-claude: subtask-1-1 - Remove amber lock indicator line from kanban resize handle (#1557) by @Andy in 83a64b88e
- fix(frontend): resolve TerminalFontSettings infinite re-render loop (#1536) by @StillKnotKnown in 1c6266025
- fix(frontend): respect hasCompletedOnboarding from ~/.claude.json (#1537) by @StillKnotKnown in 1860c2c43
- fix: prevent planner from generating invalid verification types (#1388) (#1529) by @kaigler in 94d941333
- fix(frontend): resolve Insights scroll-to-blank-space issue on macOS (ACS-382) (#1535) by @StillKnotKnown in 496b2b96a
- feat: add customizable terminal fonts with OS-specific defaults (#1412) by @StillKnotKnown in f289107b8
- Add dev mode screenshot capture warning (#1516) by @Andy in 16eeb301a
- fix: add worktree isolation warnings to prevent agent escape (ACS-394) (#1495) by @StillKnotKnown in 1e453653b
- fix: resolve flaky subprocess-spawn test on Windows CI (ACS-392) (#1494) by @StillKnotKnown in f6b264d56
- feat(task-logger): strip ANSI escape codes from logs and extend coverage (#1411) by @StillKnotKnown in 988ec0c25
- fix(frontend): use spawn() instead of exec() for Windows terminal launching (#1498) by @StillKnotKnown in 26c9083d3
- fix(api-profiles): correct z.AI China preset URL and rename provider presets (#1500) by @StillKnotKnown in 05cf0a516
- fix: validate branch pattern before worktree cleanup to prevent deleting wrong branch (#1493) by @StillKnotKnown in 8576754a1
- Real-Time Updates for Insights Chat (#1511) by @Andy in d940b6ade
- Fix Terminal UI Rendering Issues (#1514) by @Andy in 8d8306b8e
- Fix terminal content resizing on expansion (#1512) by @Andy in 9f6c0026b
- Restore Terminal Session History on App Restart (#1515) by @Andy in 63e2847fc
- Move Reference Images Above Task Title & Fix Image Display Issues (#1513) by @Andy in b269ac305
- auto-claude: 143-fix-github-integration-ui-refresh-issues (#1467) by @Andy in aa2cb4fa6
- feat: Multi-profile account swapping with token refresh and queue routing (#1496) by @Andy in 1e72c8d77
- Simplified Testing Strategy for Regression Prevention (#1379) by @Andy in ae4e48e8b
- auto-claude: 152-persist-tasks-during-roadmap-regeneration (#1463) by @Andy in 9bd3d7e3b
- Debug Kanban Memory & Add Sentry Monitoring (#1380) by @Andy in bc5f550ee
- auto-claude: 147-remove-outdated-compatibility-shims (#1465) by @Andy in 53111dbb9
- auto-claude: 162-fix-worktree-error-on-repeated-task-starts (#1453) by @Andy in b955badf7
- auto-claude: 155-fix-pr-list-diff-display-metrics (#1458) by @Andy in 31f116db5
- auto-claude: 151-fix-pr-review-agent-token-refresh-on-account-swap (#1456) by @Andy in d081af042
- auto-claude: 148-add-progress-persistence-and-status-indicators (#1464) by @Andy in 4937d5745
- auto-claude: 154-fix-task-modal-conflict-check-status-refresh (#1462) by @Andy in 0299009df
- auto-claude: 153-widen-kanban-columns-and-add-collapse-feature (#1457) by @Andy in d65973075
- auto-claude: subtask-1-1 - Add filter after map operation to remove empty str (#1466) by @Andy in 783f0fe0e
- fix: add formatReleaseNotes helper for markdown changelog rendering (#1468) by @Andy in 43a97e1b3
- feat(sidebar): add collapsible sidebar toggle (#1501) by @Michael Ludlow in d17c17887
- fix(auth): check .credentials.json for Linux profile authentication (#1492) by @StillKnotKnown in 8d2f66291
- auto-claude: subtask-1-1 - Replace ReleaseNotesRenderer with ReactMarkdown (#1454) by @Andy in 1185a558c
- auto-claude: 156-fix-electron-app-version-detection-bug (#1459) by @Andy in 9a3b48c25
- auto-claude: subtask-1-1 - Add --no-track flag to git worktree add command (#1455) by @Andy in 0c2990815
- auto-claude: subtask-1-1 - Change task.specId to taskId in 3 startSpecCreation calls (#1461) by @Andy in 91edc0e14
- fix(onboarding): align MemoryStep layout with Settings MemoryBackendSection (#1445) by @Michael Ludlow in e9de26d59
- auto-claude: subtask-1-1 - Add metadata?.requireReviewBeforeCoding check (#1460) by @Andy in 426d56571
- fix: use API profile environment variables for task title generation (#1471) by @JoshuaRileyDev in c5a0f042d
- fix(auth): Long-lived OAuth authentication with multi-profile usage display (#1443) by @Andy in 12e788417
- feat: Add screenshot capture to task creation modal (#1429) by @JoshuaRileyDev in 1a2a1b1fc
- fix: prevent queue settings modal from disappearing when tasks change (#1430) by @JoshuaRileyDev in 33acc1430
- feat: Queue System v2 with Auto-Promotion and Smart Task Management (#1203) by @JoshuaRileyDev in 3b87e24d7
- feat: Add API profile providers usage endpoints support (#1279) by @StillKnotKnown in cfe7dedd0
## Thanks to all contributors
@AndyMik90, @Andy, @Burak, @StillKnotKnown, @VDT-91, @kaigler, @Michael Ludlow, @JoshuaRileyDev, @Quentin Veys, @bu5hm4nn
## 2.7.5 - Security & Platform Improvements
### ✨ New Features
@@ -305,8 +74,6 @@
### 🐛 Bug Fixes
- Fixed task logs disappearing after app restart in development mode (issue #1657)
- Fixed Kanban board status flip-flopping and multi-location task deletion
- Fixed Windows CLI detection and version selection UX issues
@@ -1246,17 +1013,17 @@
- feat(python): bundle Python 3.12 with packaged Electron app (#284) by @Andy in 7f19c2e1
- fix: resolve spawn python ENOENT error on Linux by using getAugmentedEnv() (#281) by @Todd W. Bucy in d98e2830
- fix(ci): add write permissions to beta-release update-version job by @AndyMik90 in 0b874d4b
- chore(deps): bump @xterm/xterm from 5.5.0 to 6.0.0 in /apps/desktop (#270) by @dependabot[bot] in 50dd1078
- chore(deps): bump @xterm/xterm from 5.5.0 to 6.0.0 in /apps/frontend (#270) by @dependabot[bot] in 50dd1078
- fix(github): resolve follow-up review API issues by @AndyMik90 in f1cc5a09
- fix(security): resolve CodeQL file system race conditions and unused variables (#277) by @Andy in b005fa5c
- fix(ci): use correct electron-builder arch flags (#278) by @Andy in d79f2da4
- chore(deps): bump jsdom from 26.1.0 to 27.3.0 in /apps/desktop (#268) by @dependabot[bot] in 5ac566e2
- chore(deps): bump typescript-eslint in /apps/desktop (#269) by @dependabot[bot] in f49d4817
- chore(deps): bump jsdom from 26.1.0 to 27.3.0 in /apps/frontend (#268) by @dependabot[bot] in 5ac566e2
- chore(deps): bump typescript-eslint in /apps/frontend (#269) by @dependabot[bot] in f49d4817
- fix(ci): use develop branch for dry-run builds in beta-release workflow (#276) by @Andy in 1e1d7d9b
- fix: accept bug_fix workflow_type alias during planning (#240) by @Daniel Frey in e74a3dff
- fix(paths): normalize relative paths to posix (#239) by @Daniel Frey in 6ac8250b
- chore(deps): bump @electron/rebuild in /apps/desktop (#271) by @dependabot[bot] in a2cee694
- chore(deps): bump vitest from 4.0.15 to 4.0.16 in /apps/desktop (#272) by @dependabot[bot] in d4cad80a
- chore(deps): bump @electron/rebuild in /apps/frontend (#271) by @dependabot[bot] in a2cee694
- chore(deps): bump vitest from 4.0.15 to 4.0.16 in /apps/frontend (#272) by @dependabot[bot] in d4cad80a
- feat(github): add automated PR review with follow-up support (#252) by @Andy in 596e9513
- ci: implement enterprise-grade PR quality gates and security scanning (#266) by @Alex in d42041c5
- fix: update path resolution for ollama_model_detector.py in memory handlers (#263) by @delyethan in a3f87540
@@ -1526,17 +1293,17 @@
- feat(python): bundle Python 3.12 with packaged Electron app (#284) by @Andy in 7f19c2e1
- fix: resolve spawn python ENOENT error on Linux by using getAugmentedEnv() (#281) by @Todd W. Bucy in d98e2830
- fix(ci): add write permissions to beta-release update-version job by @AndyMik90 in 0b874d4b
- chore(deps): bump @xterm/xterm from 5.5.0 to 6.0.0 in /apps/desktop (#270) by @dependabot[bot] in 50dd1078
- chore(deps): bump @xterm/xterm from 5.5.0 to 6.0.0 in /apps/frontend (#270) by @dependabot[bot] in 50dd1078
- fix(github): resolve follow-up review API issues by @AndyMik90 in f1cc5a09
- fix(security): resolve CodeQL file system race conditions and unused variables (#277) by @Andy in b005fa5c
- fix(ci): use correct electron-builder arch flags (#278) by @Andy in d79f2da4
- chore(deps): bump jsdom from 26.1.0 to 27.3.0 in /apps/desktop (#268) by @dependabot[bot] in 5ac566e2
- chore(deps): bump typescript-eslint in /apps/desktop (#269) by @dependabot[bot] in f49d4817
- chore(deps): bump jsdom from 26.1.0 to 27.3.0 in /apps/frontend (#268) by @dependabot[bot] in 5ac566e2
- chore(deps): bump typescript-eslint in /apps/frontend (#269) by @dependabot[bot] in f49d4817
- fix(ci): use develop branch for dry-run builds in beta-release workflow (#276) by @Andy in 1e1d7d9b
- fix: accept bug_fix workflow_type alias during planning (#240) by @Daniel Frey in e74a3dff
- fix(paths): normalize relative paths to posix (#239) by @Daniel Frey in 6ac8250b
- chore(deps): bump @electron/rebuild in /apps/desktop (#271) by @dependabot[bot] in a2cee694
- chore(deps): bump vitest from 4.0.15 to 4.0.16 in /apps/desktop (#272) by @dependabot[bot] in d4cad80a
- chore(deps): bump @electron/rebuild in /apps/frontend (#271) by @dependabot[bot] in a2cee694
- chore(deps): bump vitest from 4.0.15 to 4.0.16 in /apps/frontend (#272) by @dependabot[bot] in d4cad80a
- feat(github): add automated PR review with follow-up support (#252) by @Andy in 596e9513
- ci: implement enterprise-grade PR quality gates and security scanning (#266) by @Alex in d42041c5
- fix: update path resolution for ollama_model_detector.py in memory handlers (#263) by @delyethan in a3f87540
+92 -123
View File
@@ -2,9 +2,9 @@
This file provides guidance to Claude Code when working with this repository.
Auto Claude is an autonomous multi-agent coding framework that plans, builds, and validates software for you. It's a TypeScript-first Electron desktop application with a self-contained AI agent layer (Vercel AI SDK v6). A lightweight Python sidecar provides the optional Graphiti memory system.
Auto Claude is an autonomous multi-agent coding framework that plans, builds, and validates software for you. It's a monorepo with a Python backend (CLI + agent logic) and an Electron/React frontend (desktop UI).
> **Deep-dive reference:** [ARCHITECTURE.md](shared_docs/ARCHITECTURE.md) | **Frontend contributing:** [apps/desktop/CONTRIBUTING.md](apps/desktop/CONTRIBUTING.md)
> **Deep-dive reference:** [ARCHITECTURE.md](shared_docs/ARCHITECTURE.md) | **Frontend contributing:** [apps/frontend/CONTRIBUTING.md](apps/frontend/CONTRIBUTING.md)
## Product Overview
@@ -30,92 +30,44 @@ Auto Claude is a desktop application (+ CLI) where users describe a goal and AI
## Critical Rules
**Vercel AI SDK only** — All AI interactions use the Vercel AI SDK v6 (`ai` package) via the TypeScript agent layer in `apps/desktop/src/main/ai/`. NEVER use `@anthropic-ai/sdk` or `anthropic.Anthropic()` directly. Use `createProvider()` from `ai/providers/factory.ts` and `streamText()`/`generateText()` from the `ai` package. Provider-specific adapters (e.g., `@ai-sdk/anthropic`, `@ai-sdk/openai`) are managed through the provider registry.
**Claude Agent SDK only** — All AI interactions use `claude-agent-sdk`. NEVER use `anthropic.Anthropic()` directly. Always use `create_client()` from `core.client`.
**i18n required** — All frontend user-facing text uses `react-i18next` translation keys. Hardcoded strings in JSX/TSX break localization for non-English users. Add keys to both `en/*.json` and `fr/*.json`.
**i18n required** — All frontend user-facing text MUST use `react-i18next` translation keys. Never hardcode strings in JSX/TSX. Add keys to both `en/*.json` and `fr/*.json`.
**Platform abstraction** — Never use `process.platform` directly. Import from `apps/desktop/src/main/platform/`. CI tests all three platforms.
**Platform abstraction** — Never use `process.platform` directly. Import from `apps/frontend/src/main/platform/` or `apps/backend/core/platform/`. CI tests all three platforms.
**No time estimates**Provide priority-based ordering instead of duration predictions.
**No time estimates**Never provide duration predictions. Use priority-based ordering instead.
**PR target** — Always target the `develop` branch for PRs, not `main`. Main is reserved for releases.
**No console.log in production code**`console.log` output is invisible in bundled Electron apps. Use Sentry for error tracking in production; reserve `console.log` for development only.
## Work Approach: Orchestrator-First
You are an orchestrator. Your primary role is to understand what needs to be done, break it into workstreams, and delegate execution to agent teams. This keeps your context window focused on coordination and decision-making rather than filling up with implementation details.
<orchestrator_pattern>
When given a task, follow this pattern:
1. **Investigate first** — Read the actual code before forming any hypothesis. Use targeted searches (Glob, Grep, Read) for simple lookups. For broader exploration, spawn an Explore agent.
2. **Plan the approach** — Identify what needs to change, which files are involved, and whether work can be parallelized. For multi-step tasks, create a task list to track workstreams.
3. **Delegate execution** — Spawn agent teams to do the implementation work. Each agent gets a clear, self-contained assignment with all the context it needs: relevant file paths, the specific change to make, and acceptance criteria. Run independent workstreams in parallel.
4. **Verify and integrate** — Review agent outputs, run tests, and ensure changes work together. Fix integration issues or spawn follow-up agents as needed.
</orchestrator_pattern>
**When to delegate vs. do directly:**
- Delegate: multi-file changes, research across the codebase, independent parallel workstreams, tasks that would consume significant context
- Do directly: single-file edits, simple bug fixes, quick lookups, tasks where you already have the context
**Giving agents good assignments** — Each agent works with a fresh context. Include: the specific goal, relevant file paths, code patterns to follow, and what "done" looks like. Agents perform better with explicit, complete instructions than with vague references to "the current task."
**Minimal changes only** — Prefer the simplest approach (e.g., prompt-only changes, single guard clause) before suggesting multi-component solutions. If the user asks for X, implement X — don't bundle additional fixes they didn't request.
**Default to action** — When the user's intent implies making changes, implement them rather than only suggesting. If something is unclear, read the relevant code to fill in the gaps rather than asking. Only ask when genuine ambiguity remains about what the user wants.
## Context Management
Your context window will be automatically compacted as it approaches its limit, allowing you to continue working indefinitely. Do not stop tasks early due to context concerns — instead, persist progress and keep going.
**For long-running tasks:** Use git commits, task lists, and structured notes to track state. When context compacts, review git log and any progress files to re-orient. Focus on incremental progress — complete one component before moving to the next, and commit working states along the way.
**Parallel tool calls** — When reading multiple files, running independent searches, or executing unrelated commands, make all calls in parallel rather than sequentially. This significantly speeds up investigation and implementation.
## Known Gotchas
**Electron path resolution** — For bug fixes in the Electron app, check path resolution differences between dev and production builds (`app.isPackaged`, `process.resourcesPath`). Paths that work in dev often break when Electron is bundled for production — verify both contexts.
### Resetting PR Review State
To fully clear all PR review data so reviews run fresh, delete/reset these three things in `.auto-claude/github/`:
1. `rm .auto-claude/github/pr/logs_*.json` — review log files
2. `rm .auto-claude/github/pr/review_*.json` — review result files
3. Reset `pr/index.json` to `{"reviews": [], "last_updated": null}`
4. Reset `bot_detection_state.json` to `{"reviewed_commits": {}}` — this is the gatekeeper; without clearing it, the bot detector skips already-seen commits
**PR target** — Always target the `develop` branch for PRs to AndyMik90/Auto-Claude, NOT `main`.
## Project Structure
```
autonomous-coding/
├── apps/
── desktop/ # Electron desktop application (sole app)
├── prompts/ # Agent system prompts (.md)
── backend/ # Python backend/CLI — ALL agent logic
├── core/ # client.py, auth.py, worktree.py, platform/
│ │ ├── security/ # Command allowlisting, validators, hooks
│ │ ├── agents/ # planner, coder, session management
│ │ ├── qa/ # reviewer, fixer, loop, criteria
│ │ ├── spec/ # Spec creation pipeline
│ │ ├── cli/ # CLI commands (spec, build, workspace, QA)
│ │ ├── context/ # Task context building, semantic search
│ │ ├── runners/ # Standalone runners (spec, roadmap, insights, github)
│ │ ├── services/ # Background services, recovery orchestration
│ │ ├── integrations/ # graphiti/, linear, github
│ │ ├── project/ # Project analysis, security profiles
│ │ ├── merge/ # Intent-aware semantic merge for parallel agents
│ │ └── prompts/ # Agent system prompts (.md)
│ └── frontend/ # Electron desktop UI
│ └── src/
│ ├── main/ # Electron main process
│ │ ├── ai/ # TypeScript AI agent layer (Vercel AI SDK v6)
│ │ │ ├── providers/ # Multi-provider registry + factory (9+ providers)
│ │ │ ├── tools/ # Builtin tools (Read, Write, Edit, Bash, Glob, Grep, etc.)
│ │ │ ├── security/ # Bash validator, command parser, path containment
│ │ │ ├── config/ # Agent configs (25+ types), phase config, model resolution
│ │ │ ├── session/ # streamText() agent loop, error classification, progress
│ │ │ ├── agent/ # Worker thread executor + bridge
│ │ │ ├── orchestration/ # Build pipeline (planner → coder → QA)
│ │ │ ├── runners/ # Utility runners (insights, roadmap, PR review, etc.)
│ │ │ ├── mcp/ # MCP client integration
│ │ │ ├── client/ # Client factory convenience constructors
│ │ │ └── auth/ # Token resolution (reuses claude-profile/)
│ │ ├── agent/ # Agent queue, process, state, events
│ │ ├── claude-profile/ # Multi-profile credentials, token refresh, usage
│ │ ├── terminal/ # PTY daemon, lifecycle, Claude integration
│ │ ├── platform/ # Cross-platform abstraction
│ │ ├── ipc-handlers/# 40+ handler modules by domain
│ │ ├── services/ # Session recovery, profile service
│ │ ├── services/ # SDK session recovery, profile service
│ │ └── changelog/ # Changelog generation and formatting
│ ├── preload/ # Electron preload scripts (electronAPI bridge)
│ ├── renderer/ # React UI
@@ -132,6 +84,7 @@ autonomous-coding/
│ │ └── utils/ # ANSI sanitizer, shell escape, provider detection
│ └── types/ # TypeScript type definitions
├── guides/ # Documentation
├── tests/ # Backend test suite
└── scripts/ # Build and utility scripts
```
@@ -141,15 +94,42 @@ autonomous-coding/
```bash
npm run install:all # Install all dependencies from root
# Or separately:
cd apps/desktop && npm install
cd apps/backend && uv venv && uv pip install -r requirements.txt
cd apps/frontend && npm install
```
### Backend
```bash
cd apps/backend
python spec_runner.py --interactive # Create spec interactively
python spec_runner.py --task "description" # Create from task
python run.py --spec 001 # Run autonomous build
python run.py --spec 001 --qa # Run QA validation
python run.py --spec 001 --merge # Merge completed build
python run.py --list # List all specs
```
### Frontend
```bash
cd apps/frontend
npm run dev # Dev mode (Electron + Vite HMR)
npm run build # Production build
npm run test # Vitest unit tests
npm run test:watch # Vitest watch mode
npm run lint # Biome check
npm run lint:fix # Biome auto-fix
npm run typecheck # TypeScript strict check
npm run package # Package for distribution
```
### Testing
| Stack | Command | Tool |
|-------|---------|------|
| Frontend unit | `cd apps/desktop && npm test` | Vitest |
| Frontend E2E | `cd apps/desktop && npm run test:e2e` | Playwright |
| Backend | `apps/backend/.venv/bin/pytest tests/ -v` | pytest |
| Frontend unit | `cd apps/frontend && npm test` | Vitest |
| Frontend E2E | `cd apps/frontend && npm run test:e2e` | Playwright |
| All backend | `npm run test:backend` (from root) | pytest |
### Releases
```bash
@@ -159,53 +139,38 @@ git push && gh pr create --base main # PR to main triggers release
See [RELEASE.md](RELEASE.md) for full release process.
## AI Agent Layer (`apps/desktop/src/main/ai/`)
## Backend Development
All AI agent logic lives in TypeScript using the Vercel AI SDK v6. This replaces the previous Python `claude-agent-sdk` integration.
### Claude Agent SDK Usage
### Architecture Overview
Client: `apps/backend/core/client.py``create_client()` returns a configured `ClaudeSDKClient` with security hooks, tool permissions, and MCP server integration.
- **Provider Layer** (`providers/`) — Multi-provider support via `createProviderRegistry()`. Supports Anthropic, OpenAI, Google, Bedrock, Azure, Mistral, Groq, xAI, and Ollama. Provider-specific transforms handle thinking token normalization and prompt caching.
- **Session Runtime** (`session/`) — `runAgentSession()` uses `streamText()` with `stopWhen: stepCountIs(N)` for agentic tool-use loops. Includes error classification (429/401/400) and progress tracking.
- **Worker Threads** (`agent/`) — Agent sessions run in `worker_threads` to avoid blocking the Electron main process. The `WorkerBridge` relays `postMessage()` events to the existing `AgentManagerEvents` interface.
- **Build Orchestration** (`orchestration/`) — Full planner → coder → QA pipeline. Parallel subagent execution via `Promise.allSettled()`.
- **Tools** (`tools/`) — 8 builtin tools (Read, Write, Edit, Bash, Glob, Grep, WebFetch, WebSearch) defined with Zod schemas via AI SDK `tool()`.
- **Security** (`security/`) — Bash validator, command parser, and path containment ported from Python with identical allowlist behavior.
- **Config** (`config/`) — `AGENT_CONFIGS` registry (25+ agent types), phase-aware model resolution, thinking budgets.
Model and thinking level are user-configurable (via the Electron UI settings or CLI override). Use `phase_config.py` helpers to resolve the correct values:
### Key Patterns
```python
from core.client import create_client
from phase_config import get_phase_model, get_phase_thinking_budget
```typescript
// Agent session using streamText()
import { streamText, stepCountIs } from 'ai';
# Resolve model/thinking from user settings (Electron UI or CLI override)
phase_model = get_phase_model(spec_dir, "coding", cli_model=None)
phase_thinking = get_phase_thinking_budget(spec_dir, "coding", cli_thinking=None)
const result = streamText({
model: provider,
system: systemPrompt,
messages: conversationHistory,
tools: toolRegistry.getToolsForAgent(agentType),
stopWhen: stepCountIs(1000),
onStepFinish: ({ toolCalls, text, usage }) => {
progressTracker.update(toolCalls, text);
},
});
client = create_client(
project_dir=project_dir,
spec_dir=spec_dir,
model=phase_model,
agent_type="coder", # planner | coder | qa_reviewer | qa_fixer
max_thinking_tokens=phase_thinking,
)
// Tool definition with Zod schema
import { tool } from 'ai';
import { z } from 'zod';
const readTool = tool({
description: 'Read a file from the filesystem',
inputSchema: z.object({
file_path: z.string(),
offset: z.number().optional(),
limit: z.number().optional(),
}),
execute: async ({ file_path, offset, limit }) => { /* ... */ },
});
# Run agent session (uses context manager + run_agent_session helper)
async with client:
status, response = await run_agent_session(client, prompt, spec_dir)
```
### Agent Prompts (`apps/desktop/prompts/`)
Working examples: `agents/planner.py`, `agents/coder.py`, `qa/reviewer.py`, `qa/fixer.py`, `spec/`
### Agent Prompts (`apps/backend/prompts/`)
| Prompt | Purpose |
|--------|---------|
@@ -221,13 +186,13 @@ Each spec in `.auto-claude/specs/XXX-name/` contains: `spec.md`, `requirements.j
### Memory System (Graphiti)
Graph-based semantic memory accessed via a Python MCP sidecar (lives outside `apps/desktop/`). The AI layer connects to it via `createMCPClient` from `@ai-sdk/mcp`. Configured through the Electron app's onboarding/settings UI. See [ARCHITECTURE.md](shared_docs/ARCHITECTURE.md#memory-system) for details.
Graph-based semantic memory in `integrations/graphiti/`. Configured through the Electron app's onboarding/settings UI (CLI users can alternatively set `GRAPHITI_ENABLED=true` in `.env`). See [ARCHITECTURE.md](shared_docs/ARCHITECTURE.md#memory-system) for details.
## Frontend Development
### Tech Stack
React 19, TypeScript (strict), Electron 39, Vercel AI SDK v6, Zustand 5, Tailwind CSS v4, Radix UI, xterm.js 6, Vite 7, Vitest 4, Biome 2, Motion (Framer Motion)
React 19, TypeScript (strict), Electron 39, Zustand 5, Tailwind CSS v4, Radix UI, xterm.js 6, Vite 7, Vitest 4, Biome 2, Motion (Framer Motion)
### Path Aliases (tsconfig.json)
@@ -273,9 +238,9 @@ Main ↔ Renderer communication via Electron IPC:
The frontend manages agent lifecycle end-to-end:
- **`agent-queue.ts`** — Queue routing, prioritization, spec number locking
- **`agent-process.ts`** — Spawns worker threads via `WorkerBridge` for agent execution
- **`agent-process.ts`** — Spawns and manages agent subprocess communication
- **`agent-state.ts`** — Tracks running agent state and status
- **`agent-events.ts`** — Agent lifecycle events and state transitions (structured events from worker threads)
- **`agent-events.ts`** — Agent lifecycle events and state transitions
### Claude Profile System (`src/main/claude-profile/`)
@@ -301,10 +266,13 @@ Full PTY-based terminal integration:
- **Pre-commit:** Husky + lint-staged runs Biome on staged `.ts/.tsx/.js/.jsx/.json`
- **Testing:** Vitest + React Testing Library + jsdom
### Backend
- **Linting:** Ruff
- **Testing:** pytest (`apps/backend/.venv/bin/pytest tests/ -v`)
## i18n Guidelines
All frontend UI text uses `react-i18next`. Translation files: `apps/desktop/src/shared/i18n/locales/{en,fr}/*.json`
All frontend UI text uses `react-i18next`. Translation files: `apps/frontend/src/shared/i18n/locales/{en,fr}/*.json`
**Namespaces:** `common`, `navigation`, `settings`, `dialogs`, `tasks`, `errors`, `onboarding`, `welcome`
@@ -325,7 +293,7 @@ When adding new UI text: add keys to ALL language files, use `namespace:section.
Supports Windows, macOS, Linux. CI tests all three.
**Platform modules:** `apps/desktop/src/main/platform/`
**Platform modules:** `apps/frontend/src/main/platform/` and `apps/backend/core/platform/`
| Function | Purpose |
|----------|---------|
@@ -334,26 +302,27 @@ Supports Windows, macOS, Linux. CI tests all three.
| `findExecutable(name)` | Cross-platform executable lookup |
| `requiresShell(command)` | `.cmd/.bat` shell detection (Win) |
Use `findExecutable()` and `joinPaths()` instead of hardcoded paths. See [ARCHITECTURE.md](shared_docs/ARCHITECTURE.md#cross-platform-development) for extended guide.
Never hardcode paths. Use `findExecutable()` and `joinPaths()`. See [ARCHITECTURE.md](shared_docs/ARCHITECTURE.md#cross-platform-development) for extended guide.
## E2E Testing (Electron MCP)
QA agents can interact with the running Electron app via Chrome DevTools Protocol:
1. Start app: `npm run dev:debug` (debug mode for AI self-validation via Electron MCP)
2. Enable Electron MCP in settings
3. QA runs automatically through the TypeScript agent pipeline
2. Set `ELECTRON_MCP_ENABLED=true` in `apps/backend/.env`
3. Run QA: `python run.py --spec 001 --qa`
Tools: `take_screenshot`, `click_by_text`, `fill_input`, `get_page_structure`, `send_keyboard_shortcut`, `eval`. See [ARCHITECTURE.md](shared_docs/ARCHITECTURE.md#end-to-end-testing) for full capabilities.
## Running the Application
```bash
# CLI only
cd apps/backend && python run.py --spec 001
# Desktop app
npm start # Production build + run
npm run dev # Development mode with HMR
npm run dev:debug # Debug mode with verbose output
npm run dev:mcp # Electron MCP server for AI debugging
# Project data: .auto-claude/specs/ (gitignored)
```
-348
View File
@@ -1,348 +0,0 @@
# Codex Rate Limit Monitoring — Full System Research
> Temporary research file. Delete after implementation.
## Table of Contents
1. [Codex Usage API](#1-codex-usage-api)
2. [Current System Architecture](#2-current-system-architecture)
3. [Anthropic-Hardcoded Locations](#3-anthropic-hardcoded-locations)
4. [Provider-Agnostic Parts (No Changes Needed)](#4-provider-agnostic-parts)
5. [Implementation Plan](#5-implementation-plan)
---
## 1. Codex Usage API
**Sources:** OpenAI Codex source code (`github.com/openai/codex`, Rust codebase), CodexBar macOS app (`github.com/steipete/CodexBar`), Context7 Codex developer docs.
### 1.1 Active Polling Endpoint
```
GET https://chatgpt.com/backend-api/wham/usage
```
Fallback (when base URL doesn't contain `/backend-api`):
```
GET {base_url}/api/codex/usage
```
**Required Headers:**
```http
Authorization: Bearer <access_token>
ChatGPT-Account-Id: <account_id>
Content-Type: application/json
Accept: application/json
```
- `access_token` — The OAuth access token from `auth.openai.com` (same token our `codex-oauth.ts` already obtains)
- `account_id` — Account UUID from OAuth token data. Stored in `~/.codex/auth.json` under `tokens.account_id`. Optional per CodexBar ("when available") but may be required.
### 1.2 Response Schema
From `codex-rs/codex-backend-openapi-models/src/models/rate_limit_status_payload.rs`:
```json
{
"plan_type": "plus",
"rate_limit": {
"allowed": true,
"limit_reached": false,
"primary_window": {
"used_percent": 96,
"limit_window_seconds": 18000,
"reset_after_seconds": 673,
"reset_at": 1730947200
},
"secondary_window": {
"used_percent": 70,
"limit_window_seconds": 604800,
"reset_after_seconds": 43200,
"reset_at": 1730980800
}
},
"credits": {
"has_credits": false,
"unlimited": true,
"balance": null
},
"additional_rate_limits": [
{
"limit_name": "codex_other",
"metered_feature": "codex_other",
"rate_limit": {
"allowed": true,
"limit_reached": false,
"primary_window": {
"used_percent": 70,
"limit_window_seconds": 3600,
"reset_after_seconds": 1800,
"reset_at": 1730947200
}
}
}
]
}
```
- `primary_window` = 5h session (18000s). Maps to our `sessionPercent`.
- `secondary_window` = Weekly (604800s = 7d). Maps to our `weeklyPercent`.
- `reset_at` = Unix timestamp (seconds). Convert to ms for our `sessionResetTimestamp`/`weeklyResetTimestamp`.
- `plan_type` values: `guest`, `free`, `go`, `plus`, `pro`, `free_workspace`, `team`, `business`, `education`, `quorum`, `k12`, `enterprise`, `edu`
### 1.3 Passive Headers (From API Responses)
Rate limit data is also returned in HTTP response headers on every `/v1/responses` call:
```
x-codex-primary-used-percent → float (e.g., "25.0")
x-codex-primary-window-minutes → integer (e.g., "300" for 5h)
x-codex-primary-reset-at → unix timestamp seconds
x-codex-secondary-used-percent → float (weekly)
x-codex-secondary-window-minutes → integer
x-codex-secondary-reset-at → unix timestamp seconds
x-codex-credits-has-credits → "true" or "false"
x-codex-credits-unlimited → "true" or "false"
x-codex-credits-balance → decimal string e.g. "9.99"
```
SSE event type `codex.rate_limits` also carries this data inline in streaming responses.
### 1.4 Token Details
Our `codex-oauth.ts` already uses the correct flow:
- **Client ID:** `app_EMoamEEZ73f0CkXaXp7hrann` (same as Codex CLI)
- **Auth endpoint:** `https://auth.openai.com/oauth/authorize`
- **Token endpoint:** `https://auth.openai.com/oauth/token`
- **Scopes:** `openid profile email offline_access`
- **Refresh:** `POST https://auth.openai.com/oauth/token` with `grant_type=refresh_token`
**Missing:** `account_id` for the `ChatGPT-Account-Id` header. Options:
1. Decode from the JWT access token
2. Read from `~/.codex/auth.json` (`tokens.account_id`)
3. Extract during OAuth token exchange (may be in response)
4. Try without it first (optional per CodexBar docs)
---
## 2. Current System Architecture
### 2.1 Two Parallel Account Systems
The app has TWO account management systems that don't fully integrate:
**System A: Legacy Claude Profile Manager (Main Process)**
- `claude-profile-manager.ts` — Manages OAuth profiles, rate limits, usage, auto-swap
- `claude-profiles.json` — Stores profiles with `activeProfileId`, `accountPriorityOrder`
- `usage-monitor.ts` — Polls Anthropic's `/api/oauth/usage` endpoint every 30s
- `token-refresh.ts` — Refreshes tokens via `console.anthropic.com/v1/oauth/token`
- `rate-limit-detector.ts` — Detects rate limits, triggers auto-swap
- `profile-scorer.ts` — Scores profiles by availability for auto-swap
- **100% Anthropic-specific.** Only knows about Anthropic OAuth tokens, Anthropic endpoints, Anthropic keychain format.
**System B: Multi-Provider Accounts (Renderer + Settings)**
- `ProviderAccount[]` in `settings-store.ts` — All connected accounts (any provider)
- `globalPriorityOrder: string[]` in AppSettings — Manual priority queue
- `useActiveProvider()` hook — First account in priority order = active
- **Provider-agnostic.** Works for all 10 providers. But has NO usage monitoring, NO auto-swap.
**The gap:** System A handles usage monitoring + auto-swap but only for Anthropic. System B handles multi-provider accounts but has no usage awareness.
### 2.2 Data Flow: Usage Polling
```
UsageMonitor.start() → 30s interval
checkUsageAndSwap()
├─ determineActiveProfile() ← Hardcoded: defaults to anthropic baseUrl
├─ getCredential() ← Hardcoded: reads from Anthropic keychain
│ └─ ensureValidToken(configDir) ← Hardcoded: refreshes via Anthropic endpoint
├─ fetchUsageViaAPI() ← Hardcoded: only allows anthropic/zai/zhipu domains
│ ├─ getUsageEndpoint(provider) ← Only 3 providers configured
│ ├─ Add anthropic-specific headers ← if (provider === 'anthropic') add beta headers
│ └─ Parse response ← Provider-specific normalization
├─ emit('usage-updated') → IPC 'claude:usageUpdated' → renderer
├─ emit('all-profiles-usage-updated') → IPC 'claude:allProfilesUsageUpdated' → renderer
└─ checkThresholdsExceeded()
└─ performProactiveSwap() ← Only swaps Anthropic profiles
```
### 2.3 Data Flow: Account Swapping
**Manual swap (UI):**
```
User clicks account in UsageIndicator popover
→ handleSwapAccount(accountId)
→ setQueueOrder([accountId, ...rest]) ← Reorders globalPriorityOrder
→ requestUsageUpdate() ← Refreshes usage display
```
**Automatic swap (rate limit hit):**
```
SDK operation fails with 429
→ detectRateLimit(output) ← Pattern: "Limit reached · resets..."
→ recordRateLimitEvent(profileId)
→ getBestAvailableProfileEnv()
→ profileManager.setActiveProfile() ← Only updates claude-profiles.json
→ usageMonitor.getAllProfilesUsage() ← Refreshes UI
← Returns new profile env vars
```
**Problem:** Auto-swap updates `claude-profiles.json` but NOT `globalPriorityOrder`. The renderer's priority queue may be out of sync.
### 2.4 UI Components
| Component | What it shows | Provider-specific? |
|---|---|---|
| `AuthStatusIndicator` | Provider badge (OpenAI/Anthropic) + auth type label | Codex = green "Codex", Anthropic = orange "OAuth" |
| `UsageIndicator` | Usage bars OR "Subscription" OR "Unlimited" | Anthropic OAuth = bars, Codex OAuth = "Subscription", API = "Unlimited" |
| `ProviderAccountCard` | Account card in settings with usage bars | Shows usage bars only when `account.usage` populated (Anthropic only) |
| `ProviderAccountsList` | All accounts grouped by provider | Generic, but re-auth routes differ per provider |
| `AddAccountDialog` | OAuth flow + account creation | Different flows: Codex → `codexAuthLogin()`, Anthropic → `claudeAuthLoginSubprocess()` |
| `ProviderSection` | Provider group with "Add" buttons | Button label: "Add Codex Subscription" vs "Add OAuth" |
### 2.5 Type Naming
Types use "Claude" prefix but are structurally generic:
```typescript
ClaudeUsageSnapshot { sessionPercent, weeklyPercent, resetTimestamps, profileId, ... }
ClaudeUsageData { sessionUsagePercent, weeklyUsagePercent }
ClaudeRateLimitEvent { type, hitAt, resetAt }
ProfileUsageSummary { sessionPercent, weeklyPercent, availabilityScore, ... }
AllProfilesUsage { activeProfile, allProfiles[], fetchedAt }
```
These types work perfectly for Codex data — same session/weekly model. No structural changes needed, just need to populate them.
---
## 3. Anthropic-Hardcoded Locations
### 3.1 CRITICAL — Must Change
| File | Line(s) | What's hardcoded | What to do |
|---|---|---|---|
| `usage-monitor.ts:45-49` | `ALLOWED_USAGE_API_DOMAINS` | Only `api.anthropic.com`, `api.z.ai`, `open.bigmodel.cn` | Add `chatgpt.com` |
| `usage-monitor.ts:60-73` | `PROVIDER_USAGE_ENDPOINTS` | Only anthropic/zai/zhipu paths | Add `{ provider: 'openai', usagePath: '/wham/usage' }` |
| `usage-monitor.ts:662,1069,1346,1359` | `baseUrl: 'https://api.anthropic.com'` | Hardcoded fallback for all OAuth profiles | Detect provider from account, use `chatgpt.com/backend-api` for Codex |
| `usage-monitor.ts:1424` | `if (provider === 'anthropic')` adds beta headers | Anthropic-specific `anthropic-beta` header | Add `else if (provider === 'openai')` to add `ChatGPT-Account-Id` header |
| `token-refresh.ts:31` | `ANTHROPIC_TOKEN_ENDPOINT = 'https://console.anthropic.com/v1/oauth/token'` | Only Anthropic refresh endpoint | Route to `auth.openai.com/oauth/token` for Codex |
| `token-refresh.ts:37` | `CLAUDE_CODE_CLIENT_ID = '9d1c250a-...'` | Only Anthropic client ID | Use `app_EMoamEEZ73f0CkXaXp7hrann` for Codex |
| `UsageIndicator.tsx:118` | `provider === 'anthropic' && authType === 'oauth'` | Only Anthropic gets usage bars | Add `\|\| provider === 'openai'` |
### 3.2 MODERATE — Should Change
| File | Line(s) | What's hardcoded | What to do |
|---|---|---|---|
| `usage-monitor.ts:1040-1072` | `determineActiveProfile()` | Returns `baseUrl: 'https://api.anthropic.com'` for all OAuth | Detect provider, return `chatgpt.com/backend-api` for Codex |
| `credential-utils.ts` | Keychain service names | `"Claude Code-credentials"` | Codex tokens stored differently (file-based, not keychain) |
| `usage-monitor.ts:1513` | `if (provider === 'zai' \|\| provider === 'zhipu')` | Provider-specific response unwrapping | Add Codex response parsing (different JSON structure) |
| `rate-limit-detector.ts:14` | `RATE_LIMIT_PATTERN` | Claude-specific: `"Limit reached · resets..."` | Add Codex-specific patterns |
| IPC channel names | `'claude:usageUpdated'`, `'claude:allProfilesUsageUpdated'` | "claude" prefix | Cosmetic — rename to `'usage:updated'` etc. (optional, low priority) |
### 3.3 LOW PRIORITY — Nice to Have
| Item | What | Why low priority |
|---|---|---|
| Type naming | `ClaudeUsageSnapshot``UsageSnapshot` | Structural refactor, types work as-is for Codex |
| IPC method names | `requestUsageUpdate` returns `ClaudeUsageSnapshot` | Works fine, just naming |
| `claudeProfileId` on `ProviderAccount` | Only used for Anthropic OAuth | Codex doesn't need it |
---
## 4. Provider-Agnostic Parts
These components already work for any provider and need NO changes:
| Component/Module | Why it's already generic |
|---|---|
| `profile-scorer.ts` | Scores by `billingModel`, usage thresholds, rate limit events — no provider checks |
| `rate-limit-manager.ts` | Stores/checks rate limit events — pure data, no provider logic |
| `operation-registry.ts` | Tracks running operations — no provider awareness |
| `ProviderAccount` type | Has `provider` field, `billingModel`, `usage` — works for any provider |
| `globalPriorityOrder` | Array of account IDs — provider-agnostic ordering |
| `useActiveProvider()` hook | Returns first account in priority order — generic |
| `ProviderAccountCard` | Shows usage bars when `account.usage` is populated — will work for Codex once data flows |
| `AddAccountDialog` | Already has separate Codex OAuth flow |
| `AuthStatusIndicator` | Already shows Codex-specific green badge |
| All i18n keys | Codex-specific labels already exist |
---
## 5. Implementation Plan
### Phase 1: Codex Usage Fetcher (Core)
Create `apps/desktop/src/main/claude-profile/codex-usage-fetcher.ts`:
```typescript
// Responsibilities:
// 1. Read Codex OAuth token (from our codex-auth.json)
// 2. Read account_id (from ~/.codex/auth.json or JWT decode)
// 3. Call GET https://chatgpt.com/backend-api/wham/usage
// 4. Parse response into ClaudeUsageSnapshot format
// 5. Handle 401 → refresh token via codex-oauth.ts
// 6. Handle 403 → mark as needsReauthentication
```
**Key function:**
```typescript
async function fetchCodexUsage(accessToken: string, accountId?: string): Promise<ClaudeUsageSnapshot>
```
### Phase 2: Wire into Usage Monitor
Modify `usage-monitor.ts`:
1. Add `chatgpt.com` to `ALLOWED_USAGE_API_DOMAINS`
2. Add Codex to `PROVIDER_USAGE_ENDPOINTS`
3. Update `determineActiveProfile()` to detect Codex accounts from `globalPriorityOrder`
4. Update `getCredential()` to read Codex OAuth token (from `codex-auth.json`)
5. Update `fetchUsageViaAPI()` to handle Codex response format
6. Add Codex-specific headers (`ChatGPT-Account-Id`)
7. Add Codex response parsing (different JSON structure than Anthropic)
### Phase 3: Token Refresh Routing
Modify `token-refresh.ts` or create parallel Codex path:
- When refreshing a Codex token, use `auth.openai.com/oauth/token` with Codex client ID
- When refreshing an Anthropic token, use `console.anthropic.com/v1/oauth/token` with Claude client ID
- Provider detection: check the account's `provider` field, or detect from token prefix
### Phase 4: UI Updates
1. `UsageIndicator.tsx:118` — Add `|| provider === 'openai'` to `hasUsageMonitoring`
2. That's it — the rest of the UI already handles usage bars, reset times, multi-profile display generically
### Phase 5: Auto-Swap for Codex
1. Add Codex-specific rate limit patterns to `rate-limit-detector.ts`
2. Codex returns `"codexErrorInfo": "UsageLimitExceeded"` on limit hit
3. Auto-swap logic in `profile-scorer.ts` already works — it just needs usage data populated
---
## Appendix: Comparison Table
| Aspect | Anthropic (Claude Code) | OpenAI (Codex) |
|---|---|---|
| **Usage endpoint** | `api.anthropic.com/api/oauth/usage` | `chatgpt.com/backend-api/wham/usage` |
| **Auth header** | `Bearer <oauth_token>` | `Bearer <access_token>` + `ChatGPT-Account-Id` |
| **Session window** | ~5h | Configurable (`limit_window_seconds`) |
| **Weekly window** | 7 days | Configurable (`limit_window_seconds`) |
| **Token source** | Keychain (`Claude Code-credentials`) | File (`codex-auth.json`) |
| **Token refresh** | `console.anthropic.com/v1/oauth/token` | `auth.openai.com/oauth/token` |
| **Client ID** | `9d1c250a-e61b-44d9-88ed-5944d1962f5e` | `app_EMoamEEZ73f0CkXaXp7hrann` |
| **Passive tracking** | Not available | `x-codex-*` response headers |
| **Rate limit error** | `"Limit reached · resets Dec 17..."` | `"codexErrorInfo": "UsageLimitExceeded"` |
| **Profile isolation** | `~/.claude-profiles/{name}/` dirs | Single `codex-auth.json` file |
| **Multi-account** | Multiple config dirs in keychain | Single file (no multi-account yet) |
## Appendix: Caveats
1. **Undocumented API**`chatgpt.com/backend-api/wham/usage` is internal. The Codex CLI depends on it, so it's unlikely to break silently.
2. **Account ID** — May be required. Test without it first. If needed, decode from JWT or read `~/.codex/auth.json`.
3. **CORS** — Not an issue (Electron main process = Node.js).
4. **Polling rate** — Unknown if OpenAI rate-limits `wham/usage`. Start conservatively (every 30-60s).
5. **Multi-account Codex** — Codex CLI doesn't support multiple accounts. We store one token file. If user has multiple Codex accounts, they'd need to re-auth each time (unlike Anthropic which supports multiple config dirs).
+223 -41
View File
@@ -73,11 +73,35 @@ Read the full CLA here: [CLA.md](CLA.md)
Before contributing, ensure you have the following installed:
- **Node.js 24+** - For the Electron desktop app
- **npm 10+** - Package manager (comes with Node.js)
- **CMake** - Required for building native dependencies (e.g., node-pty)
- **Python 3.12+** - For the backend framework
- **Node.js 24+** - For the Electron frontend
- **npm 10+** - Package manager for the frontend (comes with Node.js)
- **uv** (recommended) or **pip** - Python package manager
- **CMake** - Required for building native dependencies (e.g., LadybugDB)
- **Git** - Version control
### Installing Python 3.12
**Windows:**
```bash
winget install Python.Python.3.12
```
**macOS:**
```bash
brew install python@3.12
```
**Linux (Ubuntu/Debian):**
```bash
sudo apt install python3.12 python3.12-venv
```
**Linux (Fedora):**
```bash
sudo dnf install python3.12
```
### Installing Node.js 24+
**Windows:**
@@ -144,27 +168,43 @@ npm start
## Development Setup
The project is a single Electron desktop application in `apps/desktop/`. All AI agent logic lives in TypeScript using the Vercel AI SDK v6.
The project consists of two main components:
From the repository root:
1. **Python Backend** (`apps/backend/`) - The core autonomous coding framework
2. **Electron Frontend** (`apps/frontend/`) - Desktop UI
From the repository root, two commands handle everything:
```bash
# Install all dependencies
# Install all dependencies (Python backend + Electron frontend)
npm run install:all
# Start development mode (hot reload)
npm run dev
```
`npm run install:all` installs the npm dependencies for `apps/desktop/`.
`npm run install:all` automatically:
- Detects Python 3.12+ on your system
- Creates a virtual environment (`apps/backend/.venv`)
- Installs backend runtime and test dependencies
- Copies `.env.example` to `.env` (if not already present)
- Installs frontend npm dependencies
After install, configure your credentials in `apps/backend/.env`:
```bash
# Get your Claude Code OAuth token
claude setup-token
# Then edit apps/backend/.env with your token and any other provider keys
```
### Other Useful Commands
```bash
npm start # Build and run production
npm run build # Build for production
npm run build # Build frontend for production
npm run package # Package for distribution
npm test # Run frontend tests
npm run test:backend # Run Python tests
```
<details>
@@ -183,20 +223,28 @@ Auto Claude automatically downloads prebuilt binaries for Windows. If prebuilts
## Pre-commit Hooks
We use Husky + lint-staged to run Biome linting and formatting checks before each commit.
We use [pre-commit](https://pre-commit.com/) to run linting and formatting checks before each commit. This ensures code quality and consistency across the project.
### Setup
Husky is installed automatically when you run `npm install` inside `apps/desktop/`.
```bash
# Install pre-commit
pip install pre-commit
# Install the git hooks (run once after cloning)
pre-commit install
```
### What Runs on Commit
When you commit, the following checks run automatically on staged files:
When you commit, the following checks run automatically:
| Check | Scope | Description |
|-------|-------|-------------|
| **Biome** | `apps/desktop/` | TypeScript/React linter + formatter |
| **typecheck** | `apps/desktop/` | TypeScript type checking |
| **ruff** | `apps/backend/` | Python linter with auto-fix |
| **ruff-format** | `apps/backend/` | Python code formatter |
| **eslint** | `apps/frontend/` | TypeScript/React linter |
| **typecheck** | `apps/frontend/` | TypeScript type checking |
| **trailing-whitespace** | All files | Removes trailing whitespace |
| **end-of-file-fixer** | All files | Ensures files end with newline |
| **check-yaml** | All files | Validates YAML syntax |
@@ -205,29 +253,55 @@ When you commit, the following checks run automatically on staged files:
### Running Manually
```bash
cd apps/desktop
# Run all checks on all files
pre-commit run --all-files
# Run linter (Biome)
npm run lint
# Run a specific hook
pre-commit run ruff --all-files
# Auto-fix lint issues
npm run lint:fix
# Run type checking
npm run typecheck
# Skip hooks temporarily (not recommended)
git commit --no-verify -m "message"
```
### If a Check Fails
1. **Biome auto-fixes**: Run `npm run lint:fix` in `apps/desktop/`. Stage the changes and commit again.
2. **Type errors**: Resolve TypeScript type issues before committing.
1. **Ruff auto-fixes**: Some issues are fixed automatically. Stage the changes and commit again.
2. **ESLint errors**: Fix the reported issues in your code.
3. **Type errors**: Resolve TypeScript type issues before committing.
## Code Style
### Python
- Follow PEP 8 style guidelines
- Use type hints for function signatures
- Use docstrings for public functions and classes
- Keep functions focused and under 50 lines when possible
- Use meaningful variable and function names
```python
# Good
def get_next_chunk(spec_dir: Path) -> dict | None:
"""
Find the next pending chunk in the implementation plan.
Args:
spec_dir: Path to the spec directory
Returns:
The next chunk dict or None if all chunks are complete
"""
...
# Avoid
def gnc(sd):
...
```
### TypeScript/React
- Use TypeScript strict mode
- Follow the existing component patterns in `apps/desktop/src/`
- Follow the existing component patterns in `apps/frontend/src/`
- Use functional components with hooks
- Prefer named exports over default exports
- Use the UI components from `src/renderer/components/ui/`
@@ -252,12 +326,96 @@ export default function(props) {
- End files with a newline
- Keep line length under 100 characters when practical
### File Encoding (Python)
**Always specify `encoding="utf-8"` for text file operations** to ensure Windows compatibility.
Windows Python defaults to `cp1252` encoding instead of UTF-8, causing errors with:
- Emoji (🚀, ✅, ❌)
- International characters (ñ, é, 中文, العربية)
- Special symbols (™, ©, ®)
**DO:**
```python
# Reading files
with open(path, encoding="utf-8") as f:
content = f.read()
# Writing files
with open(path, "w", encoding="utf-8") as f:
f.write(content)
# Path methods
from pathlib import Path
content = Path(file).read_text(encoding="utf-8")
Path(file).write_text(content, encoding="utf-8")
# JSON files - reading
import json
with open(path, encoding="utf-8") as f:
data = json.load(f)
# JSON files - writing
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False, indent=2)
```
**DON'T:**
```python
# Wrong - platform-dependent encoding
with open(path) as f:
content = f.read()
# Wrong - Path methods without encoding
content = Path(file).read_text()
# Wrong - encoding on json.dump (not open!)
json.dump(data, f, encoding="utf-8") # ERROR
```
**Binary files - NO encoding:**
```python
with open(path, "rb") as f: # Correct
data = f.read()
```
Our pre-commit hooks automatically check for missing encoding parameters. See [PR #782](https://github.com/AndyMik90/Auto-Claude/pull/782) for the comprehensive encoding fix and [guides/windows-development.md](guides/windows-development.md) for Windows-specific development guidance.
## Testing
### Python Tests
```bash
# Run all tests (from repository root)
npm run test:backend
# Or manually with pytest
cd apps/backend
.venv/Scripts/pytest.exe ../tests -v # Windows
.venv/bin/pytest ../tests -v # macOS/Linux
# Run a specific test file
npm run test:backend -- tests/test_security.py -v
# Run a specific test
npm run test:backend -- tests/test_security.py::test_bash_command_validation -v
# Skip slow tests
npm run test:backend -- -m "not slow"
# Run with coverage
pytest tests/ --cov=apps/backend --cov-report=html
```
Test configuration is in `tests/pytest.ini`.
### Frontend Tests
```bash
cd apps/desktop
cd apps/frontend
# Run unit tests
npm test
@@ -296,22 +454,29 @@ All pull requests and pushes to `main` trigger automated CI checks via GitHub Ac
| Workflow | Trigger | What it checks |
|----------|---------|----------------|
| **CI** | Push to `main`, PRs | Frontend tests (all 3 platforms), TypeScript type check, build |
| **Lint** | Push to `main`, PRs | Biome (TypeScript/React) |
| **CI** | Push to `main`, PRs | Python tests (3.11 & 3.12), Frontend tests |
| **Lint** | Push to `main`, PRs | Ruff (Python), ESLint + TypeScript (Frontend) |
### PR Requirements
Before a PR can be merged:
1. All CI checks must pass (green checkmarks)
2. Frontend tests pass on all three platforms (Ubuntu, Windows, macOS)
3. Linting passes (no Biome errors)
4. TypeScript type checking passes
2. Python tests pass on both Python 3.11 and 3.12
3. Frontend tests pass
4. Linting passes (no ruff or eslint errors)
5. TypeScript type checking passes
### Running CI Checks Locally
```bash
cd apps/desktop
# Python tests
cd apps/backend
source .venv/bin/activate
pytest ../../tests/ -v
# Frontend tests
cd apps/frontend
npm test
npm run lint
npm run typecheck
@@ -622,7 +787,8 @@ git rebase -i origin/develop
git push --force-with-lease
# Verify everything works
cd apps/desktop && npm test && npm run lint && npm run typecheck
npm run test:backend
cd apps/frontend && npm test && npm run lint && npm run typecheck
```
**PR size:**
@@ -643,7 +809,11 @@ cd apps/desktop && npm test && npm run lint && npm run typecheck
3. **Test thoroughly**:
```bash
cd apps/desktop && npm test && npm run lint && npm run typecheck
# Python (from repository root)
npm run test:backend
# Frontend
cd apps/frontend && npm test && npm run lint && npm run typecheck
```
4. **Update documentation** if your changes affect:
@@ -681,7 +851,8 @@ When reporting a bug, include:
1. **Clear title** describing the issue
2. **Environment details**:
- OS and version
- Node.js version
- Python version
- Node.js version (for UI issues)
- Auto Claude version
3. **Steps to reproduce** the issue
4. **Expected behavior** vs **actual behavior**
@@ -699,14 +870,25 @@ When requesting a feature:
## Architecture Overview
Auto Claude is a single Electron desktop application in `apps/desktop/`.
Auto Claude consists of two main parts:
### Electron Desktop (`apps/desktop/`)
### Python Backend (`apps/backend/`)
- **AI Agent Layer** (`src/main/ai/`) - Vercel AI SDK v6 agent runtime, providers, tools, security, orchestration
- **Main Process** (`src/main/`) - IPC handlers, agent queue, terminal management, claude-profile
- **Renderer** (`src/renderer/`) - React UI components and Zustand stores
- **Shared** (`src/shared/`) - Types, i18n locales, constants, utilities
The core autonomous coding framework:
- **Entry Points**: `run.py` (build runner), `spec_runner.py` (spec creator)
- **Agent System**: `agent.py`, `client.py`, `prompts/`
- **Execution**: `coordinator.py` (parallel), `worktree.py` (isolation)
- **Memory**: `memory.py` (file-based), `graphiti_memory.py` (graph-based)
- **QA**: `qa_loop.py`, `prompts/qa_*.md`
### Electron Frontend (`apps/frontend/`)
Desktop interface:
- **Main Process**: `src/main/` - Electron main process, IPC handlers
- **Renderer**: `src/renderer/` - React UI components
- **Shared**: `src/shared/` - Types and utilities
For detailed architecture information, see [CLAUDE.md](CLAUDE.md).
-2156
View File
File diff suppressed because it is too large Load Diff
+46 -22
View File
@@ -1,14 +1,13 @@
# Aperant (formerly Auto Claude)
# Auto Claude
**Autonomous multi-agent coding framework that plans, builds, and validates software for you.**
![Aperant Kanban Board](.github/assets/Auto-Claude-Kanban.png)
![Auto Claude Kanban Board](.github/assets/Auto-Claude-Kanban.png)
[![License](https://img.shields.io/badge/license-AGPL--3.0-green?style=flat-square)](./agpl-3.0.txt)
[![Discord](https://img.shields.io/badge/Discord-Join%20Community-5865F2?style=flat-square&logo=discord&logoColor=white)](https://discord.gg/KCXaPBr4Dj)
[![YouTube](https://img.shields.io/badge/YouTube-Subscribe-FF0000?style=flat-square&logo=youtube&logoColor=white)](https://www.youtube.com/@AndreMikalsen)
[![CI](https://img.shields.io/github/actions/workflow/status/AndyMik90/Auto-Claude/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/AndyMik90/Auto-Claude/actions)
[![Mentioned in Awesome Claude Code](https://awesome.re/mentioned-badge-flat.svg)](https://github.com/hesreallyhim/awesome-claude-code)
---
@@ -17,18 +16,18 @@
### Stable Release
<!-- STABLE_VERSION_BADGE -->
[![Stable](https://img.shields.io/badge/stable-2.7.6-blue?style=flat-square)](https://github.com/AndyMik90/Auto-Claude/releases/tag/v2.7.6)
[![Stable](https://img.shields.io/badge/stable-2.7.5-blue?style=flat-square)](https://github.com/AndyMik90/Auto-Claude/releases/tag/v2.7.5)
<!-- STABLE_VERSION_BADGE_END -->
<!-- STABLE_DOWNLOADS -->
| Platform | Download |
|----------|----------|
| **Windows** | [Auto-Claude-2.7.6-win32-x64.exe](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.6/Auto-Claude-2.7.6-win32-x64.exe) |
| **macOS (Apple Silicon)** | [Auto-Claude-2.7.6-darwin-arm64.dmg](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.6/Auto-Claude-2.7.6-darwin-arm64.dmg) |
| **macOS (Intel)** | [Auto-Claude-2.7.6-darwin-x64.dmg](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.6/Auto-Claude-2.7.6-darwin-x64.dmg) |
| **Linux** | [Auto-Claude-2.7.6-linux-x86_64.AppImage](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.6/Auto-Claude-2.7.6-linux-x86_64.AppImage) |
| **Linux (Debian)** | [Auto-Claude-2.7.6-linux-amd64.deb](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.6/Auto-Claude-2.7.6-linux-amd64.deb) |
| **Linux (Flatpak)** | [Auto-Claude-2.7.6-linux-x86_64.flatpak](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.6/Auto-Claude-2.7.6-linux-x86_64.flatpak) |
| **Windows** | [Auto-Claude-2.7.5-win32-x64.exe](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.5/Auto-Claude-2.7.5-win32-x64.exe) |
| **macOS (Apple Silicon)** | [Auto-Claude-2.7.5-darwin-arm64.dmg](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.5/Auto-Claude-2.7.5-darwin-arm64.dmg) |
| **macOS (Intel)** | [Auto-Claude-2.7.5-darwin-x64.dmg](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.5/Auto-Claude-2.7.5-darwin-x64.dmg) |
| **Linux** | [Auto-Claude-2.7.5-linux-x86_64.AppImage](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.5/Auto-Claude-2.7.5-linux-x86_64.AppImage) |
| **Linux (Debian)** | [Auto-Claude-2.7.5-linux-amd64.deb](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.5/Auto-Claude-2.7.5-linux-amd64.deb) |
| **Linux (Flatpak)** | [Auto-Claude-2.7.5-linux-x86_64.flatpak](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.5/Auto-Claude-2.7.5-linux-x86_64.flatpak) |
<!-- STABLE_DOWNLOADS_END -->
### Beta Release
@@ -36,18 +35,18 @@
> ⚠️ Beta releases may contain bugs and breaking changes. [View all releases](https://github.com/AndyMik90/Auto-Claude/releases)
<!-- BETA_VERSION_BADGE -->
[![Beta](https://img.shields.io/badge/beta-2.8.0--beta.5-orange?style=flat-square)](https://github.com/AndyMik90/Auto-Claude/releases/tag/v2.8.0-beta.5)
[![Beta](https://img.shields.io/badge/beta-2.7.2--beta.10-orange?style=flat-square)](https://github.com/AndyMik90/Auto-Claude/releases/tag/v2.7.2-beta.10)
<!-- BETA_VERSION_BADGE_END -->
<!-- BETA_DOWNLOADS -->
| Platform | Download |
|----------|----------|
| **Windows** | [Aperant-2.8.0-beta.5-win32-x64.exe](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.8.0-beta.5/Aperant-2.8.0-beta.5-win32-x64.exe) |
| **macOS (Apple Silicon)** | [Aperant-2.8.0-beta.5-darwin-arm64.dmg](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.8.0-beta.5/Aperant-2.8.0-beta.5-darwin-arm64.dmg) |
| **macOS (Intel)** | [Aperant-2.8.0-beta.5-darwin-x64.dmg](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.8.0-beta.5/Aperant-2.8.0-beta.5-darwin-x64.dmg) |
| **Linux** | [Aperant-2.8.0-beta.5-linux-x86_64.AppImage](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.8.0-beta.5/Aperant-2.8.0-beta.5-linux-x86_64.AppImage) |
| **Linux (Debian)** | [Aperant-2.8.0-beta.5-linux-amd64.deb](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.8.0-beta.5/Aperant-2.8.0-beta.5-linux-amd64.deb) |
| **Linux (Flatpak)** | [Aperant-2.8.0-beta.5-linux-x86_64.flatpak](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.8.0-beta.5/Aperant-2.8.0-beta.5-linux-x86_64.flatpak) |
| **Windows** | [Auto-Claude-2.7.2-beta.10-win32-x64.exe](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.2-beta.10/Auto-Claude-2.7.2-beta.10-win32-x64.exe) |
| **macOS (Apple Silicon)** | [Auto-Claude-2.7.2-beta.10-darwin-arm64.dmg](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.2-beta.10/Auto-Claude-2.7.2-beta.10-darwin-arm64.dmg) |
| **macOS (Intel)** | [Auto-Claude-2.7.2-beta.10-darwin-x64.dmg](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.2-beta.10/Auto-Claude-2.7.2-beta.10-darwin-x64.dmg) |
| **Linux** | [Auto-Claude-2.7.2-beta.10-linux-x86_64.AppImage](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.2-beta.10/Auto-Claude-2.7.2-beta.10-linux-x86_64.AppImage) |
| **Linux (Debian)** | [Auto-Claude-2.7.2-beta.10-linux-amd64.deb](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.2-beta.10/Auto-Claude-2.7.2-beta.10-linux-amd64.deb) |
| **Linux (Flatpak)** | [Auto-Claude-2.7.2-beta.10-linux-x86_64.flatpak](https://github.com/AndyMik90/Auto-Claude/releases/download/v2.7.2-beta.10/Auto-Claude-2.7.2-beta.10-linux-x86_64.flatpak) |
<!-- BETA_DOWNLOADS_END -->
> All releases include SHA256 checksums and VirusTotal scan results for security verification.
@@ -114,15 +113,39 @@ AI-assisted feature planning with competitor analysis and audience targeting.
## Project Structure
```
Aperant/
Auto-Claude/
├── apps/
── desktop/ # Electron desktop application (TypeScript AI agent layer + UI)
── backend/ # Python agents, specs, QA pipeline
│ └── frontend/ # Electron desktop application
├── guides/ # Additional documentation
├── tests/ # Test suite
└── scripts/ # Build utilities
```
---
## CLI Usage
For headless operation, CI/CD integration, or terminal-only workflows:
```bash
cd apps/backend
# Create a spec interactively
python spec_runner.py --interactive
# Run autonomous build
python run.py --spec 001
# Review and merge
python run.py --spec 001 --review
python run.py --spec 001 --merge
```
See [guides/CLI-USAGE.md](guides/CLI-USAGE.md) for complete CLI documentation.
---
## Development
Want to build from source or contribute? See [CONTRIBUTING.md](CONTRIBUTING.md) for complete development setup instructions.
@@ -133,7 +156,7 @@ For Linux-specific builds (Flatpak, AppImage), see [guides/linux.md](guides/linu
## Security
Aperant uses a three-layer security model:
Auto Claude uses a three-layer security model:
1. **OS Sandbox** - Bash commands run in isolation
2. **Filesystem Restrictions** - Operations limited to project directory
@@ -150,7 +173,7 @@ All releases are:
| Command | Description |
|---------|-------------|
| `npm run install:all` | Install all dependencies |
| `npm run install:all` | Install backend and frontend dependencies |
| `npm start` | Build and run the desktop app |
| `npm run dev` | Run in development mode with hot reload |
| `npm run package` | Package for current platform |
@@ -160,6 +183,7 @@ All releases are:
| `npm run package:flatpak` | Package as Flatpak (see [guides/linux.md](guides/linux.md)) |
| `npm run lint` | Run linter |
| `npm test` | Run frontend tests |
| `npm run test:backend` | Run backend tests |
---
@@ -185,7 +209,7 @@ We welcome contributions! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for:
**AGPL-3.0** - GNU Affero General Public License v3.0
Aperant is free to use. If you modify and distribute it, or run it as a service, your code must also be open source under AGPL-3.0.
Auto Claude is free to use. If you modify and distribute it, or run it as a service, your code must also be open source under AGPL-3.0.
Commercial licensing available for closed-source use cases.
+3 -2
View File
@@ -66,8 +66,9 @@ node scripts/bump-version.js 2.8.0 # Set specific version
```
This will:
- Update `apps/desktop/package.json`
- Update `apps/frontend/package.json`
- Update `package.json` (root)
- Update `apps/backend/__init__.py`
- Check if `CHANGELOG.md` has an entry for the new version (warns if missing)
- Create a commit with message `chore: bump version to X.Y.Z`
@@ -194,7 +195,7 @@ The release workflow **validates** that `CHANGELOG.md` has an entry for the vers
1. Check if version in `package.json` is greater than latest tag:
```bash
git tag -l 'v*' --sort=-version:refname | head -1
cat apps/desktop/package.json | grep version
cat apps/frontend/package.json | grep version
```
2. Ensure the merge commit touched `package.json`:
+372
View File
@@ -0,0 +1,372 @@
# Auto Claude Environment Variables
# Copy this file to .env and fill in your values
# =============================================================================
# AUTHENTICATION (REQUIRED)
# =============================================================================
# Auto Claude uses Claude Code OAuth authentication.
# Direct API keys (ANTHROPIC_API_KEY) are NOT supported to prevent silent billing.
#
# Option 1: Run `claude setup-token` to save token to system keychain (recommended)
# (macOS: Keychain, Windows: Credential Manager, Linux: secret-service)
# Option 2: Set the token explicitly:
# CLAUDE_CODE_OAUTH_TOKEN=your-oauth-token-here
#
# For enterprise/proxy setups (CCR):
# ANTHROPIC_AUTH_TOKEN=sk-zcf-x-ccr
# =============================================================================
# CUSTOM API ENDPOINT (OPTIONAL)
# =============================================================================
# Override the default Anthropic API endpoint. Useful for:
# - Local proxies (ccr, litellm)
# - API gateways
# - Self-hosted Claude instances
#
# ANTHROPIC_BASE_URL=http://127.0.0.1:3456
#
# Related settings (usually set together with ANTHROPIC_BASE_URL):
# NO_PROXY=127.0.0.1
# DISABLE_TELEMETRY=true
# DISABLE_COST_WARNINGS=true
# API_TIMEOUT_MS=600000
# Model override (OPTIONAL)
# Default: claude-opus-4-5-20251101
# AUTO_BUILD_MODEL=claude-opus-4-5-20251101
# =============================================================================
# GIT/WORKTREE SETTINGS (OPTIONAL)
# =============================================================================
# Configure how Auto Claude handles git worktrees for isolated builds.
# Default base branch for worktree creation (OPTIONAL)
# If not set, Auto Claude will auto-detect main/master, or fall back to current branch.
# Common values: main, master, develop
# DEFAULT_BRANCH=main
# =============================================================================
# DEBUG MODE (OPTIONAL)
# =============================================================================
# Enable debug logging for development and troubleshooting.
# Shows detailed information about runner execution, agent calls, file operations.
# Enable debug mode (default: false)
# DEBUG=true
# Debug log level: 1=basic, 2=detailed, 3=verbose (default: 1)
# DEBUG_LEVEL=1
# Log to file instead of stdout (OPTIONAL)
# DEBUG_LOG_FILE=auto-claude/debug.log
# =============================================================================
# LINEAR INTEGRATION (OPTIONAL)
# =============================================================================
# Enable Linear integration for real-time progress tracking in Linear.
# Get your API key from: https://linear.app/YOUR-TEAM/settings/api
# Linear API Key (OPTIONAL - enables Linear integration)
# LINEAR_API_KEY=lin_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Pre-configured Team ID (OPTIONAL - will auto-detect if not set)
# LINEAR_TEAM_ID=
# Pre-configured Project ID (OPTIONAL - will create project if not set)
# LINEAR_PROJECT_ID=
# =============================================================================
# GITLAB INTEGRATION (OPTIONAL)
# =============================================================================
# Enable GitLab integration for issue tracking and merge requests.
# Supports both GitLab.com and self-hosted GitLab instances.
#
# Authentication Options (choose one):
#
# Option 1: glab CLI OAuth (Recommended)
# Install glab CLI: https://gitlab.com/gitlab-org/cli#installation
# Then run: glab auth login
# This opens your browser for OAuth authentication. Once complete,
# Auto Claude will automatically use your glab credentials (no env vars needed).
# For self-hosted: glab auth login --hostname gitlab.example.com
#
# Option 2: Personal Access Token
# Set GITLAB_TOKEN below. Token auth is used if set, otherwise falls back to glab CLI.
# GitLab Instance URL (OPTIONAL - defaults to gitlab.com)
# For self-hosted: GITLAB_INSTANCE_URL=https://gitlab.example.com
# GITLAB_INSTANCE_URL=https://gitlab.com
# GitLab Personal Access Token (OPTIONAL - only needed if not using glab CLI)
# Required scope: api (covers issues, merge requests, releases, project info)
# Optional scope: write_repository (only if creating new GitLab projects from local repos)
# Get from: https://gitlab.com/-/user_settings/personal_access_tokens
# GITLAB_TOKEN=glpat-xxxxxxxxxxxxxxxxxxxx
# GitLab Project (OPTIONAL - format: group/project or numeric ID)
# If not set, will auto-detect from git remote
# GITLAB_PROJECT=mygroup/myproject
# =============================================================================
# UI SETTINGS (OPTIONAL)
# =============================================================================
# Enable fancy terminal UI with icons, colors, and interactive menus.
# Set to "false" to use plain text output (useful for CI/CD or log files).
# Enable fancy UI (default: true)
# ENABLE_FANCY_UI=true
# =============================================================================
# ELECTRON MCP SERVER (OPTIONAL)
# =============================================================================
# Enable Electron MCP server for AI agents to interact with and validate
# Electron desktop applications. This allows QA agents to capture screenshots,
# inspect windows, and validate Electron apps during the review process.
#
# The electron-mcp-server connects via Chrome DevTools Protocol to an Electron
# app running with remote debugging enabled.
#
# Prerequisites:
# 1. Start your Electron app with remote debugging:
# ./YourElectronApp --remote-debugging-port=9222
#
# 2. For auto-claude-ui specifically (use the MCP-enabled scripts):
# cd auto-claude-ui
# pnpm run dev:mcp # Development mode with MCP debugging
# # OR for production build:
# pnpm run start:mcp # Production mode with MCP debugging
#
# Note: Only QA agents (qa_reviewer, qa_fixer) receive Electron MCP tools.
# Coder and Planner agents do NOT have access to these tools to minimize
# context token usage and keep agents focused on their roles.
#
# See: https://github.com/anthropics/anthropic-quickstarts/tree/main/mcp-electron-demo
# Enable Electron MCP integration (default: false)
# ELECTRON_MCP_ENABLED=true
# Chrome DevTools debugging port for Electron connection (default: 9222)
# ELECTRON_DEBUG_PORT=9222
# =============================================================================
# GRAPHITI MEMORY INTEGRATION (REQUIRED)
# =============================================================================
# Graphiti-based persistent memory layer for cross-session context
# retention. Uses LadybugDB as the embedded graph database.
#
# REQUIREMENTS:
# - Python 3.12 or higher
# - Install: pip install real_ladybug graphiti-core
#
# Supports multiple LLM and embedder providers:
# - OpenAI (default)
# - Anthropic (LLM only, use with Voyage for embeddings)
# - Azure OpenAI
# - Ollama (local, fully offline)
# - Google AI (Gemini)
# Graphiti is enabled by default. Set to false to disable memory features.
GRAPHITI_ENABLED=true
# =============================================================================
# GRAPHITI: Database Settings
# =============================================================================
# LadybugDB stores data in a local directory (no Docker required).
# Database name (default: auto_claude_memory)
# GRAPHITI_DATABASE=auto_claude_memory
# Database storage path (default: ~/.auto-claude/memories)
# GRAPHITI_DB_PATH=~/.auto-claude/memories
# =============================================================================
# GRAPHITI: Provider Selection
# =============================================================================
# Choose which providers to use for LLM and embeddings.
# Default is "openai" for both.
# LLM provider: openai | anthropic | azure_openai | ollama | google | openrouter
# GRAPHITI_LLM_PROVIDER=openai
# Embedder provider: openai | voyage | azure_openai | ollama | google | openrouter
# GRAPHITI_EMBEDDER_PROVIDER=openai
# =============================================================================
# GRAPHITI: OpenAI Provider (Default)
# =============================================================================
# Use OpenAI for both LLM and embeddings. This is the simplest setup.
# Required: OPENAI_API_KEY
# OpenAI API Key
# OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# OpenAI Model for LLM (default: gpt-4o-mini)
# OPENAI_MODEL=gpt-4o-mini
# OpenAI Model for embeddings (default: text-embedding-3-small)
# Available: text-embedding-3-small (1536 dim), text-embedding-3-large (3072 dim)
# OPENAI_EMBEDDING_MODEL=text-embedding-3-small
# =============================================================================
# GRAPHITI: Anthropic Provider (LLM only)
# =============================================================================
# Use Anthropic for LLM. Requires separate embedder (use Voyage or OpenAI).
# Example: GRAPHITI_LLM_PROVIDER=anthropic, GRAPHITI_EMBEDDER_PROVIDER=voyage
#
# Required: ANTHROPIC_API_KEY
# Anthropic API Key
# ANTHROPIC_API_KEY=sk-ant-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Anthropic Model (default: claude-sonnet-4-5-latest)
# GRAPHITI_ANTHROPIC_MODEL=claude-sonnet-4-5-latest
# =============================================================================
# GRAPHITI: Voyage AI Provider (Embeddings only)
# =============================================================================
# Use Voyage AI for embeddings. Commonly paired with Anthropic LLM.
# Get API key from: https://www.voyageai.com/
#
# Required: VOYAGE_API_KEY
# Voyage AI API Key
# VOYAGE_API_KEY=pa-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Voyage Embedding Model (default: voyage-3)
# Available: voyage-3 (1024 dim), voyage-3-lite (512 dim)
# VOYAGE_EMBEDDING_MODEL=voyage-3
# =============================================================================
# GRAPHITI: Google AI Provider
# =============================================================================
# Use Google AI (Gemini) for both LLM and embeddings.
# Get API key from: https://aistudio.google.com/apikey
#
# Required: GOOGLE_API_KEY
# Google AI API Key
# GOOGLE_API_KEY=AIzaSyxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Google LLM Model (default: gemini-2.0-flash)
# GOOGLE_LLM_MODEL=gemini-2.0-flash
# Google Embedding Model (default: text-embedding-004)
# GOOGLE_EMBEDDING_MODEL=text-embedding-004
# =============================================================================
# GRAPHITI: OpenRouter Provider (Multi-provider aggregator)
# =============================================================================
# Use OpenRouter to access multiple LLM providers through a single API.
# OpenRouter provides access to Anthropic, OpenAI, Google, and many other models.
# Get API key from: https://openrouter.ai/keys
#
# Required: OPENROUTER_API_KEY
# OpenRouter API Key
# OPENROUTER_API_KEY=sk-or-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# OpenRouter Base URL (default: https://openrouter.ai/api/v1)
# OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
# OpenRouter LLM Model (default: anthropic/claude-sonnet-4)
# Popular choices: anthropic/claude-sonnet-4, openai/gpt-4o, google/gemini-2.0-flash
# OPENROUTER_LLM_MODEL=anthropic/claude-sonnet-4
# OpenRouter Embedding Model (default: openai/text-embedding-3-small)
# OPENROUTER_EMBEDDING_MODEL=openai/text-embedding-3-small
# =============================================================================
# GRAPHITI: Azure OpenAI Provider
# =============================================================================
# Use Azure OpenAI for both LLM and embeddings.
# Requires Azure OpenAI deployment with appropriate models.
#
# Required: AZURE_OPENAI_API_KEY, AZURE_OPENAI_BASE_URL
# Azure OpenAI API Key
# AZURE_OPENAI_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Azure OpenAI Base URL (your Azure endpoint)
# AZURE_OPENAI_BASE_URL=https://your-resource.openai.azure.com/openai/deployments/your-deployment
# Azure OpenAI Deployment Names
# AZURE_OPENAI_LLM_DEPLOYMENT=gpt-4
# AZURE_OPENAI_EMBEDDING_DEPLOYMENT=text-embedding-3-small
# =============================================================================
# GRAPHITI: Ollama Provider (Local/Offline)
# =============================================================================
# Use Ollama for fully offline operation. No API keys required.
# Requires Ollama running locally with appropriate models pulled.
#
# Prerequisites:
# 1. Install Ollama: https://ollama.ai/
# 2. Pull models: ollama pull deepseek-r1:7b && ollama pull nomic-embed-text
# 3. Start Ollama server (usually auto-starts)
#
# Required: OLLAMA_LLM_MODEL, OLLAMA_EMBEDDING_MODEL, OLLAMA_EMBEDDING_DIM
# Ollama Server URL (default: http://localhost:11434)
# OLLAMA_BASE_URL=http://localhost:11434
# Ollama LLM Model
# Popular choices: deepseek-r1:7b, llama3.2:3b, mistral:7b, phi3:medium
# OLLAMA_LLM_MODEL=deepseek-r1:7b
# Ollama Embedding Model
# Popular choices: nomic-embed-text (768 dim), mxbai-embed-large (1024 dim)
# OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# Ollama Embedding Dimension (REQUIRED for Ollama embeddings)
# Must match your embedding model's output dimension
# Common values: nomic-embed-text=768, mxbai-embed-large=1024, all-minilm=384
# OLLAMA_EMBEDDING_DIM=768
# =============================================================================
# GRAPHITI: Example Configurations
# =============================================================================
#
# --- Example 1: OpenAI (simplest) ---
# GRAPHITI_ENABLED=true
# GRAPHITI_LLM_PROVIDER=openai
# GRAPHITI_EMBEDDER_PROVIDER=openai
# OPENAI_API_KEY=sk-xxxxxxxx
#
# --- Example 2: Anthropic + Voyage (high quality) ---
# GRAPHITI_ENABLED=true
# GRAPHITI_LLM_PROVIDER=anthropic
# GRAPHITI_EMBEDDER_PROVIDER=voyage
# ANTHROPIC_API_KEY=sk-ant-xxxxxxxx
# VOYAGE_API_KEY=pa-xxxxxxxx
#
# --- Example 3: Ollama (fully offline) ---
# GRAPHITI_ENABLED=true
# GRAPHITI_LLM_PROVIDER=ollama
# GRAPHITI_EMBEDDER_PROVIDER=ollama
# OLLAMA_LLM_MODEL=deepseek-r1:7b
# OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# OLLAMA_EMBEDDING_DIM=768
#
# --- Example 4: Azure OpenAI (enterprise) ---
# GRAPHITI_ENABLED=true
# GRAPHITI_LLM_PROVIDER=azure_openai
# GRAPHITI_EMBEDDER_PROVIDER=azure_openai
# AZURE_OPENAI_API_KEY=xxxxxxxx
# AZURE_OPENAI_BASE_URL=https://your-resource.openai.azure.com/...
# AZURE_OPENAI_LLM_DEPLOYMENT=gpt-4
# AZURE_OPENAI_EMBEDDING_DEPLOYMENT=text-embedding-3-small
#
# --- Example 5: Google AI (Gemini) ---
# GRAPHITI_ENABLED=true
# GRAPHITI_LLM_PROVIDER=google
# GRAPHITI_EMBEDDER_PROVIDER=google
# GOOGLE_API_KEY=AIzaSyxxxxxxxx
#
# --- Example 6: OpenRouter (multi-provider aggregator) ---
# GRAPHITI_ENABLED=true
# GRAPHITI_LLM_PROVIDER=openrouter
# GRAPHITI_EMBEDDER_PROVIDER=openrouter
# OPENROUTER_API_KEY=sk-or-xxxxxxxx
# OPENROUTER_LLM_MODEL=anthropic/claude-sonnet-4
# OPENROUTER_EMBEDDING_MODEL=openai/text-embedding-3-small
+67
View File
@@ -0,0 +1,67 @@
# Environment files
.env
.env.local
.env.*.local
# Virtual environment
.venv/
.venv*/
venv/
env/
# Python cache
__pycache__/
*.py[cod]
*$py.class
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
# Puppeteer / Browser automation
puppeteer_logs/
puppeteer-*.log
*.screenshot.png
screenshots/
.puppeteerrc.*
chrome-profile/
chromium-profile/
# IDE
.idea/
.vscode/
*.swp
*.swo
# OS
.DS_Store
Thumbs.db
# Git worktrees (used by parallel mode)
.worktrees/
# Claude Code settings (project-specific)
.claude_settings.json
.auto-build-security.json
# Tests (development only)
tests/
# Auto Claude data directory
.auto-claude/
/gitlab-integration-tests/
+421
View File
@@ -0,0 +1,421 @@
# Token Encryption Investigation
## Issue Summary
Auto-Claude users are experiencing API 401 errors ("Invalid bearer token") because the Python backend is passing encrypted tokens (with `enc:` prefix) directly to the Claude Agent SDK without decryption. Standalone Claude Code terminals work correctly because they decrypt these tokens before use.
**Key insight from user thehaffk:** "python cant unencrypt claude token and it launches session with CLAUDE_CODE_OAUTH_TOKEN=enc:djEwtxMGISt3tQ..."
## Token Storage Format
### Encrypted Token Format
Claude Code CLI stores OAuth tokens in an encrypted format with the prefix `enc:`:
```text
enc:djEwtxMGISt3tQ...
```
This format is used when tokens are stored in:
- **macOS**: Keychain (service: "Claude Code-credentials")
- **Linux**: Secret Service API (DBus, via secretstorage library)
- **Windows**: Credential Manager / .credentials.json files
### Decrypted Token Format
Valid Claude OAuth tokens have the format:
```text
sk-ant-oat01-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
```
## Current Token Flow (BROKEN)
1. **Token Storage**: Claude Code CLI stores encrypted token with `enc:` prefix in system keychain
2. **Token Retrieval**: `apps/backend/core/auth.py::get_auth_token()` retrieves token from:
- Environment variable `CLAUDE_CODE_OAUTH_TOKEN`
- OR system keychain via `get_token_from_keychain()`
3. **❌ NO DECRYPTION**: Token is returned as-is with `enc:` prefix intact
4. **SDK Initialization**: Encrypted token passed to Claude Agent SDK
5. **API Call Fails**: SDK sends encrypted token to API → 401 error
### Proof of Broken Flow
Test in `apps/backend`:
```python
import os
os.environ['CLAUDE_CODE_OAUTH_TOKEN'] = 'enc:test123'
from core.auth import get_auth_token
token = get_auth_token()
print(f"Token: {token}") # Output: "enc:test123"
print(f"Encrypted: {token.startswith('enc:')}") # Output: True
```
## How Standalone Claude Code CLI Handles Tokens
### Current Understanding
1. **Token Detection**: CLI checks if token starts with `enc:` prefix
2. **Decryption**: If encrypted, CLI decrypts using platform-specific keyring access
3. **Authentication**: Decrypted `sk-ant-oat01-` token is used for API calls
### Missing Documentation
Web search for "Claude Code CLI encrypted token enc: prefix decryption" found:
- Token storage formats (JSON with accessToken, refreshToken, expiresAt)
- Security issues (tokens exposed in debug logs before v2.1.0)
- Keychain access patterns for macOS/Linux/Windows
**❌ NOT FOUND**: Specific documentation on how Claude Code CLI decrypts `enc:` tokens
Sources:
- [Claude Code CLI over SSH on macOS: Fixing Keychain Access](https://phoenixtrap.com/2025/10/26/claude-code-cli-over-ssh-on-macos-fixing-keychain-access/)
- [Identity and Access Management - Claude Code Docs](https://code.claude.com/docs/en/iam)
- [Claude Code sessions should be encrypted | yoav.blog](https://yoav.blog/2026/01/09/claude-code-sessions-should-be-encrypted/)
## Decryption Approach Options
### Option 1: Claude Agent SDK Built-in Decryption
**Status**: NEEDS VERIFICATION
The Claude Agent SDK (`claude-agent-sdk>=0.1.19`) may handle decryption internally if:
- Token is passed to SDK still encrypted
- SDK detects `enc:` prefix
- SDK has access to system keyring for decryption
**Action Required**: Check if SDK has decryption capabilities by examining:
- SDK source code or documentation
- Whether SDK expects encrypted vs decrypted tokens
- If SDK requires specific environment variables for decryption
### Option 2: Python Backend Decryption (Recommended)
**Approach**: Implement decryption in `apps/backend/core/auth.py` before passing to SDK
**Implementation Pattern**:
```python
def get_auth_token() -> str | None:
"""Get authentication token (decrypted if necessary)."""
token = _retrieve_token_from_sources() # From env or keychain
if token and token.startswith("enc:"):
# Decrypt the token
token = decrypt_token(token)
return token
def decrypt_token(encrypted_token: str) -> str:
"""
Decrypt Claude Code encrypted token.
Args:
encrypted_token: Token with 'enc:' prefix
Returns:
Decrypted token in format 'sk-ant-oat01-...'
"""
# Remove 'enc:' prefix
encrypted_data = encrypted_token[4:]
# TODO: Implement decryption logic
# Questions to answer:
# 1. What encryption algorithm does Claude Code use?
# 2. Where is the decryption key stored?
# 3. Is the decryption key platform-specific (per-user)?
# 4. Can we reuse Claude Code's decryption mechanism?
raise NotImplementedError("Token decryption not yet implemented")
```
### Option 3: Call Claude Code CLI for Decryption
**Approach**: Use the Claude Code CLI binary to decrypt tokens
```python
def decrypt_token(encrypted_token: str) -> str:
"""Decrypt token by invoking Claude Code CLI."""
# Find claude binary
claude_path = shutil.which("claude") or "~/.local/bin/claude"
# Use CLI command to get decrypted token
# (if such a command exists - needs research)
result = subprocess.run(
[claude_path, "auth", "decrypt", encrypted_token],
capture_output=True,
text=True
)
return result.stdout.strip()
```
**Issues**:
- Requires Claude Code CLI to be installed
- No documented CLI command for token decryption
- Adds external dependency
## Required Investigation Steps
### 1. Verify SDK Decryption Capabilities
**Task**: Check if `claude-agent-sdk` handles `enc:` tokens automatically
**Method**:
```bash
# In environment with SDK installed
python3 << 'EOF'
import os
os.environ['CLAUDE_CODE_OAUTH_TOKEN'] = 'enc:...' # Real encrypted token
from claude_agent_sdk import Client
# Try creating client - does it decrypt internally?
client = Client()
# Check if authentication works
EOF
```
### 2. Reverse Engineer Claude Code CLI Decryption
**Task**: Understand how Claude CLI decrypts tokens
**Method**:
- Examine Claude CLI binary (if possible)
- Trace system calls when CLI runs (strace on Linux, dtruss on macOS)
- Check if CLI accesses specific keychain entries for decryption keys
- Look for encryption/decryption libraries used by CLI
### 3. Find Decryption Key Storage
**Task**: Locate where decryption keys are stored
**Hypothesis**: Decryption key stored in:
- macOS: Keychain (separate entry from encrypted token)
- Linux: Secret Service API
- Windows: Credential Manager
**Verification**:
```bash
# macOS: List all keychain entries
security find-generic-password -a "$(whoami)" | grep -i claude
# Linux: Use secretstorage to list all items
python3 -c "import secretstorage; ..."
```
## Recommended Decryption Approach for Python Backend
Based on investigation so far, the recommended approach is:
1. **Detect encrypted tokens**: Check for `enc:` prefix in `get_auth_token()`
2. **Decrypt before use**: Implement `decrypt_token()` function
3. **Platform-specific decryption**: Use appropriate keyring library:
- macOS: Use `subprocess` with `/usr/bin/security` to access decryption key
- Linux: Use `secretstorage` library to access Secret Service API
- Windows: Access Credential Manager or credentials.json
4. **Backward compatibility**: Support both encrypted and plaintext tokens
5. **Error handling**: Provide clear error messages if decryption fails
## Complete Token Flow Trace (Frontend → Backend)
### 1. Token Retrieval (Frontend)
**File**: `apps/frontend/src/main/services/profile-service.ts`
The frontend retrieves the OAuth token from the system keychain but **does not decrypt it**. When no API profile is active (OAuth mode), the frontend returns an empty environment object, which means it relies on:
- The token already being in the environment as `CLAUDE_CODE_OAUTH_TOKEN`
- OR the Python backend retrieving it from the keychain
**Key Code**:
```typescript
// Line 223: Returns empty object in OAuth mode, allowing
// CLAUDE_CODE_OAUTH_TOKEN to be used from system keychain
```
### 2. Environment Variable Passing (Frontend → PTY)
**File**: `apps/frontend/src/main/terminal/pty-manager.ts`
The PTY manager spawns the terminal shell with environment variables, including `CLAUDE_CODE_OAUTH_TOKEN`:
**Key Code** (Lines 149-152):
```typescript
// Remove ANTHROPIC_API_KEY to ensure Claude Code uses OAuth tokens
// (CLAUDE_CODE_OAUTH_TOKEN from profileEnv) instead of API keys
const { DEBUG: _DEBUG, ANTHROPIC_API_KEY: _ANTHROPIC_API_KEY, ...cleanEnv } = process.env;
```
**Important**: The frontend passes through whatever token value exists in the environment - it does NOT check for `enc:` prefix or decrypt it.
### 3. Token Retrieval (Backend)
**File**: `apps/backend/core/auth.py`
#### 3.1. get_auth_token()
This function retrieves the token from multiple sources:
```python
def get_auth_token() -> str | None:
# First check environment variables
for var in AUTH_TOKEN_ENV_VARS: # CLAUDE_CODE_OAUTH_TOKEN, ANTHROPIC_AUTH_TOKEN
token = os.environ.get(var)
if token:
return token # ❌ Returns immediately without checking for enc: prefix
# Fallback to system credential store
return get_token_from_keychain() # ❌ Also returns without decryption
```
**Issue**: Returns token as-is with `enc:` prefix intact.
#### 3.2. require_auth_token()
This function calls `get_auth_token()` and raises an error if no token is found:
```python
def require_auth_token() -> str:
token = get_auth_token() # ❌ Gets encrypted token
if not token:
raise ValueError("No OAuth token found...")
return token # ❌ Returns encrypted token
```
**Issue**: No decryption step between retrieval and return.
#### 3.3. ensure_claude_code_oauth_token()
This function ensures the environment variable is set:
```python
def ensure_claude_code_oauth_token() -> None:
if os.environ.get("CLAUDE_CODE_OAUTH_TOKEN"):
return
token = get_auth_token() # ❌ Gets encrypted token
if token:
os.environ["CLAUDE_CODE_OAUTH_TOKEN"] = token # ❌ Sets encrypted token
```
**Issue**: Propagates encrypted token to environment variable.
### 4. Token Usage in SDK Client Creation (Backend)
#### 4.1. Full Client Creation
**File**: `apps/backend/core/client.py` (see `create_client()` function)
```python
def create_client(...):
oauth_token = require_auth_token() # ❌ Gets encrypted token
# Ensure SDK can access it via its expected env var
os.environ["CLAUDE_CODE_OAUTH_TOKEN"] = oauth_token # ❌ Sets encrypted token
```
**Issue**: Encrypted token is passed to the Claude Agent SDK, which expects a decrypted `sk-ant-oat01-` token.
#### 4.2. Simple Client Creation
**File**: `apps/backend/core/simple_client.py` (see `create_simple_client()` function)
```python
def create_simple_client(...):
# Get authentication
oauth_token = require_auth_token() # ❌ Gets encrypted token
import os
os.environ["CLAUDE_CODE_OAUTH_TOKEN"] = oauth_token # ❌ Sets encrypted token
```
**Issue**: Same problem - encrypted token passed to SDK.
#### 4.3. Other Usages
**Files**:
- `apps/backend/core/workspace.py` (Line 1966) - AI merge operations
- `apps/backend/runners/insights_runner.py` - Insights analysis
- `apps/backend/runners/github/batch_issues.py` - GitHub batch operations
- `apps/backend/integrations/linear/updater.py` - Linear integration
- `apps/backend/commit_message.py` - Commit message generation
- `apps/backend/analysis/insight_extractor.py` - Code insights
- `apps/backend/merge/ai_resolver/claude_client.py` - Merge resolution
**All follow the same pattern**: Call `ensure_claude_code_oauth_token()` → encrypted token in environment → SDK receives encrypted token → API 401 error.
### 5. Where Decryption Should Be Inserted
Based on the flow analysis, decryption should be added at **the earliest point of token retrieval** to avoid duplicating decryption logic:
**RECOMMENDED INSERTION POINT**: `apps/backend/core/auth.py::get_auth_token()`
```python
def get_auth_token() -> str | None:
# First check environment variables
for var in AUTH_TOKEN_ENV_VARS:
token = os.environ.get(var)
if token:
# ✅ INSERT DECRYPTION HERE
if token.startswith("enc:"):
token = decrypt_token(token)
return token
# Fallback to system credential store
token = get_token_from_keychain()
# ✅ ALSO DECRYPT KEYCHAIN TOKENS
if token and token.startswith("enc:"):
token = decrypt_token(token)
return token
```
**Benefits of this approach**:
1. Single location for decryption logic
2. All downstream functions automatically get decrypted tokens
3. Backward compatible (plaintext tokens pass through unchanged)
4. Consistent behavior across all token sources (env vars and keychain)
**Alternative insertion points** (NOT recommended):
- `require_auth_token()` - Would need similar logic in `get_auth_token()` for non-required usage
- `create_client()` - Would need duplication in `create_simple_client()` and all other clients
- `ensure_claude_code_oauth_token()` - Would miss direct `get_auth_token()` calls
## Next Steps
1. ✅ Document current token flow and identify issue (THIS FILE - COMPLETED)
2. ✅ Trace token flow from frontend to backend (THIS FILE - COMPLETED)
3. ✅ Identify where decryption should be inserted (THIS FILE - COMPLETED)
4. ⏳ Verify if Claude Agent SDK handles decryption internally
5. ⏳ Reverse engineer or document Claude Code CLI decryption mechanism
6. ⏳ Implement `decrypt_token()` function in `apps/backend/core/auth.py`
7. ⏳ Add encryption detection and auto-decryption to `get_auth_token()`
8. ⏳ Test with real encrypted tokens on macOS and Linux
9. ⏳ Add comprehensive error handling for decryption failures
## Open Questions
1. **What encryption algorithm does Claude Code use for `enc:` tokens?**
- Possible: AES-256, ChaCha20, or similar
- Key derivation method?
2. **Where is the decryption key stored?**
- Same keychain entry as encrypted token?
- Separate keychain entry?
- Derived from system/user credentials?
3. **Does Claude Agent SDK expect encrypted or decrypted tokens?**
- If it expects decrypted: we must decrypt before passing
- If it handles encryption: we may be missing SDK configuration
4. **Is there a Claude Code CLI command to decrypt tokens?**
- `claude auth decrypt <token>`?
- `claude auth get-token`?
- No documented command found in research
5. **Can we reuse Claude Code's decryption mechanism?**
- Import decryption functions from CLI?
- Call CLI as subprocess?
- Implement decryption ourselves?
## References
- Issue: [GitHub #1223: API Error 401](https://github.com/AndyMik90/Auto-Claude/issues/1223)
- Current auth implementation: `apps/backend/core/auth.py`
- SDK client initialization: `apps/backend/core/client.py`
- Requirements: `apps/backend/requirements.txt` (includes `secretstorage>=3.3.3` for Linux)
+122
View File
@@ -0,0 +1,122 @@
# Auto Claude Backend
Autonomous coding framework powered by Claude AI. Builds software features through coordinated multi-agent sessions.
## Getting Started
### 1. Install
```bash
cd apps/backend
python -m pip install -r requirements.txt
```
### 2. Configure
```bash
cp .env.example .env
```
Authenticate with Claude Code (token auto-saved to Keychain):
```bash
claude
# Type: /login
# Press Enter to open browser
```
Token is auto-detected from macOS Keychain / Windows Credential Manager.
### 3. Run
```bash
# List available specs
python run.py --list
# Run a spec
python run.py --spec 001
```
## Requirements
- Python 3.10+
- Claude API token
## Commands
| Command | Description |
|---------|-------------|
| `--list` | List all specs |
| `--spec 001` | Run spec 001 |
| `--spec 001 --isolated` | Run in isolated workspace |
| `--spec 001 --direct` | Run directly in repo |
| `--spec 001 --merge` | Merge completed build |
| `--spec 001 --review` | Review build changes |
| `--spec 001 --discard` | Discard build |
| `--spec 001 --qa` | Run QA validation |
| `--list-worktrees` | List all worktrees |
| `--help` | Show all options |
## Configuration
Optional `.env` settings:
| Variable | Description |
|----------|-------------|
| `AUTO_BUILD_MODEL` | Override Claude model |
| `DEBUG=true` | Enable debug logging |
| `LINEAR_API_KEY` | Enable Linear integration |
| `GRAPHITI_ENABLED=true` | Enable memory system |
## Troubleshooting
**"tree-sitter not available"** - Safe to ignore, uses regex fallback.
**Missing module errors** - Run `python -m pip install -r requirements.txt`
**Debug mode** - Set `DEBUG=true DEBUG_LEVEL=2` before running.
---
## For Developers
### Project Structure
```
backend/
├── agents/ # AI agent execution
├── analysis/ # Code analysis
├── cli/ # Command-line interface
├── core/ # Core utilities
├── integrations/ # External services (Linear, Graphiti)
├── merge/ # Git merge handling
├── project/ # Project detection
├── prompts/ # Prompt templates
├── qa/ # QA validation
├── spec/ # Spec management
└── ui/ # Terminal UI
```
### Design Principles
- **SOLID** - Single responsibility, clean interfaces
- **DRY** - Shared utilities in `core/`
- **KISS** - Simple flat imports via facade modules
### Import Convention
```python
# Use facade modules for clean imports
from debug import debug, debug_error
from progress import count_subtasks
from workspace import setup_workspace
```
### Adding Features
1. Create module in appropriate folder
2. Export API in `__init__.py`
3. Add facade module at root if commonly imported
## License
AGPL-3.0
+23
View File
@@ -0,0 +1,23 @@
"""
Auto Claude Backend - Autonomous Coding Framework
==================================================
Multi-agent autonomous coding framework that builds software through
coordinated AI agent sessions.
This package provides:
- Autonomous agent execution for building features from specs
- Workspace isolation via git worktrees
- QA validation loops
- Memory management (Graphiti + file-based)
- Linear integration for project management
Quick Start:
python run.py --spec 001 # Run a spec
python run.py --list # List all specs
See README.md for full documentation.
"""
__version__ = "2.7.5"
__author__ = "Auto Claude Team"
+289
View File
@@ -0,0 +1,289 @@
"""
GitLab Test Fixtures
====================
Mock data and fixtures for GitLab integration tests.
"""
# Sample GitLab MR data
SAMPLE_MR_DATA = {
"iid": 123,
"id": 12345,
"title": "Add user authentication feature",
"description": "Implement OAuth2 login with Google and GitHub providers",
"author": {
"id": 1,
"username": "john_doe",
"name": "John Doe",
"email": "john@example.com",
},
"source_branch": "feature/oauth-auth",
"target_branch": "main",
"state": "opened",
"draft": False,
"merge_status": "can_be_merged",
"web_url": "https://gitlab.com/group/project/-/merge_requests/123",
"created_at": "2025-01-14T10:00:00.000Z",
"updated_at": "2025-01-14T12:00:00.000Z",
"labels": ["feature", "authentication"],
"assignees": [],
}
SAMPLE_MR_CHANGES = {
"id": 12345,
"iid": 123,
"project_id": 1,
"title": "Add user authentication feature",
"description": "Implement OAuth2 login",
"state": "opened",
"created_at": "2025-01-14T10:00:00.000Z",
"updated_at": "2025-01-14T12:00:00.000Z",
"merge_status": "can_be_merged",
"additions": 150,
"deletions": 20,
"changed_files_count": 5,
"changes": [
{
"old_path": "src/auth/__init__.py",
"new_path": "src/auth/__init__.py",
"diff": "@@ -0,0 +1,5 @@\n+from .oauth import OAuthHandler\n+from .providers import GoogleProvider, GitHubProvider",
"new_file": False,
"renamed_file": False,
"deleted_file": False,
},
{
"old_path": "src/auth/oauth.py",
"new_path": "src/auth/oauth.py",
"diff": "@@ -0,0 +1,50 @@\n+class OAuthHandler:\n+ def handle_callback(self, request):\n+ pass",
"new_file": True,
"renamed_file": False,
"deleted_file": False,
},
],
}
SAMPLE_MR_COMMITS = [
{
"id": "abc123def456",
"short_id": "abc123de",
"title": "Add OAuth handler",
"message": "Add OAuth handler",
"author_name": "John Doe",
"author_email": "john@example.com",
"authored_date": "2025-01-14T10:00:00.000Z",
"created_at": "2025-01-14T10:00:00.000Z",
},
{
"id": "def456ghi789",
"short_id": "def456gh",
"title": "Add Google provider",
"message": "Add Google provider",
"author_name": "John Doe",
"author_email": "john@example.com",
"authored_date": "2025-01-14T11:00:00.000Z",
"created_at": "2025-01-14T11:00:00.000Z",
},
]
# Sample GitLab issue data
SAMPLE_ISSUE_DATA = {
"iid": 42,
"id": 42,
"title": "Bug: Login button not working",
"description": "Clicking the login button does nothing",
"author": {
"id": 2,
"username": "jane_smith",
"name": "Jane Smith",
"email": "jane@example.com",
},
"state": "opened",
"labels": ["bug", "urgent"],
"assignees": [],
"milestone": None,
"web_url": "https://gitlab.com/group/project/-/issues/42",
"created_at": "2025-01-14T09:00:00.000Z",
"updated_at": "2025-01-14T09:30:00.000Z",
}
# Sample GitLab pipeline data
SAMPLE_PIPELINE_DATA = {
"id": 1001,
"iid": 1,
"project_id": 1,
"ref": "feature/oauth-auth",
"sha": "abc123def456",
"status": "success",
"source": "merge_request_event",
"created_at": "2025-01-14T10:30:00.000Z",
"updated_at": "2025-01-14T10:35:00.000Z",
"finished_at": "2025-01-14T10:35:00.000Z",
"duration": 300,
"web_url": "https://gitlab.com/group/project/-/pipelines/1001",
}
SAMPLE_PIPELINE_JOBS = [
{
"id": 5001,
"name": "test",
"stage": "test",
"status": "success",
"started_at": "2025-01-14T10:31:00.000Z",
"finished_at": "2025-01-14T10:34:00.000Z",
"duration": 180,
"allow_failure": False,
},
{
"id": 5002,
"name": "lint",
"stage": "test",
"status": "success",
"started_at": "2025-01-14T10:31:00.000Z",
"finished_at": "2025-01-14T10:32:00.000Z",
"duration": 60,
"allow_failure": False,
},
]
# Sample GitLab discussion/note data
SAMPLE_MR_DISCUSSIONS = [
{
"id": "d1",
"notes": [
{
"id": 1001,
"type": "DiscussionNote",
"author": {"username": "coderabbit[bot]"},
"body": "Consider adding error handling for OAuth failures",
"created_at": "2025-01-14T11:00:00.000Z",
"system": False,
"resolvable": True,
}
],
}
]
SAMPLE_MR_NOTES = [
{
"id": 2001,
"type": "DiscussionNote",
"author": {"username": "reviewer_user"},
"body": "LGTM, just one comment",
"created_at": "2025-01-14T12:00:00.000Z",
"system": False,
}
]
# Mock GitLab config
MOCK_GITLAB_CONFIG = {
"token": "glpat-test-token-12345",
"project": "group/project",
"instance_url": "https://gitlab.example.com",
}
def create_mock_client(project_dir=None):
"""Create a mock GitLab client for testing.
Args:
project_dir: Optional project directory path (uses temp dir if None)
Returns:
Configured GitLabClient instance
"""
import tempfile
from pathlib import Path
from runners.gitlab.glab_client import GitLabClient, GitLabConfig
if project_dir is None:
project_dir = Path(tempfile.mkdtemp())
else:
project_dir = Path(project_dir)
config = GitLabConfig(**MOCK_GITLAB_CONFIG)
return GitLabClient(project_dir=project_dir, config=config)
def mock_mr_data(**overrides):
"""Create mock MR data with optional overrides."""
import copy
data = copy.deepcopy(SAMPLE_MR_DATA)
# Handle special case for author override
if "author" in overrides:
author_value = overrides.pop("author")
if isinstance(author_value, str):
# If author is a string, update the username field
data["author"]["username"] = author_value
else:
# Otherwise, merge the author dict
data["author"].update(author_value)
data.update(overrides)
return data
def mock_mr_changes(**overrides):
"""Create mock MR changes with optional overrides."""
data = SAMPLE_MR_CHANGES.copy()
data.update(overrides)
return data
def mock_issue_data(**overrides):
"""Create mock issue data with optional overrides."""
data = SAMPLE_ISSUE_DATA.copy()
data.update(overrides)
return data
def mock_pipeline_data(**overrides):
"""Create mock pipeline data with optional overrides."""
data = SAMPLE_PIPELINE_DATA.copy()
data.update(overrides)
return data
def mock_pipeline_jobs(**overrides):
"""Create mock pipeline jobs with optional overrides."""
data = SAMPLE_PIPELINE_JOBS.copy()
if overrides:
data[0].update(overrides)
return data
def mock_mr_commits(**overrides):
"""Create mock MR commits with optional overrides."""
import copy
data = copy.deepcopy(SAMPLE_MR_COMMITS)
if overrides and data:
data[0].update(overrides)
return data
def get_mock_diff() -> str:
"""Get a mock diff string for testing."""
return """diff --git a/src/auth/oauth.py b/src/auth/oauth.py
new file mode 100644
index 0000000..abc1234
--- /dev/null
+++ b/src/auth/oauth.py
@@ -0,0 +1,50 @@
+class OAuthHandler:
+ def handle_callback(self, request):
+ pass
diff --git a/src/auth/providers.py b/src/auth/providers.py
new file mode 100644
index 0000000..def5678
--- /dev/null
+++ b/src/auth/providers.py
@@ -0,0 +1,30 @@
+class GoogleProvider:
+ pass
+
+class GitHubProvider:
+ pass
"""
@@ -0,0 +1,391 @@
"""
Tests for GitLab Auto-fix Processor
======================================
Tests for auto-fix workflow, permission verification, and state management.
"""
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
try:
from runners.gitlab.autofix_processor import AutoFixProcessor
from runners.gitlab.models import AutoFixState, AutoFixStatus, GitLabRunnerConfig
from runners.gitlab.permissions import GitLabPermissionChecker
except ImportError:
from models import AutoFixState, AutoFixStatus, GitLabRunnerConfig
from runners.gitlab.autofix_processor import AutoFixProcessor
from runners.gitlab.permissions import GitLabPermissionChecker
@pytest.fixture
def mock_config():
"""Create a mock GitLab config."""
config = MagicMock(spec=GitLabRunnerConfig)
config.project = "namespace/test-project"
config.instance_url = "https://gitlab.example.com"
config.auto_fix_enabled = True
config.auto_fix_labels = ["auto-fix", "autofix"]
config.token = "test-token"
return config
@pytest.fixture
def mock_permission_checker():
"""Create a mock permission checker."""
checker = MagicMock(spec=GitLabPermissionChecker)
checker.verify_automation_trigger = AsyncMock()
return checker
@pytest.fixture
def tmp_gitlab_dir(tmp_path):
"""Create a temporary GitLab directory."""
gitlab_dir = tmp_path / ".auto-claude" / "gitlab"
gitlab_dir.mkdir(parents=True, exist_ok=True)
return gitlab_dir
@pytest.fixture
def processor(mock_config, mock_permission_checker, tmp_path, tmp_gitlab_dir):
"""Create an AutoFixProcessor instance."""
return AutoFixProcessor(
gitlab_dir=tmp_gitlab_dir,
config=mock_config,
permission_checker=mock_permission_checker,
progress_callback=None,
)
class TestProcessIssue:
"""Tests for issue processing."""
@pytest.mark.asyncio
async def test_process_issue_success(
self, processor, mock_permission_checker, tmp_gitlab_dir
):
"""Test successful issue processing."""
issue = {
"iid": 123,
"title": "Fix this bug",
"description": "Please fix",
"labels": ["auto-fix"],
}
mock_permission_checker.verify_automation_trigger.return_value = MagicMock(
allowed=True,
username="developer",
role="MAINTAINER",
)
result = await processor.process_issue(
issue_iid=123,
issue=issue,
trigger_label="auto-fix",
)
assert result.issue_iid == 123
assert result.status == AutoFixStatus.CREATING_SPEC
@pytest.mark.asyncio
async def test_process_issue_permission_denied(
self, processor, mock_permission_checker, tmp_gitlab_dir
):
"""Test issue processing with permission denied."""
issue = {
"iid": 456,
"title": "Unauthorized fix",
"labels": ["auto-fix"],
}
mock_permission_checker.verify_automation_trigger.return_value = MagicMock(
allowed=False,
username="outsider",
role="NONE",
reason="Not a maintainer",
)
with pytest.raises(PermissionError):
await processor.process_issue(
issue_iid=456,
issue=issue,
trigger_label="auto-fix",
)
@pytest.mark.asyncio
async def test_process_issue_in_progress(
self, processor, mock_permission_checker, tmp_gitlab_dir
):
"""Test that in-progress issues are not reprocessed."""
issue = {
"iid": 789,
"title": "Already processing",
"labels": ["auto-fix"],
}
# Create existing state in progress
existing_state = AutoFixState(
issue_iid=789,
issue_url="https://gitlab.example.com/issue/789",
project="namespace/test-project",
status=AutoFixStatus.ANALYZING,
)
await existing_state.save(tmp_gitlab_dir)
result = await processor.process_issue(
issue_iid=789,
issue=issue,
trigger_label="auto-fix",
)
# Should return the existing state
assert result.status == AutoFixStatus.ANALYZING
class TestCheckLabeledIssues:
"""Tests for checking labeled issues."""
@pytest.mark.asyncio
async def test_check_labeled_issues_finds_new(
self, processor, mock_permission_checker
):
"""Test finding new labeled issues."""
all_issues = [
{
"iid": 1,
"title": "Has auto-fix label",
"labels": ["auto-fix"],
},
{
"iid": 2,
"title": "Has autofix label",
"labels": ["autofix"],
},
{
"iid": 3,
"title": "No label",
"labels": [],
},
]
# Permission checks pass
mock_permission_checker.verify_automation_trigger.return_value = MagicMock(
allowed=True
)
result = await processor.check_labeled_issues(
all_issues, verify_permissions=True
)
assert len(result) == 2
assert result[0]["issue_iid"] == 1
assert result[1]["issue_iid"] == 2
@pytest.mark.asyncio
async def test_check_labeled_issues_filters_in_queue(
self, processor, mock_permission_checker, tmp_gitlab_dir
):
"""Test that issues already in queue are filtered out."""
# Create existing state for issue 1
existing_state = AutoFixState(
issue_iid=1,
issue_url="https://gitlab.example.com/issue/1",
project="namespace/test-project",
status=AutoFixStatus.ANALYZING,
)
await existing_state.save(tmp_gitlab_dir)
all_issues = [
{
"iid": 1,
"title": "Already in queue",
"labels": ["auto-fix"],
},
{
"iid": 2,
"title": "New issue",
"labels": ["auto-fix"],
},
]
mock_permission_checker.verify_automation_trigger.return_value = MagicMock(
allowed=True
)
result = await processor.check_labeled_issues(
all_issues, verify_permissions=True
)
# Should only return issue 2 (issue 1 is already in queue)
assert len(result) == 1
assert result[0]["issue_iid"] == 2
@pytest.mark.asyncio
async def test_check_labeled_issues_permission_filtering(
self, processor, mock_permission_checker
):
"""Test that unauthorized issues are filtered out."""
all_issues = [
{
"iid": 1,
"title": "Authorized issue",
"labels": ["auto-fix"],
},
{
"iid": 2,
"title": "Unauthorized issue",
"labels": ["auto-fix"],
},
]
def make_permission_result(issue_iid, trigger_label):
if issue_iid == 1:
return MagicMock(allowed=True)
else:
return MagicMock(allowed=False, reason="Not authorized")
mock_permission_checker.verify_automation_trigger.side_effect = (
make_permission_result
)
result = await processor.check_labeled_issues(
all_issues, verify_permissions=True
)
# Should only return issue 1
assert len(result) == 1
assert result[0]["issue_iid"] == 1
class TestGetQueue:
"""Tests for getting auto-fix queue."""
@pytest.mark.asyncio
async def test_get_queue_empty(self, processor, tmp_gitlab_dir):
"""Test getting queue when empty."""
queue = await processor.get_queue()
assert queue == []
@pytest.mark.asyncio
async def test_get_queue_with_items(self, processor, tmp_gitlab_dir):
"""Test getting queue with items."""
# Create some states
for i in [1, 2, 3]:
state = AutoFixState(
issue_iid=i,
issue_url=f"https://gitlab.example.com/issue/{i}",
project="namespace/test-project",
status=AutoFixStatus.ANALYZING,
)
await state.save(tmp_gitlab_dir)
queue = await processor.get_queue()
assert len(queue) == 3
class TestAutoFixState:
"""Tests for AutoFixState model."""
def test_state_creation(self, tmp_gitlab_dir):
"""Test creating and saving state."""
state = AutoFixState(
issue_iid=123,
issue_url="https://gitlab.example.com/issue/123",
project="namespace/test-project",
status=AutoFixStatus.PENDING,
)
assert state.issue_iid == 123
assert state.status == AutoFixStatus.PENDING
def test_state_save_and_load(self, tmp_gitlab_dir):
"""Test saving and loading state."""
state = AutoFixState(
issue_iid=456,
issue_url="https://gitlab.example.com/issue/456",
project="namespace/test-project",
status=AutoFixStatus.BUILDING,
)
# Save state
import asyncio
asyncio.run(state.save(tmp_gitlab_dir))
# Load state
loaded = AutoFixState.load(tmp_gitlab_dir, 456)
assert loaded is not None
assert loaded.issue_iid == 456
assert loaded.status == AutoFixStatus.BUILDING
def test_state_transition_validation(self, tmp_gitlab_dir):
"""Test that invalid state transitions are rejected."""
state = AutoFixState(
issue_iid=789,
issue_url="https://gitlab.example.com/issue/789",
project="namespace/test-project",
status=AutoFixStatus.PENDING,
)
# Valid transition
state.update_status(AutoFixStatus.ANALYZING) # Should work
# Invalid transition
with pytest.raises(ValueError):
state.update_status(AutoFixStatus.COMPLETED) # Can't skip to completed
class TestProgressReporting:
"""Tests for progress callback handling."""
@pytest.mark.asyncio
async def test_progress_reported_during_processing(
self, mock_config, tmp_path, tmp_gitlab_dir
):
"""Test that progress callback is stored on the processor."""
progress_calls = []
def progress_callback(progress):
progress_calls.append(progress)
processor = AutoFixProcessor(
gitlab_dir=tmp_gitlab_dir,
config=mock_config,
permission_checker=MagicMock(),
progress_callback=progress_callback,
)
# Verify the callback is stored
assert processor.progress_callback is not None
assert processor.progress_callback == progress_callback
# Test that calling the callback works
processor.progress_callback({"status": "test"})
assert len(progress_calls) == 1
assert progress_calls[0] == {"status": "test"}
class TestURLConstruction:
"""Tests for URL construction."""
@pytest.mark.asyncio
async def test_issue_url_construction(self, processor, mock_config):
"""Test that issue URLs are constructed correctly."""
issue = {"iid": 123}
state = await processor.process_issue(
issue_iid=123,
issue=issue,
trigger_label=None,
)
assert (
state.issue_url
== "https://gitlab.example.com/namespace/test-project/-/issues/123"
)
@@ -0,0 +1,451 @@
"""
Tests for GitLab Batch Issues
================================
Tests for issue batching, similarity detection, and batch processing.
"""
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
try:
from runners.gitlab.batch_issues import (
ClaudeGitlabBatchAnalyzer,
GitlabBatchStatus,
GitlabIssueBatch,
GitlabIssueBatcher,
GitlabIssueBatchItem,
format_batch_summary,
)
from runners.gitlab.glab_client import GitLabConfig
except ImportError:
from glab_client import GitLabConfig
from runners.gitlab.batch_issues import (
ClaudeGitlabBatchAnalyzer,
GitlabBatchStatus,
GitlabIssueBatch,
GitlabIssueBatcher,
GitlabIssueBatchItem,
format_batch_summary,
)
@pytest.fixture
def mock_config():
"""Create a mock GitLab config."""
config = MagicMock(spec=GitLabConfig)
config.project = "namespace/test-project"
config.instance_url = "https://gitlab.example.com"
return config
@pytest.fixture
def sample_issues():
"""Sample issues for batching."""
return [
{
"iid": 1,
"title": "Login bug",
"description": "Cannot login with special characters",
"labels": ["bug", "auth"],
},
{
"iid": 2,
"title": "Signup bug",
"description": "Cannot signup with special characters",
"labels": ["bug", "auth"],
},
{
"iid": 3,
"title": "UI bug",
"description": "Button alignment issue",
"labels": ["bug", "ui"],
},
]
class TestBatchAnalyzer:
"""Tests for Claude-based batch analyzer."""
@pytest.mark.asyncio
async def test_analyze_single_issue(self, mock_config, tmp_path):
"""Test analyzing a single issue."""
analyzer = ClaudeGitlabBatchAnalyzer(project_dir=tmp_path)
issues = [{"iid": 1, "title": "Single issue"}]
with patch.object(analyzer, "_fallback_batches") as mock_fallback:
mock_fallback.return_value = [
{
"issue_iids": [1],
"theme": "Single issue",
"reasoning": "Single issue in group",
"confidence": 1.0,
}
]
result = await analyzer.analyze_and_batch_issues(issues)
assert len(result) == 1
assert result[0]["issue_iids"] == [1]
@pytest.mark.asyncio
async def test_analyze_empty_list(self, mock_config, tmp_path):
"""Test analyzing empty issue list."""
analyzer = ClaudeGitlabBatchAnalyzer(project_dir=tmp_path)
result = await analyzer.analyze_and_batch_issues([])
assert result == []
@pytest.mark.asyncio
async def test_parse_json_response(self, mock_config, tmp_path):
"""Test JSON parsing from Claude response."""
analyzer = ClaudeGitlabBatchAnalyzer(project_dir=tmp_path)
# Valid JSON
json_str = '{"batches": [{"issue_iids": [1, 2]}]}'
result = analyzer._parse_json_response(json_str)
assert "batches" in result
@pytest.mark.asyncio
async def test_parse_json_from_markdown(self, mock_config, tmp_path):
"""Test extracting JSON from markdown code blocks."""
analyzer = ClaudeGitlabBatchAnalyzer(project_dir=tmp_path)
# JSON in markdown code block
response = '```json\n{"batches": [{"issue_iids": [1, 2]}]}\n```'
result = analyzer._parse_json_response(response)
assert "batches" in result
@pytest.mark.asyncio
async def test_fallback_batches(self, mock_config, tmp_path):
"""Test fallback batching when Claude is unavailable."""
analyzer = ClaudeGitlabBatchAnalyzer(project_dir=tmp_path)
issues = [
{"iid": 1, "title": "Issue 1"},
{"iid": 2, "title": "Issue 2"},
]
result = analyzer._fallback_batches(issues)
assert len(result) == 2
assert all("confidence" in r for r in result)
class TestIssueBatchItem:
"""Tests for IssueBatchItem model."""
def test_batch_item_to_dict(self):
"""Test converting batch item to dict."""
item = GitlabIssueBatchItem(
issue_iid=123,
title="Test Issue",
body="Description",
labels=["bug"],
similarity_to_primary=0.8,
)
result = item.to_dict()
assert result["issue_iid"] == 123
assert result["similarity_to_primary"] == 0.8
def test_batch_item_from_dict(self):
"""Test creating batch item from dict."""
data = {
"issue_iid": 456,
"title": "Test",
"body": "Desc",
"labels": ["feature"],
"similarity_to_primary": 1.0,
}
result = GitlabIssueBatchItem.from_dict(data)
assert result.issue_iid == 456
class TestIssueBatch:
"""Tests for IssueBatch model."""
def test_batch_creation(self):
"""Test creating a batch."""
issues = [
GitlabIssueBatchItem(
issue_iid=1,
title="Issue 1",
body="",
),
GitlabIssueBatchItem(
issue_iid=2,
title="Issue 2",
body="",
),
]
batch = GitlabIssueBatch(
batch_id="batch-1-2",
project="namespace/test-project",
primary_issue=1,
issues=issues,
theme="Authentication issues",
)
assert batch.batch_id == "batch-1-2"
assert batch.primary_issue == 1
assert len(batch.issues) == 2
def test_batch_to_dict(self):
"""Test converting batch to dict."""
batch = GitlabIssueBatch(
batch_id="batch-1",
project="namespace/project",
primary_issue=1,
issues=[],
status=GitlabBatchStatus.PENDING,
)
result = batch.to_dict()
assert result["batch_id"] == "batch-1"
assert result["status"] == "pending"
def test_batch_from_dict(self):
"""Test creating batch from dict."""
data = {
"batch_id": "batch-1",
"project": "namespace/project",
"primary_issue": 1,
"issues": [],
"status": "pending",
"created_at": "2024-01-01T00:00:00Z",
}
result = GitlabIssueBatch.from_dict(data)
assert result.batch_id == "batch-1"
assert result.status == GitlabBatchStatus.PENDING
class TestIssueBatcher:
"""Tests for IssueBatcher class."""
def test_batcher_initialization(self, mock_config, tmp_path):
"""Test batcher initialization."""
batcher = GitlabIssueBatcher(
gitlab_dir=tmp_path / ".auto-claude" / "gitlab",
project="namespace/project",
project_dir=tmp_path,
)
assert batcher.project == "namespace/project"
@pytest.mark.asyncio
async def test_create_batches(self, mock_config, tmp_path, sample_issues):
"""Test creating batches from issues."""
batcher = GitlabIssueBatcher(
gitlab_dir=tmp_path / ".auto-claude" / "gitlab",
project="namespace/project",
project_dir=tmp_path,
)
# Patch the analyzer's analyze_and_batch_issues method
with patch.object(batcher.analyzer, "analyze_and_batch_issues") as mock_analyze:
mock_analyze.return_value = [
{
"issue_iids": [1, 2],
"theme": "Auth issues",
"confidence": 0.85,
},
{
"issue_iids": [3],
"theme": "UI bug",
"confidence": 0.9,
},
]
batches = await batcher.create_batches(sample_issues)
assert len(batches) == 2
assert batches[0].theme == "Auth issues"
assert batches[1].theme == "UI bug"
def test_generate_batch_id(self, mock_config, tmp_path):
"""Test batch ID generation."""
batcher = GitlabIssueBatcher(
gitlab_dir=tmp_path / ".auto-claude" / "gitlab",
project="namespace/project",
project_dir=tmp_path,
)
batch_id = batcher._generate_batch_id([1, 2, 3])
assert batch_id == "batch-1-2-3"
def test_save_and_load_batch(self, mock_config, tmp_path):
"""Test saving and loading batches."""
batcher = GitlabIssueBatcher(
gitlab_dir=tmp_path / ".auto-claude" / "gitlab",
project="namespace/project",
project_dir=tmp_path,
)
batch = GitlabIssueBatch(
batch_id="batch-123",
project="namespace/project",
primary_issue=123,
issues=[],
)
# Save
batcher.save_batch(batch)
# Load
loaded = batcher.load_batch(tmp_path / ".auto-claude" / "gitlab", "batch-123")
assert loaded is not None
assert loaded.batch_id == "batch-123"
def test_list_batches(self, mock_config, tmp_path):
"""Test listing all batches."""
batcher = GitlabIssueBatcher(
gitlab_dir=tmp_path / ".auto-claude" / "gitlab",
project="namespace/project",
project_dir=tmp_path,
)
# Create a couple of batches
batch1 = GitlabIssueBatch(
batch_id="batch-1",
project="namespace/project",
primary_issue=1,
issues=[],
status=GitlabBatchStatus.PENDING,
)
batch2 = GitlabIssueBatch(
batch_id="batch-2",
project="namespace/project",
primary_issue=2,
issues=[],
status=GitlabBatchStatus.COMPLETED,
)
batcher.save_batch(batch1)
batcher.save_batch(batch2)
# List
batches = batcher.list_batches()
assert len(batches) == 2
# Should be sorted by created_at descending
assert batches[0].batch_id == "batch-2"
assert batches[1].batch_id == "batch-1"
class TestBatchStatus:
"""Tests for BatchStatus enum."""
def test_status_values(self):
"""Test all status values exist."""
expected_statuses = [
GitlabBatchStatus.PENDING,
GitlabBatchStatus.ANALYZING,
GitlabBatchStatus.CREATING_SPEC,
GitlabBatchStatus.BUILDING,
GitlabBatchStatus.QA_REVIEW,
GitlabBatchStatus.MR_CREATED,
GitlabBatchStatus.COMPLETED,
GitlabBatchStatus.FAILED,
]
for status in expected_statuses:
assert status.value in [
"pending",
"analyzing",
"creating_spec",
"building",
"qa_review",
"mr_created",
"completed",
"failed",
]
class TestBatchSummaryFormatting:
"""Tests for batch summary formatting."""
def test_format_batch_summary(self):
"""Test formatting a batch summary."""
batch = GitlabIssueBatch(
batch_id="batch-auth-issues",
project="namespace/project",
primary_issue=1,
issues=[
GitlabIssueBatchItem(
issue_iid=1,
title="Login bug",
body="",
),
GitlabIssueBatchItem(
issue_iid=2,
title="Signup bug",
body="",
),
],
common_themes=["Authentication issues"],
status=GitlabBatchStatus.PENDING,
)
summary = format_batch_summary(batch)
assert "batch-auth-issues" in summary
assert "!1" in summary
assert "!2" in summary
assert "Authentication issues" in summary
class TestSimilarityThreshold:
"""Tests for similarity threshold handling."""
def test_threshold_filtering(self, mock_config, tmp_path):
"""Test that similarity threshold is respected."""
batcher = GitlabIssueBatcher(
gitlab_dir=tmp_path / ".auto-claude" / "gitlab",
project="namespace/project",
project_dir=tmp_path,
similarity_threshold=0.8, # High threshold
)
assert batcher.similarity_threshold == 0.8
class TestBatchSizeLimits:
"""Tests for batch size limits."""
def test_max_batch_size(self, mock_config, tmp_path):
"""Test that max batch size is enforced."""
batcher = GitlabIssueBatcher(
gitlab_dir=tmp_path / ".auto-claude" / "gitlab",
project="namespace/project",
project_dir=tmp_path,
max_batch_size=3,
)
assert batcher.max_batch_size == 3
def test_min_batch_size(self, mock_config, tmp_path):
"""Test min batch size setting."""
batcher = GitlabIssueBatcher(
gitlab_dir=tmp_path / ".auto-claude" / "gitlab",
project="namespace/project",
project_dir=tmp_path,
min_batch_size=2,
)
assert batcher.min_batch_size == 2
@@ -0,0 +1,253 @@
"""
GitLab Bot Detection Tests
==========================
Tests for bot detection to prevent infinite review loops.
"""
import json
from datetime import datetime, timedelta, timezone
from pathlib import Path
from unittest.mock import MagicMock, Mock, patch
import pytest
from __tests__.fixtures.gitlab import (
MOCK_GITLAB_CONFIG,
mock_mr_data,
)
class TestBotDetector:
"""Test bot detection prevents infinite loops."""
@pytest.fixture
def detector(self, tmp_path):
"""Create a BotDetector instance for testing."""
from runners.gitlab.bot_detection import BotDetector
return BotDetector(
state_dir=tmp_path,
bot_username="auto-claude-bot",
review_own_mrs=False,
)
def test_bot_detection_init(self, detector):
"""Test detector initializes correctly."""
assert detector.bot_username == "auto-claude-bot"
assert detector.review_own_mrs is False
assert detector.state.reviewed_commits == {}
def test_is_bot_mr_self_authored(self, detector):
"""Test MR authored by bot is detected."""
mr_data = mock_mr_data(author="auto-claude-bot")
assert detector.is_bot_mr(mr_data) is True
def test_is_bot_mr_pattern_match(self, detector):
"""Test MR with bot pattern in username is detected."""
mr_data = mock_mr_data(author="coderabbit[bot]")
assert detector.is_bot_mr(mr_data) is True
def test_is_bot_mr_human_authored(self, detector):
"""Test MR authored by human is not detected as bot."""
mr_data = mock_mr_data(author="john_doe")
assert detector.is_bot_mr(mr_data) is False
def test_is_bot_commit_self_authored(self, detector):
"""Test commit by bot is detected."""
commit = {
"author": {"username": "auto-claude-bot"},
"message": "Fix issue",
}
assert detector.is_bot_commit(commit) is True
def test_is_bot_commit_ai_coauthored(self, detector):
"""Test commit with AI co-authorship is detected."""
commit = {
"author": {"username": "human"},
"message": "Co-authored-by: claude <no-reply>",
}
assert detector.is_bot_commit(commit) is True
def test_is_bot_commit_human(self, detector):
"""Test human commit is not detected as bot."""
commit = {
"author": {"username": "john_doe"},
"message": "Fix bug",
}
assert detector.is_bot_commit(commit) is False
def test_should_skip_mr_bot_authored(self, detector):
"""Test should skip MR when bot authored."""
mr_data = mock_mr_data(author="auto-claude-bot")
commits = []
should_skip, reason = detector.should_skip_mr_review(123, mr_data, commits)
assert should_skip is True
assert "auto-claude-bot" in reason.lower()
def test_should_skip_mr_in_cooling_off(self, detector):
"""Test should skip MR when in cooling off period."""
# First, mark as reviewed
detector.mark_reviewed(123, "abc123")
# Immediately try to review again
mr_data = mock_mr_data()
commits = [{"id": "abc123", "sha": "abc123"}]
should_skip, reason = detector.should_skip_mr_review(123, mr_data, commits)
assert should_skip is True
assert "cooling" in reason.lower()
def test_should_skip_mr_already_reviewed(self, detector):
"""Test should skip MR when commit already reviewed."""
# Mark as reviewed
detector.mark_reviewed(123, "abc123")
# Try to review same commit
mr_data = mock_mr_data()
commits = [{"id": "abc123", "sha": "abc123"}]
# Wait past cooling off (manually update time)
detector.state.last_review_times["123"] = (
datetime.now(timezone.utc) - timedelta(minutes=10)
).isoformat()
should_skip, reason = detector.should_skip_mr_review(123, mr_data, commits)
assert should_skip is True
assert "already reviewed" in reason.lower()
def test_should_not_skip_safe_mr(self, detector):
"""Test should not skip when MR is safe to review."""
mr_data = mock_mr_data()
commits = [{"id": "new123", "sha": "new123"}]
should_skip, reason = detector.should_skip_mr_review(456, mr_data, commits)
assert should_skip is False
assert reason == ""
def test_mark_reviewed(self, detector):
"""Test marking MR as reviewed."""
detector.mark_reviewed(123, "abc123")
assert "123" in detector.state.reviewed_commits
assert "abc123" in detector.state.reviewed_commits["123"]
assert "123" in detector.state.last_review_times
def test_mark_reviewed_multiple_commits(self, detector):
"""Test marking multiple commits for same MR."""
detector.mark_reviewed(123, "commit1")
detector.mark_reviewed(123, "commit2")
detector.mark_reviewed(123, "commit3")
assert len(detector.state.reviewed_commits["123"]) == 3
def test_clear_mr_state(self, detector):
"""Test clearing MR state."""
detector.mark_reviewed(123, "abc123")
detector.clear_mr_state(123)
assert "123" not in detector.state.reviewed_commits
assert "123" not in detector.state.last_review_times
def test_get_stats(self, detector):
"""Test getting detector statistics."""
detector.mark_reviewed(123, "abc123")
detector.mark_reviewed(124, "def456")
stats = detector.get_stats()
assert stats["bot_username"] == "auto-claude-bot"
assert stats["total_mrs_tracked"] == 2
assert stats["total_reviews_performed"] == 2
def test_cleanup_stale_mrs(self, detector):
"""Test cleanup of old MR state."""
# Add an old MR (manually set old timestamp)
old_time = (datetime.now(timezone.utc) - timedelta(days=40)).isoformat()
detector.state.last_review_times["999"] = old_time
detector.state.reviewed_commits["999"] = ["old123"]
# Add a recent MR
detector.mark_reviewed(123, "abc123")
cleaned = detector.cleanup_stale_mrs(max_age_days=30)
assert cleaned == 1
assert "999" not in detector.state.reviewed_commits
assert "123" in detector.state.reviewed_commits
def test_state_persistence(self, tmp_path):
"""Test state is saved and loaded correctly."""
from runners.gitlab.bot_detection import BotDetector
# Create detector and mark as reviewed
detector1 = BotDetector(
state_dir=tmp_path,
bot_username="test-bot",
)
detector1.mark_reviewed(123, "abc123")
# Create new detector instance (should load state)
detector2 = BotDetector(
state_dir=tmp_path,
bot_username="test-bot",
)
assert "123" in detector2.state.reviewed_commits
assert "abc123" in detector2.state.reviewed_commits["123"]
class TestBotDetectionState:
"""Test BotDetectionState model."""
def test_to_dict(self):
"""Test converting state to dictionary."""
from runners.gitlab.bot_detection import BotDetectionState
state = BotDetectionState(
reviewed_commits={"123": ["abc123", "def456"]},
last_review_times={"123": "2025-01-14T10:00:00"},
)
data = state.to_dict()
assert data["reviewed_commits"]["123"] == ["abc123", "def456"]
def test_from_dict(self):
"""Test loading state from dictionary."""
from runners.gitlab.bot_detection import BotDetectionState
data = {
"reviewed_commits": {"123": ["abc123"]},
"last_review_times": {"123": "2025-01-14T10:00:00"},
}
state = BotDetectionState.from_dict(data)
assert state.reviewed_commits["123"] == ["abc123"]
assert state.last_review_times["123"] == "2025-01-14T10:00:00"
def test_save_and_load(self, tmp_path):
"""Test saving and loading state from disk."""
from runners.gitlab.bot_detection import BotDetectionState
state = BotDetectionState(
reviewed_commits={"123": ["abc123"]},
last_review_times={"123": "2025-01-14T10:00:00"},
)
state.save(tmp_path)
loaded = BotDetectionState.load(tmp_path)
assert loaded.reviewed_commits["123"] == ["abc123"]
@@ -0,0 +1,263 @@
"""
Tests for GitLab Branch Operations
====================================
Tests for branch listing, creation, deletion, and comparison.
"""
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
try:
from runners.gitlab.glab_client import GitLabClient, GitLabConfig
except ImportError:
from glab_client import GitLabClient, GitLabConfig
@pytest.fixture
def mock_config():
"""Create a mock GitLab config."""
return GitLabConfig(
token="test-token",
project="namespace/test-project",
instance_url="https://gitlab.example.com",
)
@pytest.fixture
def client(mock_config, tmp_path):
"""Create a GitLab client instance."""
return GitLabClient(
project_dir=tmp_path,
config=mock_config,
)
@pytest.fixture
def sample_branches():
"""Sample branch data."""
return [
{
"name": "main",
"merged": False,
"protected": True,
"default": True,
"developers_can_push": False,
"developers_can_merge": False,
"commit": {
"id": "abc123def456",
"short_id": "abc123d",
"title": "Stable branch",
},
"web_url": "https://gitlab.example.com/namespace/test-project/-/tree/main",
},
{
"name": "develop",
"merged": False,
"protected": False,
"default": False,
"developers_can_push": True,
"developers_can_merge": True,
"commit": {
"id": "def456abc123",
"short_id": "def456a",
"title": "Development branch",
},
"web_url": "https://gitlab.example.com/namespace/test-project/-/tree/develop",
},
]
class TestListBranches:
"""Tests for list_branches method."""
@pytest.mark.asyncio
async def test_list_all_branches(self, client, sample_branches):
"""Test listing all branches."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = sample_branches
result = client.list_branches()
assert len(result) == 2
assert result[0]["name"] == "main"
assert result[1]["name"] == "develop"
@pytest.mark.asyncio
async def test_list_branches_with_search(self, client, sample_branches):
"""Test listing branches with search filter."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = [sample_branches[0]] # Only main
result = client.list_branches(search="main")
assert len(result) == 1
assert result[0]["name"] == "main"
@pytest.mark.asyncio
async def test_list_branches_async(self, client, sample_branches):
"""Test async variant of list_branches."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = sample_branches
result = await client.list_branches_async()
assert len(result) == 2
class TestGetBranch:
"""Tests for get_branch method."""
@pytest.mark.asyncio
async def test_get_existing_branch(self, client, sample_branches):
"""Test getting an existing branch."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = sample_branches[0]
result = client.get_branch("main")
assert result["name"] == "main"
assert result["protected"] is True
assert result["commit"]["id"] == "abc123def456"
@pytest.mark.asyncio
async def test_get_branch_async(self, client, sample_branches):
"""Test async variant of get_branch."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = sample_branches[0]
result = await client.get_branch_async("main")
assert result["name"] == "main"
@pytest.mark.asyncio
async def test_get_nonexistent_branch(self, client):
"""Test getting a branch that doesn't exist."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.side_effect = Exception("404 Not Found")
with pytest.raises(Exception): # noqa: B017
client.get_branch("nonexistent")
class TestCreateBranch:
"""Tests for create_branch method."""
@pytest.mark.asyncio
async def test_create_branch_from_ref(self, client):
"""Test creating a branch from another branch."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"name": "feature-branch",
"commit": {"id": "new123"},
"protected": False,
}
result = client.create_branch(
branch_name="feature-branch",
ref="main",
)
assert result["name"] == "feature-branch"
@pytest.mark.asyncio
async def test_create_branch_from_commit(self, client):
"""Test creating a branch from a commit SHA."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"name": "fix-branch",
"commit": {"id": "fix123"},
}
result = client.create_branch(
branch_name="fix-branch",
ref="abc123def",
)
assert result["name"] == "fix-branch"
@pytest.mark.asyncio
async def test_create_branch_async(self, client):
"""Test async variant of create_branch."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {"name": "feature", "commit": {}}
result = await client.create_branch_async("feature", "main")
assert result["name"] == "feature"
class TestDeleteBranch:
"""Tests for delete_branch method."""
@pytest.mark.asyncio
async def test_delete_existing_branch(self, client):
"""Test deleting an existing branch."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = None # 204 No Content
result = client.delete_branch("feature-branch")
# Should not raise on success
assert result is None
@pytest.mark.asyncio
async def test_delete_branch_async(self, client):
"""Test async variant of delete_branch."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = None
result = await client.delete_branch_async("old-branch")
assert result is None
class TestCompareBranches:
"""Tests for compare_branches method."""
@pytest.mark.asyncio
async def test_compare_branches_basic(self, client):
"""Test comparing two branches."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"diff": "@@ -1,1 +1,1 @@",
"commits": [{"id": "abc123"}],
"compare_same_ref": False,
}
result = client.compare_branches("main", "feature")
assert "diff" in result
assert result["compare_same_ref"] is False
@pytest.mark.asyncio
async def test_compare_branches_async(self, client):
"""Test async variant of compare_branches."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"diff": "@@ -1,1 +1,1 @@",
}
result = await client.compare_branches_async("main", "feature")
assert "diff" in result
@pytest.mark.asyncio
async def test_compare_same_branch(self, client):
"""Test comparing a branch to itself."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"diff": "",
"compare_same_ref": True,
}
result = client.compare_branches("main", "main")
assert result["compare_same_ref"] is True
@@ -0,0 +1,376 @@
"""
GitLab CI Checker Tests
========================
Tests for CI/CD pipeline status checking.
"""
from datetime import datetime, timezone
from pathlib import Path
from unittest.mock import MagicMock, Mock, patch
import pytest
from __tests__.fixtures.gitlab import (
MOCK_GITLAB_CONFIG,
mock_mr_data,
mock_pipeline_data,
mock_pipeline_jobs,
)
class TestCIChecker:
"""Test CI/CD pipeline checking functionality."""
@pytest.fixture
def checker(self, tmp_path):
"""Create a CIChecker instance for testing."""
from runners.gitlab.glab_client import GitLabConfig
from runners.gitlab.services.ci_checker import CIChecker
config = GitLabConfig(
token="test-token",
project="group/project",
instance_url="https://gitlab.example.com",
)
with patch("runners.gitlab.services.ci_checker.GitLabClient"):
return CIChecker(
project_dir=tmp_path,
config=config,
)
def test_init(self, checker):
"""Test checker initializes correctly."""
assert checker.client is not None
def test_check_mr_pipeline_success(self, checker):
"""Test checking MR with successful pipeline."""
pipeline_data = mock_pipeline_data(status="success")
async def mock_get_pipelines(mr_iid):
return [pipeline_data]
async def mock_get_pipeline_status(pipeline_id):
return pipeline_data
async def mock_get_pipeline_jobs(pipeline_id):
return mock_pipeline_jobs()
# Setup async mocks
import asyncio
async def test():
with patch.object(
checker.client, "get_mr_pipelines_async", mock_get_pipelines
):
with patch.object(
checker.client,
"get_pipeline_status_async",
mock_get_pipeline_status,
):
with patch.object(
checker.client,
"get_pipeline_jobs_async",
mock_get_pipeline_jobs,
):
pipeline = await checker.check_mr_pipeline(123)
assert pipeline is not None
assert pipeline.pipeline_id == 1001
assert pipeline.status.value == "success"
assert pipeline.has_failures is False
asyncio.run(test())
def test_check_mr_pipeline_failed(self, checker):
"""Test checking MR with failed pipeline."""
pipeline_data = mock_pipeline_data(status="failed")
jobs_data = mock_pipeline_jobs()
jobs_data[0]["status"] = "failed"
import asyncio
async def test():
async def mock_get_pipelines(mr_iid):
return [pipeline_data]
async def mock_get_pipeline_status(pipeline_id):
return pipeline_data
async def mock_get_pipeline_jobs(pipeline_id):
return jobs_data
with patch.object(
checker.client, "get_mr_pipelines_async", mock_get_pipelines
):
with patch.object(
checker.client,
"get_pipeline_status_async",
mock_get_pipeline_status,
):
with patch.object(
checker.client,
"get_pipeline_jobs_async",
mock_get_pipeline_jobs,
):
pipeline = await checker.check_mr_pipeline(123)
assert pipeline.has_failures is True
assert pipeline.is_blocking is True
asyncio.run(test())
def test_check_mr_pipeline_no_pipeline(self, checker):
"""Test checking MR with no pipeline."""
import asyncio
async def test():
async def mock_get_pipelines(mr_iid):
return []
with patch.object(
checker.client, "get_mr_pipelines_async", mock_get_pipelines
):
pipeline = await checker.check_mr_pipeline(123)
assert pipeline is None
asyncio.run(test())
def test_get_blocking_reason_success(self, checker):
"""Test getting blocking reason for successful pipeline."""
from runners.gitlab.services.ci_checker import PipelineInfo, PipelineStatus
pipeline = PipelineInfo(
pipeline_id=1001,
status=PipelineStatus.SUCCESS,
ref="main",
sha="abc123",
created_at="2025-01-14T10:00:00",
updated_at="2025-01-14T10:05:00",
failed_jobs=[],
)
reason = checker.get_blocking_reason(pipeline)
assert reason == ""
def test_get_blocking_reason_failed(self, checker):
"""Test getting blocking reason for failed pipeline."""
from runners.gitlab.services.ci_checker import (
JobStatus,
PipelineInfo,
PipelineStatus,
)
pipeline = PipelineInfo(
pipeline_id=1001,
status=PipelineStatus.FAILED,
ref="main",
sha="abc123",
created_at="2025-01-14T10:00:00",
updated_at="2025-01-14T10:05:00",
failed_jobs=[
JobStatus(
name="test",
status="failed",
stage="test",
failure_reason="AssertionError",
)
],
)
reason = checker.get_blocking_reason(pipeline)
assert "failed" in reason.lower()
def test_format_pipeline_summary(self, checker):
"""Test formatting pipeline summary."""
from runners.gitlab.services.ci_checker import (
JobStatus,
PipelineInfo,
PipelineStatus,
)
pipeline = PipelineInfo(
pipeline_id=1001,
status=PipelineStatus.SUCCESS,
ref="main",
sha="abc123",
created_at="2025-01-14T10:00:00",
updated_at="2025-01-14T10:05:00",
duration=300,
jobs=[
JobStatus(
name="test",
status="success",
stage="test",
),
JobStatus(
name="lint",
status="success",
stage="lint",
),
],
)
summary = checker.format_pipeline_summary(pipeline)
assert "Pipeline #1001" in summary
assert "SUCCESS" in summary
assert "2 total" in summary
def test_security_scan_detection(self, checker):
"""Test detection of security scan failures."""
from runners.gitlab.services.ci_checker import JobStatus
jobs = [
JobStatus(
name="sast",
status="failed",
stage="test",
failure_reason="Vulnerability found",
),
JobStatus(
name="secret_detection",
status="failed",
stage="test",
failure_reason="Secret leaked",
),
JobStatus(
name="test",
status="success",
stage="test",
),
]
issues = checker._check_security_scans(jobs)
assert len(issues) == 2
assert any(i["type"] == "Static Application Security Testing" for i in issues)
assert any(i["type"] == "Secret Detection" for i in issues)
class TestPipelineStatus:
"""Test PipelineStatus enum."""
def test_status_values(self):
"""Test all status values exist."""
from runners.gitlab.services.ci_checker import PipelineStatus
assert PipelineStatus.PENDING.value == "pending"
assert PipelineStatus.RUNNING.value == "running"
assert PipelineStatus.SUCCESS.value == "success"
assert PipelineStatus.FAILED.value == "failed"
assert PipelineStatus.CANCELED.value == "canceled"
class TestJobStatus:
"""Test JobStatus model."""
def test_job_status_creation(self):
"""Test creating JobStatus."""
from runners.gitlab.services.ci_checker import JobStatus
job = JobStatus(
name="test",
status="success",
stage="test",
started_at="2025-01-14T10:00:00",
finished_at="2025-01-14T10:01:00",
duration=60,
)
assert job.name == "test"
assert job.status == "success"
assert job.duration == 60
class TestPipelineInfo:
"""Test PipelineInfo model."""
def test_pipeline_info_creation(self):
"""Test creating PipelineInfo."""
from runners.gitlab.services.ci_checker import PipelineInfo, PipelineStatus
pipeline = PipelineInfo(
pipeline_id=1001,
status=PipelineStatus.SUCCESS,
ref="main",
sha="abc123",
created_at="2025-01-14T10:00:00",
updated_at="2025-01-14T10:05:00",
)
assert pipeline.pipeline_id == 1001
assert pipeline.has_failures is False
assert pipeline.is_blocking is False
def test_has_failures_property(self):
"""Test has_failures property."""
from runners.gitlab.services.ci_checker import (
JobStatus,
PipelineInfo,
PipelineStatus,
)
pipeline = PipelineInfo(
pipeline_id=1001,
status=PipelineStatus.FAILED,
ref="main",
sha="abc123",
created_at="2025-01-14T10:00:00",
updated_at="2025-01-14T10:05:00",
failed_jobs=[
JobStatus(name="test", status="failed", stage="test"),
],
)
assert pipeline.has_failures is True
assert len(pipeline.failed_jobs) == 1
def test_is_blocking_success(self):
"""Test is_blocking for successful pipeline."""
from runners.gitlab.services.ci_checker import PipelineInfo, PipelineStatus
pipeline = PipelineInfo(
pipeline_id=1001,
status=PipelineStatus.SUCCESS,
ref="main",
sha="abc123",
created_at="2025-01-14T10:00:00",
updated_at="2025-01-14T10:05:00",
)
assert pipeline.is_blocking is False
def test_is_blocking_failed(self):
"""Test is_blocking for failed pipeline."""
from runners.gitlab.services.ci_checker import PipelineInfo, PipelineStatus
pipeline = PipelineInfo(
pipeline_id=1001,
status=PipelineStatus.FAILED,
ref="main",
sha="abc123",
created_at="2025-01-14T10:00:00",
updated_at="2025-01-14T10:05:00",
)
assert pipeline.is_blocking is True
def test_is_blocking_running(self):
"""Test is_blocking for running pipeline."""
from runners.gitlab.services.ci_checker import PipelineInfo, PipelineStatus
pipeline = PipelineInfo(
pipeline_id=1001,
status=PipelineStatus.RUNNING,
ref="main",
sha="abc123",
created_at="2025-01-14T10:00:00",
updated_at="2025-01-14T10:05:00",
)
# Running with no failed jobs is not blocking
assert pipeline.is_blocking is False
@@ -0,0 +1,322 @@
"""
Tests for GitLab Client Error Handling
=======================================
Tests for enhanced retry logic, rate limiting, and error handling.
"""
import socket
import urllib.error
from pathlib import Path
from unittest.mock import MagicMock, Mock, patch
import pytest
try:
from runners.gitlab.glab_client import GitLabClient, GitLabConfig
except ImportError:
from glab_client import GitLabClient, GitLabConfig
@pytest.fixture
def mock_config():
"""Create a mock GitLab config."""
return GitLabConfig(
token="test-token",
project="namespace/test-project",
instance_url="https://gitlab.example.com",
)
@pytest.fixture
def client(mock_config, tmp_path):
"""Create a GitLab client instance."""
return GitLabClient(
project_dir=tmp_path,
config=mock_config,
default_timeout=5.0,
)
def _create_mock_response(
status=200, content=b'{"id": 123}', content_type="application/json", headers=None
):
"""Helper to create a mock HTTP response."""
mock_resp = Mock()
mock_resp.status = status
mock_resp.read = lambda: content
# Use a real dict for headers to properly support .get() method
headers_dict = {"Content-Type": content_type}
if headers:
headers_dict.update(headers)
mock_resp.headers = headers_dict
# Support context manager protocol
mock_resp.__enter__ = Mock(return_value=mock_resp)
mock_resp.__exit__ = Mock(return_value=False)
return mock_resp
class TestRetryLogic:
"""Tests for retry logic on transient failures."""
@pytest.mark.asyncio
async def test_retry_on_429_rate_limit(self, client):
"""Test retry on HTTP 429 rate limit."""
call_count = 0
def mock_urlopen(request, timeout=None):
nonlocal call_count
call_count += 1
if call_count == 1:
# First call: rate limited
error = urllib.error.HTTPError(
url="https://example.com",
code=429,
msg="Rate limited",
hdrs={"Retry-After": "1"},
fp=None,
)
error.read = lambda: b""
raise error
# Second call: success
return _create_mock_response()
with patch("urllib.request.urlopen", mock_urlopen):
result = client._fetch("/projects/namespace%2Fproject")
assert call_count == 2 # Retried once
@pytest.mark.asyncio
async def test_retry_on_500_server_error(self, client):
"""Test retry on HTTP 500 server error."""
call_count = 0
def mock_urlopen(request, timeout=None):
nonlocal call_count
call_count += 1
if call_count < 2:
error = urllib.error.HTTPError(
url="https://example.com",
code=500,
msg="Internal server error",
hdrs={},
fp=None,
)
error.read = lambda: b""
raise error
return _create_mock_response()
with patch("urllib.request.urlopen", mock_urlopen):
result = client._fetch("/projects/namespace%2Fproject")
assert call_count == 2
@pytest.mark.asyncio
async def test_retry_on_502_bad_gateway(self, client):
"""Test retry on HTTP 502 bad gateway."""
call_count = 0
def mock_urlopen(request, timeout=None):
nonlocal call_count
call_count += 1
if call_count == 1:
error = urllib.error.HTTPError(
url="https://example.com",
code=502,
msg="Bad gateway",
hdrs={},
fp=None,
)
error.read = lambda: b""
raise error
return _create_mock_response()
with patch("urllib.request.urlopen", mock_urlopen):
result = client._fetch("/projects/namespace%2Fproject")
assert call_count == 2
@pytest.mark.asyncio
async def test_retry_on_socket_timeout(self, client):
"""Test retry on socket timeout."""
call_count = 0
def mock_urlopen(request, timeout=None):
nonlocal call_count
call_count += 1
if call_count == 1:
raise TimeoutError("Connection timed out")
return _create_mock_response()
with patch("urllib.request.urlopen", mock_urlopen):
result = client._fetch("/projects/namespace%2Fproject")
assert call_count == 2
@pytest.mark.asyncio
async def test_retry_on_connection_reset(self, client):
"""Test retry on connection reset."""
call_count = 0
def mock_urlopen(request, timeout=None):
nonlocal call_count
call_count += 1
if call_count == 1:
raise ConnectionResetError("Connection reset")
return _create_mock_response()
with patch("urllib.request.urlopen", mock_urlopen):
result = client._fetch("/projects/namespace%2Fproject")
assert call_count == 2
@pytest.mark.asyncio
async def test_no_retry_on_404_not_found(self, client):
"""Test that 404 errors are not retried."""
call_count = 0
def mock_urlopen(request, timeout=None):
nonlocal call_count
call_count += 1
error = urllib.error.HTTPError(
url="https://example.com",
code=404,
msg="Not found",
hdrs={},
fp=None,
)
error.read = lambda: b""
raise error
with patch("urllib.request.urlopen", mock_urlopen):
with pytest.raises(Exception): # noqa: B017
client._fetch("/projects/namespace%2Fproject")
assert call_count == 1 # No retry
@pytest.mark.asyncio
async def test_max_retries_exceeded(self, client):
"""Test that max retries limit is respected."""
call_count = 0
def mock_urlopen(request, timeout=None):
nonlocal call_count
call_count += 1
# Always fail
raise urllib.error.HTTPError(
url="https://example.com",
code=500,
msg="Server error",
hdrs={},
fp=None,
)
with patch("urllib.request.urlopen", mock_urlopen):
with pytest.raises(Exception, match="GitLab API error"):
client._fetch("/projects/namespace%2Fproject", max_retries=2)
# With max_retries=2, the loop runs range(2) = [0, 1], so 2 attempts total
assert call_count == 2
class TestRateLimiting:
"""Tests for rate limit handling."""
@pytest.mark.asyncio
async def test_retry_after_header_parsing(self, client):
"""Test parsing Retry-After header."""
import time
def mock_urlopen(request, timeout=None):
error = urllib.error.HTTPError(
url="https://example.com",
code=429,
msg="Rate limited",
hdrs={"Retry-After": "2"},
fp=None,
)
error.read = lambda: b""
raise error
with patch("urllib.request.urlopen", mock_urlopen):
with patch("time.sleep") as mock_sleep:
# Should fail after retries
with pytest.raises(Exception): # noqa: B017
client._fetch("/projects/namespace%2Fproject")
# Check that sleep was called with Retry-After value
mock_sleep.assert_called_with(2)
class TestErrorMessages:
"""Tests for helpful error messages."""
@pytest.mark.asyncio
async def test_gitlab_error_message_included(self, client):
"""Test that GitLab error messages are included in exceptions."""
def mock_urlopen(request, timeout=None):
error = urllib.error.HTTPError(
url="https://example.com",
code=400,
msg="Bad request",
hdrs={},
fp=None,
)
error.read = lambda: b'{"message": "Invalid branch name"}'
raise error
with patch("urllib.request.urlopen", mock_urlopen):
with pytest.raises(Exception) as exc_info:
client._fetch("/projects/namespace%2Fproject")
# Error message should include GitLab's message
assert "Invalid branch name" in str(exc_info.value)
@pytest.mark.asyncio
async def test_invalid_endpoint_raises(self, client):
"""Test that invalid endpoints are rejected."""
with pytest.raises(
ValueError, match="does not match known GitLab API patterns"
):
client._fetch("/invalid/endpoint")
class TestResponseSizeLimits:
"""Tests for response size limits."""
@pytest.mark.asyncio
async def test_large_response_rejected(self, client):
"""Test that overly large responses are rejected."""
def mock_urlopen(request, timeout=None):
# Use application/json to trigger size check (status < 400)
return _create_mock_response(
content=b"Large response",
content_type="application/json",
headers={"Content-Length": str(20 * 1024 * 1024)}, # 20MB
)
with patch("urllib.request.urlopen", mock_urlopen):
with pytest.raises(ValueError, match="Response too large"):
client._fetch("/projects/namespace%2Fproject")
class TestContentTypeHandling:
"""Tests for Content-Type validation."""
@pytest.mark.asyncio
async def test_non_json_response_handling(self, client):
"""Test handling of non-JSON responses on success."""
def mock_urlopen(request, timeout=None):
mock_resp = _create_mock_response(
content=b"Plain text response", content_type="text/plain"
)
return mock_resp
with patch("urllib.request.urlopen", mock_urlopen):
result = client._fetch("/projects/namespace%2Fproject")
# Should return raw response for non-JSON on success
assert result == "Plain text response"
@@ -0,0 +1,361 @@
"""
Tests for GitLab Client API Extensions
=========================================
Tests for new CRUD endpoints, branch operations, file operations, and webhooks.
"""
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
# Try imports with fallback for different environments
try:
from runners.gitlab.glab_client import (
GitLabClient,
GitLabConfig,
encode_project_path,
)
except ImportError:
from glab_client import GitLabClient, GitLabConfig, encode_project_path
@pytest.fixture
def mock_config():
"""Create a mock GitLab config."""
return GitLabConfig(
token="test-token",
project="namespace/test-project",
instance_url="https://gitlab.example.com",
)
@pytest.fixture
def client(mock_config, tmp_path):
"""Create a GitLab client instance."""
return GitLabClient(
project_dir=tmp_path,
config=mock_config,
)
class TestMRExtensions:
"""Tests for MR CRUD operations."""
@pytest.mark.asyncio
async def test_create_mr(self, client):
"""Test creating a merge request."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"iid": 123,
"title": "Test MR",
"source_branch": "feature",
"target_branch": "main",
}
result = client.create_mr(
source_branch="feature",
target_branch="main",
title="Test MR",
description="Test description",
)
assert mock_fetch.called
assert result["iid"] == 123
@pytest.mark.asyncio
async def test_list_mrs_filters(self, client):
"""Test listing MRs with filters."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = [
{"iid": 1, "title": "MR 1"},
{"iid": 2, "title": "MR 2"},
]
result = client.list_mrs(state="opened", labels=["bug"])
assert mock_fetch.called
assert len(result) == 2
@pytest.mark.asyncio
async def test_update_mr(self, client):
"""Test updating a merge request."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {"iid": 123, "title": "Updated"}
result = client.update_mr(
mr_iid=123,
title="Updated",
labels={"bug": True, "feature": False},
)
assert mock_fetch.called
class TestBranchOperations:
"""Tests for branch management operations."""
@pytest.mark.asyncio
async def test_list_branches(self, client):
"""Test listing branches."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = [
{"name": "main", "commit": {"id": "abc123"}},
{"name": "develop", "commit": {"id": "def456"}},
]
result = client.list_branches()
assert len(result) == 2
assert result[0]["name"] == "main"
@pytest.mark.asyncio
async def test_get_branch(self, client):
"""Test getting a specific branch."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"name": "main",
"commit": {"id": "abc123"},
"protected": True,
}
result = client.get_branch("main")
assert result["name"] == "main"
@pytest.mark.asyncio
async def test_create_branch(self, client):
"""Test creating a new branch."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"name": "feature-branch",
"commit": {"id": "abc123"},
}
result = client.create_branch(
branch_name="feature-branch",
ref="main",
)
assert result["name"] == "feature-branch"
@pytest.mark.asyncio
async def test_delete_branch(self, client):
"""Test deleting a branch."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = None # 204 No Content
result = client.delete_branch("feature-branch")
# Should not raise on success
assert result is None
@pytest.mark.asyncio
async def test_compare_branches(self, client):
"""Test comparing two branches."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"diff": "@@ -1,1 +1,1 @@",
"commits": [{"id": "abc123"}],
}
result = client.compare_branches("main", "feature")
assert "diff" in result
class TestFileOperations:
"""Tests for file operations."""
@pytest.mark.asyncio
async def test_get_file_contents(self, client):
"""Test getting file contents."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"file_name": "test.py",
"content": "ZGVmIHRlc3Q=", # base64
"encoding": "base64",
}
result = client.get_file_contents("test.py", ref="main")
assert result["file_name"] == "test.py"
@pytest.mark.asyncio
async def test_create_file(self, client):
"""Test creating a new file."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"file_path": "new_file.py",
"branch": "main",
}
result = client.create_file(
file_path="new_file.py",
content="print('hello')",
commit_message="Add new file",
branch="main",
)
assert result["file_path"] == "new_file.py"
@pytest.mark.asyncio
async def test_update_file(self, client):
"""Test updating an existing file."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"file_path": "existing.py",
"branch": "main",
}
result = client.update_file(
file_path="existing.py",
content="updated content",
commit_message="Update file",
branch="main",
)
assert result["file_path"] == "existing.py"
@pytest.mark.asyncio
async def test_delete_file(self, client):
"""Test deleting a file."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = None # 204 No Content
result = client.delete_file(
file_path="old.py",
commit_message="Remove old file",
branch="main",
)
assert result is None
class TestWebhookOperations:
"""Tests for webhook management."""
@pytest.mark.asyncio
async def test_list_webhooks(self, client):
"""Test listing webhooks."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = [
{"id": 1, "url": "https://example.com/hook"},
{"id": 2, "url": "https://example.com/another"},
]
result = client.list_webhooks()
assert len(result) == 2
@pytest.mark.asyncio
async def test_get_webhook(self, client):
"""Test getting a specific webhook."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"id": 1,
"url": "https://example.com/hook",
"push_events": True,
}
result = client.get_webhook(1)
assert result["id"] == 1
@pytest.mark.asyncio
async def test_create_webhook(self, client):
"""Test creating a webhook."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"id": 1,
"url": "https://example.com/hook",
}
result = client.create_webhook(
url="https://example.com/hook",
push_events=True,
merge_request_events=True,
)
assert result["id"] == 1
@pytest.mark.asyncio
async def test_update_webhook(self, client):
"""Test updating a webhook."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"id": 1,
"url": "https://example.com/hook-updated",
}
result = client.update_webhook(
hook_id=1,
url="https://example.com/hook-updated",
)
assert result["url"] == "https://example.com/hook-updated"
@pytest.mark.asyncio
async def test_delete_webhook(self, client):
"""Test deleting a webhook."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = None # 204 No Content
result = client.delete_webhook(1)
assert result is None
class TestAsyncMethods:
"""Tests for async method variants."""
@pytest.mark.asyncio
async def test_create_mr_async(self, client):
"""Test async variant of create_mr."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"iid": 123,
"title": "Test MR",
}
result = await client.create_mr_async(
source_branch="feature",
target_branch="main",
title="Test MR",
)
assert result["iid"] == 123
@pytest.mark.asyncio
async def test_list_branches_async(self, client):
"""Test async variant of list_branches."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = [
{"name": "main"},
]
result = await client.list_branches_async()
assert len(result) == 1
class TestEncoding:
"""Tests for URL encoding."""
def test_encode_project_path_simple(self):
"""Test encoding simple project path."""
result = encode_project_path("namespace/project")
assert result == "namespace%2Fproject"
def test_encode_project_path_with_dots(self):
"""Test encoding project path with dots."""
result = encode_project_path("group.name/project")
assert "group.name%2Fproject" in result or "group%2Ename%2Fproject" in result
def test_encode_project_path_with_slashes(self):
"""Test encoding project path with nested groups."""
result = encode_project_path("group/subgroup/project")
assert result == "group%2Fsubgroup%2Fproject"
@@ -0,0 +1,444 @@
"""
Unit Tests for GitLab MR Context Gatherer Enhancements
======================================================
Tests for enhanced context gathering including monorepo detection,
related files finding, and AI bot comment detection.
"""
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
# Try imports with fallback for different environments
try:
from runners.gitlab.services.context_gatherer import (
CONFIG_FILE_NAMES,
GITLAB_AI_BOT_PATTERNS,
MRContextGatherer,
)
except ImportError:
from runners.gitlab.context_gatherer import (
CONFIG_FILE_NAMES,
GITLAB_AI_BOT_PATTERNS,
MRContextGatherer,
)
@pytest.fixture
def mock_client():
"""Create a mock GitLab client."""
client = MagicMock()
client.get_mr_async = AsyncMock()
client.get_mr_changes_async = AsyncMock()
client.get_mr_commits_async = AsyncMock()
client.get_mr_notes_async = AsyncMock()
client.get_mr_pipeline_async = AsyncMock()
return client
@pytest.fixture
def sample_mr_data():
"""Sample MR data from GitLab API."""
return {
"iid": 123,
"title": "Add new feature",
"description": "This adds a cool feature",
"author": {"username": "developer"},
"source_branch": "feature-branch",
"target_branch": "main",
"state": "opened",
}
@pytest.fixture
def sample_changes_data():
"""Sample MR changes data."""
return {
"changes": [
{
"new_path": "src/utils/helpers.py",
"old_path": "src/utils/helpers.py",
"diff": "@@ -1,1 +1,2 @@\n def helper():\n+ return True",
"new_file": False,
"deleted_file": False,
"renamed_file": False,
},
],
"additions": 10,
"deletions": 5,
}
@pytest.fixture
def sample_commits():
"""Sample commit data."""
return [
{
"id": "abc123",
"short_id": "abc123",
"title": "Add feature",
"message": "Add feature",
}
]
@pytest.fixture
def tmp_project_dir(tmp_path):
"""Create a temporary project directory with structure."""
# Create monorepo structure
(tmp_path / "apps").mkdir()
(tmp_path / "apps" / "backend").mkdir()
(tmp_path / "apps" / "frontend").mkdir()
(tmp_path / "packages").mkdir()
(tmp_path / "packages" / "shared").mkdir()
# Create config files
(tmp_path / "package.json").write_text(
'{"workspaces": ["apps/*", "packages/*"]}', encoding="utf-8"
)
(tmp_path / "tsconfig.json").write_text(
'{"compilerOptions": {"paths": {"@/*": ["src/*"]}}}', encoding="utf-8"
)
(tmp_path / ".gitlab-ci.yml").write_text("stages:\n - test", encoding="utf-8")
# Create source files
(tmp_path / "src").mkdir()
(tmp_path / "src" / "utils").mkdir()
(tmp_path / "src" / "utils" / "helpers.py").write_text(
"def helper():\n return True", encoding="utf-8"
)
# Create test files
(tmp_path / "tests").mkdir()
(tmp_path / "tests" / "test_helpers.py").write_text(
"def test_helper():\n assert True", encoding="utf-8"
)
return tmp_path
@pytest.fixture
def gatherer(tmp_project_dir):
"""Create a context gatherer instance."""
return MRContextGatherer(
project_dir=tmp_project_dir,
mr_iid=123,
config=MagicMock(project="namespace/project", token="test-token"),
)
class TestAIBotPatterns:
"""Test AI bot pattern detection."""
def test_gitlab_ai_bot_patterns_comprehensive(self):
"""Test that AI bot patterns include major tools."""
# Check for known AI tools
assert "coderabbit" in GITLAB_AI_BOT_PATTERNS
assert "greptile" in GITLAB_AI_BOT_PATTERNS
assert "cursor" in GITLAB_AI_BOT_PATTERNS
assert "sourcery-ai" in GITLAB_AI_BOT_PATTERNS
assert "codium" in GITLAB_AI_BOT_PATTERNS
def test_config_file_names_include_gitlab_ci(self):
"""Test that GitLab CI config is included."""
assert ".gitlab-ci.yml" in CONFIG_FILE_NAMES
class TestRepoStructureDetection:
"""Test monorepo and project structure detection."""
def test_detect_monorepo_apps(self, gatherer, tmp_project_dir):
"""Test detection of apps/ directory."""
structure = gatherer._detect_repo_structure()
assert "Monorepo Apps" in structure
assert "backend" in structure
assert "frontend" in structure
def test_detect_monorepo_packages(self, gatherer, tmp_project_dir):
"""Test detection of packages/ directory."""
structure = gatherer._detect_repo_structure()
assert "Packages" in structure
assert "shared" in structure
def test_detect_workspaces(self, gatherer, tmp_project_dir):
"""Test detection of npm workspaces."""
structure = gatherer._detect_repo_structure()
assert "Workspaces" in structure
def test_detect_gitlab_ci(self, gatherer, tmp_project_dir):
"""Test detection of GitLab CI config."""
structure = gatherer._detect_repo_structure()
assert "GitLab CI" in structure
def test_detect_standard_repo(self, tmp_path):
"""Test detection of standard repo without monorepo structure."""
gatherer = MRContextGatherer(
project_dir=tmp_path,
mr_iid=123,
config=MagicMock(project="namespace/project"),
)
structure = gatherer._detect_repo_structure()
assert "Standard single-package repository" in structure
class TestRelatedFilesFinding:
"""Test finding related files for context."""
def test_find_test_files(self, gatherer, tmp_project_dir):
"""Test finding test files for a source file."""
source_path = Path("src/utils/helpers.py")
tests = gatherer._find_test_files(source_path)
# Should find the test file we created
assert "tests/test_helpers.py" in tests
def test_find_config_files(self, gatherer, tmp_project_dir):
"""Test finding config files in directory."""
directory = Path(tmp_project_dir)
configs = gatherer._find_config_files(directory)
# Should find config files in root
assert "package.json" in configs
assert "tsconfig.json" in configs
assert ".gitlab-ci.yml" in configs
def test_find_type_definitions(self, gatherer, tmp_project_dir):
"""Test finding TypeScript type definition files."""
# Create a TypeScript file
(tmp_project_dir / "src" / "types.ts").write_text(
"export type Foo = string;", encoding="utf-8"
)
(tmp_project_dir / "src" / "types.d.ts").write_text(
"export type Bar = number;", encoding="utf-8"
)
source_path = Path("src/types.ts")
type_defs = gatherer._find_type_definitions(source_path)
assert "src/types.d.ts" in type_defs
def test_find_dependents_limits_generic_names(self, gatherer, tmp_project_dir):
"""Test that generic names are skipped in dependent finding."""
# Generic names should be skipped to avoid too many matches
for stem in ["index", "main", "app", "utils", "helpers", "types", "constants"]:
result = gatherer._find_dependents(f"src/{stem}.py")
assert result == set() # Should skip generic names
def test_prioritize_related_files(self, gatherer):
"""Test prioritization of related files."""
files = {
"tests/test_utils.py", # Test file - highest priority
"src/utils.d.ts", # Type definition - high priority
"tsconfig.json", # Config - medium priority
"src/random.py", # Other - low priority
}
prioritized = gatherer._prioritize_related_files(files, limit=10)
# Test files should come first
assert prioritized[0] == "tests/test_utils.py"
assert "src/utils.d.ts" in prioritized[1:3] # Type files next
assert "tsconfig.json" in prioritized # Configs included
class TestJSONLoading:
"""Test JSON loading with comment handling."""
def test_load_json_safe_standard(self, gatherer, tmp_project_dir):
"""Test loading standard JSON without comments."""
(tmp_project_dir / "standard.json").write_text(
'{"key": "value"}', encoding="utf-8"
)
result = gatherer._load_json_safe("standard.json")
assert result == {"key": "value"}
def test_load_json_safe_with_comments(self, gatherer, tmp_project_dir):
"""Test loading JSON with tsconfig-style comments."""
(tmp_project_dir / "with-comments.json").write_text(
"{\n"
" // Single-line comment\n"
' "key": "value",\n'
" /* Multi-line\n"
" comment */\n"
' "key2": "value2"\n'
"}",
encoding="utf-8",
)
result = gatherer._load_json_safe("with-comments.json")
assert result == {"key": "value", "key2": "value2"}
def test_load_json_safe_nonexistent(self, gatherer, tmp_project_dir):
"""Test loading non-existent JSON file."""
result = gatherer._load_json_safe("nonexistent.json")
assert result is None
def test_load_tsconfig_paths(self, gatherer, tmp_project_dir):
"""Test loading tsconfig paths."""
result = gatherer._load_tsconfig_paths()
assert result is not None
assert "@/*" in result
assert "src/*" in result["@/*"]
class TestStaticMethods:
"""Test static utility methods."""
def test_find_related_files_for_root(self, tmp_project_dir):
"""Test static method for finding related files."""
changed_files = [
{"new_path": "src/utils/helpers.py", "old_path": "src/utils/helpers.py"},
]
related = MRContextGatherer.find_related_files_for_root(
changed_files=changed_files,
project_root=tmp_project_dir,
)
# Should find test file
assert "tests/test_helpers.py" in related
# Should not include the changed file itself
assert "src/utils/helpers.py" not in related
@pytest.mark.asyncio
class TestGatherIntegration:
"""Test the full gather method integration."""
async def test_gather_with_enhancements(
self, gatherer, mock_client, sample_mr_data, sample_changes_data, sample_commits
):
"""Test that gather includes repo structure and related files."""
# Setup mock responses
mock_client.get_mr_async.return_value = sample_mr_data
mock_client.get_mr_changes_async.return_value = sample_changes_data
mock_client.get_mr_commits_async.return_value = sample_commits
mock_client.get_mr_notes_async.return_value = []
mock_client.get_mr_pipeline_async.return_value = {
"id": 456,
"status": "success",
}
result = await gatherer.gather()
# Verify enhanced fields are populated
assert result.mr_iid == 123
assert result.repo_structure != ""
assert (
"Monorepo" in result.repo_structure or "Standard" in result.repo_structure
)
assert isinstance(result.related_files, list)
assert result.ci_status == "success"
assert result.ci_pipeline_id == 456
@pytest.mark.asyncio
async def test_gather_handles_missing_ci(
self, gatherer, mock_client, sample_mr_data, sample_changes_data, sample_commits
):
"""Test that gather handles missing CI pipeline gracefully."""
mock_client.get_mr_async.return_value = sample_mr_data
mock_client.get_mr_changes_async.return_value = sample_changes_data
mock_client.get_mr_commits_async.return_value = sample_commits
mock_client.get_mr_notes_async.return_value = []
mock_client.get_mr_pipeline_async.return_value = None
result = await gatherer.gather()
# Should not fail, CI fields should be None
assert result.ci_status is None
assert result.ci_pipeline_id is None
class TestAIBotCommentDetection:
"""Test AI bot comment detection and parsing."""
def test_parse_ai_comment_known_tool(self, gatherer):
"""Test parsing comment from known AI tool."""
note = {
"id": 1,
"author": {"username": "coderabbit[bot]"},
"body": "Consider using async/await here",
"created_at": "2024-01-01T00:00:00Z",
}
result = gatherer._parse_ai_comment(note)
assert result is not None
assert result.tool_name == "CodeRabbit"
assert result.author == "coderabbit[bot]"
def test_parse_ai_comment_unknown_user(self, gatherer):
"""Test parsing comment from unknown user."""
note = {
"id": 1,
"author": {"username": "developer"},
"body": "Just a regular comment",
"created_at": "2024-01-01T00:00:00Z",
}
result = gatherer._parse_ai_comment(note)
assert result is None
def test_parse_ai_comment_no_author(self, gatherer):
"""Test parsing comment with no author."""
note = {
"id": 1,
"body": "Anonymous comment",
"created_at": "2024-01-01T00:00:00Z",
}
result = gatherer._parse_ai_comment(note)
assert result is None
class TestValidation:
"""Test input validation functions."""
def test_validate_git_ref_valid(self):
"""Test validation of valid git refs."""
from runners.gitlab.services.context_gatherer import _validate_git_ref
assert _validate_git_ref("main") is True
assert _validate_git_ref("feature-branch") is True
assert _validate_git_ref("feature/branch-123") is True
assert _validate_git_ref("abc123def456") is True
def test_validate_git_ref_invalid(self):
"""Test validation rejects invalid git refs."""
from runners.gitlab.services.context_gatherer import _validate_git_ref
assert _validate_git_ref("") is False # Empty
assert _validate_git_ref("a" * 300) is False # Too long
assert _validate_git_ref("branch;rm -rf") is False # Invalid chars
def test_validate_file_path_valid(self):
"""Test validation of valid file paths."""
from runners.gitlab.services.context_gatherer import _validate_file_path
assert _validate_file_path("src/file.py") is True
assert _validate_file_path("src/utils/helpers.ts") is True
assert _validate_file_path("src/config.json") is True
def test_validate_file_path_invalid(self):
"""Test validation rejects invalid file paths."""
from runners.gitlab.services.context_gatherer import _validate_file_path
assert _validate_file_path("") is False # Empty
assert _validate_file_path("../etc/passwd") is False # Path traversal
assert _validate_file_path("/etc/passwd") is False # Absolute path
assert _validate_file_path("a" * 1100) is False # Too long
@@ -0,0 +1,422 @@
"""
GitLab File Lock Tests
=======================
Tests for file locking utilities for concurrent safety.
"""
import json
import tempfile
import threading
import time
from pathlib import Path
from unittest.mock import patch
import pytest
class TestFileLock:
"""Test FileLock for concurrent-safe operations."""
@pytest.fixture
def lock_file(self, tmp_path):
"""Create a temporary lock file path."""
return tmp_path / "test.lock"
def test_acquire_lock(self, lock_file):
"""Test acquiring a lock."""
from runners.gitlab.utils.file_lock import FileLock
with FileLock(lock_file, timeout=5.0):
# Lock is held here
assert lock_file.exists()
def test_lock_release(self, lock_file):
"""Test lock is released after context."""
from runners.gitlab.utils.file_lock import FileLock
with FileLock(lock_file, timeout=5.0):
pass
# Lock file should be cleaned up
assert not lock_file.exists()
def test_lock_timeout(self, lock_file):
"""Test lock timeout when held by another process."""
from runners.gitlab.utils.file_lock import FileLock, FileLockTimeout
# Hold lock in separate thread
def hold_lock():
with FileLock(lock_file, timeout=5.0):
time.sleep(0.5)
thread = threading.Thread(target=hold_lock)
thread.start()
# Wait a bit for lock to be acquired
time.sleep(0.1)
# Try to acquire with short timeout
with pytest.raises(FileLockTimeout):
FileLock(lock_file, timeout=0.1).acquire()
thread.join()
def test_exclusive_lock(self, lock_file):
"""Test exclusive lock prevents concurrent writes."""
from runners.gitlab.utils.file_lock import FileLock
results = []
def try_write(value):
try:
with FileLock(lock_file, timeout=1.0, exclusive=True):
with open(
lock_file.with_suffix(".txt"), "w", encoding="utf-8"
) as f:
f.write(str(value))
results.append(value)
except Exception:
results.append(None)
threads = [
threading.Thread(target=try_write, args=(1,)),
threading.Thread(target=try_write, args=(2,)),
]
for t in threads:
t.start()
for t in threads:
t.join()
# Only one should have succeeded
successful = [r for r in results if r is not None]
assert len(successful) == 1
def test_lock_cleanup_on_error(self, lock_file):
"""Test lock is cleaned up even on error."""
from runners.gitlab.utils.file_lock import FileLock
try:
with FileLock(lock_file, timeout=5.0):
raise ValueError("Simulated error")
except ValueError:
pass
# Lock should be cleaned up despite error
assert not lock_file.exists()
class TestAtomicWrite:
"""Test atomic_write for safe file writes."""
@pytest.fixture
def target_file(self, tmp_path):
"""Create a temporary target file."""
return tmp_path / "target.txt"
def test_atomic_write_creates_file(self, target_file):
"""Test atomic write creates target file."""
from runners.gitlab.utils.file_lock import atomic_write
with atomic_write(target_file) as f:
f.write("test content")
assert target_file.exists()
assert target_file.read_text(encoding="utf-8") == "test content"
def test_atomic_write_preserves_on_error(self, target_file):
"""Test atomic write doesn't corrupt on error."""
from runners.gitlab.utils.file_lock import atomic_write
# Create initial content
target_file.write_text("original content", encoding="utf-8")
try:
with atomic_write(target_file) as f:
f.write("new content")
raise ValueError("Simulated error")
except ValueError:
pass
# Original content should be preserved
assert target_file.read_text(encoding="utf-8") == "original content"
def test_atomic_write_context_manager(self, target_file):
"""Test atomic write context manager."""
from runners.gitlab.utils.file_lock import atomic_write
with atomic_write(target_file) as f:
f.write("line 1\n")
f.write("line 2\n")
content = target_file.read_text(encoding="utf-8")
assert "line 1" in content
assert "line 2" in content
class TestLockedJsonOperations:
"""Test locked JSON operations."""
@pytest.fixture
def data_file(self, tmp_path):
"""Create a temporary data file."""
return tmp_path / "data.json"
def test_locked_json_write(self, data_file):
"""Test writing JSON with file locking."""
from runners.gitlab.utils.file_lock import locked_json_write
data = {"key": "value", "number": 42}
locked_json_write(data_file, data)
assert data_file.exists()
with open(data_file, encoding="utf-8") as f:
loaded = json.load(f)
assert loaded == data
def test_locked_json_read(self, data_file):
"""Test reading JSON with file locking."""
from runners.gitlab.utils.file_lock import locked_json_read, locked_json_write
data = {"key": "value", "nested": {"item": 1}}
locked_json_write(data_file, data)
loaded = locked_json_read(data_file)
assert loaded == data
def test_locked_json_update(self, data_file):
"""Test updating JSON with file locking."""
from runners.gitlab.utils.file_lock import (
locked_json_read,
locked_json_update,
locked_json_write,
)
initial = {"key": "value"}
locked_json_write(data_file, initial)
def update_fn(data):
data["new_key"] = "new_value"
return data
locked_json_update(data_file, update_fn)
loaded = locked_json_read(data_file)
assert loaded["key"] == "value"
assert loaded["new_key"] == "new_value"
def test_locked_json_read_missing_file(self, tmp_path):
"""Test reading missing JSON file returns None."""
from runners.gitlab.utils.file_lock import locked_json_read
result = locked_json_read(tmp_path / "nonexistent.json")
assert result is None
def test_concurrent_json_writes(self, tmp_path):
"""Test concurrent JSON writes are safe."""
from runners.gitlab.utils.file_lock import (
locked_json_read,
locked_json_update,
locked_json_write,
)
data_file = tmp_path / "concurrent.json"
# Initialize
locked_json_write(data_file, {"counter": 0})
results = []
def increment():
def updater(data):
data["counter"] += 1
return data
locked_json_update(data_file, updater)
result = locked_json_read(data_file)
results.append(result["counter"])
threads = [
threading.Thread(target=increment),
threading.Thread(target=increment),
threading.Thread(target=increment),
]
for t in threads:
t.start()
for t in threads:
t.join()
# Final value should be 3
final = locked_json_read(data_file)
assert final["counter"] == 3
class TestLockedReadWrite:
"""Test general locked read/write operations."""
@pytest.fixture
def data_file(self, tmp_path):
"""Create a temporary data file."""
return tmp_path / "data.txt"
def test_locked_write(self, data_file):
"""Test writing with lock."""
from runners.gitlab.utils.file_lock import locked_write
with locked_write(data_file) as f:
f.write("test content")
assert data_file.read_text(encoding="utf-8") == "test content"
def test_locked_read(self, data_file):
"""Test reading with lock."""
from runners.gitlab.utils.file_lock import locked_read, locked_write
with locked_write(data_file) as f:
f.write("read test")
with locked_read(data_file) as f:
content = f.read()
assert content == "read test"
def test_locked_write_file_lock(self, data_file):
"""Test locked_write with custom FileLock."""
from runners.gitlab.utils.file_lock import FileLock, locked_write
with FileLock(data_file, timeout=5.0):
with locked_write(data_file, lock=None) as f:
f.write("custom lock")
assert data_file.read_text(encoding="utf-8") == "custom lock"
class TestFileLockError:
"""Test FileLockError exceptions."""
def test_file_lock_error(self):
"""Test FileLockError is raised correctly."""
from runners.gitlab.utils.file_lock import FileLockError
error = FileLockError("Custom error message")
assert str(error) == "Custom error message"
def test_file_lock_timeout(self):
"""Test FileLockTimeout is raised correctly."""
from runners.gitlab.utils.file_lock import FileLockTimeout
error = FileLockTimeout("Timeout message")
assert "Timeout" in str(error)
class TestConcurrentSafety:
"""Test concurrent safety scenarios."""
def test_multiple_readers(self, tmp_path):
"""Test multiple readers can access file concurrently."""
from runners.gitlab.utils.file_lock import locked_json_read, locked_json_write
data_file = tmp_path / "readers.json"
locked_json_write(data_file, {"value": 42})
results = []
def read_value():
data = locked_json_read(data_file)
results.append(data["value"])
threads = [threading.Thread(target=read_value) for _ in range(5)]
for t in threads:
t.start()
for t in threads:
t.join()
assert len(results) == 5
assert all(r == 42 for r in results)
def test_writers_exclusive(self, tmp_path):
"""Test writers have exclusive access."""
from runners.gitlab.utils.file_lock import (
locked_json_read,
locked_json_update,
locked_json_write,
)
data_file = tmp_path / "writers.json"
locked_json_write(data_file, {"counter": 0})
results = []
def increment():
def updater(data):
data["counter"] += 1
return data
locked_json_update(data_file, updater)
result = locked_json_read(data_file)
results.append(result["counter"])
threads = [threading.Thread(target=increment) for _ in range(10)]
for t in threads:
t.start()
for t in threads:
t.join()
# All increments should be applied
final = locked_json_read(data_file)
assert final["counter"] == 10
assert len(results) == 10
def test_reader_writer_conflict(self, tmp_path):
"""Test readers and writers don't conflict."""
from runners.gitlab.utils.file_lock import (
locked_json_read,
locked_json_update,
locked_json_write,
)
data_file = tmp_path / "rw.json"
locked_json_write(data_file, {"reads": 0, "writes": 0})
read_results = []
def reader():
for _ in range(10):
data = locked_json_read(data_file)
read_results.append(data["reads"])
def writer():
for _ in range(5):
def updater(data):
data["writes"] += 1
return data
locked_json_update(data_file, updater)
threads = [
threading.Thread(target=reader),
threading.Thread(target=writer),
]
for t in threads:
t.start()
for t in threads:
t.join()
# All operations should complete
final = locked_json_read(data_file)
assert final["writes"] == 5
assert len(read_results) == 10
@@ -0,0 +1,281 @@
"""
Tests for GitLab File Operations
===================================
Tests for file content retrieval, creation, updating, and deletion.
"""
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
try:
from runners.gitlab.glab_client import GitLabClient, GitLabConfig
except ImportError:
from glab_client import GitLabClient, GitLabConfig
@pytest.fixture
def mock_config():
"""Create a mock GitLab config."""
return GitLabConfig(
token="test-token",
project="namespace/test-project",
instance_url="https://gitlab.example.com",
)
@pytest.fixture
def client(mock_config, tmp_path):
"""Create a GitLab client instance."""
return GitLabClient(
project_dir=tmp_path,
config=mock_config,
)
class TestGetFileContents:
"""Tests for get_file_contents method."""
@pytest.mark.asyncio
async def test_get_file_contents_current_version(self, client):
"""Test getting file contents from current HEAD."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"file_name": "test.py",
"file_path": "src/test.py",
"size": 100,
"encoding": "base64",
"content": "cHJpbnQoJ2hlbGxvJyk=", # base64 for "print('hello')"
"content_sha256": "abc123",
"ref": "main",
}
result = client.get_file_contents("src/test.py")
assert result["file_name"] == "test.py"
assert result["encoding"] == "base64"
@pytest.mark.asyncio
async def test_get_file_contents_with_ref(self, client):
"""Test getting file contents from specific ref."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"file_name": "config.json",
"ref": "develop",
"content": "eyJjb25maWciOiB0cnVlfQ==",
}
result = client.get_file_contents("config.json", ref="develop")
assert result["ref"] == "develop"
@pytest.mark.asyncio
async def test_get_file_contents_async(self, client):
"""Test async variant of get_file_contents."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"file_name": "test.py",
"content": "dGVzdA==",
}
result = await client.get_file_contents_async("test.py")
assert result["file_name"] == "test.py"
class TestCreateFile:
"""Tests for create_file method."""
@pytest.mark.asyncio
async def test_create_new_file(self, client):
"""Test creating a new file."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"file_path": "new_file.py",
"branch": "main",
"commit_id": "abc123",
}
result = client.create_file(
file_path="new_file.py",
content="print('hello world')",
commit_message="Add new file",
branch="main",
)
assert result["file_path"] == "new_file.py"
@pytest.mark.asyncio
async def test_create_file_with_author(self, client):
"""Test creating a file with author information."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"file_path": "authored.py",
"commit_id": "def456",
}
result = client.create_file(
file_path="authored.py",
content="# Author: John Doe",
commit_message="Add file",
branch="main",
author_name="John Doe",
author_email="john@example.com",
)
assert result["commit_id"] == "def456"
@pytest.mark.asyncio
async def test_create_file_async(self, client):
"""Test async variant of create_file."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {"file_path": "async.py"}
result = await client.create_file_async(
file_path="async.py",
content="content",
commit_message="Add",
branch="main",
)
assert result["file_path"] == "async.py"
class TestUpdateFile:
"""Tests for update_file method."""
@pytest.mark.asyncio
async def test_update_existing_file(self, client):
"""Test updating an existing file."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"file_path": "existing.py",
"branch": "main",
"commit_id": "ghi789",
}
result = client.update_file(
file_path="existing.py",
content="updated content",
commit_message="Update file",
branch="main",
)
assert result["commit_id"] == "ghi789"
@pytest.mark.asyncio
async def test_update_file_with_author(self, client):
"""Test updating file with author info."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"file_path": "update.py",
"commit_id": "jkl012",
}
result = client.update_file(
file_path="update.py",
content="new content",
commit_message="Modify file",
branch="develop",
author_name="Jane Doe",
author_email="jane@example.com",
)
assert result["commit_id"] == "jkl012"
@pytest.mark.asyncio
async def test_update_file_async(self, client):
"""Test async variant of update_file."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {"file_path": "update.py"}
result = await client.update_file_async(
file_path="update.py",
content="new content",
commit_message="Update",
branch="main",
)
assert result["file_path"] == "update.py"
class TestDeleteFile:
"""Tests for delete_file method."""
@pytest.mark.asyncio
async def test_delete_file(self, client):
"""Test deleting a file."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"file_path": "old.py",
"branch": "main",
"commit_id": "mno345",
}
result = client.delete_file(
file_path="old.py",
commit_message="Remove old file",
branch="main",
)
assert result["commit_id"] == "mno345"
@pytest.mark.asyncio
async def test_delete_file_async(self, client):
"""Test async variant of delete_file."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {"file_path": "delete.py"}
result = await client.delete_file_async(
file_path="delete.py",
commit_message="Delete",
branch="main",
)
assert result["file_path"] == "delete.py"
class TestFileOperationErrors:
"""Tests for file operation error handling."""
@pytest.mark.asyncio
async def test_get_nonexistent_file(self, client):
"""Test getting a file that doesn't exist."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.side_effect = Exception("404 File Not Found")
with pytest.raises(Exception): # noqa: B017
client.get_file_contents("nonexistent.py")
@pytest.mark.asyncio
async def test_create_file_already_exists(self, client):
"""Test creating a file that already exists."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.side_effect = Exception("400 File already exists")
with pytest.raises(Exception): # noqa: B017
client.create_file(
file_path="existing.py",
content="content",
commit_message="Add",
branch="main",
)
@pytest.mark.asyncio
async def test_delete_nonexistent_file(self, client):
"""Test deleting a file that doesn't exist."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.side_effect = Exception("404 File Not Found")
with pytest.raises(Exception): # noqa: B017
client.delete_file(
file_path="nonexistent.py",
commit_message="Delete",
branch="main",
)
@@ -0,0 +1,455 @@
"""
Unit Tests for GitLab Follow-up MR Reviewer
============================================
Tests for FollowupReviewer class.
"""
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from runners.gitlab.models import (
AutoFixState,
AutoFixStatus,
MergeVerdict,
MRReviewFinding,
MRReviewResult,
ReviewCategory,
ReviewSeverity,
)
from runners.gitlab.services.followup_reviewer import FollowupReviewer
@pytest.fixture
def mock_client():
"""Create a mock GitLab client."""
client = MagicMock()
client.get_mr_async = AsyncMock()
client.get_mr_notes_async = AsyncMock()
return client
@pytest.fixture
def sample_previous_review():
"""Create a sample previous review result."""
return MRReviewResult(
mr_iid=123,
project="namespace/project",
success=True,
findings=[
MRReviewFinding(
id="finding-1",
severity=ReviewSeverity.HIGH,
category=ReviewCategory.SECURITY,
title="SQL Injection vulnerability",
description="User input not sanitized",
file="src/api/users.py",
line=42,
suggested_fix="Use parameterized queries",
fixable=True,
),
MRReviewFinding(
id="finding-2",
severity=ReviewSeverity.MEDIUM,
category=ReviewCategory.QUALITY,
title="Missing error handling",
description="No try-except around file I/O",
file="src/utils/file.py",
line=15,
suggested_fix="Add error handling",
fixable=True,
),
],
summary="Found 2 issues",
overall_status="request_changes",
verdict=MergeVerdict.NEEDS_REVISION,
verdict_reasoning="High severity issues must be resolved",
reviewed_commit_sha="abc123def456",
reviewed_file_blobs={"src/api/users.py": "blob1", "src/utils/file.py": "blob2"},
)
@pytest.fixture
def reviewer(sample_previous_review):
"""Create a FollowupReviewer instance."""
return FollowupReviewer(
project_dir="/tmp/project",
gitlab_dir="/tmp/project/.auto-claude/gitlab",
config=MagicMock(project="namespace/project"),
progress_callback=None,
use_ai=False,
)
@pytest.mark.asyncio
async def test_review_followup_finding_resolved(
reviewer, mock_client, sample_previous_review
):
"""Test that resolved findings are detected."""
from runners.gitlab.models import FollowupMRContext
# Create context where one finding was resolved
context = FollowupMRContext(
mr_iid=123,
previous_review=sample_previous_review,
previous_commit_sha="abc123def456",
current_commit_sha="def456abc123",
commits_since_review=[
{"id": "commit1", "message": "Fix SQL injection"},
],
files_changed_since_review=["src/api/users.py"],
diff_since_review="diff --git a/src/api/users.py b/src/api/users.py\n"
"@@ -40,7 +40,7 @@\n"
"- query = f\"SELECT * FROM users WHERE name='{name}'\"\n"
'+ query = "SELECT * FROM users WHERE name=%s"\n'
" cursor.execute(query, (name,))",
)
mock_client.get_mr_notes_async.return_value = []
result = await reviewer.review_followup(context, mock_client)
assert result.mr_iid == 123
assert len(result.resolved_findings) > 0
assert len(result.unresolved_findings) < 2 # At least one resolved
@pytest.mark.asyncio
async def test_review_followup_finding_unresolved(
reviewer, mock_client, sample_previous_review
):
"""Test that unresolved findings are tracked."""
from runners.gitlab.models import FollowupMRContext
# Create context where findings were not addressed
context = FollowupMRContext(
mr_iid=123,
previous_review=sample_previous_review,
previous_commit_sha="abc123def456",
current_commit_sha="def456abc123",
commits_since_review=[
{"id": "commit1", "message": "Update docs"},
],
files_changed_since_review=["README.md"],
diff_since_review="diff --git a/README.md b/README.md\n+ # Updated docs",
)
mock_client.get_mr_notes_async.return_value = []
result = await reviewer.review_followup(context, mock_client)
assert result.mr_iid == 123
assert len(result.unresolved_findings) == 2 # Both still unresolved
@pytest.mark.asyncio
async def test_review_followup_new_findings(
reviewer, mock_client, sample_previous_review
):
"""Test that new issues are detected."""
from runners.gitlab.models import FollowupMRContext
# Create context with TODO comment in diff
context = FollowupMRContext(
mr_iid=123,
previous_review=sample_previous_review,
previous_commit_sha="abc123def456",
current_commit_sha="def456abc123",
commits_since_review=[
{"id": "commit1", "message": "Add feature"},
],
files_changed_since_review=["src/feature.py"],
diff_since_review="diff --git a/src/feature.py b/src/feature.py\n"
"--- a/src/feature.py\n"
"+++ b/src/feature.py\n"
"@@ -0,0 +1,3 @@\n"
"+ # TODO: implement error handling\n"
"+ def feature():\n"
"+ pass",
)
mock_client.get_mr_notes_async.return_value = []
result = await reviewer.review_followup(context, mock_client)
# Should detect TODO as new finding
assert any(
f.id.startswith("followup-todo-") and "todo" in f.title.lower()
for f in result.findings
)
@pytest.mark.asyncio
async def test_determine_verdict_critical_blocks(reviewer, sample_previous_review):
"""Test that critical issues block merge."""
new_findings = [
MRReviewFinding(
id="new-1",
severity=ReviewSeverity.CRITICAL,
category=ReviewCategory.SECURITY,
title="Critical security issue",
description="Must fix",
file="src/file.py",
line=1,
)
]
verdict = reviewer._determine_verdict(
unresolved=[],
new_findings=new_findings,
mr_iid=123,
)
assert verdict == MergeVerdict.BLOCKED
@pytest.mark.asyncio
async def test_determine_verdict_high_needs_revision(reviewer, sample_previous_review):
"""Test that high issues require revision."""
new_findings = [
MRReviewFinding(
id="new-1",
severity=ReviewSeverity.HIGH,
category=ReviewCategory.SECURITY,
title="High severity issue",
description="Should fix",
file="src/file.py",
line=1,
)
]
verdict = reviewer._determine_verdict(
unresolved=[],
new_findings=new_findings,
mr_iid=123,
)
assert verdict == MergeVerdict.NEEDS_REVISION
@pytest.mark.asyncio
async def test_determine_verdict_medium_merge_with_changes(
reviewer, sample_previous_review
):
"""Test that medium issues suggest merge with changes."""
new_findings = [
MRReviewFinding(
id="new-1",
severity=ReviewSeverity.MEDIUM,
category=ReviewCategory.QUALITY,
title="Medium issue",
description="Nice to fix",
file="src/file.py",
line=1,
)
]
verdict = reviewer._determine_verdict(
unresolved=[],
new_findings=new_findings,
mr_iid=123,
)
assert verdict == MergeVerdict.MERGE_WITH_CHANGES
@pytest.mark.asyncio
async def test_determine_verdict_ready_to_merge(reviewer, sample_previous_review):
"""Test that low or no issues allow merge."""
new_findings = [
MRReviewFinding(
id="new-1",
severity=ReviewSeverity.LOW,
category=ReviewCategory.STYLE,
title="Style issue",
description="Optional fix",
file="src/file.py",
line=1,
)
]
verdict = reviewer._determine_verdict(
unresolved=[],
new_findings=new_findings,
mr_iid=123,
)
assert verdict == MergeVerdict.READY_TO_MERGE
@pytest.mark.asyncio
async def test_determine_verdict_all_clear(reviewer, sample_previous_review):
"""Test that no issues allows merge."""
verdict = reviewer._determine_verdict(
unresolved=[],
new_findings=[],
mr_iid=123,
)
assert verdict == MergeVerdict.READY_TO_MERGE
def test_is_finding_addressed_file_changed(reviewer, sample_previous_review):
"""Test finding detection when file is changed in the diff region."""
diff = (
"diff --git a/src/api/users.py b/src/api/users.py\n"
"@@ -40,7 +40,7 @@\n"
"- query = f\"SELECT * FROM users WHERE name='{name}'\"\n"
'+ query = "SELECT * FROM users WHERE name=%s"\n'
" cursor.execute(query, (name,))"
)
finding = sample_previous_review.findings[0] # Line 42 in users.py
result = reviewer._is_finding_addressed(diff, finding)
assert result is True # Line 42 is in the changed range (40-47)
def test_is_finding_addressed_file_not_changed(reviewer, sample_previous_review):
"""Test finding detection when file is not in diff."""
diff = "diff --git a/README.md b/README.md\n+ # Updated docs"
finding = sample_previous_review.findings[0] # users.py
result = reviewer._is_finding_addressed(diff, finding)
assert result is False
def test_is_finding_addressed_line_not_in_range(reviewer, sample_previous_review):
"""Test finding detection when line is outside changed range."""
diff = (
"diff --git a/src/api/users.py b/src/api/users.py\n"
"@@ -1,7 +1,7 @@\n"
" def hello():\n"
"- print('hello')\n"
"+ print('HELLO')\n"
)
finding = sample_previous_review.findings[0] # Line 42, not in range 1-8
result = reviewer._is_finding_addressed(diff, finding)
assert result is False
def test_is_finding_addressed_test_pattern_added(reviewer, sample_previous_review):
"""Test finding detection for test category when tests are added."""
diff = (
"diff --git a/tests/test_users.py b/tests/test_users.py\n"
"+ def test_sql_injection():\n"
"+ assert True"
)
test_finding = MRReviewFinding(
id="test-1",
severity=ReviewSeverity.MEDIUM,
category=ReviewCategory.TEST,
title="Missing tests",
description="Add tests for users module",
file="tests/test_users.py",
line=1,
)
result = reviewer._is_finding_addressed(diff, test_finding)
assert result is True # Pattern matches "+ def test_"
def test_is_finding_addressed_doc_pattern_added(reviewer, sample_previous_review):
"""Test finding detection for documentation category when docs are added."""
diff = (
"diff --git a/src/api/users.py b/src/api/users.py\n"
'+ """\n'
"+ User API module.\n"
'+ """'
)
doc_finding = MRReviewFinding(
id="doc-1",
severity=ReviewSeverity.LOW,
category=ReviewCategory.DOCS,
title="Missing docstring",
description="Add module docstring",
file="src/api/users.py",
line=1,
)
result = reviewer._is_finding_addressed(diff, doc_finding)
assert result is True # Pattern matches '+"""'
@pytest.mark.asyncio
async def test_review_comment_question_detection(
reviewer, mock_client, sample_previous_review
):
"""Test that questions in comments are detected."""
from runners.gitlab.models import FollowupMRContext
context = FollowupMRContext(
mr_iid=123,
previous_review=sample_previous_review,
previous_commit_sha="abc123def456",
current_commit_sha="def456abc123",
commits_since_review=[{"id": "commit1"}],
files_changed_since_review=[],
diff_since_review="",
)
mock_client.get_mr_notes_async.return_value = [
{
"id": 1,
"commit_id": "commit1",
"author": {"username": "contributor"},
"body": "Should we add error handling here?",
"created_at": "2024-01-01T00:00:00Z",
},
]
result = await reviewer.review_followup(context, mock_client)
# Should detect the question
assert any("question" in f.title.lower() for f in result.findings)
@pytest.mark.asyncio
async def test_review_comment_filters_by_commit(
reviewer, mock_client, sample_previous_review
):
"""Test that only comments from new commits are reviewed."""
from runners.gitlab.models import FollowupMRContext
context = FollowupMRContext(
mr_iid=123,
previous_review=sample_previous_review,
previous_commit_sha="abc123def456",
current_commit_sha="def456abc123",
commits_since_review=[{"id": "commit1"}],
files_changed_since_review=[],
diff_since_review="",
)
mock_client.get_mr_notes_async.return_value = [
{
"id": 1,
"commit_id": "commit1", # New commit
"author": {"username": "contributor"},
"body": "Should we add error handling?",
"created_at": "2024-01-01T00:00:00Z",
},
{
"id": 2,
"commit_id": "old-commit", # Old commit, should be ignored
"author": {"username": "contributor"},
"body": "Another question?",
"created_at": "2024-01-01T00:00:00Z",
},
]
result = await reviewer.review_followup(context, mock_client)
# Should only have one finding from the new commit
question_findings = [f for f in result.findings if "question" in f.title.lower()]
assert len(question_findings) == 1
@@ -0,0 +1,566 @@
"""
GitLab MR E2E Tests
===================
End-to-end tests for MR review lifecycle.
"""
import json
from datetime import datetime, timezone
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, Mock, patch
import pytest
from __tests__.fixtures.gitlab import (
MOCK_GITLAB_CONFIG,
mock_mr_changes,
mock_mr_commits,
mock_mr_data,
mock_pipeline_data,
mock_pipeline_jobs,
)
class TestMREndToEnd:
"""End-to-end MR review lifecycle tests."""
@pytest.fixture
def mock_orchestrator(self, tmp_path):
"""Create a mock orchestrator for testing."""
from runners.gitlab.models import GitLabRunnerConfig
from runners.gitlab.orchestrator import GitLabOrchestrator
config = GitLabRunnerConfig(
token="test-token",
project="group/project",
instance_url="https://gitlab.example.com",
model="claude-sonnet-4-20250514",
)
with patch("runners.gitlab.orchestrator.GitLabClient"):
orchestrator = GitLabOrchestrator(
project_dir=tmp_path,
config=config,
enable_bot_detection=False,
enable_ci_checking=False,
)
return orchestrator
@pytest.mark.asyncio
async def test_full_mr_review_lifecycle(self, mock_orchestrator):
"""Test complete MR review from start to finish."""
# Mock MR data
mock_orchestrator.client.get_mr_async.return_value = mock_mr_data()
mock_orchestrator.client.get_mr_commits_async.return_value = mock_mr_commits()
mock_orchestrator.client.get_mr_changes_async.return_value = mock_mr_changes()
# Mock review engine
with patch(
"runners.gitlab.services.context_gatherer.MRContextGatherer"
) as mock_gatherer:
from runners.gitlab.models import (
MergeVerdict,
MRContext,
MRReviewFinding,
ReviewCategory,
ReviewSeverity,
)
mock_gatherer.return_value.gather.return_value = MRContext(
mr_iid=123,
title="Add feature",
description="Implementation",
author="john_doe",
source_branch="feature",
target_branch="main",
state="opened",
changed_files=[],
diff="",
commits=[],
)
# Mock review engine to return findings
with patch("runners.gitlab.services.MRReviewEngine") as mock_engine:
findings = [
MRReviewFinding(
id="find-1",
severity=ReviewSeverity.MEDIUM,
category=ReviewCategory.QUALITY,
title="Code style",
description="Fix formatting",
file="file.py",
line=10,
)
]
mock_engine.return_value.run_review.return_value = (
findings,
MergeVerdict.MERGE_WITH_CHANGES,
"Consider the suggestions",
[],
)
result = await mock_orchestrator.review_mr(123)
assert result.success is True
assert result.mr_iid == 123
assert len(result.findings) == 1
assert result.verdict == MergeVerdict.MERGE_WITH_CHANGES
@pytest.mark.asyncio
async def test_mr_review_with_ci_failure(self, mock_orchestrator):
"""Test MR review blocked by CI failure."""
from runners.gitlab.services.ci_checker import PipelineInfo, PipelineStatus
# Setup CI failure
with patch("runners.gitlab.orchestrator.MRContextGatherer"):
with patch("runners.gitlab.services.ci_checker.CIChecker") as mock_checker:
pipeline_info = PipelineInfo(
pipeline_id=1001,
status=PipelineStatus.FAILED,
ref="feature",
sha="abc123",
created_at="2025-01-14T10:00:00",
updated_at="2025-01-14T10:05:00",
failed_jobs=[
Mock(
status="failed",
name="test",
stage="test",
failure_reason="Assert failed",
)
],
)
mock_checker.return_value.check_mr_pipeline.return_value = pipeline_info
mock_checker.return_value.get_blocking_reason.return_value = (
"Test job failed"
)
mock_checker.return_value.format_pipeline_summary.return_value = (
"CI Failed"
)
mock_orchestrator.client.get_mr_async.return_value = mock_mr_data()
mock_orchestrator.client.get_mr_commits_async.return_value = []
with patch("runners.gitlab.services.MRReviewEngine") as mock_engine:
from runners.gitlab.models import MergeVerdict
mock_engine.return_value.run_review.return_value = (
[],
MergeVerdict.READY_TO_MERGE,
"Looks good",
[],
)
result = await mock_orchestrator.review_mr(123)
assert result.ci_status == "failed"
assert result.ci_pipeline_id == 1001
assert "CI" in result.summary
@pytest.mark.asyncio
async def test_followup_review_lifecycle(self, mock_orchestrator):
"""Test follow-up review after initial review."""
from runners.gitlab.models import MergeVerdict, MRReviewResult
# Create initial review
initial_review = MRReviewResult(
mr_iid=123,
project="group/project",
success=True,
findings=[
Mock(id="find-1", title="Fix bug"),
Mock(id="find-2", title="Add tests"),
],
reviewed_commit_sha="abc123",
verdict=MergeVerdict.NEEDS_REVISION,
verdict_reasoning="Issues found",
blockers=["find-1"],
)
# Save initial review
initial_review.save(mock_orchestrator.gitlab_dir)
# Mock new commits
new_commits = mock_mr_commits() + [
{
"id": "new456",
"sha": "new456",
"message": "Fix the issues",
}
]
mock_orchestrator.client.get_mr_async.return_value = mock_mr_data()
mock_orchestrator.client.get_mr_commits_async.return_value = new_commits
# Mock follow-up review
with patch("runners.gitlab.orchestrator.MRContextGatherer"):
with patch("runners.gitlab.services.MRReviewEngine") as mock_engine:
mock_engine.return_value.run_review.return_value = (
[], # No new findings
MergeVerdict.READY_TO_MERGE,
"All fixed",
[],
)
result = await mock_orchestrator.followup_review_mr(123)
assert result.is_followup_review is True
assert result.reviewed_commit_sha == "new456"
@pytest.mark.asyncio
async def test_bot_detection_skips_review(self, tmp_path):
"""Test bot detection skips bot-authored MRs."""
from runners.gitlab.models import GitLabRunnerConfig
from runners.gitlab.orchestrator import GitLabOrchestrator
config = GitLabRunnerConfig(
token="test-token",
project="group/project",
)
with patch("runners.gitlab.orchestrator.GitLabClient"):
orchestrator = GitLabOrchestrator(
project_dir=tmp_path,
config=config,
bot_username="auto-claude-bot",
)
# Bot-authored MR
bot_mr = mock_mr_data(author="auto-claude-bot")
orchestrator.client.get_mr_async.return_value = bot_mr
orchestrator.client.get_mr_commits_async.return_value = []
result = await orchestrator.review_mr(123)
assert result.success is False
assert "bot" in result.error.lower()
@pytest.mark.asyncio
async def test_cooling_off_prevents_re_review(self, tmp_path):
"""Test cooling off period prevents immediate re-review."""
from runners.gitlab.models import GitLabRunnerConfig
from runners.gitlab.orchestrator import GitLabOrchestrator
config = GitLabRunnerConfig(
token="test-token",
project="group/project",
)
with patch("runners.gitlab.orchestrator.GitLabClient"):
orchestrator = GitLabOrchestrator(
project_dir=tmp_path,
config=config,
)
# First review
orchestrator.client.get_mr_async.return_value = mock_mr_data()
orchestrator.client.get_mr_commits_async.return_value = mock_mr_commits()
with patch("runners.gitlab.orchestrator.MRContextGatherer"):
with patch("runners.gitlab.services.MRReviewEngine") as mock_engine:
from runners.gitlab.models import MergeVerdict
mock_engine.return_value.run_review.return_value = (
[],
MergeVerdict.READY_TO_MERGE,
"Good",
[],
)
result1 = await orchestrator.review_mr(123)
assert result1.success is True
# Immediate second review should be skipped
result2 = await orchestrator.review_mr(123)
assert result2.success is False
assert "cooling" in result2.error.lower()
class TestMRReviewEngineIntegration:
"""Test MR review engine integration."""
@pytest.fixture
def engine(self, tmp_path):
"""Create review engine for testing."""
from runners.gitlab.models import GitLabRunnerConfig
from runners.gitlab.services.mr_review_engine import MRReviewEngine
config = GitLabRunnerConfig(
token="test-token",
project="group/project",
)
gitlab_dir = tmp_path / ".auto-claude" / "gitlab"
gitlab_dir.mkdir(parents=True, exist_ok=True)
return MRReviewEngine(
project_dir=tmp_path,
gitlab_dir=gitlab_dir,
config=config,
)
def test_engine_initialization(self, engine):
"""Test engine initializes correctly."""
assert engine.project_dir
assert engine.gitlab_dir
assert engine.config
def test_generate_summary(self, engine):
"""Test summary generation."""
from runners.gitlab.models import (
MergeVerdict,
MRReviewFinding,
ReviewCategory,
ReviewSeverity,
)
findings = [
MRReviewFinding(
id="find-1",
severity=ReviewSeverity.CRITICAL,
category=ReviewCategory.SECURITY,
title="SQL injection",
description="Vulnerability",
file="file.py",
line=10,
),
MRReviewFinding(
id="find-2",
severity=ReviewSeverity.LOW,
category=ReviewCategory.STYLE,
title="Formatting",
description="Style issue",
file="file.py",
line=20,
),
]
summary = engine.generate_summary(
findings=findings,
verdict=MergeVerdict.BLOCKED,
verdict_reasoning="Critical security issue",
blockers=["SQL injection"],
)
assert "BLOCKED" in summary
assert "SQL injection" in summary
assert "Critical" in summary
class TestMRContextGatherer:
"""Test MR context gatherer."""
@pytest.fixture
def gatherer(self, tmp_path):
"""Create context gatherer for testing."""
from runners.gitlab.glab_client import GitLabConfig
from runners.gitlab.services.context_gatherer import MRContextGatherer
config = GitLabConfig(
token="test-token",
project="group/project",
instance_url="https://gitlab.example.com",
)
with patch("runners.gitlab.services.context_gatherer.GitLabClient"):
return MRContextGatherer(
project_dir=tmp_path,
mr_iid=123,
config=config,
)
@pytest.mark.asyncio
async def test_gather_context(self, gatherer):
"""Test gathering MR context."""
from runners.gitlab.models import MRContext
# Mock client responses
gatherer.client.get_mr_async.return_value = mock_mr_data()
gatherer.client.get_mr_changes_async.return_value = mock_mr_changes()
gatherer.client.get_mr_commits_async.return_value = mock_mr_commits()
gatherer.client.get_mr_notes_async.return_value = []
context = await gatherer.gather()
assert isinstance(context, MRContext)
assert context.mr_iid == 123
assert context.title == "Add user authentication feature"
assert context.author == "john_doe"
@pytest.mark.asyncio
async def test_gather_ai_bot_comments(self, gatherer):
"""Test gathering AI bot comments."""
# Mock AI bot comments
ai_notes = [
{
"id": 1001,
"author": {"username": "coderabbit[bot]"},
"body": "Consider adding error handling",
"created_at": "2025-01-14T10:00:00",
},
{
"id": 1002,
"author": {"username": "human_user"},
"body": "Regular comment",
"created_at": "2025-01-14T11:00:00",
},
]
gatherer.client.get_mr_notes_async.return_value = ai_notes
# First call should parse comments
from runners.gitlab.services.context_gatherer import AIBotComment
# Note: _fetch_ai_bot_comments is called internally during gather()
gatherer.client.get_mr_async.return_value = mock_mr_data()
gatherer.client.get_mr_changes_async.return_value = mock_mr_changes()
gatherer.client.get_mr_commits_async.return_value = mock_mr_commits()
context = await gatherer.gather()
# Verify AI bot comments were detected (context would have them if implemented)
assert context.mr_iid == 123
class TestFollowupContextGatherer:
"""Test follow-up context gatherer."""
@pytest.fixture
def previous_review(self):
"""Create a previous review for testing."""
from runners.gitlab.models import MergeVerdict, MRReviewResult
return MRReviewResult(
mr_iid=123,
project="group/project",
success=True,
findings=[
Mock(id="find-1", title="Bug"),
],
reviewed_commit_sha="abc123",
verdict=MergeVerdict.NEEDS_REVISION,
verdict_reasoning="Issues found",
blockers=[],
)
@pytest.fixture
def gatherer(self, tmp_path, previous_review):
"""Create follow-up context gatherer."""
from runners.gitlab.glab_client import GitLabConfig
from runners.gitlab.services.context_gatherer import FollowupMRContextGatherer
config = GitLabConfig(
token="test-token",
project="group/project",
instance_url="https://gitlab.example.com",
)
with patch("runners.gitlab.services.context_gatherer.GitLabClient"):
return FollowupMRContextGatherer(
project_dir=tmp_path,
mr_iid=123,
previous_review=previous_review,
config=config,
)
@pytest.mark.asyncio
async def test_gather_followup_context(self, gatherer):
"""Test gathering follow-up context."""
from runners.gitlab.models import FollowupMRContext
# Mock new commits since previous review
new_commits = [
{
"id": "new456",
"sha": "new456",
"message": "Fix bug",
}
]
gatherer.client.get_mr_async.return_value = mock_mr_data()
gatherer.client.get_mr_commits_async.return_value = new_commits
gatherer.client.get_mr_changes_async.return_value = mock_mr_changes()
context = await gatherer.gather()
assert isinstance(context, FollowupMRContext)
assert context.mr_iid == 123
assert context.previous_commit_sha == "abc123"
assert context.current_commit_sha == "new456"
assert len(context.commits_since_review) == 1
@pytest.mark.asyncio
async def test_no_new_commits(self, gatherer):
"""Test follow-up when no new commits."""
from runners.gitlab.models import FollowupMRContext
# Same commits as previous review
gatherer.client.get_mr_async.return_value = mock_mr_data()
gatherer.client.get_mr_commits_async.return_value = mock_mr_commits()
gatherer.client.get_mr_changes_async.return_value = mock_mr_changes()
context = await gatherer.gather()
assert context.current_commit_sha == "abc123" # Same as previous
class TestAIBotComment:
"""Test AI bot comment detection."""
def test_parse_coderabbit_comment(self):
"""Test parsing CodeRabbit comment."""
from runners.gitlab.services.context_gatherer import AIBotComment
note = {
"id": 1001,
"author": {"username": "coderabbit[bot]"},
"body": "Add error handling",
"created_at": "2025-01-14T10:00:00",
}
from runners.gitlab.services.context_gatherer import MRContextGatherer
gatherer_class = MRContextGatherer.__class__
comment = gatherer_class._parse_ai_comment(None, note)
assert comment is not None
assert comment.tool_name == "CodeRabbit"
assert comment.comment_id == 1001
def test_parse_human_comment(self):
"""Test human comment is not detected as AI."""
from runners.gitlab.services.context_gatherer import MRContextGatherer
note = {
"id": 1002,
"author": {"username": "john_doe"},
"body": "Regular comment",
"created_at": "2025-01-14T10:00:00",
}
comment = MRContextGatherer._parse_ai_comment(None, note)
assert comment is None
def test_parse_greptile_comment(self):
"""Test parsing Greptile comment."""
from runners.gitlab.services.context_gatherer import AIBotComment
note = {
"id": 1003,
"author": {"username": "greptile[bot]"},
"body": "Consider this",
"created_at": "2025-01-14T10:00:00",
}
from runners.gitlab.services.context_gatherer import MRContextGatherer
comment = MRContextGatherer._parse_ai_comment(None, note)
assert comment is not None
assert comment.tool_name == "Greptile"
@@ -0,0 +1,514 @@
"""
GitLab MR Review Tests
======================
Tests for MR review models, findings, verdicts.
"""
import json
from datetime import datetime, timezone
from pathlib import Path
from unittest.mock import MagicMock, Mock, patch
import pytest
from __tests__.fixtures.gitlab import (
MOCK_GITLAB_CONFIG,
mock_issue_data,
mock_mr_data,
)
class TestMRReviewFinding:
"""Test MRReviewFinding model."""
def test_finding_creation(self):
"""Test creating a review finding."""
from runners.gitlab.models import (
MRReviewFinding,
ReviewCategory,
ReviewSeverity,
)
finding = MRReviewFinding(
id="find-1",
severity=ReviewSeverity.HIGH,
category=ReviewCategory.SECURITY,
title="SQL injection vulnerability",
description="User input not sanitized in query",
file="src/auth.py",
line=42,
end_line=45,
suggested_fix="Use parameterized query",
fixable=True,
)
assert finding.id == "find-1"
assert finding.severity == ReviewSeverity.HIGH
assert finding.category == ReviewCategory.SECURITY
assert finding.file == "src/auth.py"
assert finding.line == 42
assert finding.fixable is True
def test_finding_to_dict(self):
"""Test converting finding to dictionary."""
from runners.gitlab.models import (
MRReviewFinding,
ReviewCategory,
ReviewSeverity,
)
finding = MRReviewFinding(
id="find-1",
severity=ReviewSeverity.HIGH,
category=ReviewCategory.SECURITY,
title="SQL injection",
description="Vulnerability",
file="src/auth.py",
line=42,
)
data = finding.to_dict()
assert data["id"] == "find-1"
assert data["severity"] == "high"
assert data["category"] == "security"
def test_finding_from_dict(self):
"""Test loading finding from dictionary."""
from runners.gitlab.models import MRReviewFinding
data = {
"id": "find-1",
"severity": "high",
"category": "security",
"title": "SQL injection",
"description": "Vulnerability",
"file": "src/auth.py",
"line": 42,
"end_line": 45,
"suggested_fix": "Fix it",
"fixable": True,
}
finding = MRReviewFinding.from_dict(data)
assert finding.id == "find-1"
assert finding.severity.value == "high"
assert finding.line == 42
def test_finding_with_evidence_code(self):
"""Test finding with evidence code."""
from runners.gitlab.models import (
MRReviewFinding,
ReviewCategory,
ReviewPass,
ReviewSeverity,
)
finding = MRReviewFinding(
id="find-1",
severity=ReviewSeverity.CRITICAL,
category=ReviewCategory.SECURITY,
title="Command injection",
description="User input in subprocess",
file="src/exec.py",
line=10,
evidence_code="subprocess.call(user_input, shell=True)",
found_by_pass=ReviewPass.SECURITY,
)
assert finding.evidence_code == "subprocess.call(user_input, shell=True)"
assert finding.found_by_pass == ReviewPass.SECURITY
class TestStructuralIssue:
"""Test StructuralIssue model."""
def test_structural_issue_creation(self):
"""Test creating a structural issue."""
from runners.gitlab.models import ReviewSeverity, StructuralIssue
issue = StructuralIssue(
id="struct-1",
type="feature_creep",
title="Additional features added",
description="MR includes features beyond original scope",
severity=ReviewSeverity.MEDIUM,
files_affected=["src/auth.py", "src/users.py"],
)
assert issue.id == "struct-1"
assert issue.type == "feature_creep"
assert issue.files_affected == ["src/auth.py", "src/users.py"]
def test_structural_issue_to_dict(self):
"""Test converting structural issue to dictionary."""
from runners.gitlab.models import StructuralIssue
issue = StructuralIssue(
id="struct-1",
type="scope_change",
title="Scope increased",
description="MR scope changed significantly",
files_affected=["file1.py"],
)
data = issue.to_dict()
assert data["id"] == "struct-1"
assert data["type"] == "scope_change"
def test_structural_issue_from_dict(self):
"""Test loading structural issue from dictionary."""
from runners.gitlab.models import StructuralIssue
data = {
"id": "struct-1",
"type": "feature_creep",
"title": "Extra features",
"description": "Beyond scope",
"severity": "medium",
"files_affected": ["file.py"],
}
issue = StructuralIssue.from_dict(data)
assert issue.type == "feature_creep"
class TestAICommentTriage:
"""Test AICommentTriage model."""
def test_triage_creation(self):
"""Test creating AI comment triage."""
from runners.gitlab.models import AICommentTriage
triage = AICommentTriage(
comment_id=1001,
tool_name="CodeRabbit",
original_comment="Consider adding error handling",
triage_result="valid",
reasoning="Good point about error handling",
file="src/auth.py",
line=50,
created_at="2025-01-14T10:00:00",
)
assert triage.comment_id == 1001
assert triage.tool_name == "CodeRabbit"
assert triage.triage_result == "valid"
def test_triage_to_dict(self):
"""Test converting triage to dictionary."""
from runners.gitlab.models import AICommentTriage
triage = AICommentTriage(
comment_id=1001,
tool_name="CodeRabbit",
original_comment="Add tests",
triage_result="false_positive",
reasoning="Tests already exist",
)
data = triage.to_dict()
assert data["comment_id"] == 1001
assert data["triage_result"] == "false_positive"
def test_triage_from_dict(self):
"""Test loading triage from dictionary."""
from runners.gitlab.models import AICommentTriage
data = {
"comment_id": 1001,
"tool_name": "Cursor",
"original_comment": "Fix bug",
"triage_result": "questionable",
"reasoning": "Unclear if bug exists",
"file": "file.py",
"line": 10,
}
triage = AICommentTriage.from_dict(data)
assert triage.tool_name == "Cursor"
assert triage.triage_result == "questionable"
class TestMRReviewResult:
"""Test MRReviewResult model."""
def test_result_creation(self):
"""Test creating review result."""
from runners.gitlab.models import (
MergeVerdict,
MRReviewFinding,
MRReviewResult,
ReviewCategory,
ReviewSeverity,
)
findings = [
MRReviewFinding(
id="find-1",
severity=ReviewSeverity.HIGH,
category=ReviewCategory.SECURITY,
title="Bug",
description="Issue",
file="file.py",
line=1,
)
]
result = MRReviewResult(
mr_iid=123,
project="group/project",
success=True,
findings=findings,
summary="Review complete",
overall_status="approve",
verdict=MergeVerdict.READY_TO_MERGE,
verdict_reasoning="No issues found",
blockers=[],
)
assert result.mr_iid == 123
assert result.findings == findings
assert result.verdict == MergeVerdict.READY_TO_MERGE
def test_result_with_structural_issues(self):
"""Test result with structural issues."""
from runners.gitlab.models import (
MergeVerdict,
MRReviewResult,
StructuralIssue,
)
structural_issues = [
StructuralIssue(
id="struct-1",
type="feature_creep",
title="Extra features",
description="Beyond scope",
)
]
result = MRReviewResult(
mr_iid=123,
project="group/project",
success=True,
structural_issues=structural_issues,
verdict=MergeVerdict.MERGE_WITH_CHANGES,
verdict_reasoning="Feature creep detected",
blockers=[],
)
assert len(result.structural_issues) == 1
assert result.verdict == MergeVerdict.MERGE_WITH_CHANGES
def test_result_with_ai_triages(self):
"""Test result with AI comment triages."""
from runners.gitlab.models import (
AICommentTriage,
MergeVerdict,
MRReviewResult,
)
ai_triages = [
AICommentTriage(
comment_id=1001,
tool_name="CodeRabbit",
original_comment="Fix bug",
triage_result="valid",
reasoning="Correct",
)
]
result = MRReviewResult(
mr_iid=123,
project="group/project",
success=True,
ai_triages=ai_triages,
verdict=MergeVerdict.READY_TO_MERGE,
verdict_reasoning="All good",
blockers=[],
)
assert len(result.ai_triages) == 1
def test_result_with_ci_status(self):
"""Test result with CI/CD status."""
from runners.gitlab.models import MergeVerdict, MRReviewResult
result = MRReviewResult(
mr_iid=123,
project="group/project",
success=True,
ci_status="failed",
ci_pipeline_id=1001,
verdict=MergeVerdict.BLOCKED,
verdict_reasoning="CI failed",
blockers=["CI Pipeline Failed"],
)
assert result.ci_status == "failed"
assert result.ci_pipeline_id == 1001
assert result.verdict == MergeVerdict.BLOCKED
def test_result_to_dict(self):
"""Test converting result to dictionary."""
from runners.gitlab.models import MergeVerdict, MRReviewResult
result = MRReviewResult(
mr_iid=123,
project="group/project",
success=True,
verdict=MergeVerdict.READY_TO_MERGE,
verdict_reasoning="Good",
blockers=[],
)
data = result.to_dict()
assert data["mr_iid"] == 123
assert data["verdict"] == "ready_to_merge"
def test_result_from_dict(self):
"""Test loading result from dictionary."""
from runners.gitlab.models import MergeVerdict, MRReviewResult
data = {
"mr_iid": 123,
"project": "group/project",
"success": True,
"findings": [],
"summary": "Review",
"overall_status": "approve",
"verdict": "ready_to_merge",
"verdict_reasoning": "Good",
"blockers": [],
}
result = MRReviewResult.from_dict(data)
assert result.mr_iid == 123
assert result.verdict == MergeVerdict.READY_TO_MERGE
def test_result_save_and_load(self, tmp_path):
"""Test saving and loading result from disk."""
from runners.gitlab.models import MergeVerdict, MRReviewResult
result = MRReviewResult(
mr_iid=123,
project="group/project",
success=True,
verdict=MergeVerdict.READY_TO_MERGE,
verdict_reasoning="Good",
blockers=[],
)
result.save(tmp_path)
loaded = MRReviewResult.load(tmp_path, 123)
assert loaded is not None
assert loaded.mr_iid == 123
def test_followup_review_fields(self):
"""Test follow-up review fields."""
from runners.gitlab.models import MergeVerdict, MRReviewResult
result = MRReviewResult(
mr_iid=123,
project="group/project",
success=True,
is_followup_review=True,
reviewed_commit_sha="abc123",
resolved_findings=["find-1"],
unresolved_findings=["find-2"],
new_findings_since_last_review=["find-3"],
verdict=MergeVerdict.READY_TO_MERGE,
verdict_reasoning="Good",
blockers=[],
)
assert result.is_followup_review is True
assert result.reviewed_commit_sha == "abc123"
assert len(result.resolved_findings) == 1
class TestReviewPass:
"""Test ReviewPass enum."""
def test_all_passes_defined(self):
"""Test all review passes are defined."""
from runners.gitlab.models import ReviewPass
assert ReviewPass.QUICK_SCAN
assert ReviewPass.SECURITY
assert ReviewPass.QUALITY
assert ReviewPass.DEEP_ANALYSIS
assert ReviewPass.STRUCTURAL
assert ReviewPass.AI_COMMENT_TRIAGE
def test_pass_values(self):
"""Test pass enum values."""
from runners.gitlab.models import ReviewPass
assert ReviewPass.QUICK_SCAN.value == "quick_scan"
assert ReviewPass.SECURITY.value == "security"
assert ReviewPass.QUALITY.value == "quality"
assert ReviewPass.DEEP_ANALYSIS.value == "deep_analysis"
assert ReviewPass.STRUCTURAL.value == "structural"
assert ReviewPass.AI_COMMENT_TRIAGE.value == "ai_comment_triage"
class TestMergeVerdict:
"""Test MergeVerdict enum."""
def test_all_verdicts_defined(self):
"""Test all verdicts are defined."""
from runners.gitlab.models import MergeVerdict
assert MergeVerdict.READY_TO_MERGE
assert MergeVerdict.MERGE_WITH_CHANGES
assert MergeVerdict.NEEDS_REVISION
assert MergeVerdict.BLOCKED
def test_verdict_values(self):
"""Test verdict enum values."""
from runners.gitlab.models import MergeVerdict
assert MergeVerdict.READY_TO_MERGE.value == "ready_to_merge"
assert MergeVerdict.MERGE_WITH_CHANGES.value == "merge_with_changes"
assert MergeVerdict.NEEDS_REVISION.value == "needs_revision"
assert MergeVerdict.BLOCKED.value == "blocked"
class TestReviewSeverity:
"""Test ReviewSeverity enum."""
def test_all_severities(self):
"""Test all severity levels."""
from runners.gitlab.models import ReviewSeverity
assert ReviewSeverity.CRITICAL
assert ReviewSeverity.HIGH
assert ReviewSeverity.MEDIUM
assert ReviewSeverity.LOW
class TestReviewCategory:
"""Test ReviewCategory enum."""
def test_all_categories(self):
"""Test all categories."""
from runners.gitlab.models import ReviewCategory
assert ReviewCategory.SECURITY
assert ReviewCategory.QUALITY
assert ReviewCategory.STYLE
assert ReviewCategory.TEST
assert ReviewCategory.DOCS
assert ReviewCategory.PATTERN
assert ReviewCategory.PERFORMANCE
@@ -0,0 +1,381 @@
"""
Unit Tests for GitLab Permission System
========================================
Tests for GitLabPermissionChecker and permission verification.
"""
import logging
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from runners.gitlab.permissions import (
GitLabPermissionChecker,
GitLabRole,
PermissionCheckResult,
)
from runners.gitlab.permissions import PermissionError as GitLabPermissionError
class MockGitLabClient:
"""Mock GitLab API client for testing."""
def __init__(self):
self._fetch_async = AsyncMock()
self.get_project_members_async = AsyncMock(return_value=[])
def config(self):
"""Return mock config."""
mock_config = MagicMock()
mock_config.project = "namespace/project"
return mock_config
@pytest.fixture
def mock_glab_client():
"""Create a mock GitLab client."""
client = MockGitLabClient()
client.config = MagicMock()
client.config.project = "namespace/test-project"
return client
@pytest.fixture
def permission_checker(mock_glab_client):
"""Create a permission checker instance."""
return GitLabPermissionChecker(
glab_client=mock_glab_client,
project="namespace/test-project",
allowed_roles=["OWNER", "MAINTAINER"],
allow_external_contributors=False,
)
@pytest.mark.asyncio
async def test_verify_token_scopes_success(permission_checker, mock_glab_client):
"""Test successful token scope verification."""
mock_glab_client._fetch_async.return_value = {
"id": 123,
"name": "test-project",
"path_with_namespace": "namespace/test-project",
}
# Should not raise
await permission_checker.verify_token_scopes()
@pytest.mark.asyncio
async def test_verify_token_scopes_project_not_found(
permission_checker, mock_glab_client
):
"""Test project not found raises GitLabPermissionError."""
mock_glab_client._fetch_async.return_value = None
with pytest.raises(GitLabPermissionError, match="Cannot access project"):
await permission_checker.verify_token_scopes()
@pytest.mark.asyncio
async def test_check_label_adder_success(permission_checker, mock_glab_client):
"""Test successfully finding who added a label."""
mock_glab_client.get_project_members_async.return_value = [
{
"id": 1,
"user": {"username": "alice"},
"action": "add",
"label": {"name": "auto-fix"},
},
{
"id": 2,
"user": {"username": "bob"},
"action": "remove",
"label": {"name": "auto-fix"},
},
]
username, role = await permission_checker.check_label_adder(123, "auto-fix")
assert username == "alice"
assert role in [
"OWNER",
"MAINTAINER",
"DEVELOPER",
"REPORTER",
"GUEST",
"NONE",
]
@pytest.mark.asyncio
async def test_check_label_adder_label_not_found(permission_checker, mock_glab_client):
"""Test label not found raises GitLabPermissionError."""
mock_glab_client.get_project_members_async.return_value = [
{
"id": 1,
"user": {"username": "alice"},
"action": "add",
"label": {"name": "bug"},
},
]
with pytest.raises(GitLabPermissionError, match="not found in issue"):
await permission_checker.check_label_adder(123, "auto-fix")
@pytest.mark.asyncio
async def test_check_label_adder_no_username(permission_checker, mock_glab_client):
"""Test label event without username raises GitLabPermissionError."""
mock_glab_client.get_project_members_async.return_value = [
{
"id": 1,
"action": "add",
"label": {"name": "auto-fix"},
},
]
with pytest.raises(GitLabPermissionError, match="Could not determine who added"):
await permission_checker.check_label_adder(123, "auto-fix")
@pytest.mark.asyncio
async def test_get_user_role_project_member(permission_checker, mock_glab_client):
"""Test getting role for project member."""
mock_glab_client.get_project_members_async.return_value = [
{
"id": 1,
"username": "alice",
"access_level": 40, # MAINTAINER
},
]
role = await permission_checker.get_user_role("alice")
assert role == "MAINTAINER"
@pytest.mark.asyncio
async def test_get_user_role_owner_via_namespace(permission_checker, mock_glab_client):
"""Test getting OWNER role via namespace ownership."""
# Not a direct member
mock_glab_client._fetch_async.side_effect = [
[], # No project members
{ # Project info
"id": 123,
"namespace": {
"full_path": "namespace",
"owner_id": 999,
},
},
[ # User info matches owner
{
"id": 999,
"username": "alice",
},
],
]
role = await permission_checker.get_user_role("alice")
assert role == "OWNER"
@pytest.mark.asyncio
async def test_get_user_role_no_relationship(permission_checker, mock_glab_client):
"""Test getting role for user with no relationship."""
mock_glab_client._fetch_async.side_effect = [
[], # No project members
{ # Project info
"id": 123,
"namespace": {
"full_path": "namespace",
"owner_id": 999,
},
},
[ # User doesn't match owner
{
"id": 111,
"username": "alice",
},
],
]
role = await permission_checker.get_user_role("alice")
assert role == "NONE"
@pytest.mark.asyncio
async def test_get_user_role_uses_cache(permission_checker, mock_glab_client):
"""Test that role results are cached."""
mock_glab_client.get_project_members_async.return_value = [
{
"id": 1,
"username": "alice",
"access_level": 40,
},
]
# First call
role1 = await permission_checker.get_user_role("alice")
# Second call should use cache
role2 = await permission_checker.get_user_role("alice")
assert role1 == role2 == "MAINTAINER"
# Should only call API once
assert mock_glab_client.get_project_members_async.call_count == 1
@pytest.mark.asyncio
async def test_is_allowed_for_autofix_allowed(permission_checker, mock_glab_client):
"""Test user is allowed for auto-fix."""
mock_glab_client.get_project_members_async.return_value = [
{
"id": 1,
"username": "alice",
"access_level": 40, # MAINTAINER
},
]
result = await permission_checker.is_allowed_for_autofix("alice")
assert result.allowed is True
assert result.username == "alice"
assert result.role == "MAINTAINER"
assert result.reason is None
@pytest.mark.asyncio
async def test_is_allowed_for_autofix_denied(permission_checker, mock_glab_client):
"""Test user is denied for auto-fix."""
mock_glab_client.get_project_members_async.return_value = [
{
"id": 1,
"username": "bob",
"access_level": 20, # REPORTER (not in allowed roles)
},
]
result = await permission_checker.is_allowed_for_autofix("bob")
assert result.allowed is False
assert result.username == "bob"
assert result.role == "REPORTER"
assert "not in allowed roles" in result.reason
@pytest.mark.asyncio
async def test_verify_automation_trigger_allowed(permission_checker, mock_glab_client):
"""Test complete verification succeeds for allowed user."""
mock_glab_client._fetch_async.side_effect = [
# Label events
[
{
"id": 1,
"user": {"username": "alice"},
"action": "add",
"label": {"name": "auto-fix"},
},
],
# User role check
[
{
"id": 1,
"username": "alice",
"access_level": 40,
},
],
]
result = await permission_checker.verify_automation_trigger(123, "auto-fix")
assert result.allowed is True
@pytest.mark.asyncio
async def test_verify_automation_trigger_denied_logs_warning(
permission_checker, mock_glab_client, caplog
):
"""Test denial is logged with full context."""
mock_glab_client._fetch_async.side_effect = [
# Label events
[
{
"id": 1,
"user": {"username": "bob"},
"action": "add",
"label": {"name": "auto-fix"},
},
],
# User role check
[
{
"id": 1,
"username": "bob",
"access_level": 20, # REPORTER
},
],
]
result = await permission_checker.verify_automation_trigger(123, "auto-fix")
assert result.allowed is False
def test_log_permission_denial(permission_checker, caplog):
"""Test permission denial logging includes full context."""
with caplog.at_level(logging.INFO):
permission_checker.log_permission_denial(
action="auto-fix",
username="bob",
role="REPORTER",
issue_iid=123,
)
# Check that the log contains all relevant info
assert len(caplog.records) > 0
log_message = caplog.records[0].message
assert "auto-fix" in log_message
assert "bob" in log_message
assert "REPORTER" in log_message
assert "123" in log_message
def test_access_levels():
"""Test access level constants are correct."""
assert GitLabPermissionChecker.ACCESS_LEVELS["GUEST"] == 10
assert GitLabPermissionChecker.ACCESS_LEVELS["REPORTER"] == 20
assert GitLabPermissionChecker.ACCESS_LEVELS["DEVELOPER"] == 30
assert GitLabPermissionChecker.ACCESS_LEVELS["MAINTAINER"] == 40
assert GitLabPermissionChecker.ACCESS_LEVELS["OWNER"] == 50
@pytest.mark.asyncio
async def test_get_user_role_developer(permission_checker, mock_glab_client):
"""Test getting DEVELOPER role."""
mock_glab_client.get_project_members_async.return_value = [
{
"id": 1,
"username": "dev",
"access_level": 30,
},
]
role = await permission_checker.get_user_role("dev")
assert role == "DEVELOPER"
@pytest.mark.asyncio
async def test_get_user_role_guest(permission_checker, mock_glab_client):
"""Test getting GUEST role."""
mock_glab_client.get_project_members_async.return_value = [
{
"id": 1,
"username": "guest",
"access_level": 10,
},
]
role = await permission_checker.get_user_role("guest")
assert role == "GUEST"
@@ -0,0 +1,249 @@
"""
GitLab Provider Tests
=====================
Tests for GitLabProvider implementation of the GitProvider protocol.
"""
import json
from datetime import datetime, timezone
from enum import Enum
from pathlib import Path
from typing import Any
from unittest.mock import MagicMock, Mock, patch
import pytest
from __tests__.fixtures.gitlab import (
MOCK_GITLAB_CONFIG,
mock_issue_data,
mock_mr_data,
mock_pipeline_data,
)
# Mock ProviderType enum since GitHub runners aren't available in this branch
# Note: GitLabProvider defines its own ProviderType when GitHub runners aren't available,
# so we just use the string value for comparison
GITLAB_PROVIDER_VALUE = "gitlab" # GitHub protocol uses lowercase
# Tests for GitLabProvider
class TestGitLabProvider:
"""Test GitLabProvider implements GitProvider protocol correctly."""
@pytest.fixture
def provider(self, tmp_path):
"""Create a GitLabProvider instance for testing."""
from runners.gitlab.providers.gitlab_provider import GitLabProvider
with patch(
"runners.gitlab.providers.gitlab_provider.GitLabClient"
) as mock_client:
provider = GitLabProvider(
_repo="group/project",
_token="test-token",
_instance_url="https://gitlab.example.com",
_project_dir=tmp_path,
_glab_client=mock_client.return_value,
)
return provider
def test_provider_type_property(self, provider):
"""Test provider type is GitLab."""
# Compare the value since ProviderType may be defined in different modules
assert provider.provider_type.value == GITLAB_PROVIDER_VALUE
def test_repo_property(self, provider):
"""Test repo property returns the repository."""
assert provider.repo == "group/project"
def test_fetch_pr(self, provider):
"""Test fetching a single MR."""
# Mock client responses
provider._glab_client.get_mr.return_value = mock_mr_data()
provider._glab_client.get_mr_changes.return_value = {
"changes": [
{
"diff": "@@ -0,0 +1,10 @@\n+new line",
"new_path": "test.py",
"old_path": "test.py",
}
]
}
# Fetch MR
pr = await_if_needed(provider.fetch_pr(123))
assert pr.number == 123
assert pr.title == "Add user authentication feature"
assert pr.author == "john_doe"
assert pr.state == "opened"
assert pr.source_branch == "feature/oauth-auth"
assert pr.target_branch == "main"
assert pr.provider.name == "GITLAB"
def test_fetch_prs_with_filters(self, provider):
"""Test fetching multiple MRs with filters."""
provider._glab_client._fetch.return_value = [
mock_mr_data(iid=100),
mock_mr_data(iid=101, state="closed"),
]
prs = await_if_needed(provider.fetch_prs())
assert len(prs) == 2
def test_fetch_pr_diff(self, provider):
"""Test fetching MR diff."""
expected_diff = "diff content here"
provider._glab_client.get_mr_diff.return_value = expected_diff
diff = await_if_needed(provider.fetch_pr_diff(123))
assert diff == expected_diff
def test_fetch_issue(self, provider):
"""Test fetching a single issue."""
from __tests__.fixtures.gitlab import SAMPLE_ISSUE_DATA
provider._glab_client._fetch.return_value = SAMPLE_ISSUE_DATA
issue = await_if_needed(provider.fetch_issue(42))
assert issue.number == 42
assert issue.title == "Bug: Login button not working"
assert issue.author == "jane_smith"
assert issue.state == "opened"
def test_fetch_issues_with_filters(self, provider):
"""Test fetching issues with filters."""
provider._glab_client._fetch.return_value = [
mock_issue_data(iid=10),
mock_issue_data(iid=11),
]
issues = await_if_needed(provider.fetch_issues())
assert len(issues) == 2
def test_post_review(self, provider):
"""Test posting a review to an MR."""
# Import ReviewData from GitHub protocol (which GitLabProvider uses)
from runners.github.providers.protocol import ReviewData
provider._glab_client.post_mr_note.return_value = {"id": 999}
provider._glab_client._fetch.return_value = {} # approve MR response
review = ReviewData(
pr_number=123,
body="LGTM with minor suggestions",
event="approve",
)
note_id = await_if_needed(provider.post_review(123, review))
assert note_id == 999
provider._glab_client.post_mr_note.assert_called_once()
def test_merge_pr(self, provider):
"""Test merging an MR."""
provider._glab_client.merge_mr.return_value = {"status": "success"}
result = await_if_needed(provider.merge_pr(123, merge_method="merge"))
assert result is True
def test_close_pr(self, provider):
"""Test closing an MR."""
provider._glab_client._fetch.return_value = {}
result = await_if_needed(
provider.close_pr(123, comment="Closing as not needed")
)
assert result is True
def test_create_label(self, provider):
"""Test creating a label."""
# Use LabelData from the provider's fallback protocol
from runners.gitlab.providers.gitlab_provider import (
LabelData as GitLabLabelData,
)
# Create an alias for readability
LabelData = GitLabLabelData
provider._glab_client._fetch.return_value = {}
label = LabelData(
name="bug",
color="#ff0000",
description="Bug report",
)
await_if_needed(provider.create_label(label))
# Verify call was made (checking that it didn't raise)
provider._glab_client._fetch.assert_called()
def test_list_labels(self, provider):
"""Test listing labels."""
provider._glab_client._fetch.return_value = [
{"name": "bug", "color": "ff0000", "description": "Bug"},
{"name": "feature", "color": "00ff00", "description": "Feature"},
]
labels = await_if_needed(provider.list_labels())
assert len(labels) == 2
assert labels[0].name == "bug"
assert labels[0].color == "#ff0000"
def test_get_repository_info(self, provider):
"""Test getting repository info."""
provider._glab_client._fetch.return_value = {
"name": "project",
"path_with_namespace": "group/project",
"default_branch": "main",
}
info = await_if_needed(provider.get_repository_info())
assert info["default_branch"] == "main"
def test_get_default_branch(self, provider):
"""Test getting default branch."""
provider._glab_client._fetch.return_value = {
"default_branch": "main",
}
branch = await_if_needed(provider.get_default_branch())
assert branch == "main"
def test_api_get(self, provider):
"""Test low-level API GET."""
provider._glab_client._fetch.return_value = {"data": "value"}
result = await_if_needed(provider.api_get("/projects/1"))
assert result["data"] == "value"
def test_api_post(self, provider):
"""Test low-level API POST."""
provider._glab_client._fetch.return_value = {"id": 123}
result = await_if_needed(
provider.api_post("/projects/1/notes", {"body": "test"})
)
assert result["id"] == 123
def await_if_needed(coro_or_result):
"""Helper to await async functions if needed."""
import asyncio
if hasattr(coro_or_result, "__await__"):
return asyncio.run(coro_or_result)
return coro_or_result
@@ -0,0 +1,519 @@
"""
GitLab Rate Limiter Tests
=========================
Tests for token bucket rate limiting.
"""
import asyncio
import time
from unittest.mock import patch
import pytest
class TestTokenBucket:
"""Test TokenBucket for rate limiting."""
def test_token_bucket_initialization(self):
"""Test token bucket initializes correctly."""
from runners.gitlab.utils.rate_limiter import TokenBucket
bucket = TokenBucket(capacity=10, refill_rate=5.0)
assert bucket.capacity == 10
assert bucket.refill_rate == 5.0
assert bucket.tokens == 10
def test_token_bucket_consume_success(self):
"""Test consuming tokens when available."""
from runners.gitlab.utils.rate_limiter import TokenBucket
bucket = TokenBucket(capacity=10, refill_rate=5.0)
success = bucket.consume(1)
assert success is True
assert bucket.tokens == 9
def test_token_bucket_consume_multiple(self):
"""Test consuming multiple tokens."""
from runners.gitlab.utils.rate_limiter import TokenBucket
bucket = TokenBucket(capacity=10, refill_rate=5.0)
success = bucket.consume(5)
assert success is True
assert bucket.tokens == 5
def test_token_bucket_consume_insufficient(self):
"""Test consuming when insufficient tokens."""
from runners.gitlab.utils.rate_limiter import TokenBucket
bucket = TokenBucket(capacity=10, refill_rate=5.0)
# Consume more than available
success = bucket.consume(15)
assert success is False
assert bucket.tokens == 10 # Should not change
def test_token_bucket_refill(self):
"""Test token refill over time."""
from runners.gitlab.utils.rate_limiter import TokenBucket
bucket = TokenBucket(capacity=10, refill_rate=10.0)
# Consume all tokens
bucket.consume(10)
assert bucket.tokens == 0
# Wait for refill (0.1 seconds at 10 tokens/sec = 1 token)
time.sleep(0.11)
# Check refill
available = bucket.tokens
assert available >= 1
def test_token_bucket_refill_cap(self):
"""Test tokens don't exceed capacity."""
from runners.gitlab.utils.rate_limiter import TokenBucket
bucket = TokenBucket(capacity=10, refill_rate=100.0)
# Wait long time for refill
time.sleep(0.2)
# Should not exceed capacity
assert bucket.tokens <= 10
def test_token_bucket_wait_for_token(self):
"""Test waiting for token availability."""
from runners.gitlab.utils.rate_limiter import TokenBucket
bucket = TokenBucket(capacity=5, refill_rate=10.0)
# Consume all
bucket.consume(5)
# Should wait for refill
start = time.time()
bucket.consume(1, wait=True)
elapsed = time.time() - start
# Should have waited at least 0.1 seconds
assert elapsed >= 0.1
def test_token_bucket_wait_with_tokens(self):
"""Test wait returns immediately when tokens available."""
from runners.gitlab.utils.rate_limiter import TokenBucket
bucket = TokenBucket(capacity=10, refill_rate=5.0)
start = time.time()
bucket.consume(1, wait=True)
elapsed = time.time() - start
# Should be immediate
assert elapsed < 0.01
def test_token_bucket_get_available(self):
"""Test getting available token count."""
from runners.gitlab.utils.rate_limiter import TokenBucket
bucket = TokenBucket(capacity=10, refill_rate=5.0)
assert bucket.get_available() == 10
bucket.consume(3)
assert bucket.get_available() == 7
def test_token_bucket_reset(self):
"""Test resetting token bucket."""
from runners.gitlab.utils.rate_limiter import TokenBucket
bucket = TokenBucket(capacity=10, refill_rate=5.0)
bucket.consume(5)
assert bucket.tokens == 5
bucket.reset()
assert bucket.tokens == 10
class TestRateLimiter:
"""Test RateLimiter for API rate limiting."""
@pytest.fixture
def limiter(self):
"""Create a rate limiter for testing."""
from runners.gitlab.utils.rate_limiter import RateLimiter
return RateLimiter(
requests_per_minute=60,
burst_size=10,
)
def test_rate_limiter_initialization(self):
"""Test rate limiter initializes correctly."""
from runners.gitlab.utils.rate_limiter import RateLimiter
limiter = RateLimiter(
requests_per_minute=60,
burst_size=10,
)
assert limiter.requests_per_minute == 60
assert limiter.burst_size == 10
def test_acquire_request(self, limiter):
"""Test acquiring a request slot."""
success = limiter.acquire()
assert success is True
def test_acquire_burst(self, limiter):
"""Test burst requests."""
# Should be able to make burst_size requests immediately
for _ in range(10):
success = limiter.acquire()
assert success is True
def test_acquire_exceeds_burst(self, limiter):
"""Test exceeding burst limit."""
# Consume burst capacity
for _ in range(10):
limiter.acquire()
# Next request should fail
success = limiter.acquire()
assert success is False
def test_acquire_with_wait(self, limiter):
"""Test acquire with wait option."""
# Consume burst
for _ in range(10):
limiter.acquire()
# Should wait for refill
start = time.time()
success = limiter.acquire(wait=True)
elapsed = time.time() - start
assert success is True
# At 60 req/min, 1 request = 1 second
assert elapsed >= 0.9
def test_get_wait_time(self, limiter):
"""Test getting wait time."""
# No wait needed initially
wait_time = limiter.get_wait_time()
assert wait_time == 0
# Consume burst
for _ in range(10):
limiter.acquire()
# Should need to wait
wait_time = limiter.get_wait_time()
assert wait_time > 0
def test_reset(self, limiter):
"""Test resetting rate limiter."""
# Consume some capacity
for _ in range(5):
limiter.acquire()
limiter.reset()
# Should have full capacity
success = limiter.acquire()
assert success is True
def test_rate_limiter_state_tracking(self, limiter):
"""Test rate limiter tracks request state."""
from runners.gitlab.utils.rate_limiter import RateLimiterState
state = limiter.get_state()
assert isinstance(state, RateLimiterState)
assert state.available_tokens >= 0
assert state.available_tokens <= limiter.burst_size
def test_concurrent_requests(self, limiter):
"""Test concurrent request handling."""
import threading
results = []
def make_request():
success = limiter.acquire(wait=True)
results.append(success)
threads = [threading.Thread(target=make_request) for _ in range(15)]
for t in threads:
t.start()
for t in threads:
t.join()
# All requests should succeed (some wait for refill)
assert all(results)
def test_rate_limiter_persistence(self, limiter, tmp_path):
"""Test saving and loading rate limiter state."""
state_file = tmp_path / "rate_limiter_state.json"
# Consume some tokens
for _ in range(5):
limiter.acquire()
# Save state
limiter.save_state(state_file)
# Create new limiter and load state
from runners.gitlab.utils.rate_limiter import RateLimiter
new_limiter = RateLimiter(
requests_per_minute=60,
burst_size=10,
)
new_limiter.load_state(state_file)
# Should have same state
original_state = limiter.get_state()
loaded_state = new_limiter.get_state()
assert abs(original_state.available_tokens - loaded_state.available_tokens) < 1
class TestRateLimiterIntegration:
"""Integration tests for rate limiting with API calls."""
def test_rate_limiter_with_api_client(self):
"""Test rate limiter integrates with API client."""
from runners.gitlab.utils.rate_limiter import RateLimiter
limiter = RateLimiter(
requests_per_minute=60,
burst_size=5,
)
call_count = 0
def mock_api_call():
nonlocal call_count
if limiter.acquire(wait=True):
call_count += 1
return {"data": "success"}
return {"error": "rate limited"}
# Make several calls
results = [mock_api_call() for _ in range(8)]
# Should have made all calls successfully (some waited)
assert call_count == 8
assert all(r.get("data") for r in results)
def test_rate_limiter_respects_backoff(self):
"""Test rate limiter handles backoff correctly."""
from runners.gitlab.utils.rate_limiter import RateLimiter
limiter = RateLimiter(
requests_per_minute=30, # 0.5 req/sec
burst_size=3,
)
times = []
def track_time():
times.append(time.time())
return limiter.acquire(wait=True)
# Make burst + 1 requests
for _ in range(4):
track_time()
# First 3 should be immediate (burst)
# 4th should have waited
burst_duration = times[2] - times[0]
wait_duration = times[3] - times[2]
# 4th request should have taken longer
assert wait_duration > burst_duration
@pytest.mark.asyncio
async def test_async_rate_limiting(self):
"""Test rate limiting with async operations."""
from runners.gitlab.utils.rate_limiter import RateLimiter
limiter = RateLimiter(
requests_per_minute=60,
burst_size=5,
)
async def make_request(i):
if limiter.acquire(wait=True):
await asyncio.sleep(0.01) # Simulate API call
return f"request-{i}"
return "rate-limited"
results = await asyncio.gather(*[make_request(i) for i in range(8)])
# All should succeed
assert len(results) == 8
assert all("rate-limited" not in r for r in results)
class TestRateLimiterState:
"""Test RateLimiterState model."""
def test_state_creation(self):
"""Test creating state object."""
from runners.gitlab.utils.rate_limiter import RateLimiterState
state = RateLimiterState(
available_tokens=5.0,
last_refill_time=1234567890.0,
)
assert state.available_tokens == 5.0
assert state.last_refill_time == 1234567890.0
def test_state_to_dict(self):
"""Test converting state to dict."""
from runners.gitlab.utils.rate_limiter import RateLimiterState
state = RateLimiterState(
available_tokens=7.5,
last_refill_time=1234567890.0,
)
data = state.to_dict()
assert data["available_tokens"] == 7.5
assert data["last_refill_time"] == 1234567890.0
def test_state_from_dict(self):
"""Test loading state from dict."""
from runners.gitlab.utils.rate_limiter import RateLimiterState
data = {
"available_tokens": 8.0,
"last_refill_time": 1234567890.0,
}
state = RateLimiterState.from_dict(data)
assert state.available_tokens == 8.0
assert state.last_refill_time == 1234567890.0
class TestRateLimiterDecorators:
"""Test rate limiter decorators."""
def test_rate_limit_decorator(self):
"""Test rate limit decorator for functions."""
from runners.gitlab.utils.rate_limiter import rate_limit
limiter = type(
"MockLimiter",
(),
{
"acquire": lambda wait=True: True,
},
)()
@rate_limit(limiter)
def api_function():
return "success"
result = api_function()
assert result == "success"
def test_rate_limit_decorator_with_wait(self):
"""Test rate limit decorator respects wait parameter."""
from runners.gitlab.utils.rate_limiter import rate_limit
call_count = 0
class MockLimiter:
def acquire(self, wait=True):
nonlocal call_count
call_count += 1
return call_count <= 3 # Fail after 3 calls
limiter = MockLimiter()
@rate_limit(limiter, wait=True)
def api_function():
return "success"
# First 3 succeed
for _ in range(3):
result = api_function()
assert result == "success"
# 4th should fail (would wait but our mock returns False)
result = api_function()
assert result is None
class TestAdaptiveRateLimiting:
"""Test adaptive rate limiting based on responses."""
def test_adaptive_backoff_on_429(self):
"""Test adaptive backoff on rate limit errors."""
from runners.gitlab.utils.rate_limiter import AdaptiveRateLimiter
limiter = AdaptiveRateLimiter(
requests_per_minute=60,
burst_size=10,
)
# Simulate rate limit response
limiter.handle_response(status_code=429)
# Should reduce rate
state = limiter.get_state()
assert state.adaptive_factor < 1.0
def test_adaptive_recovery_on_success(self):
"""Test adaptive recovery on successful requests."""
from runners.gitlab.utils.rate_limiter import AdaptiveRateLimiter
limiter = AdaptiveRateLimiter(
requests_per_minute=60,
burst_size=10,
)
# Trigger backoff
limiter.handle_response(status_code=429)
# Recover with successful requests
for _ in range(10):
limiter.handle_response(status_code=200)
# Should recover rate
state = limiter.get_state()
assert state.adaptive_factor >= 0.9
def test_adaptive_minimum_rate(self):
"""Test adaptive rate has minimum floor."""
from runners.gitlab.utils.rate_limiter import AdaptiveRateLimiter
limiter = AdaptiveRateLimiter(
requests_per_minute=60,
burst_size=10,
min_adaptive_factor=0.1,
)
# Trigger many backoffs
for _ in range(100):
limiter.handle_response(status_code=429)
# Should not go below minimum
state = limiter.get_state()
assert state.adaptive_factor >= 0.1
@@ -0,0 +1,292 @@
"""
Tests for GitLab Triage Engine
=================================
Tests for AI-driven issue triage and categorization.
"""
from unittest.mock import patch
import pytest
try:
from runners.gitlab.glab_client import GitLabConfig
from runners.gitlab.models import TriageCategory, TriageResult
from runners.gitlab.services.triage_engine import TriageEngine
except ImportError:
from glab_client import GitLabConfig
from models import TriageCategory, TriageResult
from runners.gitlab.triage_engine import TriageEngine
# Mock response parser for testing
def parse_findings_from_response(response: str) -> dict:
"""Mock parser for testing triage engine."""
import json
import re
# Try to extract JSON from markdown code blocks
json_match = re.search(r"```(?:json)?\s*\n(.*?)\n```", response, re.DOTALL)
if json_match:
response = json_match.group(1)
try:
return json.loads(response)
except json.JSONDecodeError:
return {"category": "bug", "confidence": 0.5}
@pytest.fixture
def mock_config():
"""Create a mock GitLab config."""
try:
from runners.gitlab.models import GitLabRunnerConfig
return GitLabRunnerConfig(
token="test-token",
project="namespace/test-project",
instance_url="https://gitlab.example.com",
model="claude-sonnet-4-5-20250929",
)
except ImportError:
# Fallback to simple config with model attribute
config = GitLabConfig(
token="test-token",
project="namespace/test-project",
instance_url="https://gitlab.example.com",
)
config.model = "claude-sonnet-4-5-20250929"
return config
@pytest.fixture
def sample_issue():
"""Sample issue data."""
return {
"iid": 123,
"title": "Fix authentication bug",
"description": "Users cannot log in when using special characters in password",
"labels": ["bug", "critical"],
"author": {"username": "reporter"},
"state": "opened",
}
@pytest.fixture
def engine(mock_config, tmp_path):
"""Create a triage engine instance."""
return TriageEngine(
project_dir=tmp_path,
gitlab_dir=tmp_path / ".auto-claude" / "gitlab",
config=mock_config,
)
class TestTriageEngineBasic:
"""Tests for triage engine initialization and basic operations."""
def test_engine_initialization(self, engine):
"""Test that engine initializes correctly."""
assert engine is not None
assert engine.project_dir is not None
def test_supported_categories(self, engine):
"""Test that engine supports all required categories."""
expected_categories = {
TriageCategory.BUG,
TriageCategory.FEATURE,
TriageCategory.DUPLICATE,
TriageCategory.QUESTION,
TriageCategory.SPAM,
TriageCategory.INVALID,
TriageCategory.WONTFIX,
}
# Engine should handle all categories
for category in expected_categories:
assert category in TriageCategory
class ResponseParserTests:
"""Tests for response parsing utilities."""
def test_parse_findings_valid_json(self, engine):
"""Test parsing valid JSON response with findings."""
response = """```json
{
"category": "bug",
"confidence": 0.9,
"duplicate_of": null,
"reasoning": "Clear bug report with reproduction steps",
"suggested_labels": ["bug", "critical"]
}
```"""
result = parse_findings_from_response(response)
assert result["category"] == "bug"
assert result["confidence"] == 0.9
def test_parse_findings_with_duplicate(self, engine):
"""Test parsing response with duplicate reference."""
response = """```json
{
"category": "duplicate",
"confidence": 0.95,
"duplicate_of": 42,
"reasoning": "Same as issue #42",
"suggested_labels": ["duplicate"]
}
```"""
result = parse_findings_from_response(response)
assert result["category"] == "duplicate"
assert result["duplicate_of"] == 42
def test_parse_findings_with_question(self, engine):
"""Test parsing response for question-type issue."""
response = """```json
{
"category": "question",
"confidence": 0.8,
"reasoning": "User is asking for help, not reporting a bug",
"suggested_response": "Please provide more details"
}
```"""
result = parse_findings_from_response(response)
assert result["category"] == "question"
assert "suggested_response" in result
def test_parse_findings_markdown_only(self, engine):
"""Test parsing response without JSON code blocks."""
response = """{"category": "feature", "confidence": 0.7}"""
result = parse_findings_from_response(response)
assert result["category"] == "feature"
def test_parse_findings_invalid_json(self, engine):
"""Test parsing invalid JSON response."""
response = "This is not valid JSON at all"
result = parse_findings_from_response(response)
# Should return defaults for invalid response
assert "category" in result
class TestTriageCategorization:
"""Tests for issue categorization."""
def test_triage_categories_exist(self):
"""Test that all triage categories are defined."""
expected_categories = {
TriageCategory.BUG,
TriageCategory.FEATURE,
TriageCategory.DUPLICATE,
TriageCategory.QUESTION,
TriageCategory.SPAM,
TriageCategory.INVALID,
TriageCategory.WONTFIX,
}
# Verify categories exist
assert TriageCategory.BUG in expected_categories
assert TriageCategory.FEATURE in expected_categories
class TestTriageContextBuilding:
"""Tests for context building."""
def test_build_triage_context_basic(self, engine, sample_issue):
"""Test building basic triage context."""
context = engine.build_triage_context(sample_issue, [])
assert "Issue #123" in context
assert "Fix authentication bug" in context
# The description contains "Users cannot log in" not "Cannot login"
assert "Users cannot log in" in context
def test_build_triage_context_with_duplicates(self, engine):
"""Test building context with potential duplicates."""
issue = {
"iid": 1,
"title": "Login bug",
"description": "Cannot login",
"author": {"username": "user1"},
"created_at": "2024-01-01T00:00:00Z",
"labels": ["bug"],
}
all_issues = [
issue,
{
"iid": 2,
"title": "Login issue",
"description": "Login not working",
"author": {"username": "user2"},
"created_at": "2024-01-02T00:00:00Z",
"labels": [],
},
]
context = engine.build_triage_context(issue, all_issues)
# Should include potential duplicates section
assert "Potential Duplicates" in context
assert "#2" in context
def test_build_triage_context_no_duplicates(self, engine, sample_issue):
"""Test building context without duplicates."""
context = engine.build_triage_context(sample_issue, [])
# Should NOT include duplicates section
assert "Potential Duplicates" not in context
class TestTriageErrors:
"""Tests for error handling in triage."""
def test_triage_result_default_values(self):
"""Test TriageResult can be created with default values."""
result = TriageResult(
issue_iid=1,
project="test/project",
category=TriageCategory.FEATURE,
confidence=0.0,
)
assert result.issue_iid == 1
assert result.category == TriageCategory.FEATURE
assert result.confidence == 0.0
class TestTriageResult:
"""Tests for TriageResult model."""
def test_triage_result_creation(self):
"""Test creating a triage result."""
result = TriageResult(
issue_iid=123,
project="namespace/project",
category=TriageCategory.BUG,
confidence=0.9,
)
assert result.issue_iid == 123
assert result.category == TriageCategory.BUG
assert result.confidence == 0.9
def test_triage_result_with_duplicate(self):
"""Test creating a triage result with duplicate reference."""
result = TriageResult(
issue_iid=456,
project="namespace/project",
category=TriageCategory.DUPLICATE,
confidence=0.95,
duplicate_of=123,
)
assert result.duplicate_of == 123
assert result.category == TriageCategory.DUPLICATE
+400
View File
@@ -0,0 +1,400 @@
"""
Tests for GitLab TypedDict Definitions
========================================
Tests for type definitions and TypedDict usage.
"""
import pytest
try:
from runners.gitlab.types import (
GitLabCommit,
GitLabIssue,
GitLabLabel,
GitLabMR,
GitLabPipeline,
GitLabUser,
)
except ImportError:
from runners.gitlab.types import (
GitLabCommit,
GitLabIssue,
GitLabLabel,
GitLabMR,
GitLabPipeline,
GitLabUser,
)
class TestGitLabUserTypedDict:
"""Tests for GitLabUser TypedDict."""
def test_user_dict_structure(self):
"""Test that user dict conforms to expected structure."""
user: GitLabUser = {
"id": 123,
"username": "testuser",
"name": "Test User",
"email": "test@example.com",
"avatar_url": "https://example.com/avatar.png",
"web_url": "https://gitlab.example.com/testuser",
}
assert user["id"] == 123
assert user["username"] == "testuser"
def test_user_dict_optional_fields(self):
"""Test user dict with optional fields omitted."""
user: GitLabUser = {
"id": 456,
"username": "minimal",
"name": "Minimal User",
}
assert user["id"] == 456
# Should work without email, avatar_url, web_url
class TestGitLabLabelTypedDict:
"""Tests for GitLabLabel TypedDict."""
def test_label_dict_structure(self):
"""Test that label dict conforms to expected structure."""
label: GitLabLabel = {
"id": 1,
"name": "bug",
"color": "#FF0000",
"description": "Bug report",
}
assert label["name"] == "bug"
assert label["color"] == "#FF0000"
def test_label_dict_optional_description(self):
"""Test label dict without description."""
label: GitLabLabel = {
"id": 2,
"name": "enhancement",
"color": "#00FF00",
}
assert label["name"] == "enhancement"
class TestGitLabMRTypedDict:
"""Tests for GitLabMR TypedDict."""
def test_mr_dict_structure(self):
"""Test that MR dict conforms to expected structure."""
mr: GitLabMR = {
"iid": 123,
"id": 456,
"title": "Test MR",
"description": "Test description",
"state": "opened",
"created_at": "2024-01-01T00:00:00Z",
"updated_at": "2024-01-01T01:00:00Z",
"merged_at": None,
"author": {
"id": 1,
"username": "author",
"name": "Author",
},
"assignees": [],
"reviewers": [],
"source_branch": "feature",
"target_branch": "main",
"web_url": "https://gitlab.example.com/merge_requests/123",
}
assert mr["iid"] == 123
assert mr["state"] == "opened"
def test_mr_dict_with_merge_status(self):
"""Test MR dict with merge status."""
mr: GitLabMR = {
"iid": 456,
"id": 789,
"title": "Merged MR",
"state": "merged",
"merged_at": "2024-01-02T00:00:00Z",
"author": {"id": 1, "username": "dev"},
"assignees": [],
"reviewers": [],
"diff_refs": {
"base_sha": "abc123",
"head_sha": "def456",
"start_sha": "abc123",
"head_commit": {"id": "def456"},
},
"labels": [],
}
assert mr["state"] == "merged"
assert mr["merged_at"] is not None
class TestGitLabIssueTypedDict:
"""Tests for GitLabIssue TypedDict."""
def test_issue_dict_structure(self):
"""Test that issue dict conforms to expected structure."""
issue: GitLabIssue = {
"iid": 123,
"id": 456,
"title": "Test Issue",
"description": "Test description",
"state": "opened",
"created_at": "2024-01-01T00:00:00Z",
"updated_at": "2024-01-01T01:00:00Z",
"closed_at": None,
"author": {
"id": 1,
"username": "reporter",
"name": "Reporter",
},
"assignees": [],
"labels": [],
"web_url": "https://gitlab.example.com/issues/123",
}
assert issue["iid"] == 123
assert issue["state"] == "opened"
def test_issue_dict_with_labels(self):
"""Test issue dict with labels."""
issue: GitLabIssue = {
"iid": 789,
"id": 101,
"title": "Labeled Issue",
"labels": [
{
"id": 1,
"name": "bug",
"color": "#FF0000",
},
{
"id": 2,
"name": "critical",
"color": "#00FF00",
},
],
}
assert len(issue["labels"]) == 2
assert issue["labels"][0]["name"] == "bug"
class TestGitLabCommitTypedDict:
"""Tests for GitLabCommit TypedDict."""
def test_commit_dict_structure(self):
"""Test that commit dict conforms to expected structure."""
commit: GitLabCommit = {
"id": "abc123def456",
"short_id": "abc123",
"title": "Test commit",
"message": "Test commit message",
"author_name": "Developer",
"author_email": "dev@example.com",
"authored_date": "2024-01-01T00:00:00Z",
"committed_date": "2024-01-01T00:00:01Z",
"web_url": "https://gitlab.example.com/commit/abc123",
}
assert commit["id"] == "abc123def456"
assert commit["short_id"] == "abc123"
assert commit["author_name"] == "Developer"
class TestGitLabPipelineTypedDict:
"""Tests for GitLabPipeline TypedDict."""
def test_pipeline_dict_structure(self):
"""Test that pipeline dict conforms to expected structure."""
pipeline: GitLabPipeline = {
"id": 123,
"iid": 456,
"project_id": 789,
"sha": "abc123",
"ref": "main",
"status": "success",
"created_at": "2024-01-01T00:00:00Z",
"updated_at": "2024-01-01T01:00:00Z",
"finished_at": "2024-01-01T02:00:00Z",
"duration": 120,
"web_url": "https://gitlab.example.com/pipelines/123",
}
assert pipeline["id"] == 123
assert pipeline["status"] == "success"
assert pipeline["duration"] == 120
def test_pipeline_dict_optional_fields(self):
"""Test pipeline dict with optional fields omitted."""
pipeline: GitLabPipeline = {
"id": 456,
"iid": 789,
"project_id": 101,
"sha": "def456",
"ref": "develop",
"status": "running",
"created_at": "2024-01-01T00:00:00Z",
"updated_at": "2024-01-01T01:00:00Z",
"finished_at": None,
"duration": None,
}
assert pipeline["status"] == "running"
assert pipeline["finished_at"] is None
class TestTotalFalseBehavior:
"""Tests for total=False behavior in TypedDict (all fields optional)."""
def test_mr_minimal_dict(self):
"""Test creating MR with minimal required fields."""
# In practice, GitLab API always returns certain fields
# But TypedDict with total=False allows flexibility
mr: GitLabMR = {
"iid": 123,
"id": 456,
"title": "Minimal MR",
"state": "opened",
}
assert mr["iid"] == 123
def test_issue_minimal_dict(self):
"""Test creating issue with minimal required fields."""
issue: GitLabIssue = {
"iid": 456,
"id": 789,
"title": "Minimal Issue",
"state": "opened",
}
assert issue["iid"] == 456
class TestNestedTypedDicts:
"""Tests for nested TypedDict structures."""
def test_mr_with_nested_user(self):
"""Test MR with nested user objects."""
mr: GitLabMR = {
"iid": 123,
"id": 456,
"title": "MR with author",
"state": "opened",
"author": {
"id": 1,
"username": "dev",
"name": "Developer",
},
"assignees": [
{
"id": 2,
"username": "assignee1",
"name": "Assignee One",
}
],
}
assert mr["author"]["username"] == "dev"
assert len(mr["assignees"]) == 1
def test_issue_with_nested_labels(self):
"""Test issue with nested label objects."""
issue: GitLabIssue = {
"iid": 123,
"id": 456,
"title": "Issue with labels",
"state": "opened",
"labels": [
{"id": 1, "name": "bug", "color": "#FF0000"},
{"id": 2, "name": "critical", "color": "#00FF00"},
],
}
assert issue["labels"][0]["name"] == "bug"
assert len(issue["labels"]) == 2
class TestTypeCompatibility:
"""Tests for type compatibility and validation."""
def test_mr_type_accepts_all_states(self):
"""Test that MR type accepts all valid GitLab MR states."""
valid_states = ["opened", "closed", "locked", "merged"]
for state in valid_states:
mr: GitLabMR = {
"iid": 1,
"id": 1,
"title": f"MR in {state} state",
"state": state,
}
assert mr["state"] == state
def test_pipeline_type_accepts_all_statuses(self):
"""Test that pipeline type accepts all valid GitLab pipeline statuses."""
valid_statuses = [
"pending",
"running",
"success",
"failed",
"canceled",
"skipped",
"manual",
"scheduled",
]
for status in valid_statuses:
pipeline: GitLabPipeline = {
"id": 1,
"iid": 1,
"project_id": 1,
"sha": "abc",
"ref": "main",
"status": status,
}
assert pipeline["status"] == status
class TestDocumentation:
"""Tests that types are self-documenting."""
def test_user_fields_are_documented(self):
"""Test that user fields match documentation."""
# GitLabUser should have: id, username, name, email, avatar_url, web_url
user: GitLabUser = {
"id": 1,
"username": "test",
"name": "Test",
"email": "test@example.com",
"avatar_url": "https://example.com/avatar.png",
"web_url": "https://gitlab.example.com/test",
}
# Verify expected fields exist
expected_fields = ["id", "username", "name", "email", "avatar_url", "web_url"]
for field in expected_fields:
assert field in user
def test_mr_fields_are_documented(self):
"""Test that MR fields match documentation."""
# Key MR fields
mr: GitLabMR = {
"iid": 123,
"id": 456,
"title": "Test",
"state": "opened",
"created_at": "2024-01-01T00:00:00Z",
"updated_at": "2024-01-01T01:00:00Z",
}
expected_fields = ["iid", "id", "title", "state", "created_at", "updated_at"]
for field in expected_fields:
assert field in mr
@@ -0,0 +1,318 @@
"""
Tests for GitLab Webhook Operations
======================================
Tests for webhook listing, creation, updating, and deletion.
"""
from unittest.mock import MagicMock, patch
import pytest
try:
from runners.gitlab.glab_client import GitLabClient, GitLabConfig
except ImportError:
from glab_client import GitLabClient, GitLabConfig
@pytest.fixture
def mock_config():
"""Create a mock GitLab config."""
return GitLabConfig(
token="test-token",
project="namespace/test-project",
instance_url="https://gitlab.example.com",
)
@pytest.fixture
def client(mock_config, tmp_path):
"""Create a GitLab client instance."""
return GitLabClient(
project_dir=tmp_path,
config=mock_config,
)
@pytest.fixture
def sample_webhooks():
"""Sample webhook data."""
return [
{
"id": 1,
"url": "https://example.com/webhook",
"project_id": 123,
"push_events": True,
"issues_events": False,
"merge_requests_events": True,
"wiki_page_events": False,
"repository_update_events": False,
"tag_push_events": False,
"note_events": False,
"confidential_note_events": False,
"job_events": False,
"pipeline_events": False,
"deployment_events": False,
"release_events": False,
},
{
"id": 2,
"url": "https://hooks.example.com/another",
"project_id": 123,
"push_events": False,
"issues_events": True,
"merge_requests_events": True,
},
]
class TestListWebhooks:
"""Tests for list_webhooks method."""
@pytest.mark.asyncio
async def test_list_all_webhooks(self, client, sample_webhooks):
"""Test listing all webhooks."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = sample_webhooks
result = client.list_webhooks()
assert len(result) == 2
assert result[0]["id"] == 1
assert result[0]["url"] == "https://example.com/webhook"
@pytest.mark.asyncio
async def test_list_webhooks_empty(self, client):
"""Test listing webhooks when none exist."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = []
result = client.list_webhooks()
assert result == []
@pytest.mark.asyncio
async def test_list_webhooks_async(self, client, sample_webhooks):
"""Test async variant of list_webhooks."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = sample_webhooks
result = await client.list_webhooks_async()
assert len(result) == 2
class TestGetWebhook:
"""Tests for get_webhook method."""
@pytest.mark.asyncio
async def test_get_existing_webhook(self, client, sample_webhooks):
"""Test getting an existing webhook."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = sample_webhooks[0]
result = client.get_webhook(1)
assert result["id"] == 1
assert result["url"] == "https://example.com/webhook"
@pytest.mark.asyncio
async def test_get_webhook_async(self, client, sample_webhooks):
"""Test async variant of get_webhook."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = sample_webhooks[0]
result = await client.get_webhook_async(1)
assert result["id"] == 1
@pytest.mark.asyncio
async def test_get_nonexistent_webhook(self, client):
"""Test getting a webhook that doesn't exist."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.side_effect = Exception("404 Not Found")
with pytest.raises(Exception): # noqa: B017
client.get_webhook(999)
class TestCreateWebhook:
"""Tests for create_webhook method."""
@pytest.mark.asyncio
async def test_create_webhook_basic(self, client):
"""Test creating a webhook with basic settings."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"id": 3,
"url": "https://example.com/new-hook",
}
result = client.create_webhook(
url="https://example.com/new-hook",
)
assert result["id"] == 3
assert result["url"] == "https://example.com/new-hook"
@pytest.mark.asyncio
async def test_create_webhook_with_events(self, client):
"""Test creating a webhook with specific events."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"id": 4,
"url": "https://example.com/push-hook",
"push_events": True,
"issues_events": True,
}
result = client.create_webhook(
url="https://example.com/push-hook",
push_events=True,
issues_events=True,
)
assert result["push_events"] is True
@pytest.mark.asyncio
async def test_create_webhook_with_all_events(self, client):
"""Test creating a webhook that listens to all events."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {"id": 5}
result = client.create_webhook(
url="https://example.com/all-events",
push_events=True,
merge_request_events=True,
issues_events=True,
note_events=True,
job_events=True,
pipeline_events=True,
wiki_page_events=True,
)
assert result["id"] == 5
@pytest.mark.asyncio
async def test_create_webhook_async(self, client):
"""Test async variant of create_webhook."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {"id": 6}
result = await client.create_webhook_async(
url="https://example.com/async-hook",
)
assert result["id"] == 6
class TestUpdateWebhook:
"""Tests for update_webhook method."""
@pytest.mark.asyncio
async def test_update_webhook_url(self, client):
"""Test updating webhook URL."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"id": 1,
"url": "https://example.com/updated-url",
}
result = client.update_webhook(
hook_id=1,
url="https://example.com/updated-url",
)
assert result["url"] == "https://example.com/updated-url"
@pytest.mark.asyncio
async def test_update_webhook_events(self, client):
"""Test updating webhook events."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {
"id": 1,
"push_events": False, # Disabled
"issues_events": True, # Enabled
}
result = client.update_webhook(
hook_id=1,
push_events=False,
issues_events=True,
)
assert result["push_events"] is False
@pytest.mark.asyncio
async def test_update_webhook_async(self, client):
"""Test async variant of update_webhook."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = {"id": 1, "url": "new"}
result = await client.update_webhook_async(
hook_id=1,
url="new",
)
assert result["url"] == "new"
class TestDeleteWebhook:
"""Tests for delete_webhook method."""
@pytest.mark.asyncio
async def test_delete_webhook(self, client):
"""Test deleting a webhook."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = None # 204 No Content
result = client.delete_webhook(1)
# Should not raise on success
assert result is None
@pytest.mark.asyncio
async def test_delete_webhook_async(self, client):
"""Test async variant of delete_webhook."""
# Patch _fetch instead of _fetch_async since _fetch_async calls _fetch
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.return_value = None
result = await client.delete_webhook_async(2)
assert result is None
class TestWebhookErrors:
"""Tests for webhook error handling."""
@pytest.mark.asyncio
async def test_get_invalid_webhook_id(self, client):
"""Test getting webhook with invalid ID."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.side_effect = Exception("404 Not Found")
with pytest.raises(Exception): # noqa: B017
client.get_webhook(0)
@pytest.mark.asyncio
async def test_create_webhook_invalid_url(self, client):
"""Test creating webhook with invalid URL."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.side_effect = Exception("400 Invalid URL")
with pytest.raises(Exception): # noqa: B017
client.create_webhook(url="not-a-url")
@pytest.mark.asyncio
async def test_delete_nonexistent_webhook(self, client):
"""Test deleting webhook that doesn't exist."""
with patch.object(client, "_fetch") as mock_fetch:
mock_fetch.side_effect = Exception("404 Not Found")
with pytest.raises(Exception): # noqa: B017
client.delete_webhook(999)
+733
View File
@@ -0,0 +1,733 @@
"""
GitLab Client Tests
===================
Tests for GitLab client timeout, retry, and async operations.
"""
import asyncio
import json
from unittest.mock import AsyncMock, MagicMock, Mock, patch
import pytest
from requests.exceptions import ConnectionError, Timeout
class TestGitLabClient:
"""Test GitLab client basic operations."""
@pytest.fixture
def client(self):
"""Create a GitLab client for testing."""
from __tests__.fixtures.gitlab import create_mock_client
return create_mock_client()
def test_client_initialization(self, client):
"""Test client initializes correctly."""
assert client.config.token == "glpat-test-token-12345"
assert client.config.project == "group/project"
assert client.config.instance_url == "https://gitlab.example.com"
assert client.default_timeout == 30.0
def test_client_custom_timeout(self):
"""Test client with custom timeout."""
from __tests__.fixtures.gitlab import create_mock_client
client = create_mock_client()
assert client.default_timeout == 30.0 # Uses default
def test_client_custom_retries(self):
"""Test client with custom retry count."""
from __tests__.fixtures.gitlab import create_mock_client
client = create_mock_client()
# Uses default max_retries of 3
assert client.default_timeout == 30.0
def test_build_url(self, client):
"""Test URL building."""
url = client._api_url("/projects/group%2Fproject/merge_requests")
assert "group%2Fproject" in url
assert "merge_requests" in url
assert "/api/v4/" in url
def test_build_url_with_params(self, client):
"""Test URL building with query parameters."""
from urllib.parse import parse_qs, urlencode, urlparse
base_url = client._api_url("/projects/group%2Fproject/merge_requests")
query_string = urlencode({"state": "opened", "per_page": 50}, doseq=True)
full_url = f"{base_url}?{query_string}"
parsed = urlparse(full_url)
params = parse_qs(parsed.query)
assert "state=opened" in full_url or params.get("state") == ["opened"]
assert "per_page=50" in full_url or params.get("per_page") == ["50"]
class TestGitLabClientRetry:
"""Test GitLab client retry logic."""
@pytest.fixture
def client(self):
"""Create a GitLab client for testing."""
import dataclasses
from __tests__.fixtures.gitlab import create_mock_client
client = create_mock_client()
return client
def test_retry_on_timeout(self, client):
"""Test retry on timeout exception."""
from socket import timeout
call_count = 0
def mock_urlopen_side_effect(*args, **kwargs):
nonlocal call_count
call_count += 1
if call_count < 3:
raise TimeoutError("Request timed out")
# Return successful response
mock_resp = Mock()
mock_resp.read.return_value = b'{"iid": 123}'
mock_resp.headers = {"Content-Type": "application/json"}
mock_resp.status = 200
mock_resp.__enter__ = Mock(return_value=mock_resp)
mock_resp.__exit__ = Mock(return_value=False)
return mock_resp
with patch("urllib.request.urlopen", side_effect=mock_urlopen_side_effect):
result = client.get_mr(123)
assert call_count == 3 # Initial + 2 retries
assert result["iid"] == 123
def test_retry_on_connection_error(self, client):
"""Test retry on connection error."""
from urllib.error import URLError
call_count = 0
def mock_urlopen_side_effect(*args, **kwargs):
nonlocal call_count
call_count += 1
if call_count < 2:
raise URLError("Connection failed")
# Return successful response
mock_resp = Mock()
mock_resp.read.return_value = b'{"iid": 123}'
mock_resp.headers = {"Content-Type": "application/json"}
mock_resp.status = 200
mock_resp.__enter__ = Mock(return_value=mock_resp)
mock_resp.__exit__ = Mock(return_value=False)
return mock_resp
with patch("urllib.request.urlopen", side_effect=mock_urlopen_side_effect):
result = client.get_mr(123)
assert call_count == 2 # Initial + 1 retry
assert result["iid"] == 123
def test_retry_exhausted(self, client):
"""Test failure after retry exhaustion."""
from urllib.error import URLError
def mock_urlopen_side_effect(*args, **kwargs):
raise URLError("Request timed out")
with patch("urllib.request.urlopen", side_effect=mock_urlopen_side_effect):
with pytest.raises(Exception, match="GitLab API network error"):
client.get_mr(123)
def test_retry_with_backoff(self, client):
"""Test retry uses exponential backoff."""
import time
from socket import timeout
call_times = []
def mock_urlopen_side_effect(*args, **kwargs):
call_times.append(time.time())
if len(call_times) < 3:
raise TimeoutError("Request timed out")
# Return successful response
mock_resp = Mock()
mock_resp.read.return_value = b'{"iid": 123}'
mock_resp.headers = {"Content-Type": "application/json"}
mock_resp.status = 200
mock_resp.__enter__ = Mock(return_value=mock_resp)
mock_resp.__exit__ = Mock(return_value=False)
return mock_resp
with patch("urllib.request.urlopen", side_effect=mock_urlopen_side_effect):
client.get_mr(123)
# Check delays between retries increase (exponential backoff)
if len(call_times) > 2:
delay1 = call_times[1] - call_times[0]
delay2 = call_times[2] - call_times[1]
# Second delay should be longer (exponential backoff)
assert delay2 > delay1
def test_no_retry_on_client_error(self, client):
"""Test no retry on 4xx client errors."""
from urllib.error import HTTPError
call_count = 0
def mock_urlopen_side_effect(*args, **kwargs):
nonlocal call_count
call_count += 1
# 404 should not be retried (not in RETRYABLE_STATUS_CODES)
raise HTTPError("url", 404, "Not Found", {}, None)
with patch("urllib.request.urlopen", side_effect=mock_urlopen_side_effect):
with pytest.raises(Exception, match="GitLab API error"):
client.get_mr(123)
# Should only be called once (no retry for 4xx)
assert call_count == 1
def test_retry_on_server_error(self, client):
"""Test retry on 5xx server errors."""
from urllib.error import HTTPError
call_count = 0
def mock_urlopen_side_effect(*args, **kwargs):
nonlocal call_count
call_count += 1
if call_count < 2:
raise HTTPError(None, 503, "Service Unavailable", {}, None)
# Return successful response
mock_resp = Mock()
mock_resp.read.return_value = b'{"iid": 123}'
mock_resp.headers = {"Content-Type": "application/json"}
mock_resp.status = 200
mock_resp.__enter__ = Mock(return_value=mock_resp)
mock_resp.__exit__ = Mock(return_value=False)
return mock_resp
with patch("urllib.request.urlopen", side_effect=mock_urlopen_side_effect):
result = client.get_mr(123)
assert call_count == 2
assert result["iid"] == 123
class TestGitLabClientAsync:
"""Test GitLab client async operations."""
@pytest.fixture
def client(self):
"""Create a GitLab client for testing."""
from __tests__.fixtures.gitlab import create_mock_client
return create_mock_client()
@pytest.mark.asyncio
async def test_get_mr_async(self, client):
"""Test async get MR."""
mock_data = {
"iid": 123,
"title": "Test MR",
"state": "opened",
}
with patch.object(client, "_fetch_async", return_value=mock_data):
result = await client.get_mr_async(123)
assert result["iid"] == 123
assert result["title"] == "Test MR"
@pytest.mark.asyncio
async def test_get_mr_changes_async(self, client):
"""Test async get MR changes."""
mock_data = {
"changes": [
{
"old_path": "file.py",
"new_path": "file.py",
"diff": "@@ -1,1 +1,2 @@",
}
]
}
with patch.object(client, "_fetch_async", return_value=mock_data):
result = await client.get_mr_changes_async(123)
assert len(result["changes"]) == 1
@pytest.mark.asyncio
async def test_get_mr_commits_async(self, client):
"""Test async get MR commits."""
mock_data = [
{"id": "abc123", "message": "Commit 1"},
{"id": "def456", "message": "Commit 2"},
]
with patch.object(client, "_fetch_async", return_value=mock_data):
result = await client.get_mr_commits_async(123)
assert len(result) == 2
assert result[0]["id"] == "abc123"
@pytest.mark.asyncio
async def test_get_mr_notes_async(self, client):
"""Test async get MR notes."""
mock_data = [
{"id": 1001, "body": "Comment 1"},
{"id": 1002, "body": "Comment 2"},
]
with patch.object(client, "_fetch_async", return_value=mock_data):
result = await client.get_mr_notes_async(123)
assert len(result) == 2
@pytest.mark.asyncio
async def test_get_mr_pipelines_async(self, client):
"""Test async get MR pipelines."""
mock_data = [
{"id": 1001, "status": "success"},
{"id": 1002, "status": "failed"},
]
with patch.object(client, "_fetch_async", return_value=mock_data):
result = await client.get_mr_pipelines_async(123)
assert len(result) == 2
@pytest.mark.asyncio
async def test_get_issue_async(self, client):
"""Test async get issue."""
mock_data = {
"iid": 456,
"title": "Test Issue",
"state": "opened",
}
with patch.object(client, "_fetch_async", return_value=mock_data):
result = await client.get_issue_async(456)
assert result["iid"] == 456
@pytest.mark.asyncio
async def test_get_pipeline_async(self, client):
"""Test async get pipeline."""
mock_data = {
"id": 1001,
"status": "running",
"ref": "main",
}
with patch.object(client, "_fetch_async", return_value=mock_data):
result = await client.get_pipeline_status_async(1001)
assert result["id"] == 1001
@pytest.mark.asyncio
async def test_get_pipeline_jobs_async(self, client):
"""Test async get pipeline jobs."""
mock_data = [
{"id": 2001, "name": "test", "status": "success"},
{"id": 2002, "name": "build", "status": "failed"},
]
with patch.object(client, "_fetch_async", return_value=mock_data):
result = await client.get_pipeline_jobs_async(1001)
assert len(result) == 2
@pytest.mark.asyncio
async def test_concurrent_async_requests(self, client):
"""Test concurrent async requests."""
async def fetch_mr(iid):
return await client.get_mr_async(iid)
mock_data = {
"iid": 123,
"title": "Test MR",
}
with patch.object(client, "_fetch_async", return_value=mock_data):
results = await asyncio.gather(
fetch_mr(123),
fetch_mr(456),
fetch_mr(789),
)
assert len(results) == 3
@pytest.mark.asyncio
async def test_async_error_handling(self, client):
"""Test async error handling."""
with patch.object(client, "_fetch_async", side_effect=Exception("API Error")):
with pytest.raises(Exception, match="API Error"):
await client.get_mr_async(123)
class TestGitLabClientAPI:
"""Test GitLab client API methods."""
@pytest.fixture
def client(self):
"""Create a GitLab client for testing."""
from __tests__.fixtures.gitlab import create_mock_client
return create_mock_client()
def test_get_mr(self, client):
"""Test getting MR details."""
mock_response = {
"iid": 123,
"title": "Test MR",
"description": "Test description",
"state": "opened",
"author": {"username": "john_doe"},
}
with patch.object(client, "_fetch", return_value=mock_response):
result = client.get_mr(123)
assert result["iid"] == 123
assert result["title"] == "Test MR"
def test_get_mr_changes(self, client):
"""Test getting MR changes."""
mock_response = {
"changes": [
{
"old_path": "src/file.py",
"new_path": "src/file.py",
"diff": "@@ -1,1 +1,2 @@",
}
]
}
with patch.object(client, "_fetch", return_value=mock_response):
result = client.get_mr_changes(123)
assert len(result["changes"]) == 1
def test_get_mr_commits(self, client):
"""Test getting MR commits."""
mock_response = [
{"id": "abc123", "message": "First commit"},
{"id": "def456", "message": "Second commit"},
]
with patch.object(client, "_fetch", return_value=mock_response):
result = client.get_mr_commits(123)
assert len(result) == 2
def test_get_mr_notes(self, client):
"""Test getting MR discussion notes."""
mock_response = [
{"id": 1001, "body": "Review comment", "author": {"username": "reviewer"}},
]
with patch.object(client, "_fetch", return_value=mock_response):
result = client.get_mr_notes(123)
assert len(result) == 1
def test_post_mr_note(self, client):
"""Test posting note to MR."""
mock_response = {"id": 1002, "body": "New comment"}
with patch.object(client, "_fetch", return_value=mock_response):
result = client.post_mr_note(123, "New comment")
assert result["id"] == 1002
def test_get_mr_pipelines(self, client):
"""Test getting MR pipelines."""
mock_response = [
{"id": 1001, "status": "success", "ref": "feature"},
]
with patch.object(client, "_fetch", return_value=mock_response):
result = client.get_mr_pipelines(123)
assert len(result) == 1
def test_get_pipeline(self, client):
"""Test getting pipeline details."""
mock_response = {
"id": 1001,
"status": "success",
"ref": "main",
"sha": "abc123",
}
with patch.object(client, "_fetch", return_value=mock_response):
result = client.get_pipeline_status(1001)
assert result["id"] == 1001
def test_get_pipeline_jobs(self, client):
"""Test getting pipeline jobs."""
mock_response = [
{"id": 2001, "name": "test", "stage": "test", "status": "passed"},
{"id": 2002, "name": "build", "stage": "build", "status": "failed"},
]
with patch.object(client, "_fetch", return_value=mock_response):
result = client.get_pipeline_jobs(1001)
assert len(result) == 2
assert result[1]["status"] == "failed"
def test_get_issue(self, client):
"""Test getting issue details."""
mock_response = {
"iid": 456,
"title": "Test Issue",
"description": "Issue description",
"state": "opened",
}
with patch.object(client, "_fetch", return_value=mock_response):
result = client.get_issue(456)
assert result["iid"] == 456
def test_list_issues(self, client):
"""Test listing issues."""
mock_response = [
{"iid": 456, "title": "Issue 1"},
{"iid": 457, "title": "Issue 2"},
]
with patch.object(client, "_fetch", return_value=mock_response):
result = client.list_issues(state="opened")
assert len(result) == 2
def test_post_issue_note(self, client):
"""Test posting note to issue."""
mock_response = {"id": 2001, "body": "Issue comment"}
with patch.object(client, "_fetch", return_value=mock_response):
result = client.post_issue_note(456, "Issue comment")
assert result["id"] == 2001
def test_get_file(self, client):
"""Test getting file from repository."""
mock_response = {
"file_name": "README.md",
"content": "SGVsbG8gV29ybGQ=", # Base64 encoded
"encoding": "base64",
}
with patch.object(client, "_fetch", return_value=mock_response):
result = client.get_file_contents("README.md", ref="main")
assert result["file_name"] == "README.md"
def test_list_projects(self, client):
"""Test listing projects - removed in new API."""
# This method was removed from the new GitLabClient API
# Projects are now specified via the config
assert client.config.project is not None
class TestGitLabClientAuth:
"""Test GitLab client authentication."""
def test_token_in_headers(self):
"""Test token is included in request headers."""
import dataclasses
from __tests__.fixtures.gitlab import create_mock_client
client = create_mock_client()
client.config = dataclasses.replace(client.config, token="test-token-12345")
with patch("urllib.request.urlopen") as mock_urlopen:
# Mock response object with proper attributes
mock_response = Mock()
mock_response.read.return_value = b'{"iid": 123}'
mock_response.headers = {"Content-Type": "application/json"}
mock_response.status = 200
mock_response.__enter__ = Mock(return_value=mock_response)
mock_response.__exit__ = Mock(return_value=False)
mock_urlopen.return_value = mock_response
client.get_mr(123)
# Check that urlopen was called
assert mock_urlopen.called
# Get the request object that was passed to urlopen
call_args = mock_urlopen.call_args[0]
request = call_args[0]
# Check the PRIVATE-TOKEN header (case-insensitive check)
assert (
"PRIVATE-TOKEN" in request.headers or "Private-token" in request.headers
)
# Use get() with case-insensitive fallback
token_value = request.headers.get(
"PRIVATE-TOKEN", request.headers.get("Private-token")
)
assert token_value == "test-token-12345"
def test_custom_instance_url(self):
"""Test custom instance URL."""
import dataclasses
from __tests__.fixtures.gitlab import create_mock_client
client = create_mock_client()
client.config = dataclasses.replace(
client.config, instance_url="https://gitlab.custom.com"
)
with patch("urllib.request.urlopen") as mock_urlopen:
# Mock response object with proper attributes
mock_response = Mock()
mock_response.read.return_value = b'{"iid": 123}'
mock_response.headers = {"Content-Type": "application/json"}
mock_response.status = 200
mock_response.__enter__ = Mock(return_value=mock_response)
mock_response.__exit__ = Mock(return_value=False)
mock_urlopen.return_value = mock_response
client.get_mr(123)
# Check that urlopen was called with correct URL
call_args = mock_urlopen.call_args[0]
request = call_args[0]
assert "gitlab.custom.com" in request.full_url
class TestGitLabClientConfig:
"""Test GitLab configuration model."""
def test_config_creation(self):
"""Test creating GitLab config."""
from runners.gitlab.glab_client import GitLabConfig
config = GitLabConfig(
token="test-token",
project="group/project",
instance_url="https://gitlab.example.com",
)
assert config.token == "test-token"
assert config.project == "group/project"
def test_config_defaults(self):
"""Test config has sensible defaults."""
import tempfile
from pathlib import Path
from runners.gitlab.glab_client import GitLabClient, GitLabConfig
project_dir = Path(tempfile.mkdtemp())
config = GitLabConfig(
token="test-token",
project="group/project",
instance_url="https://gitlab.com",
)
client = GitLabClient(project_dir=project_dir, config=config)
assert client.config.instance_url == "https://gitlab.com"
assert client.default_timeout == 30.0
def test_config_to_dict(self):
"""Test converting config to dict using dataclasses."""
import dataclasses
from runners.gitlab.glab_client import GitLabConfig
config = GitLabConfig(
token="test-token",
project="group/project",
instance_url="https://gitlab.com",
)
data = dataclasses.asdict(config)
assert data["token"] == "test-token"
assert data["project"] == "group/project"
def test_config_from_dict(self):
"""Test loading config from dict using dataclasses."""
import dataclasses
from runners.gitlab.glab_client import GitLabConfig
data = {
"token": "test-token",
"project": "group/project",
"instance_url": "https://gitlab.example.com",
}
config = GitLabConfig(**data)
assert config.token == "test-token"
assert config.instance_url == "https://gitlab.example.com"
class TestGitLabClientErrorHandling:
"""Test GitLab client error handling."""
@pytest.fixture
def client(self):
"""Create a GitLab client for testing."""
from __tests__.fixtures.gitlab import create_mock_client
return create_mock_client()
def test_http_404_handling(self, client):
"""Test 404 error handling."""
from urllib.error import HTTPError
def mock_request(*args, **kwargs):
raise HTTPError(None, 404, "404 Not Found", {}, None)
with patch.object(client, "_fetch", mock_request):
with pytest.raises(HTTPError):
client.get_mr(99999)
def test_http_403_handling(self, client):
"""Test 403 forbidden error handling."""
from urllib.error import HTTPError
def mock_request(*args, **kwargs):
raise HTTPError(None, 403, "403 Forbidden", {}, None)
with patch.object(client, "_fetch", mock_request):
with pytest.raises(HTTPError):
client.get_mr(123)
def test_network_error_handling(self, client):
"""Test network error handling."""
with patch.object(
client, "_fetch", side_effect=ConnectionError("Network error")
):
with pytest.raises(ConnectionError):
client.get_mr(123)
def test_timeout_handling(self, client):
"""Test timeout handling."""
from socket import timeout
with patch.object(
client, "_fetch", side_effect=TimeoutError("Request timed out")
):
with pytest.raises(timeout):
client.get_mr(123)
+3
View File
@@ -0,0 +1,3 @@
"""Backward compatibility shim - import from core.agent instead."""
from core.agent import * # noqa: F403
+152
View File
@@ -0,0 +1,152 @@
# Agents Module
Modular agent system for autonomous coding. This module refactors the original monolithic `agent.py` (1,446 lines) into focused, maintainable modules.
## Architecture
The agent system is now organized by concern:
```
auto-claude/agents/
├── __init__.py # Public API exports
├── base.py # Shared constants and imports
├── utils.py # Git operations and plan management
├── memory.py # Memory management (Graphiti + file-based)
├── session.py # Agent session execution
├── planner.py # Follow-up planner logic
└── coder.py # Main autonomous agent loop
```
## Modules
### `base.py` (352 bytes)
- Shared constants (`AUTO_CONTINUE_DELAY_SECONDS`, `HUMAN_INTERVENTION_FILE`)
- Common imports and logging setup
### `utils.py` (3.6 KB)
- Git operations: `get_latest_commit()`, `get_commit_count()`
- Plan management: `load_implementation_plan()`, `find_subtask_in_plan()`, `find_phase_for_subtask()`
- Workspace sync: `sync_spec_to_source()`
### `memory.py` (13 KB)
- Dual-layer memory system (Graphiti primary, file-based fallback)
- `debug_memory_system_status()` - Memory system diagnostics
- `get_graphiti_context()` - Retrieve relevant context for subtasks
- `save_session_memory()` - Save session insights to memory
- `save_session_to_graphiti()` - Backwards compatibility wrapper
### `session.py` (17 KB)
- `run_agent_session()` - Execute a single agent session
- `post_session_processing()` - Process results and update memory
- Session logging and tool tracking
- Recovery manager integration
### `planner.py` (5.4 KB)
- `run_followup_planner()` - Add new subtasks to completed specs
- Follow-up planning workflow
- Plan validation and status updates
### `coder.py` (16 KB)
- `run_autonomous_agent()` - Main autonomous agent loop
- Planning and coding phase management
- Linear integration
- Recovery and stuck subtask handling
## Public API
The `agents` module exports a clean public API:
```python
from agents import (
# Main functions
run_autonomous_agent,
run_followup_planner,
# Memory functions
save_session_memory,
get_graphiti_context,
# Session management
run_agent_session,
post_session_processing,
# Utilities
get_latest_commit,
load_implementation_plan,
sync_spec_to_source,
)
```
## Backwards Compatibility
The original `agent.py` is now a facade that re-exports everything from the `agents` module:
```python
# Old code still works
from agent import run_autonomous_agent, save_session_memory
# New code can use modular imports
from agents.coder import run_autonomous_agent
from agents.memory import save_session_memory
```
All existing imports continue to work without changes.
## Benefits
1. **Separation of Concerns**: Each module has a clear, focused responsibility
2. **Maintainability**: Easier to understand and modify individual components
3. **Testability**: Modules can be tested in isolation
4. **Backwards Compatible**: No breaking changes to existing code
5. **Scalability**: Easy to add new agent types or features
## Module Dependencies
```
coder.py
├── session.py (run_agent_session, post_session_processing)
├── memory.py (get_graphiti_context, debug_memory_system_status)
└── utils.py (git operations, plan management)
session.py
├── memory.py (save_session_memory)
└── utils.py (git operations, plan management)
planner.py
└── session.py (run_agent_session)
memory.py
└── base.py (constants, logging)
```
## Testing
Run the verification script to test the refactoring:
```bash
python3 auto-claude/agents/test_refactoring.py
```
This verifies:
- Module structure is correct
- All imports work
- Public API is accessible
- Backwards compatibility is maintained
## Migration Guide
No migration needed! The refactoring maintains 100% backwards compatibility.
### For new code:
```python
# Use focused imports for clarity
from agents.coder import run_autonomous_agent
from agents.memory import save_session_memory, get_graphiti_context
from agents.session import run_agent_session
```
### For existing code:
```python
# Old imports continue to work
from agent import run_autonomous_agent, save_session_memory
```
+96
View File
@@ -0,0 +1,96 @@
"""
Agents Module
=============
Modular agent system for autonomous coding.
This module provides:
- run_autonomous_agent: Main coder agent loop
- run_followup_planner: Follow-up planner for completed specs
- Memory management (Graphiti + file-based fallback)
- Session management and post-processing
- Utility functions for git and plan management
Uses lazy imports to avoid circular dependencies.
"""
# Explicit import required by CodeQL static analysis
# (CodeQL doesn't recognize __getattr__ dynamic exports)
from .utils import sync_spec_to_source
__all__ = [
# Main API
"run_autonomous_agent",
"run_followup_planner",
# Memory
"debug_memory_system_status",
"get_graphiti_context",
"save_session_memory",
"save_session_to_graphiti",
# Session
"run_agent_session",
"post_session_processing",
# Utils
"get_latest_commit",
"get_commit_count",
"load_implementation_plan",
"find_subtask_in_plan",
"find_phase_for_subtask",
"sync_spec_to_source",
# Constants
"AUTO_CONTINUE_DELAY_SECONDS",
"HUMAN_INTERVENTION_FILE",
]
def __getattr__(name):
"""Lazy imports to avoid circular dependencies."""
if name in ("AUTO_CONTINUE_DELAY_SECONDS", "HUMAN_INTERVENTION_FILE"):
from .base import AUTO_CONTINUE_DELAY_SECONDS, HUMAN_INTERVENTION_FILE
return locals()[name]
elif name == "run_autonomous_agent":
from .coder import run_autonomous_agent
return run_autonomous_agent
elif name in (
"debug_memory_system_status",
"get_graphiti_context",
"save_session_memory",
"save_session_to_graphiti",
):
from .memory_manager import (
debug_memory_system_status,
get_graphiti_context,
save_session_memory,
save_session_to_graphiti,
)
return locals()[name]
elif name == "run_followup_planner":
from .planner import run_followup_planner
return run_followup_planner
elif name in ("post_session_processing", "run_agent_session"):
from .session import post_session_processing, run_agent_session
return locals()[name]
elif name in (
"find_phase_for_subtask",
"find_subtask_in_plan",
"get_commit_count",
"get_latest_commit",
"load_implementation_plan",
"sync_spec_to_source",
):
from .utils import (
find_phase_for_subtask,
find_subtask_in_plan,
get_commit_count,
get_latest_commit,
load_implementation_plan,
sync_spec_to_source,
)
return locals()[name]
raise AttributeError(f"module 'agents' has no attribute '{name}'")
+15
View File
@@ -0,0 +1,15 @@
"""
Base Module for Agent System
=============================
Shared imports, types, and constants used across agent modules.
"""
import logging
# Configure logging
logger = logging.getLogger(__name__)
# Configuration constants
AUTO_CONTINUE_DELAY_SECONDS = 3
HUMAN_INTERVENTION_FILE = "PAUSE"
+616
View File
@@ -0,0 +1,616 @@
"""
Coder Agent Module
==================
Main autonomous agent loop that runs the coder agent to implement subtasks.
"""
import asyncio
import logging
import os
from pathlib import Path
from core.client import create_client
from linear_updater import (
LinearTaskState,
is_linear_enabled,
linear_build_complete,
linear_task_started,
linear_task_stuck,
)
from phase_config import get_phase_model, get_phase_thinking_budget
from phase_event import ExecutionPhase, emit_phase
from progress import (
count_subtasks,
count_subtasks_detailed,
get_current_phase,
get_next_subtask,
is_build_complete,
print_build_complete_banner,
print_progress_summary,
print_session_header,
)
from prompt_generator import (
format_context_for_prompt,
generate_planner_prompt,
generate_subtask_prompt,
load_subtask_context,
)
from prompts import is_first_run
from recovery import RecoveryManager
from security.constants import PROJECT_DIR_ENV_VAR
from task_logger import (
LogPhase,
get_task_logger,
)
from ui import (
BuildState,
Icons,
StatusManager,
bold,
box,
highlight,
icon,
muted,
print_key_value,
print_status,
)
from .base import AUTO_CONTINUE_DELAY_SECONDS, HUMAN_INTERVENTION_FILE
from .memory_manager import debug_memory_system_status, get_graphiti_context
from .session import post_session_processing, run_agent_session
from .utils import (
find_phase_for_subtask,
get_commit_count,
get_latest_commit,
load_implementation_plan,
sync_spec_to_source,
)
logger = logging.getLogger(__name__)
async def run_autonomous_agent(
project_dir: Path,
spec_dir: Path,
model: str,
max_iterations: int | None = None,
verbose: bool = False,
source_spec_dir: Path | None = None,
) -> None:
"""
Run the autonomous agent loop with automatic memory management.
The agent can use subagents (via Task tool) for parallel execution if needed.
This is decided by the agent itself based on the task complexity.
Args:
project_dir: Root directory for the project
spec_dir: Directory containing the spec (auto-claude/specs/001-name/)
model: Claude model to use
max_iterations: Maximum number of iterations (None for unlimited)
verbose: Whether to show detailed output
source_spec_dir: Original spec directory in main project (for syncing from worktree)
"""
# Set environment variable for security hooks to find the correct project directory
# This is needed because os.getcwd() may return the wrong directory in worktree mode
os.environ[PROJECT_DIR_ENV_VAR] = str(project_dir.resolve())
# Initialize recovery manager (handles memory persistence)
recovery_manager = RecoveryManager(spec_dir, project_dir)
# Initialize status manager for ccstatusline
status_manager = StatusManager(project_dir)
status_manager.set_active(spec_dir.name, BuildState.BUILDING)
# Initialize task logger for persistent logging
task_logger = get_task_logger(spec_dir)
# Debug: Print memory system status at startup
debug_memory_system_status()
# Update initial subtask counts
subtasks = count_subtasks_detailed(spec_dir)
status_manager.update_subtasks(
completed=subtasks["completed"],
total=subtasks["total"],
in_progress=subtasks["in_progress"],
)
# Check Linear integration status
linear_task = None
if is_linear_enabled():
linear_task = LinearTaskState.load(spec_dir)
if linear_task and linear_task.task_id:
print_status("Linear integration: ENABLED", "success")
print_key_value("Task", linear_task.task_id)
print_key_value("Status", linear_task.status)
print()
else:
print_status("Linear enabled but no task created for this spec", "warning")
print()
# Check if this is a fresh start or continuation
first_run = is_first_run(spec_dir)
# Track which phase we're in for logging
current_log_phase = LogPhase.CODING
is_planning_phase = False
planning_retry_context: str | None = None
planning_validation_failures = 0
max_planning_validation_retries = 3
def _validate_and_fix_implementation_plan() -> tuple[bool, list[str]]:
from spec.validate_pkg import SpecValidator, auto_fix_plan
spec_validator = SpecValidator(spec_dir)
result = spec_validator.validate_implementation_plan()
if result.valid:
return True, []
fixed = auto_fix_plan(spec_dir)
if fixed:
result = spec_validator.validate_implementation_plan()
if result.valid:
return True, []
return False, result.errors
if first_run:
print_status(
"Fresh start - will use Planner Agent to create implementation plan", "info"
)
content = [
bold(f"{icon(Icons.GEAR)} PLANNER SESSION"),
"",
f"Spec: {highlight(spec_dir.name)}",
muted("The agent will analyze your spec and create a subtask-based plan."),
]
print()
print(box(content, width=70, style="heavy"))
print()
# Update status for planning phase
status_manager.update(state=BuildState.PLANNING)
emit_phase(ExecutionPhase.PLANNING, "Creating implementation plan")
is_planning_phase = True
current_log_phase = LogPhase.PLANNING
# Start planning phase in task logger
if task_logger:
task_logger.start_phase(
LogPhase.PLANNING, "Starting implementation planning..."
)
# Update Linear to "In Progress" when build starts
if linear_task and linear_task.task_id:
print_status("Updating Linear task to In Progress...", "progress")
await linear_task_started(spec_dir)
else:
print(f"Continuing build: {highlight(spec_dir.name)}")
print_progress_summary(spec_dir)
# Check if already complete
if is_build_complete(spec_dir):
print_build_complete_banner(spec_dir)
status_manager.update(state=BuildState.COMPLETE)
return
# Start/continue coding phase in task logger
if task_logger:
task_logger.start_phase(LogPhase.CODING, "Continuing implementation...")
# Emit phase event when continuing build
emit_phase(ExecutionPhase.CODING, "Continuing implementation")
# Show human intervention hint
content = [
bold("INTERACTIVE CONTROLS"),
"",
f"Press {highlight('Ctrl+C')} once {icon(Icons.ARROW_RIGHT)} Pause and optionally add instructions",
f"Press {highlight('Ctrl+C')} twice {icon(Icons.ARROW_RIGHT)} Exit immediately",
]
print(box(content, width=70, style="light"))
print()
# Main loop
iteration = 0
while True:
iteration += 1
# Check for human intervention (PAUSE file)
pause_file = spec_dir / HUMAN_INTERVENTION_FILE
if pause_file.exists():
print("\n" + "=" * 70)
print(" PAUSED BY HUMAN")
print("=" * 70)
pause_content = pause_file.read_text(encoding="utf-8").strip()
if pause_content:
print(f"\nMessage: {pause_content}")
print("\nTo resume, delete the PAUSE file:")
print(f" rm {pause_file}")
print("\nThen run again:")
print(f" python auto-claude/run.py --spec {spec_dir.name}")
return
# Check max iterations
if max_iterations and iteration > max_iterations:
print(f"\nReached max iterations ({max_iterations})")
print("To continue, run the script again without --max-iterations")
break
# Get the next subtask to work on (planner sessions shouldn't bind to a subtask)
next_subtask = None if first_run else get_next_subtask(spec_dir)
subtask_id = next_subtask.get("id") if next_subtask else None
phase_name = next_subtask.get("phase_name") if next_subtask else None
# Update status for this session
status_manager.update_session(iteration)
if phase_name:
current_phase = get_current_phase(spec_dir)
if current_phase:
status_manager.update_phase(
current_phase.get("name", ""),
current_phase.get("phase", 0),
current_phase.get("total", 0),
)
status_manager.update_subtasks(in_progress=1)
# Print session header
print_session_header(
session_num=iteration,
is_planner=first_run,
subtask_id=subtask_id,
subtask_desc=next_subtask.get("description") if next_subtask else None,
phase_name=phase_name,
attempt=recovery_manager.get_attempt_count(subtask_id) + 1
if subtask_id
else 1,
)
# Capture state before session for post-processing
commit_before = get_latest_commit(project_dir)
commit_count_before = get_commit_count(project_dir)
# Get the phase-specific model and thinking level (respects task_metadata.json configuration)
# first_run means we're in planning phase, otherwise coding phase
current_phase = "planning" if first_run else "coding"
phase_model = get_phase_model(spec_dir, current_phase, model)
phase_thinking_budget = get_phase_thinking_budget(spec_dir, current_phase)
# Create client (fresh context) with phase-specific model and thinking
# Use appropriate agent_type for correct tool permissions and thinking budget
client = create_client(
project_dir,
spec_dir,
phase_model,
agent_type="planner" if first_run else "coder",
max_thinking_tokens=phase_thinking_budget,
)
# Generate appropriate prompt
if first_run:
prompt = generate_planner_prompt(spec_dir, project_dir)
if planning_retry_context:
prompt += "\n\n" + planning_retry_context
# Retrieve Graphiti memory context for planning phase
# This gives the planner knowledge of previous patterns, gotchas, and insights
planner_context = await get_graphiti_context(
spec_dir,
project_dir,
{
"description": "Planning implementation for new feature",
"id": "planner",
},
)
if planner_context:
prompt += "\n\n" + planner_context
print_status("Graphiti memory context loaded for planner", "success")
first_run = False
current_log_phase = LogPhase.PLANNING
# Set session info in logger
if task_logger:
task_logger.set_session(iteration)
else:
# Switch to coding phase after planning
just_transitioned_from_planning = False
if is_planning_phase:
just_transitioned_from_planning = True
is_planning_phase = False
current_log_phase = LogPhase.CODING
emit_phase(ExecutionPhase.CODING, "Starting implementation")
if task_logger:
task_logger.end_phase(
LogPhase.PLANNING,
success=True,
message="Implementation plan created",
)
task_logger.start_phase(
LogPhase.CODING, "Starting implementation..."
)
# In worktree mode, the UI prefers planning logs from the main spec dir.
# Ensure the planning->coding transition is immediately reflected there.
if sync_spec_to_source(spec_dir, source_spec_dir):
print_status("Phase transition synced to main project", "success")
if not next_subtask:
# FIX for Issue #495: Race condition after planning phase
# The implementation_plan.json may not be fully flushed to disk yet,
# or there may be a brief delay before subtasks become available.
# Retry with exponential backoff before giving up.
if just_transitioned_from_planning:
print_status(
"Waiting for implementation plan to be ready...", "progress"
)
for retry_attempt in range(3):
delay = (retry_attempt + 1) * 2 # 2s, 4s, 6s
await asyncio.sleep(delay)
next_subtask = get_next_subtask(spec_dir)
if next_subtask:
# Update subtask_id and phase_name after successful retry
subtask_id = next_subtask.get("id")
phase_name = next_subtask.get("phase_name")
print_status(
f"Found subtask {subtask_id} after {delay}s delay",
"success",
)
break
print_status(
f"Retry {retry_attempt + 1}/3: No subtask found yet...",
"warning",
)
if not next_subtask:
print("No pending subtasks found - build may be complete!")
break
# Get attempt count for recovery context
attempt_count = recovery_manager.get_attempt_count(subtask_id)
recovery_hints = (
recovery_manager.get_recovery_hints(subtask_id)
if attempt_count > 0
else None
)
# Find the phase for this subtask
plan = load_implementation_plan(spec_dir)
phase = find_phase_for_subtask(plan, subtask_id) if plan else {}
# Generate focused, minimal prompt for this subtask
prompt = generate_subtask_prompt(
spec_dir=spec_dir,
project_dir=project_dir,
subtask=next_subtask,
phase=phase or {},
attempt_count=attempt_count,
recovery_hints=recovery_hints,
)
# Load and append relevant file context
context = load_subtask_context(spec_dir, project_dir, next_subtask)
if context.get("patterns") or context.get("files_to_modify"):
prompt += "\n\n" + format_context_for_prompt(context)
# Retrieve and append Graphiti memory context (if enabled)
graphiti_context = await get_graphiti_context(
spec_dir, project_dir, next_subtask
)
if graphiti_context:
prompt += "\n\n" + graphiti_context
print_status("Graphiti memory context loaded", "success")
# Show what we're working on
print(f"Working on: {highlight(subtask_id)}")
print(f"Description: {next_subtask.get('description', 'No description')}")
if attempt_count > 0:
print_status(f"Previous attempts: {attempt_count}", "warning")
print()
# Set subtask info in logger
if task_logger and subtask_id:
task_logger.set_subtask(subtask_id)
task_logger.set_session(iteration)
# Run session with async context manager
async with client:
status, response = await run_agent_session(
client, prompt, spec_dir, verbose, phase=current_log_phase
)
plan_validated = False
if is_planning_phase and status != "error":
valid, errors = _validate_and_fix_implementation_plan()
if valid:
plan_validated = True
planning_retry_context = None
else:
planning_validation_failures += 1
if planning_validation_failures >= max_planning_validation_retries:
print_status(
"implementation_plan.json validation failed too many times",
"error",
)
for err in errors:
print(f" - {err}")
status_manager.update(state=BuildState.ERROR)
return
print_status(
"implementation_plan.json invalid - retrying planner", "warning"
)
for err in errors:
print(f" - {err}")
planning_retry_context = (
"## IMPLEMENTATION PLAN VALIDATION ERRORS\n\n"
"The previous `implementation_plan.json` is INVALID.\n"
"You MUST rewrite it to match the required schema:\n"
"- Top-level: `feature`, `workflow_type`, `phases`\n"
"- Each phase: `id` (or `phase`) and `name`, and `subtasks`\n"
"- Each subtask: `id`, `description`, `status` (use `pending` for not started)\n\n"
"Validation errors:\n" + "\n".join(f"- {e}" for e in errors)
)
# Stay in planning mode for the next iteration
first_run = True
status = "continue"
# === POST-SESSION PROCESSING (100% reliable) ===
# Only run post-session processing for coding sessions.
if subtask_id and current_log_phase == LogPhase.CODING:
linear_is_enabled = (
linear_task is not None and linear_task.task_id is not None
)
success = await post_session_processing(
spec_dir=spec_dir,
project_dir=project_dir,
subtask_id=subtask_id,
session_num=iteration,
commit_before=commit_before,
commit_count_before=commit_count_before,
recovery_manager=recovery_manager,
linear_enabled=linear_is_enabled,
status_manager=status_manager,
source_spec_dir=source_spec_dir,
)
# Check for stuck subtasks
attempt_count = recovery_manager.get_attempt_count(subtask_id)
if not success and attempt_count >= 3:
recovery_manager.mark_subtask_stuck(
subtask_id, f"Failed after {attempt_count} attempts"
)
print()
print_status(
f"Subtask {subtask_id} marked as STUCK after {attempt_count} attempts",
"error",
)
print(muted("Consider: manual intervention or skipping this subtask"))
# Record stuck subtask in Linear (if enabled)
if linear_is_enabled:
await linear_task_stuck(
spec_dir=spec_dir,
subtask_id=subtask_id,
attempt_count=attempt_count,
)
print_status("Linear notified of stuck subtask", "info")
elif plan_validated and source_spec_dir:
# After planning phase, sync the newly created implementation plan back to source
if sync_spec_to_source(spec_dir, source_spec_dir):
print_status("Implementation plan synced to main project", "success")
# Handle session status
if status == "complete":
# Don't emit COMPLETE here - subtasks are done but QA hasn't run yet
# QA loop will emit COMPLETE after actual approval
print_build_complete_banner(spec_dir)
status_manager.update(state=BuildState.COMPLETE)
if task_logger:
task_logger.end_phase(
LogPhase.CODING,
success=True,
message="All subtasks completed successfully",
)
if linear_task and linear_task.task_id:
await linear_build_complete(spec_dir)
print_status("Linear notified: build complete, ready for QA", "success")
break
elif status == "continue":
print(
muted(
f"\nAgent will auto-continue in {AUTO_CONTINUE_DELAY_SECONDS}s..."
)
)
print_progress_summary(spec_dir)
# Update state back to building
status_manager.update(
state=BuildState.PLANNING if is_planning_phase else BuildState.BUILDING
)
# Show next subtask info
next_subtask = get_next_subtask(spec_dir)
if next_subtask:
subtask_id = next_subtask.get("id")
print(
f"\nNext: {highlight(subtask_id)} - {next_subtask.get('description')}"
)
attempt_count = recovery_manager.get_attempt_count(subtask_id)
if attempt_count > 0:
print_status(
f"WARNING: {attempt_count} previous attempt(s)", "warning"
)
await asyncio.sleep(AUTO_CONTINUE_DELAY_SECONDS)
elif status == "error":
emit_phase(ExecutionPhase.FAILED, "Session encountered an error")
print_status("Session encountered an error", "error")
print(muted("Will retry with a fresh session..."))
status_manager.update(state=BuildState.ERROR)
await asyncio.sleep(AUTO_CONTINUE_DELAY_SECONDS)
# Small delay between sessions
if max_iterations is None or iteration < max_iterations:
print("\nPreparing next session...\n")
await asyncio.sleep(1)
# Final summary
content = [
bold(f"{icon(Icons.SESSION)} SESSION SUMMARY"),
"",
f"Project: {project_dir}",
f"Spec: {highlight(spec_dir.name)}",
f"Sessions completed: {iteration}",
]
print()
print(box(content, width=70, style="heavy"))
print_progress_summary(spec_dir)
# Show stuck subtasks if any
stuck_subtasks = recovery_manager.get_stuck_subtasks()
if stuck_subtasks:
print()
print_status("STUCK SUBTASKS (need manual intervention):", "error")
for stuck in stuck_subtasks:
print(f" {icon(Icons.ERROR)} {stuck['subtask_id']}: {stuck['reason']}")
# Instructions
completed, total = count_subtasks(spec_dir)
if completed < total:
content = [
bold(f"{icon(Icons.PLAY)} NEXT STEPS"),
"",
f"{total - completed} subtasks remaining.",
f"Run again: {highlight(f'python auto-claude/run.py --spec {spec_dir.name}')}",
]
else:
content = [
bold(f"{icon(Icons.SUCCESS)} NEXT STEPS"),
"",
"All subtasks completed!",
" 1. Review the auto-claude/* branch",
" 2. Run manual tests",
" 3. Merge to main",
]
print()
print(box(content, width=70, style="light"))
print()
# Set final status
if completed == total:
status_manager.update(state=BuildState.COMPLETE)
else:
status_manager.update(state=BuildState.PAUSED)
+494
View File
@@ -0,0 +1,494 @@
"""
Memory Management for Agent System
===================================
Handles session memory storage using dual-layer approach:
- PRIMARY: Graphiti (when enabled) - semantic search, cross-session context
- FALLBACK: File-based memory - zero dependencies, always available
"""
import logging
from pathlib import Path
from core.sentry import capture_exception
from debug import (
debug,
debug_detailed,
debug_error,
debug_section,
debug_success,
debug_warning,
is_debug_enabled,
)
from graphiti_config import get_graphiti_status, is_graphiti_enabled
# Import from parent memory package
# Now safe since this module is named memory_manager (not memory)
from memory import save_session_insights as save_file_based_memory
from memory.graphiti_helpers import get_graphiti_memory
logger = logging.getLogger(__name__)
def debug_memory_system_status() -> None:
"""
Print memory system status for debugging.
Called at startup when DEBUG=true to show memory configuration.
"""
if not is_debug_enabled():
return
debug_section("memory", "Memory System Status")
# Get Graphiti status
graphiti_status = get_graphiti_status()
debug(
"memory",
"Memory system configuration",
primary_system="Graphiti"
if graphiti_status.get("available")
else "File-based (fallback)",
graphiti_enabled=graphiti_status.get("enabled"),
graphiti_available=graphiti_status.get("available"),
)
if graphiti_status.get("enabled"):
debug_detailed(
"memory",
"Graphiti configuration",
host=graphiti_status.get("host"),
port=graphiti_status.get("port"),
database=graphiti_status.get("database"),
llm_provider=graphiti_status.get("llm_provider"),
embedder_provider=graphiti_status.get("embedder_provider"),
)
if not graphiti_status.get("available"):
debug_warning(
"memory",
"Graphiti not available",
reason=graphiti_status.get("reason"),
errors=graphiti_status.get("errors"),
)
debug("memory", "Will use file-based memory as fallback")
else:
debug_success("memory", "Graphiti ready as PRIMARY memory system")
else:
debug(
"memory",
"Graphiti disabled, using file-based memory only",
note="Set GRAPHITI_ENABLED=true to enable Graphiti",
)
async def get_graphiti_context(
spec_dir: Path,
project_dir: Path,
subtask: dict,
) -> str | None:
"""
Retrieve relevant context from Graphiti for the current subtask.
This searches the knowledge graph for context relevant to the subtask's
task description, returning past insights, patterns, and gotchas.
Args:
spec_dir: Spec directory
project_dir: Project root directory
subtask: The current subtask being worked on
Returns:
Formatted context string or None if unavailable
"""
if is_debug_enabled():
debug(
"memory",
"Retrieving Graphiti context for subtask",
subtask_id=subtask.get("id", "unknown"),
subtask_desc=subtask.get("description", "")[:100],
)
if not is_graphiti_enabled():
if is_debug_enabled():
debug("memory", "Graphiti not enabled, skipping context retrieval")
return None
memory = None
try:
# Use centralized helper for GraphitiMemory instantiation (async)
memory = await get_graphiti_memory(spec_dir, project_dir)
if memory is None:
if is_debug_enabled():
debug_warning(
"memory", "GraphitiMemory not available for context retrieval"
)
return None
# Build search query from subtask description
subtask_desc = subtask.get("description", "")
subtask_id = subtask.get("id", "")
query = f"{subtask_desc} {subtask_id}".strip()
if not query:
if is_debug_enabled():
debug_warning("memory", "Empty query, skipping context retrieval")
return None
if is_debug_enabled():
debug_detailed(
"memory",
"Searching Graphiti knowledge graph",
query=query[:200],
num_results=5,
)
# Get relevant context
context_items = await memory.get_relevant_context(query, num_results=5)
# Get patterns and gotchas specifically (THE FIX for learning loop!)
# This retrieves PATTERN and GOTCHA episode types for cross-session learning
patterns, gotchas = await memory.get_patterns_and_gotchas(
query, num_results=3, min_score=0.5
)
# Also get recent session history
session_history = await memory.get_session_history(limit=3)
if is_debug_enabled():
debug(
"memory",
"Graphiti context retrieval complete",
context_items_found=len(context_items) if context_items else 0,
patterns_found=len(patterns) if patterns else 0,
gotchas_found=len(gotchas) if gotchas else 0,
session_history_found=len(session_history) if session_history else 0,
)
if not context_items and not session_history and not patterns and not gotchas:
if is_debug_enabled():
debug("memory", "No relevant context found in Graphiti")
return None
# Format the context
sections = ["## Graphiti Memory Context\n"]
sections.append("_Retrieved from knowledge graph for this subtask:_\n")
if context_items:
sections.append("### Relevant Knowledge\n")
for item in context_items:
content = item.get("content", "")[:500] # Truncate
item_type = item.get("type", "unknown")
sections.append(f"- **[{item_type}]** {content}\n")
# Add patterns section (cross-session learning)
if patterns:
sections.append("### Learned Patterns\n")
sections.append("_Patterns discovered in previous sessions:_\n")
for p in patterns:
pattern_text = p.get("pattern", "")
applies_to = p.get("applies_to", "")
if applies_to:
sections.append(
f"- **Pattern**: {pattern_text}\n _Applies to:_ {applies_to}\n"
)
else:
sections.append(f"- **Pattern**: {pattern_text}\n")
# Add gotchas section (cross-session learning)
if gotchas:
sections.append("### Known Gotchas\n")
sections.append("_Pitfalls to avoid:_\n")
for g in gotchas:
gotcha_text = g.get("gotcha", "")
solution = g.get("solution", "")
if solution:
sections.append(
f"- **Gotcha**: {gotcha_text}\n _Solution:_ {solution}\n"
)
else:
sections.append(f"- **Gotcha**: {gotcha_text}\n")
if session_history:
sections.append("### Recent Session Insights\n")
for session in session_history[:2]: # Only show last 2
session_num = session.get("session_number", "?")
recommendations = session.get("recommendations_for_next_session", [])
if recommendations:
sections.append(f"**Session {session_num} recommendations:**")
for rec in recommendations[:3]: # Limit to 3
sections.append(f"- {rec}")
sections.append("")
if is_debug_enabled():
debug_success(
"memory", "Graphiti context formatted", total_sections=len(sections)
)
return "\n".join(sections)
except Exception as e:
logger.warning(f"Failed to get Graphiti context: {e}")
if is_debug_enabled():
debug_error("memory", "Graphiti context retrieval failed", error=str(e))
# Capture exception to Sentry with full context
capture_exception(
e,
operation="get_graphiti_context",
subtask_id=subtask.get("id", "unknown"),
subtask_desc=subtask.get("description", "")[:200],
spec_dir=str(spec_dir),
project_dir=str(project_dir),
)
return None
finally:
# Always close the memory connection (swallow exceptions to avoid overriding)
if memory is not None:
try:
await memory.close()
except Exception as e:
logger.debug(
"Failed to close Graphiti memory connection", exc_info=True
)
async def save_session_memory(
spec_dir: Path,
project_dir: Path,
subtask_id: str,
session_num: int,
success: bool,
subtasks_completed: list[str],
discoveries: dict | None = None,
) -> tuple[bool, str]:
"""
Save session insights to memory.
Memory Strategy:
- PRIMARY: Graphiti (when enabled) - provides semantic search, cross-session context
- FALLBACK: File-based (when Graphiti is disabled) - zero dependencies, always works
This is called after each session to persist learnings.
Args:
spec_dir: Spec directory
project_dir: Project root directory
subtask_id: The subtask that was worked on
session_num: Current session number
success: Whether the subtask was completed successfully
subtasks_completed: List of subtask IDs completed this session
discoveries: Optional dict with file discoveries, patterns, gotchas
Returns:
Tuple of (success, storage_type) where storage_type is "graphiti" or "file"
"""
# Debug: Log memory save start
if is_debug_enabled():
debug_section("memory", f"Saving Session {session_num} Memory")
debug(
"memory",
"Memory save initiated",
subtask_id=subtask_id,
session_num=session_num,
success=success,
subtasks_completed=subtasks_completed,
spec_dir=str(spec_dir),
)
# Build insights structure (same format for both storage systems)
insights = {
"subtasks_completed": subtasks_completed,
"discoveries": discoveries
or {
"files_understood": {},
"patterns_found": [],
"gotchas_encountered": [],
},
"what_worked": [f"Implemented subtask: {subtask_id}"] if success else [],
"what_failed": [] if success else [f"Failed to complete subtask: {subtask_id}"],
"recommendations_for_next_session": [],
}
if is_debug_enabled():
debug_detailed("memory", "Insights structure built", insights=insights)
# Check Graphiti status for debugging
graphiti_enabled = is_graphiti_enabled()
if is_debug_enabled():
graphiti_status = get_graphiti_status()
debug(
"memory",
"Graphiti status check",
enabled=graphiti_status.get("enabled"),
available=graphiti_status.get("available"),
host=graphiti_status.get("host"),
port=graphiti_status.get("port"),
database=graphiti_status.get("database"),
llm_provider=graphiti_status.get("llm_provider"),
embedder_provider=graphiti_status.get("embedder_provider"),
reason=graphiti_status.get("reason") or "OK",
)
# PRIMARY: Try Graphiti if enabled
if graphiti_enabled:
if is_debug_enabled():
debug("memory", "Attempting PRIMARY storage: Graphiti")
memory = None
try:
# Use centralized helper for GraphitiMemory instantiation (async)
memory = await get_graphiti_memory(spec_dir, project_dir)
if memory is None:
if is_debug_enabled():
debug_warning("memory", "GraphitiMemory not available")
debug(
"memory",
"get_graphiti_memory() returned None - this usually means Graphiti is disabled or provider config is invalid",
)
# Continue to file-based fallback
if memory is not None and memory.is_enabled:
if is_debug_enabled():
debug("memory", "Saving to Graphiti...")
# Use structured insights if we have rich extracted data
if discoveries and discoveries.get("file_insights"):
# Rich insights from insight_extractor
if is_debug_enabled():
debug(
"memory",
"Using save_structured_insights (rich data available)",
)
result = await memory.save_structured_insights(discoveries)
else:
# Fallback to basic session insights
result = await memory.save_session_insights(session_num, insights)
if result:
logger.info(
f"Session {session_num} insights saved to Graphiti (primary)"
)
if is_debug_enabled():
debug_success(
"memory",
f"Session {session_num} saved to Graphiti (PRIMARY)",
storage_type="graphiti",
subtasks_saved=len(subtasks_completed),
)
return True, "graphiti"
else:
logger.warning(
"Graphiti save returned False, falling back to file-based"
)
if is_debug_enabled():
debug_warning(
"memory", "Graphiti save returned False, using FALLBACK"
)
elif memory is None:
if is_debug_enabled():
debug_warning(
"memory", "GraphitiMemory not available, using FALLBACK"
)
else:
# memory is not None but memory.is_enabled is False
logger.warning(
"GraphitiMemory.is_enabled=False, falling back to file-based"
)
if is_debug_enabled():
debug_warning("memory", "GraphitiMemory disabled, using FALLBACK")
except Exception as e:
logger.warning(f"Graphiti save failed: {e}, falling back to file-based")
if is_debug_enabled():
debug_error("memory", "Graphiti save failed", error=str(e))
# Capture exception to Sentry with full context
capture_exception(
e,
operation="save_session_memory_graphiti",
subtask_id=subtask_id,
session_num=session_num,
success=success,
subtasks_completed=subtasks_completed,
spec_dir=str(spec_dir),
project_dir=str(project_dir),
)
finally:
# Always close the memory connection (swallow exceptions to avoid overriding)
if memory is not None:
try:
await memory.close()
except Exception as e:
logger.debug(
"Failed to close Graphiti memory connection", exc_info=e
)
else:
if is_debug_enabled():
debug("memory", "Graphiti not enabled, skipping to FALLBACK")
# FALLBACK: File-based memory (when Graphiti is disabled or fails)
if is_debug_enabled():
debug("memory", "Attempting FALLBACK storage: File-based")
try:
memory_dir = spec_dir / "memory" / "session_insights"
if is_debug_enabled():
debug_detailed(
"memory",
"File-based memory path",
memory_dir=str(memory_dir),
session_file=f"session_{session_num:03d}.json",
)
save_file_based_memory(spec_dir, session_num, insights)
logger.info(
f"Session {session_num} insights saved to file-based memory (fallback)"
)
if is_debug_enabled():
debug_success(
"memory",
f"Session {session_num} saved to file-based (FALLBACK)",
storage_type="file",
file_path=str(memory_dir / f"session_{session_num:03d}.json"),
subtasks_saved=len(subtasks_completed),
)
return True, "file"
except Exception as e:
logger.error(f"File-based memory save also failed: {e}")
if is_debug_enabled():
debug_error("memory", "File-based memory save FAILED", error=str(e))
# Capture exception to Sentry with full context
capture_exception(
e,
operation="save_session_memory_file",
subtask_id=subtask_id,
session_num=session_num,
success=success,
subtasks_completed=subtasks_completed,
spec_dir=str(spec_dir),
project_dir=str(project_dir),
)
return False, "none"
# Keep the old function name as an alias for backwards compatibility
async def save_session_to_graphiti(
spec_dir: Path,
project_dir: Path,
subtask_id: str,
session_num: int,
success: bool,
subtasks_completed: list[str],
discoveries: dict | None = None,
) -> bool:
"""Backwards compatibility wrapper for save_session_memory."""
result, _ = await save_session_memory(
spec_dir,
project_dir,
subtask_id,
session_num,
success,
subtasks_completed,
discoveries,
)
return result
+183
View File
@@ -0,0 +1,183 @@
"""
Planner Agent Module
====================
Handles follow-up planner sessions for adding new subtasks to completed specs.
"""
import logging
from pathlib import Path
from core.client import create_client
from phase_config import get_phase_model, get_phase_thinking_budget
from phase_event import ExecutionPhase, emit_phase
from task_logger import (
LogPhase,
get_task_logger,
)
from ui import (
BuildState,
Icons,
StatusManager,
bold,
box,
highlight,
icon,
muted,
print_status,
)
from .session import run_agent_session
logger = logging.getLogger(__name__)
async def run_followup_planner(
project_dir: Path,
spec_dir: Path,
model: str,
verbose: bool = False,
) -> bool:
"""
Run the follow-up planner to add new subtasks to a completed spec.
This is a simplified version of run_autonomous_agent that:
1. Creates a client
2. Loads the followup planner prompt
3. Runs a single planning session
4. Returns after the plan is updated (doesn't enter coding loop)
The planner agent will:
- Read FOLLOWUP_REQUEST.md for the new task
- Read the existing implementation_plan.json
- Add new phase(s) with pending subtasks
- Update the plan status back to in_progress
Args:
project_dir: Root directory for the project
spec_dir: Directory containing the completed spec
model: Claude model to use
verbose: Whether to show detailed output
Returns:
bool: True if planning completed successfully
"""
from implementation_plan import ImplementationPlan
from prompts import get_followup_planner_prompt
# Initialize status manager for ccstatusline
status_manager = StatusManager(project_dir)
status_manager.set_active(spec_dir.name, BuildState.PLANNING)
emit_phase(ExecutionPhase.PLANNING, "Follow-up planning")
# Initialize task logger for persistent logging
task_logger = get_task_logger(spec_dir)
# Show header
content = [
bold(f"{icon(Icons.GEAR)} FOLLOW-UP PLANNER SESSION"),
"",
f"Spec: {highlight(spec_dir.name)}",
muted("Adding follow-up work to completed spec."),
"",
muted("The agent will read your FOLLOWUP_REQUEST.md and add new subtasks."),
]
print()
print(box(content, width=70, style="heavy"))
print()
# Start planning phase in task logger
if task_logger:
task_logger.start_phase(LogPhase.PLANNING, "Starting follow-up planning...")
task_logger.set_session(1)
# Create client with phase-specific model and thinking budget
# Respects task_metadata.json configuration when no CLI override
planning_model = get_phase_model(spec_dir, "planning", model)
planning_thinking_budget = get_phase_thinking_budget(spec_dir, "planning")
client = create_client(
project_dir,
spec_dir,
planning_model,
max_thinking_tokens=planning_thinking_budget,
)
# Generate follow-up planner prompt
prompt = get_followup_planner_prompt(spec_dir)
print_status("Running follow-up planner...", "progress")
print()
try:
# Run single planning session
async with client:
status, response = await run_agent_session(
client, prompt, spec_dir, verbose, phase=LogPhase.PLANNING
)
# End planning phase in task logger
if task_logger:
task_logger.end_phase(
LogPhase.PLANNING,
success=(status != "error"),
message="Follow-up planning session completed",
)
if status == "error":
print()
print_status("Follow-up planning failed", "error")
status_manager.update(state=BuildState.ERROR)
return False
# Verify the plan was updated (should have pending subtasks now)
plan_file = spec_dir / "implementation_plan.json"
if plan_file.exists():
plan = ImplementationPlan.load(plan_file)
# Check if there are any pending subtasks
all_subtasks = [c for p in plan.phases for c in p.subtasks]
pending_subtasks = [c for c in all_subtasks if c.status.value == "pending"]
if pending_subtasks:
# Reset the plan status to in_progress (in case planner didn't)
plan.reset_for_followup()
await plan.async_save(plan_file)
print()
content = [
bold(f"{icon(Icons.SUCCESS)} FOLLOW-UP PLANNING COMPLETE"),
"",
f"New pending subtasks: {highlight(str(len(pending_subtasks)))}",
f"Total subtasks: {len(all_subtasks)}",
"",
muted("Next steps:"),
f" Run: {highlight(f'python auto-claude/run.py --spec {spec_dir.name}')}",
]
print(box(content, width=70, style="heavy"))
print()
status_manager.update(state=BuildState.PAUSED)
return True
else:
print()
print_status(
"Warning: No pending subtasks found after planning", "warning"
)
print(muted("The planner may not have added new subtasks."))
print(muted("Check implementation_plan.json manually."))
status_manager.update(state=BuildState.PAUSED)
return False
else:
print()
print_status(
"Error: implementation_plan.json not found after planning", "error"
)
status_manager.update(state=BuildState.ERROR)
return False
except Exception as e:
print()
print_status(f"Follow-up planning error: {e}", "error")
if task_logger:
task_logger.log_error(f"Follow-up planning error: {e}", LogPhase.PLANNING)
status_manager.update(state=BuildState.ERROR)
return False
+554
View File
@@ -0,0 +1,554 @@
"""
Agent Session Management
========================
Handles running agent sessions and post-session processing including
memory updates, recovery tracking, and Linear integration.
"""
import logging
from pathlib import Path
from claude_agent_sdk import ClaudeSDKClient
from debug import debug, debug_detailed, debug_error, debug_section, debug_success
from insight_extractor import extract_session_insights
from linear_updater import (
linear_subtask_completed,
linear_subtask_failed,
)
from progress import (
count_subtasks_detailed,
is_build_complete,
)
from recovery import RecoveryManager
from security.tool_input_validator import get_safe_tool_input
from task_logger import (
LogEntryType,
LogPhase,
get_task_logger,
)
from ui import (
StatusManager,
muted,
print_key_value,
print_status,
)
from .memory_manager import save_session_memory
from .utils import (
find_subtask_in_plan,
get_commit_count,
get_latest_commit,
load_implementation_plan,
sync_spec_to_source,
)
logger = logging.getLogger(__name__)
async def post_session_processing(
spec_dir: Path,
project_dir: Path,
subtask_id: str,
session_num: int,
commit_before: str | None,
commit_count_before: int,
recovery_manager: RecoveryManager,
linear_enabled: bool = False,
status_manager: StatusManager | None = None,
source_spec_dir: Path | None = None,
) -> bool:
"""
Process session results and update memory automatically.
This runs in Python (100% reliable) instead of relying on agent compliance.
Args:
spec_dir: Spec directory containing memory/
project_dir: Project root for git operations
subtask_id: The subtask that was being worked on
session_num: Current session number
commit_before: Git commit hash before session
commit_count_before: Number of commits before session
recovery_manager: Recovery manager instance
linear_enabled: Whether Linear integration is enabled
status_manager: Optional status manager for ccstatusline
source_spec_dir: Original spec directory (for syncing back from worktree)
Returns:
True if subtask was completed successfully
"""
print()
print(muted("--- Post-Session Processing ---"))
# Sync implementation plan back to source (for worktree mode)
if sync_spec_to_source(spec_dir, source_spec_dir):
print_status("Implementation plan synced to main project", "success")
# Check if implementation plan was updated
plan = load_implementation_plan(spec_dir)
if not plan:
print(" Warning: Could not load implementation plan")
return False
subtask = find_subtask_in_plan(plan, subtask_id)
if not subtask:
print(f" Warning: Subtask {subtask_id} not found in plan")
return False
subtask_status = subtask.get("status", "pending")
# Check for new commits
commit_after = get_latest_commit(project_dir)
commit_count_after = get_commit_count(project_dir)
new_commits = commit_count_after - commit_count_before
print_key_value("Subtask status", subtask_status)
print_key_value("New commits", str(new_commits))
if subtask_status == "completed":
# Success! Record the attempt and good commit
print_status(f"Subtask {subtask_id} completed successfully", "success")
# Update status file
if status_manager:
subtasks = count_subtasks_detailed(spec_dir)
status_manager.update_subtasks(
completed=subtasks["completed"],
total=subtasks["total"],
in_progress=0,
)
# Record successful attempt
recovery_manager.record_attempt(
subtask_id=subtask_id,
session=session_num,
success=True,
approach=f"Implemented: {subtask.get('description', 'subtask')[:100]}",
)
# Record good commit for rollback safety
if commit_after and commit_after != commit_before:
recovery_manager.record_good_commit(commit_after, subtask_id)
print_status(f"Recorded good commit: {commit_after[:8]}", "success")
# Record Linear session result (if enabled)
if linear_enabled:
# Get progress counts for the comment
subtasks_detail = count_subtasks_detailed(spec_dir)
await linear_subtask_completed(
spec_dir=spec_dir,
subtask_id=subtask_id,
completed_count=subtasks_detail["completed"],
total_count=subtasks_detail["total"],
)
print_status("Linear progress recorded", "success")
# Extract rich insights from session (LLM-powered analysis)
try:
extracted_insights = await extract_session_insights(
spec_dir=spec_dir,
project_dir=project_dir,
subtask_id=subtask_id,
session_num=session_num,
commit_before=commit_before,
commit_after=commit_after,
success=True,
recovery_manager=recovery_manager,
)
insight_count = len(extracted_insights.get("file_insights", []))
pattern_count = len(extracted_insights.get("patterns_discovered", []))
if insight_count > 0 or pattern_count > 0:
print_status(
f"Extracted {insight_count} file insights, {pattern_count} patterns",
"success",
)
except Exception as e:
logger.warning(f"Insight extraction failed: {e}")
extracted_insights = None
# Save session memory (Graphiti=primary, file-based=fallback)
try:
save_success, storage_type = await save_session_memory(
spec_dir=spec_dir,
project_dir=project_dir,
subtask_id=subtask_id,
session_num=session_num,
success=True,
subtasks_completed=[subtask_id],
discoveries=extracted_insights,
)
if save_success:
if storage_type == "graphiti":
print_status("Session saved to Graphiti memory", "success")
else:
print_status(
"Session saved to file-based memory (fallback)", "info"
)
else:
print_status("Failed to save session memory", "warning")
except Exception as e:
logger.warning(f"Error saving session memory: {e}")
print_status("Memory save failed", "warning")
return True
elif subtask_status == "in_progress":
# Session ended without completion
print_status(f"Subtask {subtask_id} still in progress", "warning")
recovery_manager.record_attempt(
subtask_id=subtask_id,
session=session_num,
success=False,
approach="Session ended with subtask in_progress",
error="Subtask not marked as completed",
)
# Still record commit if one was made (partial progress)
if commit_after and commit_after != commit_before:
recovery_manager.record_good_commit(commit_after, subtask_id)
print_status(
f"Recorded partial progress commit: {commit_after[:8]}", "info"
)
# Record Linear session result (if enabled)
if linear_enabled:
attempt_count = recovery_manager.get_attempt_count(subtask_id)
await linear_subtask_failed(
spec_dir=spec_dir,
subtask_id=subtask_id,
attempt=attempt_count,
error_summary="Session ended without completion",
)
# Extract insights even from failed sessions (valuable for future attempts)
try:
extracted_insights = await extract_session_insights(
spec_dir=spec_dir,
project_dir=project_dir,
subtask_id=subtask_id,
session_num=session_num,
commit_before=commit_before,
commit_after=commit_after,
success=False,
recovery_manager=recovery_manager,
)
except Exception as e:
logger.debug(f"Insight extraction failed for incomplete session: {e}")
extracted_insights = None
# Save failed session memory (to track what didn't work)
try:
await save_session_memory(
spec_dir=spec_dir,
project_dir=project_dir,
subtask_id=subtask_id,
session_num=session_num,
success=False,
subtasks_completed=[],
discoveries=extracted_insights,
)
except Exception as e:
logger.debug(f"Failed to save incomplete session memory: {e}")
return False
else:
# Subtask still pending or failed
print_status(
f"Subtask {subtask_id} not completed (status: {subtask_status})", "error"
)
recovery_manager.record_attempt(
subtask_id=subtask_id,
session=session_num,
success=False,
approach="Session ended without progress",
error=f"Subtask status is {subtask_status}",
)
# Record Linear session result (if enabled)
if linear_enabled:
attempt_count = recovery_manager.get_attempt_count(subtask_id)
await linear_subtask_failed(
spec_dir=spec_dir,
subtask_id=subtask_id,
attempt=attempt_count,
error_summary=f"Subtask status: {subtask_status}",
)
# Extract insights even from completely failed sessions
try:
extracted_insights = await extract_session_insights(
spec_dir=spec_dir,
project_dir=project_dir,
subtask_id=subtask_id,
session_num=session_num,
commit_before=commit_before,
commit_after=commit_after,
success=False,
recovery_manager=recovery_manager,
)
except Exception as e:
logger.debug(f"Insight extraction failed for failed session: {e}")
extracted_insights = None
# Save failed session memory (to track what didn't work)
try:
await save_session_memory(
spec_dir=spec_dir,
project_dir=project_dir,
subtask_id=subtask_id,
session_num=session_num,
success=False,
subtasks_completed=[],
discoveries=extracted_insights,
)
except Exception as e:
logger.debug(f"Failed to save failed session memory: {e}")
return False
async def run_agent_session(
client: ClaudeSDKClient,
message: str,
spec_dir: Path,
verbose: bool = False,
phase: LogPhase = LogPhase.CODING,
) -> tuple[str, str]:
"""
Run a single agent session using Claude Agent SDK.
Args:
client: Claude SDK client
message: The prompt to send
spec_dir: Spec directory path
verbose: Whether to show detailed output
phase: Current execution phase for logging
Returns:
(status, response_text) where status is:
- "continue" if agent should continue working
- "complete" if all subtasks complete
- "error" if an error occurred
"""
debug_section("session", f"Agent Session - {phase.value}")
debug(
"session",
"Starting agent session",
spec_dir=str(spec_dir),
phase=phase.value,
prompt_length=len(message),
prompt_preview=message[:200] + "..." if len(message) > 200 else message,
)
print("Sending prompt to Claude Agent SDK...\n")
# Get task logger for this spec
task_logger = get_task_logger(spec_dir)
current_tool = None
message_count = 0
tool_count = 0
try:
# Send the query
debug("session", "Sending query to Claude SDK...")
await client.query(message)
debug_success("session", "Query sent successfully")
# Collect response text and show tool use
response_text = ""
debug("session", "Starting to receive response stream...")
async for msg in client.receive_response():
msg_type = type(msg).__name__
message_count += 1
debug_detailed(
"session",
f"Received message #{message_count}",
msg_type=msg_type,
)
# Handle AssistantMessage (text and tool use)
if msg_type == "AssistantMessage" and hasattr(msg, "content"):
for block in msg.content:
block_type = type(block).__name__
if block_type == "TextBlock" and hasattr(block, "text"):
response_text += block.text
print(block.text, end="", flush=True)
# Log text to task logger (persist without double-printing)
if task_logger and block.text.strip():
task_logger.log(
block.text,
LogEntryType.TEXT,
phase,
print_to_console=False,
)
elif block_type == "ToolUseBlock" and hasattr(block, "name"):
tool_name = block.name
tool_input_display = None
tool_count += 1
# Safely extract tool input (handles None, non-dict, etc.)
inp = get_safe_tool_input(block)
# Extract meaningful tool input for display
if inp:
if "pattern" in inp:
tool_input_display = f"pattern: {inp['pattern']}"
elif "file_path" in inp:
fp = inp["file_path"]
if len(fp) > 50:
fp = "..." + fp[-47:]
tool_input_display = fp
elif "command" in inp:
cmd = inp["command"]
if len(cmd) > 50:
cmd = cmd[:47] + "..."
tool_input_display = cmd
elif "path" in inp:
tool_input_display = inp["path"]
debug(
"session",
f"Tool call #{tool_count}: {tool_name}",
tool_input=tool_input_display,
full_input=str(inp)[:500] if inp else None,
)
# Log tool start (handles printing too)
if task_logger:
task_logger.tool_start(
tool_name,
tool_input_display,
phase,
print_to_console=True,
)
else:
print(f"\n[Tool: {tool_name}]", flush=True)
if verbose and hasattr(block, "input"):
input_str = str(block.input)
if len(input_str) > 300:
print(f" Input: {input_str[:300]}...", flush=True)
else:
print(f" Input: {input_str}", flush=True)
current_tool = tool_name
# Handle UserMessage (tool results)
elif msg_type == "UserMessage" and hasattr(msg, "content"):
for block in msg.content:
block_type = type(block).__name__
if block_type == "ToolResultBlock":
result_content = getattr(block, "content", "")
is_error = getattr(block, "is_error", False)
# Check if this is an error (not just content containing "blocked")
if is_error and "blocked" in str(result_content).lower():
# Actual blocked command by security hook
debug_error(
"session",
f"Tool BLOCKED: {current_tool}",
result=str(result_content)[:300],
)
print(f" [BLOCKED] {result_content}", flush=True)
if task_logger and current_tool:
task_logger.tool_end(
current_tool,
success=False,
result="BLOCKED",
detail=str(result_content),
phase=phase,
)
elif is_error:
# Show errors (truncated)
error_str = str(result_content)[:500]
debug_error(
"session",
f"Tool error: {current_tool}",
error=error_str[:200],
)
print(f" [Error] {error_str}", flush=True)
if task_logger and current_tool:
# Store full error in detail for expandable view
task_logger.tool_end(
current_tool,
success=False,
result=error_str[:100],
detail=str(result_content),
phase=phase,
)
else:
# Tool succeeded
debug_detailed(
"session",
f"Tool success: {current_tool}",
result_length=len(str(result_content)),
)
if verbose:
result_str = str(result_content)[:200]
print(f" [Done] {result_str}", flush=True)
else:
print(" [Done]", flush=True)
if task_logger and current_tool:
# Store full result in detail for expandable view (only for certain tools)
# Skip storing for very large outputs like Glob results
detail_content = None
if current_tool in (
"Read",
"Grep",
"Bash",
"Edit",
"Write",
):
result_str = str(result_content)
# Only store if not too large (detail truncation happens in logger)
if (
len(result_str) < 50000
): # 50KB max before truncation
detail_content = result_str
task_logger.tool_end(
current_tool,
success=True,
detail=detail_content,
phase=phase,
)
current_tool = None
print("\n" + "-" * 70 + "\n")
# Check if build is complete
if is_build_complete(spec_dir):
debug_success(
"session",
"Session completed - build is complete",
message_count=message_count,
tool_count=tool_count,
response_length=len(response_text),
)
return "complete", response_text
debug_success(
"session",
"Session completed - continuing",
message_count=message_count,
tool_count=tool_count,
response_length=len(response_text),
)
return "continue", response_text
except Exception as e:
debug_error(
"session",
f"Session error: {e}",
exception_type=type(e).__name__,
message_count=message_count,
tool_count=tool_count,
)
print(f"Error during agent session: {e}")
if task_logger:
task_logger.log_error(f"Session error: {e}", phase)
return "error", str(e)
+159
View File
@@ -0,0 +1,159 @@
#!/usr/bin/env python3
"""
Verification script for agent module refactoring.
This script verifies that:
1. All modules can be imported
2. All public API functions are accessible
3. Backwards compatibility is maintained
"""
import sys
from pathlib import Path
# Add parent directory to path
sys.path.insert(0, str(Path(__file__).parent.parent))
def test_imports():
"""Test that all modules can be imported."""
print("Testing module imports...")
# Test base module
from agents import base
assert hasattr(base, "AUTO_CONTINUE_DELAY_SECONDS")
assert hasattr(base, "HUMAN_INTERVENTION_FILE")
print(" ✓ agents.base")
# Test utils module
from agents import utils
assert hasattr(utils, "get_latest_commit")
assert hasattr(utils, "load_implementation_plan")
print(" ✓ agents.utils")
# Test memory module
from agents import memory
assert hasattr(memory, "save_session_memory")
assert hasattr(memory, "get_graphiti_context")
print(" ✓ agents.memory")
# Test session module
from agents import session
assert hasattr(session, "run_agent_session")
assert hasattr(session, "post_session_processing")
print(" ✓ agents.session")
# Test planner module
from agents import planner
assert hasattr(planner, "run_followup_planner")
print(" ✓ agents.planner")
# Test coder module
from agents import coder
assert hasattr(coder, "run_autonomous_agent")
print(" ✓ agents.coder")
print("\n✓ All module imports successful!\n")
def test_public_api():
"""Test that the public API is accessible."""
print("Testing public API...")
# Test main agent module exports
import agents
required_functions = [
"run_autonomous_agent",
"run_followup_planner",
"save_session_memory",
"get_graphiti_context",
"run_agent_session",
"post_session_processing",
"get_latest_commit",
"load_implementation_plan",
]
for func_name in required_functions:
assert hasattr(agents, func_name), f"Missing function: {func_name}"
print(f" ✓ agents.{func_name}")
print("\n✓ All public API functions accessible!\n")
def test_backwards_compatibility():
"""Test that the old agent.py facade maintains backwards compatibility."""
print("Testing backwards compatibility...")
# Test that agent.py can be imported
import agent
required_functions = [
"run_autonomous_agent",
"run_followup_planner",
"save_session_memory",
"save_session_to_graphiti",
"run_agent_session",
"post_session_processing",
]
for func_name in required_functions:
assert hasattr(agent, func_name), (
f"Missing function in agent module: {func_name}"
)
print(f" ✓ agent.{func_name}")
print("\n✓ Backwards compatibility maintained!\n")
def test_module_structure():
"""Test that the module structure is correct."""
print("Testing module structure...")
from pathlib import Path
agents_dir = Path(__file__).parent
required_files = [
"__init__.py",
"base.py",
"utils.py",
"memory.py",
"session.py",
"planner.py",
"coder.py",
]
for filename in required_files:
filepath = agents_dir / filename
assert filepath.exists(), f"Missing file: {filename}"
print(f" ✓ agents/{filename}")
print("\n✓ Module structure correct!\n")
if __name__ == "__main__":
try:
test_module_structure()
test_imports()
test_public_api()
test_backwards_compatibility()
print("=" * 60)
print("✓ ALL TESTS PASSED - Refactoring verified!")
print("=" * 60)
except AssertionError as e:
print(f"\n✗ TEST FAILED: {e}")
sys.exit(1)
except ImportError as e:
print(f"\n✗ IMPORT ERROR: {e}")
print("Note: Some imports may fail due to missing dependencies.")
print("This is expected in test environments.")
sys.exit(0) # Don't fail on import errors (expected in test env)
+91
View File
@@ -0,0 +1,91 @@
"""
Custom MCP Tools for Auto-Claude Agents
========================================
This module provides custom MCP tools that agents can use for reliable
operations on auto-claude data structures. These tools replace prompt-based
JSON manipulation with guaranteed-correct operations.
Benefits:
- 100% reliable JSON operations (no malformed output)
- Reduced context usage (tool definitions << prompt instructions)
- Type-safe with proper error handling
- Each agent only sees tools relevant to their role via allowed_tools
Usage:
from auto_claude_tools import create_auto_claude_mcp_server, get_allowed_tools
# Create the MCP server
mcp_server = create_auto_claude_mcp_server(spec_dir, project_dir)
# Get allowed tools for a specific agent type
allowed_tools = get_allowed_tools("coder")
# Use in ClaudeAgentOptions
options = ClaudeAgentOptions(
mcp_servers={"auto-claude": mcp_server},
allowed_tools=allowed_tools,
...
)
"""
from .models import (
# Agent configuration registry
AGENT_CONFIGS,
# Base tools
BASE_READ_TOOLS,
BASE_WRITE_TOOLS,
# MCP tool lists
CONTEXT7_TOOLS,
ELECTRON_TOOLS,
GRAPHITI_MCP_TOOLS,
LINEAR_TOOLS,
PUPPETEER_TOOLS,
# Auto-Claude tool names
TOOL_GET_BUILD_PROGRESS,
TOOL_GET_SESSION_CONTEXT,
TOOL_RECORD_DISCOVERY,
TOOL_RECORD_GOTCHA,
TOOL_UPDATE_QA_STATUS,
TOOL_UPDATE_SUBTASK_STATUS,
WEB_TOOLS,
# Config functions
get_agent_config,
get_default_thinking_level,
get_required_mcp_servers,
is_electron_mcp_enabled,
)
from .permissions import get_all_agent_types, get_allowed_tools
from .registry import create_auto_claude_mcp_server, is_tools_available
__all__ = [
# Main API
"create_auto_claude_mcp_server",
"get_allowed_tools",
"is_tools_available",
# Agent configuration registry
"AGENT_CONFIGS",
"get_agent_config",
"get_required_mcp_servers",
"get_default_thinking_level",
"get_all_agent_types",
# Base tool lists
"BASE_READ_TOOLS",
"BASE_WRITE_TOOLS",
"WEB_TOOLS",
# MCP tool lists
"CONTEXT7_TOOLS",
"LINEAR_TOOLS",
"GRAPHITI_MCP_TOOLS",
"ELECTRON_TOOLS",
"PUPPETEER_TOOLS",
# Auto-Claude tool name constants
"TOOL_UPDATE_SUBTASK_STATUS",
"TOOL_GET_BUILD_PROGRESS",
"TOOL_RECORD_DISCOVERY",
"TOOL_RECORD_GOTCHA",
"TOOL_GET_SESSION_CONTEXT",
"TOOL_UPDATE_QA_STATUS",
# Config
"is_electron_mcp_enabled",
]
+512
View File
@@ -0,0 +1,512 @@
"""
Tool Models and Constants
==========================
Defines tool name constants and configuration for auto-claude MCP tools.
This module is the single source of truth for all tool definitions used by
the Claude Agent SDK client. Tool lists are organized by category:
- Base tools: Core file operations (Read, Write, Edit, etc.)
- Web tools: Documentation and research (WebFetch, WebSearch)
- MCP tools: External integrations (Context7, Linear, Graphiti, etc.)
- Auto-Claude tools: Custom build management tools
"""
import os
# =============================================================================
# Base Tools (Built-in Claude Code tools)
# =============================================================================
# Core file operation tools
BASE_READ_TOOLS = ["Read", "Glob", "Grep"]
BASE_WRITE_TOOLS = ["Write", "Edit", "Bash"]
# Web tools for documentation lookup and research
# Always available to all agents for accessing external information
WEB_TOOLS = ["WebFetch", "WebSearch"]
# =============================================================================
# Auto-Claude MCP Tools (Custom build management)
# =============================================================================
# Auto-Claude MCP tool names (prefixed with mcp__auto-claude__)
TOOL_UPDATE_SUBTASK_STATUS = "mcp__auto-claude__update_subtask_status"
TOOL_GET_BUILD_PROGRESS = "mcp__auto-claude__get_build_progress"
TOOL_RECORD_DISCOVERY = "mcp__auto-claude__record_discovery"
TOOL_RECORD_GOTCHA = "mcp__auto-claude__record_gotcha"
TOOL_GET_SESSION_CONTEXT = "mcp__auto-claude__get_session_context"
TOOL_UPDATE_QA_STATUS = "mcp__auto-claude__update_qa_status"
# =============================================================================
# External MCP Tools
# =============================================================================
# Context7 MCP tools for documentation lookup (always enabled)
CONTEXT7_TOOLS = [
"mcp__context7__resolve-library-id",
"mcp__context7__get-library-docs",
]
# Linear MCP tools for project management (when LINEAR_API_KEY is set)
LINEAR_TOOLS = [
"mcp__linear-server__list_teams",
"mcp__linear-server__get_team",
"mcp__linear-server__list_projects",
"mcp__linear-server__get_project",
"mcp__linear-server__create_project",
"mcp__linear-server__update_project",
"mcp__linear-server__list_issues",
"mcp__linear-server__get_issue",
"mcp__linear-server__create_issue",
"mcp__linear-server__update_issue",
"mcp__linear-server__list_comments",
"mcp__linear-server__create_comment",
"mcp__linear-server__list_issue_statuses",
"mcp__linear-server__list_issue_labels",
"mcp__linear-server__list_users",
"mcp__linear-server__get_user",
]
# Graphiti MCP tools for knowledge graph memory (when GRAPHITI_MCP_URL is set)
# See: https://github.com/getzep/graphiti
GRAPHITI_MCP_TOOLS = [
"mcp__graphiti-memory__search_nodes", # Search entity summaries
"mcp__graphiti-memory__search_facts", # Search relationships between entities
"mcp__graphiti-memory__add_episode", # Add data to knowledge graph
"mcp__graphiti-memory__get_episodes", # Retrieve recent episodes
"mcp__graphiti-memory__get_entity_edge", # Get specific entity/relationship
]
# =============================================================================
# Browser Automation MCP Tools (QA agents only)
# =============================================================================
# Puppeteer MCP tools for web browser automation
# Used for web frontend validation (non-Electron web apps)
# NOTE: Screenshots must be compressed (1280x720, quality 60, JPEG) to stay under
# Claude SDK's 1MB JSON message buffer limit. See GitHub issue #74.
PUPPETEER_TOOLS = [
"mcp__puppeteer__puppeteer_connect_active_tab",
"mcp__puppeteer__puppeteer_navigate",
"mcp__puppeteer__puppeteer_screenshot",
"mcp__puppeteer__puppeteer_click",
"mcp__puppeteer__puppeteer_fill",
"mcp__puppeteer__puppeteer_select",
"mcp__puppeteer__puppeteer_hover",
"mcp__puppeteer__puppeteer_evaluate",
]
# Electron MCP tools for desktop app automation (when ELECTRON_MCP_ENABLED is set)
# Uses electron-mcp-server to connect to Electron apps via Chrome DevTools Protocol.
# Electron app must be started with --remote-debugging-port=9222 (or ELECTRON_DEBUG_PORT).
# These tools are only available to QA agents (qa_reviewer, qa_fixer), not Coder/Planner.
# NOTE: Screenshots must be compressed to stay under Claude SDK's 1MB JSON message buffer limit.
ELECTRON_TOOLS = [
"mcp__electron__get_electron_window_info", # Get info about running Electron windows
"mcp__electron__take_screenshot", # Capture screenshot of Electron window
"mcp__electron__send_command_to_electron", # Send commands (click, fill, evaluate JS)
"mcp__electron__read_electron_logs", # Read console logs from Electron app
]
# =============================================================================
# Configuration
# =============================================================================
def is_electron_mcp_enabled() -> bool:
"""
Check if Electron MCP server integration is enabled.
Requires ELECTRON_MCP_ENABLED to be set to 'true'.
When enabled, QA agents can use Electron MCP tools to connect to Electron apps
via Chrome DevTools Protocol on the configured debug port.
"""
return os.environ.get("ELECTRON_MCP_ENABLED", "").lower() == "true"
# =============================================================================
# Agent Configuration Registry
# =============================================================================
# Single source of truth for phase → tools → MCP servers mapping.
# This enables phase-aware tool control and context window optimization.
AGENT_CONFIGS = {
# ═══════════════════════════════════════════════════════════════════════
# SPEC CREATION PHASES (Minimal tools, fast startup)
# ═══════════════════════════════════════════════════════════════════════
"spec_gatherer": {
"tools": BASE_READ_TOOLS + WEB_TOOLS,
"mcp_servers": [], # No MCP needed - just reads project
"auto_claude_tools": [],
"thinking_default": "medium",
},
"spec_researcher": {
"tools": BASE_READ_TOOLS + WEB_TOOLS,
"mcp_servers": ["context7"], # Needs docs lookup
"auto_claude_tools": [],
"thinking_default": "medium",
},
"spec_writer": {
"tools": BASE_READ_TOOLS + BASE_WRITE_TOOLS,
"mcp_servers": [], # Just writes spec.md
"auto_claude_tools": [],
"thinking_default": "high",
},
"spec_critic": {
"tools": BASE_READ_TOOLS,
"mcp_servers": [], # Self-critique, no external tools
"auto_claude_tools": [],
"thinking_default": "ultrathink",
},
"spec_discovery": {
"tools": BASE_READ_TOOLS + WEB_TOOLS,
"mcp_servers": [],
"auto_claude_tools": [],
"thinking_default": "medium",
},
"spec_context": {
"tools": BASE_READ_TOOLS,
"mcp_servers": [],
"auto_claude_tools": [],
"thinking_default": "medium",
},
"spec_validation": {
"tools": BASE_READ_TOOLS,
"mcp_servers": [],
"auto_claude_tools": [],
"thinking_default": "high",
},
"spec_compaction": {
"tools": BASE_READ_TOOLS + BASE_WRITE_TOOLS,
"mcp_servers": [],
"auto_claude_tools": [],
"thinking_default": "medium",
},
# ═══════════════════════════════════════════════════════════════════════
# BUILD PHASES (Full tools + Graphiti memory)
# Note: "linear" is conditional on project setting "update_linear_with_tasks"
# ═══════════════════════════════════════════════════════════════════════
"planner": {
"tools": BASE_READ_TOOLS + BASE_WRITE_TOOLS + WEB_TOOLS,
"mcp_servers": ["context7", "graphiti", "auto-claude"],
"mcp_servers_optional": ["linear"], # Only if project setting enabled
"auto_claude_tools": [
TOOL_GET_BUILD_PROGRESS,
TOOL_GET_SESSION_CONTEXT,
TOOL_RECORD_DISCOVERY,
],
"thinking_default": "high",
},
"coder": {
"tools": BASE_READ_TOOLS + BASE_WRITE_TOOLS + WEB_TOOLS,
"mcp_servers": ["context7", "graphiti", "auto-claude"],
"mcp_servers_optional": ["linear"],
"auto_claude_tools": [
TOOL_UPDATE_SUBTASK_STATUS,
TOOL_GET_BUILD_PROGRESS,
TOOL_RECORD_DISCOVERY,
TOOL_RECORD_GOTCHA,
TOOL_GET_SESSION_CONTEXT,
],
"thinking_default": "none", # Coding doesn't use extended thinking
},
# ═══════════════════════════════════════════════════════════════════════
# QA PHASES (Read + test + browser + Graphiti memory)
# ═══════════════════════════════════════════════════════════════════════
"qa_reviewer": {
# Read + Write/Edit (for QA reports and plan updates) + Bash (for tests)
# Note: Reviewer writes to spec directory only (qa_report.md, implementation_plan.json)
"tools": BASE_READ_TOOLS + BASE_WRITE_TOOLS + WEB_TOOLS,
"mcp_servers": ["context7", "graphiti", "auto-claude", "browser"],
"mcp_servers_optional": ["linear"], # For updating issue status
"auto_claude_tools": [
TOOL_GET_BUILD_PROGRESS,
TOOL_UPDATE_QA_STATUS,
TOOL_GET_SESSION_CONTEXT,
],
"thinking_default": "high",
},
"qa_fixer": {
"tools": BASE_READ_TOOLS + BASE_WRITE_TOOLS + WEB_TOOLS,
"mcp_servers": ["context7", "graphiti", "auto-claude", "browser"],
"mcp_servers_optional": ["linear"],
"auto_claude_tools": [
TOOL_UPDATE_SUBTASK_STATUS,
TOOL_GET_BUILD_PROGRESS,
TOOL_UPDATE_QA_STATUS,
TOOL_RECORD_GOTCHA,
],
"thinking_default": "medium",
},
# ═══════════════════════════════════════════════════════════════════════
# UTILITY PHASES (Minimal, no MCP)
# ═══════════════════════════════════════════════════════════════════════
"insights": {
"tools": BASE_READ_TOOLS + WEB_TOOLS,
"mcp_servers": [],
"auto_claude_tools": [],
# Note: Default to "none" because insight_extractor uses Haiku which doesn't support thinking
# If using Sonnet/Opus models, override max_thinking_tokens in create_simple_client()
"thinking_default": "none",
},
"merge_resolver": {
"tools": [], # Text-only analysis
"mcp_servers": [],
"auto_claude_tools": [],
"thinking_default": "low",
},
"commit_message": {
"tools": [],
"mcp_servers": [],
"auto_claude_tools": [],
"thinking_default": "low",
},
"pr_reviewer": {
"tools": BASE_READ_TOOLS + WEB_TOOLS, # Read-only
"mcp_servers": ["context7"],
"auto_claude_tools": [],
"thinking_default": "high",
},
"pr_orchestrator_parallel": {
"tools": BASE_READ_TOOLS + WEB_TOOLS, # Read-only for parallel PR orchestrator
"mcp_servers": ["context7"],
"auto_claude_tools": [],
"thinking_default": "high",
},
"pr_followup_parallel": {
"tools": BASE_READ_TOOLS
+ WEB_TOOLS, # Read-only for parallel followup reviewer
"mcp_servers": ["context7"],
"auto_claude_tools": [],
"thinking_default": "high",
},
# ═══════════════════════════════════════════════════════════════════════
# ANALYSIS PHASES
# ═══════════════════════════════════════════════════════════════════════
"analysis": {
"tools": BASE_READ_TOOLS + WEB_TOOLS,
"mcp_servers": ["context7"],
"auto_claude_tools": [],
"thinking_default": "medium",
},
"batch_analysis": {
"tools": BASE_READ_TOOLS + WEB_TOOLS,
"mcp_servers": [],
"auto_claude_tools": [],
"thinking_default": "low",
},
"batch_validation": {
"tools": BASE_READ_TOOLS,
"mcp_servers": [],
"auto_claude_tools": [],
"thinking_default": "low",
},
# ═══════════════════════════════════════════════════════════════════════
# ROADMAP & IDEATION
# ═══════════════════════════════════════════════════════════════════════
"roadmap_discovery": {
"tools": BASE_READ_TOOLS + WEB_TOOLS,
"mcp_servers": ["context7"],
"auto_claude_tools": [],
"thinking_default": "high",
},
"competitor_analysis": {
"tools": BASE_READ_TOOLS + WEB_TOOLS,
"mcp_servers": ["context7"], # WebSearch for competitor research
"auto_claude_tools": [],
"thinking_default": "high",
},
"ideation": {
"tools": BASE_READ_TOOLS + WEB_TOOLS,
"mcp_servers": [],
"auto_claude_tools": [],
"thinking_default": "high",
},
}
# =============================================================================
# Agent Config Helper Functions
# =============================================================================
def get_agent_config(agent_type: str) -> dict:
"""
Get full configuration for an agent type.
Args:
agent_type: The agent type identifier (e.g., 'coder', 'planner', 'qa_reviewer')
Returns:
Configuration dict containing tools, mcp_servers, auto_claude_tools, thinking_default
Raises:
ValueError: If agent_type is not found in AGENT_CONFIGS (strict mode)
"""
if agent_type not in AGENT_CONFIGS:
raise ValueError(
f"Unknown agent type: '{agent_type}'. "
f"Valid types: {sorted(AGENT_CONFIGS.keys())}"
)
return AGENT_CONFIGS[agent_type]
def _map_mcp_server_name(
name: str, custom_server_ids: list[str] | None = None
) -> str | None:
"""
Map user-friendly MCP server names to internal identifiers.
Also accepts custom server IDs directly.
Args:
name: User-provided MCP server name
custom_server_ids: List of custom server IDs to accept as-is
Returns:
Internal server identifier or None if not recognized
"""
if not name:
return None
mappings = {
"context7": "context7",
"graphiti-memory": "graphiti",
"graphiti": "graphiti",
"linear": "linear",
"electron": "electron",
"puppeteer": "puppeteer",
"auto-claude": "auto-claude",
}
# Check if it's a known mapping
mapped = mappings.get(name.lower().strip())
if mapped:
return mapped
# Check if it's a custom server ID (accept as-is)
if custom_server_ids and name in custom_server_ids:
return name
return None
def get_required_mcp_servers(
agent_type: str,
project_capabilities: dict | None = None,
linear_enabled: bool = False,
mcp_config: dict | None = None,
) -> list[str]:
"""
Get MCP servers required for this agent type.
Handles dynamic server selection:
- "browser" electron (if is_electron) or puppeteer (if is_web_frontend)
- "linear" only if in mcp_servers_optional AND linear_enabled is True
- "graphiti" only if GRAPHITI_MCP_URL is set
- Respects per-project MCP config overrides from .auto-claude/.env
- Applies per-agent ADD/REMOVE overrides from AGENT_MCP_<agent>_ADD/REMOVE
Args:
agent_type: The agent type identifier
project_capabilities: Dict from detect_project_capabilities() or None
linear_enabled: Whether Linear integration is enabled for this project
mcp_config: Per-project MCP server toggles from .auto-claude/.env
Keys: CONTEXT7_ENABLED, LINEAR_MCP_ENABLED, ELECTRON_MCP_ENABLED,
PUPPETEER_MCP_ENABLED, AGENT_MCP_<agent>_ADD/REMOVE
Returns:
List of MCP server names to start
"""
config = get_agent_config(agent_type)
servers = list(config.get("mcp_servers", []))
# Load per-project config (or use defaults)
if mcp_config is None:
mcp_config = {}
# Filter context7 if explicitly disabled by project config
if "context7" in servers:
context7_enabled = mcp_config.get("CONTEXT7_ENABLED", "true")
if str(context7_enabled).lower() == "false":
servers = [s for s in servers if s != "context7"]
# Handle optional servers (e.g., Linear if project setting enabled)
optional = config.get("mcp_servers_optional", [])
if "linear" in optional and linear_enabled:
# Also check per-project LINEAR_MCP_ENABLED override
linear_mcp_enabled = mcp_config.get("LINEAR_MCP_ENABLED", "true")
if str(linear_mcp_enabled).lower() != "false":
servers.append("linear")
# Handle dynamic "browser" → electron/puppeteer based on project type and config
if "browser" in servers:
servers = [s for s in servers if s != "browser"]
if project_capabilities:
is_electron = project_capabilities.get("is_electron", False)
is_web_frontend = project_capabilities.get("is_web_frontend", False)
# Check per-project overrides (default false for both)
electron_enabled = mcp_config.get("ELECTRON_MCP_ENABLED", "false")
puppeteer_enabled = mcp_config.get("PUPPETEER_MCP_ENABLED", "false")
# Electron: enabled by project config OR global env var
if is_electron and (
str(electron_enabled).lower() == "true" or is_electron_mcp_enabled()
):
servers.append("electron")
# Puppeteer: enabled by project config (no global env var)
elif is_web_frontend and not is_electron:
if str(puppeteer_enabled).lower() == "true":
servers.append("puppeteer")
# Filter graphiti if not enabled
if "graphiti" in servers:
if not os.environ.get("GRAPHITI_MCP_URL"):
servers = [s for s in servers if s != "graphiti"]
# ========== Apply per-agent MCP overrides ==========
# Format: AGENT_MCP_<agent_type>_ADD=server1,server2
# AGENT_MCP_<agent_type>_REMOVE=server1,server2
add_key = f"AGENT_MCP_{agent_type}_ADD"
remove_key = f"AGENT_MCP_{agent_type}_REMOVE"
# Extract custom server IDs for mapping (allows custom servers to be recognized)
custom_servers = mcp_config.get("CUSTOM_MCP_SERVERS", [])
custom_server_ids = [s.get("id") for s in custom_servers if s.get("id")]
# Process additions
if add_key in mcp_config:
additions = [
s.strip() for s in str(mcp_config[add_key]).split(",") if s.strip()
]
for server in additions:
mapped = _map_mcp_server_name(server, custom_server_ids)
if mapped and mapped not in servers:
servers.append(mapped)
# Process removals (but never remove auto-claude)
if remove_key in mcp_config:
removals = [
s.strip() for s in str(mcp_config[remove_key]).split(",") if s.strip()
]
for server in removals:
mapped = _map_mcp_server_name(server, custom_server_ids)
if mapped and mapped != "auto-claude": # auto-claude cannot be removed
servers = [s for s in servers if s != mapped]
return servers
def get_default_thinking_level(agent_type: str) -> str:
"""
Get default thinking level string for agent type.
This returns the thinking level name (e.g., 'medium', 'high'), not the token budget.
To convert to tokens, use phase_config.get_thinking_budget(level).
Args:
agent_type: The agent type identifier
Returns:
Thinking level string (none, low, medium, high, ultrathink)
"""
config = get_agent_config(agent_type)
return config.get("thinking_default", "medium")
@@ -0,0 +1,120 @@
"""
Agent Tool Permissions
======================
Manages which tools are allowed for each agent type to prevent context
pollution and accidental misuse.
Supports dynamic tool filtering based on project capabilities to optimize
context window usage. For example, Electron tools are only included for
Electron projects, not for Next.js or CLI projects.
This module now uses AGENT_CONFIGS from models.py as the single source of truth
for tool permissions. The get_allowed_tools() function remains the primary API
for backwards compatibility.
"""
from .models import (
AGENT_CONFIGS,
CONTEXT7_TOOLS,
ELECTRON_TOOLS,
GRAPHITI_MCP_TOOLS,
LINEAR_TOOLS,
PUPPETEER_TOOLS,
get_agent_config,
get_required_mcp_servers,
)
from .registry import is_tools_available
def get_allowed_tools(
agent_type: str,
project_capabilities: dict | None = None,
linear_enabled: bool = False,
mcp_config: dict | None = None,
) -> list[str]:
"""
Get the list of allowed tools for a specific agent type.
This ensures each agent only sees tools relevant to their role,
preventing context pollution and accidental misuse.
Uses AGENT_CONFIGS as the single source of truth for tool permissions.
Dynamic MCP tools are added based on project capabilities and required servers.
Args:
agent_type: Agent type identifier (e.g., 'coder', 'planner', 'qa_reviewer')
project_capabilities: Optional dict from detect_project_capabilities()
containing flags like is_electron, is_web_frontend, etc.
linear_enabled: Whether Linear integration is enabled for this project
mcp_config: Per-project MCP server toggles from .auto-claude/.env
Returns:
List of allowed tool names
Raises:
ValueError: If agent_type is not found in AGENT_CONFIGS
"""
# Get agent configuration (raises ValueError if unknown type)
config = get_agent_config(agent_type)
# Start with base tools from config
tools = list(config.get("tools", []))
# Get required MCP servers for this agent
required_servers = get_required_mcp_servers(
agent_type,
project_capabilities,
linear_enabled,
mcp_config,
)
# Add auto-claude tools ONLY if the MCP server is available
# This prevents allowing tools that won't work because the server isn't running
if "auto-claude" in required_servers and is_tools_available():
tools.extend(config.get("auto_claude_tools", []))
# Add MCP tool names based on required servers
tools.extend(_get_mcp_tools_for_servers(required_servers))
return tools
def _get_mcp_tools_for_servers(servers: list[str]) -> list[str]:
"""
Get the list of MCP tools for a list of required servers.
Maps server names to their corresponding tool lists.
Args:
servers: List of MCP server names (e.g., ['context7', 'linear', 'electron'])
Returns:
List of MCP tool names for all specified servers
"""
tools = []
for server in servers:
if server == "context7":
tools.extend(CONTEXT7_TOOLS)
elif server == "linear":
tools.extend(LINEAR_TOOLS)
elif server == "graphiti":
tools.extend(GRAPHITI_MCP_TOOLS)
elif server == "electron":
tools.extend(ELECTRON_TOOLS)
elif server == "puppeteer":
tools.extend(PUPPETEER_TOOLS)
# auto-claude tools are already added via config["auto_claude_tools"]
return tools
def get_all_agent_types() -> list[str]:
"""
Get all registered agent types.
Returns:
Sorted list of all agent type identifiers
"""
return sorted(AGENT_CONFIGS.keys())
+72
View File
@@ -0,0 +1,72 @@
"""
Tool Registry
=============
Central registry for creating and managing auto-claude MCP tools.
"""
from pathlib import Path
try:
from claude_agent_sdk import create_sdk_mcp_server
SDK_TOOLS_AVAILABLE = True
except ImportError:
SDK_TOOLS_AVAILABLE = False
create_sdk_mcp_server = None
from .tools import (
create_memory_tools,
create_progress_tools,
create_qa_tools,
create_subtask_tools,
)
def create_all_tools(spec_dir: Path, project_dir: Path) -> list:
"""
Create all custom tools with the given spec and project directories.
Args:
spec_dir: Path to the spec directory
project_dir: Path to the project root
Returns:
List of all tool functions
"""
if not SDK_TOOLS_AVAILABLE:
return []
all_tools = []
# Create tools by category
all_tools.extend(create_subtask_tools(spec_dir, project_dir))
all_tools.extend(create_progress_tools(spec_dir, project_dir))
all_tools.extend(create_memory_tools(spec_dir, project_dir))
all_tools.extend(create_qa_tools(spec_dir, project_dir))
return all_tools
def create_auto_claude_mcp_server(spec_dir: Path, project_dir: Path):
"""
Create an MCP server with auto-claude custom tools.
Args:
spec_dir: Path to the spec directory
project_dir: Path to the project root
Returns:
MCP server instance, or None if SDK tools not available
"""
if not SDK_TOOLS_AVAILABLE:
return None
tools = create_all_tools(spec_dir, project_dir)
return create_sdk_mcp_server(name="auto-claude", version="1.0.0", tools=tools)
def is_tools_available() -> bool:
"""Check if SDK tools functionality is available."""
return SDK_TOOLS_AVAILABLE
@@ -0,0 +1,18 @@
"""
Auto-Claude MCP Tools
=====================
Individual tool implementations organized by functionality.
"""
from .memory import create_memory_tools
from .progress import create_progress_tools
from .qa import create_qa_tools
from .subtask import create_subtask_tools
__all__ = [
"create_subtask_tools",
"create_progress_tools",
"create_memory_tools",
"create_qa_tools",
]
@@ -0,0 +1,356 @@
"""
Session Memory Tools
====================
Tools for recording and retrieving session memory, including discoveries,
gotchas, and patterns.
Dual-storage approach:
- File-based: Always available, works offline, spec-specific
- LadybugDB: When Graphiti is enabled, also saves to graph database for
cross-session retrieval and Memory UI display
"""
import asyncio
import json
import logging
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
try:
from claude_agent_sdk import tool
SDK_TOOLS_AVAILABLE = True
except ImportError:
SDK_TOOLS_AVAILABLE = False
tool = None
logger = logging.getLogger(__name__)
async def _save_to_graphiti_async(
spec_dir: Path,
project_dir: Path,
save_type: str,
data: dict,
) -> bool:
"""
Save data to Graphiti/LadybugDB (async implementation).
Args:
spec_dir: Spec directory for GraphitiMemory initialization
project_dir: Project root directory
save_type: Type of save - 'discovery', 'gotcha', or 'pattern'
data: Data to save
Returns:
True if save succeeded, False otherwise
"""
try:
# Use centralized helper for GraphitiMemory instantiation
# The helper handles enablement checks internally
from memory.graphiti_helpers import get_graphiti_memory
memory = await get_graphiti_memory(spec_dir, project_dir)
if memory is None:
return False
try:
if save_type == "discovery":
# Save as codebase discovery
# Format: {file_path: description}
result = await memory.save_codebase_discoveries(
{data["file_path"]: data["description"]}
)
elif save_type == "gotcha":
# Save as gotcha
gotcha_text = data["gotcha"]
if data.get("context"):
gotcha_text += f" (Context: {data['context']})"
result = await memory.save_gotcha(gotcha_text)
elif save_type == "pattern":
# Save as pattern
result = await memory.save_pattern(data["pattern"])
else:
result = False
return result
finally:
# Always close the memory connection (swallow exceptions to avoid overriding)
try:
await memory.close()
except Exception as e:
logger.debug(
"Failed to close Graphiti memory connection", exc_info=True
)
except Exception as e:
logger.warning(f"Failed to save to Graphiti: {e}")
return False
def _save_to_graphiti_sync(
spec_dir: Path,
project_dir: Path,
save_type: str,
data: dict,
) -> bool:
"""
Save data to Graphiti/LadybugDB (synchronous wrapper for sync contexts only).
NOTE: This should only be called from synchronous code. For async callers,
use _save_to_graphiti_async() directly to ensure proper resource cleanup.
Args:
spec_dir: Spec directory for GraphitiMemory initialization
project_dir: Project root directory
save_type: Type of save - 'discovery', 'gotcha', or 'pattern'
data: Data to save
Returns:
True if save succeeded, False otherwise
"""
try:
# Check if we're already in an async context
try:
asyncio.get_running_loop()
# We're in an async context - caller should use _save_to_graphiti_async
# Log a warning and return False to avoid the resource leak bug
logger.warning(
"_save_to_graphiti_sync called from async context. "
"Use _save_to_graphiti_async instead for proper cleanup."
)
return False
except RuntimeError:
# No running loop - safe to create one
return asyncio.run(
_save_to_graphiti_async(spec_dir, project_dir, save_type, data)
)
except Exception as e:
logger.warning(f"Failed to save to Graphiti: {e}")
return False
def create_memory_tools(spec_dir: Path, project_dir: Path) -> list:
"""
Create session memory tools.
Args:
spec_dir: Path to the spec directory
project_dir: Path to the project root
Returns:
List of memory tool functions
"""
if not SDK_TOOLS_AVAILABLE:
return []
tools = []
# -------------------------------------------------------------------------
# Tool: record_discovery
# -------------------------------------------------------------------------
@tool(
"record_discovery",
"Record a codebase discovery to session memory. Use this when you learn something important about the codebase.",
{"file_path": str, "description": str, "category": str},
)
async def record_discovery(args: dict[str, Any]) -> dict[str, Any]:
"""Record a discovery to the codebase map (file + Graphiti)."""
file_path = args["file_path"]
description = args["description"]
category = args.get("category", "general")
memory_dir = spec_dir / "memory"
memory_dir.mkdir(exist_ok=True)
codebase_map_file = memory_dir / "codebase_map.json"
saved_to_graphiti = False
try:
# PRIMARY: Save to file-based storage (always works)
# Load existing map or create new
if codebase_map_file.exists():
with open(codebase_map_file, encoding="utf-8") as f:
codebase_map = json.load(f)
else:
codebase_map = {
"discovered_files": {},
"last_updated": None,
}
# Add or update the discovery
codebase_map["discovered_files"][file_path] = {
"description": description,
"category": category,
"discovered_at": datetime.now(timezone.utc).isoformat(),
}
codebase_map["last_updated"] = datetime.now(timezone.utc).isoformat()
with open(codebase_map_file, "w", encoding="utf-8") as f:
json.dump(codebase_map, f, indent=2)
# SECONDARY: Also save to Graphiti/LadybugDB (for Memory UI)
saved_to_graphiti = await _save_to_graphiti_async(
spec_dir,
project_dir,
"discovery",
{
"file_path": file_path,
"description": f"[{category}] {description}",
},
)
storage_note = " (also saved to memory graph)" if saved_to_graphiti else ""
return {
"content": [
{
"type": "text",
"text": f"Recorded discovery for '{file_path}': {description}{storage_note}",
}
]
}
except Exception as e:
return {
"content": [{"type": "text", "text": f"Error recording discovery: {e}"}]
}
tools.append(record_discovery)
# -------------------------------------------------------------------------
# Tool: record_gotcha
# -------------------------------------------------------------------------
@tool(
"record_gotcha",
"Record a gotcha or pitfall to avoid. Use this when you encounter something that future sessions should know.",
{"gotcha": str, "context": str},
)
async def record_gotcha(args: dict[str, Any]) -> dict[str, Any]:
"""Record a gotcha to session memory (file + Graphiti)."""
gotcha = args["gotcha"]
context = args.get("context", "")
memory_dir = spec_dir / "memory"
memory_dir.mkdir(exist_ok=True)
gotchas_file = memory_dir / "gotchas.md"
saved_to_graphiti = False
try:
# PRIMARY: Save to file-based storage (always works)
timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M")
entry = f"\n## [{timestamp}]\n{gotcha}"
if context:
entry += f"\n\n_Context: {context}_"
entry += "\n"
with open(gotchas_file, "a", encoding="utf-8") as f:
if not gotchas_file.exists() or gotchas_file.stat().st_size == 0:
f.write(
"# Gotchas & Pitfalls\n\nThings to watch out for in this codebase.\n"
)
f.write(entry)
# SECONDARY: Also save to Graphiti/LadybugDB (for Memory UI)
saved_to_graphiti = await _save_to_graphiti_async(
spec_dir,
project_dir,
"gotcha",
{"gotcha": gotcha, "context": context},
)
storage_note = " (also saved to memory graph)" if saved_to_graphiti else ""
return {
"content": [
{"type": "text", "text": f"Recorded gotcha: {gotcha}{storage_note}"}
]
}
except Exception as e:
return {
"content": [{"type": "text", "text": f"Error recording gotcha: {e}"}]
}
tools.append(record_gotcha)
# -------------------------------------------------------------------------
# Tool: get_session_context
# -------------------------------------------------------------------------
@tool(
"get_session_context",
"Get context from previous sessions including discoveries, gotchas, and patterns.",
{},
)
async def get_session_context(args: dict[str, Any]) -> dict[str, Any]:
"""Get accumulated session context."""
memory_dir = spec_dir / "memory"
if not memory_dir.exists():
return {
"content": [
{
"type": "text",
"text": "No session memory found. This appears to be the first session.",
}
]
}
result_parts = []
# Load codebase map
codebase_map_file = memory_dir / "codebase_map.json"
if codebase_map_file.exists():
try:
with open(codebase_map_file, encoding="utf-8") as f:
codebase_map = json.load(f)
discoveries = codebase_map.get("discovered_files", {})
if discoveries:
result_parts.append("## Codebase Discoveries")
for path, info in list(discoveries.items())[:20]: # Limit to 20
desc = info.get("description", "No description")
result_parts.append(f"- `{path}`: {desc}")
except Exception:
pass
# Load gotchas
gotchas_file = memory_dir / "gotchas.md"
if gotchas_file.exists():
try:
content = gotchas_file.read_text(encoding="utf-8")
if content.strip():
result_parts.append("\n## Gotchas")
# Take last 1000 chars to avoid too much context
result_parts.append(
content[-1000:] if len(content) > 1000 else content
)
except Exception:
pass
# Load patterns
patterns_file = memory_dir / "patterns.md"
if patterns_file.exists():
try:
content = patterns_file.read_text(encoding="utf-8")
if content.strip():
result_parts.append("\n## Patterns")
result_parts.append(
content[-1000:] if len(content) > 1000 else content
)
except Exception:
pass
if not result_parts:
return {
"content": [
{"type": "text", "text": "No session context available yet."}
]
}
return {"content": [{"type": "text", "text": "\n".join(result_parts)}]}
tools.append(get_session_context)
return tools
@@ -0,0 +1,142 @@
"""
Build Progress Tools
====================
Tools for tracking and reporting build progress.
"""
import json
from pathlib import Path
from typing import Any
try:
from claude_agent_sdk import tool
SDK_TOOLS_AVAILABLE = True
except ImportError:
SDK_TOOLS_AVAILABLE = False
tool = None
def create_progress_tools(spec_dir: Path, project_dir: Path) -> list:
"""
Create build progress tracking tools.
Args:
spec_dir: Path to the spec directory
project_dir: Path to the project root
Returns:
List of progress tool functions
"""
if not SDK_TOOLS_AVAILABLE:
return []
tools = []
# -------------------------------------------------------------------------
# Tool: get_build_progress
# -------------------------------------------------------------------------
@tool(
"get_build_progress",
"Get the current build progress including completed subtasks, pending subtasks, and next subtask to work on.",
{},
)
async def get_build_progress(args: dict[str, Any]) -> dict[str, Any]:
"""Get current build progress."""
plan_file = spec_dir / "implementation_plan.json"
if not plan_file.exists():
return {
"content": [
{
"type": "text",
"text": "No implementation plan found. Run the planner first.",
}
]
}
try:
with open(plan_file, encoding="utf-8") as f:
plan = json.load(f)
stats = {
"total": 0,
"completed": 0,
"in_progress": 0,
"pending": 0,
"failed": 0,
}
phases_summary = []
next_subtask = None
for phase in plan.get("phases", []):
phase_id = phase.get("id") or phase.get("phase")
phase_name = phase.get("name", phase_id)
phase_subtasks = phase.get("subtasks", [])
phase_stats = {"completed": 0, "total": len(phase_subtasks)}
for subtask in phase_subtasks:
stats["total"] += 1
status = subtask.get("status", "pending")
if status == "completed":
stats["completed"] += 1
phase_stats["completed"] += 1
elif status == "in_progress":
stats["in_progress"] += 1
elif status == "failed":
stats["failed"] += 1
else:
stats["pending"] += 1
# Track next subtask to work on
if next_subtask is None:
next_subtask = {
"id": subtask.get("id"),
"description": subtask.get("description"),
"phase": phase_name,
}
phases_summary.append(
f" {phase_name}: {phase_stats['completed']}/{phase_stats['total']}"
)
progress_pct = (
(stats["completed"] / stats["total"] * 100) if stats["total"] > 0 else 0
)
result = f"""Build Progress: {stats["completed"]}/{stats["total"]} subtasks ({progress_pct:.0f}%)
Status breakdown:
Completed: {stats["completed"]}
In Progress: {stats["in_progress"]}
Pending: {stats["pending"]}
Failed: {stats["failed"]}
Phases:
{chr(10).join(phases_summary)}"""
if next_subtask:
result += f"""
Next subtask to work on:
ID: {next_subtask["id"]}
Phase: {next_subtask["phase"]}
Description: {next_subtask["description"]}"""
elif stats["completed"] == stats["total"]:
result += "\n\nAll subtasks completed! Build is ready for QA."
return {"content": [{"type": "text", "text": result}]}
except Exception as e:
return {
"content": [
{"type": "text", "text": f"Error reading build progress: {e}"}
]
}
tools.append(get_build_progress)
return tools
+208
View File
@@ -0,0 +1,208 @@
"""
QA Management Tools
===================
Tools for managing QA status and sign-off in implementation_plan.json.
"""
import json
import logging
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from core.file_utils import write_json_atomic
from spec.validate_pkg.auto_fix import auto_fix_plan
try:
from claude_agent_sdk import tool
SDK_TOOLS_AVAILABLE = True
except ImportError:
SDK_TOOLS_AVAILABLE = False
tool = None
def _apply_qa_update(
plan: dict[str, Any],
status: str,
issues: list[Any],
tests_passed: dict[str, Any],
) -> int:
"""
Apply QA update to the plan and return the new QA session number.
Args:
plan: The implementation plan dict
status: QA status (pending, in_review, approved, rejected, fixes_applied)
issues: List of issues found
tests_passed: Dict of test results
Returns:
The new QA session number
"""
# Get current QA session number
current_qa = plan.get("qa_signoff", {})
qa_session = current_qa.get("qa_session", 0)
if status in ["in_review", "rejected"]:
qa_session += 1
plan["qa_signoff"] = {
"status": status,
"qa_session": qa_session,
"issues_found": issues,
"tests_passed": tests_passed,
"timestamp": datetime.now(timezone.utc).isoformat(),
"ready_for_qa_revalidation": status == "fixes_applied",
}
# Update plan status to match QA result
# This ensures the UI shows the correct column after QA
if status == "approved":
plan["status"] = "human_review"
plan["planStatus"] = "review"
elif status == "rejected":
plan["status"] = "human_review"
plan["planStatus"] = "review"
plan["last_updated"] = datetime.now(timezone.utc).isoformat()
return qa_session
def create_qa_tools(spec_dir: Path, project_dir: Path) -> list:
"""
Create QA management tools.
Args:
spec_dir: Path to the spec directory
project_dir: Path to the project root
Returns:
List of QA tool functions
"""
if not SDK_TOOLS_AVAILABLE:
return []
tools = []
# -------------------------------------------------------------------------
# Tool: update_qa_status
# -------------------------------------------------------------------------
@tool(
"update_qa_status",
"Update the QA sign-off status in implementation_plan.json. Use after QA review.",
{"status": str, "issues": str, "tests_passed": str},
)
async def update_qa_status(args: dict[str, Any]) -> dict[str, Any]:
"""Update QA status in the implementation plan."""
status = args["status"]
issues_str = args.get("issues", "[]")
tests_str = args.get("tests_passed", "{}")
valid_statuses = [
"pending",
"in_review",
"approved",
"rejected",
"fixes_applied",
]
if status not in valid_statuses:
return {
"content": [
{
"type": "text",
"text": f"Error: Invalid QA status '{status}'. Must be one of: {valid_statuses}",
}
]
}
plan_file = spec_dir / "implementation_plan.json"
if not plan_file.exists():
return {
"content": [
{
"type": "text",
"text": "Error: implementation_plan.json not found",
}
]
}
try:
# Parse issues and tests
try:
issues = json.loads(issues_str) if issues_str else []
except json.JSONDecodeError:
issues = [{"description": issues_str}] if issues_str else []
try:
tests_passed = json.loads(tests_str) if tests_str else {}
except json.JSONDecodeError:
tests_passed = {}
with open(plan_file, encoding="utf-8") as f:
plan = json.load(f)
qa_session = _apply_qa_update(plan, status, issues, tests_passed)
# Use atomic write to prevent file corruption
write_json_atomic(plan_file, plan, indent=2)
return {
"content": [
{
"type": "text",
"text": f"Updated QA status to '{status}' (session {qa_session})",
}
]
}
except json.JSONDecodeError as e:
# Attempt to auto-fix the plan and retry
if auto_fix_plan(spec_dir):
# Retry after fix
try:
with open(plan_file, encoding="utf-8") as f:
plan = json.load(f)
qa_session = _apply_qa_update(plan, status, issues, tests_passed)
write_json_atomic(plan_file, plan, indent=2)
return {
"content": [
{
"type": "text",
"text": f"Updated QA status to '{status}' (session {qa_session}) (after auto-fix)",
}
]
}
except Exception as retry_err:
logging.warning(
f"QA update retry failed after auto-fix: {retry_err} (original error: {e})"
)
return {
"content": [
{
"type": "text",
"text": f"Error: QA update failed after auto-fix: {retry_err} (original JSON error: {e})",
}
]
}
return {
"content": [
{
"type": "text",
"text": f"Error: Invalid JSON in implementation_plan.json: {e}",
}
]
}
except Exception as e:
return {
"content": [{"type": "text", "text": f"Error updating QA status: {e}"}]
}
tools.append(update_qa_status)
return tools
@@ -0,0 +1,204 @@
"""
Subtask Management Tools
========================
Tools for managing subtask status in implementation_plan.json.
"""
import json
import logging
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from core.file_utils import write_json_atomic
from spec.validate_pkg.auto_fix import auto_fix_plan
try:
from claude_agent_sdk import tool
SDK_TOOLS_AVAILABLE = True
except ImportError:
SDK_TOOLS_AVAILABLE = False
tool = None
def _update_subtask_in_plan(
plan: dict[str, Any],
subtask_id: str,
status: str,
notes: str,
) -> bool:
"""
Update a subtask in the plan.
Args:
plan: The implementation plan dict
subtask_id: ID of the subtask to update
status: New status (pending, in_progress, completed, failed)
notes: Optional notes to add
Returns:
True if subtask was found and updated, False otherwise
"""
subtask_found = False
for phase in plan.get("phases", []):
for subtask in phase.get("subtasks", []):
if subtask.get("id") == subtask_id:
subtask["status"] = status
if notes:
subtask["notes"] = notes
subtask["updated_at"] = datetime.now(timezone.utc).isoformat()
subtask_found = True
break
if subtask_found:
break
if subtask_found:
plan["last_updated"] = datetime.now(timezone.utc).isoformat()
return subtask_found
def create_subtask_tools(spec_dir: Path, project_dir: Path) -> list:
"""
Create subtask management tools.
Args:
spec_dir: Path to the spec directory
project_dir: Path to the project root
Returns:
List of subtask tool functions
"""
if not SDK_TOOLS_AVAILABLE:
return []
tools = []
# -------------------------------------------------------------------------
# Tool: update_subtask_status
# -------------------------------------------------------------------------
@tool(
"update_subtask_status",
"Update the status of a subtask in implementation_plan.json. Use this when completing or starting a subtask.",
{"subtask_id": str, "status": str, "notes": str},
)
async def update_subtask_status(args: dict[str, Any]) -> dict[str, Any]:
"""Update subtask status in the implementation plan."""
subtask_id = args["subtask_id"]
status = args["status"]
notes = args.get("notes", "")
valid_statuses = ["pending", "in_progress", "completed", "failed"]
if status not in valid_statuses:
return {
"content": [
{
"type": "text",
"text": f"Error: Invalid status '{status}'. Must be one of: {valid_statuses}",
}
]
}
plan_file = spec_dir / "implementation_plan.json"
if not plan_file.exists():
return {
"content": [
{
"type": "text",
"text": "Error: implementation_plan.json not found",
}
]
}
try:
with open(plan_file, encoding="utf-8") as f:
plan = json.load(f)
subtask_found = _update_subtask_in_plan(plan, subtask_id, status, notes)
if not subtask_found:
return {
"content": [
{
"type": "text",
"text": f"Error: Subtask '{subtask_id}' not found in implementation plan",
}
]
}
# Use atomic write to prevent file corruption
write_json_atomic(plan_file, plan, indent=2)
return {
"content": [
{
"type": "text",
"text": f"Successfully updated subtask '{subtask_id}' to status '{status}'",
}
]
}
except json.JSONDecodeError as e:
# Attempt to auto-fix the plan and retry
if auto_fix_plan(spec_dir):
# Retry after fix
try:
with open(plan_file, encoding="utf-8") as f:
plan = json.load(f)
subtask_found = _update_subtask_in_plan(
plan, subtask_id, status, notes
)
if subtask_found:
write_json_atomic(plan_file, plan, indent=2)
return {
"content": [
{
"type": "text",
"text": f"Successfully updated subtask '{subtask_id}' to status '{status}' (after auto-fix)",
}
]
}
else:
return {
"content": [
{
"type": "text",
"text": f"Error: Subtask '{subtask_id}' not found in implementation plan (after auto-fix)",
}
]
}
except Exception as retry_err:
logging.warning(
f"Subtask update retry failed after auto-fix: {retry_err}"
)
return {
"content": [
{
"type": "text",
"text": f"Error: Subtask update failed after auto-fix: {retry_err}",
}
]
}
return {
"content": [
{
"type": "text",
"text": f"Error: Invalid JSON in implementation_plan.json: {e}",
}
]
}
except Exception as e:
return {
"content": [
{"type": "text", "text": f"Error updating subtask status: {e}"}
]
}
tools.append(update_subtask_status)
return tools
+181
View File
@@ -0,0 +1,181 @@
"""
Utility Functions for Agent System
===================================
Helper functions for git operations, plan management, and file syncing.
"""
import json
import logging
import shutil
from pathlib import Path
from core.git_executable import run_git
logger = logging.getLogger(__name__)
def get_latest_commit(project_dir: Path) -> str | None:
"""Get the hash of the latest git commit."""
result = run_git(
["rev-parse", "HEAD"],
cwd=project_dir,
timeout=10,
)
if result.returncode == 0:
return result.stdout.strip()
return None
def get_commit_count(project_dir: Path) -> int:
"""Get the total number of commits."""
result = run_git(
["rev-list", "--count", "HEAD"],
cwd=project_dir,
timeout=10,
)
if result.returncode == 0:
try:
return int(result.stdout.strip())
except ValueError:
return 0
return 0
def load_implementation_plan(spec_dir: Path) -> dict | None:
"""Load the implementation plan JSON."""
plan_file = spec_dir / "implementation_plan.json"
if not plan_file.exists():
return None
try:
with open(plan_file, encoding="utf-8") as f:
return json.load(f)
except (OSError, json.JSONDecodeError, UnicodeDecodeError):
return None
def find_subtask_in_plan(plan: dict, subtask_id: str) -> dict | None:
"""Find a subtask by ID in the plan."""
for phase in plan.get("phases", []):
for subtask in phase.get("subtasks", []):
if subtask.get("id") == subtask_id:
return subtask
return None
def find_phase_for_subtask(plan: dict, subtask_id: str) -> dict | None:
"""Find the phase containing a subtask."""
for phase in plan.get("phases", []):
for subtask in phase.get("subtasks", []):
if subtask.get("id") == subtask_id:
return phase
return None
def sync_spec_to_source(spec_dir: Path, source_spec_dir: Path | None) -> bool:
"""
Sync ALL spec files from worktree back to source spec directory.
When running in isolated mode (worktrees), the agent creates and updates
many files inside the worktree's spec directory. This function syncs ALL
of them back to the main project's spec directory.
IMPORTANT: Since .auto-claude/ is gitignored, this sync happens to the
local filesystem regardless of what branch the user is on. The worktree
may be on a different branch (e.g., auto-claude/093-task), but the sync
target is always the main project's .auto-claude/specs/ directory.
Files synced (all files in spec directory):
- implementation_plan.json - Task status and subtask completion
- build-progress.txt - Session-by-session progress notes
- task_logs.json - Execution logs
- review_state.json - QA review state
- critique_report.json - Spec critique findings
- suggested_commit_message.txt - Commit suggestions
- REGRESSION_TEST_REPORT.md - Test regression report
- spec.md, context.json, etc. - Original spec files (for completeness)
- memory/ directory - Codebase map, patterns, gotchas, session insights
Args:
spec_dir: Current spec directory (inside worktree)
source_spec_dir: Original spec directory in main project (outside worktree)
Returns:
True if sync was performed, False if not needed or failed
"""
# Skip if no source specified or same path (not in worktree mode)
if not source_spec_dir:
return False
# Resolve paths and check if they're different
spec_dir_resolved = spec_dir.resolve()
source_spec_dir_resolved = source_spec_dir.resolve()
if spec_dir_resolved == source_spec_dir_resolved:
return False # Same directory, no sync needed
synced_any = False
# Ensure source directory exists
source_spec_dir.mkdir(parents=True, exist_ok=True)
try:
# Sync all files and directories from worktree spec to source spec
for item in spec_dir.iterdir():
# Skip symlinks to prevent path traversal attacks
if item.is_symlink():
logger.warning(f"Skipping symlink during sync: {item.name}")
continue
source_item = source_spec_dir / item.name
if item.is_file():
# Copy file (preserves timestamps)
shutil.copy2(item, source_item)
logger.debug(f"Synced {item.name} to source")
synced_any = True
elif item.is_dir():
# Recursively sync directory
_sync_directory(item, source_item)
synced_any = True
except Exception as e:
logger.warning(f"Failed to sync spec directory to source: {e}")
return synced_any
def _sync_directory(source_dir: Path, target_dir: Path) -> None:
"""
Recursively sync a directory from source to target.
Args:
source_dir: Source directory (in worktree)
target_dir: Target directory (in main project)
"""
# Create target directory if needed
target_dir.mkdir(parents=True, exist_ok=True)
for item in source_dir.iterdir():
# Skip symlinks to prevent path traversal attacks
if item.is_symlink():
logger.warning(
f"Skipping symlink during sync: {source_dir.name}/{item.name}"
)
continue
target_item = target_dir / item.name
if item.is_file():
shutil.copy2(item, target_item)
logger.debug(f"Synced {source_dir.name}/{item.name} to source")
elif item.is_dir():
# Recurse into subdirectories
_sync_directory(item, target_item)
# Keep the old name as an alias for backward compatibility
def sync_plan_to_source(spec_dir: Path, source_spec_dir: Path | None) -> bool:
"""Alias for sync_spec_to_source for backward compatibility."""
return sync_spec_to_source(spec_dir, source_spec_dir)
+41
View File
@@ -0,0 +1,41 @@
"""
Analysis Module
===============
Code analysis and project scanning tools.
"""
# Import from analyzers subpackage (these are the modular analyzers)
from __future__ import annotations
from .analyzers import (
ProjectAnalyzer as ModularProjectAnalyzer,
)
from .analyzers import (
ServiceAnalyzer,
analyze_project,
analyze_service,
)
from .ci_discovery import CIDiscovery
# Import from analysis module root (these are other analysis tools)
from .project_analyzer import ProjectAnalyzer
from .risk_classifier import RiskClassifier
from .security_scanner import SecurityScanner
from .test_discovery import TestDiscovery
# insight_extractor is a module with functions, not a class, so don't import it here
# Import it directly when needed: from analysis import insight_extractor
__all__ = [
"ProjectAnalyzer",
"ModularProjectAnalyzer",
"ServiceAnalyzer",
"analyze_project",
"analyze_service",
"RiskClassifier",
"SecurityScanner",
"CIDiscovery",
"TestDiscovery",
]
+102
View File
@@ -0,0 +1,102 @@
#!/usr/bin/env python3
"""
Codebase Analyzer
=================
Automatically detects project structure, frameworks, and services.
Supports monorepos with multiple services.
Usage:
# Index entire project (creates project_index.json)
python auto-claude/analyzer.py --index
# Analyze specific service
python auto-claude/analyzer.py --service backend
# Output to specific file
python auto-claude/analyzer.py --index --output path/to/output.json
The analyzer will:
1. Detect if this is a monorepo or single project
2. Find all services/packages and analyze each separately
3. Map interdependencies between services
4. Identify infrastructure (Docker, CI/CD)
5. Document conventions (linting, testing)
This module now serves as a facade to the modular analyzer system in the analyzers/ package.
All actual implementation is in focused submodules for better maintainability.
"""
from __future__ import annotations
import json
from pathlib import Path
# Import from the new modular structure
from .analyzers import (
ProjectAnalyzer,
ServiceAnalyzer,
analyze_project,
analyze_service,
)
# Re-export for backward compatibility
__all__ = [
"ServiceAnalyzer",
"ProjectAnalyzer",
"analyze_project",
"analyze_service",
]
def main():
"""CLI entry point."""
import argparse
parser = argparse.ArgumentParser(
description="Analyze project structure, frameworks, and services"
)
parser.add_argument(
"--project-dir",
type=Path,
default=Path.cwd(),
help="Project directory to analyze (default: current directory)",
)
parser.add_argument(
"--index",
action="store_true",
help="Create full project index (default behavior)",
)
parser.add_argument(
"--service",
type=str,
default=None,
help="Analyze a specific service only",
)
parser.add_argument(
"--output",
type=Path,
default=None,
help="Output file for JSON results",
)
parser.add_argument(
"--quiet",
action="store_true",
help="Only output JSON, no status messages",
)
args = parser.parse_args()
# Determine what to analyze
if args.service:
results = analyze_service(args.project_dir, args.service, args.output)
else:
results = analyze_project(args.project_dir, args.output)
# Print results
if not args.quiet or not args.output:
print(json.dumps(results, indent=2))
if __name__ == "__main__":
main()
@@ -0,0 +1,94 @@
"""
Analyzers Package
=================
Modular analyzer system for detecting project structure, frameworks, and services.
Main exports:
- ServiceAnalyzer: Analyzes a single service/package
- ProjectAnalyzer: Analyzes entire projects (single or monorepo)
- analyze_project: Convenience function for project analysis
- analyze_service: Convenience function for service analysis
"""
from __future__ import annotations
from pathlib import Path
from typing import Any
from .project_analyzer_module import ProjectAnalyzer
from .service_analyzer import ServiceAnalyzer
# Re-export main classes
__all__ = [
"ServiceAnalyzer",
"ProjectAnalyzer",
"analyze_project",
"analyze_service",
]
def analyze_project(project_dir: Path, output_file: Path | None = None) -> dict:
"""
Analyze a project and optionally save results.
Args:
project_dir: Path to the project root
output_file: Optional path to save JSON output
Returns:
Project index as a dictionary
"""
import json
analyzer = ProjectAnalyzer(project_dir)
results = analyzer.analyze()
if output_file:
output_file.parent.mkdir(parents=True, exist_ok=True)
with open(output_file, "w", encoding="utf-8") as f:
json.dump(results, f, indent=2)
print(f"Project index saved to: {output_file}")
return results
def analyze_service(
project_dir: Path, service_name: str, output_file: Path | None = None
) -> dict:
"""
Analyze a specific service within a project.
Args:
project_dir: Path to the project root
service_name: Name of the service to analyze
output_file: Optional path to save JSON output
Returns:
Service analysis as a dictionary
"""
import json
# Find the service
service_path = project_dir / service_name
if not service_path.exists():
# Check common locations
for parent in ["packages", "apps", "services"]:
candidate = project_dir / parent / service_name
if candidate.exists():
service_path = candidate
break
if not service_path.exists():
raise ValueError(f"Service '{service_name}' not found in {project_dir}")
analyzer = ServiceAnalyzer(service_path, service_name)
results = analyzer.analyze()
if output_file:
output_file.parent.mkdir(parents=True, exist_ok=True)
with open(output_file, "w", encoding="utf-8") as f:
json.dump(results, f, indent=2)
print(f"Service analysis saved to: {output_file}")
return results
+151
View File
@@ -0,0 +1,151 @@
"""
Base Analyzer Module
====================
Provides common constants, utilities, and base functionality shared across all analyzers.
"""
from __future__ import annotations
import json
from pathlib import Path
# Directories to skip during analysis
SKIP_DIRS = {
"node_modules",
".git",
"__pycache__",
".venv",
"venv",
".env",
"env",
"dist",
"build",
".next",
".nuxt",
"target",
"vendor",
".idea",
".vscode",
".pytest_cache",
".mypy_cache",
"coverage",
".coverage",
"htmlcov",
"eggs",
"*.egg-info",
".turbo",
".cache",
".worktrees", # Skip git worktrees directory
".auto-claude", # Skip auto-claude metadata directory
}
# Common service directory names
SERVICE_INDICATORS = {
"backend",
"frontend",
"api",
"web",
"app",
"server",
"client",
"worker",
"workers",
"services",
"packages",
"apps",
"libs",
"scraper",
"crawler",
"proxy",
"gateway",
"admin",
"dashboard",
"mobile",
"desktop",
"cli",
"sdk",
"core",
"shared",
"common",
}
# Files that indicate a service root
SERVICE_ROOT_FILES = {
"package.json",
"requirements.txt",
"pyproject.toml",
"Cargo.toml",
"go.mod",
"Gemfile",
"composer.json",
"pom.xml",
"build.gradle",
"Makefile",
"Dockerfile",
}
class BaseAnalyzer:
"""Base class with common utilities for all analyzers."""
def __init__(self, path: Path):
self.path = path.resolve()
def _exists(self, path: str) -> bool:
"""Check if a file exists relative to the analyzer's path."""
return (self.path / path).exists()
def _read_file(self, path: str) -> str:
"""Read a file relative to the analyzer's path."""
try:
return (self.path / path).read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
return ""
def _read_json(self, path: str) -> dict | None:
"""Read and parse a JSON file relative to the analyzer's path."""
content = self._read_file(path)
if content:
try:
return json.loads(content)
except json.JSONDecodeError:
return None
return None
def _infer_env_var_type(self, value: str) -> str:
"""Infer the type of an environment variable from its value."""
if not value:
return "string"
# Boolean
if value.lower() in ["true", "false", "1", "0", "yes", "no"]:
return "boolean"
# Number
if value.isdigit():
return "number"
# URL
if value.startswith(
(
"http://",
"https://",
"postgres://",
"postgresql://",
"mysql://",
"mongodb://",
"redis://",
)
):
return "url"
# Email
if "@" in value and "." in value:
return "email"
# Path
if "/" in value or "\\" in value:
return "path"
return "string"
@@ -0,0 +1,26 @@
"""
Context Analyzer Package
=========================
Contains specialized detectors for comprehensive project context analysis.
"""
from __future__ import annotations
from .api_docs_detector import ApiDocsDetector
from .auth_detector import AuthDetector
from .env_detector import EnvironmentDetector
from .jobs_detector import JobsDetector
from .migrations_detector import MigrationsDetector
from .monitoring_detector import MonitoringDetector
from .services_detector import ServicesDetector
__all__ = [
"ApiDocsDetector",
"AuthDetector",
"EnvironmentDetector",
"JobsDetector",
"MigrationsDetector",
"MonitoringDetector",
"ServicesDetector",
]
@@ -0,0 +1,95 @@
"""
API Documentation Detector Module
==================================
Detects API documentation tools and configurations:
- OpenAPI/Swagger (FastAPI auto-generated, swagger-ui-express)
- GraphQL playground
- API documentation endpoints
"""
from __future__ import annotations
from pathlib import Path
from typing import Any
from ..base import BaseAnalyzer
class ApiDocsDetector(BaseAnalyzer):
"""Detects API documentation setup."""
def __init__(self, path: Path, analysis: dict[str, Any]):
super().__init__(path)
self.analysis = analysis
def detect(self) -> None:
"""
Detect API documentation setup.
Detects: OpenAPI/Swagger, GraphQL playground, API docs endpoints.
"""
docs_info = {}
# Detect OpenAPI/Swagger
openapi_info = self._detect_fastapi() or self._detect_swagger_nodejs()
if openapi_info:
docs_info.update(openapi_info)
# Detect GraphQL
graphql_info = self._detect_graphql()
if graphql_info:
docs_info["graphql"] = graphql_info
if docs_info:
self.analysis["api_documentation"] = docs_info
def _detect_fastapi(self) -> dict[str, Any] | None:
"""Detect FastAPI auto-generated OpenAPI docs."""
if self.analysis.get("framework") != "FastAPI":
return None
return {
"type": "openapi",
"auto_generated": True,
"docs_url": "/docs",
"redoc_url": "/redoc",
"openapi_url": "/openapi.json",
}
def _detect_swagger_nodejs(self) -> dict[str, Any] | None:
"""Detect Swagger for Node.js projects."""
if not self._exists("package.json"):
return None
pkg = self._read_json("package.json")
if not pkg:
return None
deps = {**pkg.get("dependencies", {}), **pkg.get("devDependencies", {})}
if "swagger-ui-express" in deps or "swagger-jsdoc" in deps:
return {
"type": "openapi",
"library": "swagger-ui-express",
"docs_url": "/api-docs",
}
return None
def _detect_graphql(self) -> dict[str, str] | None:
"""Detect GraphQL API and playground."""
if not self._exists("package.json"):
return None
pkg = self._read_json("package.json")
if not pkg:
return None
deps = {**pkg.get("dependencies", {}), **pkg.get("devDependencies", {})}
if "graphql" in deps or "apollo-server" in deps or "@apollo/server" in deps:
return {
"playground_url": "/graphql",
"library": "apollo-server" if "apollo-server" in deps else "graphql",
}
return None
@@ -0,0 +1,141 @@
"""
Authentication Patterns Detector Module
========================================
Detects authentication and authorization patterns:
- JWT authentication
- OAuth providers
- Session-based authentication
- API key authentication
- User models
- Auth middleware and decorators
"""
from __future__ import annotations
import re
from pathlib import Path
from typing import Any
from ..base import BaseAnalyzer
class AuthDetector(BaseAnalyzer):
"""Detects authentication and authorization patterns."""
JWT_LIBS = ["python-jose", "pyjwt", "jsonwebtoken", "jose"]
OAUTH_LIBS = ["authlib", "passport", "next-auth", "@auth/core", "oauth2"]
SESSION_LIBS = ["flask-login", "express-session", "django.contrib.auth"]
USER_MODEL_FILES = [
"models/user.py",
"models/User.py",
"app/models/user.py",
"models/user.ts",
"models/User.ts",
"src/models/user.ts",
]
def __init__(self, path: Path, analysis: dict[str, Any]):
super().__init__(path)
self.analysis = analysis
def detect(self) -> None:
"""
Detect authentication and authorization patterns.
Detects: JWT, OAuth, session-based, API keys, user models, protected routes.
"""
auth_info = {
"strategies": [],
"libraries": [],
"user_model": None,
"middleware": [],
}
# Get all dependencies
all_deps = self._get_all_dependencies()
# Detect auth strategies and libraries
self._detect_jwt(all_deps, auth_info)
self._detect_oauth(all_deps, auth_info)
self._detect_session(all_deps, auth_info)
# Find user model
auth_info["user_model"] = self._find_user_model()
# Detect auth middleware/decorators
auth_info["middleware"] = self._find_auth_middleware()
# Remove duplicates from strategies
auth_info["strategies"] = list(set(auth_info["strategies"]))
if auth_info["strategies"] or auth_info["libraries"]:
self.analysis["auth"] = auth_info
def _get_all_dependencies(self) -> set[str]:
"""Extract all dependencies from Python and Node.js projects."""
all_deps = set()
if self._exists("requirements.txt"):
content = self._read_file("requirements.txt")
all_deps.update(re.findall(r"^([a-zA-Z0-9_-]+)", content, re.MULTILINE))
pkg = self._read_json("package.json")
if pkg:
all_deps.update(pkg.get("dependencies", {}).keys())
return all_deps
def _detect_jwt(self, all_deps: set[str], auth_info: dict[str, Any]) -> None:
"""Detect JWT authentication libraries."""
for lib in self.JWT_LIBS:
if lib in all_deps:
auth_info["strategies"].append("jwt")
auth_info["libraries"].append(lib)
break
def _detect_oauth(self, all_deps: set[str], auth_info: dict[str, Any]) -> None:
"""Detect OAuth authentication libraries."""
for lib in self.OAUTH_LIBS:
if lib in all_deps:
auth_info["strategies"].append("oauth")
auth_info["libraries"].append(lib)
break
def _detect_session(self, all_deps: set[str], auth_info: dict[str, Any]) -> None:
"""Detect session-based authentication libraries."""
for lib in self.SESSION_LIBS:
if lib in all_deps:
auth_info["strategies"].append("session")
auth_info["libraries"].append(lib)
break
def _find_user_model(self) -> str | None:
"""Find the user model file."""
for model_file in self.USER_MODEL_FILES:
if self._exists(model_file):
return model_file
return None
def _find_auth_middleware(self) -> list[str]:
"""Detect auth middleware and decorators from Python files."""
# Limit to first 20 files for performance
all_py_files = list(self.path.glob("**/*.py"))[:20]
auth_decorators = set()
for py_file in all_py_files:
try:
content = py_file.read_text(encoding="utf-8")
# Find custom decorators
if (
"@require" in content
or "@login_required" in content
or "@authenticate" in content
):
decorators = re.findall(r"@(\w*(?:require|auth|login)\w*)", content)
auth_decorators.update(decorators)
except (OSError, UnicodeDecodeError):
continue
return list(auth_decorators) if auth_decorators else []
@@ -0,0 +1,223 @@
"""
Environment Variable Detector Module
=====================================
Detects and analyzes environment variables from multiple sources:
- .env files and variants
- .env.example files
- docker-compose.yml
- Source code (os.getenv, process.env)
"""
from __future__ import annotations
import re
from pathlib import Path
from typing import Any
from ..base import BaseAnalyzer
class EnvironmentDetector(BaseAnalyzer):
"""Detects environment variables and their configurations."""
def __init__(self, path: Path, analysis: dict[str, Any]):
super().__init__(path)
self.analysis = analysis
def detect(self) -> None:
"""
Discover all environment variables from multiple sources.
Extracts from: .env files, docker-compose, example files.
Categorizes as required/optional and detects sensitive data.
"""
env_vars = {}
required_vars = set()
optional_vars = set()
# Parse various sources
self._parse_env_files(env_vars)
self._parse_env_example(env_vars, required_vars)
self._parse_docker_compose(env_vars)
self._parse_code_references(env_vars, optional_vars)
# Mark required vs optional
for key in env_vars:
if "required" not in env_vars[key]:
env_vars[key]["required"] = key in required_vars
if env_vars:
self.analysis["environment"] = {
"variables": env_vars,
"required_count": len(required_vars),
"optional_count": len(optional_vars),
"detected_count": len(env_vars),
}
def _parse_env_files(self, env_vars: dict[str, Any]) -> None:
"""Parse .env files and variants."""
env_files = [
".env",
".env.local",
".env.development",
".env.production",
".env.dev",
".env.prod",
".env.test",
".env.staging",
"config/.env",
"../.env",
]
for env_file in env_files:
content = self._read_file(env_file)
if not content:
continue
for line in content.split("\n"):
line = line.strip()
if not line or line.startswith("#"):
continue
# Parse KEY=value or KEY="value" or KEY='value'
match = re.match(r"^([A-Z_][A-Z0-9_]*)\s*=\s*(.*)$", line)
if match:
key = match.group(1)
value = match.group(2).strip().strip('"').strip("'")
# Detect if sensitive
is_sensitive = self._is_sensitive_key(key)
# Detect type
var_type = self._infer_env_var_type(value)
env_vars[key] = {
"value": "<REDACTED>" if is_sensitive else value,
"source": env_file,
"type": var_type,
"sensitive": is_sensitive,
}
def _parse_env_example(
self, env_vars: dict[str, Any], required_vars: set[str]
) -> None:
"""Parse .env.example to find required variables."""
example_content = self._read_file(".env.example") or self._read_file(
".env.sample"
)
if not example_content:
return
for line in example_content.split("\n"):
line = line.strip()
if not line or line.startswith("#"):
continue
match = re.match(r"^([A-Z_][A-Z0-9_]*)\s*=", line)
if match:
key = match.group(1)
required_vars.add(key)
if key not in env_vars:
env_vars[key] = {
"value": None,
"source": ".env.example",
"type": "string",
"sensitive": self._is_sensitive_key(key),
"required": True,
}
def _parse_docker_compose(self, env_vars: dict[str, Any]) -> None:
"""Parse docker-compose.yml environment section."""
for compose_file in ["docker-compose.yml", "../docker-compose.yml"]:
content = self._read_file(compose_file)
if not content:
continue
# Look for environment variables in docker-compose
in_env_section = False
for line in content.split("\n"):
if "environment:" in line:
in_env_section = True
continue
if in_env_section:
# Check if we left the environment section
if line and not line.startswith((" ", "\t", "-")):
in_env_section = False
continue
# Parse - KEY=value or - KEY
match = re.match(r"^\s*-\s*([A-Z_][A-Z0-9_]*)", line)
if match:
key = match.group(1)
if key not in env_vars:
env_vars[key] = {
"value": None,
"source": compose_file,
"type": "string",
"sensitive": False,
}
def _parse_code_references(
self, env_vars: dict[str, Any], optional_vars: set[str]
) -> None:
"""Scan code for os.getenv() / process.env usage to find optional vars."""
entry_files = [
"app.py",
"main.py",
"config.py",
"settings.py",
"src/config.py",
"src/settings.py",
"index.js",
"index.ts",
"config.js",
"config.ts",
]
for entry_file in entry_files:
content = self._read_file(entry_file)
if not content:
continue
# Python: os.getenv("VAR") or os.environ.get("VAR")
python_patterns = [
r'os\.getenv\(["\']([A-Z_][A-Z0-9_]*)["\']',
r'os\.environ\.get\(["\']([A-Z_][A-Z0-9_]*)["\']',
r'os\.environ\[["\']([A-Z_][A-Z0-9_]*)["\']',
]
# JavaScript: process.env.VAR
js_patterns = [
r"process\.env\.([A-Z_][A-Z0-9_]*)",
]
for pattern in python_patterns + js_patterns:
matches = re.findall(pattern, content)
for var_name in matches:
if var_name not in env_vars:
optional_vars.add(var_name)
env_vars[var_name] = {
"value": None,
"source": f"code:{entry_file}",
"type": "string",
"sensitive": self._is_sensitive_key(var_name),
"required": False,
}
@staticmethod
def _is_sensitive_key(key: str) -> bool:
"""Determine if an environment variable key contains sensitive data."""
sensitive_keywords = [
"secret",
"key",
"password",
"token",
"api_key",
"private",
"credential",
"auth",
]
return any(keyword in key.lower() for keyword in sensitive_keywords)
@@ -0,0 +1,118 @@
"""
Background Jobs Detector Module
================================
Detects background job and task queue systems:
- Celery (Python)
- BullMQ/Bull (Node.js)
- Sidekiq (Ruby)
- Scheduled tasks and cron jobs
"""
from __future__ import annotations
import re
from pathlib import Path
from typing import Any
from ..base import BaseAnalyzer
class JobsDetector(BaseAnalyzer):
"""Detects background job and task queue systems."""
def __init__(self, path: Path, analysis: dict[str, Any]):
super().__init__(path)
self.analysis = analysis
def detect(self) -> None:
"""
Detect background job/task queue systems.
Detects: Celery, BullMQ, Sidekiq, cron jobs, scheduled tasks.
"""
jobs_info = None
# Try each job system in order
jobs_info = (
self._detect_celery() or self._detect_bullmq() or self._detect_sidekiq()
)
if jobs_info:
self.analysis["background_jobs"] = jobs_info
def _detect_celery(self) -> dict[str, Any] | None:
"""Detect Celery (Python) task queue."""
celery_files = list(self.path.glob("**/celery.py")) + list(
self.path.glob("**/tasks.py")
)
if not celery_files:
return None
tasks = []
for task_file in celery_files:
try:
content = task_file.read_text(encoding="utf-8")
# Find @celery.task or @shared_task decorators
task_pattern = r"@(?:celery\.task|shared_task|app\.task)\s*(?:\([^)]*\))?\s*def\s+(\w+)"
task_matches = re.findall(task_pattern, content)
for task_name in task_matches:
tasks.append(
{
"name": task_name,
"file": str(task_file.relative_to(self.path)),
}
)
except (OSError, UnicodeDecodeError):
continue
if not tasks:
return None
return {
"system": "celery",
"tasks": tasks,
"total_tasks": len(tasks),
"worker_command": "celery -A app worker",
}
def _detect_bullmq(self) -> dict[str, Any] | None:
"""Detect BullMQ/Bull (Node.js) task queue."""
if not self._exists("package.json"):
return None
pkg = self._read_json("package.json")
if not pkg:
return None
deps = pkg.get("dependencies", {})
if "bullmq" in deps:
return {
"system": "bullmq",
"tasks": [],
"worker_command": "node worker.js",
}
elif "bull" in deps:
return {
"system": "bull",
"tasks": [],
"worker_command": "node worker.js",
}
return None
def _detect_sidekiq(self) -> dict[str, Any] | None:
"""Detect Sidekiq (Ruby) background jobs."""
if not self._exists("Gemfile"):
return None
gemfile = self._read_file("Gemfile")
if "sidekiq" not in gemfile.lower():
return None
return {
"system": "sidekiq",
"worker_command": "bundle exec sidekiq",
}
@@ -0,0 +1,129 @@
"""
Database Migrations Detector Module
====================================
Detects database migration tools and configurations:
- Alembic (Python)
- Django migrations
- Knex (Node.js)
- TypeORM
- Prisma
"""
from __future__ import annotations
from pathlib import Path
from typing import Any
from ..base import BaseAnalyzer
class MigrationsDetector(BaseAnalyzer):
"""Detects database migration setup and tools."""
def __init__(self, path: Path, analysis: dict[str, Any]):
super().__init__(path)
self.analysis = analysis
def detect(self) -> None:
"""
Detect database migration setup.
Detects: Alembic, Django migrations, Knex, TypeORM, Prisma migrations.
"""
migration_info = None
# Try each migration tool in order
migration_info = (
self._detect_alembic()
or self._detect_django()
or self._detect_knex()
or self._detect_typeorm()
or self._detect_prisma()
)
if migration_info:
self.analysis["migrations"] = migration_info
def _detect_alembic(self) -> dict[str, Any] | None:
"""Detect Alembic (Python) migrations."""
if not (self._exists("alembic.ini") or self._exists("alembic")):
return None
return {
"tool": "alembic",
"directory": "alembic/versions"
if self._exists("alembic/versions")
else "alembic",
"config_file": "alembic.ini",
"commands": {
"upgrade": "alembic upgrade head",
"downgrade": "alembic downgrade -1",
"create": "alembic revision --autogenerate -m 'message'",
},
}
def _detect_django(self) -> dict[str, Any] | None:
"""Detect Django migrations."""
if not self._exists("manage.py"):
return None
migration_dirs = list(self.path.glob("**/migrations"))
if not migration_dirs:
return None
return {
"tool": "django",
"directories": [str(d.relative_to(self.path)) for d in migration_dirs],
"commands": {
"migrate": "python manage.py migrate",
"makemigrations": "python manage.py makemigrations",
},
}
def _detect_knex(self) -> dict[str, Any] | None:
"""Detect Knex (Node.js) migrations."""
if not (self._exists("knexfile.js") or self._exists("knexfile.ts")):
return None
return {
"tool": "knex",
"directory": "migrations",
"config_file": "knexfile.js",
"commands": {
"migrate": "knex migrate:latest",
"rollback": "knex migrate:rollback",
"create": "knex migrate:make migration_name",
},
}
def _detect_typeorm(self) -> dict[str, Any] | None:
"""Detect TypeORM migrations."""
if not (self._exists("ormconfig.json") or self._exists("data-source.ts")):
return None
return {
"tool": "typeorm",
"directory": "migrations",
"commands": {
"run": "typeorm migration:run",
"revert": "typeorm migration:revert",
"create": "typeorm migration:create",
},
}
def _detect_prisma(self) -> dict[str, Any] | None:
"""Detect Prisma migrations."""
if not self._exists("prisma/schema.prisma"):
return None
return {
"tool": "prisma",
"directory": "prisma/migrations",
"config_file": "prisma/schema.prisma",
"commands": {
"migrate": "prisma migrate deploy",
"dev": "prisma migrate dev",
"create": "prisma migrate dev --name migration_name",
},
}
@@ -0,0 +1,109 @@
"""
Monitoring Detector Module
===========================
Detects monitoring and observability setup:
- Health check endpoints
- Prometheus metrics endpoints
- APM tools (Sentry, Datadog, New Relic)
- Logging infrastructure
"""
from __future__ import annotations
from pathlib import Path
from typing import Any
from ..base import BaseAnalyzer
class MonitoringDetector(BaseAnalyzer):
"""Detects monitoring and observability setup."""
def __init__(self, path: Path, analysis: dict[str, Any]):
super().__init__(path)
self.analysis = analysis
def detect(self) -> None:
"""
Detect monitoring and observability setup.
Detects: Health checks, metrics endpoints, APM tools, logging.
"""
monitoring_info = {}
# Detect health check endpoints from existing API analysis
health_checks = self._detect_health_checks()
if health_checks:
monitoring_info["health_checks"] = health_checks
# Detect Prometheus metrics
metrics_info = self._detect_prometheus()
if metrics_info:
monitoring_info.update(metrics_info)
# Reference APM tools from services analysis
apm_tools = self._get_apm_tools()
if apm_tools:
monitoring_info["apm_tools"] = apm_tools
if monitoring_info:
self.analysis["monitoring"] = monitoring_info
def _detect_health_checks(self) -> list[str] | None:
"""Detect health check endpoints from API routes."""
if "api" not in self.analysis:
return None
routes = self.analysis["api"].get("routes", [])
health_routes = [
r["path"]
for r in routes
if "health" in r["path"].lower() or "ping" in r["path"].lower()
]
return health_routes if health_routes else None
def _detect_prometheus(self) -> dict[str, str] | None:
"""Detect Prometheus metrics endpoint."""
# Look for actual Prometheus imports/usage, not just keywords
all_files = (
list(self.path.glob("**/*.py"))[:30] + list(self.path.glob("**/*.js"))[:30]
)
for file_path in all_files:
# Skip analyzer files to avoid self-detection
if "analyzers" in str(file_path) or "analyzer.py" in str(file_path):
continue
try:
content = file_path.read_text(encoding="utf-8")
# Look for actual Prometheus imports or usage patterns
prometheus_patterns = [
"from prometheus_client import",
"import prometheus_client",
"prometheus_client.",
"@app.route('/metrics')", # Flask
"app.get('/metrics'", # Express/Fastify
"router.get('/metrics'", # Express Router
]
if any(pattern in content for pattern in prometheus_patterns):
return {
"metrics_endpoint": "/metrics",
"metrics_type": "prometheus",
}
except (OSError, UnicodeDecodeError):
continue
return None
def _get_apm_tools(self) -> list[str] | None:
"""Get APM tools from existing services analysis."""
if (
"services" not in self.analysis
or "monitoring" not in self.analysis["services"]
):
return None
return [s["type"] for s in self.analysis["services"]["monitoring"]]
@@ -0,0 +1,215 @@
"""
External Services Detector Module
==================================
Detects external service integrations based on dependencies:
- Databases (PostgreSQL, MySQL, MongoDB, Redis, SQLite)
- Cache services (Redis, Memcached)
- Message queues (Celery, BullMQ, Kafka, RabbitMQ)
- Email services (SendGrid, Mailgun, Postmark)
- Payment processors (Stripe, PayPal, Square)
- Storage services (AWS S3, Google Cloud Storage, Azure)
- Auth providers (OAuth, JWT)
- Monitoring tools (Sentry, Datadog, New Relic)
"""
from __future__ import annotations
import re
from pathlib import Path
from typing import Any
from ..base import BaseAnalyzer
class ServicesDetector(BaseAnalyzer):
"""Detects external service integrations."""
# Service indicator mappings
DATABASE_INDICATORS = {
"psycopg2": "postgresql",
"psycopg2-binary": "postgresql",
"pg": "postgresql",
"mysql": "mysql",
"mysql2": "mysql",
"pymongo": "mongodb",
"mongodb": "mongodb",
"mongoose": "mongodb",
"redis": "redis",
"redis-py": "redis",
"ioredis": "redis",
"sqlite3": "sqlite",
"better-sqlite3": "sqlite",
}
CACHE_INDICATORS = ["redis", "memcached", "node-cache"]
QUEUE_INDICATORS = {
"celery": "celery",
"bullmq": "bullmq",
"bull": "bull",
"kafka-python": "kafka",
"kafkajs": "kafka",
"amqplib": "rabbitmq",
"amqp": "rabbitmq",
}
EMAIL_INDICATORS = {
"sendgrid": "sendgrid",
"@sendgrid/mail": "sendgrid",
"nodemailer": "smtp",
"mailgun": "mailgun",
"postmark": "postmark",
}
PAYMENT_INDICATORS = {
"stripe": "stripe",
"paypal": "paypal",
"square": "square",
"braintree": "braintree",
}
STORAGE_INDICATORS = {
"boto3": "aws_s3",
"@aws-sdk/client-s3": "aws_s3",
"aws-sdk": "aws_s3",
"@google-cloud/storage": "google_cloud_storage",
"azure-storage-blob": "azure_blob_storage",
}
AUTH_INDICATORS = {
"authlib": "oauth",
"python-jose": "jwt",
"pyjwt": "jwt",
"jsonwebtoken": "jwt",
"passport": "oauth",
"next-auth": "oauth",
"@auth/core": "oauth",
}
MONITORING_INDICATORS = {
"sentry-sdk": "sentry",
"@sentry/node": "sentry",
"datadog": "datadog",
"newrelic": "new_relic",
"loguru": "logging",
"winston": "logging",
"pino": "logging",
}
def __init__(self, path: Path, analysis: dict[str, Any]):
super().__init__(path)
self.analysis = analysis
def detect(self) -> None:
"""
Detect external service integrations.
Detects: databases, cache, email, payments, storage, monitoring, etc.
"""
services = {
"databases": [],
"cache": [],
"message_queues": [],
"email": [],
"payments": [],
"storage": [],
"auth_providers": [],
"monitoring": [],
}
# Get all dependencies
all_deps = self._get_all_dependencies()
# Detect each service category
self._detect_databases(all_deps, services["databases"])
self._detect_cache(all_deps, services["cache"])
self._detect_message_queues(all_deps, services["message_queues"])
self._detect_email(all_deps, services["email"])
self._detect_payments(all_deps, services["payments"])
self._detect_storage(all_deps, services["storage"])
self._detect_auth_providers(all_deps, services["auth_providers"])
self._detect_monitoring(all_deps, services["monitoring"])
# Remove empty categories
services = {k: v for k, v in services.items() if v}
if services:
self.analysis["services"] = services
def _get_all_dependencies(self) -> set[str]:
"""Extract all dependencies from Python and Node.js projects."""
all_deps = set()
# Python dependencies
if self._exists("requirements.txt"):
content = self._read_file("requirements.txt")
all_deps.update(re.findall(r"^([a-zA-Z0-9_-]+)", content, re.MULTILINE))
# Node.js dependencies
pkg = self._read_json("package.json")
if pkg:
all_deps.update(pkg.get("dependencies", {}).keys())
all_deps.update(pkg.get("devDependencies", {}).keys())
return all_deps
def _detect_databases(
self, all_deps: set[str], databases: list[dict[str, str]]
) -> None:
"""Detect database clients."""
for dep, db_type in self.DATABASE_INDICATORS.items():
if dep in all_deps:
databases.append({"type": db_type, "client": dep})
def _detect_cache(self, all_deps: set[str], cache: list[dict[str, str]]) -> None:
"""Detect cache services."""
for indicator in self.CACHE_INDICATORS:
if indicator in all_deps:
cache.append({"type": indicator})
def _detect_message_queues(
self, all_deps: set[str], queues: list[dict[str, str]]
) -> None:
"""Detect message queue systems."""
for dep, queue_type in self.QUEUE_INDICATORS.items():
if dep in all_deps:
queues.append({"type": queue_type, "client": dep})
def _detect_email(self, all_deps: set[str], email: list[dict[str, str]]) -> None:
"""Detect email service providers."""
for dep, email_type in self.EMAIL_INDICATORS.items():
if dep in all_deps:
email.append({"provider": email_type, "client": dep})
def _detect_payments(
self, all_deps: set[str], payments: list[dict[str, str]]
) -> None:
"""Detect payment processors."""
for dep, payment_type in self.PAYMENT_INDICATORS.items():
if dep in all_deps:
payments.append({"provider": payment_type, "client": dep})
def _detect_storage(
self, all_deps: set[str], storage: list[dict[str, str]]
) -> None:
"""Detect storage services."""
for dep, storage_type in self.STORAGE_INDICATORS.items():
if dep in all_deps:
storage.append({"provider": storage_type, "client": dep})
def _detect_auth_providers(
self, all_deps: set[str], auth: list[dict[str, str]]
) -> None:
"""Detect authentication providers."""
for dep, auth_type in self.AUTH_INDICATORS.items():
if dep in all_deps:
auth.append({"type": auth_type, "client": dep})
def _detect_monitoring(
self, all_deps: set[str], monitoring: list[dict[str, str]]
) -> None:
"""Detect monitoring and observability tools."""
for dep, monitoring_type in self.MONITORING_INDICATORS.items():
if dep in all_deps:
monitoring.append({"type": monitoring_type, "client": dep})
@@ -0,0 +1,102 @@
"""
Context Analyzer Module
=======================
Orchestrates comprehensive project context analysis including:
- Environment variables and configuration
- External service integrations
- Authentication patterns
- Database migrations
- Background jobs/task queues
- API documentation
- Monitoring and observability
This module delegates to specialized detectors for clean separation of concerns.
"""
from __future__ import annotations
from pathlib import Path
from typing import Any
from .base import BaseAnalyzer
from .context import (
ApiDocsDetector,
AuthDetector,
EnvironmentDetector,
JobsDetector,
MigrationsDetector,
MonitoringDetector,
ServicesDetector,
)
class ContextAnalyzer(BaseAnalyzer):
"""Orchestrates project context and configuration analysis."""
def __init__(self, path: Path, analysis: dict[str, Any]):
super().__init__(path)
self.analysis = analysis
def detect_environment_variables(self) -> None:
"""
Discover all environment variables from multiple sources.
Delegates to EnvironmentDetector for actual detection logic.
"""
detector = EnvironmentDetector(self.path, self.analysis)
detector.detect()
def detect_external_services(self) -> None:
"""
Detect external service integrations.
Delegates to ServicesDetector for actual detection logic.
"""
detector = ServicesDetector(self.path, self.analysis)
detector.detect()
def detect_auth_patterns(self) -> None:
"""
Detect authentication and authorization patterns.
Delegates to AuthDetector for actual detection logic.
"""
detector = AuthDetector(self.path, self.analysis)
detector.detect()
def detect_migrations(self) -> None:
"""
Detect database migration setup.
Delegates to MigrationsDetector for actual detection logic.
"""
detector = MigrationsDetector(self.path, self.analysis)
detector.detect()
def detect_background_jobs(self) -> None:
"""
Detect background job/task queue systems.
Delegates to JobsDetector for actual detection logic.
"""
detector = JobsDetector(self.path, self.analysis)
detector.detect()
def detect_api_documentation(self) -> None:
"""
Detect API documentation setup.
Delegates to ApiDocsDetector for actual detection logic.
"""
detector = ApiDocsDetector(self.path, self.analysis)
detector.detect()
def detect_monitoring(self) -> None:
"""
Detect monitoring and observability setup.
Delegates to MonitoringDetector for actual detection logic.
"""
detector = MonitoringDetector(self.path, self.analysis)
detector.detect()
@@ -0,0 +1,316 @@
"""
Database Detector Module
========================
Detects database models and schemas across different ORMs:
- Python: SQLAlchemy, Django ORM
- JavaScript/TypeScript: Prisma, TypeORM, Drizzle, Mongoose
"""
from __future__ import annotations
import re
from pathlib import Path
from .base import BaseAnalyzer
class DatabaseDetector(BaseAnalyzer):
"""Detects database models across multiple ORMs."""
def __init__(self, path: Path):
super().__init__(path)
def detect_all_models(self) -> dict:
"""Detect all database models across different ORMs."""
models = {}
# Python SQLAlchemy
models.update(self._detect_sqlalchemy_models())
# Python Django
models.update(self._detect_django_models())
# Prisma schema
models.update(self._detect_prisma_models())
# TypeORM entities
models.update(self._detect_typeorm_models())
# Drizzle schema
models.update(self._detect_drizzle_models())
# Mongoose models
models.update(self._detect_mongoose_models())
return models
def _detect_sqlalchemy_models(self) -> dict:
"""Detect SQLAlchemy models."""
models = {}
py_files = list(self.path.glob("**/*.py"))
for file_path in py_files:
try:
content = file_path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
# Find class definitions that inherit from Base or db.Model
class_pattern = (
r"class\s+(\w+)\([^)]*(?:Base|db\.Model|DeclarativeBase)[^)]*\):"
)
matches = re.finditer(class_pattern, content)
for match in matches:
model_name = match.group(1)
# Extract table name if defined
table_match = re.search(r'__tablename__\s*=\s*["\'](\w+)["\']', content)
table_name = (
table_match.group(1) if table_match else model_name.lower() + "s"
)
# Extract columns
fields = {}
column_pattern = r"(\w+)\s*=\s*Column\((.*?)\)"
column_matches = re.finditer(
column_pattern, content[match.end() : match.end() + 2000]
)
for col_match in column_matches:
field_name = col_match.group(1)
field_def = col_match.group(2)
# Detect field properties
is_primary = "primary_key=True" in field_def
is_unique = "unique=True" in field_def
is_nullable = "nullable=False" not in field_def
# Extract type
type_match = re.search(
r"(Integer|String|Text|Boolean|DateTime|Float|JSON)", field_def
)
field_type = type_match.group(1) if type_match else "Unknown"
fields[field_name] = {
"type": field_type,
"primary_key": is_primary,
"unique": is_unique,
"nullable": is_nullable,
}
if fields: # Only add if we found fields
models[model_name] = {
"table": table_name,
"fields": fields,
"file": str(file_path.relative_to(self.path)),
"orm": "SQLAlchemy",
}
return models
def _detect_django_models(self) -> dict:
"""Detect Django models."""
models = {}
model_files = list(self.path.glob("**/models.py")) + list(
self.path.glob("**/models/*.py")
)
for file_path in model_files:
try:
content = file_path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
# Find class definitions that inherit from models.Model
class_pattern = r"class\s+(\w+)\(models\.Model\):"
matches = re.finditer(class_pattern, content)
for match in matches:
model_name = match.group(1)
table_name = model_name.lower()
# Extract fields
fields = {}
field_pattern = r"(\w+)\s*=\s*models\.(\w+Field)\((.*?)\)"
field_matches = re.finditer(
field_pattern, content[match.end() : match.end() + 2000]
)
for field_match in field_matches:
field_name = field_match.group(1)
field_type = field_match.group(2)
field_args = field_match.group(3)
fields[field_name] = {
"type": field_type,
"unique": "unique=True" in field_args,
"nullable": "null=True" in field_args,
}
if fields:
models[model_name] = {
"table": table_name,
"fields": fields,
"file": str(file_path.relative_to(self.path)),
"orm": "Django",
}
return models
def _detect_prisma_models(self) -> dict:
"""Detect Prisma models from schema.prisma."""
models = {}
schema_file = self.path / "prisma" / "schema.prisma"
if not schema_file.exists():
return models
try:
content = schema_file.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
return models
# Find model definitions
model_pattern = r"model\s+(\w+)\s*\{([^}]+)\}"
matches = re.finditer(model_pattern, content, re.MULTILINE)
for match in matches:
model_name = match.group(1)
model_body = match.group(2)
fields = {}
# Parse fields: id Int @id @default(autoincrement())
field_pattern = r"(\w+)\s+(\w+)([^/\n]*)"
field_matches = re.finditer(field_pattern, model_body)
for field_match in field_matches:
field_name = field_match.group(1)
field_type = field_match.group(2)
field_attrs = field_match.group(3)
fields[field_name] = {
"type": field_type,
"primary_key": "@id" in field_attrs,
"unique": "@unique" in field_attrs,
"nullable": "?" in field_type,
}
if fields:
models[model_name] = {
"table": model_name.lower(),
"fields": fields,
"file": "prisma/schema.prisma",
"orm": "Prisma",
}
return models
def _detect_typeorm_models(self) -> dict:
"""Detect TypeORM entities."""
models = {}
ts_files = list(self.path.glob("**/*.entity.ts")) + list(
self.path.glob("**/entities/*.ts")
)
for file_path in ts_files:
try:
content = file_path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
# Find @Entity() class declarations
entity_pattern = r"@Entity\([^)]*\)\s*(?:export\s+)?class\s+(\w+)"
matches = re.finditer(entity_pattern, content)
for match in matches:
model_name = match.group(1)
# Extract columns
fields = {}
column_pattern = (
r"@(PrimaryGeneratedColumn|Column)\(([^)]*)\)\s+(\w+):\s*(\w+)"
)
column_matches = re.finditer(column_pattern, content)
for col_match in column_matches:
decorator = col_match.group(1)
options = col_match.group(2)
field_name = col_match.group(3)
field_type = col_match.group(4)
fields[field_name] = {
"type": field_type,
"primary_key": decorator == "PrimaryGeneratedColumn",
"unique": "unique: true" in options,
}
if fields:
models[model_name] = {
"table": model_name.lower(),
"fields": fields,
"file": str(file_path.relative_to(self.path)),
"orm": "TypeORM",
}
return models
def _detect_drizzle_models(self) -> dict:
"""Detect Drizzle ORM schemas."""
models = {}
schema_files = list(self.path.glob("**/schema.ts")) + list(
self.path.glob("**/db/schema.ts")
)
for file_path in schema_files:
try:
content = file_path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
# Find table definitions: export const users = pgTable('users', {...})
table_pattern = r'export\s+const\s+(\w+)\s*=\s*(?:pg|mysql|sqlite)Table\(["\'](\w+)["\']'
matches = re.finditer(table_pattern, content)
for match in matches:
const_name = match.group(1)
table_name = match.group(2)
models[const_name] = {
"table": table_name,
"fields": {}, # Would need more parsing for fields
"file": str(file_path.relative_to(self.path)),
"orm": "Drizzle",
}
return models
def _detect_mongoose_models(self) -> dict:
"""Detect Mongoose models."""
models = {}
model_files = list(self.path.glob("**/models/*.js")) + list(
self.path.glob("**/models/*.ts")
)
for file_path in model_files:
try:
content = file_path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
# Find mongoose.model() or new Schema()
model_pattern = r'mongoose\.model\(["\'](\w+)["\']'
matches = re.finditer(model_pattern, content)
for match in matches:
model_name = match.group(1)
models[model_name] = {
"table": model_name.lower(),
"fields": {},
"file": str(file_path.relative_to(self.path)),
"orm": "Mongoose",
}
return models
@@ -0,0 +1,413 @@
"""
Framework Analyzer Module
=========================
Detects programming languages, frameworks, and related technologies across different ecosystems.
Supports Python, Node.js/TypeScript, Go, Rust, and Ruby frameworks.
"""
from __future__ import annotations
from pathlib import Path
from typing import Any
from .base import BaseAnalyzer
class FrameworkAnalyzer(BaseAnalyzer):
"""Analyzes and detects programming languages and frameworks."""
def __init__(self, path: Path, analysis: dict[str, Any]):
super().__init__(path)
self.analysis = analysis
def detect_language_and_framework(self) -> None:
"""Detect primary language and framework."""
# Python detection
if self._exists("requirements.txt"):
self.analysis["language"] = "Python"
self.analysis["package_manager"] = "pip"
deps = self._read_file("requirements.txt")
self._detect_python_framework(deps)
elif self._exists("pyproject.toml"):
self.analysis["language"] = "Python"
content = self._read_file("pyproject.toml")
if "[tool.poetry]" in content:
self.analysis["package_manager"] = "poetry"
elif "[tool.uv]" in content:
self.analysis["package_manager"] = "uv"
else:
self.analysis["package_manager"] = "pip"
self._detect_python_framework(content)
elif self._exists("Pipfile"):
self.analysis["language"] = "Python"
self.analysis["package_manager"] = "pipenv"
content = self._read_file("Pipfile")
self._detect_python_framework(content)
# Node.js/TypeScript detection
elif self._exists("package.json"):
pkg = self._read_json("package.json")
if pkg:
# Check if TypeScript
deps = {**pkg.get("dependencies", {}), **pkg.get("devDependencies", {})}
if "typescript" in deps:
self.analysis["language"] = "TypeScript"
else:
self.analysis["language"] = "JavaScript"
self.analysis["package_manager"] = self._detect_node_package_manager()
self._detect_node_framework(pkg)
# Go detection
elif self._exists("go.mod"):
self.analysis["language"] = "Go"
self.analysis["package_manager"] = "go mod"
content = self._read_file("go.mod")
self._detect_go_framework(content)
# Rust detection
elif self._exists("Cargo.toml"):
self.analysis["language"] = "Rust"
self.analysis["package_manager"] = "cargo"
content = self._read_file("Cargo.toml")
self._detect_rust_framework(content)
# Swift/iOS detection (check BEFORE Ruby - iOS projects often have Gemfile for CocoaPods/Fastlane)
elif self._exists("Package.swift") or any(self.path.glob("*.xcodeproj")):
self.analysis["language"] = "Swift"
if self._exists("Package.swift"):
self.analysis["package_manager"] = "Swift Package Manager"
else:
self.analysis["package_manager"] = "Xcode"
self._detect_swift_framework()
# Ruby detection
elif self._exists("Gemfile"):
self.analysis["language"] = "Ruby"
self.analysis["package_manager"] = "bundler"
content = self._read_file("Gemfile")
self._detect_ruby_framework(content)
def _detect_python_framework(self, content: str) -> None:
"""Detect Python framework."""
from .port_detector import PortDetector
content_lower = content.lower()
# Web frameworks (with conventional defaults)
frameworks = {
"fastapi": {"name": "FastAPI", "type": "backend", "port": 8000},
"flask": {"name": "Flask", "type": "backend", "port": 5000},
"django": {"name": "Django", "type": "backend", "port": 8000},
"starlette": {"name": "Starlette", "type": "backend", "port": 8000},
"litestar": {"name": "Litestar", "type": "backend", "port": 8000},
}
for key, info in frameworks.items():
if key in content_lower:
self.analysis["framework"] = info["name"]
self.analysis["type"] = info["type"]
# Try to detect actual port, fall back to default
port_detector = PortDetector(self.path, self.analysis)
detected_port = port_detector.detect_port_from_sources(info["port"])
self.analysis["default_port"] = detected_port
break
# Task queues
if "celery" in content_lower:
self.analysis["task_queue"] = "Celery"
if not self.analysis.get("type"):
self.analysis["type"] = "worker"
elif "dramatiq" in content_lower:
self.analysis["task_queue"] = "Dramatiq"
elif "huey" in content_lower:
self.analysis["task_queue"] = "Huey"
# ORM
if "sqlalchemy" in content_lower:
self.analysis["orm"] = "SQLAlchemy"
elif "tortoise" in content_lower:
self.analysis["orm"] = "Tortoise ORM"
elif "prisma" in content_lower:
self.analysis["orm"] = "Prisma"
def _detect_node_framework(self, pkg: dict) -> None:
"""Detect Node.js/TypeScript framework."""
from .port_detector import PortDetector
deps = {**pkg.get("dependencies", {}), **pkg.get("devDependencies", {})}
deps_lower = {k.lower(): k for k in deps.keys()}
# Frontend frameworks
frontend_frameworks = {
"next": {"name": "Next.js", "type": "frontend", "port": 3000},
"nuxt": {"name": "Nuxt", "type": "frontend", "port": 3000},
"react": {"name": "React", "type": "frontend", "port": 3000},
"vue": {"name": "Vue", "type": "frontend", "port": 5173},
"svelte": {"name": "Svelte", "type": "frontend", "port": 5173},
"@sveltejs/kit": {"name": "SvelteKit", "type": "frontend", "port": 5173},
"angular": {"name": "Angular", "type": "frontend", "port": 4200},
"@angular/core": {"name": "Angular", "type": "frontend", "port": 4200},
"solid-js": {"name": "SolidJS", "type": "frontend", "port": 3000},
"astro": {"name": "Astro", "type": "frontend", "port": 4321},
}
# Backend frameworks
backend_frameworks = {
"express": {"name": "Express", "type": "backend", "port": 3000},
"fastify": {"name": "Fastify", "type": "backend", "port": 3000},
"koa": {"name": "Koa", "type": "backend", "port": 3000},
"hono": {"name": "Hono", "type": "backend", "port": 3000},
"elysia": {"name": "Elysia", "type": "backend", "port": 3000},
"@nestjs/core": {"name": "NestJS", "type": "backend", "port": 3000},
}
port_detector = PortDetector(self.path, self.analysis)
# Check frontend first (Next.js includes React, etc.)
for key, info in frontend_frameworks.items():
if key in deps_lower:
self.analysis["framework"] = info["name"]
self.analysis["type"] = info["type"]
detected_port = port_detector.detect_port_from_sources(info["port"])
self.analysis["default_port"] = detected_port
break
# If no frontend, check backend
if not self.analysis.get("framework"):
for key, info in backend_frameworks.items():
if key in deps_lower:
self.analysis["framework"] = info["name"]
self.analysis["type"] = info["type"]
detected_port = port_detector.detect_port_from_sources(info["port"])
self.analysis["default_port"] = detected_port
break
# Build tool
if "vite" in deps_lower:
self.analysis["build_tool"] = "Vite"
if not self.analysis.get("default_port"):
detected_port = port_detector.detect_port_from_sources(5173)
self.analysis["default_port"] = detected_port
elif "webpack" in deps_lower:
self.analysis["build_tool"] = "Webpack"
elif "esbuild" in deps_lower:
self.analysis["build_tool"] = "esbuild"
elif "turbopack" in deps_lower:
self.analysis["build_tool"] = "Turbopack"
# Styling
if "tailwindcss" in deps_lower:
self.analysis["styling"] = "Tailwind CSS"
elif "styled-components" in deps_lower:
self.analysis["styling"] = "styled-components"
elif "@emotion/react" in deps_lower:
self.analysis["styling"] = "Emotion"
# State management
if "zustand" in deps_lower:
self.analysis["state_management"] = "Zustand"
elif "@reduxjs/toolkit" in deps_lower or "redux" in deps_lower:
self.analysis["state_management"] = "Redux"
elif "jotai" in deps_lower:
self.analysis["state_management"] = "Jotai"
elif "pinia" in deps_lower:
self.analysis["state_management"] = "Pinia"
# Task queues
if "bullmq" in deps_lower or "bull" in deps_lower:
self.analysis["task_queue"] = "BullMQ"
if not self.analysis.get("type"):
self.analysis["type"] = "worker"
# ORM
if "@prisma/client" in deps_lower or "prisma" in deps_lower:
self.analysis["orm"] = "Prisma"
elif "typeorm" in deps_lower:
self.analysis["orm"] = "TypeORM"
elif "drizzle-orm" in deps_lower:
self.analysis["orm"] = "Drizzle"
elif "mongoose" in deps_lower:
self.analysis["orm"] = "Mongoose"
# Scripts
scripts = pkg.get("scripts", {})
if "dev" in scripts:
self.analysis["dev_command"] = "npm run dev"
elif "start" in scripts:
self.analysis["dev_command"] = "npm run start"
def _detect_go_framework(self, content: str) -> None:
"""Detect Go framework."""
from .port_detector import PortDetector
frameworks = {
"gin-gonic/gin": {"name": "Gin", "port": 8080},
"labstack/echo": {"name": "Echo", "port": 8080},
"gofiber/fiber": {"name": "Fiber", "port": 3000},
"go-chi/chi": {"name": "Chi", "port": 8080},
}
for key, info in frameworks.items():
if key in content:
self.analysis["framework"] = info["name"]
self.analysis["type"] = "backend"
port_detector = PortDetector(self.path, self.analysis)
detected_port = port_detector.detect_port_from_sources(info["port"])
self.analysis["default_port"] = detected_port
break
def _detect_rust_framework(self, content: str) -> None:
"""Detect Rust framework."""
from .port_detector import PortDetector
frameworks = {
"actix-web": {"name": "Actix Web", "port": 8080},
"axum": {"name": "Axum", "port": 3000},
"rocket": {"name": "Rocket", "port": 8000},
}
for key, info in frameworks.items():
if key in content:
self.analysis["framework"] = info["name"]
self.analysis["type"] = "backend"
port_detector = PortDetector(self.path, self.analysis)
detected_port = port_detector.detect_port_from_sources(info["port"])
self.analysis["default_port"] = detected_port
break
def _detect_ruby_framework(self, content: str) -> None:
"""Detect Ruby framework."""
from .port_detector import PortDetector
port_detector = PortDetector(self.path, self.analysis)
if "rails" in content.lower():
self.analysis["framework"] = "Ruby on Rails"
self.analysis["type"] = "backend"
detected_port = port_detector.detect_port_from_sources(3000)
self.analysis["default_port"] = detected_port
elif "sinatra" in content.lower():
self.analysis["framework"] = "Sinatra"
self.analysis["type"] = "backend"
detected_port = port_detector.detect_port_from_sources(4567)
self.analysis["default_port"] = detected_port
if "sidekiq" in content.lower():
self.analysis["task_queue"] = "Sidekiq"
def _detect_swift_framework(self) -> None:
"""Detect Swift/iOS framework and dependencies."""
try:
# Scan Swift files for imports, excluding hidden/vendor dirs
swift_files = []
for swift_file in self.path.rglob("*.swift"):
# Skip hidden directories, node_modules, .worktrees, etc.
if any(
part.startswith(".") or part in ("node_modules", "Pods", "Carthage")
for part in swift_file.parts
):
continue
swift_files.append(swift_file)
if len(swift_files) >= 50: # Limit for performance
break
imports = set()
for swift_file in swift_files:
try:
content = swift_file.read_text(encoding="utf-8", errors="ignore")
for line in content.split("\n"):
line = line.strip()
if line.startswith("import "):
module = line.replace("import ", "").split()[0]
imports.add(module)
except Exception:
continue
# Detect UI framework
if "SwiftUI" in imports:
self.analysis["framework"] = "SwiftUI"
self.analysis["type"] = "mobile"
elif "UIKit" in imports:
self.analysis["framework"] = "UIKit"
self.analysis["type"] = "mobile"
elif "AppKit" in imports:
self.analysis["framework"] = "AppKit"
self.analysis["type"] = "desktop"
# Detect iOS/Apple frameworks
apple_frameworks = []
framework_map = {
"Combine": "Combine",
"CoreData": "CoreData",
"MapKit": "MapKit",
"WidgetKit": "WidgetKit",
"CoreLocation": "CoreLocation",
"StoreKit": "StoreKit",
"CloudKit": "CloudKit",
"ActivityKit": "ActivityKit",
"UserNotifications": "UserNotifications",
}
for key, name in framework_map.items():
if key in imports:
apple_frameworks.append(name)
if apple_frameworks:
self.analysis["apple_frameworks"] = apple_frameworks
# Detect SPM dependencies from Package.swift or xcodeproj
dependencies = self._detect_spm_dependencies()
if dependencies:
self.analysis["spm_dependencies"] = dependencies
except Exception:
# Silently fail if Swift detection has issues
pass
def _detect_spm_dependencies(self) -> list[str]:
"""Detect Swift Package Manager dependencies."""
dependencies = []
# Try Package.swift first
if self._exists("Package.swift"):
content = self._read_file("Package.swift")
# Look for .package(url: "...", patterns
import re
urls = re.findall(r'\.package\s*\([^)]*url:\s*"([^"]+)"', content)
for url in urls:
# Extract package name from URL
name = url.rstrip("/").split("/")[-1].replace(".git", "")
if name:
dependencies.append(name)
# Also check xcodeproj for XCRemoteSwiftPackageReference
for xcodeproj in self.path.glob("*.xcodeproj"):
pbxproj = xcodeproj / "project.pbxproj"
if pbxproj.exists():
try:
content = pbxproj.read_text(encoding="utf-8", errors="ignore")
import re
# Match repositoryURL patterns
urls = re.findall(r'repositoryURL\s*=\s*"([^"]+)"', content)
for url in urls:
name = url.rstrip("/").split("/")[-1].replace(".git", "")
if name and name not in dependencies:
dependencies.append(name)
except Exception:
continue
return dependencies
def _detect_node_package_manager(self) -> str:
"""Detect Node.js package manager."""
if self._exists("pnpm-lock.yaml"):
return "pnpm"
elif self._exists("yarn.lock"):
return "yarn"
elif self._exists("bun.lockb") or self._exists("bun.lock"):
return "bun"
return "npm"
@@ -0,0 +1,337 @@
"""
Port Detector Module
====================
Detects application ports from multiple sources including entry points,
environment files, Docker Compose, configuration files, and scripts.
"""
from __future__ import annotations
import re
from pathlib import Path
from typing import Any
from .base import BaseAnalyzer
class PortDetector(BaseAnalyzer):
"""Detects application ports from various configuration sources."""
def __init__(self, path: Path, analysis: dict[str, Any]):
super().__init__(path)
self.analysis = analysis
def detect_port_from_sources(self, default_port: int) -> int:
"""
Robustly detect the actual port by checking multiple sources.
Checks in order of priority:
1. Entry point files (app.py, main.py, etc.) for uvicorn.run(), app.run(), etc.
2. Environment files (.env, .env.local, .env.development)
3. Docker Compose port mappings
4. Configuration files (config.py, settings.py, etc.)
5. Package.json scripts (for Node.js)
6. Makefile/shell scripts
7. Falls back to default_port if nothing found
Args:
default_port: The framework's conventional default port
Returns:
Detected port or default_port if not found
"""
# 1. Check entry point files for explicit port definitions
port = self._detect_port_in_entry_points()
if port:
return port
# 2. Check environment files
port = self._detect_port_in_env_files()
if port:
return port
# 3. Check Docker Compose
port = self._detect_port_in_docker_compose()
if port:
return port
# 4. Check configuration files
port = self._detect_port_in_config_files()
if port:
return port
# 5. Check package.json scripts (for Node.js)
if self.analysis.get("language") in ["JavaScript", "TypeScript"]:
port = self._detect_port_in_package_scripts()
if port:
return port
# 6. Check Makefile/shell scripts
port = self._detect_port_in_scripts()
if port:
return port
# Fall back to default
return default_port
def _detect_port_in_entry_points(self) -> int | None:
"""Detect port in entry point files."""
entry_files = [
"app.py",
"main.py",
"server.py",
"__main__.py",
"asgi.py",
"wsgi.py",
"src/app.py",
"src/main.py",
"src/server.py",
"index.js",
"index.ts",
"server.js",
"server.ts",
"main.js",
"main.ts",
"src/index.js",
"src/index.ts",
"src/server.js",
"src/server.ts",
"main.go",
"cmd/main.go",
"src/main.rs",
]
# Patterns to search for ports
patterns = [
# Python: uvicorn.run(app, host="0.0.0.0", port=8050)
r"uvicorn\.run\([^)]*port\s*=\s*(\d+)",
# Python: app.run(port=8050, host="0.0.0.0")
r"\.run\([^)]*port\s*=\s*(\d+)",
# Python: port = 8050 or PORT = 8050
r"^\s*[Pp][Oo][Rr][Tt]\s*=\s*(\d+)",
# Python: os.getenv("PORT", 8050) or os.environ.get("PORT", 8050)
r'getenv\(\s*["\']PORT["\']\s*,\s*(\d+)',
r'environ\.get\(\s*["\']PORT["\']\s*,\s*(\d+)',
# JavaScript/TypeScript: app.listen(8050)
r"\.listen\(\s*(\d+)",
# JavaScript/TypeScript: const PORT = 8050 or let port = 8050
r"(?:const|let|var)\s+[Pp][Oo][Rr][Tt]\s*=\s*(\d+)",
# JavaScript/TypeScript: process.env.PORT || 8050
r"process\.env\.PORT\s*\|\|\s*(\d+)",
# JavaScript/TypeScript: Number(process.env.PORT) || 8050
r"Number\(process\.env\.PORT\)\s*\|\|\s*(\d+)",
# Go: :8050 or ":8050"
r':\s*(\d+)(?:["\s]|$)',
# Rust: .bind("127.0.0.1:8050")
r'\.bind\(["\'][\d.]+:(\d+)',
]
for entry_file in entry_files:
content = self._read_file(entry_file)
if not content:
continue
for pattern in patterns:
matches = re.findall(pattern, content, re.MULTILINE)
if matches:
# Return the first valid port found
for match in matches:
try:
port = int(match)
if 1000 <= port <= 65535: # Valid port range
return port
except ValueError:
continue
return None
def _detect_port_in_env_files(self) -> int | None:
"""Detect port in environment files."""
env_files = [
".env",
".env.local",
".env.development",
".env.dev",
"config/.env",
"config/.env.local",
"../.env",
]
patterns = [
r"^\s*PORT\s*=\s*(\d+)",
r"^\s*API_PORT\s*=\s*(\d+)",
r"^\s*SERVER_PORT\s*=\s*(\d+)",
r"^\s*APP_PORT\s*=\s*(\d+)",
]
for env_file in env_files:
content = self._read_file(env_file)
if not content:
continue
for pattern in patterns:
matches = re.findall(pattern, content, re.MULTILINE)
if matches:
try:
port = int(matches[0])
if 1000 <= port <= 65535:
return port
except ValueError:
continue
return None
def _detect_port_in_docker_compose(self) -> int | None:
"""Detect port from docker-compose.yml mappings."""
compose_files = [
"docker-compose.yml",
"docker-compose.yaml",
"../docker-compose.yml",
"../docker-compose.yaml",
]
service_name = self.path.name.lower()
for compose_file in compose_files:
content = self._read_file(compose_file)
if not content:
continue
# Look for port mappings like "8050:8000" or "8050:8050"
# Match the service name if possible
pattern = r'^\s*-\s*["\']?(\d+):\d+["\']?'
in_service = False
in_ports = False
for line in content.split("\n"):
# Check if we're in the right service block
if re.match(rf"^\s*{re.escape(service_name)}\s*:", line):
in_service = True
continue
# Check if we hit another service
if (
in_service
and re.match(r"^\s*\w+\s*:", line)
and "ports:" not in line
):
in_service = False
in_ports = False
continue
# Check if we're in the ports section
if in_service and "ports:" in line:
in_ports = True
continue
# Extract port mapping
if in_ports:
match = re.match(pattern, line)
if match:
try:
port = int(match.group(1))
if 1000 <= port <= 65535:
return port
except ValueError:
continue
return None
def _detect_port_in_config_files(self) -> int | None:
"""Detect port in configuration files."""
config_files = [
"config.py",
"settings.py",
"config/settings.py",
"src/config.py",
"config.json",
"settings.json",
"config/config.json",
"config.toml",
"settings.toml",
]
for config_file in config_files:
content = self._read_file(config_file)
if not content:
continue
# Python config patterns
patterns = [
r"[Pp][Oo][Rr][Tt]\s*=\s*(\d+)",
r'["\']port["\']\s*:\s*(\d+)',
]
for pattern in patterns:
matches = re.findall(pattern, content)
if matches:
try:
port = int(matches[0])
if 1000 <= port <= 65535:
return port
except ValueError:
continue
return None
def _detect_port_in_package_scripts(self) -> int | None:
"""Detect port in package.json scripts."""
pkg = self._read_json("package.json")
if not pkg:
return None
scripts = pkg.get("scripts", {})
# Look for port specifications in scripts
# e.g., "dev": "next dev -p 3001"
# e.g., "start": "node server.js --port 8050"
patterns = [
r"-p\s+(\d+)",
r"--port\s+(\d+)",
r"PORT=(\d+)",
]
for script in scripts.values():
if not isinstance(script, str):
continue
for pattern in patterns:
matches = re.findall(pattern, script)
if matches:
try:
port = int(matches[0])
if 1000 <= port <= 65535:
return port
except ValueError:
continue
return None
def _detect_port_in_scripts(self) -> int | None:
"""Detect port in Makefile or shell scripts."""
script_files = ["Makefile", "start.sh", "run.sh", "dev.sh"]
patterns = [
r"PORT=(\d+)",
r"--port\s+(\d+)",
r"-p\s+(\d+)",
]
for script_file in script_files:
content = self._read_file(script_file)
if not content:
continue
for pattern in patterns:
matches = re.findall(pattern, content)
if matches:
try:
port = int(matches[0])
if 1000 <= port <= 65535:
return port
except ValueError:
continue
return None
@@ -0,0 +1,292 @@
"""
Project Analyzer Module
=======================
Analyzes entire projects, detecting monorepo structures, services, infrastructure, and conventions.
"""
from __future__ import annotations
from pathlib import Path
from typing import Any
from .base import SERVICE_INDICATORS, SERVICE_ROOT_FILES, SKIP_DIRS
from .service_analyzer import ServiceAnalyzer
class ProjectAnalyzer:
"""Analyzes an entire project, detecting monorepo structure and all services."""
def __init__(self, project_dir: Path):
self.project_dir = project_dir.resolve()
self.index = {
"project_root": str(self.project_dir),
"project_type": "single", # or "monorepo"
"services": {},
"infrastructure": {},
"conventions": {},
}
def analyze(self) -> dict[str, Any]:
"""Run full project analysis."""
self._detect_project_type()
self._find_and_analyze_services()
self._analyze_infrastructure()
self._detect_conventions()
self._map_dependencies()
return self.index
def _detect_project_type(self) -> None:
"""Detect if this is a monorepo or single project."""
monorepo_indicators = [
"pnpm-workspace.yaml",
"lerna.json",
"nx.json",
"turbo.json",
"rush.json",
]
for indicator in monorepo_indicators:
if (self.project_dir / indicator).exists():
self.index["project_type"] = "monorepo"
self.index["monorepo_tool"] = indicator.replace(".json", "").replace(
".yaml", ""
)
return
# Check for packages/apps directories
if (self.project_dir / "packages").exists() or (
self.project_dir / "apps"
).exists():
self.index["project_type"] = "monorepo"
return
# Check for multiple service directories
service_dirs_found = 0
for item in self.project_dir.iterdir():
if not item.is_dir():
continue
if item.name in SKIP_DIRS or item.name.startswith("."):
continue
# Check if this directory has service root files
if any((item / f).exists() for f in SERVICE_ROOT_FILES):
service_dirs_found += 1
# If we have 2+ directories with service root files, it's likely a monorepo
if service_dirs_found >= 2:
self.index["project_type"] = "monorepo"
def _find_and_analyze_services(self) -> None:
"""Find all services and analyze each."""
services = {}
if self.index["project_type"] == "monorepo":
# Look for services in common locations
service_locations = [
self.project_dir,
self.project_dir / "packages",
self.project_dir / "apps",
self.project_dir / "services",
]
for location in service_locations:
if not location.exists():
continue
for item in location.iterdir():
if not item.is_dir():
continue
if item.name in SKIP_DIRS:
continue
if item.name.startswith("."):
continue
# Check if this looks like a service
has_root_file = any((item / f).exists() for f in SERVICE_ROOT_FILES)
is_service_name = item.name.lower() in SERVICE_INDICATORS
if has_root_file or (
location == self.project_dir and is_service_name
):
analyzer = ServiceAnalyzer(item, item.name)
service_info = analyzer.analyze()
if service_info.get(
"language"
): # Only include if we detected something
services[item.name] = service_info
else:
# Single project - analyze root
analyzer = ServiceAnalyzer(self.project_dir, "main")
service_info = analyzer.analyze()
if service_info.get("language"):
services["main"] = service_info
self.index["services"] = services
def _analyze_infrastructure(self) -> None:
"""Analyze infrastructure configuration."""
infra = {}
# Docker
if (self.project_dir / "docker-compose.yml").exists():
infra["docker_compose"] = "docker-compose.yml"
compose_content = self._read_file("docker-compose.yml")
infra["docker_services"] = self._parse_compose_services(compose_content)
elif (self.project_dir / "docker-compose.yaml").exists():
infra["docker_compose"] = "docker-compose.yaml"
compose_content = self._read_file("docker-compose.yaml")
infra["docker_services"] = self._parse_compose_services(compose_content)
if (self.project_dir / "Dockerfile").exists():
infra["dockerfile"] = "Dockerfile"
# Docker directory
docker_dir = self.project_dir / "docker"
if docker_dir.exists():
dockerfiles = list(docker_dir.glob("Dockerfile*")) + list(
docker_dir.glob("*.Dockerfile")
)
if dockerfiles:
infra["docker_directory"] = "docker/"
infra["dockerfiles"] = [
str(f.relative_to(self.project_dir)) for f in dockerfiles
]
# CI/CD
if (self.project_dir / ".github" / "workflows").exists():
infra["ci"] = "GitHub Actions"
workflows = list((self.project_dir / ".github" / "workflows").glob("*.yml"))
infra["ci_workflows"] = [f.name for f in workflows]
elif (self.project_dir / ".gitlab-ci.yml").exists():
infra["ci"] = "GitLab CI"
elif (self.project_dir / ".circleci").exists():
infra["ci"] = "CircleCI"
# Deployment
deployment_files = {
"vercel.json": "Vercel",
"netlify.toml": "Netlify",
"fly.toml": "Fly.io",
"render.yaml": "Render",
"railway.json": "Railway",
"Procfile": "Heroku",
"app.yaml": "Google App Engine",
"serverless.yml": "Serverless Framework",
}
for file, platform in deployment_files.items():
if (self.project_dir / file).exists():
infra["deployment"] = platform
break
self.index["infrastructure"] = infra
def _parse_compose_services(self, content: str) -> list[str]:
"""Extract service names from docker-compose content."""
services = []
in_services = False
for line in content.split("\n"):
if line.strip() == "services:":
in_services = True
continue
if in_services:
# Service names are at 2-space indent
if (
line.startswith(" ")
and not line.startswith(" ")
and line.strip().endswith(":")
):
service_name = line.strip().rstrip(":")
services.append(service_name)
elif line and not line.startswith(" "):
break # End of services section
return services
def _detect_conventions(self) -> None:
"""Detect project-wide conventions."""
conventions = {}
# Python linting
if (self.project_dir / "ruff.toml").exists() or self._has_in_pyproject("ruff"):
conventions["python_linting"] = "Ruff"
elif (self.project_dir / ".flake8").exists():
conventions["python_linting"] = "Flake8"
elif (self.project_dir / "pylintrc").exists():
conventions["python_linting"] = "Pylint"
# Python formatting
if (self.project_dir / "pyproject.toml").exists():
content = self._read_file("pyproject.toml")
if "[tool.black]" in content:
conventions["python_formatting"] = "Black"
# JavaScript/TypeScript linting
eslint_files = [
".eslintrc",
".eslintrc.js",
".eslintrc.json",
".eslintrc.yml",
"eslint.config.js",
]
if any((self.project_dir / f).exists() for f in eslint_files):
conventions["js_linting"] = "ESLint"
# Prettier
prettier_files = [
".prettierrc",
".prettierrc.js",
".prettierrc.json",
"prettier.config.js",
]
if any((self.project_dir / f).exists() for f in prettier_files):
conventions["formatting"] = "Prettier"
# TypeScript
if (self.project_dir / "tsconfig.json").exists():
conventions["typescript"] = True
# Git hooks
if (self.project_dir / ".husky").exists():
conventions["git_hooks"] = "Husky"
elif (self.project_dir / ".pre-commit-config.yaml").exists():
conventions["git_hooks"] = "pre-commit"
self.index["conventions"] = conventions
def _map_dependencies(self) -> None:
"""Map dependencies between services."""
services = self.index.get("services", {})
for service_name, service_info in services.items():
consumes = []
# Check for API client patterns
if service_info.get("type") == "frontend":
# Frontend typically consumes backend
for other_name, other_info in services.items():
if other_info.get("type") == "backend":
consumes.append(f"{other_name}.api")
# Check for shared libraries
if service_info.get("dependencies"):
deps = service_info["dependencies"]
for other_name in services.keys():
if other_name in deps or f"@{other_name}" in str(deps):
consumes.append(other_name)
if consumes:
service_info["consumes"] = consumes
def _has_in_pyproject(self, tool: str) -> bool:
"""Check if a tool is configured in pyproject.toml."""
if (self.project_dir / "pyproject.toml").exists():
content = self._read_file("pyproject.toml")
return f"[tool.{tool}]" in content
return False
def _read_file(self, path: str) -> str:
try:
return (self.project_dir / path).read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
return ""
@@ -0,0 +1,418 @@
"""
Route Detector Module
=====================
Detects API routes and endpoints across different frameworks:
- Python: FastAPI, Flask, Django
- Node.js: Express, Next.js
- Go: Gin, Echo, Chi, Fiber
- Rust: Axum, Actix
"""
from __future__ import annotations
import re
from pathlib import Path
from .base import BaseAnalyzer
class RouteDetector(BaseAnalyzer):
"""Detects API routes across multiple web frameworks."""
# Directories to exclude from route detection
EXCLUDED_DIRS = {"node_modules", ".venv", "venv", "__pycache__", ".git"}
def __init__(self, path: Path):
super().__init__(path)
def _should_include_file(self, file_path: Path) -> bool:
"""Check if file should be included (not in excluded directories)."""
return not any(part in self.EXCLUDED_DIRS for part in file_path.parts)
def detect_all_routes(self) -> list[dict]:
"""Detect all API routes across different frameworks."""
routes = []
# Python FastAPI
routes.extend(self._detect_fastapi_routes())
# Python Flask
routes.extend(self._detect_flask_routes())
# Python Django
routes.extend(self._detect_django_routes())
# Node.js Express/Fastify/Koa
routes.extend(self._detect_express_routes())
# Next.js (file-based routing)
routes.extend(self._detect_nextjs_routes())
# Go Gin/Echo/Chi
routes.extend(self._detect_go_routes())
# Rust Axum/Actix
routes.extend(self._detect_rust_routes())
return routes
def _detect_fastapi_routes(self) -> list[dict]:
"""Detect FastAPI routes."""
routes = []
files_to_check = [
f for f in self.path.glob("**/*.py") if self._should_include_file(f)
]
for file_path in files_to_check:
try:
content = file_path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
# Pattern: @app.get("/path") or @router.post("/path", dependencies=[...])
patterns = [
(
r'@(?:app|router)\.(get|post|put|delete|patch)\(["\']([^"\']+)["\']',
"decorator",
),
(
r'@(?:app|router)\.api_route\(["\']([^"\']+)["\'][^)]*methods\s*=\s*\[([^\]]+)\]',
"api_route",
),
]
for pattern, pattern_type in patterns:
matches = re.finditer(pattern, content, re.MULTILINE)
for match in matches:
if pattern_type == "decorator":
method = match.group(1).upper()
path = match.group(2)
methods = [method]
else:
path = match.group(1)
methods_str = match.group(2)
methods = [
m.strip().strip('"').strip("'").upper()
for m in methods_str.split(",")
]
# Check if route requires auth (has Depends in the decorator)
line_start = content.rfind("\n", 0, match.start()) + 1
line_end = content.find("\n", match.end())
route_definition = content[
line_start : line_end if line_end != -1 else len(content)
]
requires_auth = (
"Depends" in route_definition
or "require" in route_definition.lower()
)
routes.append(
{
"path": path,
"methods": methods,
"file": str(file_path.relative_to(self.path)),
"framework": "FastAPI",
"requires_auth": requires_auth,
}
)
return routes
def _detect_flask_routes(self) -> list[dict]:
"""Detect Flask routes."""
routes = []
files_to_check = [
f for f in self.path.glob("**/*.py") if self._should_include_file(f)
]
for file_path in files_to_check:
try:
content = file_path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
# Pattern: @app.route("/path", methods=["GET", "POST"])
pattern = r'@(?:app|bp|blueprint)\.route\(["\']([^"\']+)["\'](?:[^)]*methods\s*=\s*\[([^\]]+)\])?'
matches = re.finditer(pattern, content, re.MULTILINE)
for match in matches:
path = match.group(1)
methods_str = match.group(2)
if methods_str:
methods = [
m.strip().strip('"').strip("'").upper()
for m in methods_str.split(",")
]
else:
methods = ["GET"] # Flask default
# Check for @login_required decorator
decorator_start = content.rfind("@", 0, match.start())
decorator_section = content[decorator_start : match.end()]
requires_auth = (
"login_required" in decorator_section
or "require" in decorator_section.lower()
)
routes.append(
{
"path": path,
"methods": methods,
"file": str(file_path.relative_to(self.path)),
"framework": "Flask",
"requires_auth": requires_auth,
}
)
return routes
def _detect_django_routes(self) -> list[dict]:
"""Detect Django routes from urls.py files."""
routes = []
url_files = [
f for f in self.path.glob("**/urls.py") if self._should_include_file(f)
]
for file_path in url_files:
try:
content = file_path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
# Pattern: path('users/<int:id>/', views.user_detail)
patterns = [
r'path\(["\']([^"\']+)["\']',
r're_path\([r]?["\']([^"\']+)["\']',
]
for pattern in patterns:
matches = re.finditer(pattern, content)
for match in matches:
path = match.group(1)
routes.append(
{
"path": f"/{path}" if not path.startswith("/") else path,
"methods": ["GET", "POST"], # Django allows both by default
"file": str(file_path.relative_to(self.path)),
"framework": "Django",
"requires_auth": False, # Can't easily detect without middleware analysis
}
)
return routes
def _detect_express_routes(self) -> list[dict]:
"""Detect Express/Fastify/Koa routes."""
routes = []
js_files = [
f for f in self.path.glob("**/*.js") if self._should_include_file(f)
]
ts_files = [
f for f in self.path.glob("**/*.ts") if self._should_include_file(f)
]
files_to_check = js_files + ts_files
for file_path in files_to_check:
try:
content = file_path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
# Pattern: app.get('/path', handler) or router.post('/path', middleware, handler)
pattern = (
r'(?:app|router)\.(get|post|put|delete|patch|use)\(["\']([^"\']+)["\']'
)
matches = re.finditer(pattern, content)
for match in matches:
method = match.group(1).upper()
path = match.group(2)
if method == "USE":
# .use() is middleware, might be a route prefix
continue
# Check for auth middleware in the route definition
line_start = content.rfind("\n", 0, match.start()) + 1
line_end = content.find("\n", match.end())
route_line = content[
line_start : line_end if line_end != -1 else len(content)
]
requires_auth = any(
keyword in route_line.lower()
for keyword in ["auth", "authenticate", "protect", "require"]
)
routes.append(
{
"path": path,
"methods": [method],
"file": str(file_path.relative_to(self.path)),
"framework": "Express",
"requires_auth": requires_auth,
}
)
return routes
def _detect_nextjs_routes(self) -> list[dict]:
"""Detect Next.js file-based routes."""
routes = []
# Next.js App Router (app directory)
app_dir = self.path / "app"
if app_dir.exists():
# Find all route.ts/js files
route_files = [
f
for f in app_dir.glob("**/route.{ts,js,tsx,jsx}")
if self._should_include_file(f)
]
for route_file in route_files:
# Convert file path to route path
# app/api/users/[id]/route.ts -> /api/users/:id
relative_path = route_file.parent.relative_to(app_dir)
route_path = "/" + str(relative_path).replace("\\", "/")
# Convert [id] to :id
route_path = re.sub(r"\[([^\]]+)\]", r":\1", route_path)
try:
content = route_file.read_text(encoding="utf-8")
# Detect exported methods: export async function GET(request)
methods = re.findall(
r"export\s+(?:async\s+)?function\s+(GET|POST|PUT|DELETE|PATCH)",
content,
)
if methods:
routes.append(
{
"path": route_path,
"methods": methods,
"file": str(route_file.relative_to(self.path)),
"framework": "Next.js",
"requires_auth": "auth" in content.lower(),
}
)
except (OSError, UnicodeDecodeError):
continue
# Next.js Pages Router (pages/api directory)
pages_api = self.path / "pages" / "api"
if pages_api.exists():
api_files = [
f
for f in pages_api.glob("**/*.{ts,js,tsx,jsx}")
if self._should_include_file(f)
]
for api_file in api_files:
if api_file.name.startswith("_"):
continue
# Convert file path to route
relative_path = api_file.relative_to(pages_api)
route_path = "/api/" + str(relative_path.with_suffix("")).replace(
"\\", "/"
)
# Convert [id] to :id
route_path = re.sub(r"\[([^\]]+)\]", r":\1", route_path)
routes.append(
{
"path": route_path,
"methods": [
"GET",
"POST",
], # Next.js API routes handle all methods
"file": str(api_file.relative_to(self.path)),
"framework": "Next.js",
"requires_auth": False,
}
)
return routes
def _detect_go_routes(self) -> list[dict]:
"""Detect Go framework routes (Gin, Echo, Chi, Fiber)."""
routes = []
go_files = [
f for f in self.path.glob("**/*.go") if self._should_include_file(f)
]
for file_path in go_files:
try:
content = file_path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
# Gin: r.GET("/path", handler)
# Echo: e.POST("/path", handler)
# Chi: r.Get("/path", handler)
# Fiber: app.Get("/path", handler)
pattern = r'(?:r|e|app|router)\.(GET|POST|PUT|DELETE|PATCH|Get|Post|Put|Delete|Patch)\(["\']([^"\']+)["\']'
matches = re.finditer(pattern, content)
for match in matches:
method = match.group(1).upper()
path = match.group(2)
routes.append(
{
"path": path,
"methods": [method],
"file": str(file_path.relative_to(self.path)),
"framework": "Go",
"requires_auth": False,
}
)
return routes
def _detect_rust_routes(self) -> list[dict]:
"""Detect Rust framework routes (Axum, Actix)."""
routes = []
rust_files = [
f for f in self.path.glob("**/*.rs") if self._should_include_file(f)
]
for file_path in rust_files:
try:
content = file_path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
# Axum: .route("/path", get(handler))
# Actix: web::get().to(handler)
patterns = [
r'\.route\(["\']([^"\']+)["\'],\s*(get|post|put|delete|patch)',
r"web::(get|post|put|delete|patch)\(\)",
]
for pattern in patterns:
matches = re.finditer(pattern, content)
for match in matches:
if len(match.groups()) == 2:
path = match.group(1)
method = match.group(2).upper()
else:
path = "/" # Can't determine path from web:: syntax
method = match.group(1).upper()
routes.append(
{
"path": path,
"methods": [method],
"file": str(file_path.relative_to(self.path)),
"framework": "Rust",
"requires_auth": False,
}
)
return routes
@@ -0,0 +1,313 @@
"""
Service Analyzer Module
=======================
Main ServiceAnalyzer class that coordinates all analysis for a single service/package.
Integrates framework detection, route analysis, database models, and context extraction.
"""
from __future__ import annotations
import re
from pathlib import Path
from typing import Any
from .base import BaseAnalyzer
from .context_analyzer import ContextAnalyzer
from .database_detector import DatabaseDetector
from .framework_analyzer import FrameworkAnalyzer
from .route_detector import RouteDetector
class ServiceAnalyzer(BaseAnalyzer):
"""Analyzes a single service/package within a project."""
def __init__(self, service_path: Path, service_name: str):
super().__init__(service_path)
self.name = service_name
self.analysis = {
"name": service_name,
"path": str(service_path),
"language": None,
"framework": None,
"type": None, # backend, frontend, worker, library, etc.
}
def analyze(self) -> dict[str, Any]:
"""Run full analysis on this service."""
self._detect_language_and_framework()
self._detect_service_type()
self._find_key_directories()
self._find_entry_points()
self._detect_dependencies()
self._detect_testing()
self._find_dockerfile()
# Comprehensive context extraction
self._detect_environment_variables()
self._detect_api_routes()
self._detect_database_models()
self._detect_external_services()
self._detect_auth_patterns()
self._detect_migrations()
self._detect_background_jobs()
self._detect_api_documentation()
self._detect_monitoring()
return self.analysis
def _detect_language_and_framework(self) -> None:
"""Detect primary language and framework."""
framework_analyzer = FrameworkAnalyzer(self.path, self.analysis)
framework_analyzer.detect_language_and_framework()
def _detect_service_type(self) -> None:
"""Infer service type from name and content if not already set."""
if self.analysis.get("type"):
return
name_lower = self.name.lower()
# Infer from name
if any(kw in name_lower for kw in ["frontend", "client", "web", "ui", "app"]):
self.analysis["type"] = "frontend"
elif any(kw in name_lower for kw in ["backend", "api", "server", "service"]):
self.analysis["type"] = "backend"
elif any(
kw in name_lower for kw in ["worker", "job", "queue", "task", "celery"]
):
self.analysis["type"] = "worker"
elif any(kw in name_lower for kw in ["scraper", "crawler", "spider"]):
self.analysis["type"] = "scraper"
elif any(kw in name_lower for kw in ["proxy", "gateway", "router"]):
self.analysis["type"] = "proxy"
elif any(
kw in name_lower for kw in ["lib", "shared", "common", "core", "utils"]
):
self.analysis["type"] = "library"
else:
# Try to infer from language and content if name doesn't match
language = self.analysis.get("language")
if language == "Python":
# Check if it's a CLI tool, framework, or backend service
has_run_py = (self.path / "run.py").exists()
has_main_py = (self.path / "main.py").exists()
has_main_module = (self.path / "__main__.py").exists()
# Check for agent/automation framework patterns
has_agent_files = any(
(self.path / f).exists()
for f in ["agent.py", "agents", "runner.py", "runners"]
)
if has_run_py or has_main_py or has_main_module or has_agent_files:
# It's a backend tool/framework/CLI
self.analysis["type"] = "backend"
return
# Default to unknown if no clear indicators
self.analysis["type"] = "unknown"
def _find_key_directories(self) -> None:
"""Find important directories within this service."""
key_dirs = {}
# Common directory patterns
patterns = {
"src": "Source code",
"lib": "Library code",
"app": "Application code",
"api": "API endpoints",
"routes": "Route handlers",
"controllers": "Controllers",
"models": "Data models",
"schemas": "Schemas/DTOs",
"services": "Business logic",
"components": "UI components",
"pages": "Page components",
"views": "Views/templates",
"hooks": "Custom hooks",
"utils": "Utilities",
"helpers": "Helper functions",
"middleware": "Middleware",
"tests": "Tests",
"test": "Tests",
"__tests__": "Tests",
"config": "Configuration",
"tasks": "Background tasks",
"jobs": "Background jobs",
"workers": "Worker processes",
}
for dir_name, purpose in patterns.items():
dir_path = self.path / dir_name
if dir_path.exists() and dir_path.is_dir():
key_dirs[dir_name] = {
"path": str(dir_path.relative_to(self.path)),
"purpose": purpose,
}
if key_dirs:
self.analysis["key_directories"] = key_dirs
def _find_entry_points(self) -> None:
"""Find main entry point files."""
entry_patterns = [
"main.py",
"app.py",
"__main__.py",
"server.py",
"wsgi.py",
"asgi.py",
"index.ts",
"index.js",
"main.ts",
"main.js",
"server.ts",
"server.js",
"app.ts",
"app.js",
"src/index.ts",
"src/index.js",
"src/main.ts",
"src/app.ts",
"src/server.ts",
"src/App.tsx",
"src/App.jsx",
"pages/_app.tsx",
"pages/_app.js", # Next.js
"main.go",
"cmd/main.go",
"src/main.rs",
"src/lib.rs",
]
for pattern in entry_patterns:
if self._exists(pattern):
self.analysis["entry_point"] = pattern
break
def _detect_dependencies(self) -> None:
"""Extract key dependencies."""
if self._exists("package.json"):
pkg = self._read_json("package.json")
if pkg:
deps = pkg.get("dependencies", {})
dev_deps = pkg.get("devDependencies", {})
self.analysis["dependencies"] = list(deps.keys())[:20] # Top 20
self.analysis["dev_dependencies"] = list(dev_deps.keys())[:10]
elif self._exists("requirements.txt"):
content = self._read_file("requirements.txt")
deps = []
for line in content.split("\n"):
line = line.strip()
if line and not line.startswith("#") and not line.startswith("-"):
match = re.match(r"^([a-zA-Z0-9_-]+)", line)
if match:
deps.append(match.group(1))
self.analysis["dependencies"] = deps[:20]
def _detect_testing(self) -> None:
"""Detect testing framework and configuration."""
if self._exists("package.json"):
pkg = self._read_json("package.json")
if pkg:
deps = {**pkg.get("dependencies", {}), **pkg.get("devDependencies", {})}
if "vitest" in deps:
self.analysis["testing"] = "Vitest"
elif "jest" in deps:
self.analysis["testing"] = "Jest"
if "@playwright/test" in deps:
self.analysis["e2e_testing"] = "Playwright"
elif "cypress" in deps:
self.analysis["e2e_testing"] = "Cypress"
elif self._exists("pytest.ini") or self._exists("pyproject.toml"):
self.analysis["testing"] = "pytest"
# Find test directory
for test_dir in ["tests", "test", "__tests__", "spec"]:
if self._exists(test_dir):
self.analysis["test_directory"] = test_dir
break
def _find_dockerfile(self) -> None:
"""Find Dockerfile for this service."""
dockerfile_patterns = [
"Dockerfile",
f"Dockerfile.{self.name}",
f"docker/{self.name}.Dockerfile",
f"docker/Dockerfile.{self.name}",
"../docker/Dockerfile." + self.name,
]
for pattern in dockerfile_patterns:
if self._exists(pattern):
self.analysis["dockerfile"] = pattern
break
def _detect_environment_variables(self) -> None:
"""Detect environment variables."""
context = ContextAnalyzer(self.path, self.analysis)
context.detect_environment_variables()
def _detect_api_routes(self) -> None:
"""Detect API routes."""
route_detector = RouteDetector(self.path)
routes = route_detector.detect_all_routes()
if routes:
self.analysis["api"] = {
"routes": routes,
"total_routes": len(routes),
"methods": list(
set(method for r in routes for method in r.get("methods", []))
),
"protected_routes": [
r["path"] for r in routes if r.get("requires_auth")
],
}
def _detect_database_models(self) -> None:
"""Detect database models."""
db_detector = DatabaseDetector(self.path)
models = db_detector.detect_all_models()
if models:
self.analysis["database"] = {
"models": models,
"total_models": len(models),
"model_names": list(models.keys()),
}
def _detect_external_services(self) -> None:
"""Detect external services."""
context = ContextAnalyzer(self.path, self.analysis)
context.detect_external_services()
def _detect_auth_patterns(self) -> None:
"""Detect authentication patterns."""
context = ContextAnalyzer(self.path, self.analysis)
context.detect_auth_patterns()
def _detect_migrations(self) -> None:
"""Detect database migrations."""
context = ContextAnalyzer(self.path, self.analysis)
context.detect_migrations()
def _detect_background_jobs(self) -> None:
"""Detect background jobs."""
context = ContextAnalyzer(self.path, self.analysis)
context.detect_background_jobs()
def _detect_api_documentation(self) -> None:
"""Detect API documentation."""
context = ContextAnalyzer(self.path, self.analysis)
context.detect_api_documentation()
def _detect_monitoring(self) -> None:
"""Detect monitoring setup."""
context = ContextAnalyzer(self.path, self.analysis)
context.detect_monitoring()
+589
View File
@@ -0,0 +1,589 @@
#!/usr/bin/env python3
"""
CI Discovery Module
===================
Parses CI/CD configuration files to extract test commands and workflows.
Supports GitHub Actions, GitLab CI, CircleCI, and Jenkins.
The CI discovery results are used by:
- QA Agent: To understand existing CI test patterns
- Validation Strategy: To match CI commands
- Planner: To align verification with CI
Usage:
from ci_discovery import CIDiscovery
discovery = CIDiscovery()
result = discovery.discover(project_dir)
if result:
print(f"CI System: {result.ci_system}")
print(f"Test Commands: {result.test_commands}")
"""
from __future__ import annotations
import json
import re
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
# Try to import yaml, fall back gracefully
try:
import yaml
HAS_YAML = True
except ImportError:
HAS_YAML = False
# =============================================================================
# DATA CLASSES
# =============================================================================
@dataclass
class CIWorkflow:
"""
Represents a CI workflow or job.
Attributes:
name: Name of the workflow/job
trigger: What triggers this workflow (push, pull_request, etc.)
steps: List of step names or commands
test_related: Whether this appears to be test-related
"""
name: str
trigger: list[str] = field(default_factory=list)
steps: list[str] = field(default_factory=list)
test_related: bool = False
@dataclass
class CIConfig:
"""
Result of CI configuration discovery.
Attributes:
ci_system: Name of CI system (github_actions, gitlab, circleci, jenkins)
config_files: List of CI config files found
test_commands: Extracted test commands by type
coverage_command: Coverage command if found
workflows: List of discovered workflows
environment_variables: Environment variables used
"""
ci_system: str
config_files: list[str] = field(default_factory=list)
test_commands: dict[str, str] = field(default_factory=dict)
coverage_command: str | None = None
workflows: list[CIWorkflow] = field(default_factory=list)
environment_variables: list[str] = field(default_factory=list)
# =============================================================================
# CI PARSERS
# =============================================================================
class CIDiscovery:
"""
Discovers CI/CD configurations in a project.
Analyzes:
- GitHub Actions (.github/workflows/*.yml)
- GitLab CI (.gitlab-ci.yml)
- CircleCI (.circleci/config.yml)
- Jenkins (Jenkinsfile)
"""
def __init__(self) -> None:
"""Initialize CI discovery."""
self._cache: dict[str, CIConfig | None] = {}
def discover(self, project_dir: Path) -> CIConfig | None:
"""
Discover CI configuration in the project.
Args:
project_dir: Path to the project root
Returns:
CIConfig if CI found, None otherwise
"""
project_dir = Path(project_dir)
cache_key = str(project_dir.resolve())
if cache_key in self._cache:
return self._cache[cache_key]
# Try each CI system
result = None
# GitHub Actions
github_workflows = project_dir / ".github" / "workflows"
if github_workflows.exists():
result = self._parse_github_actions(github_workflows)
# GitLab CI
if not result:
gitlab_ci = project_dir / ".gitlab-ci.yml"
if gitlab_ci.exists():
result = self._parse_gitlab_ci(gitlab_ci)
# CircleCI
if not result:
circleci = project_dir / ".circleci" / "config.yml"
if circleci.exists():
result = self._parse_circleci(circleci)
# Jenkins
if not result:
jenkinsfile = project_dir / "Jenkinsfile"
if jenkinsfile.exists():
result = self._parse_jenkinsfile(jenkinsfile)
self._cache[cache_key] = result
return result
def _parse_github_actions(self, workflows_dir: Path) -> CIConfig:
"""Parse GitHub Actions workflow files."""
result = CIConfig(ci_system="github_actions")
workflow_files = list(workflows_dir.glob("*.yml")) + list(
workflows_dir.glob("*.yaml")
)
for wf_file in workflow_files:
result.config_files.append(
str(wf_file.relative_to(workflows_dir.parent.parent))
)
try:
content = wf_file.read_text(encoding="utf-8")
workflow_data = self._parse_yaml(content)
if not workflow_data:
continue
# Get workflow name
wf_name = workflow_data.get("name", wf_file.stem)
# Get triggers
triggers = []
on_trigger = workflow_data.get("on", {})
if isinstance(on_trigger, str):
triggers = [on_trigger]
elif isinstance(on_trigger, list):
triggers = on_trigger
elif isinstance(on_trigger, dict):
triggers = list(on_trigger.keys())
# Parse jobs
jobs = workflow_data.get("jobs", {})
for job_name, job_config in jobs.items():
if not isinstance(job_config, dict):
continue
steps = job_config.get("steps", [])
step_commands = []
test_related = False
for step in steps:
if not isinstance(step, dict):
continue
# Get step name or command
step_name = step.get("name", "")
run_cmd = step.get("run", "")
uses = step.get("uses", "")
if step_name:
step_commands.append(step_name)
if run_cmd:
step_commands.append(run_cmd)
# Extract test commands
self._extract_test_commands(run_cmd, result)
if uses:
step_commands.append(f"uses: {uses}")
# Check if test-related
test_keywords = ["test", "pytest", "jest", "vitest", "coverage"]
if any(kw in str(step).lower() for kw in test_keywords):
test_related = True
result.workflows.append(
CIWorkflow(
name=f"{wf_name}/{job_name}",
trigger=triggers,
steps=step_commands,
test_related=test_related,
)
)
# Extract environment variables
env = workflow_data.get("env", {})
if isinstance(env, dict):
result.environment_variables.extend(env.keys())
except Exception:
continue
return result
def _parse_gitlab_ci(self, config_file: Path) -> CIConfig:
"""Parse GitLab CI configuration."""
result = CIConfig(
ci_system="gitlab",
config_files=[".gitlab-ci.yml"],
)
try:
content = config_file.read_text(encoding="utf-8")
data = self._parse_yaml(content)
if not data:
return result
# Parse jobs (top-level keys that aren't special keywords)
special_keys = {
"stages",
"variables",
"image",
"services",
"before_script",
"after_script",
"cache",
"include",
"default",
"workflow",
}
for key, value in data.items():
if key.startswith(".") or key in special_keys:
continue
if not isinstance(value, dict):
continue
job_config = value
script = job_config.get("script", [])
if isinstance(script, str):
script = [script]
test_related = any(
kw in str(script).lower()
for kw in ["test", "pytest", "jest", "vitest", "coverage"]
)
result.workflows.append(
CIWorkflow(
name=key,
trigger=job_config.get("only", [])
or job_config.get("rules", []),
steps=script,
test_related=test_related,
)
)
# Extract test commands
for cmd in script:
if isinstance(cmd, str):
self._extract_test_commands(cmd, result)
# Extract variables
variables = data.get("variables", {})
if isinstance(variables, dict):
result.environment_variables.extend(variables.keys())
except Exception:
pass
return result
def _parse_circleci(self, config_file: Path) -> CIConfig:
"""Parse CircleCI configuration."""
result = CIConfig(
ci_system="circleci",
config_files=[".circleci/config.yml"],
)
try:
content = config_file.read_text(encoding="utf-8")
data = self._parse_yaml(content)
if not data:
return result
# Parse jobs
jobs = data.get("jobs", {})
for job_name, job_config in jobs.items():
if not isinstance(job_config, dict):
continue
steps = job_config.get("steps", [])
step_commands = []
test_related = False
for step in steps:
if isinstance(step, str):
step_commands.append(step)
elif isinstance(step, dict):
if "run" in step:
run = step["run"]
if isinstance(run, str):
step_commands.append(run)
self._extract_test_commands(run, result)
elif isinstance(run, dict):
cmd = run.get("command", "")
step_commands.append(cmd)
self._extract_test_commands(cmd, result)
if any(
kw in str(step).lower()
for kw in ["test", "pytest", "jest", "coverage"]
):
test_related = True
result.workflows.append(
CIWorkflow(
name=job_name,
trigger=[],
steps=step_commands,
test_related=test_related,
)
)
except Exception:
pass
return result
def _parse_jenkinsfile(self, jenkinsfile: Path) -> CIConfig:
"""Parse Jenkinsfile (basic extraction)."""
result = CIConfig(
ci_system="jenkins",
config_files=["Jenkinsfile"],
)
try:
content = jenkinsfile.read_text(encoding="utf-8")
# Extract sh commands using regex
sh_pattern = re.compile(r'sh\s+[\'"]([^\'"]+)[\'"]')
matches = sh_pattern.findall(content)
steps = []
test_related = False
for cmd in matches:
steps.append(cmd)
self._extract_test_commands(cmd, result)
if any(
kw in cmd.lower() for kw in ["test", "pytest", "jest", "coverage"]
):
test_related = True
# Extract stage names
stage_pattern = re.compile(r'stage\s*\([\'"]([^\'"]+)[\'"]\)')
stages = stage_pattern.findall(content)
for stage in stages:
result.workflows.append(
CIWorkflow(
name=stage,
trigger=[],
steps=steps if "test" in stage.lower() else [],
test_related="test" in stage.lower(),
)
)
except Exception:
pass
return result
def _parse_yaml(self, content: str) -> dict | None:
"""Parse YAML content, with fallback to basic parsing if yaml not available."""
if HAS_YAML:
try:
return yaml.safe_load(content)
except Exception:
return None
# Basic fallback for simple YAML (very limited)
# This won't work for complex structures
return None
def _extract_test_commands(self, cmd: str, result: CIConfig) -> None:
"""Extract test commands from a command string."""
cmd_lower = cmd.lower()
# Python pytest
if "pytest" in cmd_lower:
if "pytest" not in result.test_commands:
result.test_commands["unit"] = cmd.strip()
if "--cov" in cmd_lower:
result.coverage_command = cmd.strip()
# Node.js test commands
if (
"npm test" in cmd_lower
or "yarn test" in cmd_lower
or "pnpm test" in cmd_lower
):
if "unit" not in result.test_commands:
result.test_commands["unit"] = cmd.strip()
# Jest/Vitest
if "jest" in cmd_lower or "vitest" in cmd_lower:
if "unit" not in result.test_commands:
result.test_commands["unit"] = cmd.strip()
if "--coverage" in cmd_lower:
result.coverage_command = cmd.strip()
# E2E testing
if "playwright" in cmd_lower:
result.test_commands["e2e"] = cmd.strip()
if "cypress" in cmd_lower:
result.test_commands["e2e"] = cmd.strip()
# Integration tests
if "integration" in cmd_lower:
result.test_commands["integration"] = cmd.strip()
# Go tests
if "go test" in cmd_lower:
if "unit" not in result.test_commands:
result.test_commands["unit"] = cmd.strip()
# Rust tests
if "cargo test" in cmd_lower:
if "unit" not in result.test_commands:
result.test_commands["unit"] = cmd.strip()
def to_dict(self, result: CIConfig) -> dict[str, Any]:
"""Convert result to dictionary for JSON serialization."""
return {
"ci_system": result.ci_system,
"config_files": result.config_files,
"test_commands": result.test_commands,
"coverage_command": result.coverage_command,
"workflows": [
{
"name": w.name,
"trigger": w.trigger,
"steps": w.steps,
"test_related": w.test_related,
}
for w in result.workflows
],
"environment_variables": result.environment_variables,
}
def clear_cache(self) -> None:
"""Clear the internal cache."""
self._cache.clear()
# =============================================================================
# CONVENIENCE FUNCTIONS
# =============================================================================
def discover_ci(project_dir: Path) -> CIConfig | None:
"""
Convenience function to discover CI configuration.
Args:
project_dir: Path to project root
Returns:
CIConfig if found, None otherwise
"""
discovery = CIDiscovery()
return discovery.discover(project_dir)
def get_ci_test_commands(project_dir: Path) -> dict[str, str]:
"""
Get test commands from CI configuration.
Args:
project_dir: Path to project root
Returns:
Dictionary of test type to command
"""
discovery = CIDiscovery()
result = discovery.discover(project_dir)
if result:
return result.test_commands
return {}
def get_ci_system(project_dir: Path) -> str | None:
"""
Get the CI system name if configured.
Args:
project_dir: Path to project root
Returns:
CI system name or None
"""
discovery = CIDiscovery()
result = discovery.discover(project_dir)
if result:
return result.ci_system
return None
# =============================================================================
# CLI
# =============================================================================
def main() -> None:
"""CLI entry point for testing."""
import argparse
parser = argparse.ArgumentParser(description="Discover CI configuration")
parser.add_argument("project_dir", type=Path, help="Path to project root")
parser.add_argument("--json", action="store_true", help="Output as JSON")
args = parser.parse_args()
discovery = CIDiscovery()
result = discovery.discover(args.project_dir)
if not result:
print("No CI configuration found")
return
if args.json:
print(json.dumps(discovery.to_dict(result), indent=2))
else:
print(f"CI System: {result.ci_system}")
print(f"Config Files: {', '.join(result.config_files)}")
print("\nTest Commands:")
for test_type, cmd in result.test_commands.items():
print(f" {test_type}: {cmd}")
if result.coverage_command:
print(f"\nCoverage Command: {result.coverage_command}")
print(f"\nWorkflows ({len(result.workflows)}):")
for w in result.workflows:
marker = "[TEST]" if w.test_related else ""
print(f" - {w.name} {marker}")
if w.trigger:
print(f" Triggers: {', '.join(str(t) for t in w.trigger)}")
if result.environment_variables:
print(f"\nEnvironment Variables: {', '.join(result.environment_variables)}")
if __name__ == "__main__":
main()
+643
View File
@@ -0,0 +1,643 @@
"""
Insight Extractor
=================
Automatically extracts structured insights from completed coding sessions.
Runs after each session to capture rich, actionable knowledge for Graphiti memory.
Uses the Claude Agent SDK (same as the rest of the system) for extraction.
Falls back to generic insights if extraction fails (never blocks the build).
"""
from __future__ import annotations
import json
import logging
import os
import subprocess
from pathlib import Path
from typing import Any
logger = logging.getLogger(__name__)
# Check for Claude SDK availability
try:
from claude_agent_sdk import ClaudeAgentOptions, ClaudeSDKClient
SDK_AVAILABLE = True
except ImportError:
SDK_AVAILABLE = False
ClaudeAgentOptions = None
ClaudeSDKClient = None
from core.auth import ensure_claude_code_oauth_token, get_auth_token
# Default model for insight extraction (fast and cheap)
# Note: Using Haiku 4.5 for fast, cheap extraction. Haiku does not support
# extended thinking, so thinking_default is set to "none" in models.py
DEFAULT_EXTRACTION_MODEL = "claude-haiku-4-5-20251001"
# Maximum diff size to send to the LLM (avoid context limits)
MAX_DIFF_CHARS = 15000
# Maximum attempt history entries to include
MAX_ATTEMPTS_TO_INCLUDE = 3
def is_extraction_enabled() -> bool:
"""Check if insight extraction is enabled."""
# Extraction requires Claude SDK and authentication token
if not SDK_AVAILABLE:
return False
if not get_auth_token():
return False
enabled_str = os.environ.get("INSIGHT_EXTRACTION_ENABLED", "true").lower()
return enabled_str in ("true", "1", "yes")
def get_extraction_model() -> str:
"""Get the model to use for insight extraction."""
return os.environ.get("INSIGHT_EXTRACTOR_MODEL", DEFAULT_EXTRACTION_MODEL)
# =============================================================================
# Git Helpers
# =============================================================================
def get_session_diff(
project_dir: Path,
commit_before: str | None,
commit_after: str | None,
) -> str:
"""
Get the git diff between two commits.
Args:
project_dir: Project root directory
commit_before: Commit hash before session (or None)
commit_after: Commit hash after session (or None)
Returns:
Diff text (truncated if too large)
"""
if not commit_before or not commit_after:
return "(No commits to diff)"
if commit_before == commit_after:
return "(No changes - same commit)"
try:
result = subprocess.run(
["git", "diff", commit_before, commit_after],
cwd=project_dir,
capture_output=True,
text=True,
timeout=30,
)
diff = result.stdout
if len(diff) > MAX_DIFF_CHARS:
# Truncate and add note
diff = (
diff[:MAX_DIFF_CHARS] + f"\n\n... (truncated, {len(diff)} chars total)"
)
return diff if diff else "(Empty diff)"
except subprocess.TimeoutExpired:
logger.warning("Git diff timed out")
return "(Git diff timed out)"
except Exception as e:
logger.warning(f"Failed to get git diff: {e}")
return f"(Failed to get diff: {e})"
def get_changed_files(
project_dir: Path,
commit_before: str | None,
commit_after: str | None,
) -> list[str]:
"""
Get list of files changed between two commits.
Args:
project_dir: Project root directory
commit_before: Commit hash before session
commit_after: Commit hash after session
Returns:
List of changed file paths
"""
if not commit_before or not commit_after or commit_before == commit_after:
return []
try:
result = subprocess.run(
["git", "diff", "--name-only", commit_before, commit_after],
cwd=project_dir,
capture_output=True,
text=True,
timeout=10,
)
files = [f.strip() for f in result.stdout.strip().split("\n") if f.strip()]
return files
except Exception as e:
logger.warning(f"Failed to get changed files: {e}")
return []
def get_commit_messages(
project_dir: Path,
commit_before: str | None,
commit_after: str | None,
) -> str:
"""Get commit messages between two commits."""
if not commit_before or not commit_after or commit_before == commit_after:
return "(No commits)"
try:
result = subprocess.run(
["git", "log", "--oneline", f"{commit_before}..{commit_after}"],
cwd=project_dir,
capture_output=True,
text=True,
timeout=10,
)
return result.stdout.strip() if result.stdout.strip() else "(No commits)"
except Exception as e:
logger.warning(f"Failed to get commit messages: {e}")
return f"(Failed: {e})"
# =============================================================================
# Input Gathering
# =============================================================================
def gather_extraction_inputs(
spec_dir: Path,
project_dir: Path,
subtask_id: str,
session_num: int,
commit_before: str | None,
commit_after: str | None,
success: bool,
recovery_manager: Any,
) -> dict:
"""
Gather all inputs needed for insight extraction.
Args:
spec_dir: Spec directory
project_dir: Project root
subtask_id: The subtask that was worked on
session_num: Session number
commit_before: Commit before session
commit_after: Commit after session
success: Whether session succeeded
recovery_manager: Recovery manager with attempt history
Returns:
Dict with all inputs for the extractor
"""
# Get subtask description from implementation plan
subtask_description = _get_subtask_description(spec_dir, subtask_id)
# Get git diff
diff = get_session_diff(project_dir, commit_before, commit_after)
# Get changed files
changed_files = get_changed_files(project_dir, commit_before, commit_after)
# Get commit messages
commit_messages = get_commit_messages(project_dir, commit_before, commit_after)
# Get attempt history
attempt_history = _get_attempt_history(recovery_manager, subtask_id)
return {
"subtask_id": subtask_id,
"subtask_description": subtask_description,
"session_num": session_num,
"success": success,
"diff": diff,
"changed_files": changed_files,
"commit_messages": commit_messages,
"attempt_history": attempt_history,
}
def _get_subtask_description(spec_dir: Path, subtask_id: str) -> str:
"""Get subtask description from implementation plan."""
plan_file = spec_dir / "implementation_plan.json"
if not plan_file.exists():
return f"Subtask: {subtask_id}"
try:
with open(plan_file, encoding="utf-8") as f:
plan = json.load(f)
# Search through phases for the subtask
for phase in plan.get("phases", []):
for subtask in phase.get("subtasks", []):
if subtask.get("id") == subtask_id:
return subtask.get("description", f"Subtask: {subtask_id}")
return f"Subtask: {subtask_id}"
except Exception as e:
logger.warning(f"Failed to load subtask description: {e}")
return f"Subtask: {subtask_id}"
def _get_attempt_history(recovery_manager: Any, subtask_id: str) -> list[dict]:
"""Get previous attempt history for this subtask."""
if not recovery_manager:
return []
try:
history = recovery_manager.get_subtask_history(subtask_id)
attempts = history.get("attempts", [])
# Limit to recent attempts
return attempts[-MAX_ATTEMPTS_TO_INCLUDE:]
except Exception as e:
logger.warning(f"Failed to get attempt history: {e}")
return []
# =============================================================================
# LLM Extraction
# =============================================================================
def _build_extraction_prompt(inputs: dict) -> str:
"""Build the prompt for insight extraction."""
prompt_file = Path(__file__).parent / "prompts" / "insight_extractor.md"
if prompt_file.exists():
base_prompt = prompt_file.read_text(encoding="utf-8")
else:
# Fallback if prompt file missing
base_prompt = """Extract structured insights from this coding session.
Output ONLY valid JSON with: file_insights, patterns_discovered, gotchas_discovered, approach_outcome, recommendations"""
# Build session context
session_context = f"""
---
## SESSION DATA
### Subtask
- **ID**: {inputs["subtask_id"]}
- **Description**: {inputs["subtask_description"]}
- **Session Number**: {inputs["session_num"]}
- **Outcome**: {"SUCCESS" if inputs["success"] else "FAILED"}
### Files Changed
{chr(10).join(f"- {f}" for f in inputs["changed_files"]) if inputs["changed_files"] else "(No files changed)"}
### Commit Messages
{inputs["commit_messages"]}
### Git Diff
```diff
{inputs["diff"]}
```
### Previous Attempts
{_format_attempt_history(inputs["attempt_history"])}
---
Now analyze this session and output ONLY the JSON object.
"""
return base_prompt + session_context
def _format_attempt_history(attempts: list[dict]) -> str:
"""Format attempt history for the prompt."""
if not attempts:
return "(First attempt - no previous history)"
lines = []
for i, attempt in enumerate(attempts, 1):
success = "SUCCESS" if attempt.get("success") else "FAILED"
approach = attempt.get("approach", "Unknown approach")
error = attempt.get("error", "")
lines.append(f"**Attempt {i}** ({success}): {approach}")
if error:
lines.append(f" Error: {error}")
return "\n".join(lines)
async def run_insight_extraction(
inputs: dict, project_dir: Path | None = None
) -> dict | None:
"""
Run the insight extraction using Claude Agent SDK.
Args:
inputs: Gathered session inputs
project_dir: Project directory for SDK context (optional)
Returns:
Extracted insights dict or None if failed
"""
if not SDK_AVAILABLE:
logger.warning("Claude SDK not available, skipping insight extraction")
return None
if not get_auth_token():
logger.warning("No authentication token found, skipping insight extraction")
return None
# Ensure SDK can find the token
ensure_claude_code_oauth_token()
model = get_extraction_model()
prompt = _build_extraction_prompt(inputs)
# Use current directory if project_dir not specified
cwd = str(project_dir.resolve()) if project_dir else os.getcwd()
try:
# Use simple_client for insight extraction
from pathlib import Path
from core.simple_client import create_simple_client
client = create_simple_client(
agent_type="insights",
model=model,
system_prompt=(
"You are an expert code analyst. You extract structured insights from coding sessions. "
"Always respond with valid JSON only, no markdown formatting or explanations."
),
cwd=Path(cwd) if cwd else None,
)
# Use async context manager
async with client:
await client.query(prompt)
# Collect the response
response_text = ""
message_count = 0
text_blocks_found = 0
async for msg in client.receive_response():
msg_type = type(msg).__name__
message_count += 1
if msg_type == "AssistantMessage" and hasattr(msg, "content"):
for block in msg.content:
# Must check block type - only TextBlock has .text attribute
block_type = type(block).__name__
if block_type == "TextBlock" and hasattr(block, "text"):
text_blocks_found += 1
if block.text: # Only add non-empty text
response_text += block.text
else:
logger.debug(
f"Found empty TextBlock in response (block #{text_blocks_found})"
)
# Log response collection summary
logger.debug(
f"Insight extraction response: {message_count} messages, "
f"{text_blocks_found} text blocks, {len(response_text)} chars collected"
)
# Validate we received content before parsing
if not response_text.strip():
logger.warning(
f"Insight extraction returned empty response. "
f"Messages received: {message_count}, TextBlocks found: {text_blocks_found}. "
f"This may indicate the AI model did not respond with text content."
)
return None
# Parse JSON from response
return parse_insights(response_text)
except Exception as e:
logger.warning(f"Insight extraction failed: {e}")
return None
def parse_insights(response_text: str) -> dict | None:
"""
Parse the LLM response into structured insights.
Args:
response_text: Raw LLM response
Returns:
Parsed insights dict or None if parsing failed
"""
# Try to extract JSON from the response
text = response_text.strip()
# Early validation - check for empty response
if not text:
logger.warning("Cannot parse insights: response text is empty")
return None
# Handle markdown code blocks
if text.startswith("```"):
# Remove code block markers
lines = text.split("\n")
# Remove first line (```json or ```)
if lines[0].startswith("```"):
lines = lines[1:]
# Remove last line if it's ```
if lines and lines[-1].strip() == "```":
lines = lines[:-1]
text = "\n".join(lines).strip()
# Check again after removing code blocks
if not text:
logger.warning(
"Cannot parse insights: response contained only markdown code block markers with no content"
)
return None
try:
insights = json.loads(text)
# Validate structure
if not isinstance(insights, dict):
logger.warning(
f"Insights is not a dict, got type: {type(insights).__name__}"
)
return None
# Ensure required keys exist with defaults
insights.setdefault("file_insights", [])
insights.setdefault("patterns_discovered", [])
insights.setdefault("gotchas_discovered", [])
insights.setdefault("approach_outcome", {})
insights.setdefault("recommendations", [])
return insights
except json.JSONDecodeError as e:
logger.warning(f"Failed to parse insights JSON: {e}")
# Show more context in the error message
preview_length = min(500, len(text))
logger.warning(
f"Response text preview (first {preview_length} chars): {text[:preview_length]}"
)
if len(text) > preview_length:
logger.warning(f"... (total length: {len(text)} chars)")
return None
# =============================================================================
# Main Entry Point
# =============================================================================
async def extract_session_insights(
spec_dir: Path,
project_dir: Path,
subtask_id: str,
session_num: int,
commit_before: str | None,
commit_after: str | None,
success: bool,
recovery_manager: Any,
) -> dict:
"""
Extract insights from a completed coding session.
This is the main entry point called from post_session_processing().
Falls back to generic insights if extraction fails.
Args:
spec_dir: Spec directory
project_dir: Project root
subtask_id: Subtask that was worked on
session_num: Session number
commit_before: Commit before session
commit_after: Commit after session
success: Whether session succeeded
recovery_manager: Recovery manager with attempt history
Returns:
Insights dict (rich if extraction succeeded, generic if failed)
"""
# Check if extraction is enabled
if not is_extraction_enabled():
logger.info("Insight extraction disabled")
return _get_generic_insights(subtask_id, success)
# Check for no changes
if commit_before == commit_after:
logger.info("No changes to extract insights from")
return _get_generic_insights(subtask_id, success)
try:
# Gather inputs
inputs = gather_extraction_inputs(
spec_dir=spec_dir,
project_dir=project_dir,
subtask_id=subtask_id,
session_num=session_num,
commit_before=commit_before,
commit_after=commit_after,
success=success,
recovery_manager=recovery_manager,
)
# Run extraction
extracted = await run_insight_extraction(inputs, project_dir=project_dir)
if extracted:
# Add metadata
extracted["subtask_id"] = subtask_id
extracted["session_num"] = session_num
extracted["success"] = success
extracted["changed_files"] = inputs["changed_files"]
logger.info(
f"Extracted insights: {len(extracted.get('file_insights', []))} file insights, "
f"{len(extracted.get('patterns_discovered', []))} patterns, "
f"{len(extracted.get('gotchas_discovered', []))} gotchas"
)
return extracted
else:
logger.warning("Extraction returned no results, using generic insights")
return _get_generic_insights(subtask_id, success)
except Exception as e:
logger.warning(f"Insight extraction failed: {e}, using generic insights")
return _get_generic_insights(subtask_id, success)
def _get_generic_insights(subtask_id: str, success: bool) -> dict:
"""Return generic insights when extraction fails or is disabled."""
return {
"file_insights": [],
"patterns_discovered": [],
"gotchas_discovered": [],
"approach_outcome": {
"success": success,
"approach_used": f"Implemented subtask: {subtask_id}",
"why_it_worked": None,
"why_it_failed": None,
"alternatives_tried": [],
},
"recommendations": [],
"subtask_id": subtask_id,
"success": success,
"changed_files": [],
}
# =============================================================================
# CLI for Testing
# =============================================================================
if __name__ == "__main__":
import argparse
import asyncio
parser = argparse.ArgumentParser(description="Test insight extraction")
parser.add_argument("--spec-dir", type=Path, required=True, help="Spec directory")
parser.add_argument(
"--project-dir", type=Path, required=True, help="Project directory"
)
parser.add_argument(
"--commit-before", type=str, required=True, help="Commit before session"
)
parser.add_argument(
"--commit-after", type=str, required=True, help="Commit after session"
)
parser.add_argument(
"--subtask-id", type=str, default="test-subtask", help="Subtask ID"
)
args = parser.parse_args()
async def main():
insights = await extract_session_insights(
spec_dir=args.spec_dir,
project_dir=args.project_dir,
subtask_id=args.subtask_id,
session_num=1,
commit_before=args.commit_before,
commit_after=args.commit_after,
success=True,
recovery_manager=None,
)
print(json.dumps(insights, indent=2))
asyncio.run(main())
+109
View File
@@ -0,0 +1,109 @@
"""
Smart Project Analyzer for Dynamic Security Profiles
=====================================================
FACADE MODULE: This module re-exports all functionality from the
auto-claude/project/ package for backward compatibility.
The implementation has been refactored into focused modules:
- project/command_registry.py - Command registries
- project/models.py - Data structures
- project/config_parser.py - Config file parsing
- project/stack_detector.py - Stack detection
- project/framework_detector.py - Framework detection
- project/structure_analyzer.py - Project structure analysis
- project/analyzer.py - Main orchestration
This file maintains the original API so existing imports continue to work.
This system:
1. Detects languages, frameworks, databases, and infrastructure
2. Parses package.json scripts, Makefile targets, pyproject.toml scripts
3. Builds a tailored security profile for the specific project
4. Caches the profile for subsequent runs
5. Can re-analyze when project structure changes
The goal: Allow an AI developer to run any command that's legitimately
needed for the detected tech stack, while blocking dangerous operations.
"""
# Re-export all public API from the project module
from __future__ import annotations
from project import (
# Command registries
BASE_COMMANDS,
VALIDATED_COMMANDS,
CustomScripts,
# Main classes
ProjectAnalyzer,
SecurityProfile,
TechnologyStack,
# Utility functions
get_or_create_profile,
is_command_allowed,
needs_validation,
)
# Also re-export command registries for backward compatibility
from project.command_registry import (
CLOUD_COMMANDS,
CODE_QUALITY_COMMANDS,
DATABASE_COMMANDS,
FRAMEWORK_COMMANDS,
INFRASTRUCTURE_COMMANDS,
LANGUAGE_COMMANDS,
PACKAGE_MANAGER_COMMANDS,
VERSION_MANAGER_COMMANDS,
)
__all__ = [
# Main classes
"ProjectAnalyzer",
"SecurityProfile",
"TechnologyStack",
"CustomScripts",
# Utility functions
"get_or_create_profile",
"is_command_allowed",
"needs_validation",
# Base command sets
"BASE_COMMANDS",
"VALIDATED_COMMANDS",
# Technology-specific command sets
"LANGUAGE_COMMANDS",
"PACKAGE_MANAGER_COMMANDS",
"FRAMEWORK_COMMANDS",
"DATABASE_COMMANDS",
"INFRASTRUCTURE_COMMANDS",
"CLOUD_COMMANDS",
"CODE_QUALITY_COMMANDS",
"VERSION_MANAGER_COMMANDS",
]
# =============================================================================
# CLI for testing
# =============================================================================
if __name__ == "__main__":
import sys
from pathlib import Path
if len(sys.argv) < 2:
print("Usage: python project_analyzer.py <project_dir> [--force]")
sys.exit(1)
project_dir = Path(sys.argv[1])
force = "--force" in sys.argv
if not project_dir.exists():
print(f"Error: {project_dir} does not exist")
sys.exit(1)
profile = get_or_create_profile(project_dir, force_reanalyze=force)
print("\nAllowed commands:")
for cmd in sorted(profile.get_all_allowed_commands()):
print(f" {cmd}")
+591
View File
@@ -0,0 +1,591 @@
#!/usr/bin/env python3
"""
Risk Classifier Module
======================
Reads the AI-generated complexity_assessment.json and provides programmatic
access to risk classification and validation recommendations.
This module serves as the bridge between the AI complexity assessor prompt
and the rest of the validation system.
Usage:
from risk_classifier import RiskClassifier
classifier = RiskClassifier()
assessment = classifier.load_assessment(spec_dir)
if classifier.should_skip_validation(spec_dir):
print("Validation can be skipped for this task")
test_types = classifier.get_required_test_types(spec_dir)
"""
from __future__ import annotations
import json
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
# =============================================================================
# DATA CLASSES
# =============================================================================
@dataclass
class ScopeAnalysis:
"""Analysis of task scope."""
estimated_files: int = 0
estimated_services: int = 0
is_cross_cutting: bool = False
notes: str = ""
@dataclass
class IntegrationAnalysis:
"""Analysis of external integrations."""
external_services: list[str] = field(default_factory=list)
new_dependencies: list[str] = field(default_factory=list)
research_needed: bool = False
notes: str = ""
@dataclass
class InfrastructureAnalysis:
"""Analysis of infrastructure requirements."""
docker_changes: bool = False
database_changes: bool = False
config_changes: bool = False
notes: str = ""
@dataclass
class KnowledgeAnalysis:
"""Analysis of knowledge requirements."""
patterns_exist: bool = True
research_required: bool = False
unfamiliar_tech: list[str] = field(default_factory=list)
notes: str = ""
@dataclass
class RiskAnalysis:
"""Analysis of task risk."""
level: str = "low" # low, medium, high
concerns: list[str] = field(default_factory=list)
notes: str = ""
@dataclass
class ComplexityAnalysis:
"""Full complexity analysis from the AI assessor."""
scope: ScopeAnalysis = field(default_factory=ScopeAnalysis)
integrations: IntegrationAnalysis = field(default_factory=IntegrationAnalysis)
infrastructure: InfrastructureAnalysis = field(
default_factory=InfrastructureAnalysis
)
knowledge: KnowledgeAnalysis = field(default_factory=KnowledgeAnalysis)
risk: RiskAnalysis = field(default_factory=RiskAnalysis)
@dataclass
class ValidationRecommendations:
"""Validation recommendations from the AI assessor."""
risk_level: str = "medium" # trivial, low, medium, high, critical
skip_validation: bool = False
minimal_mode: bool = False
test_types_required: list[str] = field(default_factory=lambda: ["unit"])
security_scan_required: bool = False
staging_deployment_required: bool = False
reasoning: str = ""
@dataclass
class AssessmentFlags:
"""Flags indicating special requirements."""
needs_research: bool = False
needs_self_critique: bool = False
needs_infrastructure_setup: bool = False
@dataclass
class RiskAssessment:
"""Complete risk assessment from complexity_assessment.json."""
complexity: str # simple, standard, complex
workflow_type: str # feature, refactor, investigation, migration, simple
confidence: float
reasoning: str
analysis: ComplexityAnalysis
recommended_phases: list[str]
flags: AssessmentFlags
validation: ValidationRecommendations
created_at: str | None = None
@property
def risk_level(self) -> str:
"""Get the risk level from validation recommendations."""
return self.validation.risk_level
# =============================================================================
# RISK CLASSIFIER
# =============================================================================
class RiskClassifier:
"""
Reads AI-generated complexity_assessment.json and provides risk classification.
The complexity_assessment.json is generated by the AI complexity assessor
agent using the complexity_assessor.md prompt. This module parses that output
and provides programmatic access to the risk classification.
"""
def __init__(self) -> None:
"""Initialize the risk classifier."""
self._cache: dict[str, RiskAssessment] = {}
def load_assessment(self, spec_dir: Path) -> RiskAssessment | None:
"""
Load complexity_assessment.json from spec directory.
Args:
spec_dir: Path to the spec directory containing complexity_assessment.json
Returns:
RiskAssessment object if file exists and is valid, None otherwise
"""
spec_dir = Path(spec_dir)
cache_key = str(spec_dir.resolve())
# Return cached result if available
if cache_key in self._cache:
return self._cache[cache_key]
assessment_file = spec_dir / "complexity_assessment.json"
if not assessment_file.exists():
return None
try:
with open(assessment_file, encoding="utf-8") as f:
data = json.load(f)
assessment = self._parse_assessment(data)
self._cache[cache_key] = assessment
return assessment
except (json.JSONDecodeError, KeyError, TypeError) as e:
# Log error but don't crash - return None to allow fallback behavior
print(f"Warning: Failed to parse complexity_assessment.json: {e}")
return None
def _parse_assessment(self, data: dict[str, Any]) -> RiskAssessment:
"""Parse raw JSON data into a RiskAssessment object."""
# Parse analysis sections
analysis_data = data.get("analysis", {})
analysis = ComplexityAnalysis(
scope=self._parse_scope(analysis_data.get("scope", {})),
integrations=self._parse_integrations(
analysis_data.get("integrations", {})
),
infrastructure=self._parse_infrastructure(
analysis_data.get("infrastructure", {})
),
knowledge=self._parse_knowledge(analysis_data.get("knowledge", {})),
risk=self._parse_risk(analysis_data.get("risk", {})),
)
# Parse flags
flags_data = data.get("flags", {})
flags = AssessmentFlags(
needs_research=flags_data.get("needs_research", False),
needs_self_critique=flags_data.get("needs_self_critique", False),
needs_infrastructure_setup=flags_data.get(
"needs_infrastructure_setup", False
),
)
# Parse validation recommendations
validation_data = data.get("validation_recommendations", {})
validation = self._parse_validation_recommendations(validation_data, analysis)
return RiskAssessment(
complexity=data.get("complexity", "standard"),
workflow_type=data.get("workflow_type", "feature"),
confidence=float(data.get("confidence", 0.5)),
reasoning=data.get("reasoning", ""),
analysis=analysis,
recommended_phases=data.get("recommended_phases", []),
flags=flags,
validation=validation,
created_at=data.get("created_at"),
)
def _parse_scope(self, data: dict[str, Any]) -> ScopeAnalysis:
"""Parse scope analysis section."""
return ScopeAnalysis(
estimated_files=int(data.get("estimated_files", 0)),
estimated_services=int(data.get("estimated_services", 0)),
is_cross_cutting=bool(data.get("is_cross_cutting", False)),
notes=str(data.get("notes", "")),
)
def _parse_integrations(self, data: dict[str, Any]) -> IntegrationAnalysis:
"""Parse integrations analysis section."""
return IntegrationAnalysis(
external_services=list(data.get("external_services", [])),
new_dependencies=list(data.get("new_dependencies", [])),
research_needed=bool(data.get("research_needed", False)),
notes=str(data.get("notes", "")),
)
def _parse_infrastructure(self, data: dict[str, Any]) -> InfrastructureAnalysis:
"""Parse infrastructure analysis section."""
return InfrastructureAnalysis(
docker_changes=bool(data.get("docker_changes", False)),
database_changes=bool(data.get("database_changes", False)),
config_changes=bool(data.get("config_changes", False)),
notes=str(data.get("notes", "")),
)
def _parse_knowledge(self, data: dict[str, Any]) -> KnowledgeAnalysis:
"""Parse knowledge analysis section."""
return KnowledgeAnalysis(
patterns_exist=bool(data.get("patterns_exist", True)),
research_required=bool(data.get("research_required", False)),
unfamiliar_tech=list(data.get("unfamiliar_tech", [])),
notes=str(data.get("notes", "")),
)
def _parse_risk(self, data: dict[str, Any]) -> RiskAnalysis:
"""Parse risk analysis section."""
return RiskAnalysis(
level=str(data.get("level", "low")),
concerns=list(data.get("concerns", [])),
notes=str(data.get("notes", "")),
)
def _parse_validation_recommendations(
self, data: dict[str, Any], analysis: ComplexityAnalysis
) -> ValidationRecommendations:
"""
Parse validation recommendations section.
If validation_recommendations is not present in the JSON (older assessments),
infer appropriate values from the analysis.
"""
if data:
# New format with explicit validation recommendations
return ValidationRecommendations(
risk_level=str(data.get("risk_level", "medium")),
skip_validation=bool(data.get("skip_validation", False)),
minimal_mode=bool(data.get("minimal_mode", False)),
test_types_required=list(data.get("test_types_required", ["unit"])),
security_scan_required=bool(data.get("security_scan_required", False)),
staging_deployment_required=bool(
data.get("staging_deployment_required", False)
),
reasoning=str(data.get("reasoning", "")),
)
else:
# Infer from analysis (backward compatibility)
return self._infer_validation_recommendations(analysis)
def _infer_validation_recommendations(
self, analysis: ComplexityAnalysis
) -> ValidationRecommendations:
"""
Infer validation recommendations from analysis when not explicitly provided.
This provides backward compatibility with older complexity assessments
that don't have the validation_recommendations section.
"""
risk_level = analysis.risk.level
# Map old risk levels to new ones
risk_mapping = {
"low": "low",
"medium": "medium",
"high": "high",
}
normalized_risk = risk_mapping.get(risk_level, "medium")
# Infer test types based on risk
test_types_map = {
"low": ["unit"],
"medium": ["unit", "integration"],
"high": ["unit", "integration", "e2e"],
}
test_types = test_types_map.get(normalized_risk, ["unit", "integration"])
# Security scan for high risk or security-related concerns
security_keywords = [
"security",
"auth",
"password",
"credential",
"token",
"api key",
]
has_security_concerns = any(
kw in str(analysis.risk.concerns).lower() for kw in security_keywords
)
security_scan_required = normalized_risk == "high" or has_security_concerns
# Staging for database or infrastructure changes
staging_required = (
analysis.infrastructure.database_changes
and normalized_risk in ["medium", "high"]
)
# Minimal mode for simple changes
minimal_mode = (
analysis.scope.estimated_files <= 2
and analysis.scope.estimated_services <= 1
and not analysis.integrations.external_services
)
return ValidationRecommendations(
risk_level=normalized_risk,
skip_validation=False, # Never skip by inference
minimal_mode=minimal_mode,
test_types_required=test_types,
security_scan_required=security_scan_required,
staging_deployment_required=staging_required,
reasoning="Inferred from complexity analysis (no explicit recommendations found)",
)
def should_skip_validation(self, spec_dir: Path) -> bool:
"""
Quick check if validation can be skipped entirely.
Args:
spec_dir: Path to the spec directory
Returns:
True if validation can be skipped (trivial changes), False otherwise
"""
assessment = self.load_assessment(spec_dir)
if not assessment:
return False # When in doubt, don't skip
return assessment.validation.skip_validation
def should_use_minimal_mode(self, spec_dir: Path) -> bool:
"""
Check if minimal validation mode should be used.
Args:
spec_dir: Path to the spec directory
Returns:
True if minimal mode is recommended, False otherwise
"""
assessment = self.load_assessment(spec_dir)
if not assessment:
return False
return assessment.validation.minimal_mode
def get_required_test_types(self, spec_dir: Path) -> list[str]:
"""
Get list of required test types based on risk.
Args:
spec_dir: Path to the spec directory
Returns:
List of test types (e.g., ["unit", "integration", "e2e"])
"""
assessment = self.load_assessment(spec_dir)
if not assessment:
return ["unit"] # Default to unit tests
return assessment.validation.test_types_required
def requires_security_scan(self, spec_dir: Path) -> bool:
"""
Check if security scanning is required.
Args:
spec_dir: Path to the spec directory
Returns:
True if security scan is required, False otherwise
"""
assessment = self.load_assessment(spec_dir)
if not assessment:
return False
return assessment.validation.security_scan_required
def requires_staging_deployment(self, spec_dir: Path) -> bool:
"""
Check if staging deployment is required.
Args:
spec_dir: Path to the spec directory
Returns:
True if staging deployment is required, False otherwise
"""
assessment = self.load_assessment(spec_dir)
if not assessment:
return False
return assessment.validation.staging_deployment_required
def get_risk_level(self, spec_dir: Path) -> str:
"""
Get the risk level for the task.
Args:
spec_dir: Path to the spec directory
Returns:
Risk level string (trivial, low, medium, high, critical)
"""
assessment = self.load_assessment(spec_dir)
if not assessment:
return "medium" # Default to medium when unknown
return assessment.validation.risk_level
def get_complexity(self, spec_dir: Path) -> str:
"""
Get the complexity level for the task.
Args:
spec_dir: Path to the spec directory
Returns:
Complexity level string (simple, standard, complex)
"""
assessment = self.load_assessment(spec_dir)
if not assessment:
return "standard" # Default to standard when unknown
return assessment.complexity
def get_validation_summary(self, spec_dir: Path) -> dict[str, Any]:
"""
Get a summary of validation requirements.
Args:
spec_dir: Path to the spec directory
Returns:
Dictionary with validation summary
"""
assessment = self.load_assessment(spec_dir)
if not assessment:
return {
"risk_level": "unknown",
"complexity": "unknown",
"skip_validation": False,
"minimal_mode": False,
"test_types": ["unit"],
"security_scan": False,
"staging_deployment": False,
"confidence": 0.0,
}
return {
"risk_level": assessment.validation.risk_level,
"complexity": assessment.complexity,
"skip_validation": assessment.validation.skip_validation,
"minimal_mode": assessment.validation.minimal_mode,
"test_types": assessment.validation.test_types_required,
"security_scan": assessment.validation.security_scan_required,
"staging_deployment": assessment.validation.staging_deployment_required,
"confidence": assessment.confidence,
"reasoning": assessment.validation.reasoning,
}
def clear_cache(self) -> None:
"""Clear the internal cache of loaded assessments."""
self._cache.clear()
# =============================================================================
# CONVENIENCE FUNCTIONS
# =============================================================================
def load_risk_assessment(spec_dir: Path) -> RiskAssessment | None:
"""
Convenience function to load a risk assessment.
Args:
spec_dir: Path to the spec directory
Returns:
RiskAssessment object or None
"""
classifier = RiskClassifier()
return classifier.load_assessment(spec_dir)
def get_validation_requirements(spec_dir: Path) -> dict[str, Any]:
"""
Convenience function to get validation requirements.
Args:
spec_dir: Path to the spec directory
Returns:
Dictionary with validation requirements
"""
classifier = RiskClassifier()
return classifier.get_validation_summary(spec_dir)
# =============================================================================
# CLI
# =============================================================================
def main() -> None:
"""CLI entry point for testing."""
import argparse
parser = argparse.ArgumentParser(description="Load and display risk assessment")
parser.add_argument(
"spec_dir",
type=Path,
help="Path to spec directory with complexity_assessment.json",
)
parser.add_argument("--json", action="store_true", help="Output as JSON")
args = parser.parse_args()
classifier = RiskClassifier()
summary = classifier.get_validation_summary(args.spec_dir)
if args.json:
print(json.dumps(summary, indent=2))
else:
print(f"Risk Level: {summary['risk_level']}")
print(f"Complexity: {summary['complexity']}")
print(f"Skip Validation: {summary['skip_validation']}")
print(f"Minimal Mode: {summary['minimal_mode']}")
print(f"Test Types: {', '.join(summary['test_types'])}")
print(f"Security Scan: {summary['security_scan']}")
print(f"Staging Deployment: {summary['staging_deployment']}")
print(f"Confidence: {summary['confidence']:.2f}")
if summary.get("reasoning"):
print(f"Reasoning: {summary['reasoning']}")
if __name__ == "__main__":
main()
+599
View File
@@ -0,0 +1,599 @@
#!/usr/bin/env python3
"""
Security Scanner Module
=======================
Consolidates security scanning including secrets detection and SAST tools.
This module integrates the existing scan_secrets.py and provides a unified
interface for all security scanning.
The security scanner is used by:
- QA Agent: To verify no secrets are committed
- Validation Strategy: To run security scans for high-risk changes
Usage:
from analysis.security_scanner import SecurityScanner
scanner = SecurityScanner()
results = scanner.scan(project_dir, spec_dir)
if results.has_critical_issues:
print("Security issues found - blocking QA approval")
"""
from __future__ import annotations
import json
import subprocess
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
# Import the existing secrets scanner
try:
from security.scan_secrets import SecretMatch, get_all_tracked_files, scan_files
HAS_SECRETS_SCANNER = True
except ImportError:
HAS_SECRETS_SCANNER = False
SecretMatch = None
# =============================================================================
# DATA CLASSES
# =============================================================================
@dataclass
class SecurityVulnerability:
"""
Represents a security vulnerability found during scanning.
Attributes:
severity: Severity level (critical, high, medium, low, info)
source: Which scanner found this (secrets, bandit, npm_audit, etc.)
title: Short title of the vulnerability
description: Detailed description
file: File where vulnerability was found (if applicable)
line: Line number (if applicable)
cwe: CWE identifier if available
"""
severity: str # critical, high, medium, low, info
source: str # secrets, bandit, npm_audit, semgrep, etc.
title: str
description: str
file: str | None = None
line: int | None = None
cwe: str | None = None
@dataclass
class SecurityScanResult:
"""
Result of a security scan.
Attributes:
secrets: List of detected secrets
vulnerabilities: List of security vulnerabilities
scan_errors: List of errors during scanning
has_critical_issues: Whether any critical issues were found
should_block_qa: Whether these results should block QA approval
"""
secrets: list[dict[str, Any]] = field(default_factory=list)
vulnerabilities: list[SecurityVulnerability] = field(default_factory=list)
scan_errors: list[str] = field(default_factory=list)
has_critical_issues: bool = False
should_block_qa: bool = False
# =============================================================================
# SECURITY SCANNER
# =============================================================================
class SecurityScanner:
"""
Consolidates all security scanning operations.
Integrates:
- scan_secrets.py for secrets detection
- Bandit for Python SAST (if available)
- npm audit for JavaScript vulnerabilities (if applicable)
"""
def __init__(self) -> None:
"""Initialize the security scanner."""
self._bandit_available: bool | None = None
self._npm_available: bool | None = None
def scan(
self,
project_dir: Path,
spec_dir: Path | None = None,
changed_files: list[str] | None = None,
run_secrets: bool = True,
run_sast: bool = True,
run_dependency_audit: bool = True,
) -> SecurityScanResult:
"""
Run all applicable security scans.
Args:
project_dir: Path to the project root
spec_dir: Path to the spec directory (for storing results)
changed_files: Optional list of files to scan (if None, scans all)
run_secrets: Whether to run secrets scanning
run_sast: Whether to run SAST tools
run_dependency_audit: Whether to run dependency audits
Returns:
SecurityScanResult with all findings
"""
project_dir = Path(project_dir)
result = SecurityScanResult()
# Run secrets scan
if run_secrets:
self._run_secrets_scan(project_dir, changed_files, result)
# Run SAST based on project type
if run_sast:
self._run_sast_scans(project_dir, result)
# Run dependency audits
if run_dependency_audit:
self._run_dependency_audits(project_dir, result)
# Determine if should block QA
result.has_critical_issues = (
any(v.severity in ["critical", "high"] for v in result.vulnerabilities)
or len(result.secrets) > 0
)
# Any secrets always block, critical vulnerabilities block
result.should_block_qa = len(result.secrets) > 0 or any(
v.severity == "critical" for v in result.vulnerabilities
)
# Save results if spec_dir provided
if spec_dir:
self._save_results(spec_dir, result)
return result
def _run_secrets_scan(
self,
project_dir: Path,
changed_files: list[str] | None,
result: SecurityScanResult,
) -> None:
"""Run secrets scanning using scan_secrets.py."""
if not HAS_SECRETS_SCANNER:
result.scan_errors.append("scan_secrets module not available")
return
try:
# Get files to scan
if changed_files:
files_to_scan = changed_files
else:
files_to_scan = get_all_tracked_files()
# Run scan
matches = scan_files(files_to_scan, project_dir)
# Convert matches to result format
for match in matches:
result.secrets.append(
{
"file": match.file_path,
"line": match.line_number,
"pattern": match.pattern_name,
"matched_text": self._redact_secret(match.matched_text),
}
)
# Also add as vulnerability
result.vulnerabilities.append(
SecurityVulnerability(
severity="critical",
source="secrets",
title=f"Potential secret: {match.pattern_name}",
description=f"Found potential {match.pattern_name} in file",
file=match.file_path,
line=match.line_number,
)
)
except Exception as e:
result.scan_errors.append(f"Secrets scan error: {str(e)}")
def _run_sast_scans(self, project_dir: Path, result: SecurityScanResult) -> None:
"""Run SAST tools based on project type."""
# Python SAST with Bandit
if self._is_python_project(project_dir):
self._run_bandit(project_dir, result)
# JavaScript/Node.js - npm audit
# (handled in dependency audits for Node projects)
def _run_bandit(self, project_dir: Path, result: SecurityScanResult) -> None:
"""Run Bandit security scanner for Python projects."""
if not self._check_bandit_available():
return
try:
# Find Python source directories
src_dirs = []
for candidate in ["src", "app", project_dir.name, "."]:
candidate_path = project_dir / candidate
if (
candidate_path.exists()
and (candidate_path / "__init__.py").exists()
):
src_dirs.append(str(candidate_path))
if not src_dirs:
# Try to find any Python files
py_files = list(project_dir.glob("**/*.py"))
if not py_files:
return
src_dirs = ["."]
# Run bandit
cmd = [
"bandit",
"-r",
*src_dirs,
"-f",
"json",
"--exit-zero", # Don't fail on findings
]
proc = subprocess.run(
cmd,
cwd=project_dir,
capture_output=True,
text=True,
timeout=120,
)
if proc.stdout:
try:
bandit_output = json.loads(proc.stdout)
for finding in bandit_output.get("results", []):
severity = finding.get("issue_severity", "MEDIUM").lower()
if severity == "high":
severity = "high"
elif severity == "medium":
severity = "medium"
else:
severity = "low"
result.vulnerabilities.append(
SecurityVulnerability(
severity=severity,
source="bandit",
title=finding.get("issue_text", "Unknown issue"),
description=finding.get("issue_text", ""),
file=finding.get("filename"),
line=finding.get("line_number"),
cwe=finding.get("issue_cwe", {}).get("id"),
)
)
except json.JSONDecodeError:
result.scan_errors.append("Failed to parse Bandit output")
except subprocess.TimeoutExpired:
result.scan_errors.append("Bandit scan timed out")
except FileNotFoundError:
result.scan_errors.append("Bandit not found")
except Exception as e:
result.scan_errors.append(f"Bandit error: {str(e)}")
def _run_dependency_audits(
self, project_dir: Path, result: SecurityScanResult
) -> None:
"""Run dependency vulnerability audits."""
# npm audit for JavaScript projects
if (project_dir / "package.json").exists():
self._run_npm_audit(project_dir, result)
# pip-audit for Python projects (if available)
if self._is_python_project(project_dir):
self._run_pip_audit(project_dir, result)
def _run_npm_audit(self, project_dir: Path, result: SecurityScanResult) -> None:
"""Run npm audit for JavaScript projects."""
try:
cmd = ["npm", "audit", "--json"]
proc = subprocess.run(
cmd,
cwd=project_dir,
capture_output=True,
text=True,
timeout=120,
)
if proc.stdout:
try:
audit_output = json.loads(proc.stdout)
# npm audit v2+ format
vulnerabilities = audit_output.get("vulnerabilities", {})
for pkg_name, vuln_info in vulnerabilities.items():
severity = vuln_info.get("severity", "moderate")
if severity == "critical":
severity = "critical"
elif severity == "high":
severity = "high"
elif severity == "moderate":
severity = "medium"
else:
severity = "low"
result.vulnerabilities.append(
SecurityVulnerability(
severity=severity,
source="npm_audit",
title=f"Vulnerable dependency: {pkg_name}",
description=vuln_info.get("via", [{}])[0].get(
"title", ""
)
if isinstance(vuln_info.get("via"), list)
and vuln_info.get("via")
else str(vuln_info.get("via", "")),
file="package.json",
)
)
except json.JSONDecodeError:
pass # npm audit may return invalid JSON on no findings
except subprocess.TimeoutExpired:
result.scan_errors.append("npm audit timed out")
except FileNotFoundError:
pass # npm not available
except Exception as e:
result.scan_errors.append(f"npm audit error: {str(e)}")
def _run_pip_audit(self, project_dir: Path, result: SecurityScanResult) -> None:
"""Run pip-audit for Python projects (if available)."""
try:
cmd = ["pip-audit", "--format", "json"]
proc = subprocess.run(
cmd,
cwd=project_dir,
capture_output=True,
text=True,
timeout=120,
)
if proc.stdout:
try:
audit_output = json.loads(proc.stdout)
for vuln in audit_output:
severity = "high" if vuln.get("fix_versions") else "medium"
result.vulnerabilities.append(
SecurityVulnerability(
severity=severity,
source="pip_audit",
title=f"Vulnerable package: {vuln.get('name')}",
description=vuln.get("description", ""),
cwe=vuln.get("aliases", [""])[0]
if vuln.get("aliases")
else None,
)
)
except json.JSONDecodeError:
pass
except FileNotFoundError:
pass # pip-audit not available
except subprocess.TimeoutExpired:
pass
except Exception:
pass
def _is_python_project(self, project_dir: Path) -> bool:
"""Check if this is a Python project."""
indicators = [
project_dir / "pyproject.toml",
project_dir / "requirements.txt",
project_dir / "setup.py",
project_dir / "setup.cfg",
]
return any(p.exists() for p in indicators)
def _check_bandit_available(self) -> bool:
"""Check if Bandit is available."""
if self._bandit_available is None:
try:
subprocess.run(
["bandit", "--version"],
capture_output=True,
timeout=5,
)
self._bandit_available = True
except (FileNotFoundError, subprocess.TimeoutExpired):
self._bandit_available = False
return self._bandit_available
def _redact_secret(self, text: str) -> str:
"""Redact a secret for safe logging."""
if len(text) <= 8:
return "*" * len(text)
return text[:4] + "*" * (len(text) - 8) + text[-4:]
def _save_results(self, spec_dir: Path, result: SecurityScanResult) -> None:
"""Save scan results to spec directory."""
spec_dir = Path(spec_dir)
spec_dir.mkdir(parents=True, exist_ok=True)
output_file = spec_dir / "security_scan_results.json"
output_data = self.to_dict(result)
with open(output_file, "w", encoding="utf-8") as f:
json.dump(output_data, f, indent=2)
def to_dict(self, result: SecurityScanResult) -> dict[str, Any]:
"""Convert result to dictionary for JSON serialization."""
return {
"secrets": result.secrets,
"vulnerabilities": [
{
"severity": v.severity,
"source": v.source,
"title": v.title,
"description": v.description,
"file": v.file,
"line": v.line,
"cwe": v.cwe,
}
for v in result.vulnerabilities
],
"scan_errors": result.scan_errors,
"has_critical_issues": result.has_critical_issues,
"should_block_qa": result.should_block_qa,
"summary": {
"total_secrets": len(result.secrets),
"total_vulnerabilities": len(result.vulnerabilities),
"critical_count": sum(
1 for v in result.vulnerabilities if v.severity == "critical"
),
"high_count": sum(
1 for v in result.vulnerabilities if v.severity == "high"
),
"medium_count": sum(
1 for v in result.vulnerabilities if v.severity == "medium"
),
"low_count": sum(
1 for v in result.vulnerabilities if v.severity == "low"
),
},
}
# =============================================================================
# CONVENIENCE FUNCTIONS
# =============================================================================
def scan_for_security_issues(
project_dir: Path,
spec_dir: Path | None = None,
changed_files: list[str] | None = None,
) -> SecurityScanResult:
"""
Convenience function to run security scan.
Args:
project_dir: Path to project root
spec_dir: Optional spec directory to save results
changed_files: Optional list of files to scan
Returns:
SecurityScanResult with all findings
"""
scanner = SecurityScanner()
return scanner.scan(project_dir, spec_dir, changed_files)
def has_security_issues(project_dir: Path) -> bool:
"""
Quick check if project has security issues.
Args:
project_dir: Path to project root
Returns:
True if any critical/high issues found
"""
scanner = SecurityScanner()
result = scanner.scan(project_dir, run_sast=False, run_dependency_audit=False)
return result.has_critical_issues
def scan_secrets_only(
project_dir: Path,
changed_files: list[str] | None = None,
) -> list[dict[str, Any]]:
"""
Scan only for secrets (quick scan).
Args:
project_dir: Path to project root
changed_files: Optional list of files to scan
Returns:
List of detected secrets
"""
scanner = SecurityScanner()
result = scanner.scan(
project_dir,
changed_files=changed_files,
run_sast=False,
run_dependency_audit=False,
)
return result.secrets
# =============================================================================
# CLI
# =============================================================================
def main() -> None:
"""CLI entry point for testing."""
import argparse
parser = argparse.ArgumentParser(description="Run security scans")
parser.add_argument("project_dir", type=Path, help="Path to project root")
parser.add_argument("--spec-dir", type=Path, help="Path to spec directory")
parser.add_argument(
"--secrets-only", action="store_true", help="Only scan for secrets"
)
parser.add_argument("--json", action="store_true", help="Output as JSON")
args = parser.parse_args()
scanner = SecurityScanner()
result = scanner.scan(
args.project_dir,
spec_dir=args.spec_dir,
run_sast=not args.secrets_only,
run_dependency_audit=not args.secrets_only,
)
if args.json:
print(json.dumps(scanner.to_dict(result), indent=2))
else:
print(f"Secrets Found: {len(result.secrets)}")
print(f"Vulnerabilities: {len(result.vulnerabilities)}")
print(f"Has Critical Issues: {result.has_critical_issues}")
print(f"Should Block QA: {result.should_block_qa}")
if result.secrets:
print("\nSecrets Detected:")
for secret in result.secrets:
print(f" - {secret['pattern']} in {secret['file']}:{secret['line']}")
if result.vulnerabilities:
print(f"\nVulnerabilities ({len(result.vulnerabilities)}):")
for v in result.vulnerabilities:
print(f" [{v.severity.upper()}] {v.title}")
if v.file:
print(f" File: {v.file}:{v.line or ''}")
if result.scan_errors:
print(f"\nScan Errors ({len(result.scan_errors)}):")
for error in result.scan_errors:
print(f" - {error}")
if __name__ == "__main__":
main()
+690
View File
@@ -0,0 +1,690 @@
#!/usr/bin/env python3
"""
Test Discovery Module
=====================
Detects test frameworks, test commands, and test directories in a project.
This module analyzes project configuration files to discover how tests
should be run.
The test discovery results are used by:
- QA Agent: To determine what test commands to run
- Test Creator: To know what framework to use when creating tests
- Planner: To include correct test commands in verification strategy
Usage:
from test_discovery import TestDiscovery
discovery = TestDiscovery()
result = discovery.discover(project_dir)
print(f"Test frameworks: {result['frameworks']}")
print(f"Test command: {result['test_command']}")
"""
from __future__ import annotations
import json
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
# =============================================================================
# DATA CLASSES
# =============================================================================
@dataclass
class TestFramework:
"""
Represents a detected test framework.
Attributes:
name: Name of the framework (e.g., "pytest", "jest", "vitest")
type: Type of testing (unit, integration, e2e, all)
command: Command to run tests
config_file: Configuration file if found
version: Version if detected
coverage_command: Command for coverage if available
"""
__test__ = False # Prevent pytest from collecting this as a test class
name: str
type: str # unit, integration, e2e, all
command: str
config_file: str | None = None
version: str | None = None
coverage_command: str | None = None
@dataclass
class TestDiscoveryResult:
"""
Result of test framework discovery.
Attributes:
frameworks: List of detected test frameworks
test_command: Primary test command to run
test_directories: Discovered test directories
package_manager: Detected package manager
has_tests: Whether any test files were found
coverage_command: Command for coverage if available
"""
__test__ = False # Prevent pytest from collecting this as a test class
frameworks: list[TestFramework] = field(default_factory=list)
test_command: str = ""
test_directories: list[str] = field(default_factory=list)
package_manager: str = ""
has_tests: bool = False
coverage_command: str | None = None
# =============================================================================
# FRAMEWORK DETECTORS
# =============================================================================
# Pattern-based framework detection
FRAMEWORK_PATTERNS = {
# JavaScript/TypeScript
"jest": {
"config_files": [
"jest.config.js",
"jest.config.ts",
"jest.config.mjs",
"jest.config.cjs",
],
"package_key": "jest",
"type": "unit",
"command": "npx jest",
"coverage_command": "npx jest --coverage",
},
"vitest": {
"config_files": ["vitest.config.js", "vitest.config.ts", "vitest.config.mjs"],
"package_key": "vitest",
"type": "unit",
"command": "npx vitest run",
"coverage_command": "npx vitest run --coverage",
},
"mocha": {
"config_files": [
".mocharc.js",
".mocharc.json",
".mocharc.yaml",
".mocharc.yml",
],
"package_key": "mocha",
"type": "unit",
"command": "npx mocha",
"coverage_command": "npx nyc mocha",
},
"playwright": {
"config_files": ["playwright.config.js", "playwright.config.ts"],
"package_key": "@playwright/test",
"type": "e2e",
"command": "npx playwright test",
"coverage_command": None,
},
"cypress": {
"config_files": ["cypress.config.js", "cypress.config.ts", "cypress.json"],
"package_key": "cypress",
"type": "e2e",
"command": "npx cypress run",
"coverage_command": None,
},
# Python
"pytest": {
"config_files": ["pytest.ini", "pyproject.toml", "setup.cfg", "conftest.py"],
"pyproject_key": "pytest",
"requirements_key": "pytest",
"type": "all",
"command": "pytest",
"coverage_command": "pytest --cov",
},
"unittest": {
"config_files": [],
"type": "unit",
"command": "python -m unittest discover",
"coverage_command": "coverage run -m unittest discover",
},
# Rust
"cargo_test": {
"config_files": ["Cargo.toml"],
"type": "all",
"command": "cargo test",
"coverage_command": "cargo tarpaulin",
},
# Go
"go_test": {
"config_files": ["go.mod"],
"type": "all",
"command": "go test ./...",
"coverage_command": "go test -cover ./...",
},
# Ruby
"rspec": {
"config_files": [".rspec", "spec/spec_helper.rb"],
"gemfile_key": "rspec",
"type": "all",
"command": "bundle exec rspec",
"coverage_command": "bundle exec rspec --format documentation",
},
"minitest": {
"config_files": [],
"gemfile_key": "minitest",
"type": "unit",
"command": "bundle exec rake test",
"coverage_command": None,
},
}
# =============================================================================
# TEST DISCOVERY
# =============================================================================
class TestDiscovery:
"""
Discovers test frameworks and configurations in a project.
Analyzes:
- Package files (package.json, pyproject.toml, Cargo.toml, etc.)
- Configuration files (jest.config.js, pytest.ini, etc.)
- Directory structure (tests/, spec/, __tests__/)
"""
__test__ = False # Prevent pytest from collecting this as a test class
def __init__(self) -> None:
"""Initialize the test discovery."""
self._cache: dict[str, TestDiscoveryResult] = {}
def discover(self, project_dir: Path) -> TestDiscoveryResult:
"""
Discover test frameworks and configuration in the project.
Args:
project_dir: Path to the project root
Returns:
TestDiscoveryResult with detected frameworks and commands
"""
project_dir = Path(project_dir)
cache_key = str(project_dir.resolve())
if cache_key in self._cache:
return self._cache[cache_key]
result = TestDiscoveryResult()
# Detect package manager
result.package_manager = self._detect_package_manager(project_dir)
# Discover frameworks based on project type
if (project_dir / "package.json").exists():
self._discover_js_frameworks(project_dir, result)
# Check for Python project indicators
python_indicators = [
project_dir / "pyproject.toml",
project_dir / "requirements.txt",
project_dir / "setup.py",
project_dir / "pytest.ini",
project_dir / "conftest.py",
project_dir / "tests" / "conftest.py",
]
if any(p.exists() for p in python_indicators):
self._discover_python_frameworks(project_dir, result)
if (project_dir / "Cargo.toml").exists():
self._discover_rust_frameworks(project_dir, result)
if (project_dir / "go.mod").exists():
self._discover_go_frameworks(project_dir, result)
if (project_dir / "Gemfile").exists():
self._discover_ruby_frameworks(project_dir, result)
# Find test directories
result.test_directories = self._find_test_directories(project_dir)
# Check if tests exist
result.has_tests = self._has_test_files(project_dir, result.test_directories)
# Set primary test command
if result.frameworks:
result.test_command = result.frameworks[0].command
# Set coverage command from first framework that has one
if not result.coverage_command:
for framework in result.frameworks:
if framework.coverage_command:
result.coverage_command = framework.coverage_command
break
self._cache[cache_key] = result
return result
def _detect_package_manager(self, project_dir: Path) -> str:
"""Detect the package manager used by the project."""
if (project_dir / "pnpm-lock.yaml").exists():
return "pnpm"
if (project_dir / "yarn.lock").exists():
return "yarn"
if (project_dir / "package-lock.json").exists():
return "npm"
if (project_dir / "bun.lockb").exists() or (project_dir / "bun.lock").exists():
return "bun"
if (project_dir / "uv.lock").exists():
return "uv"
if (project_dir / "poetry.lock").exists():
return "poetry"
if (project_dir / "Pipfile.lock").exists():
return "pipenv"
if (project_dir / "Cargo.lock").exists():
return "cargo"
if (project_dir / "go.sum").exists():
return "go"
if (project_dir / "Gemfile.lock").exists():
return "bundler"
return ""
def _discover_js_frameworks(
self, project_dir: Path, result: TestDiscoveryResult
) -> None:
"""Discover JavaScript/TypeScript test frameworks."""
package_json = project_dir / "package.json"
if not package_json.exists():
return
try:
with open(package_json, encoding="utf-8") as f:
pkg = json.load(f)
except (OSError, json.JSONDecodeError, UnicodeDecodeError):
return
deps = pkg.get("dependencies", {})
dev_deps = pkg.get("devDependencies", {})
all_deps = {**deps, **dev_deps}
scripts = pkg.get("scripts", {})
# Check for test frameworks in dependencies
for name, pattern in FRAMEWORK_PATTERNS.items():
if "package_key" not in pattern:
continue
if pattern["package_key"] in all_deps:
# Check for config file
config_file = None
for cf in pattern.get("config_files", []):
if (project_dir / cf).exists():
config_file = cf
break
# Get version
version = all_deps.get(pattern["package_key"], "")
if version.startswith("^") or version.startswith("~"):
version = version[1:]
# Determine command - prefer npm scripts if available
command = pattern["command"]
if "test" in scripts and pattern["package_key"] in scripts.get(
"test", ""
):
command = f"{result.package_manager or 'npm'} test"
result.frameworks.append(
TestFramework(
name=name,
type=pattern["type"],
command=command,
config_file=config_file,
version=version,
coverage_command=pattern.get("coverage_command"),
)
)
# Check npm scripts for test commands
if not result.frameworks and "test" in scripts:
test_script = scripts["test"]
if (
test_script
and test_script != 'echo "Error: no test specified" && exit 1'
):
# Try to infer framework from script
framework_name = "npm_test"
framework_type = "unit"
if "jest" in test_script:
framework_name = "jest"
elif "vitest" in test_script:
framework_name = "vitest"
elif "mocha" in test_script:
framework_name = "mocha"
elif "playwright" in test_script:
framework_name = "playwright"
framework_type = "e2e"
elif "cypress" in test_script:
framework_name = "cypress"
framework_type = "e2e"
result.frameworks.append(
TestFramework(
name=framework_name,
type=framework_type,
command=f"{result.package_manager or 'npm'} test",
config_file=None,
)
)
def _discover_python_frameworks(
self, project_dir: Path, result: TestDiscoveryResult
) -> None:
"""Discover Python test frameworks."""
# Check for pytest.ini first (explicit pytest config)
if (project_dir / "pytest.ini").exists():
if not any(f.name == "pytest" for f in result.frameworks):
result.frameworks.append(
TestFramework(
name="pytest",
type="all",
command="pytest",
config_file="pytest.ini",
)
)
# Check pyproject.toml
pyproject = project_dir / "pyproject.toml"
if pyproject.exists():
content = pyproject.read_text(encoding="utf-8")
# Check for pytest
if "pytest" in content:
if not any(f.name == "pytest" for f in result.frameworks):
config_file = (
"pyproject.toml" if "[tool.pytest" in content else None
)
result.frameworks.append(
TestFramework(
name="pytest",
type="all",
command="pytest",
config_file=config_file,
)
)
# Check requirements.txt
requirements = project_dir / "requirements.txt"
if requirements.exists():
content = requirements.read_text(encoding="utf-8").lower()
if "pytest" in content and not any(
f.name == "pytest" for f in result.frameworks
):
result.frameworks.append(
TestFramework(
name="pytest",
type="all",
command="pytest",
config_file=None,
)
)
# Check for conftest.py (pytest marker)
conftest_root = project_dir / "conftest.py"
conftest_tests = project_dir / "tests" / "conftest.py"
if conftest_root.exists() or conftest_tests.exists():
if not any(f.name == "pytest" for f in result.frameworks):
result.frameworks.append(
TestFramework(
name="pytest",
type="all",
command="pytest",
config_file="conftest.py",
)
)
# Fall back to unittest if test files exist but no framework detected
if not result.frameworks:
test_dirs = self._find_test_directories(project_dir)
if test_dirs:
result.frameworks.append(
TestFramework(
name="unittest",
type="unit",
command="python -m unittest discover",
config_file=None,
)
)
def _discover_rust_frameworks(
self, project_dir: Path, result: TestDiscoveryResult
) -> None:
"""Discover Rust test frameworks."""
cargo_toml = project_dir / "Cargo.toml"
if cargo_toml.exists():
result.frameworks.append(
TestFramework(
name="cargo_test",
type="all",
command="cargo test",
config_file="Cargo.toml",
)
)
def _discover_go_frameworks(
self, project_dir: Path, result: TestDiscoveryResult
) -> None:
"""Discover Go test frameworks."""
go_mod = project_dir / "go.mod"
if go_mod.exists():
result.frameworks.append(
TestFramework(
name="go_test",
type="all",
command="go test ./...",
config_file="go.mod",
)
)
def _discover_ruby_frameworks(
self, project_dir: Path, result: TestDiscoveryResult
) -> None:
"""Discover Ruby test frameworks."""
gemfile = project_dir / "Gemfile"
if not gemfile.exists():
return
content = gemfile.read_text(encoding="utf-8").lower()
if "rspec" in content or (project_dir / ".rspec").exists():
result.frameworks.append(
TestFramework(
name="rspec",
type="all",
command="bundle exec rspec",
config_file=".rspec" if (project_dir / ".rspec").exists() else None,
)
)
elif "minitest" in content:
result.frameworks.append(
TestFramework(
name="minitest",
type="unit",
command="bundle exec rake test",
config_file=None,
)
)
def _find_test_directories(self, project_dir: Path) -> list[str]:
"""Find test directories in the project."""
test_dir_patterns = [
"tests",
"test",
"spec",
"__tests__",
"specs",
"test_*",
]
found_dirs = []
for pattern in test_dir_patterns:
if pattern.endswith("*"):
# Glob pattern
for d in project_dir.glob(pattern):
if d.is_dir():
found_dirs.append(str(d.relative_to(project_dir)))
else:
# Exact name
test_dir = project_dir / pattern
if test_dir.is_dir():
found_dirs.append(pattern)
return found_dirs
def _has_test_files(self, project_dir: Path, test_directories: list[str]) -> bool:
"""Check if any test files exist."""
test_file_patterns = [
"**/test_*.py",
"**/*_test.py",
"**/*.test.js",
"**/*.test.ts",
"**/*.test.tsx",
"**/*.spec.js",
"**/*.spec.ts",
"**/*.spec.tsx",
"**/test_*.go",
"**/*_test.go",
"**/*_test.rs",
"**/spec/**/*_spec.rb",
]
# Check in test directories
for test_dir in test_directories:
test_path = project_dir / test_dir
if test_path.exists():
for pattern in test_file_patterns:
if list(test_path.glob(pattern.replace("**/", ""))):
return True
# Check project-wide
for pattern in test_file_patterns:
if list(project_dir.glob(pattern)):
return True
return False
def to_dict(self, result: TestDiscoveryResult) -> dict[str, Any]:
"""Convert result to dictionary for JSON serialization."""
return {
"frameworks": [
{
"name": f.name,
"type": f.type,
"command": f.command,
"config_file": f.config_file,
"version": f.version,
"coverage_command": f.coverage_command,
}
for f in result.frameworks
],
"test_command": result.test_command,
"test_directories": result.test_directories,
"package_manager": result.package_manager,
"has_tests": result.has_tests,
"coverage_command": result.coverage_command,
}
def clear_cache(self) -> None:
"""Clear the internal cache."""
self._cache.clear()
# =============================================================================
# CONVENIENCE FUNCTIONS
# =============================================================================
def discover_tests(project_dir: Path) -> TestDiscoveryResult:
"""
Convenience function to discover tests in a project.
Args:
project_dir: Path to project root
Returns:
TestDiscoveryResult with detected frameworks
"""
discovery = TestDiscovery()
return discovery.discover(project_dir)
def get_test_command(project_dir: Path) -> str:
"""
Get the primary test command for a project.
Args:
project_dir: Path to project root
Returns:
Test command string, or empty string if not found
"""
discovery = TestDiscovery()
result = discovery.discover(project_dir)
return result.test_command
def get_test_frameworks(project_dir: Path) -> list[str]:
"""
Get list of test framework names in a project.
Args:
project_dir: Path to project root
Returns:
List of framework names
"""
discovery = TestDiscovery()
result = discovery.discover(project_dir)
return [f.name for f in result.frameworks]
# =============================================================================
# CLI
# =============================================================================
def main() -> None:
"""CLI entry point for testing."""
import argparse
parser = argparse.ArgumentParser(description="Discover test frameworks")
parser.add_argument("project_dir", type=Path, help="Path to project root")
parser.add_argument("--json", action="store_true", help="Output as JSON")
args = parser.parse_args()
discovery = TestDiscovery()
result = discovery.discover(args.project_dir)
if args.json:
print(json.dumps(discovery.to_dict(result), indent=2))
else:
print(f"Package Manager: {result.package_manager or 'unknown'}")
print(f"Has Tests: {result.has_tests}")
print(f"Test Command: {result.test_command or 'none'}")
print(f"Test Directories: {', '.join(result.test_directories) or 'none'}")
print(f"Coverage Command: {result.coverage_command or 'none'}")
print(f"\nFrameworks ({len(result.frameworks)}):")
for f in result.frameworks:
print(f" - {f.name} ({f.type})")
print(f" Command: {f.command}")
if f.config_file:
print(f" Config: {f.config_file}")
if f.version:
print(f" Version: {f.version}")
if __name__ == "__main__":
main()
+26
View File
@@ -0,0 +1,26 @@
#!/usr/bin/env python3
"""
Analyzer facade module.
Provides backward compatibility for scripts that import from analyzer.py at the root.
Actual implementation is in analysis/analyzer.py.
"""
from analysis.analyzer import (
ProjectAnalyzer,
ServiceAnalyzer,
analyze_project,
analyze_service,
main,
)
__all__ = [
"ServiceAnalyzer",
"ProjectAnalyzer",
"analyze_project",
"analyze_service",
"main",
]
if __name__ == "__main__":
main()
+36
View File
@@ -0,0 +1,36 @@
"""
Auto Claude tools module facade.
Provides MCP tools for agent operations.
Re-exports from agents.tools_pkg for clean imports.
"""
from agents.tools_pkg.models import ( # noqa: F401
ELECTRON_TOOLS,
TOOL_GET_BUILD_PROGRESS,
TOOL_GET_SESSION_CONTEXT,
TOOL_RECORD_DISCOVERY,
TOOL_RECORD_GOTCHA,
TOOL_UPDATE_QA_STATUS,
TOOL_UPDATE_SUBTASK_STATUS,
is_electron_mcp_enabled,
)
from agents.tools_pkg.permissions import get_allowed_tools # noqa: F401
from agents.tools_pkg.registry import ( # noqa: F401
create_auto_claude_mcp_server,
is_tools_available,
)
__all__ = [
"create_auto_claude_mcp_server",
"get_allowed_tools",
"is_tools_available",
"TOOL_UPDATE_SUBTASK_STATUS",
"TOOL_GET_BUILD_PROGRESS",
"TOOL_RECORD_DISCOVERY",
"TOOL_RECORD_GOTCHA",
"TOOL_GET_SESSION_CONTEXT",
"TOOL_UPDATE_QA_STATUS",
"ELECTRON_TOOLS",
"is_electron_mcp_enabled",
]
+21
View File
@@ -0,0 +1,21 @@
"""Backward compatibility shim - import from analysis.ci_discovery instead."""
from analysis.ci_discovery import (
HAS_YAML,
CIConfig,
CIDiscovery,
CIWorkflow,
discover_ci,
get_ci_system,
get_ci_test_commands,
)
__all__ = [
"CIConfig",
"CIWorkflow",
"CIDiscovery",
"discover_ci",
"get_ci_test_commands",
"get_ci_system",
"HAS_YAML",
]
+18
View File
@@ -0,0 +1,18 @@
"""
Auto Claude CLI Package
=======================
Command-line interface for the Auto Claude autonomous coding framework.
This package provides a modular CLI structure:
- main.py: Argument parsing and command routing
- spec_commands.py: Spec listing and management
- build_commands.py: Build execution and follow-up tasks
- workspace_commands.py: Workspace management (merge, review, discard)
- qa_commands.py: QA validation commands
- utils.py: Shared utilities and configuration
"""
from .main import main
__all__ = ["main"]
+266
View File
@@ -0,0 +1,266 @@
"""
Batch Task Management Commands
==============================
Commands for creating and managing multiple tasks from batch files.
"""
import json
import shutil
import subprocess
from pathlib import Path
from ui import highlight, print_status
def handle_batch_create_command(batch_file: str, project_dir: str) -> bool:
"""
Create multiple tasks from a batch JSON file.
Args:
batch_file: Path to JSON file with task definitions
project_dir: Project directory
Returns:
True if successful
"""
batch_path = Path(batch_file)
if not batch_path.exists():
print_status(f"Batch file not found: {batch_file}", "error")
return False
try:
with open(batch_path, encoding="utf-8") as f:
batch_data = json.load(f)
except json.JSONDecodeError as e:
print_status(f"Invalid JSON in batch file: {e}", "error")
return False
tasks = batch_data.get("tasks", [])
if not tasks:
print_status("No tasks found in batch file", "warning")
return False
print_status(f"Creating {len(tasks)} tasks from batch file", "info")
print()
specs_dir = Path(project_dir) / ".auto-claude" / "specs"
specs_dir.mkdir(parents=True, exist_ok=True)
# Find next spec ID
existing_specs = [d.name for d in specs_dir.iterdir() if d.is_dir()]
next_id = (
max([int(s.split("-")[0]) for s in existing_specs if s[0].isdigit()] or [0]) + 1
)
created_specs = []
for idx, task in enumerate(tasks, 1):
spec_id = f"{next_id:03d}"
task_title = task.get("title", f"Task {idx}")
task_slug = task_title.lower().replace(" ", "-")[:50]
spec_name = f"{spec_id}-{task_slug}"
spec_dir = specs_dir / spec_name
spec_dir.mkdir(exist_ok=True)
# Create requirements.json
requirements = {
"task_description": task.get("description", task_title),
"description": task.get("description", task_title),
"workflow_type": task.get("workflow_type", "feature"),
"services_involved": task.get("services", ["frontend"]),
"priority": task.get("priority", 5),
"complexity_inferred": task.get("complexity", "standard"),
"inferred_from": {},
"created_at": Path(spec_dir).stat().st_mtime,
"estimate": {
"estimated_hours": task.get("estimated_hours", 4.0),
"estimated_days": task.get("estimated_days", 0.5),
},
}
req_file = spec_dir / "requirements.json"
with open(req_file, "w", encoding="utf-8") as f:
json.dump(requirements, f, indent=2, default=str)
created_specs.append(
{
"id": spec_id,
"name": spec_name,
"title": task_title,
"status": "pending_spec_creation",
}
)
print_status(
f"[{idx}/{len(tasks)}] Created {spec_id} - {task_title}", "success"
)
next_id += 1
print()
print_status(f"Created {len(created_specs)} spec(s) successfully", "success")
print()
# Show summary
print(highlight("Next steps:"))
print(" 1. Generate specs: spec_runner.py --continue <spec_id>")
print(" 2. Approve specs and build them")
print(" 3. Run: python run.py --spec <id> to execute")
return True
def handle_batch_status_command(project_dir: str) -> bool:
"""
Show status of all specs in project.
Args:
project_dir: Project directory
Returns:
True if successful
"""
specs_dir = Path(project_dir) / ".auto-claude" / "specs"
if not specs_dir.exists():
print_status("No specs found in project", "warning")
return True
specs = sorted([d for d in specs_dir.iterdir() if d.is_dir()])
if not specs:
print_status("No specs found", "warning")
return True
print_status(f"Found {len(specs)} spec(s)", "info")
print()
for spec_dir in specs:
spec_name = spec_dir.name
req_file = spec_dir / "requirements.json"
status = "unknown"
title = spec_name
if req_file.exists():
try:
with open(req_file, encoding="utf-8") as f:
req = json.load(f)
title = req.get("task_description", title)
except json.JSONDecodeError:
pass
# Determine status
if (spec_dir / "spec.md").exists():
status = "spec_created"
elif (spec_dir / "implementation_plan.json").exists():
status = "building"
elif (spec_dir / "qa_report.md").exists():
status = "qa_approved"
else:
status = "pending_spec"
status_icon = {
"pending_spec": "",
"spec_created": "📋",
"building": "⚙️",
"qa_approved": "",
"unknown": "",
}.get(status, "")
print(f"{status_icon} {spec_name:<40} {title}")
return True
def handle_batch_cleanup_command(project_dir: str, dry_run: bool = True) -> bool:
"""
Clean up completed specs and worktrees.
Args:
project_dir: Project directory
dry_run: If True, show what would be deleted
Returns:
True if successful
"""
specs_dir = Path(project_dir) / ".auto-claude" / "specs"
worktrees_dir = Path(project_dir) / ".auto-claude" / "worktrees" / "tasks"
if not specs_dir.exists():
print_status("No specs directory found", "info")
return True
# Find completed specs
completed = []
for spec_dir in specs_dir.iterdir():
if spec_dir.is_dir() and (spec_dir / "qa_report.md").exists():
completed.append(spec_dir.name)
if not completed:
print_status("No completed specs to clean up", "info")
return True
print_status(f"Found {len(completed)} completed spec(s)", "info")
if dry_run:
print()
print("Would remove:")
for spec_name in completed:
print(f" - {spec_name}")
wt_path = worktrees_dir / spec_name
if wt_path.exists():
print(f" └─ .auto-claude/worktrees/tasks/{spec_name}/")
print()
print("Run with --no-dry-run to actually delete")
else:
# Actually delete specs and worktrees
deleted_count = 0
for spec_name in completed:
spec_path = specs_dir / spec_name
wt_path = worktrees_dir / spec_name
# Remove worktree first (if exists)
if wt_path.exists():
try:
result = subprocess.run(
["git", "worktree", "remove", "--force", str(wt_path)],
cwd=project_dir,
capture_output=True,
text=True,
timeout=30,
)
if result.returncode == 0:
print_status(f"Removed worktree: {spec_name}", "success")
else:
# Fallback: remove directory manually if git fails
shutil.rmtree(wt_path, ignore_errors=True)
print_status(
f"Removed worktree directory: {spec_name}", "success"
)
except subprocess.TimeoutExpired:
# Timeout: fall back to manual removal
shutil.rmtree(wt_path, ignore_errors=True)
print_status(
f"Worktree removal timed out, removed directory: {spec_name}",
"warning",
)
except Exception as e:
print_status(
f"Failed to remove worktree {spec_name}: {e}", "warning"
)
# Remove spec directory
if spec_path.exists():
try:
shutil.rmtree(spec_path)
print_status(f"Removed spec: {spec_name}", "success")
deleted_count += 1
except Exception as e:
print_status(f"Failed to remove spec {spec_name}: {e}", "error")
print()
print_status(f"Cleaned up {deleted_count} spec(s)", "info")
return True

Some files were not shown because too many files have changed in this diff Show More