Files
Aperant/tests/test_ci_discovery.py
T
StillKnotKnown ed93df698b test: improve backend agent test coverage to 94% (#1779)
* fix: add mock reset fixtures and resolve async iterator mock issues

- Add pytest_runtest_setup and pytest_runtest_teardown hooks to reset
  shared module-level mocks between tests
- Add module-specific mock reset fixtures for test_qa_fixer and
  test_qa_reviewer to prevent test interference
- Fix async iterator mock for receive_response to properly return
  an AsyncIteratorMock instance
- Update test_qa_fixer.py and test_qa_reviewer.py with proper mock
  setup for isolated test execution

* docs(agents): add CLAUDE.md documentation for agents module

Documents the agents module architecture including:
- Module components (coder, planner, session, memory_manager, base)
- Single-agent architecture without external parallelism
- Subagent architecture clarification

* Revert "docs(agents): add CLAUDE.md documentation for agents module"

This reverts commit bf1ddd7da08f2f34352d11a5d823da981f1a98bb.

* chore: update gitignore to allow agents/tests/

* fix(tests): resolve mock isolation and path permission issues

- Fix test_tool_concurrency_error_detection by patching where functions
  are used (qa.fixer) instead of where they're defined
- Add Path.exists/is_dir/glob mocks to avoid permission errors on
  nonexistent directories in test_validation_strategy.py
- Add helper function clean_project_index_files() to reduce code
  duplication in prereqs_validator tests
- Add comprehensive tests for spec validation validators
  (context, prereqs, spec_document)
- Fix similar mock/path issues in test_qa_reviewer.py,
  test_service_orchestrator.py, test_ci_discovery.py,
  test_prompt_generator.py, test_security_scanner.py

All 2103 tests now pass.

* fix(tests): remove unused imports and fix double assignment

- Remove unused 'patch' import from validator test files
- Remove unused 'pytest' import where not needed
- Fix double assignment typo in test_error_message_includes_filename

* fix(tests): move agents tests to tests/agents/ directory

- Move test_agent_architecture.py, test_agent_configs.py, and
  test_agent_flow.py from apps/backend/agents/tests/ to tests/agents/
- Fix path resolution to work from new location
- Remove gitignore exception for agents/tests/ (no longer needed)

This resolves the issue where tests were not included in the PR
because they were in an untracked location.

* fix(tests): simplify conftest.py mock management

- Remove redundant pytest_runtest_teardown and pytest_runtest_call hooks
  (autouse fixtures in test files already handle mock reset)
- Add prompts_pkg.project_context to potentially mocked modules list
- Remove prompts_pkg from test_qa_fixer entry (not used there)

This reduces maintenance burden by having mock reset in one place.

* refactor(tests): consolidate duplicate mock setup into shared helper

- Create tests/qa_test_helpers.py with shared mock infrastructure:
  - AsyncIteratorMock and ReceiveResponseMock classes
  - setup_qa_mocks(), cleanup_qa_mocks(), reset_qa_mocks() functions
  - Mock response creation helpers
  - Accessor functions for mock objects
- Refactor test_qa_fixer.py to use shared helpers
- Reduces ~80 lines of duplicated code per test file
- Fixes potential mock binding issues by using accessor functions

This addresses code quality issues identified in PR review:
- Duplicate mock setup between test_qa_fixer.py and test_qa_reviewer.py
- Duplicated _AsyncIteratorMock class across files

* refactor(tests): consolidate test_qa_reviewer.py with shared helpers

- Refactor test_qa_reviewer.py to use shared qa_test_helpers
- Remove ~170 lines of duplicated mock setup and helper functions
- Fix unused imports in test_qa_fixer.py (json, sys, MagicMock, etc.)
- Fix rate limit error detection tests to patch where functions are used
- Consolidate duplicated _create_*_response helper methods to module level

Addresses CodeQL warnings about unused imports and reduces code
duplication between test_qa_fixer.py and test_qa_reviewer.py.

* fix(tests): remove unused Path import from test_qa_reviewer.py

* fix(tests): address all PR review findings

PR Review Fixes:
- Remove unused create_mock_qa_approved_response/rejected_response functions
- Guard against overwriting _original_modules on second setup_qa_mocks() call
- Clear _original_modules in cleanup_qa_mocks() to prevent stale state
- Add prompts_pkg.project_context to test_qa_reviewer preserved_mocks in conftest
- Convert asyncio.run() pattern to native async tests in test_agent_flow.py
- Remove redundant @pytest.mark.asyncio decorators (asyncio_mode=auto)
- Remove unused pytest import from qa_test_helpers.py
- Fix structural duplication by keeping fixtures in test files

Code Quality:
- Removed ~100 lines of duplicated/unused code
- Consistent async test patterns across all QA test files
- Proper mock state management to prevent test pollution

* fix(tests): save original modules individually in setup_qa_mocks

The boolean guard `setup_done` prevented saving original modules on
subsequent calls with different parameters. When setup_qa_mocks was
called first with include_prompts_pkg=False, then with True, the
prompts_pkg modules were never saved to _original_modules. During
cleanup, these unsaved modules were deleted from sys.modules instead
of being restored, causing ModuleNotFoundError in subsequent tests.

Now checks each module individually before mocking, ensuring all
originals are saved across multiple setup calls.

* fix(tests): address all PR review findings including low priority

- Fix path in test_no_subtask_worker_config (parent.parent.parent)
- Add guard to prevent double setup in setup_qa_mocks()
- Don't clear _original_modules in cleanup to fix multi-module cleanup

* fix(tests): address PR review follow-up findings

- Fix module-level mock setup ordering dependency: now tracks
  include_prompts_pkg config and allows incremental setup when
  test_qa_fixer.py (False) is imported before test_qa_reviewer.py (True)
- Remove unused asyncio import from test_agent_flow.py
- Replace os.chdir() with monkeypatch.chdir() in prereqs validator
  tests for safe parallel test execution

---------

Co-authored-by: StillKnotKnown <stillknotknown@users.noreply.github.com>
2026-02-12 15:06:25 +01:00

675 lines
18 KiB
Python

#!/usr/bin/env python3
"""
Tests for the ci_discovery module.
Tests cover:
- GitHub Actions parsing
- GitLab CI parsing
- CircleCI parsing
- Jenkins parsing
- Test command extraction
"""
import json
import tempfile
from pathlib import Path
from unittest.mock import patch
import pytest
# Add auto-claude to path for imports
import sys
sys.path.insert(0, str(Path(__file__).parent.parent / "apps" / "backend"))
from ci_discovery import (
CIConfig,
CIWorkflow,
CIDiscovery,
discover_ci,
get_ci_test_commands,
get_ci_system,
HAS_YAML,
)
# Skip tests that require YAML parsing when PyYAML is not installed
requires_yaml = pytest.mark.skipif(not HAS_YAML, reason="PyYAML not installed")
# =============================================================================
# FIXTURES
# =============================================================================
@pytest.fixture
def temp_dir():
"""Create a temporary directory for tests."""
with tempfile.TemporaryDirectory() as tmpdir:
yield Path(tmpdir)
@pytest.fixture
def discovery():
"""Create a CIDiscovery instance."""
return CIDiscovery()
# =============================================================================
# GITHUB ACTIONS
# =============================================================================
class TestGitHubActions:
"""Tests for GitHub Actions parsing."""
def test_detect_github_actions(self, discovery, temp_dir):
"""Test GitHub Actions detection (basic file presence)."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
workflow_content = """
name: CI
on: push
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm test
"""
(workflows / "ci.yml").write_text(workflow_content)
result = discovery.discover(temp_dir)
assert result is not None
assert result.ci_system == "github_actions"
assert len(result.config_files) > 0
@requires_yaml
def test_extract_test_commands(self, discovery, temp_dir):
"""Test extracting test commands from GitHub Actions."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
workflow_content = """
name: Test
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm install
- run: npm test
- run: pytest tests/
"""
(workflows / "test.yml").write_text(workflow_content)
result = discovery.discover(temp_dir)
assert "unit" in result.test_commands
@requires_yaml
def test_detect_test_related_workflow(self, discovery, temp_dir):
"""Test detecting test-related workflows."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
workflow_content = """
name: Test Suite
on: push
jobs:
unit-tests:
runs-on: ubuntu-latest
steps:
- run: pytest tests/
"""
(workflows / "test.yml").write_text(workflow_content)
result = discovery.discover(temp_dir)
test_workflows = [w for w in result.workflows if w.test_related]
assert len(test_workflows) > 0
@requires_yaml
def test_extract_environment_variables(self, discovery, temp_dir):
"""Test extracting environment variables."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
workflow_content = """
name: CI
on: push
env:
NODE_ENV: test
CI: true
jobs:
test:
runs-on: ubuntu-latest
steps:
- run: echo test
"""
(workflows / "ci.yml").write_text(workflow_content)
result = discovery.discover(temp_dir)
assert "NODE_ENV" in result.environment_variables or "CI" in result.environment_variables
@requires_yaml
def test_handle_multiple_workflows(self, discovery, temp_dir):
"""Test handling multiple workflow files."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "ci.yml").write_text("""
name: CI
on: push
jobs:
build:
runs-on: ubuntu-latest
steps:
- run: npm build
""")
(workflows / "test.yml").write_text("""
name: Test
on: pull_request
jobs:
test:
runs-on: ubuntu-latest
steps:
- run: npm test
""")
result = discovery.discover(temp_dir)
assert len(result.config_files) == 2
assert len(result.workflows) >= 2
# =============================================================================
# GITLAB CI
# =============================================================================
class TestGitLabCI:
"""Tests for GitLab CI parsing."""
def test_detect_gitlab_ci(self, discovery, temp_dir):
"""Test GitLab CI detection."""
gitlab_ci = """
stages:
- test
- build
test:
stage: test
script:
- npm test
"""
(temp_dir / ".gitlab-ci.yml").write_text(gitlab_ci)
result = discovery.discover(temp_dir)
assert result is not None
assert result.ci_system == "gitlab"
@requires_yaml
def test_extract_gitlab_test_commands(self, discovery, temp_dir):
"""Test extracting test commands from GitLab CI."""
gitlab_ci = """
test:
script:
- pytest tests/
integration:
script:
- pytest tests/integration/
"""
(temp_dir / ".gitlab-ci.yml").write_text(gitlab_ci)
result = discovery.discover(temp_dir)
assert "unit" in result.test_commands or len(result.test_commands) > 0
def test_detect_gitlab_variables(self, discovery, temp_dir):
"""Test extracting GitLab CI variables."""
gitlab_ci = """
variables:
DATABASE_URL: postgres://localhost
NODE_ENV: test
test:
script:
- npm test
"""
(temp_dir / ".gitlab-ci.yml").write_text(gitlab_ci)
result = discovery.discover(temp_dir)
# May not work without yaml module, but should not crash
assert result.ci_system == "gitlab"
# =============================================================================
# CIRCLECI
# =============================================================================
class TestCircleCI:
"""Tests for CircleCI parsing."""
def test_detect_circleci(self, discovery, temp_dir):
"""Test CircleCI detection."""
circleci_dir = temp_dir / ".circleci"
circleci_dir.mkdir()
config = """
version: 2.1
jobs:
test:
docker:
- image: node:18
steps:
- checkout
- run: npm test
"""
(circleci_dir / "config.yml").write_text(config)
result = discovery.discover(temp_dir)
assert result is not None
assert result.ci_system == "circleci"
def test_extract_circleci_commands(self, discovery, temp_dir):
"""Test extracting commands from CircleCI."""
circleci_dir = temp_dir / ".circleci"
circleci_dir.mkdir()
config = """
version: 2.1
jobs:
test:
docker:
- image: python:3.11
steps:
- checkout
- run:
name: Run tests
command: pytest tests/ --cov
"""
(circleci_dir / "config.yml").write_text(config)
result = discovery.discover(temp_dir)
# Should find pytest command
assert result.ci_system == "circleci"
# =============================================================================
# JENKINS
# =============================================================================
class TestJenkins:
"""Tests for Jenkinsfile parsing."""
def test_detect_jenkins(self, discovery, temp_dir):
"""Test Jenkinsfile detection."""
jenkinsfile = """
pipeline {
agent any
stages {
stage('Test') {
steps {
sh 'npm test'
}
}
}
}
"""
(temp_dir / "Jenkinsfile").write_text(jenkinsfile)
result = discovery.discover(temp_dir)
assert result is not None
assert result.ci_system == "jenkins"
def test_extract_jenkins_commands(self, discovery, temp_dir):
"""Test extracting sh commands from Jenkinsfile."""
jenkinsfile = """
pipeline {
agent any
stages {
stage('Test') {
steps {
sh 'pytest tests/'
}
}
}
}
"""
(temp_dir / "Jenkinsfile").write_text(jenkinsfile)
result = discovery.discover(temp_dir)
# Should extract sh command
assert result.ci_system == "jenkins"
def test_extract_jenkins_stages(self, discovery, temp_dir):
"""Test extracting stages from Jenkinsfile."""
jenkinsfile = """
pipeline {
agent any
stages {
stage('Build') {
steps {
sh 'npm build'
}
}
stage('Test') {
steps {
sh 'npm test'
}
}
}
}
"""
(temp_dir / "Jenkinsfile").write_text(jenkinsfile)
result = discovery.discover(temp_dir)
workflow_names = [w.name for w in result.workflows]
assert "Build" in workflow_names or "Test" in workflow_names
# =============================================================================
# TEST COMMAND EXTRACTION
# =============================================================================
class TestCommandExtraction:
"""Tests for test command extraction (requires YAML parsing)."""
@requires_yaml
def test_extract_pytest(self, discovery, temp_dir):
"""Test pytest command extraction."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "test.yml").write_text("""
name: Test
on: push
jobs:
test:
runs-on: ubuntu-latest
steps:
- run: pytest tests/ -v
""")
result = discovery.discover(temp_dir)
assert "pytest" in str(result.test_commands)
@requires_yaml
def test_extract_coverage_command(self, discovery, temp_dir):
"""Test coverage command extraction."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "test.yml").write_text("""
name: Test
on: push
jobs:
test:
runs-on: ubuntu-latest
steps:
- run: pytest tests/ --cov=src
""")
result = discovery.discover(temp_dir)
# Coverage command should be extracted
assert result.coverage_command is not None or "cov" in str(result.test_commands)
@requires_yaml
def test_extract_npm_test(self, discovery, temp_dir):
"""Test npm test command extraction."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "ci.yml").write_text("""
name: CI
on: push
jobs:
test:
runs-on: ubuntu-latest
steps:
- run: npm test
""")
result = discovery.discover(temp_dir)
assert "npm" in str(result.test_commands) or "unit" in result.test_commands
@requires_yaml
def test_extract_e2e_playwright(self, discovery, temp_dir):
"""Test Playwright E2E command extraction."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "e2e.yml").write_text("""
name: E2E
on: push
jobs:
e2e:
runs-on: ubuntu-latest
steps:
- run: npx playwright test
""")
result = discovery.discover(temp_dir)
assert "e2e" in result.test_commands
@requires_yaml
def test_extract_integration_tests(self, discovery, temp_dir):
"""Test integration test command extraction."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "test.yml").write_text("""
name: Test
on: push
jobs:
integration:
runs-on: ubuntu-latest
steps:
- run: pytest tests/integration/
""")
result = discovery.discover(temp_dir)
assert "integration" in result.test_commands
# =============================================================================
# SERIALIZATION
# =============================================================================
class TestSerialization:
"""Tests for result serialization."""
def test_to_dict(self, discovery, temp_dir):
"""Test converting result to dictionary."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "ci.yml").write_text("""
name: CI
on: push
jobs:
test:
runs-on: ubuntu-latest
steps:
- run: npm test
""")
result = discovery.discover(temp_dir)
result_dict = discovery.to_dict(result)
assert isinstance(result_dict, dict)
assert "ci_system" in result_dict
assert "config_files" in result_dict
assert "test_commands" in result_dict
assert "workflows" in result_dict
def test_json_serializable(self, discovery, temp_dir):
"""Test that result is JSON serializable."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "ci.yml").write_text("""
name: CI
on: push
jobs:
test:
runs-on: ubuntu-latest
steps:
- run: npm test
""")
result = discovery.discover(temp_dir)
result_dict = discovery.to_dict(result)
# Should not raise
json_str = json.dumps(result_dict)
assert isinstance(json_str, str)
# =============================================================================
# CONVENIENCE FUNCTIONS
# =============================================================================
class TestConvenienceFunctions:
"""Tests for convenience functions."""
def test_discover_ci(self, temp_dir):
"""Test discover_ci function."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "ci.yml").write_text("name: CI\non: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps:\n - run: npm test\n")
result = discover_ci(temp_dir)
assert result is not None
assert isinstance(result, CIConfig)
def test_discover_ci_no_config(self, temp_dir):
"""Test discover_ci when no CI config exists."""
result = discover_ci(temp_dir)
assert result is None
def test_get_ci_test_commands(self, temp_dir):
"""Test get_ci_test_commands function."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "ci.yml").write_text("name: CI\non: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps:\n - run: pytest tests/\n")
commands = get_ci_test_commands(temp_dir)
assert isinstance(commands, dict)
def test_get_ci_system(self, temp_dir):
"""Test get_ci_system function."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "ci.yml").write_text("name: CI\non: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps:\n - run: npm test\n")
system = get_ci_system(temp_dir)
assert system == "github_actions"
def test_get_ci_system_not_found(self, temp_dir):
"""Test get_ci_system when no CI exists."""
system = get_ci_system(temp_dir)
assert system is None
# =============================================================================
# EDGE CASES
# =============================================================================
class TestEdgeCases:
"""Tests for edge cases."""
def test_invalid_yaml(self, discovery, temp_dir):
"""Test handling of invalid YAML."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "bad.yml").write_text("invalid: yaml: content: [")
# Should not raise
result = discovery.discover(temp_dir)
assert result is not None
def test_empty_workflow_file(self, discovery, temp_dir):
"""Test handling of empty workflow file."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "empty.yml").write_text("")
# Should not raise
result = discovery.discover(temp_dir)
assert result is not None
def test_nonexistent_directory(self, discovery):
"""Test handling of non-existent directory."""
fake_dir = Path("/nonexistent/path")
# Should not raise - mock exists to avoid permission error
with patch.object(Path, 'exists', return_value=False):
result = discovery.discover(fake_dir)
assert result is None
def test_ci_priority_github_first(self, discovery, temp_dir):
"""Test that GitHub Actions takes priority."""
# Create both GitHub and GitLab configs
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "ci.yml").write_text("name: CI\non: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps:\n - run: npm test\n")
(temp_dir / ".gitlab-ci.yml").write_text("test:\n script:\n - npm test\n")
result = discovery.discover(temp_dir)
# GitHub Actions should be detected (checked first)
assert result.ci_system == "github_actions"
def test_caching(self, discovery, temp_dir):
"""Test that results are cached."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "ci.yml").write_text("name: CI\non: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps:\n - run: npm test\n")
result1 = discovery.discover(temp_dir)
result2 = discovery.discover(temp_dir)
assert result1 is result2
def test_clear_cache(self, discovery, temp_dir):
"""Test cache clearing."""
workflows = temp_dir / ".github" / "workflows"
workflows.mkdir(parents=True)
(workflows / "ci.yml").write_text("name: CI\non: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps:\n - run: npm test\n")
result1 = discovery.discover(temp_dir)
discovery.clear_cache()
result2 = discovery.discover(temp_dir)
assert result1 is not result2