Files
Aperant/tests/test_client.py
T
Andy d8f4de9a06 feat(pr-review): add validation pipeline, context enrichment, and cross-validation (#1354)
* docs(phase-1): research core validation pipeline

Phase 1: Core Validation Pipeline
- Finding-validator pattern from follow-up reviews documented
- Orchestrator integration points identified
- Context bug at line 1288 analyzed
- Prompt patterns for Read tool instructions catalogued
- Evidence/scope validation strategies defined

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(01-01): include AI reviews in follow-up context

- Fixed ai_bot_comments_since_review to include ai_reviews
- Mirrors contributor_comments + contributor_reviews pattern
- AI formal reviews (CodeRabbit, Cursor) now available to follow-up agents

* feat(01-02): add tool usage instructions to follow-up agent prompts

- Add "CRITICAL: Full Context Analysis" section to follow-up prompts
- Require Read tool usage before reporting findings
- Require +-20 lines context around flagged lines
- Require actual code evidence, not descriptions
- Require Grep search for mitigations

Files: pr_followup_resolution_agent.md, pr_followup_newcode_agent.md

* test(01-01): add tests for AI reviews inclusion in follow-up context

- Test AI bot patterns include known bots (CodeRabbit, Gemini, Copilot)
- Test FollowupReviewContext has ai_bot_comments_since_review field
- Test FollowupContextGatherer.gather() includes AI formal reviews
- Test AI reviews are correctly separated from contributor reviews

* feat(01-03): add finding-validator agent to parallel orchestrator

- Load pr_finding_validator.md prompt in _define_specialist_agents()
- Add finding-validator AgentDefinition with tools [Read, Grep, Glob]
- Description instructs to validate ALL findings after specialist agents

* feat(01-03): add Phase 3.5 validation step to orchestrator prompt

- Add finding-validator to Available Specialist Agents section
- Add Phase 3.5: Finding Validation (CRITICAL - Prevent False Positives)
- Instructions to invoke validator for ALL findings after synthesis
- Filter based on validation status (confirmed_valid, dismissed_false_positive)
- Re-calculate verdict based only on validated findings

* feat(01-03): add validation fields to orchestrator output format

- Add validation_summary top-level field (total, confirmed, dismissed, needs_review)
- Add validation_status field per finding (confirmed_valid, dismissed_false_positive, needs_human_review)
- Add validation_evidence field per finding with actual code snippet
- Document that dismissed findings should be removed from output

* feat(01-04): add evidence validation function for PR findings

- Add _validate_finding_evidence() helper to validate evidence quality
- Rejects findings with no evidence or very short evidence (<10 chars)
- Filters findings that start with description patterns (not code)
- Requires code syntax characters in evidence to pass validation

* feat(01-04): add scope pre-filter function for PR findings

- Add _is_finding_in_scope() to verify findings are within PR scope
- Rejects findings for files not in changed files list
- Allows impact findings (affect/break/depend) for unchanged files
- Rejects findings with invalid line numbers (<= 0)

* feat(01-04): integrate evidence and scope filters into finding processing

- Apply _validate_finding_evidence to filter findings with poor evidence
- Apply _is_finding_in_scope to filter findings outside PR scope
- Log filtered findings with reasons for debugging
- Replace unique_findings with validated_findings for verdict/summary

* docs(02): create phase 2 plans for context enrichment

Phase 02: Context Enrichment
- 3 plans in 2 waves
- Plans 01 & 02 parallel (Wave 1), Plan 03 sequential (Wave 2)
- Ready for execution

Plan details:
- 02-01: JS/TS import analysis (path aliases, CommonJS, re-exports)
- 02-02: Python import analysis via AST
- 02-03: Related files enhancement (limit 50, prioritization, reverse deps)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(02-02): add Python import resolution methods

- Add ast import for Python AST parsing
- Add _resolve_python_import() to resolve module names to file paths
- Add _find_python_imports() to extract imports using AST
- Handles relative imports (from . import, from .. import)
- Handles absolute imports that map to project files
- Gracefully handles SyntaxError in Python files

* feat(02-02): integrate Python import detection into _find_imports

- Replace TODO comment with actual Python import detection
- Call _find_python_imports() for .py files in _find_imports()
- Python files now have their imports resolved to file paths

* fix(02-01): prevent _load_json_safe from mangling path patterns with /*

The regex-based comment stripping was incorrectly removing path patterns
like "@/*" from tsconfig.json because /* looks like a multi-line comment.

Fix:
- Try standard JSON parse first (most tsconfigs don't have comments)
- Fall back to smarter comment stripping that checks if // appears
  outside of strings by counting quotes before the comment position

This ensures path aliases like "@/*": ["src/*"] are preserved.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(02-03): add reverse dependency detection

- Add _find_dependents() method to find files that import a given file
- Use grep with recursive search for import/from statements
- Skip generic names (index, main, utils) to avoid too many matches
- 5-second timeout protection prevents hanging on large repos
- Exclude common non-code directories (node_modules, .git, __pycache__)
- Limit results to prevent overwhelming context

* feat(02-03): add smart file prioritization

- Add _prioritize_related_files() method for relevance-based ordering
- Priority: tests > type definitions > configs > other files
- Sort alphabetically within each category for consistency
- Supports limit parameter (default 50)
- Fix .d.ts detection using name_lower.endswith('.d.ts')

* feat(02-03): update _find_related_files with reverse deps and prioritization

- Add reverse dependency detection call to _find_related_files()
- Replace simple sorting with _prioritize_related_files()
- Increase limit from 20 to 50 files
- Update find_related_files_for_root() static method limit to 50
- Tests pass (1616 passed, 11 skipped)

* docs(03): research phase 3 cross-validation domain

Phase 3: Cross-Validation
- Confidence threshold routing (REQ-011)
- Multi-agent cross-validation (REQ-012)
- Standard stack identified (built-in Python, existing Pydantic models)
- Architecture patterns documented
- Common pitfalls catalogued

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* docs(03): create phase 3 plans for cross-validation

Phase 03: Cross-Validation
- 2 plans in 2 waves
- Plan 03-01: Confidence threshold routing (Wave 1)
- Plan 03-02: Multi-agent agreement and confidence boost (Wave 2)
- Ready for execution

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(03): revise plans based on checker feedback

Address checker issues:
- 03-01: Add Task 0 to add confidence, source_agents, cross_validated fields to PRReviewFinding dataclass
- 03-02: Update Task 1 to clarify it uses the new PRReviewFinding fields (not just pydantic model)
- 03-02: Document that AgentAgreement is logged for monitoring, not persisted to PRReviewResult

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(03-01): add cross-validation fields to PRReviewFinding

- Add confidence: float = 0.5 field for confidence scoring
- Add source_agents: list[str] field to track which agents reported finding
- Add cross_validated: bool field to track multi-agent agreement
- Update to_dict() to include all three new fields
- Update from_dict() to handle all three new fields with defaults
- Fix output_validator to treat confidence=0.5 as default (not explicit)

* feat(03-01): add confidence routing function

- Add ConfidenceTier class with HIGH/MEDIUM/LOW constants (0.8/0.5 thresholds)
- Add _apply_confidence_routing() method to ParallelOrchestratorReviewer
- HIGH (>=0.8): Include finding as-is
- MEDIUM (0.5-0.8): Include with '[Potential]' prefix in title
- LOW (<0.5): Log and exclude from output
- Handle missing confidence gracefully (default to 0.5)
- Log tier distribution after routing

* feat(03-01): wire confidence routing into review pipeline

- Call _apply_confidence_routing() after evidence/scope validation
- Log routing results: included count vs dropped (low confidence)
- Use routed findings for verdict and summary generation
- Confidence routing happens AFTER validation, BEFORE verdict

* docs(03-01): update orchestrator prompt with confidence tier guidance

- Add 'Confidence Tiers' section after Phase 3.5
- Document tier thresholds: HIGH (>=0.8), MEDIUM (0.5-0.8), LOW (<0.5)
- Include guidelines for assigning confidence scores
- Provide examples of confidence score assignments
- Placed between validation section and output format

* docs(03-01): complete confidence threshold routing plan

Tasks completed: 4/4
- Task 0: Add cross-validation fields to PRReviewFinding model
- Task 1: Add confidence routing function
- Task 2: Wire confidence routing into review pipeline
- Task 3: Update orchestrator prompt with confidence tier guidance

SUMMARY: .planning/phases/03-cross-validation/03-01-SUMMARY.md

* feat(03-02): add _cross_validate_findings method

- Groups findings by (file, line, category) for multi-agent agreement detection
- Boosts confidence by 0.15 (capped at 0.95) when 2+ agents agree
- Sets cross_validated=True and populates source_agents on PRReviewFinding
- Returns AgentAgreement tracking object with agreed_findings list
- Uses collections.defaultdict for efficient grouping
- Merges evidence with '---' separator, keeps highest severity

* feat(03-02): wire cross-validation into review pipeline

- Call _cross_validate_findings after deduplication
- Cross-validated findings flow through evidence/scope validation
- Cross-validated findings flow through confidence routing
- Log AgentAgreement: info level for summary, debug level for full JSON
- Pipeline order: deduplicate -> cross-validate -> validate evidence/scope -> confidence route

* docs(03-02): add multi-agent agreement documentation to orchestrator prompt

- Add 'Multi-Agent Agreement' section documenting confidence boost behavior
- Document +0.15 confidence boost when 2+ agents agree (max 0.95)
- Add example showing merged finding with cross_validated and source_agents
- Document agent_agreement tracking and logging behavior
- Update Phase 3: Synthesis to reference cross-validation and confidence routing

* docs(04): create phase plan for integration testing

Phase 04: Integration Testing
- 1 plan in 1 wave
- Tests all Phase 1-3 features
- Ready for execution

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test(04-01): add Phase 1 feature tests - confidence, evidence, scope

- Add TestConfidenceTierRouting with 7 tests for tier boundaries
- Add TestEvidenceValidation with 6 tests for code syntax detection
- Add TestScopeFiltering with 6 tests for scope filtering logic
- Import ConfidenceTier, _validate_finding_evidence, _is_finding_in_scope
- All 18 Phase 1 tests passing

* test(04-01): add Phase 2 and Phase 3 feature tests

Phase 2 - Import Detection (5 tests):
- Path alias detection (@/utils -> src/utils.ts)
- CommonJS require('./utils') detection
- Re-export (export * from) detection
- Python relative import via AST
- Python absolute import resolution

Phase 2 - Reverse Dependencies (3 tests):
- Grep-based dependent file detection
- Generic name skipping (index, main, utils)
- Timeout handling for large repos

Phase 3 - Cross-Validation (7 tests):
- Multi-agent agreement confidence boost (+0.15)
- Confidence cap at 0.95
- cross_validated flag on merged findings
- Grouping by (file, line, category) tuple
- Description combination with ' | ' separator
- Single-agent findings not boosted
- Highest severity preserved on merge

All 33 tests passing

* test(04-01): add integration pipeline verification tests

TestIntegrationPipeline (9 tests):
- Full pipeline flow: high confidence + valid evidence + in scope
- Low confidence filtering behavior documentation
- Cross-validation elevating MEDIUM to HIGH tier
- Invalid evidence rejection regardless of confidence
- Out-of-scope rejection
- Impact finding allowance for unchanged files
- End-to-end review scenario with multiple agents
- Empty findings handling
- Confidence tier routing documentation

Total: 42 integration tests passing

* gitignore planning for GSD test

* chore: remove .planning/ from git tracking

These files are in .gitignore but were committed before the ignore
rule was added. Removing from tracking to keep planning files local.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: cross-platform _find_dependents and improved test assertions

- Replace grep subprocess with pure Python os.walk() + re.compile()
  for cross-platform compatibility (Windows, macOS, Linux)
- Add debug logging to _load_json_safe() for troubleshooting
- Fix test assertion type (set instead of list)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: address all PR review findings (10 issues)

HIGH priority fixes:
- Fix path alias resolution to use project root instead of relative path
- Rewrite test to mock os.walk instead of subprocess.run
- Extract duplicated 'Full Context Analysis' to partials/ with sync comments

MEDIUM priority fixes:
- Extract _resolve_any_import() helper to eliminate DRY violation
- Improve path alias test to verify actual resolution
- Add guard for empty target_paths in tsconfig
- Convert ConfidenceTier to str, Enum pattern
- Add block comment stripping in _load_json_safe

LOW priority fixes:
- Remove unused tempfile import
- Remove duplicate .planning/ gitignore entry

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: restore phase_config module after mock to prevent test pollution

The test_integration_phase4.py was mocking phase_config at module level
during import, which polluted sys.modules for subsequent tests. This
caused test_agent_configs::test_thinking_defaults_are_valid to fail
because THINKING_BUDGET_MAP.keys() returned empty from the MagicMock.

Fix: Save and restore the original phase_config module after loading
the orchestrator module.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: add env cleanup fixture to test_client.py for test isolation

Add autouse fixture to clear AUTH_TOKEN_ENV_VARS before and after each
test in TestClientTokenValidation. This ensures test isolation and
prevents env var pollution from previous tests in the suite.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: mock decrypt_token in encrypted token rejection tests

Also mock decrypt_token to raise ValueError, ensuring the encrypted
token flows through to validate_token_not_encrypted regardless of
whether the CI environment has a claude CLI available that might
attempt decryption.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: restore all mocked modules in test_integration_phase4.py

The test was mocking core.client, phase_config, and other modules at
module level but only restoring phase_config. This caused core.client
to remain as a MagicMock, which made validate_token_not_encrypted a
MagicMock that never raised ValueError.

Now all mocked modules are saved before mocking and restored after
the orchestrator module is loaded.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: normalize path separators for cross-platform test compatibility

Windows returns paths with backslashes (src\utils.ts) while the test
expected forward slashes (src/utils.ts). Normalize to forward slashes
for comparison.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: normalize path separators in all import detection tests

Apply the same Windows path normalization fix to:
- test_commonjs_require_detection
- test_reexport_detection
- test_python_relative_import
- test_python_absolute_import

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 18:22:57 +01:00

138 lines
5.8 KiB
Python

#!/usr/bin/env python3
"""
Tests for Client Creation and Token Validation
===============================================
Tests the client.py and simple_client.py module functionality including:
- Token validation before SDK initialization
- Encrypted token rejection
- Client creation with valid tokens
"""
import os
from unittest.mock import MagicMock, patch
import pytest
# Auth token env vars that need to be cleared between tests
AUTH_TOKEN_ENV_VARS = [
"CLAUDE_CODE_OAUTH_TOKEN",
"ANTHROPIC_AUTH_TOKEN",
]
class TestClientTokenValidation:
"""Tests for client token validation."""
@pytest.fixture(autouse=True)
def clear_env(self):
"""Clear auth environment variables before and after each test."""
for var in AUTH_TOKEN_ENV_VARS:
os.environ.pop(var, None)
yield
for var in AUTH_TOKEN_ENV_VARS:
os.environ.pop(var, None)
def test_create_client_rejects_encrypted_tokens(self, tmp_path, monkeypatch):
"""Verify create_client() rejects encrypted tokens."""
from core.client import create_client
monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", "enc:test123456789012")
# Mock keychain to ensure encrypted token is the only source
monkeypatch.setattr("core.auth.get_token_from_keychain", lambda: None)
# Mock decrypt_token to raise ValueError (simulates decryption failure)
# This ensures the encrypted token flows through to validate_token_not_encrypted
monkeypatch.setattr(
"core.auth.decrypt_token",
lambda t: (_ for _ in ()).throw(ValueError("Decryption not supported")),
)
with pytest.raises(ValueError, match="encrypted format"):
create_client(tmp_path, tmp_path, "claude-sonnet-4", "coder")
def test_create_simple_client_rejects_encrypted_tokens(self, monkeypatch):
"""Verify create_simple_client() rejects encrypted tokens."""
from core.simple_client import create_simple_client
monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", "enc:test123456789012")
# Mock keychain to ensure encrypted token is the only source
monkeypatch.setattr("core.auth.get_token_from_keychain", lambda: None)
# Mock decrypt_token to raise ValueError (simulates decryption failure)
monkeypatch.setattr(
"core.auth.decrypt_token",
lambda t: (_ for _ in ()).throw(ValueError("Decryption not supported")),
)
with pytest.raises(ValueError, match="encrypted format"):
create_simple_client(agent_type="merge_resolver")
def test_create_client_accepts_valid_plaintext_token(self, tmp_path, monkeypatch):
"""Verify create_client() accepts valid plaintext tokens and creates SDK client."""
valid_token = "sk-ant-oat01-valid-plaintext-token"
monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", valid_token)
monkeypatch.setattr("core.auth.get_token_from_keychain", lambda: None)
# Mock the SDK client to avoid actual initialization
mock_sdk_client = MagicMock()
with patch("core.client.ClaudeSDKClient", return_value=mock_sdk_client):
from core.client import create_client
client = create_client(tmp_path, tmp_path, "claude-sonnet-4", "coder")
# Verify SDK client was created
assert client is mock_sdk_client
def test_create_simple_client_accepts_valid_plaintext_token(self, monkeypatch):
"""Verify create_simple_client() accepts valid plaintext tokens and creates SDK client."""
valid_token = "sk-ant-oat01-valid-plaintext-token"
monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", valid_token)
monkeypatch.setattr("core.auth.get_token_from_keychain", lambda: None)
# Mock the SDK client to avoid actual initialization
mock_sdk_client = MagicMock()
with patch(
"core.simple_client.ClaudeSDKClient", return_value=mock_sdk_client
):
from core.simple_client import create_simple_client
client = create_simple_client(agent_type="merge_resolver")
# Verify SDK client was created
assert client is mock_sdk_client
def test_create_client_validates_token_before_sdk_init(
self, tmp_path, monkeypatch
):
"""Verify create_client() validates token format before SDK initialization."""
valid_token = "sk-ant-oat01-valid-token"
monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", valid_token)
monkeypatch.setattr("core.auth.get_token_from_keychain", lambda: None)
# Mock validate_token_not_encrypted to verify it's called
with patch(
"core.client.validate_token_not_encrypted"
) as mock_validate, patch("core.client.ClaudeSDKClient"):
from core.client import create_client
create_client(tmp_path, tmp_path, "claude-sonnet-4", "coder")
# Verify validation was called with the token
mock_validate.assert_called_once_with(valid_token)
def test_create_simple_client_validates_token_before_sdk_init(self, monkeypatch):
"""Verify create_simple_client() validates token format before SDK initialization."""
valid_token = "sk-ant-oat01-valid-token"
monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", valid_token)
monkeypatch.setattr("core.auth.get_token_from_keychain", lambda: None)
# Mock validate_token_not_encrypted to verify it's called
with patch(
"core.simple_client.validate_token_not_encrypted"
) as mock_validate, patch("core.simple_client.ClaudeSDKClient"):
from core.simple_client import create_simple_client
create_simple_client(agent_type="merge_resolver")
# Verify validation was called with the token
mock_validate.assert_called_once_with(valid_token)