Files
Aperant/apps/backend/context/main.py
T
TamerineSky 6a6247bbf2 Fix Windows UTF-8 encoding errors across entire backend (251 instances) (#782)
* Fix UTF-8 encoding for Priorities 1-2 (Core & Agents - 18 instances)

Add encoding="utf-8" to file operations in:
- Priority 1: Core Infrastructure (8 instances)
  - core/progress.py (6 read operations)
  - core/debug.py (1 append operation)
  - core/workspace/setup.py (1 read operation)

- Priority 2: Agent System (10 instances)
  - agents/utils.py (1 read)
  - agents/tools_pkg/tools/subtask.py (1 read, 1 write)
  - agents/tools_pkg/tools/memory.py (2 read, 1 write, 1 append)
  - agents/tools_pkg/tools/qa.py (1 read, 1 write)
  - agents/tools_pkg/tools/progress.py (1 read)

All changes use double quotes for ruff format compliance.

* Fix UTF-8 encoding for Priorities 3-4 (Spec & Project - 26 instances)

Add encoding="utf-8" to file operations in:
- Priority 3: Spec Pipeline (21 instances)
  - spec/context.py (4: 2 read, 2 write)
  - spec/complexity.py (3: 2 read, 1 write)
  - spec/requirements.py (3: 2 read, 1 write)
  - spec/validator.py (3 write operations)
  - spec/writer.py (2: 1 read, 1 write)
  - spec/discovery.py (1 read)
  - spec/pipeline/orchestrator.py (2 read)
  - spec/phases/requirements_phases.py (1 write)
  - spec/validate_pkg/auto_fix.py (2: 1 read, 1 write)

- Priority 4: Project Analyzer (5 instances)
  - project/analyzer.py (2: 1 read, 1 write)
  - project/config_parser.py (2 read operations)
  - project/stack_detector.py (1 read)

All changes use double quotes for ruff format compliance.

* Fix UTF-8 encoding for Priorities 5-7 (Services, Analysis, Ideation - 43 instances)

Add encoding="utf-8" to file operations in:
- Priority 5: Services (12 instances)
  - services/recovery.py (8: 4 read, 4 write)
  - services/context.py (4 read operations)

- Priority 6: Analysis & QA (6 instances)
  - analysis/analyzers/__init__.py (2 write)
  - analysis/insight_extractor.py (1 read)
  - qa/criteria.py (2: 1 read, 1 write)
  - qa/report.py (1 read)

- Priority 7: Ideation & Roadmap (25 instances)
  - ideation/analyzer.py (3 read)
  - ideation/formatter.py (4 read, 1 write)
  - ideation/phase_executor.py (5: 3 read, 2 write)
  - ideation/runner.py (1 read)
  - runners/roadmap/competitor_analyzer.py (3: 1 read, 2 write)
  - runners/roadmap/graph_integration.py (3 write)
  - runners/roadmap/orchestrator.py (1 read)
  - runners/roadmap/phases.py (2 read)
  - runners/insights_runner.py (3 read)

All changes use double quotes for ruff format compliance.

* Fix UTF-8 encoding for Priorities 8-14 (All remaining - 85+ instances)

Add encoding="utf-8" to file operations across all remaining modules:

Priorities 8-10 (Merge, Memory, Integrations - 26 instances):
- merge/ (4 files)
- memory/ (3 files)
- context/ (3 files)
- integrations/ (4 files)

Priorities 11-14 (GitHub, GitLab, AI, Other - 59 instances):
- runners/github/ (19 files)
- runners/gitlab/ (3 files)
- runners/ai_analyzer/ (1 file)

All changes use double quotes for ruff format compliance.
Applied using Python regex script for efficiency.

* Fix UTF-8 encoding for missed instances (23 instances)

Fix remaining instances missed by batch script:
- cli/batch_commands.py (3 instances)
- cli/followup_commands.py (1 instance)
- core/client.py (1 instance)
- phase_config.py (1 instance)
- planner_lib/context.py (4 instances)
- prediction/main.py (1 instance)
- prediction/memory_loader.py (1 instance)
- prompts_pkg/prompts.py (2 instances)
- review/formatters.py (1 instance)
- review/state.py (2 instances)
- spec/phases/spec_phases.py (1 instance)
- spec/pipeline/models.py (1 instance)
- spec/validate_pkg/validators/context_validator.py (1 instance)
- spec/validate_pkg/validators/implementation_plan_validator.py (1 instance)
- ui/status.py (2 instances)

All encoding parameters use double quotes for ruff format compliance.
Verified: 0 instances without encoding remain in source code.

* Fix missed os.fdopen() calls and duplicate encoding bug

Thorough verification found 3 additional issues:
- runners/github/file_lock.py:462 - os.fdopen missing encoding
- runners/github/trust.py:442 - os.fdopen missing encoding
- runners/insights_runner.py:372 - duplicate encoding parameter

All fixed. Final count: 251 instances with encoding="utf-8"

* Fix missed Path.read_text() and Path.write_text() encoding (99 instances)

Gemini Code Assist review found instances we missed:
- Path.read_text() without encoding: 77 instances → fixed
- Path.write_text() without encoding: 22 instances → fixed

Total UTF-8 encoding fixes: 350 instances across codebase
- open() operations: 251 instances
- Path.read_text(): 98 instances
- Path.write_text(): 30 instances

All text file operations now explicitly use encoding="utf-8".

Addresses feedback from PR #782 review.

* Fix critical syntax errors from CodeRabbit review

- Fix os.getpid() syntax error in core/workspace/models.py (2 instances)
  Changed: os.getpid(, encoding="utf-8") -> str(os.getpid())

- Fix json.dumps invalid encoding parameter (3 instances)
  json.dumps() doesn't accept encoding parameter
  Changed: json.dumps(data, encoding="utf-8") -> json.dumps(data)
  Files: runners/ai_analyzer/cache_manager.py, runners/github/test_file_lock.py

- Fix tempfile.NamedTemporaryFile missing encoding
  Added encoding="utf-8" to spec/requirements.py:22

- Fix subprocess.run text=True to encoding
  Changed: text=True -> encoding="utf-8" in core/workspace/setup.py:375

All critical syntax errors from CodeRabbit review resolved.

* Fix critical syntax errors in test_context_gatherer.py

- Line 78: Move encoding="utf-8" outside of JS string content
  Changed: write_text("...encoding="utf-8"...")
  To: write_text("...", encoding="utf-8")

- Line 102: Move encoding="utf-8" outside of JS string content
  Changed: write_text("...encoding="utf-8"...")
  To: write_text("...", encoding="utf-8")

Fixes syntax errors where encoding parameter was incorrectly placed
inside the JavaScript code string instead of as write_text() parameter.

* Fix CodeRabbit issues: UnicodeDecodeError handling and trailing newlines

- Add UnicodeDecodeError to exception handling in agents/utils.py and spec/validate_pkg/auto_fix.py
- Fix trailing newline preservation in merge/file_merger.py (2 locations)
- Add encoding parameter to atomic_write() in runners/github/file_lock.py

These fixes ensure robust error handling for malformed UTF-8 files
and preserve file formatting during merge operations.

* Fix test fixture to use UTF-8 encoding consistently

Update spec_file fixture in tests/conftest.py to write spec file
with encoding="utf-8" to match how it's read in validators.

This ensures consistency between test fixtures and production code.

* Fix linting errors and security vulnerabilities from merge

- Remove unused tree-sitter methods in semantic_analyzer.py that caused F821 undefined name errors
- Fix regex injection vulnerability in bump-version.js by properly escaping all regex special characters
- Add escapeRegex() function to prevent security issues when version string is used in RegExp constructor

Resolves ruff linting failures and CodeQL security alerts.

* Fix code formatting for ruff compliance

Apply formatting fixes to meet line length requirements:
- context/builder.py: Split long line with array slicing
- planner_lib/context.py: Split long ternary expression
- spec/requirements.py: Split long tempfile.NamedTemporaryFile call

Resolves ruff format check failures.

* Fix missing UTF-8 encoding in init.py gitignore operations

Found by pre-commit hook testing in PR #795:
- Line 96: Path.read_text() without encoding
- Line 122: Path.write_text() without encoding

These handle .gitignore file operations and could fail on Windows
with special characters in gitignore comments or entries.

Total fixes in PR #782: 253 instances (was 251, +2 from init.py)

* Add pre-commit hook for UTF-8 encoding enforcement

1. Encoding Check Script (scripts/check_encoding.py):
   - Validates all file operations have encoding="utf-8"
   - Checks open(), Path.read_text(), Path.write_text()
   - Checks json.load/dump with open()
   - Allows binary mode without encoding
   - Windows-compatible emoji output with UTF-8 reconfiguration

2. Pre-commit Config (.pre-commit-config.yaml):
   - Added check-file-encoding hook for apps/backend/
   - Runs automatically before commits
   - Scoped to backend Python files only

3. Tests (tests/test_check_encoding.py):
   - Comprehensive test coverage (10 tests, all passing)
   - Tests detection of missing encoding
   - Tests allowlist for binary files
   - Tests multiple issues in single file
   - Tests file type filtering

Purpose:
- Prevent regression of 251 UTF-8 encoding fixes from PR #782
- Catch missing encoding in new code during development
- Fast feedback loop for developers

Implementation Notes:
- Hook scoped to apps/backend/ to avoid false positives in test code
- Uses simple regex matching for speed
- Compatible with existing pre-commit infrastructure
- Already caught 6 real issues in apps/backend/core/progress.py

Related: PR #782 - Fix Windows UTF-8 encoding errors

* Address CodeRabbit and Gemini review feedback

Fixes based on automated review comments:

1. Binary Mode Detection (Critical Fix):
   - Replaced brittle regex with robust pattern: r'["'][rwax+]*b[rwax+]*["']'
   - Now correctly detects all binary modes: rb, wb, ab, r+b, w+b, etc.
   - Prevents false positives on text mode 'w' without 'b'
   - Added comprehensive tests for wb, ab, and text w modes

2. Encoding Detection Robustness (Critical Fix):
   - Changed from 'encoding=' string match to word boundary regex: r'\bencoding\s*='
   - Now handles encoding with spaces: encoding = "utf-8"
   - Prevents false matches of substrings containing 'encoding='
   - Applied across all checks (open, read_text, write_text, json.load, json.dump)
   - Added test for spaces around equals sign

3. Test Coverage Improvements:
   - Added json.dump() with encoding test (passing case)
   - Added json.dump() without encoding test (failing case)
   - Fixed test assertions to match actual behavior (== 1 not == 2)
   - Added 6 new tests for improved binary/text mode coverage
   - Total tests increased from 10 to 16, all passing 

4. Code Cleanup:
   - Removed unused pytest import (CodeQL warning)
   - Simplified check_files() to remove unused variable tracking

All changes validated with comprehensive test suite (16/16 passing).

Related: PR #795 review feedback from CodeRabbit and Gemini Code Assist

* docs: Add UTF-8 encoding guidelines and Windows development guide

1. CONTRIBUTING.md:
   - Added concise file encoding section after Code Style
   - DO/DON'T examples for common file operations
   - Covers open(), Path methods, json operations
   - References PR #782 and windows-development.md

2. guides/windows-development.md (NEW):
   - Comprehensive Windows development guide
   - File encoding (cp1252 vs UTF-8 issue)
   - Line endings, path separators, shell commands
   - Development environment recommendations
   - Common pitfalls and solutions
   - Testing guidelines

3. .github/PULL_REQUEST_TEMPLATE.md:
   - Added encoding checklist item for Python PRs
   - Helps catch missing encoding during review

4. guides/README.md:
   - Added windows-development.md to guide index
   - Organized with CLI-USAGE and linux guides

Purpose: Educate developers about UTF-8 encoding requirements to prevent
regressions of the 251 encoding issues fixed in PR #782. Automated checking
via pre-commit hooks (PR #795) + developer education ensures long-term
Windows compatibility.

Related:
- PR #782: Fix Windows UTF-8 encoding errors (251 instances)
- PR #795: Add pre-commit hooks for encoding enforcement

* Address review comments from CodeRabbit and Gemini

1. Fix CONTRIBUTING.md markdown linting issues
   - Add blank lines around code blocks (MD031)
   - Add JSON write example with ensure_ascii=False (Gemini suggestion)

2. Fix guides/windows-development.md markdown linting (39 violations)
   - Rename duplicate headings: "The Problem"/"The Solution" → "Problem"/"Solution" (MD024)
   - Add blank lines around all code blocks (MD031)
   - Add language specifiers to code blocks (MD040)
   - Add blank lines before/after headings (MD022)
   - Wrap long lines to <=80 characters (MD013)
   - Add blank line before list (MD032)
   - Use Gemini's idiomatic line ending normalization pattern

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix additional UTF-8 encoding issues and improve encoding check script

- Add encoding="utf-8" to 5 files that were missing it:
  - cli/workspace_commands.py: read_text for worktree config
  - context/pattern_discovery.py: read_text with errors param
  - context/search.py: read_text with errors param
  - core/sentry.py: open for package.json version detection
  - core/workspace/setup.py: open for security profile JSON

- Improve check_encoding.py script to reduce false positives:
  - Use negative lookbehind to exclude os.open(), urlopen(), etc.
  - Handle nested parentheses correctly when checking args
  - Skip self.method.read_text() calls (custom methods, not Path)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix missing UTF-8 encoding in locked_write() function

Add encoding parameter to locked_write() async context manager and
use it in os.fdopen() call. This fixes HIGH priority issue from PR review
where locked_write() was missing UTF-8 encoding support, which could cause
encoding errors on Windows when writing files with non-ASCII content.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add UnicodeDecodeError handling for file loading resilience

Address CodeRabbit review feedback:
- runner.py: Add UnicodeDecodeError to exception handling when loading batch files
- trust.py: Add exception handling in get_state() and get_all_states() to
  gracefully handle corrupted state files instead of failing completely

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix atomic_write to handle binary mode correctly

The atomic_write function was unconditionally passing encoding to os.fdopen,
which would crash with ValueError if called with binary mode (e.g., 'wb').
Apply the same fix used in locked_write: only pass encoding for text modes.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix run_git() call with invalid parameters in setup.py

Remove capture_output and encoding kwargs from run_git() call - these
parameters are already handled internally by run_git() and passing them
causes TypeError since the function doesn't accept them.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix CodeQL warnings and potential double-newline bug

- Remove unused is_path_call variables in check_encoding.py
- Remove unused failed_count variable in check_encoding.py
- Remove unused escapeRegex function in bump-version.js
- Fix potential double-newline when adding imports in file_merger.py
  (strip trailing newlines from content_after before inserting)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix Ruff formatting: wrap long line in file_merger.py

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add UnicodeDecodeError handling to all JSON file loading

Comprehensively add UnicodeDecodeError to exception handlers across
the codebase to handle legacy-encoded or corrupted files gracefully:

- 32+ locations now catch UnicodeDecodeError alongside OSError and
  json.JSONDecodeError
- context/builder.py: Regenerate index on decode failure
- planner_lib/context.py: Use empty dicts on decode failure
- check_encoding.py: Handle OSError for unreadable files
- cleanup.py: Handle decode errors in index pruning

This ensures the codebase is robust against non-UTF-8 files that may
exist from previous Windows runs with cp1252 encoding.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add explanatory comments to empty except clauses

Address CodeQL notices about empty except clauses with just 'pass'
by adding explanatory comments describing the intent.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix review issues from Andy's Auto Claude PR Review

1. [HIGH] Fix double-close bug in trust.py:449
   - Remove try/except around os.fdopen since it takes ownership of fd
   - The with statement handles closing, no need for explicit os.close()

2. [LOW] Fix dead code in file_merger.py:87,159
   - Simplify endswith check to just '\n' since content is already
     normalized to LF at that point

3. [LOW] Fix escaped backslash-n in test_context_gatherer.py:150
   - Change "\n" (literal backslash-n) to "\n" (actual newline)

4. [LOW] Fix coder.md examples missing encoding parameter
   - Add encoding="utf-8" to read_text() and open() calls in examples

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: TamerineSky <TamerineSky@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 22:22:55 +01:00

145 lines
3.6 KiB
Python

#!/usr/bin/env python3
"""
Task Context Builder
====================
Builds focused context for a specific task by searching relevant services.
This is the "RAG-like" component that finds what files matter for THIS task.
Usage:
# Find context for a task across specific services
python auto-claude/context.py \
--services backend,scraper \
--keywords "retry,error,proxy" \
--task "Add retry logic when proxies fail" \
--output auto-claude/specs/001-retry/context.json
# Use project index to auto-suggest services
python auto-claude/context.py \
--task "Add retry logic when proxies fail" \
--output context.json
The context builder will:
1. Load project index (from analyzer)
2. Search specified services for relevant files
3. Find similar implementations to reference
4. Output focused context for AI agents
"""
import json
from pathlib import Path
from context import (
ContextBuilder,
FileMatch,
TaskContext,
)
from context.serialization import serialize_context
# Backward compatibility exports
__all__ = [
"ContextBuilder",
"FileMatch",
"TaskContext",
"build_task_context",
]
def build_task_context(
project_dir: Path,
task: str,
services: list[str] | None = None,
keywords: list[str] | None = None,
output_file: Path | None = None,
) -> dict:
"""
Build context for a task and optionally save to file.
Args:
project_dir: Path to project root
task: Task description
services: Services to search (None = auto-detect)
keywords: Keywords to search for (None = extract from task)
output_file: Optional path to save JSON output
Returns:
Context as a dictionary
"""
builder = ContextBuilder(project_dir)
context = builder.build_context(task, services, keywords)
result = serialize_context(context)
if output_file:
output_file.parent.mkdir(parents=True, exist_ok=True)
with open(output_file, "w", encoding="utf-8") as f:
json.dump(result, f, indent=2)
print(f"Task context saved to: {output_file}")
return result
def main():
"""CLI entry point."""
import argparse
parser = argparse.ArgumentParser(
description="Build task-specific context by searching the codebase"
)
parser.add_argument(
"--project-dir",
type=Path,
default=Path.cwd(),
help="Project directory (default: current directory)",
)
parser.add_argument(
"--task",
type=str,
required=True,
help="Description of the task",
)
parser.add_argument(
"--services",
type=str,
default=None,
help="Comma-separated list of services to search",
)
parser.add_argument(
"--keywords",
type=str,
default=None,
help="Comma-separated list of keywords to search for",
)
parser.add_argument(
"--output",
type=Path,
default=None,
help="Output file for JSON results",
)
parser.add_argument(
"--quiet",
action="store_true",
help="Only output JSON, no status messages",
)
args = parser.parse_args()
# Parse comma-separated args
services = args.services.split(",") if args.services else None
keywords = args.keywords.split(",") if args.keywords else None
result = build_task_context(
args.project_dir,
args.task,
services,
keywords,
args.output,
)
if not args.quiet or not args.output:
print(json.dumps(result, indent=2))
if __name__ == "__main__":
main()