Files
Aperant/apps/backend/integrations/graphiti/tests/test_providers_llm_ollama.py
T
StillKnotKnown 4f1b7b2a95 test: improve backend memory system test coverage to 100% (#1780)
* test: add comprehensive test suite for backend memory system

Add 25 test files covering the integrations/graphiti memory system:
- Core module tests (client, queries, search, graphiti, schema)
- Migration tests (migrate_embeddings, kuzu_driver_patched)
- Provider tests (6 embedder + 6 LLM providers)
- Cross-encoder and config tests

Coverage achievements:
- 134 passing tests for core modules
- graphiti.py: 95%, queries.py: 87%, client.py: 96%
- cross_encoder.py: 74%, search.py: 95%, config.py: 94%
- Overall: 51% coverage (up from 46%)

Tests were moved from apps/backend/tests/ (gitignored) to
tests/integrations/ to be included in version control.

* test: add pytest configuration with markers for long-running tests

Add pyproject.toml for backend testing with:
- pytest markers for slow/integration/smoke tests
- optimized test configuration (maxfail, -v, -m "not slow")
- coverage settings with HTML and terminal reporting
- mypy configuration for type checking

This ensures long-running tests are excluded from default CI runs
while maintaining comprehensive test coverage reporting.

* fix: resolve F821 undefined name errors in test_kuzu_driver_patched.py

Fixed 14 F821 undefined name errors for mock_kuzu_driver_module by
adding proper local definitions before each patch.dict call in test
methods that use the mock.

Also fixed encoding issue in test_config.py (added encoding='utf-8' to
open() call).

All 426 tests now pass with pre-commit hooks successful.

* test: add tests for __init__.py and providers.py modules

Added comprehensive test coverage for:
- integrations/graphiti/__init__.py: Test lazy import __getattr__ functionality
- integrations/graphiti/providers.py: Test re-exported items from graphiti_providers

These modules now have 100% test coverage.

* test: add error path tests for cross_encoder.py

Added tests for:
- ImportError when graphiti_core modules not available
- Exception during reranker creation

cross_encoder.py now has 100% test coverage (23 statements).

* test: add test for Windows non-pywin32 import error

Added test for Windows-specific import error that is not a pywin32 error,
which logs a debug message instead of an error.

client.py coverage improved from 95.9% to 96.7% (4 lines remaining).

* test: add fast success path tests for azure_openai_llm and openrouter_llm

Added fast (non-slow) tests for the success paths in:
- azure_openai_llm.py: Now 100% coverage (was 83.3%)
- openrouter_llm.py: Now 100% coverage (was 83.3%)

Both files now have complete test coverage without relying on slow test markers.

* test: add fast success path tests for azure_openai and openai embedders

Added fast (non-slow) tests for the success paths in:
- azure_openai_embedder.py: Now 100% coverage (was 87.5%)
- openai_embedder.py: Now 100% coverage (was 81.8%)

Both embedder files now have complete test coverage without relying on slow test markers.

* test: add fast success path tests for voyage, openrouter, and ollama embedders

Added fast (non-slow) tests for the success paths in:
- voyage_embedder.py: Now 100% coverage (was 81.8%)
- openrouter_embedder.py: Now 100% coverage (was 81.8%)
- ollama_embedder.py: Now 100% coverage (was 76.0%)

All embedder files now have complete test coverage without relying on slow test markers.

* test: add fast success path tests for ollama, openai, and anthropic LLM providers

Added fast (non-slow) tests for the success paths in:
- ollama_llm.py: Now 100% coverage (was 66.7%)
- openai_llm.py: Now 93.8% coverage (was 56.2%)
- anthropic_llm.py: Now 91.7% coverage (was 58.3%)

All LLM providers now have comprehensive test coverage without relying on slow test markers.

* test: improve backend memory system test coverage to 55.8%

- 100% coverage for 26 files including:
  - All embedder providers (ollama, openai, azure_openai, voyage, openrouter)
  - All LLM providers (ollama, openai, azure_openai, anthropic, openrouter)
  - validators.py, utils.py, search.py, client.py, schema.py
  - All __init__.py modules in providers_pkg

- Added comprehensive tests for:
  - validator functions (validate_embedding_config, test_llm_connection,
    test_embedder_connection, test_ollama_connection)
  - search methods (non-dict content handling, JSON decode errors)
  - provider exceptions and error handling
  - Fast test variants for slow-marked tests

- Fixed namespace package mocking for google providers
- Improved test patterns for local imports and exception handlers

507 tests passing

* test: improve queries.py coverage to 100%

- Added tests for duplicate_facts exception handling in:
  - gotchas_discovered (lines 418-419)
  - approach_outcome (lines 457-458)
  - recommendations (lines 488-489)

- Added test for outer exception handler (lines 499-523)
- Removed duplicate test definition
- All tests passing with comprehensive exception coverage

42 tests passing, 100% coverage for queries.py

* test: improve google_embedder.py, google_llm.py, migrate_embeddings.py coverage

- google_embedder.py: 100% coverage (was 42.9%)
- google_llm.py: 100% coverage (was 39.6%)
- migrate_embeddings.py: 61.5% coverage (was 33.3%)

Changes:
- Added fast variants of async tests without @pytest.mark.slow
- Added tests for assistant role handling in google_llm.py
- Added tests for JSON decode error handling in google_llm.py
- Added tests for timestamp parsing in migrate_embeddings.py
- Added tests for target exception handler in EmbeddingMigrator.initialize
- Fixed automatic_migration test config mocking to use side_effect

Overall coverage: 63.3% (30 files at 100%)

* test: improve kuzu_driver_patched.py coverage to 34.2%

- Added fast variant of execute_query test without @pytest.mark.slow
- Added fast variant of empty results test
- Fixed graphiti_core.graph_queries mocking in fast test
- Renamed slow variant to avoid duplicate test name

kuzu_driver_patched.py: 34.2% coverage (was 22.8%)
Overall coverage: 63.8% (30 files at 100%)

* test: improve backend memory system test coverage to 100%

- Add pragma: no cover comments for unreachable defensive code in config.py,
  memory.py, and kuzu_driver_patched.py (hard-to-test import-time fallbacks)

- Add comprehensive test files:
  - test___init__.py: Tests for lazy import pattern in __init__.py
  - test_graphiti.py: Comprehensive tests for GraphitiMemory class (100% coverage)
  - test_memory.py: Tests for memory.py facade functions
  - test_providers_facade.py: Tests for providers.py re-export facade

- Enhance existing test files:
  - test_config.py: Add test_get_graphiti_status_invalid_config_sets_reason
  - test_kuzu_driver_patched.py: Add tests for create_patched_kuzu_driver
  - test_migrate_embeddings.py: Add tests for migration scenarios

Coverage results:
- 684 tests passing, 7 skipped
- 93.1% overall coverage
- All core memory system files at 100% line coverage:
  - config.py, memory.py, migrate_embeddings.py
  - graphiti.py, kuzu_driver_patched.py, queries.py
  - client.py, search.py, schema.py
  - __init__.py, providers.py

* fix: address CodeRabbit AI review feedback

Fix all 21 test files as reported by CodeRabbit AI:

1. test___init__.py - Replace exec-based dynamic imports with importlib.import_module + getattr
2. test_client.py - Remove unused "result" assignments, remove unused imports
3. test_cross_encoder.py - Update test to actually call create_cross_encoder and assert base_url is preserved
4. test_graphiti_memory.py - Replace /tmp paths with tempfile.mkdtemp(), change datetime.now() to datetime.now(timezone.utc)
5. test_kuzu_driver_patched.py - Add assertions that install_calls and load_calls are non-empty after setup_schema
6. test_memory.py - Remove unused AsyncMock import, fix test to re-raise AssertionError
7. test_migrate_embeddings.py - Remove unused imports, remove duplicate slow tests
8. test_provider_naming.py - Remove sys.path.insert, fix imports properly, add assertions to verify behavior
9. test_providers_facade.py - Make assertion count derive from expected_exports list
10. test_providers_google.py - Remove duplicate slow tests, add assertion for embed_content call, remove unused AsyncMock
11. test_providers_llm_anthropic.py - Replace custom __getattr__ stub with ModuleType
12. test_providers_llm_azure_openai.py - Remove unused sys import
13. test_providers_llm_google.py - Remove unused AsyncMock import
14. test_providers_llm_openai.py - Add assertions for reasoning/verbosity parameters in GPT-5/O1/O3 tests
15. test_providers_llm_openrouter.py - Replace builtins.__import__ with sys.modules patch, remove redundant test
16. test_providers_voyage.py - Clear sys.modules cache before import test, instantiate MagicMocks properly
17. test_queries.py - Remove unused datetime, timezone imports
18. test_schema.py - Fix MAX_RETRIES test consistency (change >= 0 to > 0)
19. test_search.py - Fix non-dict content test, rename unused result to _result, remove unused Path import

* fix: address remaining CodeRabbit AI feedback

Fixed multiple test file issues reported by CodeRabbit AI:
- test_provider_naming.py: Removed excessive print statements
- test___init__.py: Updated lazy import test to handle ImportError gracefully
- test_client.py: Renamed test to match assertion (test_returns_true_if_already_initialized)
- test_cross_encoder.py: Added underscore prefix to unused result variable
- test_kuzu_driver_patched.py: Removed unused imports (re, Mock)
- test_memory.py: Removed unused Path import
- test_migrate_embeddings.py: Updated test to use caplog, attached mock_target_client
- test_providers_facade.py: Fixed EMBEDDING_DIMENSIONS test to check model names not providers
- test_providers_google.py: Added comment to DEFAULT_GOOGLE_EMBEDDING_MODEL test
- test_providers_llm_anthropic.py: Removed dead skipped test
- test_providers_llm_azure_openai.py: Removed unused LLMConfig import
- test_providers_llm_openai.py: Fixed patch path to target graphiti_core module
- test_providers_llm_openrouter.py: Fixed patches for create_openrouter_llm_client imports
- test_queries.py: Parametrized repetitive tests, improved autouse fixture cleanup
- test_search.py: Added underscore prefix to unused local variables

All tests pass (683 passed, 6 skipped) and ruff lint reports no errors.

* fix: address AndyMik90 PR review feedback - code duplication

Fixes:
- Extract repeated sys.modules cleanup into isolate_kuzu_module fixture in test_client.py
- Add _build_sys_modules_dict helper to eliminate 25-line sys.modules patching duplication in test_kuzu_driver_patched.py
- Fix inconsistent pragma in memory.py (lines 95-96 now both marked)
- Update testpaths in pyproject.toml to include "integrations/graphiti/tests"
- Remove duplicate test___init__.py file
- Remove coverage.json from git and add to .gitignore

Code reduction: 598 deletions vs 310 insertions
All 666 tests passing.

* fix: address detailed PR review feedback on test files

Fixes:
- test_client.py: Removed redundant _apply_ladybug_monkeypatch() call, fixed convoluted pywin32 assertion, used call.kwargs directly
- test_cross_encoder.py: Extracted duplicate sys.modules mocking into graphiti_core_mocks fixture
- test_kuzu_driver_patched.py: Parameterized slow tests, split test_execute_query_handles_empty_results, updated build_indices assertions to check SQL strings
- test_memory.py: Fixed fragile import mocking to only raise for graphiti_core imports
- test_migrate_embeddings.py: Created distinct MagicMock instances per iteration to avoid mutation issues
- test_provider_naming.py: Removed print statements and script-entry guard, used explicit config values, strengthened assertions
- test_providers_facade.py: Extracted expected_exports list into module-level constant
- test_providers_google.py: Extracted repeated MagicMock setup into google_genai_mock fixture
- test_providers_llm_openai.py: Replaced tautological assertions with concrete expectations and parametrized slow tests
- search.py: Fixed min_score filtering to handle None scores by normalizing to 0.0

All 667 tests passing.

* fix: address additional detailed PR review feedback

Fixes:
- search.py: Normalized result.score in get_patterns_and_gotchas and get_similar_task_outcomes to handle None values
- test_client.py: Fixed test_returns_false_when_ladybug_unavailable to ensure graphiti_core is present, extracted repeated boilerplate into graphiti_mocks fixture
- test_cross_encoder.py: Added concrete assertion for base_url value, removed original_func indirection
- test_kuzu_driver_patched.py: Added module-level MockKuzuDriver class, added DROP_FTS_INDEX assertion to test_build_indices_with_delete_existing
- test_memory.py: Fixed tautological else branch with concrete assertion
- test_migrate_embeddings.py: Renamed mock configs to match actual roles (current_config, source_config, target_config)
- test_provider_naming.py: Removed unused pytest import and unused embedding_model variable
- test_providers_google.py: Added sys.modules patching to test_google_embedder_init_import_error
- test_providers_llm_openai.py: Fixed patch target path for OpenAIClient to use consuming module's namespace

All 667 tests passing.

* fix: remove duplicate tests and improve test coverage

Fixes:
- test_client.py: Removed duplicate test_initialize_returns_false_on_ladybug_unavailable
- test_client.py: Removed duplicate test_updates_state_with_init_info
- test_cross_encoder.py: Changed unused result variable to _ discard
- test_kuzu_driver_patched.py: Removed duplicate test_execute_query_returns_rows
- test_memory.py: Added pytest.importorskip guards for graphiti_providers package
- test_provider_naming.py: Changed `if dim:` to `if dim is not None:`, converted for-loop to pytest.mark.parametrize

All 668 tests passing.

* fix: address PR review feedback - score normalization and code duplication

- Fix score normalization to correctly handle score of 0 vs None
  - Changed `getattr(result, "score", None) or 0.0` to explicit None check
  - This prevents treating a legitimate score of 0 as None

- Refactor test_client.py to eliminate code duplication
  - Created _make_mock_config() helper function for consistent mock config creation
  - Extended graphiti_mocks fixture with better documentation
  - Converted 15+ tests to use the fixture instead of duplicated boilerplate
  - Removed ~330 net lines of duplicated setup/teardown code

Addresses HIGH and MEDIUM severity issues from PR review.

* fix: address remaining medium severity PR review issues

1. Move standalone test scripts out of tests/ directory
   - Renamed test_graphiti_memory.py -> run_graphiti_memory_test.py
   - Renamed test_ollama_embedding_memory.py -> run_ollama_embedding_test.py
   - These are standalone executable scripts with argparse, not pytest tests

2. Remove fragile pytest_collection_modifyitems filtering
   - No longer needed since standalone scripts moved out of tests/
   - Only keep validator function filtering (legitimate use case)

3. Rename shadowing fixtures in test_graphiti.py
   - temp_spec_dir -> graphiti_test_spec_dir
   - temp_project_dir -> graphiti_test_project_dir
   - mock_config -> mock_graphiti_config
   - mock_state -> mock_graphiti_state
   - Names now indicate intentional difference from conftest fixtures

Addresses 3 MEDIUM severity issues from PR review.

* fix: update test_graphiti_connection for embedded LadybugDB

The function was using outdated FalkorDB configuration attributes
(falkordb_host, falkordb_port, falkordb_password) that no longer exist
on GraphitiConfig. Updated to use embedded LadybugDB via
create_patched_kuzu_driver with db_path instead.

- Replace FalkorDriver with patched KuzuDriver for embedded DB
- Use config.get_db_path() instead of host/port credentials
- Update tests to mock the new driver creation path
- Rename test to reflect new driver type

* fix: address PR review feedback on conftest fixtures and test comments

- Fix mock_config fixture to use actual GraphitiConfig fields (database
  instead of dataset_name, openai_model instead of llm_model, etc.)
- Fix mock_state fixture to use actual GraphitiState fields
- Fix mock_env_vars to use correct env var names (GRAPHITI_DATABASE,
  OPENAI_MODEL, OPENAI_EMBEDDING_MODEL)
- Fix test_search.py comments to accurately describe None->0.0 score
  conversion, add assertion to verify the behavior
- Update pyproject.toml testpaths to include core/workspace/tests
  and remove non-existent 'tests' directory

* fix: address all remaining PR review feedback including LOW severity

MEDIUM fixes:
- Update usage docs in run_graphiti_memory_test.py to reference new filename
- Update usage docs in run_ollama_embedding_test.py to reference new filename

LOW fixes:
- Fix get_relevant_context docstring: add min_score param, correct
  include_project_context description (works in SPEC mode, not PROJECT mode)
- Make mock_embedder fixture deterministic using [0.1] * 1536 instead of
  random values for reproducibility
- Add test coverage for None score handling in get_similar_task_outcomes
  and get_patterns_and_gotchas methods

---------

Co-authored-by: StillKnotKnown <stillknotknown@users.noreply.github.com>
2026-02-12 11:40:54 +02:00

182 lines
7.0 KiB
Python

"""
Unit tests for Ollama LLM provider.
Tests cover:
- create_ollama_llm_client factory function
- ProviderNotInstalled exception handling
- ProviderError for missing configuration
"""
from unittest.mock import MagicMock, patch
import pytest
from integrations.graphiti.providers_pkg.exceptions import (
ProviderError,
ProviderNotInstalled,
)
from integrations.graphiti.providers_pkg.llm_providers.ollama_llm import (
create_ollama_llm_client,
)
# =============================================================================
# Test create_ollama_llm_client
# =============================================================================
class TestCreateOllamaLLMClient:
"""Test create_ollama_llm_client factory function."""
@pytest.fixture
def mock_config(self):
"""Create a mock GraphitiConfig."""
config = MagicMock()
config.ollama_llm_model = "llama3.2"
config.ollama_base_url = "http://localhost:11434"
return config
@pytest.mark.slow
def test_create_ollama_llm_client_success(self, mock_config):
"""Test create_ollama_llm_client returns client with valid config."""
mock_client = MagicMock()
with patch(
"integrations.graphiti.providers_pkg.llm_providers.ollama_llm.OpenAIGenericClient",
return_value=mock_client,
):
result = create_ollama_llm_client(mock_config)
assert result == mock_client
def test_create_ollama_llm_client_success_fast(self, mock_config):
"""Fast test for create_ollama_llm_client success path."""
mock_llm_client = MagicMock()
# Create the config mock
mock_config_module = MagicMock()
mock_config_module.LLMConfig = MagicMock
# Mock the graphiti_core imports
with patch.dict(
"sys.modules",
{
"graphiti_core": MagicMock(),
"graphiti_core.llm_client": MagicMock(),
"graphiti_core.llm_client.config": mock_config_module,
"graphiti_core.llm_client.openai_generic_client": MagicMock(),
},
):
from graphiti_core.llm_client.openai_generic_client import (
OpenAIGenericClient,
)
OpenAIGenericClient.return_value = mock_llm_client
result = create_ollama_llm_client(mock_config)
# Verify the client was created and returned
OpenAIGenericClient.assert_called_once()
assert result == mock_llm_client
def test_create_ollama_llm_client_missing_model(self, mock_config):
"""Test create_ollama_llm_client raises ProviderError for missing model."""
mock_config.ollama_llm_model = None
with pytest.raises(ProviderError) as exc_info:
create_ollama_llm_client(mock_config)
assert "OLLAMA_LLM_MODEL" in str(exc_info.value)
def test_create_ollama_llm_client_import_error(self, mock_config):
"""Test create_ollama_llm_client raises ProviderNotInstalled on ImportError."""
import builtins
original_import = builtins.__import__
def mock_import(name, *args, **kwargs):
if name.startswith("graphiti_core.llm_client"):
raise ImportError("graphiti-core not installed")
return original_import(name, *args, **kwargs)
with patch("builtins.__import__", side_effect=mock_import):
with pytest.raises(ProviderNotInstalled) as exc_info:
create_ollama_llm_client(mock_config)
assert "graphiti-core" in str(exc_info.value)
@pytest.mark.slow
def test_create_ollama_llm_client_base_url_without_v1(self, mock_config):
"""Test create_ollama_llm_client appends /v1 to base URL if missing."""
mock_config.ollama_base_url = "http://localhost:11434"
mock_client = MagicMock()
with patch(
"integrations.graphiti.providers_pkg.llm_providers.ollama_llm.LLMConfig",
) as mock_config_class:
with patch(
"integrations.graphiti.providers_pkg.llm_providers.ollama_llm.OpenAIGenericClient",
return_value=mock_client,
):
create_ollama_llm_client(mock_config)
# Verify base_url has /v1 appended
call_kwargs = mock_config_class.call_args.kwargs
assert call_kwargs["base_url"] == "http://localhost:11434/v1"
@pytest.mark.slow
def test_create_ollama_llm_client_base_url_with_v1(self, mock_config):
"""Test create_ollama_llm_client doesn't duplicate /v1 in base URL."""
mock_config.ollama_base_url = "http://localhost:11434/v1"
mock_client = MagicMock()
with patch(
"integrations.graphiti.providers_pkg.llm_providers.ollama_llm.LLMConfig",
) as mock_config_class:
with patch(
"integrations.graphiti.providers_pkg.llm_providers.ollama_llm.OpenAIGenericClient",
return_value=mock_client,
):
create_ollama_llm_client(mock_config)
# Verify base_url is not duplicated
call_kwargs = mock_config_class.call_args.kwargs
assert call_kwargs["base_url"] == "http://localhost:11434/v1"
@pytest.mark.slow
def test_create_ollama_llm_client_base_url_with_trailing_slash(self, mock_config):
"""Test create_ollama_llm_client handles trailing slash correctly."""
mock_config.ollama_base_url = "http://localhost:11434/"
mock_client = MagicMock()
with patch(
"integrations.graphiti.providers_pkg.llm_providers.ollama_llm.LLMConfig",
) as mock_config_class:
with patch(
"integrations.graphiti.providers_pkg.llm_providers.ollama_llm.OpenAIGenericClient",
return_value=mock_client,
):
create_ollama_llm_client(mock_config)
# Verify trailing slash is handled
call_kwargs = mock_config_class.call_args.kwargs
assert call_kwargs["base_url"] == "http://localhost:11434/v1"
@pytest.mark.slow
def test_create_ollama_llm_client_passes_config_correctly(self, mock_config):
"""Test create_ollama_llm_client passes config values correctly."""
mock_config.ollama_llm_model = "qwen2.5"
mock_client = MagicMock()
with patch(
"integrations.graphiti.providers_pkg.llm_providers.ollama_llm.LLMConfig",
) as mock_config_class:
with patch(
"integrations.graphiti.providers_pkg.llm_providers.ollama_llm.OpenAIGenericClient",
return_value=mock_client,
):
create_ollama_llm_client(mock_config)
# Verify LLMConfig was called with correct arguments
call_kwargs = mock_config_class.call_args.kwargs
assert call_kwargs["api_key"] == "ollama"
assert call_kwargs["model"] == "qwen2.5"
assert call_kwargs["small_model"] == "qwen2.5"