Skip to content

[P0][B] Build complete test suite targeting 100% line+branch coverage #9

Description

@POWDER-RANGER

Problem

No tests/ directory exists. CI currently runs pytest tests/ ... || echo "Tests directory not found..." — meaning CI silently passes even with zero tests. There is literally no test coverage of any library module.

Scope definition for "100%":

  • 100% line + branch coverage of oblisk/agents, oblisk/core, oblisk/vault
  • Exclusions allowed only for:
    • if __name__ == "__main__": guards
    • # pragma: no cover with documented justification
    • Unreachable defensive raise NotImplementedError in ABCs
  • oblisk_cli.py and build_oblisk.py targeted at 80%+ (not 100%)

Test File Structure

tests/
├── __init__.py
├── conftest.py              # shared fixtures
├── unit/
│   ├── __init__.py
│   ├── test_agent.py           # Agent class full coverage
│   ├── test_agent_manager.py   # AgentManager full coverage
│   ├── test_task.py            # Task dataclass + status
│   ├── test_governance_engine.py
│   ├── test_symbolic_planner.py
│   ├── test_vault_crypto.py    # (defined in issue #8)
│   └── test_vault.py           # (defined in issue #8)
├── integration/
│   ├── __init__.py
│   ├── test_agent_vault_integration.py
│   └── test_governance_agent_integration.py
└── smoke/
    └── test_examples.py        # runs all examples as subprocesses

Shared Fixtures (conftest.py)

# tests/conftest.py
import pytest
from oblisk.vault import Vault
from oblisk.agents import Agent, AgentManager
from oblisk.core import GovernanceEngine, SymbolicPlanner

VAULT_KEY = b'x' * 32  # test-only key

@pytest.fixture
def vault():
    return Vault(key=VAULT_KEY)

@pytest.fixture
def agent(vault):
    a = Agent(name="test-agent", vault=vault)
    yield a
    if a.status == "running":
        a.stop()

@pytest.fixture
def agent_manager(vault):
    return AgentManager(vault=vault)

@pytest.fixture
def governance(vault):
    return GovernanceEngine(vault=vault)

@pytest.fixture
def planner():
    return SymbolicPlanner()

Unit Test Specs

tests/unit/test_agent.py

class TestAgentInitialization:
    def test_default_status_is_idle(self, agent): ...
    def test_name_stored_correctly(self, agent): ...
    def test_invalid_name_raises(self): ...  # empty string, None
    def test_vault_reference_stored(self, agent, vault): ...

class TestAgentLifecycle:
    def test_start_changes_status_to_running(self, agent): ...
    def test_stop_changes_status_to_idle(self, agent): ...
    def test_start_already_running_raises(self, agent): ...
    def test_stop_already_idle_raises(self, agent): ...
    def test_double_stop_raises(self, agent): ...

class TestAgentTaskExecution:
    def test_execute_task_returns_result_dict(self, agent): ...
    def test_assign_task_returns_task_id(self, agent): ...
    def test_wait_for_task_returns_completed_task(self, agent): ...
    def test_wait_for_task_timeout_raises(self, agent): ...
    def test_task_failure_captured_in_result(self, agent): ...
    def test_execute_nonexistent_task_raises(self, agent): ...

class TestAgentState:
    def test_state_alias_equals_status(self, agent): ...
    def test_state_transitions_correctly(self, agent): ...

tests/unit/test_agent_manager.py

class TestAgentManagerCreation:
    def test_empty_manager_has_zero_agents(self, agent_manager): ...
    def test_register_adds_agent(self, agent_manager, agent): ...
    def test_register_duplicate_name_raises(self, agent_manager, agent): ...
    def test_unregister_removes_agent(self, agent_manager, agent): ...
    def test_unregister_nonexistent_raises(self, agent_manager): ...

class TestAgentManagerLifecycle:
    def test_start_all_starts_each_agent(self, agent_manager, vault): ...
    def test_stop_all_stops_each_agent(self, agent_manager, vault): ...
    def test_get_agent_by_name(self, agent_manager, agent): ...
    def test_get_nonexistent_agent_returns_none(self, agent_manager): ...

class TestAgentManagerTaskDistribution:
    def test_distribute_task_to_idle_agent(self, agent_manager, vault): ...
    def test_distribute_when_all_busy_raises(self, agent_manager, vault): ...
    def test_list_agents_returns_all(self, agent_manager, vault): ...
    def test_list_agents_empty(self, agent_manager): ...

class TestAgentManagerAudit:
    def test_audit_log_records_task_assignments(self, agent_manager, vault): ...
    def test_audit_log_empty_on_new_manager(self, agent_manager): ...

tests/unit/test_governance_engine.py

class TestGovernancePolicies:
    def test_evaluate_allowed_action_returns_true(self, governance): ...
    def test_evaluate_denied_action_returns_false(self, governance): ...
    def test_add_policy_persists(self, governance): ...
    def test_remove_policy(self, governance): ...
    def test_list_policies(self, governance): ...
    def test_duplicate_policy_raises(self, governance): ...

class TestGovernanceAudit:
    def test_audit_log_records_evaluation(self, governance): ...
    def test_audit_log_includes_timestamp(self, governance): ...
    def test_audit_log_includes_agent_id(self, governance): ...
    def test_export_audit_log_json(self, governance, tmp_path): ...
    def test_export_audit_log_creates_file(self, governance, tmp_path): ...
    def test_audit_log_empty_on_init(self, governance): ...

class TestGovernanceEdgeCases:
    def test_evaluate_with_no_policies_defaults_to_allow(self, governance): ...
    def test_evaluate_unknown_agent(self, governance): ...
    def test_policy_priority_order(self, governance): ...  # deny > allow

tests/unit/test_symbolic_planner.py

class TestPlanCreation:
    def test_simple_plan_succeeds(self, planner): ...
    def test_plan_returns_list_of_steps(self, planner): ...
    def test_plan_with_no_goal_raises(self, planner): ...
    def test_plan_with_unsatisfiable_goal_raises(self, planner): ...
    def test_plan_step_has_required_fields(self, planner): ...  # name, preconditions, effects

class TestPlanExecution:
    def test_execute_plan_runs_all_steps(self, planner): ...
    def test_execute_plan_tracks_state_changes(self, planner): ...
    def test_execute_returns_final_state(self, planner): ...
    def test_step_failure_aborts_plan(self, planner): ...

class TestPlannerState:
    def test_initial_state_empty(self, planner): ...
    def test_add_fact_persists(self, planner): ...
    def test_remove_fact_persists(self, planner): ...
    def test_query_fact_returns_boolean(self, planner): ...
    def test_reset_clears_state(self, planner): ...

tests/integration/test_agent_vault_integration.py

class TestAgentVaultIntegration:
    def test_agent_stores_result_in_vault(self, agent, vault): ...
    def test_agent_reads_config_from_vault(self, agent, vault): ...
    def test_vault_key_unavailable_raises(self, agent): ...
    def test_multiple_agents_share_vault(self, vault): ...

tests/smoke/test_examples.py

import subprocess
import sys
import pytest

@pytest.mark.parametrize("example", [
    "examples/simple_agent.py",
    "examples/vault_demo.py",
    "examples/governance_demo.py",
    "examples/multi_agent.py",
])
def test_example_runs_without_error(example):
    result = subprocess.run(
        [sys.executable, example],
        capture_output=True,
        timeout=30
    )
    assert result.returncode == 0, f"{example} failed:\n{result.stderr.decode()}"

CI Coverage Configuration (pyproject.toml)

[tool.pytest.ini_options]
testpaths = ["tests"]
asyncio_mode = "auto"
addopts = """
    --cov=oblisk
    --cov-branch
    --cov-report=term-missing
    --cov-report=xml:coverage.xml
    --cov-report=html:htmlcov
    --cov-fail-under=100
    -v
"""

[tool.coverage.run]
branch = true
source = ["oblisk"]
omit = [
    "oblisk/__main__.py",
    "**/migrations/**",
]

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "if __name__ == .__main__.:",
    "raise NotImplementedError",
    "@(abc.)?abstractmethod",
]
fail_under = 100
show_missing = true

Acceptance Criteria

  • tests/ directory exists with structure above
  • pytest runs successfully (no import errors, no fixture failures)
  • pytest --cov --cov-branch --cov-fail-under=100 exits 0
  • ✅ Coverage XML uploaded to Codecov (see CI issue [P0][B3+D1] Harden CI: real coverage enforcement, Ruff, MyPy, pre-commit #10)
  • ✅ All unit test classes populated with working test methods
  • ✅ Integration tests pass with real vault + agent interaction
  • ✅ Smoke tests run all example scripts and pass
  • ✅ CI fails (not just warns) if coverage drops below 100%
  • ✅ Coverage badge in README shows 100%

Test Count Estimate

~70 unit tests, ~8 integration tests, ~4 smoke tests = ~82 total tests

Dependencies

Requires: #6 (package layout), #7 (Task API), #8 (vault crypto)
Blocks: #10 (CI hardening needs real tests to enforce)
Estimated effort: 12-16 hours

Priority: P0 — CI is currently meaningless without this
Labels: testing, coverage, P0

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions