plugin

agent_cover.plugin

Pytest plugin for Agent Coverage integration.

This module implements the pytest hooks necessary to integrate Agent Coverage into the test execution flow. It orchestrates the entire lifecycle: 1. Setup: Parses CLI options (--agent-cov, etc.). 2. Instrumentation: Initializes the AgentCoverage manager before tests start. 3. Teardown: Generates reports (JSON/XML/HTML) after tests finish.

CLI Options

The plugin adds the following flags to the pytest command line:

Option	Description	Default
--agent-cov	Enable the plugin. Must be present to run instrumentation.	False
--agent-cov-html=<dir>	Directory to generate the static HTML report.	None
--agent-cov-xml=<file>	Path to generate the Cobertura XML report (CI/CD).	None
--agent-cov-json=<file>	Path to generate the raw JSON data dump.	agent_coverage.json
--agent-source-dir=<dir>	Root directory to scan for python modules (Discovery phase).	Current Working Dir
--prompt-prefixes=<list>	Comma-separated list of variable prefixes to scan as Raw Strings.	None
--prompt-suffixes=<list>	Comma-separated list of variable suffixes to scan as Raw Strings.	None
--agent-cov-verbose	Enable debug logging specifically for AgentCover internals.	False

Hooks Implemented

• pytest_addoption: Registers custom CLI flags.
• pytest_configure: Starts instrumentation and scans for static definitions.
• pytest_sessionfinish: Generates reports using data collected during the run.
• pytest_unconfigure: Clean up and stop instrumentation.

Classes

Functions

pytest_addoption(parser)

Registers the command-line options for agent coverage.

These options allow users to enable coverage, specify report paths, and customize heuristic scanning without changing code.

Parameters:

Name	Type	Description	Default
parser		The pytest parser object.	required

Source code in src/agent_cover/plugin.py

def pytest_addoption(parser):
    """Registers the command-line options for agent coverage.

    These options allow users to enable coverage, specify report paths, and
    customize heuristic scanning without changing code.

    Args:
        parser: The pytest parser object.
    """
    group = parser.getgroup("agent-cover", "Agent Coverage Reporting")

    group.addoption(
        "--agent-cov",
        action="store_true",
        default=False,
        help="Enable Agent Coverage reporting.",
    )
    group.addoption(
        "--agent-cov-json",
        action="store",
        default="agent_coverage.json",
        help="Path to save the JSON report.",
    )
    group.addoption(
        "--agent-cov-xml",
        action="store",
        default=None,
        help="Path to save the Cobertura XML report.",
    )
    group.addoption(
        "--agent-cov-html",
        action="store",
        default=None,
        help="Directory where to save the HTML report.",
    )
    group.addoption(
        "--agent-source-dir",
        action="store",
        default=None,
        help="Directory to scan for python modules (discovery).",
    )
    group.addoption(
        "--prompt-prefixes",
        action="store",
        default=None,
        help="Comma-separated list of variable prefixes to treat as prompts (e.g. 'MY_PROMPT_,TEXT_').",
    )
    group.addoption(
        "--prompt-suffixes",
        action="store",
        default=None,
        help="Comma-separated list of variable suffixes (e.g. '_PROMPT,_TEXT').",
    )

    group.addoption(
        "--agent-cov-verbose",
        action="store_true",
        default=False,
        help="Enable debug logging specifically for AgentCover.",
    )

pytest_configure(config)

Initializes the AgentCoverage manager, loads the config, and applies patches.

This hook runs before any test is executed. It is responsible for: 1. Loading agent-cover.yaml. 2. Starting the Instrumentation. 3. Performing an initial static scan of the codebase to find Prompts and Tools.

Parameters:

Name	Type	Description	Default
config		The pytest configuration object.	required

Source code in src/agent_cover/plugin.py

def pytest_configure(config):
    """Initializes the AgentCoverage manager, loads the config, and applies patches.

    This hook runs **before** any test is executed. It is responsible for:
    1.  Loading `agent-cover.yaml`.
    2.  Starting the [Instrumentation][agent_cover.manager.AgentCoverage.start].
    3.  Performing an initial static scan of the codebase to find Prompts and Tools.

    Args:
        config: The pytest configuration object.
    """
    # VERBOSE MANAGEMENT
    if config.getoption("--agent-cov-verbose"):
        if config.option.capture != "no":
            config.option.capture = "no"

        # Configure your package's ROOT logger
        pkg_logger = logging.getLogger("agent_cover")
        pkg_logger.setLevel(logging.DEBUG)

        # Detach the logger from the parent hierarchy.
        # This prevents Pytest or Promptflow from blocking your DEBUG logs.
        pkg_logger.propagate = False

        # Clean up old/inherited handlers
        if pkg_logger.hasHandlers():
            pkg_logger.handlers.clear()

        # Create a new handler on STDOUT (safe with -s flag)
        handler = logging.StreamHandler(sys.stdout)
        handler.setLevel(logging.DEBUG)

        formatter = logging.Formatter("[\033[96mAgentCover\033[0m] %(message)s")
        handler.setFormatter(formatter)

        pkg_logger.addHandler(handler)

        pkg_logger.debug("Logger configured and handler attached successfully.")

    if not config.getoption("--agent-cov"):
        return

    logger.debug("Plugin enabled via CLI.")

    # --- 1. Environment Setup and Configuration ---
    source_dir = config.getoption("--agent-source-dir") or os.getcwd()
    abs_source = os.path.abspath(source_dir)

    if abs_source not in sys.path:
        sys.path.insert(0, abs_source)

    # Load YAML configuration
    agent_config = load_config(abs_source)

    # Prefixes/Suffixes Configuration (Raw Strings)
    prefixes_str = config.getoption("--prompt-prefixes")
    if prefixes_str:
        set_custom_prefixes(prefixes_str.split(","))

    suffixes_str = config.getoption("--prompt-suffixes")
    if suffixes_str:
        set_custom_suffixes(suffixes_str.split(","))

    # --- 2. Start Manager ---
    cov_manager = AgentCoverage(config=agent_config)

    try:
        cov_manager.start()

        # Save the manager in the stash for later retrieval
        config.stash[MANAGER_KEY] = cov_manager

        logger.debug("Instrumentation applied via Manager.")
    except Exception as e:
        logger.critical(f"Critical Error starting instrumentation: {e}", exc_info=True)

    # --- 3. Initial Static Scan (PromptFlow) ---
    try:
        scan_promptflow_definitions(root_path=abs_source, registry=cov_manager.registry)
    except Exception as e:
        logger.warning(f"PromptFlow scan warning: {e}", exc_info=True)

pytest_sessionfinish(session, exitstatus)

Generates reports at the end of the session.

This hook runs after all tests have completed. It: 1. Runs a final discovery sweep (to catch dynamically defined objects). 2. Extracts execution data from the AgentRegistry. 3. Calls the reporting modules to generate JSON, XML, and HTML files.

Parameters:

Name	Type	Description	Default
session		The pytest session object.	required
exitstatus		The exit status of the test run.	required

Source code in src/agent_cover/plugin.py

def pytest_sessionfinish(session, exitstatus):
    """Generates reports at the end of the session.

    This hook runs **after** all tests have completed. It:
    1.  Runs a final discovery sweep (to catch dynamically defined objects).
    2.  Extracts execution data from the [`AgentRegistry`][agent_cover.registry.AgentRegistry].
    3.  Calls the reporting modules to generate JSON, XML, and HTML files.

    Args:
        session: The pytest session object.
        exitstatus: The exit status of the test run.
    """
    manager = session.config.stash.get(MANAGER_KEY, None)
    if not manager:
        return

    source_dir = session.config.getoption("--agent-source-dir") or os.getcwd()

    # --- 1. Final Discovery and Static Scan ---
    try:
        logger.debug("Running final discovery...")
        discover_repository_modules(root_path=os.path.abspath(source_dir))

        scan_static_definitions(registry=manager.registry, config=manager.config)
    except Exception as e:
        logger.warning(f"Final discovery warning: {e}", exc_info=True)

    # --- 2. Data Extraction ---
    registry = manager.registry
    config_obj = manager.config

    definitions = registry.definitions
    executions = registry.executions
    decision_hits = registry.decision_hits

    # --- 3. Common Timestamp Setup ---
    # Generate a single timestamp for consistency across all reports
    common_timestamp = get_timestamp()

    # --- 4. JSON Report Generation ---
    json_path = session.config.getoption("--agent-cov-json")
    _save_json_report(
        definitions,
        executions,
        config_obj,
        decision_hits,
        json_path,
        timestamp=common_timestamp,
    )

    # --- 5. XML Report Generation ---
    xml_path = session.config.getoption("--agent-cov-xml")
    if xml_path:
        generate_cobertura_xml(
            definitions,
            executions,
            config_obj,
            decision_hits,
            xml_path,
            timestamp=common_timestamp,
        )
        logger.debug(f"XML Report generated: {xml_path}")

    # --- 6. HTML Report Generation ---
    html_dir = session.config.getoption("--agent-cov-html")
    if html_dir:
        report_path = generate_html_report(
            definitions,
            executions,
            html_dir,
            decision_config=config_obj,
            decision_hits=decision_hits,
            timestamp=common_timestamp,
        )
        logger.debug(f"HTML Report generated: {report_path}/index.html")

pytest_terminal_summary(terminalreporter, exitstatus, config)

Outputs a coverage summary to the terminal.

Parameters:

Name	Type	Description	Default
terminalreporter		The pytest terminal reporter object.	required
exitstatus		The exit status of the test run.	required
config		The pytest configuration object.	required

Source code in src/agent_cover/plugin.py

def pytest_terminal_summary(terminalreporter, exitstatus, config):
    """Outputs a coverage summary to the terminal.

    Args:
        terminalreporter: The pytest terminal reporter object.
        exitstatus: The exit status of the test run.
        config: The pytest configuration object.
    """
    manager = config.stash.get(MANAGER_KEY, None)
    if not manager:
        return

    registry = manager.registry
    agent_config = manager.config

    if not registry.definitions and (not agent_config or not agent_config.decisions):
        return

    definitions = registry.definitions
    executions = registry.executions
    decision_hits = registry.decision_hits

    prompts = {
        k: v
        for k, v in definitions.items()
        if v.get("kind") == "PROMPT" or v.get("type") == "PROMPT"
    }

    tools = {
        k: v
        for k, v in definitions.items()
        if v.get("kind") == "TOOL" or v.get("type") == "TOOL"
    }

    if prompts:
        _print_coverage_table(
            terminalreporter, "Prompt Coverage", prompts, executions, color="purple"
        )

    if tools or (agent_config and agent_config.decisions):
        _print_coverage_table(
            terminalreporter,
            "Completion Coverage",
            tools,
            executions,
            extra_decisions=agent_config,
            decision_hits=decision_hits,
            color="yellow",
        )

pytest_unconfigure(config)

Cleans up resources when the pytest session ends.

Parameters:

Name	Type	Description	Default
config		The pytest configuration object.	required

Source code in src/agent_cover/plugin.py

def pytest_unconfigure(config):
    """Cleans up resources when the pytest session ends.

    Args:
        config: The pytest configuration object.
    """
    manager = config.stash.get(MANAGER_KEY, None)
    if manager:
        logger.debug("Stopping instrumentation (Cleanup)...")
        manager.stop()