User Guide & Integrations

AgentCover is designed to fit into your existing workflow with zero to minimal code changes. Depending on your environment, you can choose one of three integration strategies.

🛠 Usage Strategies

1. Pytest Plugin (Recommended)

The easiest way to use AgentCover during local development or in CI/CD. Simply add the --agent-cov flag to your pytest command.

pytest --agent-cov --agent-cov-html=reports/

Best for: Unit and integration tests using LangChain or LlamaIndex.
Auto-discovery: It automatically scans your repo for PROMPT_ variables and Pydantic models.

2. CLI Wrapper (For Distributed/Multi-process)

Some frameworks, like Microsoft PromptFlow, execute logic in isolated worker processes. Standard instrumentation won't reach them. Use the run command:

agent-cover run --source-dir ./src -- pf run create --flow ./my-flow

How it works: It injects a bootstrap script via PYTHONPATH that initializes instrumentation inside every child process spawned by the target command.
Aggregation: Once the command finishes, it merges all fragments from different workers into one consolidated report.

3. SDK (Manual Control)

For notebooks, custom scripts, or non-standard test runners.

from agent_cover.manager import AgentCoverage

with AgentCoverage() as cov:
    # Your agent code here
    pass
# Coverage is automatically calculated and patches removed here

🔌 Supported Frameworks & API

AgentCover patches these libraries dynamically. We support multiple versions via a "best-effort" strategy.

🕵️ Agents Orchestrators

Module	Class	Intercepted Methods
`langchain.agents`	`AgentExecutor` (min: 0.1.0)	`invoke` (sync), `stream` (sync_gen), `astream` (async_gen)
`langchain.chains`	`LLMChain`	`invoke` (sync)
`llama_index.core.agent`	`AgentRunner`	`chat` (sync), `achat` (async)
`llama_index.agent`	`AgentRunner`	`chat` (sync), `achat` (async)
`llama_index.agent.runner.base`	`AgentRunner`	`chat` (sync), `achat` (async)

🤖 LLM Providers

Module	Class	Intercepted Methods
`openai.resources.chat.completions.completions`	`Completions.create`	Patch Type: `class_method`
`openai.resources.chat.completions`	`Completions.create` (min: 1.0.0)	Patch Type: `class_method`
`openai.resources.completions.completions`	`Completions.create`	Patch Type: `class_method`
`openai.resources.completions`	`Completions.create`	Patch Type: `class_method`
`openai`	`ChatCompletion.create` (max: 1.0.0)	Patch Type: `function`
`openai`	`Completion.create` (max: 1.0.0)	Patch Type: `function`

📝 Prompts Frameworks

Module	Class	Intercepted Methods
`langchain.prompts`	`PromptTemplate`	`format`
`langchain.prompts`	`ChatPromptTemplate`	`format_messages`, `format`
`langchain_core.prompts`	`PromptTemplate`	`format`
`langchain_core.prompts`	`ChatPromptTemplate`	`format_messages`, `format`
`llama_index.core.prompts`	`PromptTemplate`	`format`
`llama_index.core.prompts`	`SelectorPromptTemplate`	`format`
`llama_index.core.prompts`	`ChatPromptTemplate`	`format`
`llama_index.prompts`	`PromptTemplate`	`format`
`llama_index.prompts`	`SelectorPromptTemplate`	`format`
`llama_index.prompts`	`ChatPromptTemplate`	`format`

🛠️ Tools Frameworks

Module	Class	Intercepted Methods
`langchain_core.tools`	`BaseTool`	`invoke`, `ainvoke`, `run`, `arun`
`langchain.tools`	`BaseTool`	`invoke`, `run`
`langchain_core.tools`	`StructuredTool`	`_run`, `_arun`
`langchain.tools`	`StructuredTool`	`_run`, `_arun`
`llama_index.core.tools`	`BaseTool`	`__call__`, `call`, `acall`
`llama_index.core.tools`	`FunctionTool`	`__call__`, `call`
`llama_index.core.tools.function_tool`	`FunctionTool`	`__call__`, `call`
`llama_index.core.tools.types`	`BaseTool`	`__call__`, `call`, `acall`
`llama_index.tools`	`BaseTool`	`__call__`, `call`, `acall`
`llama_index.tools`	`FunctionTool`	`__call__`, `call`
`llama_index.tools.function_tool`	`FunctionTool`	`__call__`, `call`
`llama_index.tools.types`	`BaseTool`	`__call__`, `call`, `acall`

🌊 Microsoft PromptFlow

Module	Class	Intercepted Methods
`promptflow.tools.common`	`render_jinja_template`	Patch Type: `render`
`promptflow.core.tools.common`	`render_jinja_template`	Patch Type: `render`
`promptflow.core`	`tool`	Patch Type: `decorator`
`promptflow.tools.tool`	`tool`	Patch Type: `decorator`

⚙️ Configuration (`agent-cover.yaml`)

Define your business rules to track Decision Coverage.

decisions:
  - id: "refund_policy"
    target_field: "refund_status"
    expected_values: ["APPROVED", "REJECTED", "NEEDS_REVIEW"]