Guidelines for writing and running tests in the Agent Framework Python codebase. Use this when creating, modifying, or running tests.
We strive for at least 85% test coverage across the codebase, with a focus on core packages and critical paths. Tests should be fast, reliable, and maintainable.
When adding new code, check that the relevant sections of the codebase are covered by tests, and add new tests as needed. When modifying existing code, update or add tests to cover the changes.
We run tests in two stages, for a PR each commit is tested with unit tests only (using -m "not integration"), and the full suite including integration tests is run when merging.
# Run tests for all packages in parallel
uv run poe test
# Run tests for a specific workspace package
uv run poe test -P core
# Run all selected tests in a single pytest invocation
uv run poe test -A
# With coverage
uv run poe test -A -C
uv run poe test -P core -C
# Run only unit tests (exclude integration tests)
uv run poe test -A -m "not integration"
# Run only integration tests
uv run poe test -A -m integration
Direct package execution still works when you need it:
uv run --directory packages/core poe test
asyncio_mode = "auto" is enabled — do NOT use @pytest.mark.asyncio, but do mark tests with async def and use await for async callsimportlib for cross-package isolationpytest-xdist (-n auto --dist worksteal) in their poe test task. The aggregate uv run poe test -A sweep also uses xdist across the selected packages.Test directories must NOT contain __init__.py files.
Non-core packages must place tests in a uniquely-named subdirectory:
packages/anthropic/
├── tests/
│ └── anthropic/ # Unique subdirectory matching package name
│ ├── conftest.py
│ └── test_client.py
Core package can use tests/ directly with topic subdirectories:
packages/core/
├── tests/
│ ├── conftest.py
│ ├── core/
│ │ └── test_agents.py
│ └── openai/
│ └── test_client.py
conftest.py for shared fixtures within a test directorymapper, test_request, mock_clienttest_ are test files — do not use this prefix for helpersconftest.py for shared utilitiesIntegration tests require external services (OpenAI, Azure, etc.) and are controlled by three markers:
@pytest.mark.flaky — marks the test as potentially flaky since it depends on external services@pytest.mark.integration — used for test selection, so integration tests can be included/excluded with -m integration / -m "not integration"@skip_if_..._integration_tests_disabled decorator — skips the test when the required API keys or service endpoints are missingAll three markers must be applied to every new integration test:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_openai_chat_completion() -> None:
...
For test files where all tests are integration tests (e.g., Azure Functions, Durable Task), use the module-level pytestmark list:
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("01_single_agent"),
pytest.mark.usefixtures("function_app_for_test"),
]
The merge CI workflow (python-merge-tests.yml) splits integration tests into parallel jobs by provider with change-based detection:
packages/core/agent_framework/openai/ or core infrastructure changespackages/core/agent_framework/azure/ or core changespackages/foundry/ or core changesCore infrastructure changes (e.g., _agents.py, _types.py) trigger all integration test jobs. Scheduled and manual runs always execute all jobs.
Two workflow files define the same set of parallel test jobs:
python-merge-tests.yml — runs on PRs, merge queue, schedule, and manual dispatch. Uses path-based change detection to skip unaffected integration jobs.python-integration-tests.yml — called from the manual integration test orchestrator (integration-tests-manual.yml). Always runs all jobs (no path filtering).These workflows must be kept in sync. When you add, remove, or modify a test job, update both files. The job structure, pytest commands, and xdist flags should match between them. The only difference is that python-merge-tests.yml has path filters and conditional job execution, while python-integration-tests.yml does not.
When adding integration tests for a new provider package, you must update both python-merge-tests.yml and python-integration-tests.yml:
paths-filter job in python-merge-tests.yml so the CI knows which file changes should trigger those tests.python-tests-misc-integration job, or create a dedicated job if the provider:
The python-tests-misc-integration job is intended for small integration test suites that don't need dedicated infrastructure. When a provider's integration tests grow large or gain special requirements, split them out into their own job (like python-tests-functions was split out for Azure Functions + Durable Task).