Save and reopen executions

Direct SDK use keeps data in memory unless you choose a persistent store. The CLI chooses SQLite automatically.

CLI-managed history

m3 test -- tests/test_m3_starter.py

This writes .m3/executions.sqlite and a feedback bundle under .m3/reports/<run-id>/feedback.json. The CLI prints the run ID and report path. The CLI prints M3 feedback: .m3/reports/RUN_ID/feedback.json. The bundle's top-level fields include summary, limitations, and evaluation_stats. summary.executions counts executions, while summary.failures counts failed or errored cases plus collection errors. summary.test_outcome_counts and summary.effective_verdict_counts include xfailed beside skipped, and summary.xfailed_tests counts expected failures. summary.skipped_tests counts only genuine skips. evaluation_stats groups explicit saved evaluations by name.

Each tests[] entry has node_id, outcome, verdict, effective_verdict, tool_result, and execution_ids. outcome is the pytest case outcome. verdict distinguishes assertion, protocol, setup, teardown, and other results. effective_verdict applies the required- evaluation policy without relabeling outcome. All three are xfailed for a pytest expected failure: the test ran and failed as expected. This differs from skipped, where the test did not run. An xfailed test is not a failure and does not need attention. A test with an xfail marker also has a top-level xfail_reason, which is "" for a bare @pytest.mark.xfail. Runs saved before this field existed report skipped xfail tests as xfailed.

The failures[] array includes failed pytest cases copied from tests[]. Those entries include node_id, outcome, verdict, effective_verdict, and execution_ids; they do not have kind or execution_id. Required evaluation failures may be attached to a failed case under evaluations[]. When they appear as separate entries, they have kind: "evaluation", evaluation_id, execution_id, evaluator, case_id, and status, plus evaluation details such as required, score, rationale, and message. Detached evaluation entries have execution_id: null. Collection and worker failures have kind: "collection" or kind: "worker" plus fields from their manifest reports; persistence failures have kind: "persistence" and message. These sources do not necessarily have node_id, verdict, or execution_ids, so inspect each entry's source-specific fields rather than filtering on one set of fields. An executions[].outcome value of completed describes lifecycle only.

File maps are relative to the report directory. Each maps an ID to a file path; a map is {} when the run has nothing of that kind:

Map	Key	File holds
execution_files	execution ID	The execution report (executions/). Large payload values that the trace also holds are replaced by __m3_ref__ markers; see below.
spec_files	execution ID	The execution spec, when one was recorded (specs/).
catalog_files	execution ID	Tool-catalog versions observed through tools/list (catalogs/); their tool sets are shared files under catalogs/tools/.
trace_files	execution ID	The trace view (traces/).
evidence_files	evidence ID	Raw captured evidence referenced by trace events; truncated captures are listed in unavailable_references instead (evidence/).
artifact_files	artifact ID	The bytes of each artifact recorded on the execution (artifacts/).
diagnostic_files	attempt ID	Diagnostics attached to a test result (diagnostics/).
test_run_files	run ID	The saved test-run manifest for this run and its baseline run, if any (diagnostics/).
test_result_files	attempt ID	One saved test result per attempt (diagnostics/).

unavailable_references lists references that could not be included.

A catalog file has execution_id and versions[]. Each version has server, connection, complete, pages[], fingerprint, and tools_file. complete is false when the cursor chain stopped before its last page. A version with reason records tools/list pages that M3 could not place in a chain; it has no pages and no tools. tools_file names catalogs/tools/<fingerprint>.json, which holds fingerprint, complete, and tools, an object keyed by tool name. Versions with the same tools and complete value share one file, so a run that lists the same tools in every execution writes them once. Each page's tool_names lists its tools in the order the server returned them; look each name up in tools. A page keeps a tools array instead when a tool has no name or differs from the entry with that name, such as a tool that a later page returns again with other content. A version without tools_file keeps its tools object inline. fingerprint is the catalog fingerprint that catalog_distribution entries in comparison.interface_changes count under before.counts and after.counts.

To print a tool's input schema for one execution, run this from the report directory with EXECUTION_ID set to an ID from catalog_files and TOOL set to the tool name:

jq -r --arg id "$EXECUTION_ID" '.catalog_files[$id]' feedback.json | xargs jq -r '.versions[].tools_file // empty' | sort -u | xargs jq --arg tool "$TOOL" '.tools[$tool].inputSchema // empty'

The command prints one schema for each distinct tool set the execution observed that contains the tool.

In an execution report, an event payload value of 16 KiB or more is written once, in the trace, when the trace holds a byte-identical copy. The report keeps a marker in its place:

{"__m3_ref__": {"file": "traces/<name>.json", "pointer": "/timeline/3/arguments/value", "size_bytes": 5242896}}

file is relative to the report directory. pointer is a JSON Pointer into that file. size_bytes is the size of the value's compact, key-sorted UTF-8 JSON. The payload's top-level keys always stay in the report, and values without an identical trace copy stay inline. To read a referenced value, run this from the report directory with the marker's fields:

jq --arg ptr "$POINTER" 'reduce ($ptr | ltrimstr("/") | split("/")[] | gsub("~1"; "/") | gsub("~0"; "~")) as $k (.; if type == "array" then .[$k | tonumber] else .[$k] end)' "$FILE"

A payload object whose only key is __m3_ref__ or __m3_literal__ is written as {"__m3_literal__": <object>}, so payload data never reads as a marker. When reading a payload, treat an object whose only key is __m3_ref__ as a reference and replace an object whose only key is __m3_literal__ with its value, whose members follow the same rules. A referenced value is plain data.

When a run sets M3_TIMINGS=1, .m3/reports/<run-id>/ also has a timings/ folder with summary.json, trace.json, and the raw per-process records. The bundle does not reference it. See Find slow steps in a test run.

Trace timeline[] entries use these shapes:

• tool_call: server_binding, status, tool, arguments, result, and provenance
• diagnostic: code, status, stage, operation, elapsed_seconds, timeout_seconds, message, and limitations

With the plugin, pytest outcomes are saved as run records and M3 matcher checks as execution evaluations. Other Python assertion results and aggregate summary rows are not saved. Add .m3/ to the project's Git ignore rules.

Direct SDK storage

Install storage support:

uv add "sf-m3[storage]"

Then pass an explicit store:

import sys
from pathlib import Path

import pytest

from m3 import MCPTestKit, StdioServer
from m3.storage import SQLiteExecutionStore

pytestmark = pytest.mark.m3(suite_name="shipping")


def test_reopen_saved_execution(tmp_path: Path) -> None:
    root = Path(__file__).parents[1]
    server = StdioServer(
        name="shipping",
        command=sys.executable,
        args=(str(root / "shipping_server.py"),),
        cwd=str(root),
    )
    store = SQLiteExecutionStore(tmp_path / "executions.sqlite")
    try:
        with MCPTestKit(store=store, env={}) as kit:
            with kit.direct(server) as client:
                client.call_tool("shipping_quote", {"weight_kg": 2, "zone": "local"})
            execution_id = client.final_trace.execution_id

        saved = store.get_snapshot(execution_id)
        assert saved is not None
    finally:
        store.close()

Run this as tests/test_persistence.py in the first-test project. A temporary database prevents tests from sharing history. Closing a kit does not remove records from an explicitly selected SQLite store; close the store after its last query.

The results database uses SQLite's write-ahead log with synchronous=NORMAL. An application crash loses nothing. A power loss or operating-system crash can lose the most recent commits, but the database stays consistent.