Short answer
Run Codex against your server inside a pytest test, restrict it to the tools you name, and assert on the recorded call. With SineFrame M3:
- Install the M3 CLI with uv tool install sf-m3-cli, then run m3 init and m3 setup.
- Select the codex harness with OPENAI_API_KEY or CODEX_API_KEY, or reuse your ChatGPT login.
- Call agent.run(...) with tools=["server:tool"] and assert with expect(result).to_have_tool_call(...).
- Pin the Codex version with m3 test --runtime managed --harness codex@VERSION=MODEL.
- Check the schema with m3 lint --harness codex, because Codex rewrites large schemas.
What does a Codex test look like?
The test starts your server, gives Codex one goal, limits it to the shipping_quote tool, and asserts on the call M3 recorded: tool name, server, arguments, a successful result, and exactly one call.
import os
import sys
from pathlib import Path
from m3 import ExecutionOutcome, MCPTestKit, expect
from m3.types import StdioServer
HERE = Path(__file__).resolve().parent
def test_codex_calls_the_shipping_tool() -> None:
server = StdioServer(
name="shipping",
command=sys.executable,
args=(str(HERE / "shipping_server.py"),),
cwd=str(HERE),
)
with MCPTestKit(env={}) as kit:
agent = kit.agents(
[
{
"harness": "codex",
"models": [os.environ["M3_DOCS_CODEX_MODEL"]],
"credential_env": {"OPENAI_API_KEY": "MY_OPENAI_KEY"},
}
]
)[0]
result = agent.run(
"Call shipping_quote once for weight_kg 2 in zone local.",
server=server,
tools=["shipping:shipping_quote"],
timeout=120,
)
assert result.snapshot.outcome is ExecutionOutcome.COMPLETED, result.error
expect(result).to_have_tool_call(
"shipping_quote",
server="shipping",
arguments={"weight_kg": 2, "zone": "local"},
status="success",
count=1,
)How does Codex log in during a test?
Map OPENAI_API_KEY or CODEX_API_KEY and M3 passes the key to Codex’s login flow, kept in memory only. With no mapping, M3 copies the auth.json from your Codex login, and credential_env={} forces that reuse of a ChatGPT login. When neither exists, startup fails with "Codex has no login".
How are Codex approvals handled?
M3 answers Codex’s MCP tool approval prompts from the test’s tools selection: a tool outside the selection is declined, even when permission_policy is "allow". Command, sandbox, and file approvals go to permission_policy, which denies by default, and the trace records each one as a request and response pair. Set Codex’s sandbox and permission profile with Codex(permissions=...).
Agent-driven elicitation is supported on Codex and tested on version 0.156.1, with at most 9 rounds per action.
What does Codex do differently with MCP servers?
Codex prefers its own web search. Across 21 MCP servers, Codex 0.165.0 on GPT-6 Luna scored 56.5%, and 17.2% on the 10 servers that overlap a built-in tool. On Microsoft Learn, Terraform, DeepWiki, and Exa it did not call the server in a single trial that needed it.
Codex also changes what the model sees. It compacts input schemas over 5,000 bytes and removes descriptions first; in our tests a 14 KB schema lost its required nested fields, and the call came back once the schema budget was raised. Codex does not act on list_changed, so a tool added after startup was never called. And one prompt line, "If a tool is not available, say which one and stop", broke every Codex run.
- 21 servers: Codex 56.5% (Claude Code 65.4%, OpenCode 82.0% on Haiku 5.5).
- Should-call trials where Codex skipped the server: 44.5%.
- Well-built vs flawed issue-tracker server: GPT-6 Luna 85.6% vs 50.0%, GPT-6.1 Sol 99.0% vs 95.0%.
How do I pin a Codex version?
Pass --runtime managed with a selector of the form codex@VERSION=MODEL. M3 downloads that release from the openai/codex repository and records the requested and resolved version with each execution. Name two Codex selectors to compare versions of the same agent on the same tests.
Frequently asked questions
- Why does Codex not call my MCP server?
- Usually because a built-in tool can do the job: Codex used its own web search instead of search and docs servers in our runs. It can also miss tools whose schemas it compacted or that appeared after startup. An M3 agent test makes each case a failing assertion.
- Can I use my ChatGPT login instead of an API key?
- Yes. Set credential_env={} and M3 reuses the auth.json from your Codex login. With OPENAI_API_KEY or CODEX_API_KEY mapped, it uses the key instead and keeps it in memory.
- Will Codex run shell commands during my test?
- Only if you allow it. Command, sandbox, and file approvals go to the test’s permission_policy, which denies by default, and each request and answer is recorded in the trace.
Related
Try it on your own MCP server
SineFrame M3 is free and open source (Apache-2.0). Direct tests need no model or API key, and runs stay on your machine unless you publish them.