OpenAI Evals shutdown: what to save and where multi-turn tests go

OpenAI Evals became read-only on 31 October 2026, and its dashboard and API are scheduled to shut down on 30 November, according to OpenAI's deprecations page. Prompt objects and Agent Builder are scheduled to shut down the same day, and the post on those two covers what to move from them. The migration guide linked from the Evals entry shows how to recreate a dataset eval in Promptfoo. It never mentions a conversation or a turn, and for a tool, an agent or a custom provider it has one table row, which says to configure that workflow in Promptfoo and test it. That fits a guide about datasets and graders, and it leaves a conversation, such as an assistant that must ask a question before it answers, for the reader to solve.

I would split the migration where the guide stops. A dataset eval is a table of inputs, outputs and grader verdicts, and what has to leave OpenAI before the API closes is its definitions and its results. A multi-turn test is a scenario with a user, a goal and a stop rule, and it has to be written again, in Promptfoo or, when the assistant is an AIVAX AI Gateway, in AIVAX Agentic Tests. Agentic Tests does not import OpenAI evals and does not replace dataset graders.

The API holds the grader definitions that the JSONL export leaves out

Pull the definitions and the results through the API before 30 November. OpenAI's Evals guide says existing content "remains available during the transition window", and the evals became read-only on 31 October. Neither the guide nor the deprecations page says which calls read-only turned off. I would assume new runs and edits stopped that day while reads keep working until the shutdown, and use what is left of November to compare the recreated evals with the stored results. Three calls cover the pull:

1. Get an eval returns the data source configuration and the testing_criteria, which is where the graders are defined.
2. Get eval runs lists the runs of that eval.
3. Get eval run output items returns one record per row: the datasource_item, the model's sample with its output messages, and a results list with each grader's name, passed and score. It returns 20 items a page unless you set limit, so pass the last item's id as after until has_more is false.

Keep the raw JSON of all three, and download the dashboard's JSONL for every run as well.

The JSONL holds less than the API does. In OpenAI's forum thread about the shutdown, a reader following the cookbook found a JSONL download where the guide described a runnable Promptfoo configuration. OpenAI support answered on 3 July that the JSONL "only contains the results of a completed evaluation" and that not every evaluation type can export a configuration. The guide has since lost those steps: pull request 2869 removed them in August because the export path had not been verified, which leaves manual recreation as the documented path.

The JSONL is still worth keeping, because promptfoo import accepts the dashboard's eval_items_*.jsonl as a historical eval, keeping each source item under vars.item and the grader values and pass states on the result (pull request 9305, released in 0.121.12). It does not take the API's output-items response either: both sample responses in the API reference, whole or as one item per line, stopped with Unsupported eval export results format. It does not infer assertions or rebuild a runnable configuration either. I imported the two-row fixture from that pull request, not a real export, with 0.125.1: the eval was listed as imported from an OpenAI run, and its configuration held a description, the tests and metadata, with no providers, prompts or assertions. A new Promptfoo run stays separate from the old OpenAI runs, as the cookbook says, and the imported results can sit beside it as history.

The stored outputs can check each recreated grader

The cookbook says to recreate each grader as a Promptfoo assertion and to "verify its behavior before relying on it", and it warns that similarity scores may differ numerically between systems. It also says to compare the recreated prompts, assertions and outputs before relying on the eval, and gives no procedure for that. I found no mapping from OpenAI grader types to Promptfoo assertion types in either company's documentation.

The saved output items can serve as the answer key, because each holds the model's output next to the old grader's verdict. Promptfoo's echo provider returns its prompt as the output, so a saved output can go through a new assertion without a model call. The file below uses two invented outputs to show the shape: one the old grader passed, with the assertion it became, and one the old grader failed, with the same assertion negated by a not- prefix, so a row passes when the new assertion agrees with the old verdict.

prompts:
  - "{{output}}"
providers:
  - echo
tests:
  - description: the old grader passed this output
    vars:
      output: "Team fits eight people. Open Settings > Plans and select Team."
    assert:
      - type: icontains
        value: "Settings > Plans"
  - description: the old grader failed this output
    vars:
      output: "Starter is the cheapest option."
    assert:
      - type: not-icontains
        value: "Settings > Plans"

promptfoo eval -c replay.yaml exits with 0 when every row agrees with the old verdict and with 100 when one does not, the code Promptfoo uses for any failed test case. I changed the second output to end with "Open Settings > Plans to compare." and got 100. A row that disagrees says only that the two graders differ, not which one is right, and a model-graded assertion such as llm-rubric calls a judge model on every replay, so read those rows one by one.

A multi-turn test has to be written again

Promptfoo's simulated-user provider, promptfoo:simulated-user, plays the user from an instructions string and talks to your assistant for up to maxTurns turns, 10 by default, ending early when the simulated user writes ###STOP###. An assertion such as llm-rubric grades the transcript. initialMessages starts the conversation partway and does not count toward maxTurns, and stateful: true sends only the newest user message once a session exists, for targets that keep the history themselves. The target has to accept messages in OpenAI chat format, and a function-calling agent needs its function definitions in the provider configuration.

The configuration below uses the fictional support policy from the multi-turn Agentic Tests walkthrough, which is not AIVAX's plan rules, as the assistant's system prompt.

prompts:
  - >-
    Ask for team size before recommending a plan. For one to five people,
    recommend Starter; for six or more, recommend Team. Explain the
    recommendation using the stated team size. The next signup step is to
    open Settings > Plans and select the recommended plan. Do not claim
    signup is complete.
providers:
  - openai:gpt-6-luna
defaultTest:
  provider:
    id: promptfoo:simulated-user
    config:
      maxTurns: 10
tests:
  - vars:
      instructions: >-
        You are choosing a plan for an eight-person team. Explain your
        team size when asked, ask which plan fits, and continue until you
        understand the recommendation and how to sign up.
    assert:
      - type: llm-rubric
        value: >-
          The assistant asks for team size before recommending a plan,
          recommends Team for eight people, explains why it fits, and gives
          Settings > Plans > Team as the next signup step. It does not claim
          signup is complete.

Promptfoo's simulated user runs on Promptfoo's servers by default

The documentation says the target always runs locally and that "only simulated user responses are generated remotely". In the 0.125.1 source, generating a simulated-user message sends the instructions and the whole conversation so far, your assistant's replies included, along with the account email if one is configured, to Promptfoo's hosted endpoint, and the documentation points to the privacy notice for what is sent.

The documented switch, PROMPTFOO_DISABLE_REMOTE_GENERATION=true, does not move the simulated user onto your machine. I ran the configuration above with it set and a placeholder OpenAI key, and the test ended with Remote generation is disabled. SimulatedUser requires Promptfoo's conversation simulation models. before any request was made. The provider's configuration type declares a userProvider option, and I found nothing in the package that reads it. The endpoint comes from PROMPTFOO_REMOTE_GENERATION_URL when that variable is set, so the request can be pointed at a server you operate.

Agentic Tests only reaches assistants that run through an AIVAX gateway

If the assistant is an AIVAX AI Gateway, Agentic Tests calls the gateway itself, so a change to its model, instructions, tools, skills or knowledge changes the result, and there is no adapter to write. A test names a gateway or an integrated model, not a URL. A gateway can use an external OpenAI-compatible endpoint with your own key, so a service that speaks Chat Completions could sit behind one, but the test would then exercise the gateway's configuration as well as your service.

If the assistant is your own service, Promptfoo reaches it through a provider with no gateway in between, and nothing here argues for routing it through AIVAX just to run these tests; what you accept in exchange is the hosted simulated user described in the previous section.

The scenario translates field by field. The simulated user's instructions become goal, which the judge reads too, and the rubric becomes validation_criteria, which only the judge receives. These are two fields of a test, in the dashboard form or in the body of POST /api/v1/agentic-tests, not a complete request. A script or CI job can send them, with the gateway's slug as model, to the direct evaluation endpoint, which streams one ephemeral run as server-sent events; the walkthrough has a complete request.

{
  "goal": "You are choosing a plan for an eight-person team. Explain your team size when asked, ask which plan fits, and continue until you understand the recommendation and how to sign up.",
  "validation_criteria": "Validate whether the assistant asks for team size before recommending a plan, recommends Team for eight people, explains why it fits, and gives Settings > Plans > Team as the next signup step. It must not claim signup is complete."
}

The plan policy belongs in the gateway's instructions, where production reads it, so for this test put the policy from the Promptfoo configuration above in those instructions.

Shared reference material goes in resources, up to 16 Text or RemoteResource items, and each item's visibility decides who reads it: Internal, the default, shows it to the simulated user and the judge, Gateway to the gateway alone, and Both to all three (API reference). A Gateway or Both item reaches the gateway as a system message ahead of the conversation on every turn. That makes it easy to pass a test on text that production never sees, so use Gateway only for context your application injects at run time, such as a customer record. An Internal item is not sent to the gateway and adds no knowledge to it; Gateway and Both items are, as a system message on every turn.

A test conversation goes through the gateway like any other request, and I found no dry-run or sandbox switch, so treat its tools as live. Point a test gateway at fixtures or a staging backend, and check a signup or a ticket in the system that owns it.

A run is billed for the gateway's inference plus the simulated user and the judge: each turn adds a simulated-user message, and each judged turn adds a judge call (pricing and limits). New runs are capped per account at five a minute on Free, twenty on Pro and a hundred on Max, with no plan cap on Ultimate, shared by manual runs, scheduled runs and direct evaluations, so a suite that starts fifty scenarios at once gets 429s. Pace the starts and retry with backoff. Succeeded and failed runs are kept for one month and cancelled runs for one day (retention), so export the JSON of any run you need for longer.

A succeeded run can still end in loss

The judge scores each judged turn between 0 and 1. A run ends in success at the first score at or above base_threshold, 0.9 by default. It ends in loss when the last two scores and the running trajectory are all at or below loss_threshold, 0.2 by default, so one bad turn does not end a run that can still recover. The two scores are not configurable, and the trajectory, conversation_delta in the events, starts at 0.5, rises after a score above 0.5, falls after one below it and stays between 0 and 1. A run that reaches max_turns (10 by default, 2 to 64) without either is incomplete, and so is one where the simulated user ends the conversation before the judge is satisfied. The judge-trajectory post uses the same per-turn values to watch live traffic.

A run's state is pending, running, succeeded, failed or cancelled, and it says whether execution finished. The verdict is the outcome in the run's result, and a run in the succeeded state can end in loss or incomplete (judge states and outcomes). Failure notifications follow the state, so a scheduled test that ends in loss every night sends no alert, and AIVAX's weekly testing summary, if you enabled it, counts every test with a succeeded or failed run in the past seven days as passing unless its latest such run is failed. A release gate has to read outcome and require success, as the walkthrough does on the chat.validation.end event of a direct evaluation. That endpoint leaves no test or run in the dashboard, so the calling code has to keep the events.

For a check that has to be exact, such as a forbidden phrase in an assistant message, I would not leave it to the judge. A persistent test can call validation hooks with the conversation after each gateway response, and a non-2xx answer from your service stops the run. That run is stored as failed, not as loss, so unlike a judged loss it counts toward the failure notification.

As read on 29 October 2026; the OpenAI dates were reread on 2 November. I ran Promptfoo 0.125.1 locally for the two replay files, the imports and the run with remote generation disabled, and validated the simulated-user file. Three details come from reading AIVAX's code rather than its documentation: how resources reach the gateway, how the loss trajectory is computed and how the weekly summary counts. On November 10, 2026 I ran the goal and validation_criteria shown above through one direct evaluation against the live AIVAX API, on a temporary gateway that carried the plan policy in its instructions and has since been deleted. The run took two turns: the assistant asked for the team size, then recommended Team for eight people and gave Settings > Plans, and the stream ended with chat.validation.end and outcome success at a score of 0.95, after turn scores of 0.65 and 0.95. I also created a persistent test with the same fields and deleted it: it came back with the defaults the post lists (10 turns, thresholds 0.9 and 0.2, medium profile). The loss trajectory, the run caps, the retention periods and the weekly summary come from the documentation and the code. I did not call the Evals API or Promptfoo's hosted simulated user, did not point PROMPTFOO_REMOTE_GENERATION_URL at a server of my own, did not route a test through a gateway with an external endpoint, and did not read how the gateway dispatches tools.