Web search API for LLM agents: why you also need a fetch step

Search for "web search API for LLM" and the first page is comparison articles: Braintrust, TinyFish, You.com and a few vendor pages rank providers by index quality, latency and price. You.com's own summary gets closest to the decision that matters: whether you need search metadata or page-level content in the same request. Our answer is that an agent needs both, from two different calls. Whichever provider you pick, a result's title and snippet tell the model where to look, and they rarely contain the date or the exception that decides whether a claim is true.

So our research pipelines have two calls instead of one. Search discovers candidate URLs, fetch reads the pages we chose, and both outputs are kept so every claim in the answer can be traced to text the application actually read. The same split works for scheduled monitors and dataset enrichment jobs that never call a model.

Snippets answer where, not whether

A title, URL and snippet answer a navigational question well: where is the release note or the documentation page? They answer factual questions badly, because the extract was picked for matching the query and can drop the conditions around it. A pricing page snippet can show the new price without the sentence saying it applies only to annual plans.

AIVAX Web Search returns the same result shape for every request: url, title and text. Some search services return longer page content in that field, and the length varies with the underlying provider, so we never infer completeness from it. When a detail matters, we open the source with Fetch and OCR, check its publication date, prefer the primary source and compare conflicting accounts before storing anything.

Chaining search and fetch in one script

For a backend-controlled workflow, call POST /api/v1/web/search, select candidate URLs, then call POST /api/v1/web/fetch. Both need an authenticated secret API key and a positive account balance. Public pk-aiv- keys cannot use either endpoint, so this code belongs on a server. No model inference is involved.

Search takes a required query and optional topn, country, language and includeDomains. topn defaults to 10 and accepts 1 to 25; it is a requested maximum, and some queries return fewer results. Country and language support depends on the search provider behind the request, so treat those fields as hints. Domain filters narrow discovery and do nothing to restrict what your code fetches later.

Fetch takes a non-empty contents array of URLs or base64 data URIs. Each entry in its data.results has a zero-based index into that array, plus extractedText, processingUnits and error, and the optional JSON-conversion fields described in the Fetch reference. Always join results back to inputs on index.

Save the script as research.mjs and run it with Bun or Node.js 18 or later, with AIVAX_API_KEY set in the environment. It reads the first five results, which is an illustrative selection policy; a production caller should apply its source rules before fetching.

const apiKey = process.env.AIVAX_API_KEY;

if (!apiKey) {
  throw new Error("AIVAX_API_KEY is required");
}

const startedAt = new Date().toISOString();
const searchRequest = {
  query: "browser automation release notes September 2026",
  topn: 10,
  includeDomains: ["developer.chrome.com", "playwright.dev"],
};
const headers = {
  Authorization: `Bearer ${apiKey}`,
  "Content-Type": "application/json",
};

const searchResponse = await fetch(
  "https://inference.aivax.net/api/v1/web/search",
  {
    method: "POST",
    headers,
    body: JSON.stringify(searchRequest),
  },
);

if (!searchResponse.ok) {
  throw new Error(`Search failed with HTTP ${searchResponse.status}`);
}

const searchPayload = await searchResponse.json();
const discovered = searchPayload.data.results;
const candidates = discovered.slice(0, 5);

if (candidates.length === 0) {
  throw new Error("Search returned no candidate sources");
}

const fetchResponse = await fetch(
  "https://inference.aivax.net/api/v1/web/fetch",
  {
    method: "POST",
    headers,
    body: JSON.stringify({
      contents: candidates.map((result) => result.url),
      returnErrors: true,
    }),
  },
);

if (!fetchResponse.ok) {
  throw new Error(`Fetch failed with HTTP ${fetchResponse.status}`);
}

const fetchPayload = await fetchResponse.json();
const evidence = {
  startedAt,
  collectedAt: new Date().toISOString(),
  searchRequest,
  discovered,
  selectionPolicy: "First five returned results",
  read: fetchPayload.data.results.map((result) => ({
    index: result.index,
    url: candidates[result.index].url,
    extractedText: result.extractedText,
    processingUnits: result.processingUnits,
    error: result.error,
  })),
};

console.log(JSON.stringify(evidence, null, 2));

The script stops before summarization on purpose. An HTTP 200 from fetch can still contain failed items, so check each error before passing text to a model. returnErrors defaults to true; with false, failed items disappear from the response, which makes a missing source look like an absent one.

Keep an evidence record you can audit later

The script prints one record. In production we persist discovery before fetching, so a later failure does not erase which URLs were found, and we keep the extracted text next to the URL because pages change after an answer is written. The record needs enough to reconstruct how a claim reached the answer:

• the question or monitor rule and the exact search request
• every returned URL, title and text, not only the selected ones
• the selection policy and the URLs sent to fetch
• each extraction's status, timestamp and text, or a durable reference to it
• the passages that support each downstream claim, with their source URLs
• retry history and the record or decision that consumed the evidence

If findings become long-lived agent context, the record is also what lets you decide which claims may be stored as trusted memory; protecting agent memory from poisoning covers the provenance and write controls we use for that.

Grade discovery, extraction and the answer separately

We grade three things separately, because one overall score hides which step broke. For discovery, take representative questions and check whether the returned URLs include the primary sources you would expect, noting duplicates and irrelevant results. For extraction, compare the fetched text with the original page or document, paying attention to dates, tables and identifiers; OCR and layout extraction can lose structure or characters, so consequential names, numbers and quotations get checked against the source. For the answer, trace each material claim to a retained passage, and count a citation that does not support its claim as a failure.

No relevant URL means the query or source strategy needs work. An empty extraction means an access or format problem, and does not prove the source lacks the information. An unsupported answer built from correct text means the generation or validation step is wrong. For recurring jobs, bound concurrency and retry only failures that can recover; the batch guide covers admission control when many jobs compete for the same quota.

The web-to-text extraction guide goes deeper on that reading step, and PDF to Markdown for RAG covers document-heavy sources.

Fetched pages are untrusted input

A page your agent reads was written by someone else, and some pages are written for agents. Keep fetched text apart from system instructions, label source passages when you pass them to a model, and enforce tool permissions in the application. A fetched page must never be able to change credentials, destination allowlists, retry policy or write permissions. Labels help the model tell sources from instructions, though on their own they do not stop prompt injection. Fetch also does not bypass authentication or bot blocking on the destination.

What each step costs and where the Free plan stops

Search and extraction are billed differently, so budget them separately. Web Search costs $5 per 1,000 searches. Fetch is metered in processing units, with a daily allowance included in every plan and per-plan rates after it; the optional responseSchema conversion has its own jsonProcessingUnits and is billed separately. PU cost depends on content, not text length: the limits page estimates about 1 PU for a native-text PDF, 10 PUs per second of page rendering and 400 to 500 PUs for a ten-page scanned PDF. Each item can be up to 50 MB.

Request quotas matter more than price on Free. The direct search endpoint counts against the plan's daily web search limit, 15 per day on Free, and both search and fetch requests count against the 30 general service actions per day that Free shares across services (Plans and limits). Fetch counts that quota per request, not per URL, so sending the five candidates in one contents array costs one action instead of five.

Letting the agent decide when to read

The script above is a backend pipeline: your code decides what to search and what to open. When an agent should make that choice itself, the Web Foundation MCP exposes the same two steps as web_search and fetch_url, and the harness integration guide shows how to attach it to a coding agent. The MCP tool reads one to five public URLs per call and does not accept data URIs; use the Fetch API for inline files and for documents above the MCP tool's smaller download size. The evidence record is still your responsibility in that setup, because the agent's transcript is not a durable store.

Last verified on 9 November 2026 against the AIVAX source and documentation. The script passes a syntax check but was not run against the live API.