Research is a pipeline: search discovers, fetch reads

A search result is a lead, not evidence. Its title and snippet can tell a backend which pages may matter, but they rarely contain enough context to support a consequential claim. Treating that lookup as completed research collapses two different operations: discovering candidate sources and reading what those sources actually say.

For scheduled monitors, dataset enrichment, and context preparation before inference, the distinction belongs in the server-side architecture. Search should produce candidates. Fetch should extract readable content from the selected URLs. The application should retain both steps so a reviewer can see which pages were discovered, which were read, and which extracted passages support the downstream result.

That is a research pipeline even when no model or agent participates.

A lookup answers “where”; research also asks “what does the source say?”

A quick lookup can be sufficient when the output is only navigational: find the current documentation page, locate a release note, or identify likely sources for later review. It is not sufficient when the system will store a fact, trigger an alert, enrich a record, or provide context to a model.

AIVAX Web Search documents this division directly: use search to discover relevant pages, then use Fetch and OCR when a URL is known or a source needs to be read in more detail. Search results help locate evidence; they do not guarantee that a page is accurate or current.

A practical backend therefore has at least two explicit artifacts:

1. A discovery record containing the query, filters, result URLs, titles, and snippets.
2. A reading record containing each selected URL, its extraction outcome, and the extracted text or a durable reference to it.

The second artifact does not make a source true. It makes the basis for a later decision inspectable.

The direct API keeps the model out of the control path

AIVAX documents three Web Search integrations: built-in tools for model inference, the Web Utilities MCP server for compatible agents and automation clients, and the direct API for backend workflows where no model or agent is involved. This article focuses on the third option.

The current server exposes authenticated POST /api/v1/web/search and POST /api/v1/web/fetch routes. Both require a positive account balance. Search accepts a query plus optional result count, country, language, and domain filters; it returns url, title, and text for each result. Fetch accepts a non-empty contents array of URLs or base64 data URIs and returns per-item index, extractedText, processingUnits, and error fields.

The index is operationally important: it lets the caller associate every extraction result with the input URL without relying on completion order or text matching.

Here is a minimal server-side JavaScript example. It searches a bounded set of documentation domains, fetches the first five discovered URLs, and emits one evidence record. The environment variable is a prerequisite; never place an API key in source code.

const apiKey = process.env.AIVAX_API_KEY;

if (!apiKey) {
  throw new Error("AIVAX_API_KEY is required");
}

const query = "browser automation release notes September 2026";
const headers = {
  Authorization: `Bearer ${apiKey}`,
  "Content-Type": "application/json",
};

const searchResponse = await fetch(
  "https://inference.aivax.net/api/v1/web/search",
  {
    method: "POST",
    headers,
    body: JSON.stringify({
      query,
      topn: 10,
      language: "en",
      includeDomains: ["developer.chrome.com", "playwright.dev"],
    }),
  },
);

if (!searchResponse.ok) {
  throw new Error(`Search failed with HTTP ${searchResponse.status}`);
}

const searchPayload = await searchResponse.json();
const candidates = searchPayload.data.results.slice(0, 5);

if (candidates.length === 0) {
  throw new Error("Search returned no candidate sources");
}

const fetchResponse = await fetch(
  "https://inference.aivax.net/api/v1/web/fetch",
  {
    method: "POST",
    headers,
    body: JSON.stringify({
      contents: candidates.map((result) => result.url),
      returnErrors: true,
    }),
  },
);

if (!fetchResponse.ok) {
  throw new Error(`Fetch failed with HTTP ${fetchResponse.status}`);
}

const fetchPayload = await fetchResponse.json();
const evidence = {
  query,
  collectedAt: new Date().toISOString(),
  discovered: candidates,
  read: fetchPayload.data.results.map((result) => ({
    url: candidates[result.index]?.url ?? null,
    extractedText: result.extractedText,
    processingUnits: result.processingUnits,
    error: result.error,
  })),
};

console.log(JSON.stringify(evidence, null, 2));

This example deliberately stops before summarization or inference. A monitor can compare the extracted text with a previous run. An enrichment job can pass validated fields to its dataset writer. A separate inference request can receive only the selected passages after the application has applied its own source and content policies.

Preserve the chain of evidence

Observability here is an application design choice, not a side effect of calling an endpoint. Keep enough data to reconstruct the run:

• the question or monitor rule that initiated it;
• the exact search query and filters;
• the returned URLs, titles, and snippets;
• the policy used to select URLs for fetch;
• each URL's extraction status and extracted content;
• timestamps, retries, and the downstream record or decision that consumed the evidence.

That separation also makes evaluation more precise. Discovery can be evaluated for source coverage and relevance. Reading can be evaluated for extraction success and fidelity against the original page or document. A downstream transformation can be evaluated for whether every material claim is supported by a retained source passage and cited URL.

A single “research succeeded” metric hides where the pipeline failed. No relevant URL is a discovery failure. A relevant URL with an empty or malformed extraction is a reading failure. A correct extraction paired with an unsupported summary is a downstream reasoning failure. Those cases require different fixes.

For recurring or high-volume work, apply queueing, concurrency, and retry policy outside these endpoints. The same admission-control concerns described in Batch is an admission-control problem, not a queue apply when many monitors wake up together.

Retrieved text is data, never instruction

Both AIVAX web guides state the same trust boundary: retrieved or extracted text is external, untrusted source material, not instructions for an application or agent. A page can contain prompt-like text, malicious directives, stale claims, or content unrelated to the query. Fetching it changes its availability, not its authority.

The backend should therefore keep control data separate from retrieved content. Do not concatenate a fetched page into a system instruction. Do not let a page alter destination allowlists, retry policy, credentials, or write permissions. If extracted text later enters inference, label it as source material and require the model's output to remain grounded in the URLs and passages the application selected.

Domain filters can narrow search discovery, but they do not replace source verification or the caller's outbound-access policy. Fetch also does not bypass authentication or a destination that blocks automated access.

If extracted findings are persisted as long-lived agent context, they become a separate governance problem: Persistent memory is a write path, not a notebook explains why provenance and write controls matter after evidence outlives the run that collected it.

Fetch returns text, not certainty

The Fetch API extracts readable text from web pages and supported documents. The documentation says web markup and non-content elements are removed, while OCR can recover text from supported images and scanned PDFs. The result is not a generated summary and not a pixel-perfect copy.

That boundary matters for tables, layouts, charts, scans, and exact identifiers. OCR and extraction can lose structure or characters. Use returnErrors: true when a batch must preserve an explicit result for each failed item, and verify consequential names, numbers, and quotations against the original source before publishing or acting on them. A failed extraction is not evidence that the source lacks the information.

The public contract supports items up to 10 MB. Current pricing should be read from the AIVAX Pricing page, and account quotas and rate limits from Plans and limits. Web Search is billed per search; Fetch and OCR is metered in processing units. This pipeline makes both operations explicit, so the backend can budget and schedule them independently without inventing a cost estimate from text length.

Provider choice should not leak into the pipeline contract

The current server implementation defines a provider-independent web research interface. Its default pool contains Tavily, Linkup, and Jina clients; Exa and AIVAX researcher implementations also exist in the same subsystem but are not members of that default pool.

That is an implementation detail, not a response contract for callers. Backend code should depend on the documented results shape and retain source URLs, rather than branching on a provider name or assuming identical snippets across runs. Search is the candidate-generation stage regardless of which configured client serves a request.

Build research as two accountable stages

The useful distinction is not “search versus AI.” It is discovery versus reading. Search widens the field; fetch turns selected URLs into content that can be inspected, checked, and attached to a later result.

Use the direct API when the backend owns that control flow. Use the MCP-based coding harness pattern when an agent should decide when to search or fetch. In either architecture, preserve the same boundary: URLs and extracted text form an evidence chain, while the application decides what to trust and what to do next.