PDF to Markdown for RAG: extract, chunk, and import JSONL

A PDF is a poor source for a RAG collection as it comes. Plain-text extraction strips the headings and table structure that chunk boundaries need, and one embedding per page blurs unrelated topics together, so a long manual matches many queries weakly. Converting the PDF to Markdown first gives the chunker real boundaries to follow — headings, paragraphs, tables — instead of cutting every N characters mid-sentence.

With AIVAX the conversion is one API call, POST /api/v1/web/fetch, and the splitting is another, POST /api/v1/generations/segment; a short script writes the chunks as JSONL with a stable ID per line so you can import and re-import them. If you would rather see it work first, AIVAX Toys runs both calls in a browser, free and without sign-up. The toy enables sanitization by default; the script below leaves it off.

The pipeline: extract, segment, import

Each step can be swapped. If you already parse PDFs locally, start at step two; if your text is already one idea per unit, the segmentation section below explains when to skip it.

Try it in the browser first

Media → Markdown → RAG segments is the toy in AIVAX Toys, a set of small, open-source demos of AIVAX features. You paste a web page link or drop a PDF or image (PNG, JPEG, WebP, TIFF or BMP), read the Markdown, then split it and download the JSONL.

The Worker behind the page uses AIVAX's own key, so the visitor is not billed: each visitor gets 5 documents per minute and 20 per hour, enforced per IP address by a Durable Object, and a Cloudflare Turnstile check runs before each call. Files are capped at 10 MB in the toy; the Fetch API itself accepts up to 50 MB per item. A "For developers" drawer lists every API call with its latency and processing units, and an estimate based on public list prices; you can switch the plan used for OCR pricing. Applying the same list prices to the PUs of your own calls is a quick way to estimate a whole corpus from a few samples.

The Worker does not write the upload anywhere; it forwards it to the Fetch API and returns the result, which you can verify in src/worker.js of the Apache-2.0 repository aivaxlabs/toys. Even so, do not upload documents you are not allowed to send to a third-party service, in the toy or in your own pipeline. The toy is a demonstration of the two calls below; the rest of this post is about what you would build yourself.

How do you convert a PDF to Markdown?

Send the file to the Fetch API as a base64 data URI with its real MIME type. Web pages go in as plain URLs, and one request can mix several items — links, PDFs and PNG, JPEG, WebP, TIFF or BMP images.

curl https://inference.aivax.net/api/v1/web/fetch \
  -H "Authorization: Bearer $AIVAX_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"contents\": [\"data:application/pdf;base64,$(base64 -w0 manual.pdf)\"],
    \"returnErrors\": true
  }"

The response has one entry in results per item, matched by index:

{ "data": { "results": [ {
  "index": 0,
  "extractedText": "# Title\n\nParagraph...",
  "processingUnits": 7,
  "jsonProcessingUnits": 0,
  "error": null
} ] } }

The values above are placeholders that show the shape. Digital and scanned pages are both handled: the Fetch and OCR docs describe text from digital PDFs and OCR text from scanned pages, and mixed PDFs can combine both. Scan quality and complex layouts affect the OCR result, and the docs state that layout, table structure, charts and embedded images are not guaranteed to be reproduced, so read a sample of your own pages before you trust the output; the ingestion quality checklist covers how.

With returnErrors true, a failed item has error set and extractedText null. Check it before the next step: a failed extraction says nothing about whether the PDF contains the answer.

Extraction is metered in processing units (PUs), not tokens; the docs advise against estimating cost from text length and say to read processingUnits from the response. When the daily allowance does not cover an extraction, the list rate is $0.15 per 1,000 PUs on Free, $0.05 on Pro, $0.03 on Max and $0.02 on Ultimate — check Pricing for current values. For the broader picture of what Fetch reads, see Fetch turns the messy web into text your code can use.

How the segmenter splits the Markdown

Pass the Markdown to the segmentation endpoint as an element of documents:

curl https://inference.aivax.net/api/v1/generations/segment \
  -H "Authorization: Bearer $AIVAX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "documents": ["# Title\n\nParagraph..."], "sanitize": false }'

{ "data": {
  "result": [ { "index": 0, "count": 8, "segments": ["...", "..."] } ],
  "usage": { "processing_units": 2516, "cost": 0 }
} }

Again, the numbers are placeholders. Without sanitize, segments target about 300 tokens, cover the whole document in source order, and do not overlap; a document of about that size or smaller comes back as one segment. The endpoint finds boundaries from the document structure and the semantic similarity of neighboring passages, so it is a boundary-finding step rather than a size dial. If you need a specific chunk size or overlap between chunks, add a pass of your own after it; the endpoint does not produce overlap.

sanitize trades completeness for cleanliness. It is processed by a language model, takes longer, and omits content it judges useless for retrieval, such as front matter and link references. The API default is false; the toy's checkbox is on, because a demo benefits from dropping clutter such as page headers and link lists. Use it only when what is omitted is genuinely irrelevant; when exact wording or full traceability matters, leave it off and review the source.

Clean the Markdown before segmenting. The Text Segmentation docs list the failure modes: tables lose their headers, OCR drops line structure, and exported documents repeat running headers on every page. Restore table headers and remove repeated page headers first, rather than expecting the segmenter to work around them. Segmentation does not fix OCR mistakes, and it does not embed anything: it returns strings to your application and stores nothing.

Skipping the split and importing the whole PDF as one document brings back the problem from the opening: one embedding covers every topic in the manual, and each added page makes it a weaker match for any single question. The docs recommend the opposite shortcut too — for text that is already one idea per unit, such as FAQ answers, import it directly, because each unnecessary split adds indexing work and can separate statements that only make sense together.

Without sanitize, segmentation is billed at $0.10 per million document tokens before the plan's inference tax; with sanitize on, the charge follows the language model's usage instead. Segmentation also has its own account quota that counts every entry in documents, so a batch of documents consumes it faster than a single request suggests. See Plans and limits.

The JSONL format for a collection import

A collection import takes one JSON object per line. These are the documented fields, with the values the toy writes:

Put anything people will search for in text. Tags and metadata are for filtering, audits and source links, and metadata is not part of the semantic text.

For new integrations, import through the JSONL WebSocket endpoint, which replaces the multipart upload and accepts ten times as many documents per import. The multipart upload below remains available for existing integrations. Upload the file as the multipart field documents:

curl https://inference.aivax.net/api/v1/collections/$COLLECTION_ID/documents \
  -H "Authorization: Bearer $AIVAX_API_KEY" \
  -F "documents=@manual.pdf.jsonl"

Re-importing the same PDF

Because docid is the match key, the import is an upsert. Per the collection docs, a line with an existing docid and changed text updates that document and queues it for reindexing; a line with unchanged text is skipped, and a line that changes only __ref or __tags is skipped too. Indexing is billed on document text tokens when documents are created or their text changes, so re-importing an unchanged file should not trigger reindexing.

Plan for how the source#number scheme behaves. The example uses the file name as the source, so two files that share a name collide; use a source identifier that is unique within the collection. And when a new version of the PDF splits at different boundaries, the numbering shifts and many chunks change text at once. If the new version has fewer chunks, the old extra ones stay in the collection. The WebSocket import takes an insert-mode=sync query parameter and the multipart upload an insert-mode form field; either deletes documents in the collection whose names are not in the uploaded file. Use it only when the collection holds nothing but that one source, because it compares against every document in the collection, not just those with the same __ref.

Per-request line limits and daily insertion limits depend on your plan; split large files into several imports.

A script for the whole pipeline

This script reads a PDF or image, calls both endpoints, and prints JSONL to stdout, with the chunk and PU counts on stderr. It runs on Bun or Node 18+ and uses no dependencies.

import { readFile } from "node:fs/promises";
import { basename, extname } from "node:path";

const API = "https://inference.aivax.net/api/v1";
const MAX_BYTES = 50 * 1024 * 1024;
const MEDIA_TYPES = {
  ".pdf": "application/pdf",
  ".png": "image/png",
  ".jpg": "image/jpeg",
  ".jpeg": "image/jpeg",
  ".webp": "image/webp",
  ".tif": "image/tiff",
  ".tiff": "image/tiff",
  ".bmp": "image/bmp",
};

const args = process.argv.slice(2);
const sanitize = args.includes("--sanitize");
const [path] = args.filter((arg) => !arg.startsWith("--"));
const key = process.env.AIVAX_API_KEY;

if (!path || !key) {
  console.error("Usage: AIVAX_API_KEY=... bun pdf-to-rag.mjs <file.pdf|image> [--sanitize] > output.jsonl");
  process.exit(1);
}

const mediaType = MEDIA_TYPES[extname(path).toLowerCase()];
if (!mediaType) throw new Error("Use a PDF or a PNG, JPEG, WebP, TIFF or BMP image.");

const bytes = await readFile(path);
if (bytes.length > MAX_BYTES) throw new Error("Fetch accepts at most 50 MB per item. Split the file first.");

async function post(route, body) {
  const response = await fetch(`${API}${route}`, {
    method: "POST",
    headers: { authorization: `Bearer ${key}`, "content-type": "application/json" },
    body: JSON.stringify(body),
  });
  const payload = await response.json().catch(() => ({}));

  if (!response.ok) throw new Error(`${route} failed with HTTP ${response.status}: ${payload.message ?? "no message"}`);
  return payload.data;
}

const fetched = await post("/web/fetch", {
  contents: [`data:${mediaType};base64,${bytes.toString("base64")}`],
  returnErrors: true,
});
const [item] = fetched.results;
if (item.error) throw new Error(`Fetch could not read the file: ${item.error}`);

const segmented = await post("/generations/segment", { documents: [item.extractedText], sanitize });
const [{ segments }] = segmented.result;

const source = basename(path);
segments.forEach((text, index) => {
  console.log(
    JSON.stringify({
      docid: `${source}#${index + 1}`,
      text,
      __ref: source.slice(0, 64),
      __tags: ["pdf-to-rag"],
      __meta: { source, index: index + 1 },
    }),
  );
});

console.error(`${segments.length} chunks | extraction ${item.processingUnits} PU | segmentation ${segmented.usage.processing_units} PU`);

Run it with AIVAX_API_KEY=... bun pdf-to-rag.mjs manual.pdf > manual.pdf.jsonl, then upload the file as shown above. The script is deliberately short: it handles one file, stops on the first error, and has no retries.

The toy can hide its key because it calls the API with AIVAX's own; your integration cannot. Keep the key on a server or in your shell, and never ship it in browser code.

What should you check before you index the chunks?

Open the Markdown next to the PDF for a handful of pages, and open the chunks next to the Markdown:

• Headings in the PDF are headings in the Markdown, so the chunk boundaries have structure to follow.
• Table headers still sit with their rows, and numbers match the page.
• Running headers and footers are not repeated inside the text.
• Chunk sizes are in the same range, with no chunk that is a single orphaned line or a whole chapter.
• Pick three questions whose answers you know, search the collection, and confirm the right chunk comes back before you attach it to anything in production.

The PDF ingestion checklist turns this into an acceptance matrix, and A vector database is not a RAG system places the preparation step among the other jobs a RAG system has.

When a different approach fits better

The Fetch and segment route is hosted and metered; it is not the right answer for every corpus.

These can be combined. A local parser can produce the Markdown and the AIVAX segmentation endpoint can split it, or you can chunk with your own splitter and use only the JSONL import.

When you need a few named fields from the PDF rather than text to retrieve, the extract endpoint returns JSON shaped by your schema with no conversion step of your own; extracting structured data from a PDF covers what a request with several documents bills when one of them fails, and turning PDFs and images into JSON with one API call walks through the request itself. For a short description of each PDF before you decide what to index, Summarization returns one summary per document. If readers need the PDF in another language, the same extracted Markdown can be split at headings and translated in batches.

A note on what we verified: we ran the script above against simulated API responses to confirm the request bodies, the sanitize flag and the shape of the JSONL lines, but not against live OCR for this post, so the chunk counts and PU values on your own files are the ones to trust. We have not benchmarked the Markdown quality of the local parsers in the table above against each other; that comparison is about control and operations, not accuracy.