When a 200-page PDF goes into a RAG system, search results often get worse, not better. A table gets separated from its caption. A section header ends up in a different chunk than the section itself. The model retrieves fragments that don’t actually answer the question. Chunking fixes this by splitting a document into smaller, coherent pieces sized for accurate embedding and retrieval, rather than feeding in the whole file or splitting it at arbitrary page breaks.
Good chunking follows a document’s natural structure (headings, paragraphs, sections) instead of fixed page or character counts, keeps a bit of overlap between chunks to preserve context across boundaries, and strips out repeated boilerplate like headers and footers before anything gets embedded. PDFs add their own complications on top of that, including multi-column layouts, tables that span pages, and scanned pages that need OCR before they can be chunked at all. This guide covers the core chunking principles, the techniques worth knowing, and the PDF-specific problems that a generic chunking strategy usually misses.
What Chunking Means in a RAG System
Chunking is the process of dividing a large document into smaller, coherent text pieces that a retrieval-augmented generation system can index, embed, and retrieve efficiently at query time. Documents are too large to embed as a single unit, so chunking lets the system focus on the specific section that’s actually relevant to a question instead of the entire file, which improves both search precision and answer quality.
Why PDFs Are Harder to Chunk Than Plain Text
Most chunking advice is written for clean text or Markdown, and PDFs rarely cooperate with that. A few problems show up specifically with PDFs:
- Multi-column layouts. Text extraction can read across columns instead of down them, scrambling sentence order before chunking even starts.
- Tables that span pages. A table split across a page break can get chunked into two pieces that make no sense independently, or lose its header row entirely.
- Scanned or image-based pages. These need OCR before any text-based chunking is possible, and OCR quality directly affects chunk quality.
- Repeated headers and footers. Page numbers, document titles, and boilerplate legal text repeat on every page and can get chunked in as noise unless they’re filtered out first.
- Embedded images and charts. Meaningful information can live in a figure or chart, not in nearby text, which plain text chunking simply skips.
None of this means PDFs are unusable for RAG. It means the extraction step, not just the chunking step, needs to account for structure before splitting begins, a gap that recent retrieval research has specifically flagged as heterogeneous formats like PDFs become more common in enterprise knowledge bases.
Core Chunking Principles
- Chunk size. A common starting point is 400 to 800 tokens with roughly 10 to 20 percent overlap, though the right size depends on the type of question and how your retrieval pipeline is set up. Short factual queries perform well with smaller chunks, while questions that need broader context benefit from larger ones.
- Semantic coherence. Split along logical boundaries, sections, paragraphs, or topics, rather than a fixed character count that can cut a sentence in half.
- Metadata retention. Keep titles, headings, and page numbers attached to each chunk so retrieved results stay traceable back to their source.
- Overlap. A small overlap between adjacent chunks (10 to 20 percent is a reasonable range) helps preserve continuity for content that spans a chunk boundary.
- Noise removal. Strip boilerplate, repeated headers and footers, and irrelevant formatting before chunking, not after, so it never gets embedded in the first place.
Chunking Techniques Compared
| Technique | Description | Best for | Tradeoff |
| Fixed-length | Splits by a set token or character count | Simple, uniform pipelines | Can split mid-sentence, losing meaning |
| Rule-based | Splits on delimiters like headings or page breaks | Structured documents with clear sections | Can miss subtle topic shifts within a section |
| Page-level | Treats each PDF page as one chunk | Paginated documents where page boundaries carry meaning | Pages don’t always align with topic boundaries |
| Semantic | Uses NLP to detect topic boundaries | Best contextual accuracy | Requires more compute and tooling |
| Hybrid | Combines rule-based structure with semantic or fixed-length splitting within sections | Long, mixed-structure documents | More complex to set up and tune |
Page-level chunking is worth calling out specifically for PDFs, since page boundaries are already built into the file format and often align reasonably well with how the document is organized, which Weaviate’s research on chunking strategies also points to as a fast default before investing in semantic chunking.
Handling Tables, Images, and Scanned Pages
This is where most generic chunking advice falls short for PDFs specifically.
For tables, extract them as a distinct element rather than flattening them into surrounding paragraph text. Tools like Unstructured can detect table structure during extraction and keep a table’s header row attached to its rows even if the table spans a page break.
For scanned or image-based PDFs, OCR has to run before chunking is possible at all, and OCR errors propagate directly into chunk quality, so it’s worth checking OCR output on a sample before processing an entire document set.
For embedded charts and figures, a short generated description of the image, produced by a vision-capable model, gives the chunk something retrievable even though the original information isn’t in text form.
How Chunking Affects RAG Performance
Well-designed chunks improve retrieval accuracy by matching queries to genuinely relevant content, which in turn reduces hallucinations by giving the model focused, context-rich input instead of noisy or incomplete passages, the kind of grounding covered in sources and citations more broadly. Chunk size also has to balance two constraints at once. Small enough for precise retrieval, large enough that the chunk still makes sense without its surrounding context. Good chunking additionally makes ongoing content updates easier, since a well-scoped chunk can be re-indexed on its own without reprocessing an entire document.
Tools for Chunking PDFs
- pypdf, the actively maintained successor to the now-deprecated PyPDF2, for basic text and structure extraction.
- pdfplumber, for extracting tables and layout-aware text.
- Unstructured, for PDF-specific partitioning that handles tables, images, and OCR in one pipeline.
- spaCy or NLTK, for detecting sentence and topic boundaries once text has been extracted.
- AI platforms like CustomGPT.ai that automate extraction and semantic chunking together, without requiring a custom pipeline.
Getting Started With CustomGPT.ai
CustomGPT.ai handles PDF extraction and semantic chunking automatically, including tables and multi-column layouts, so large PDFs turn into structured, retrievable knowledge without building a custom RAG pipeline from scratch. It pairs that with source citations on every answer, which matters most for the exact failure mode poor chunking causes, a confident answer built from an incomplete or mismatched chunk.
The Bottom Line
Chunking is a foundational step in building RAG systems for PDFs, and the documents themselves are usually the hard part, not the algorithm. Structure-aware extraction, sensible chunk sizes with overlap, and explicit handling for tables, images, and scanned pages together make the difference between a RAG system that retrieves accurately and one that returns confident, disconnected fragments.
Try CustomGPT.ai’s smart chunking free →
Frequently Asked Questions
What chunk size should I use for PDF documents in a RAG system?
400 to 800 tokens with 10 to 20 percent overlap is a reasonable default. Shorter factual queries tend to do better with smaller chunks, while questions needing broader context benefit from larger ones. Testing 2 to 3 chunk sizes against your actual documents and queries beats guessing a single number upfront.
Does chunk overlap actually make a difference?
Yes, for content that spans a chunk boundary. Without overlap, a sentence or idea that crosses two chunks can lose meaning in both. A modest overlap, often 10 to 20 percent of the chunk size, is usually enough without significantly increasing storage or retrieval cost.
How should I handle PDFs with heavy tables or scanned pages?
Extract tables as distinct structured elements rather than flattening them into paragraph text, and run OCR before chunking anything from a scanned page. Check OCR output quality on a sample of pages before processing a full document set, since OCR errors carry directly into chunk quality and, eventually, into retrieval accuracy.
Is page-level chunking or semantic chunking better for PDFs?
Page-level chunking is a fast, reasonable default for paginated documents since the boundaries already exist in the file. Semantic chunking generally retrieves more accurately but costs more to run. Many teams start with page-level or rule-based chunking and move to semantic chunking only for documents where retrieval quality actually falls short.
How do I get started with chunking large PDFs without building a custom pipeline?
Platforms like CustomGPT.ai handle extraction, table and layout detection, and semantic chunking automatically, so a large PDF collection can be ready for retrieval without custom scripting.
Related Reading
- The RAG implementation guide covers building a retrieval pipeline end to end.
- Components of a RAG system breaks down what determines retrieval quality overall.
- RAG evaluation metrics covers measuring retrieval accuracy once a system is live.
- Anti-hallucination AI explains how grounded, cited answers reduce unsupported output.
- What are alphanumeric characters covers a related retrieval challenge with structured identifiers and codes.
- Data connectors covers getting PDFs and other content into a system in the first place.
- Security and trust details how uploaded documents are handled once ingested.
- The no-code GPT builder covers the setup path for teams without a dedicated engineering team.
- Enterprise knowledge search applies these chunking concepts to internal documentation at scale.
- How to pick the right AI model covers the model side of a RAG stack, alongside retrieval.
Arooj Ejaz is the Marketing Operations Lead at CustomGPT.ai, where she works on content, growth operations, and go-to-market programs for AI agent and chatbot solutions.