Hastimal Jangid 
Every RAG tutorial makes it look like three steps: chunk your documents, embed them, and let the model do the rest. In production on Vertex AI, the three steps that actually decide whether the thing works are chunking strategy, grounding quality, and knowing when retrieval is the wrong tool entirely. This is what I learned shipping a retrieval pipeline for an internal knowledge system on Google Cloud.
The architecture, roughly
The stack settled into something simple: documents land in Cloud Storage, get chunked and embedded through a Cloud Run job, and land in a vector index — I tested both Vertex AI Search and a self-managed pgvector setup on Cloud SQL before landing on Vertex AI Search for the managed indexing and freshness handling. Gemini sits on top doing retrieval-augmented generation with function calling for structured lookups the vector index isn't good at, like exact policy numbers or dates.
Nothing about that is exotic. The interesting part is everything that had to change once real documents and real questions started hitting it.
Chunking is the whole game
Fixed-size chunking (say, 500 tokens with 50-token overlap) is the default in every tutorial, and it's the first thing that fails on real documents. Tables get split mid-row. A numbered procedure gets separated from its own numbering. A section header ends up in one chunk while its content ends up in the next, so neither chunk means anything on its own.
What worked better was structure-aware chunking: split on the document's actual headings first, then only fall back to fixed-size splitting inside a section if that section is too large to embed as one chunk. It's more work upfront — you need a parser that understands your document format, not just a token counter — but retrieval quality improved more from this than from any embedding model swap.
Grounding is a UX problem, not just a model problem
Gemini's grounding with Google Search and with your own retrieved context is good at citing sources. It's less good, out of the box, at telling you when it shouldn't answer because nothing retrieved is actually relevant. The failure mode isn't hallucination in the dramatic sense — it's confident synthesis from a weakly related chunk that technically matched the query's keywords but not its intent.
The fix that helped most wasn't a smarter model. It was a relevance threshold on the retrieval step itself: if the top retrieved chunk's similarity score falls below a set bar, the system says it doesn't have a confident answer instead of asking Gemini to make the best of a bad chunk. Users trust "I don't know" far more than they trust a wrong answer delivered with total confidence.
Cost and latency trade off differently than you'd guess
The expensive part wasn't Gemini generation — it was re-embedding on every document update. Documents that changed frequently (policy pages, changelogs) were getting fully re-chunked and re-embedded on every edit, most of which touched one paragraph. Moving to incremental re-embedding — diffing the document and only re-processing changed sections — cut embedding costs by more than half and, as a side effect, cut end-to-end latency for fresh-content queries since the index updated faster.
What I'd do differently
- Start with structure-aware chunking from day one instead of retrofitting it after fixed-size chunking underperformed.
- Build the "don't answer" path before the demo, not after the first bad answer in front of a stakeholder.
- Instrument retrieval quality separately from generation quality — a bad answer can come from a bad retrieval or a bad generation, and you need to know which before you can fix it.
Takeaway
RAG on Vertex AI works well once you stop treating it as a single system and start treating it as two systems — a retrieval system with its own quality bar, and a generation system that's only as good as what it's given. Most of the engineering effort belongs on the retrieval side, not the prompt.