Contextual Compression, Cross-Encoder Reranking, and Late Chunking

In production Retrieval-Augmented Generation (RAG) pipelines, initial vector retrieval (Bi-Encoders) prioritizes high recall at low latency, but often returns noisy, redundant, or partially relevant chunks. Rerankers (Cross-Encoders), Contextual Compression, and Late Chunking dramatically improve precision and eliminate hallucination.

⚡ Quick Dive

Bi-Encoder vs. Cross-Encoder (Reranker)

Feature Bi-Encoder (Dense Vector Search) Cross-Encoder (Reranker)
Input Processing Encodes Query and Document independently Encodes (Query, Document) pair jointly
Attention Scope Query and Document tokens never attend to each other Full cross-attention between Query & Doc tokens
Speed ⚡ Ultra-fast (Milliseconds over 10M vectors via ANN) Slower (Requires forward pass per candidate)
Accuracy Good (Used for Initial Recall: Top-50) 🎯 Maximum (Used for Precision Rerank: Top-5)

📖 Extended Guide

1. The Two-Stage Hybrid Retrieval Pipeline

User Query ──► [ Stage 1: Fast Candidate Retrieval (Top 50) ]
                     │ (Combines Dense Vector Search + BM25 Lexical Search via RRF)
                     ▼
               [ Stage 2: Cross-Encoder Reranker (Top 5) ]
                     │ (Scores full semantic interaction; discards noise)
                     ▼
               [ Prompt Context Assembly ──► LLM Generation ]

Reciprocal Rank Fusion (RRF) Formula:

$$\text{RRF_Score}(d) = \sum_{m \in M} \frac{1}{k + r_m(d)}$$ Where $r_m(d)$ is the rank of document $d$ in search system $m$, and $k \approx 60$ is a smoothing constant.

⌂ All Guides