RAG & Knowledge Systems

ColPali: Efficient Document Retrieval by Embedding Page Images with a Vision-Language Model

Faysse, M., Sibille, H., Wu, T., Omrani, B., Viaud, G., Hudelot, C., Colombo, P.

ICLR 2025 · arXiv 2407.01449

An engineering dissection of ColPali — a document-retrieval system that skips the OCR/layout/chunking pipeline and embeds page images directly with a vision-language model, matching queries via ColBERT-style late interaction. Covers why the ingestion pipeline (not the embedding model) is the real retrieval bottleneck, the multi-vector late-interaction operator, the PaliGemma-based architecture, the contrastive training recipe, the storage/latency trade-offs of multi-vector retrieval, and token pooling as the compression lever.

• Embeds document pages directly from their image, deleting the brittle OCR → layout-detection → chunking → captioning ingestion pipeline that is the real latency bottleneck in RAG indexing
• Uses ColBERT-style late interaction: one embedding per image patch and query token, scored by a sum-of-max-dot-products operator that is fully differentiable and end-to-end trainable
• Built on PaliGemma-3B with a projection to 128-dim vectors; trained contrastively with LoRA adapters on 118k query-page pairs for a single epoch
• Multi-vector storage costs ~257.5 KB/page, but token pooling cuts vectors by 66.7% while keeping 97.8% of retrieval quality