cal/claude-memory
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

store: Embedding model size barely affects speed — GPU memory bandwidth is the bottleneck

Browse Source
This commit is contained in:
Cal Corum 2026-02-19 14:53:16 -06:00
parent 8250be4b9c
commit 99114ea561
1 changed files with 12 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
graph/insights/embedding-model-size-barely-affects-speed-gpu-memory-bandwid-329d3c.md Normal file
View File

@ -0,0 +1,12 @@
---
id: 329d3c3d-4cb5-4274-8613-df8bdfa9e3b2
type: insight
title: "Embedding model size barely affects speed — GPU memory bandwidth is the bottleneck"
tags: [ollama, embedding, performance, gpu, insight]
importance: 0.7
confidence: 0.8
created: "2026-02-19T20:53:16.955487+00:00"
updated: "2026-02-19T20:53:16.955487+00:00"
---
nomic-embed-text (137M, F16) and qwen3-embedding:8b (7.6B, Q4_K_M) embed 430 memories in roughly the same time (~27-30s) on RTX 4080 SUPER. Reason: embedding is a single forward pass per batch (not autoregressive generation), texts are short (50-100 tokens), batched 50 at a time (only ~9 batches). GPU memory bandwidth, not compute, is the bottleneck. Quantized 8B model fits in ~5.7GB VRAM. This means there's no practical speed penalty for using the highest quality embedding model that fits in VRAM.