Embedding Infrastructure Across 15 Servers — CPU Selection to Architecture Tiers

As a RAG system scales, embedding computation becomes the bottleneck. Adding a GPU is costly, but assigning embedding to an arbitrary server left us without a clear selection criterion. We benchmarked embedding throughput on all 15 servers in the cluster and derived CPU selection criteria, the irrelevance of RAM and NVMe, and server placement priority.This is an infrastructure design record, not a tool introduction.

Why Design an Embedding Infrastructure

Once local LLM services stabilized, the next bottleneck was embedding. Converting user queries to vectors, pre-indexing documents, and supporting real-time search all require the embedding server to respond reliably.

Initially, embedding ran on the same GPU servers as LLM inference. But GPUs were saturated by LLM workloads, causing embedding requests to queue. At the same time, several CPU servers sat idle.Switching to CPU embedding frees the GPU exclusively for LLM inference and distributes embedding across idle CPU servers.The prerequisite was measuring “which CPU handles embedding well, and by how much.”

GPU Is Not Needed for Embedding

The first question was whether CPU embedding is practical. We calculated the fraction of total response time that embedding occupies relative to LLM inference.

Excluding the N100, every server kept embedding latency below 6% of LLM inference time. The bottleneck is LLM inference, not embedding. Assigning GPU resources to embedding is wasteful.CPU embedding is sufficient; the GPU should stay dedicated to LLM inference.

The N100 is the exception. At 481ms, embedding consumes up to 24% of the wait time relative to LLM inference. This becomes a perceptible bottleneck. The N100 is reserved for proxy and lightweight services; it is not placed as an embedding server.

CPU Architecture Performance Comparison

The 15 servers span 7 CPU types. Ranked by BGE-M3 short text (30 chars) p50 latency and sustained throughput.

9950X3D — Undisputed #1

The 16-core Zen 5 9950X3D leads at 65.0ms and 148.7 items/s. The gap to #2 H255 (70.9ms) is only 8.3% in latency, but throughput is 49% higher. The core count advantage grows more pronounced in batched parallel workloads. It currently shares an AI GPU server and handles both embedding and LLM inference simultaneously.

H255 — Balanced #2

Four H255 units average 70.9ms — close to the 9950X3D despite being embedded in mini PCs. Throughput approaches 100 items/s on BGE-M3, making it the top candidate for dedicated embedding servers after the 9950X3D.

5825U — Workhorse (6 units)

The Zen 3-based 5825U is the most numerous at 6 units. At 118ms it is 66% slower than the H255, but 46 items/s is sufficient for standard request loads. When one unit falls short, traffic can be distributed across multiple units.

N100 — Not Recommended for Embedding

The N100 (4-core low-power) recorded 481ms and 5.8 items/s —6.8x slower than the #2 H255. Suitable for lightweight services and reverse proxying, but not viable as an embedding server.

RAM and NVMe Have No Impact

Each server varies in RAM capacity, brand, and NVMe type. Does this affect embedding performance? We grouped servers by CPU and compared within each group.

5825U group (6 units)

H255 group (4 units)

The reason is straightforward. Embedding inference repeatedly computes with model weights that reside in the CPU L3 cache. The memory bus and NVMe are barely touched during inference, so RAM and NVMe specs do not affect throughput.NVMe only matters during the initial model load. Once the server is up, it has no bearing on inference speed.

Practical conclusion: when provisioning a new embedding server, do not spend budget on RAM brand or NVMe tier.Focus budget on CPU selection.

Latency by Text Length

Embedding model latency scales nonlinearly with input text length. Three ranges were measured: short (30 chars), medium (150 chars), and long (500 chars).

The 9950X3D shows the most linear scaling — only 1.7x increase from short to long text. The N100, by contrast, balloons to 2,650ms at 500 chars, a 5.5x increase. For document indexing workloads that process large volumes of long text, the CPU gap widens further.

RAG system design note: It is advisable to separate real-time user query embedding (short, latency-sensitive) from document indexing (long, batch). Assign low-latency servers for real-time queries and high-throughput servers for batch indexing.

The Hyperthreading Trap

What happens when you enable HT (hyperthreading) in multithreaded embedding? Increasing threads to the logical core count seems like it should improve performance — in practice, the results were the opposite.

On the 9950X3D, pushing threads to the logical core count (32T) causes a 22x throughput degradation versus physical cores (16T). Embedding computation is cache- and ALU-intensive per thread; HT threads sharing a physical core compete with each other and reduce total throughput.

Configuration rule: threads = physical core count.When deploying an embedding server, explicitly set OMP_NUM_THREADS or torch.set_num_threads()to the physical core count. Relying on defaults — which typically use logical core count — inverts performance.

# Check physical core count (excluding HT)
lscpu | grep "Core(s) per socket"

# Set thread count when starting the embedding server
OMP_NUM_THREADS=8 python embedding_server.py

# Set within Python code
import torch
torch.set_num_threads(8)  # physical core count

Temperature and Cooling — Safe Under Sustained Load?

Embedding servers run under 24/7 sustained load. Exceeding thermal limits triggers throttling and degrades performance. Temperature changes under 30 seconds of continuous embedding load were measured.

Every server remained well below the thermal throttling threshold (typically 90–100°C) after 30 seconds of sustained load. Notably, the 5825U and N100 — running mini PC built-in cooling — rose only +5°C and +4°C respectively. Their low TDPs are well within the capacity of compact cooling solutions.

The 9950X3D runs at 58°C at idle because it shares a chassis with a GPU server and ambient temperature is elevated. Under load it reaches 73°C, leaving comfortable headroom. That said, in a hot server room, a tower cooler or better is recommended.

Server Tier Classification — Tier 1/2/3

Based on the benchmark data, servers were classified into three tiers by embedding workload suitability.

Embedding Infrastructure Design Checklist

Design principles derived from this benchmark.

Designing an embedding infrastructure is simpler than it looks. Pick the right CPU and the rest of the variables do not matter.Avoid CPUs at the extreme low end like the N100, and most servers in the fleet can handle embedding without becoming a bottleneck in the LLM pipeline.

The benchmark tool construction and script details are in the CPU Embedding Benchmark Tool post. The overall server monitoring setup is covered in the Grafana + Prometheus Monitoring post.