BGE-M3 vs mE5-Large - Comparing Korean RAG Search Quality Head to Head

In our earlier write-up on building a RAG pipeline, picking an embedding model was just one step among many. In practice, though, it deserves its own investigation. Most RAG failures originate upstream in retrieval, not in the generation model. Even when the necessary document was never found, the LLM will still produce an answer — and users don’t read that as a retrieval failure. They read it as the AI confidently getting it wrong.

Korean is especially prone to this because question phrasing varies so much. The document says “주차 가능” (parking available), but users ask “차 댈 수 있어요?”, “주차할 수 있나요?”, or “근처에 세울 데 있나요?” — three colloquial ways of asking “can I park here?” If the embedding model can’t map that colloquial register onto the same meaning, retrieval collapses easily.

We deliberately kept the test small. Before adopting a model in production, what matters more than a giant public benchmark is whether it passes the 10–50 questions your own documents actually get. Here we split 4 sample cafe documents into 9 chunks and threw 10 natural-language Korean questions at them.

The scoring rule is simple. For a question like “주차할 수 있나요?” (is there parking?), the retrieved chunks need to contain key facts such as whether parking is available, how many spaces there are, and any alternative parking guidance. If those keywords aren’t in the Top-3, the LLM has no grounding left to answer from.

Both produce 1,024-dimensional vectors, so storage cost is identical. The differences showed up in retrieval quality and embedding speed. mE5-Large embedded the 9 chunks in 0.05s versus 0.18s for BGE-M3. But average Recall was 100.0% for BGE-M3 and 86.7% for mE5-Large.

Look only at the 86.7% average and mE5-Large seems good enough. But in RAG you have to look past the average at the character of the failures. The biggest gap in this test was “주차할 수 있나요?” (is there parking?). BGE-M3 pulled the relevant chunk; mE5-Large never got the expected keywords into the Top-3.

This is the crux of model selection. Rather than a model that is “right most of the time,” you want a model that doesn’t miss the questions that are dangerous for your service. Pricing, opening hours, refunds, reservations, shipping, parking, legal notices — collect the questions that hurt when they’re wrong and test those separately.

mE5-Large is fast. In this test it embedded 9 chunks in 0.05s, versus 0.18s for BGE-M3 — more than a 3x gap on paper. But in production you shouldn’t translate that difference directly into user-facing latency.

Document embedding is normally done ahead of time. At query time you embed a single question, retrieve Top-K from the vector DB, and hand it to the LLM. Service bottlenecks usually live in LLM generation time, the network, permission checks, and post-processing. So when picking an embedding model, the cost of a missed retrieval matters far more than a 0.1s difference.

By our criteria, BGE-M3 was closer to the default choice. Vector dimensions are the same, retrieval quality was more consistent, and at this scale the speed difference wasn’t enough to flip the decision. That said, every service has different documents and different questions, so the final call has to be verified against your own data.

Embedding Model Evaluation Checklist

Before reaching for public benchmarks, build a small internal evaluation set. It doesn’t need to be elaborate at the start. Pick 10 good questions and the differences between models already show up.

Conclusion

In this test, BGE-M3 recovered the expected keywords in the Top-3 for all 10 colloquial Korean questions. mE5-Large was faster, but it had misses on the parking and Wi-Fi questions. If you’re attaching RAG to customer support or workflow automation, that difference is not small.

So our current rule is simple. If you’re building Korean RAG for the first time, make BGE-M3 the default candidate and run your own Recall test with 20–50 questions drawn from your service documents. If the result holds up, move on to the vector DB, reranker, and hybrid search. What matters more than the model name is whether it actually stops missing the grounding for your service’s questions.

The full RAG pipeline build is covered in Building a Local RAG Pipeline.