All postsAI

How to roll out a new embedding model in pgvector without breaking retrieval

A new encoder means new dimensions, new scores and new neighbours. A schema, a shadow-read handler in Go and a CI gate that make the swap reversible.

Johnny Unar10 min read

01

why an encoder swap is a migration

Swapping the embedding model behind a search or RAG feature looks like a config change, and that's why it goes wrong. Nothing crashes when you do it. Latency stays flat, error rates stay flat, and the only signal is that results get slightly worse in ways that show up weeks later as support tickets about search "not understanding" invoice numbers. Every vector in the old index was produced by one model, every query vector by whatever the code currently calls, and the moment those two disagree you're comparing coordinates from different maps.

So it helps to treat a model change the way you'd treat a schema migration on a table people depend on. The old and new vectors live side by side, new content gets written to both, the candidate gets real query traffic without serving it, a judged evaluation set decides whether it's better, and the cutover is one reversible switch. None of the steps is hard, but skipping any of them turns a rollback into "re-embed the whole corpus again", which isn't a rollback.

Two facts drive most of the design. Similarity scores from different models aren't comparable, so a threshold like "answer directly if cosine is above 0.82" is tied to one model and has to be versioned or deleted. And models don't share dimensions: OpenAI's text-embedding-3-small returns 1,536 dimensions by default and text-embedding-3-large returns 3,072, according to OpenAI's embeddings guide. That second fact breaks the most common schema.

02

a schema that can hold two models at once

The usual table has an embedding vector(1536) column. A typed column rejects anything else, so the moment the candidate has a different size you can't store it next to the old one, and pgvector says so plainly: ERROR: expected 1536 dimensions, not 1024. The fix is to keep the column untyped and put the type into a partial expression index per model version.

sql
1create extension if not exists vector;
2
3create table document_chunks (
4 id bigint generated always as identity primary key,
5 semantic_key text not null unique,
6 document_id uuid not null,
7 content text not null,
8 content_sha256 bytea not null
9);
10
11create table embedding_versions (
12 id int primary key,
13 version_key text not null unique,
14 model text not null,
15 dimensions int not null,
16 status text not null default 'candidate'
17 check (status in ('candidate', 'shadow', 'active', 'retired')),
18 created_at timestamptz not null default now()
19);
20
21create unique index one_active_version
22 on embedding_versions ((true)) where status = 'active';
23
24create table chunk_embeddings (
25 chunk_id bigint not null references document_chunks (id) on delete cascade,
26 version_id int not null references embedding_versions (id),
27 embedding vector not null,
28 created_at timestamptz not null default now(),
29 primary key (version_id, chunk_id)
30);

Each version then gets its own HNSW index, restricted to its own rows and cast to its own dimensions:

sql
1insert into embedding_versions (id, version_key, model, dimensions, status) values
2 (1, 'te3-small-1536', 'text-embedding-3-small', 1536, 'active'),
3 (2, 'te3-large-3072', 'text-embedding-3-large', 3072, 'candidate');
4
5create index chunk_embeddings_v1_hnsw on chunk_embeddings
6 using hnsw ((embedding::vector(1536)) vector_cosine_ops)
7 where version_id = 1;
8
9create index chunk_embeddings_v2_hnsw on chunk_embeddings
10 using hnsw ((embedding::halfvec(3072)) halfvec_cosine_ops)
11 where version_id = 2;

The second index uses halfvec because pgvector's HNSW index supports vector only up to 2,000 dimensions and halfvec up to 4,000. Try a plain HNSW index on a vector(3072) column and you get column cannot have more than 2000 dimensions for hnsw index. The other way out is OpenAI's dimensions request parameter, which shortens text-embedding-3-large output to a size you choose, but then the shortened vectors are their own version and need their own evaluation.

The partial indexes do two jobs. Queries only use one if they repeat its expression and predicate exactly, so retrieval for version 2 has to be written as where version_id = 2 order by embedding::halfvec(3072) <=> $1::halfvec(3072). Running both shapes through EXPLAIN on Postgres 18 with pgvector 0.8.7, each one picks its own index. The cast inside the index also works as a dimension check: inserting a 1,024-dimension vector under version_id = 1 fails with the same "expected 1536 dimensions" error, because Postgres evaluates the index expression for every row that matches the predicate. The one_active_version index makes a second active version a unique violation instead of a bug you find in production.

If you'd rather not have the cast discipline in every query, one table per version works too. What doesn't work is declarative partitioning by version_id, because every partition has to share the parent's column types, so you're back to an untyped column anyway.

The semantic_key is the other half of the design. Embeddings should hang off something that survives re-chunking, imports and table rewrites, otherwise you can't tell whether the candidate retrieved worse or you changed identity underneath it. A key derived from the document plus a hash of the normalized chunk text does that; the post on document pipelines for RAG covers how to make those IDs stable.

03

backfill and dual write

Once the candidate version row exists, the backfill is a loop over chunks that don't have a vector for it yet:

sql
1select dc.id, dc.content
2from document_chunks dc
3where not exists (
4 select 1 from chunk_embeddings ce
5 where ce.version_id = 2 and ce.chunk_id = dc.id
6)
7order by dc.id
8limit 1000;

A worker embeds each batch and writes it with COPY or a multi-row insert. Because the primary key starts with version_id, the anti-join is an index lookup per chunk, so the query stays cheap as the backfill progresses and it resumes after a crash without bookkeeping. Throttle it against your provider's rate limit and log progress per batch.

While the backfill runs, the ingestion path writes new and changed chunks under every version whose status isn't retired. If you skip dual writes, the candidate quietly misses everything that arrived during the backfill and looks worse than it is in the evaluation, or better if new content happens to be easy.

Build the candidate's HNSW index before you start sending it queries, and give the build enough maintenance_work_mem, because pgvector warns when the graph no longer fits in memory and the build slows down a lot from that point.

04

shadow reads in go

Offline evaluation sets are always cleaner than real traffic. Real users paste half a contract clause, an IBAN with spaces, a stack trace or a Czech query against English documents, so the candidate should see live queries before anyone sees its results. A small handler in front of retrieval answers from the active version and fires the same query at the candidate in parallel.

The version of this that ships in a lot of codebases passes r.Context() to both calls. The Go net/http documentation says the request context is cancelled when ServeHTTP returns, so the slower shadow query gets killed as soon as the user has their answer, and the comparison log fills up with context canceled. context.WithoutCancel (Go 1.21 and later) keeps the request's values but drops its cancellation, and a separate timeout bounds the shadow call:

go
1package shadow
2
3import (
4 "context"
5 "encoding/json"
6 "net/http"
7 "time"
8)
9
10type SearchRequest struct {
11 Query string `json:"query"`
12 K int `json:"k"`
13}
14
15type Hit struct {
16 SemanticKey string `json:"semantic_key"`
17 Score float64 `json:"score"`
18}
19
20type SearchResponse struct {
21 Hits []Hit `json:"hits"`
22 Version string `json:"version"`
23}
24
25type SearchFunc func(ctx context.Context, version string, req SearchRequest) (SearchResponse, error)
26
27type Handler struct {
28 Search SearchFunc
29 Primary string
30 Candidate string
31 ShadowTimeout time.Duration
32 Slots chan struct{} // caps concurrent shadow queries
33 Log func(req SearchRequest, primary, shadow SearchResponse, shadowErr error)
34}
35
36type result struct {
37 resp SearchResponse
38 err error
39}
40
41func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
42 var req SearchRequest
43 if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&req); err != nil {
44 http.Error(w, "invalid request body", http.StatusBadRequest)
45 return
46 }
47
48 var shadowCh chan result
49 select {
50 case h.Slots <- struct{}{}:
51 shadowCh = make(chan result, 1)
52 // r.Context() is cancelled as soon as ServeHTTP returns, so the
53 // shadow query gets a context that keeps request values but not
54 // the cancellation, bounded by its own timeout.
55 ctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), h.ShadowTimeout)
56 go func() {
57 defer func() { cancel(); <-h.Slots }()
58 resp, err := h.Search(ctx, h.Candidate, req)
59 shadowCh <- result{resp, err}
60 }()
61 default:
62 // all slots busy: skip shadowing this request rather than queue it
63 }
64
65 primary, err := h.Search(r.Context(), h.Primary, req)
66 if err != nil {
67 http.Error(w, "search failed", http.StatusBadGateway)
68 return
69 }
70 w.Header().Set("Content-Type", "application/json")
71 _ = json.NewEncoder(w).Encode(primary)
72
73 if shadowCh != nil {
74 go func() {
75 s := <-shadowCh
76 h.Log(req, primary, s.resp, s.err)
77 }()
78 }
79}

The Slots channel matters as much as the context fix. Without it every request spawns an unbounded goroutine against the candidate, which doubles the load on the database and the embedding API exactly when traffic peaks. With it, shadowing degrades to sampling under load instead of competing with production. Tested with httptest and the race detector, the shadow query completes after the response is written, and a control test that passes r.Context() instead gets context.Canceled every time.

Log the query, both top-k lists of semantic keys, latency for each side and the overlap. Don't log the raw scores as a comparison, because a cosine of 0.71 from one model and 0.48 from another tell you nothing about which result is better. Ranks, overlap and judged relevance are comparable across models, and scores aren't.

05

judging the candidate

Comparing the candidate's top-k to the old top-k only measures how similar the two systems are, which rewards the candidate for copying the old one's mistakes. You need relevance judgments. A cross-encoder reranker is a good automatic judge for shadow samples, because it scores a query and a passage together instead of comparing two independently computed vectors, and it doesn't care which embedding model produced the candidates. Back it with a hand-labelled set of a few hundred real queries with known relevant chunks, and keep that set alive as the corpus changes.

Averages hide the regressions that hurt. If the candidate improves long natural-language questions and gets worse at exact invoice numbers, the overall NDCG can still go up while the people who look up invoices all day get a worse product. So the gate should have per-segment floors and a maximum drop against the active version, with the important segments allowed to veto the release:

yaml
1gates:
2 overall:
3 min_recall_at_50: 0.92
4 min_ndcg_at_10: 0.75
5 max_drop_vs_active: 0.01
6 segments:
7 invoice_numbers:
8 min_recall_at_50: 0.98
9 max_drop_vs_active: 0.0
10 czech_queries:
11 max_drop_vs_active: 0.02

The numbers are placeholders; set yours from the active version's current results. The check itself is a few lines that run in CI on the fixed set and nightly on fresh shadow samples:

python
1def check(gates: dict, active: dict, candidate: dict) -> list[str]:
2 failures = []
3 for segment, rules in [("overall", gates["overall"]), *gates["segments"].items()]:
4 a, c = active[segment], candidate[segment]
5 for metric in ("recall_at_50", "ndcg_at_10"):
6 floor = rules.get(f"min_{metric}")
7 if floor is not None and c[metric] < floor:
8 failures.append(f"{segment}: {metric} {c[metric]:.3f} < {floor}")
9 drop = a[metric] - c[metric]
10 if "max_drop_vs_active" in rules and drop > rules["max_drop_vs_active"]:
11 failures.append(f"{segment}: {metric} dropped {drop:.3f} vs active")
12 return failures

Fed with a candidate that wins overall but loses two points of recall on invoice numbers, it returns three failures, all in invoice_numbers, and the job exits non-zero. That's the behaviour you want: a better average isn't allowed to buy a worse result for the queries the business depends on.

06

cutover, rollback and what to watch afterwards

With the schema above, the cutover is one transaction, and the unique index guarantees there's never a moment with two active versions:

sql
1begin;
2update embedding_versions set status = 'retired' where status = 'active';
3update embedding_versions set status = 'active' where id = 2;
4commit;

The service reads the active version with a short cache, and query embedding, index choice and any score thresholds all key off it. Rollback is the same transaction in reverse, which only works if the old vectors and their index are still there, so leave the retired version in place until the new one has survived a few weeks of real traffic. Keep the reranker independent of the embedding version for the same reason; if it has model-specific heuristics baked in, rolling back the encoder gives you a different class of failure instead of the old behaviour. When you finally drop the old version, drop index concurrently its HNSW index first and delete its rows in batches.

After the cutover, retrieval needs ongoing checks, because a frozen model doesn't mean frozen results. Three are cheap and catch real problems. Run the labelled canary queries on a schedule and alert on a sustained drop in recall@k, which catches corpus changes, chunking bugs and anything else that moves results. After every index rebuild or change to hnsw.ef_search, sample queries and compare the index result with the exact result (run the same query inside a transaction with set local enable_indexscan = off), which catches approximate-search settings that quietly trade recall for latency. And re-embed a fixed set of canary texts with the active model now and then and compare against the stored vectors: within one model the cosine between the two should be very close to 1, so a drop means the provider changed something or your preprocessing did.

Two checks people often propose don't add much. Comparing the cosine between old-model and new-model vectors of the same text is meaningless, because the two spaces are unrelated and often don't even have the same number of dimensions. And norm monitoring is mostly moot for OpenAI embeddings, which the embeddings guide says are normalized to length 1; it's worth it only for self-hosted models where a missing normalization step is a real failure. What does pay off is a hard check at query time that the query vector's model and dimensions match the version being searched, failing loudly when they don't, because a query embedded with one model against an index built with another still returns plausible-looking results.

Thanks for reading.
Johnny Unar

Written by

Johnny Unar

Keep reading

Stop patching.Start building.

Tell us what's slowing you down. We'll bring coffee and opinions.