LLMOps: Semantic Caching for Latency & Cost Optimization

In production LLM systems, latency and token costs are the two primary bottlenecks. A single complex agent step can take 2–5 seconds and cost upwards of ₹1.50 depending on the context length and model choice.

While traditional exact-match caching (like Redis key-value storage) works for deterministic APIs, it fails for natural language queries because users rarely type the exact same prompt twice.

Semantic caching solves this by storing previous LLM prompts and completions as vector embeddings. When a new query arrives, we check if it is semantically close enough to a previously cached query. If it falls within a specific similarity threshold, we serve the cached answer instantly, reducing latency to < 50ms and token costs to ₹0.

The Semantic Cache Architecture

A production-ready semantic cache consists of three core components:

1. Embedding Generator: Converts incoming prompts into vector representations (e.g., using text-embedding-3-small).
2. Vector Index: Performs fast similarity search (e.g., using pgvector with an HNSW index, or a Redis Vector Search index).
3. Evaluation Node: Computes the exact similarity score and determines whether to trigger a cache hit or fetch from the model.

Incoming Query ──> [ Embedding Model ] ──> Vector Representation
                                                    │
                                                    ▼
Cached LLM Answer <── [ Similarity Check ] <── [ Vector Index Search ]
(Similarity >= 0.92)         │
                             └──> [ Similarity < 0.92 ] ──> Route to LLM

Setting the Similarity Threshold

Determining the threshold for a cache hit is a delicate balance: - Too high (e.g., > 0.95): You will rarely get cache hits, defeating the purpose of the cache. - Too low (e.g., < 0.88): You will trigger false cache hits, serving irrelevant cached responses to users.

For cosine distance on OpenAI embeddings, the optimal threshold for general Q&A is usually between 0.91 and 0.93.

Step-by-Step Implementation with pgvector

Here is how to set up a semantic cache using PostgreSQL and the pgvector extension.

1. Database Schema

First, we create a table to store prompts, completions, and their corresponding embedding vectors.

-- Enable the vector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create the cache table CREATE TABLE IF NOT EXISTS semantic_cache ( id SERIAL PRIMARY KEY, prompt TEXT NOT NULL, completion TEXT NOT NULL, embedding VECTOR(1536) NOT NULL, created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP );

-- Build an HNSW index for cosine distance searches CREATE INDEX ON semantic_cache USING hnsw (embedding vector_cosine_ops); ```

2. Python Cache Manager

Here is the implementation of the cache lookup and write logic using Python and psycopg2 / pgvector.

import numpy as np
from openai import OpenAI
import psycopg2
from psycopg2.extras import RealDictCursor

client = OpenAI() conn = psycopg2.connect("postgresql://user:pass@localhost:5432/ai_db", cursor_factory=RealDictCursor)

def get_embedding(text: str) -> list[float]: response = client.embeddings.create( model="text-embedding-3-small", input=text ) return response.data[0].embedding

def check_cache(prompt: str, threshold: float = 0.92) -> str | None: embedding = get_embedding(prompt) with conn.cursor() as cur: # pgvector uses <=> for cosine distance (1 - cosine_similarity) # So we check if cosine_distance <= (1 - threshold) cur.execute(""" SELECT prompt, completion, (1 - (embedding <=> %s::vector)) as similarity FROM semantic_cache WHERE (1 - (embedding <=> %s::vector)) >= %s ORDER BY embedding <=> %s::vector LIMIT 1; """, (embedding, embedding, threshold, embedding)) row = cur.fetchone() if row: print(f"Cache Hit! Similarity: {row['similarity']:.4f}") return row['completion'] return None

def write_to_cache(prompt: str, completion: str): embedding = get_embedding(prompt) with conn.cursor() as cur: cur.execute(""" INSERT INTO semantic_cache (prompt, completion, embedding) VALUES (%s, %s, %s::vector); """, (prompt, completion, embedding)) conn.commit() ```

Cache Invalidation Strategies

A static semantic cache can serve stale answers if your underlying database changes. Implement these three eviction strategies:

1. TTL (Time to Live): Clear entries older than 7 days using an automated Cron job.
2. Explicit Eviction: When updating articles or course materials, run a vector similarity match on the cache and delete cache records matching the updated topic scope.
3. Dynamic Feedback: If a user clicks a thumbs-down button on a response, immediately delete the cache entry to prevent serving a bad response to subsequent queries.

By combining similarity checks with proper invalidation rules, you can scale your AI agent workflows confidently while keeping operational costs virtually flat.