KB Agent System Design — arxiv-sanity Style

Original Home

Zero-infrastructure KB Agent for Banking Voice Agents

Pickle everything at build time. Load into memory at startup. Stateless workers for K8s/Docker scale-out. No Redis, no MongoDB, no vector DB.

Design Mapping

arxiv-sanity

db.p (papers)

tfidf.p, tfidf_meta.p

sim_dict.p

SEARCH_DICT

SQLite (users)

MongoDB (comments)

KB Agent

kb.p (structured KB)

bm25.p, meta.p

sim.p

BM25_INDEX (inverted)

SQLite (sessions, ephemeral)

— (dropped)

1. Build Pipeline (Once)

flowchart LR subgraph Input["Input"] MD[KB Markdown Files] end subgraph Build["build_kb.py"] Parse[parse_kb_files] Embed[build_embeddings] BM25[build_bm25_index] Sim[build_similarities] end subgraph Output["Pickles"] KBP[kb.p] CP[chunks.p] EP[embeddings.npy] BP[bm25.p] SP[sim.p] MP[meta.p] end MD --> Parse Parse --> KBP Parse --> CP CP --> Embed --> EP CP --> BM25 --> BP EP --> Sim --> SP BM25 --> MP

# Like arxiv-sanity pipeline:
# fetch_papers.py  → parse_kb.py
# analyze.py       → build_bm25_index()
# make_cache.py    → build_all()

python build_kb.py --kb-dir ./kb_files --output ./data

# Output:
# data/kb.p           ← Structured KB (product:field → value)
# data/chunks.p       ← KB chunks for RAG
# data/embeddings.npy ← Precomputed vectors
# data/bm25.p         ← BM25 inverted index
# data/sim.p          ← Precomputed similarities
# data/meta.p         ← Vocab, IDF, config

2. Runtime Architecture

flowchart TB subgraph Docker["Docker Container / K8s Pod"] subgraph Startup["Startup"] Load["load_data()
KB = pickle.load()
BM25_INDEX = pickle.load()"] end subgraph Runtime["Runtime"] Query[Query] Router["QueryRouter"] H1["CRITICAL
KB dict O(1)"] H2["ESCALATE
Template"] H3["PROCEDURAL
BM25 search"] Resp[Response] end subgraph Storage["Ephemeral"] SQL["/tmp/sessions.db"] end Load --> Router Query --> Router Router --> H1 --> Resp Router --> H2 --> Resp Router --> H3 --> Resp Resp --> SQL end subgraph Scale["K8s Scale-Out"] Pod1["Pod 1"] Pod2["Pod 2"] Pod3["Pod N"] end Docker -.-> Pod1 Docker -.-> Pod2 Docker -.-> Pod3

3. Query Flow Sequence

sequenceDiagram autonumber participant U as User participant F as Flask participant R as Router participant KB as KB Dict participant BM as BM25_INDEX participant SQL as SQLite Note over F: Startup: load_data() F->>KB: pickle.load(kb.p) F->>BM: pickle.load(bm25.p) Note over F: All in memory U->>F: POST /query F->>R: route(query) R->>R: Pattern match alt CRITICAL + product R->>KB: KB["be_cake:interest_rate"] KB-->>R: KBEntry("44%/năm") R-->>F: Template response Note over R,KB: O(1) dict lookup end alt CRITICAL no product R-->>F: "Thẻ nào ạ?" end alt ESCALATE R-->>F: "Chuyển chuyên viên?" end alt PROCEDURAL R->>BM: search_bm25(query) Note over BM: Inverted index
O(terms) BM-->>R: chunks[] R->>R: verify_numbers() R-->>F: RAG response end F->>SQL: log_query() F-->>U: JSON response

4. Memory Layout (Per Worker)

┌─────────────────────────────────────────────────────────────────┐
│                     Python Process Memory                        │
├─────────────────────────────────────────────────────────────────┤
│  GLOBALS (loaded once at startup)                                │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  KB: dict[str, KBEntry]           ~1 KB                  │   │
│  │      "be_cake:interest_rate" → KBEntry(display="44%/năm")│   │
│  └──────────────────────────────────────────────────────────┘   │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  CHUNKS: list[Chunk]              ~10-100 KB             │   │
│  └──────────────────────────────────────────────────────────┘   │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  EMBEDDINGS: np.ndarray           ~10-50 MB              │   │
│  │      shape: (num_chunks, 384 or 1536)                    │   │
│  └──────────────────────────────────────────────────────────┘   │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  BM25_INDEX: dict[str, list]      ~1-5 MB                │   │
│  │      "lãi" → [(idx=0, score=2.3), ...]                   │   │
│  │      (inverted index, like SEARCH_DICT but inverted)     │   │
│  └──────────────────────────────────────────────────────────┘   │
├─────────────────────────────────────────────────────────────────┤
│  Total: ~50-100 MB per worker                                   │
│  ✓ Fits in memory    ✓ No DB calls    ✓ Workers independent    │
└─────────────────────────────────────────────────────────────────┘

5. BM25 Inverted Index

❌ arxiv-sanity (linear)

def papers_search(qraw):
  for pid,p in db.items():
    score = sum(
      SEARCH_DICT[pid].get(q,0)
      for q in qparts
    )
# O(num_docs) per query

✓ KB Agent (inverted)

def search_bm25(query):
  for token in tokens:
    for idx, score in BM25_INDEX[token]:
      scores[idx] += score

# O(terms × posting_len)

6. Deployment

flowchart TB subgraph Build["Build Stage"] KB[kb_files/*.md] BP[build_kb.py] Data[data/*.p] KB --> BP --> Data end subgraph Image["Docker Image"] Data2["/app/data/*.p"] Server[server.py] Gun["gunicorn --workers 4"] end subgraph K8s["Kubernetes"] HPA["HPA 2-20"] Deploy["Deployment"] Svc["Service"] Ing["Ingress"] HPA --> Deploy --> Svc --> Ing end Data --> Data2 --> Server --> Gun --> Deploy

# Build & Deploy
docker build -t kb-agent .
kubectl apply -f deployment.yaml
kubectl scale deployment kb-agent --replicas=10

# Each pod independent:
# - Same pickles from image
# - No shared state
# - SQLite ephemeral
# - Scale by replicas

7. Latency

CRITICAL

0.2ms

Dict lookup O(1)

ESCALATE

0.05ms

Template only

PROCEDURAL

1-5ms

BM25 + verify

Key Insight

Karpathy's arxiv-sanity works because he accepted constraints: single server, papers fit in memory, batch updates daily. Same philosophy adapted for stateless scale-out.

Constraints accepted:

Single server → Stateless workers
MongoDB comments → Dropped
In-memory rate limits → Accept reset

P(works) = P(fits_memory) × P(stateless) × P(no_deps) = 97%