Traffic Law RAG
A question-answering tool for Taiwan's traffic laws. Ask in plain Chinese and get an answer grounded only in the statute text, with citations for every claim or a clear refusal.
- Python / FastAPI
- Chroma
- Voyage embeddings
- Claude API
- SSE
- React / TypeScript
- pytest
- Vitest
- Playwright
- Docker
- GitHub Actions
- Role
- Solo, end to end (spec, retrieval and evaluation, backend, frontend, deployment)
- articles across five regulations
- 462
- evaluation questions
- 40
- automated tests
- 442
- Architecture Decision Records
- 9
Why I built it
Laws are written in legal language; people ask in everyday words. 「紅燈右轉罰多少？」 shares almost no keywords with the article that answers it. A general chatbot may answer anyway, sometimes with an outdated or invented fine. I wanted each answer to be checkable: the source article sits beside the answer, every number must appear in the cited text, and “these laws don’t answer that” is an acceptable result.
Coming from frontend, I also wanted to own what I knew least: retrieval, evaluation, and keeping a public demo affordable when every question is billed to my own API keys.
What it does
- Streams answers in Traditional Chinese, with a citation badge such as
道交處罰條例 §53after each claim - Lists retrieved articles beside the answer; clicking one opens the full text and links to the official law database
- Shows the five pipeline steps: rewrite, retrieve, relevance check, generate, and verify, with status and timing
- Refuses out-of-scope questions and explains why, including the similarity score against the threshold
- Publishes evaluation metrics and all 40 expandable test questions
- Exposes the same pipeline to Claude Code as an MCP server with three read-only tools
Architecture
One Docker image runs FastAPI and serves the built React app, so the SSE stream needs no CORS. Law data comes from Taiwan’s open-data portal because the official site forbids scraping; it is indexed offline in Chroma.
flowchart LR
Data["data.gov.tw open data"] -.->|"parse, chunk, embed"| Index["Chroma index<br/>(GitHub Release)"]
UI["React SPA"] -->|"POST /api/ask"| Ret["Vector search, top 5"]
Index --> Ret
Ret --> Gate{"Top similarity ≥ 0.40?"}
Gate -- No --> Refuse["Refusal event"]
Gate -- Yes --> Gen["Claude reads whole articles"]
Gen -->|"[[REFUSE]]"| Refuse
Gen --> Val["Verify citations and numbers"]
Val -->|"SSE events"| UI
Refuse -->|"SSE events"| UI- Legal boundaries define chunks. Each article is one chunk. Long articles split only at paragraph or item boundaries, so every chunk still maps to an article number.
- The index is versioned outside the repository. Each data version is a GitHub Release. CD downloads the latest one, deploys it, and smoke-tests that the live data version and chunk count match.
Engineering highlights
A RAG demo can look convincing on five hand-picked questions, but choosing retrieval options required data. I built a 40-question evaluation set: 28 answerable questions and 12 that fall outside the covered laws. AI drafted the questions; I checked every expected answer against the statute text by hand. A change shipped only if it improved the results.
That rule ruled out hybrid search. The familiar vector-plus-BM25 combination dropped MRR from 0.845 to 0.749, partly because jieba’s default dictionary splits 罰鍰 into 罰 / 鍰. I also left query rewriting off: it helped four questions, hurt four, and increased token use by 27%.
The final retrieval results were Recall@5 of 1.000 and MRR of 0.845. They have an important limitation: the person writing the questions had read the laws. Perfect recall means this set was too easy, not that the retriever is perfect.
I expected a similarity threshold to distinguish in-scope from out-of-scope questions. Then I added seven questions that were about traffic but not covered by these laws: license-exam fees, parking-lot charges, and tolls. Their top scores reached 0.62, while some answerable questions scored as low as 0.42. No single threshold handled both cases: at 0.55, it wrongly refused three answerable questions and let one out-of-scope question through.
The 0.40 threshold now filters only clearly unrelated questions, returning in about 0.26 seconds without a model call. For the ambiguous cases, the model reads the retrieved articles and returns [[REFUSE]] when they are insufficient. All 12 out-of-scope questions were refused, six at each layer, and none of the 28 answerable questions was wrongly refused. One apparent failure was my own mislabeled example: the model found an article I had missed.
For a legal tool, a confident wrong fine or a made-up article is the most serious failure. The model uses [[article_id]] markers rather than Anthropic’s native Citations, which lets the application parse them during streaming and apply its own validation rules. Every cited ID must belong to the retrieved articles. Every amount is normalized, so 一千八百元 becomes 1,800 元, and must appear in the article it cites.
Hand-checking answers exposed another problem. When the model saw only the best-matching chunk, it claimed drunk driving had no base fine. I changed the prompt to send whole articles instead. Key-amount coverage rose from 0.894 to 0.985; citation and number correctness reached 1.000. The remaining gap is semantic: these checks establish that a citation exists, not that it supports the sentence. Citation precision is 0.653 because the model over-cites.
Streaming creates awkward failure modes: errors can arrive after a 200 response, people can close the tab mid-answer, and a citation marker can be split across chunks. I used SSE over POST, read with fetch and ReadableStream. EventSource cannot POST, and a two-way WebSocket channel was unnecessary.
A small state machine ensures that no text chunk contains half of a [[…]] marker; tests split a marker at every position. Failures become typed error events, and a disconnect closes the model stream so the user is not charged for an unread answer. The first token arrives in about one second and a full answer in about 2.4 seconds. If the connection fails, the UI preserves the partial answer and labels it incomplete.
How I worked
I wrote a 13-step implementation spec with acceptance criteria for each step, recorded nine ADRs, and built with Claude Code. AI wrote most of the code and tests; I chose the scope and data source, verified the evaluation answers, made the trade-offs, and accepted each step. Rather than reviewing backend code line by line, I used 442 tests, CI, and evaluation reruns after every prompt or retrieval change as the gate. A separate reviewer agent’s observation that the out-of-scope questions were too easy reshaped the evaluation set.