All projects

Case study

2024 – 2025

Enterprise Knowledge RAG

Production Jira/Confluence RAG preferred over the incumbent tool 240–30–13 in a blind pairwise evaluation over 283 real questions.

Read the repository 

My role

Built the retrieval pipeline, the citation construction, the async serving layer and the blind evaluation that cleared it for production.

Designed and built the system end to end — retrieval pipeline, citation construction, async serving layer and the blind evaluation that cleared it for production.

Blind pairwise, LLM judge
240–30–13Blind pairwise, LLM judge
Human expert judge
246–34–3Human expert judge
Judge agreement
91.5%Judge agreement
Jira issues indexed
30–60KJira issues indexed

The problem

The company's operational knowledge lived in Jira and Confluence. Answering a routine "how does this flow work" question meant interrupting an engineer who happened to remember the right ticket, and the existing search tool returned documents rather than answers.

What I built

A production retrieval-augmented generation service over the Jira/Confluence corpus: LangChain + LangGraph over Pinecone with dual knowledge bases, migrated off ChromaDB once the corpus outgrew it. It indexes 30–60K Jira issues across an enumeration reaching 40 projects, and answers with numbered source citations.

The citations are generated by construction: the answer is assembled from retrieved chunks that carry their own source identity, so a citation cannot reference a document that was never retrieved. Hallucinated references are prevented structurally instead of being detected afterwards.

How it works

Serving is async Flask with Celery workers and CI deployment. Conversation state uses Redis-cached rolling summarisation so long threads stay inside the context budget; a database-backed feedback loop and latency analytics record what users actually accepted, and the retrieval configuration is tuned against that record.

Evidence

The system was validated by blind pairwise evaluation over 283 real questions against the incumbent tool: preferred 240–30–13 by an LLM judge and 246–34–3 by a human domain expert, with the two judges agreeing 91.5% of the time.

One measurement is worth reporting because it went the other way. An early corpus count of ~178K "tickets" was wrong — raw JQL had been interpolated into a URL without percent-encoding, and the pagination bug inflated the row count roughly 50×. The real figure is 30–60K distinct issues. I corrected the number rather than keeping the impressive one.

What I owned

I designed and built this system end to end: the retrieval pipeline and the ChromaDB → Pinecone migration, the by-construction citation layer, the async Flask + Celery serving path with its Redis-cached rolling summarisation, and the blind pairwise evaluation harness that decided whether it was allowed to replace the incumbent tool.

Stack

LangChain · LangGraph · Pinecone (migrated from ChromaDB) · Flask (async) · Celery · Redis · Python

 All projectsAsk me about this