A LangGraph agent that plans, decomposes, retrieves, reasons, and checks its own draft for hallucination before answering questions about SEC filings. Hybrid dense and sparse retrieval fused with reciprocal rank fusion and cross-encoder reranking, a 3-tier token budget engineered to survive Groq free-tier rate limits, and a RAGAS evaluation that reports its own quality gate failing rather than hiding it.
≥ 0.70 faithfulness
ragas_evaluator.py, checked after every run
0.60 faithfulness
3-question RAGAS run, real CSV output
behaviorally correct
hedged, flagged, and refused, exactly as intended
Two of three low scores come from RAGAS metrics penalizing correct hedging and correct refusal, not from the agent doing anything wrong.
Nodes in the state machine
plan, rewrite, retrieve, reason, reflect, respond
Budget degradation tiers
GREEN / YELLOW / RED, tuned for Groq free tier
Retrieval signals fused
dense (Qdrant) + sparse (BM25) via RRF, k=60
Independent cache layers
semantic response, retrieval result, embedding LRU
Session token budget
GREEN 0 to 60%, YELLOW 60 to 85%, RED 85%+
Semantic cache threshold
cosine similarity, full pipeline bypass on hit
Chunks indexed
single NVIDIA FY2026 10-K, current corpus footprint
Reflect loops at full budget
GREEN tier only, see the budget panel below
Embedding a question and returning the nearest chunk fails on financial documents in three specific, recurring ways. Multi-hop questions, identify the company that acquired Figma, then its CEO, then the language that CEO's company built, depend on facts that only exist after an earlier hop resolves, so a single vector search over the raw question returns nothing useful for the second or third fact.
Naive RAG also has no mechanism to catch itself fabricating a plausible dollar figure, which is the worst failure mode in a domain where the reader has no easy way to spot it. And a design that always calls the largest available model on every step of a multi-loop agent will hit a free-tier rate limit mid-session and simply stop answering, a constraint this project hits repeatedly on Groq's free tier, not a hypothetical one.
| Capability | Naive RAG | LangChain / LlamaIndex RAG | FinSight-Alpha |
|---|---|---|---|
| Retrieval | Single-query dense only | Dense or sparse | Hybrid dense + sparse + RRF + cross-encoder |
| Query handling | Single pass | Single pass | Multi-hop decomposition with planning |
| Self-correction | None | None | Reflector, then a conditional routing loop |
| Token management | None | None | 3-tier budget plus adaptive model routing |
| Caching | None | Basic | 3-layer: semantic, result, embedding |
| Error handling | Crash | Exception bubbles | Graceful degradation, fallback LLM |
| Evaluation | None | Optional | Built-in RAGAS: faithfulness + relevancy |
The project's own comparative framing (README section 6.3), not an independently benchmarked result.
NVIDIA's FY2026 10-K, the raw SEC EDGAR submission format: a multi-document container with embedded HTML and inline XBRL, not a pre-cleaned PDF.
The current corpus is intentionally small: 288 chunks from a single filing, one Qdrant collection. The parsing, chunking, retrieval, agent loop, and evaluation harness are all built to be dataset-agnostic and multi-collection, five parser formats already supported (SEC EDGAR, PDF, HTML, text, JSONL), but the data footprint on disk today is a single-filing proof of the architecture, not a funded ingestion effort.
Every retained chunk carries a 12-field metadata object, written to the Qdrant payload for provenance and future filtering. It is not yet used to narrow a query at retrieval time, that metadata exists today for citation display, not active filtering.
Verified directly against git history, not the README: the parsing layer was rebuilt once to fix a real production incident, and again to generalize the design.
partition_html / partition_text / chunk_by_title off the shelf, no custom noise filtering.
Produced a poisoned index: HTML remnants and XBRL schema noise made it into the live vector store.
A from-scratch BeautifulSoup rewrite with explicit XBRL stripping, plus rebuild_index.py to wipe and reindex from raw source.
rebuild_index.py's own docstring: "Run this after fixing the document processor to replace the poisoned index."
Strategy-pattern parser registry (SEC / PDF / HTML / text), content-hash incremental indexing, multi-collection isolation, async batch embedding.
Live today. document_processor.py still sits in the repo, no longer imported by the pipeline.
Six LangGraph nodes, Plan, Rewrite, Retrieve, Reason, Reflect, Respond. The Reflector can route the graph back for another pass before anything reaches the user.
$ build_graph().compile()
Planner
Llama 3.1 8B
Formulates a 1 to 3 step research plan from the raw query
Skipped entirely under YELLOW / RED, saves about 800 tokens
Query Rewriter
Llama 3.1 8B
Decomposes the query into atomic, sequentially-ordered sub-queries
Reflection feedback from a prior loop is injected here
Retriever
HybridRetriever
Runs hybrid search per sub-query, deduplicates by exact text match
top_n set by the current budget tier: 8, 5, or 4
Reasoner
Llama 3.3 70B (or 8B under RED)
Synthesizes a cited draft answer from every accumulated chunk
Every claim must carry a [Doc X: source] citation
Reflector
Llama 3.1 8B
Checks the draft for hallucination and missing coverage
Returns strict JSON: is_grounded, needs_more_info, feedback
Responder
Llama 3.3 70B (fallback only)
Packages the verified draft, or degrades gracefully on error
On any upstream error: direct LLM answer, no RAG, no citations
is_grounded = true, or loop_count reaches the tier max
→ Responder, finalize
needs_more_info = true
→ Query Rewriter, fetch more context
is_grounded = false, needs_more_info = false
→ Reasoner, re-answer on the same context
error set at any node, checked first, every time
→ Responder, direct fallback LLM
1def route_reflection(state):2 if state.get("error"):3 return "responder" # global bypass, checked first4 5 is_grounded = state.get("is_grounded", False)6 loop_count = state.get("loop_count", 0)7 max_loops = budget_manager.get_max_iterations() // 28 9 if is_grounded or loop_count >= max_loops:10 return "responder"11 if "needs_retrieval" in reflection:12 return "query_rewriter"13 return "reasoner"Dense embedding search finds paraphrases. Sparse BM25 finds exact figures and accession numbers. Reciprocal rank fusion combines both without needing to normalize two incomparable scales.
HybridRetriever.search(query)
all-MiniLM-L6-v2 embeds the query, Qdrant cosine search
384-dim, fetch_k = 50
BM25Okapi over text.lower().split() tokens
fetch_k = 50, no stemming
Reciprocal Rank Fusion merges both ranked lists
score = sum of 1 / (k + rank + 1), k = 60
Cross-encoder jointly scores each query/chunk pair
ms-marco-MiniLM-L-6-v2, top ~20 candidates only
Reranking only runs on the fused candidate set, roughly 20 chunks, not the full fetch_k pool from either path. The cross-encoder jointly encodes each query and chunk pair, which is strictly more expensive than comparing independent embeddings, so it is deliberately scoped to a small candidate set rather than the full corpus.
A 3-tier token budget that degrades the pipeline, not the request, as a session's cumulative cost climbs. Every threshold is a real, tested value in token_budget.py.
Llama 3.3 70B
Llama 3.3 70B
Llama 3.1 8B (all tasks)
max_iterations is the tuning knob in config.py. The reflect loop cap the router actually enforces is max_iterations divided by 2, integer division, which means YELLOW and RED both cap at a single reflect loop, not three. Only GREEN gets the full 3-loop self-correction budget.
query arrives at POST /chat
Cosine sim >= 0.92 against all cached embeddings in one np.dot() call. Full pipeline bypass on hit: zero LLM calls, zero retrieval. 500 entries, 1 hour TTL, thread-safe RLock.
Avoids repeating dense + sparse + rerank for the same query within a session. MD5 key, 5 minute TTL.
OrderedDict, O(1) lookup and eviction, 256 entry cap. Avoids re-encoding the same query string across agent loops.
The RAGAS test suite hardcoded in the evaluator is small, three questions, deliberately chosen to stress three different things rather than three similar ones: a real retrieval question about supply-chain strategy, an adversarial arithmetic question built on a premise that does not appear in any filing, and a hallucination trap asking about a corporate acquisition that never happened.
n=3 is not statistically meaningful, and the project's own README says so. What it is meaningful for is exposing exactly how a hallucination-averse agent should be graded, which turns out to be more interesting than the raw average suggests.
The evaluator's own quality gate requires average faithfulness of at least 0.70. This run averages roughly 0.60. Every individual answer was still the behaviorally correct one.
“What are NVIDIA's primary strategies for mitigating supply chain constraints?”
The agent hedged correctly: it cited related-but-partial context (supplier diversification, domestic manufacturing) instead of inventing a clean answer. RAGAS's AnswerRelevancy metric generates synthetic questions from the answer and compares them to the query, and a hedged answer scores near zero even when the hedge is the right call.
The source repo's own Streamlit dashboard ships mock data for its charts, self-documented as such. This case study checks every claim against the code and excludes anything that is not real.
langgraph_agent.py, build_graph() / route_reflection()
hybrid_retriever.py
data/reports/ragas_evaluation_report.csv
git commit 0a28963, rebuild_index.py docstring
data/processed/, data/.registry_sec_filings.json
src/ui/components/data.py, self-documented as demo data
README framing; the class is real and tested but is never called from the live agent path
What was chosen, why, what it cost, and what happened. Specific to this project, not a repeat of the portfolio's global decision log.
Recovered from git history, not the README: what broke, how it was found, and what changed.
The poisoned index: a parsing library let noise into the vector store
The unstructured-library-based parser produced chunks contaminated with HTML remnants and XBRL schema fragments, degrading retrieval quality in a way only visible as a pattern across many queries, not any single obviously-broken result.
A full rewrite of the parsing layer with BeautifulSoup and explicit XBRL stripping, plus a dedicated rebuild_index.py script whose entire job was wiping data/processed/ and rebuilding Qdrant and BM25 from the raw source files.
A silent context-formatting bug: literal backslash-n instead of real newlines
The reasoner built its context string with escaped "\n" characters standing in for line breaks, not actual newlines. No exception, no failed test, just a harder-to-parse block of text handed to the model.
Extracted a _format_context() helper using real newlines and a clear "---" separator between chunks, fixed in the same audit commit that caught the other three issues below.
An overly conservative context window was quietly starving the reasoner
context_top_k=3, max_chunk_chars=800, a 2000 token cap, and a 0.25 relevance floor, tuned defensively for token conservation at the cost of answer quality.
Loosened to top_k=8, 1500 characters, a 6000 token cap, and a 0.15 floor, alongside raising the per-tier retrieval top_n (RED 2 to 4, YELLOW 3 to 5, GREEN 5 to 8).
DynamicContextWindow: instantiated, retuned, never actually called
A commit literally titled "...DynamicContextWindow never integrated" added the module's import and object, and retuned its config thresholds, but never added the call site that would make it run inside the reasoning path.
Partially fixed. As of the current codebase, the class is real, unit-tested, and correct in isolation, but reasoner_node still formats the raw, unfiltered chunk list directly. Disclosed here as a still-open gap rather than smoothed over.
The ingestion pipeline was rewritten a second time, for a different reason
After the poisoned-index fix stabilized correctness, the hand-rolled DocumentProcessor was still a single-purpose tool: one format detector, no incremental indexing, no multi-collection support.
Replaced again with a Strategy-pattern ParserRegistry, an async IngestionPipeline, content-hash incremental indexing, and multi-collection isolation via CollectionManager, the architecture live today.
Every parameter moved in the same direction, toward more context, in the same audit commit. Strong circumstantial evidence the original defaults were tuned defensively for token conservation, at a real cost to answer quality.
The dashboard is live. The agent needs a local Groq key and a GPU.