LangChain vs LlamaIndex vs Custom RAG: What Actually Differs When You Build the Same Bot 3 Times

We did LangChain vs LlamaIndex vs Custom RAG. Our first scorecard said the frameworks hallucinated 83 to 92% of the time. It was wrong, and so was the judge that produced it.

We caught it by reading the answers by hand. That correction set the tone for the whole project: the real differences between LangChain, LlamaIndex, and plain code were not where the comparison posts point.

How We Compared LangChain, LlamaIndex and Custom RAG

We built the same documentation Q&A bot three times: plain SDK code, LangChain and LlamaIndex. The corpus was 20,882 chunks from the docs of 7 Python libraries, and we asked 60 questions. Then we measured it the way a production team would, with 836 paid API calls that cost about $1.13 (an estimate).

We ran two modes. In controlled mode, all three got identical prompts and identical retrieval, the step that finds the doc passages most relevant to a question. In default mode, each framework used its own quickstart settings.

The central result is simple. Given identical inputs, the three sent the same request to the model, byte for byte, and gave the same answer quality. What separated them was their defaults, their behavior when things fail, and the data underneath.

Do LangChain, LlamaIndex and Custom RAG Give the Same Answers?

Yes. With identical prompts and retrieval, all three sent the same request and gave the same answer quality.

Before comparing anything, we ran a parity check, meaning we proved all three retrieve the same chunks and send the same request. It failed on 1 of 60 questions.

The culprit was not a framework. It was Chroma, our vector store: 402 vectors had never been written to disk, so each process rebuilt them slightly differently. We wrote them out, and parity passed 60 of 60.

Related Read: Pinecone vs Weaviate vs Pgvector

If your results look random, check the storage layer before you blame the library. 🤔

Hidden Defaults in LangChain and LlamaIndex That Change Your Results

Both frameworks silently changed temperature for gpt-5.x models (as OpenAI reasoning models don’t support the temperature param), LlamaIndex’s default prompt cut abstentions to 6 of 12, and LangChain’s capped answers at three sentences.

Nobody ships a framework with identical inputs. They ship its defaults, so that is where we looked next.

The quiet one first. Both frameworks silently changed temperature=0 for gpt-5.x models: LangChain drops it, LlamaIndex forces 1.0. We had to set it again after construction.

# LangChain: we had to set temperature again after construction
self.llm = ChatOpenAI(model=config.LLM_MODEL, temperature=0,
                      max_retries=2, timeout=30)
self.llm.temperature = 0  # ChatOpenAI silently drops temperature != 1 for gpt-5.x

# LangChain: we had to set temperature again after construction
self.llm = ChatOpenAI(model=config.LLM_MODEL, temperature=0,
                      max_retries=2, timeout=30)
self.llm.temperature = 0  # ChatOpenAI silently drops temperature != 1 for gpt-5.x

A team that never checked would run at temperature 1, thinking it was 0.

The louder default is a prompt. LlamaIndex’s quickstart is one line, as_query_engine, and it brings its own system prompt.

LlamaIndex default system prompt (excerpt):
"""Always answer the query using the provided context information, and not prior knowledge.
1. Never directly reference the given context in your answer.
2. Avoid statements like 'Based on the context, ...'"""

LlamaIndex default system prompt (excerpt):
"""Always answer the query using the provided context information, and not prior knowledge.
1. Never directly reference the given context in your answer.
2. Avoid statements like 'Based on the context, ...'"""

That pushes the model away from admitting the docs are silent. On our 12 unanswerable questions, LlamaIndex abstained, meaning it said it did not know, on only 6. Plain code abstained on 11 and LangChain on 9. With our shared prompt, all three managed 11.

Read by hand, most of LlamaIndex’s misses were confident “No, it doesn’t support X” answers. One was a plain invention: a recipe for composite primary keys that the docs never describe.

# Shared prompt so all three approaches are compared fairly

SYSTEM_PROMPT = (
    "You are a technical documentation assistant. "
    "Answer the question using ONLY the provided context. "
    "If the context does not contain the answer, say you don't know. "
    "Include short code examples when the context has them."
)

USER_PROMPT_TEMPLATE = """Context:
{context}

Question: {question}

Answer:"""

# Shared prompt so all three approaches are compared fairly

SYSTEM_PROMPT = (
    "You are a technical documentation assistant. "
    "Answer the question using ONLY the provided context. "
    "If the context does not contain the answer, say you don't know. "
    "Include short code examples when the context has them."
)

USER_PROMPT_TEMPLATE = """Context:
{context}

Question: {question}

Answer:"""

LangChain has its own default. Its quickstart prompt says “three sentences maximum”, so its answers averaged 276 characters against 562 for plain code.

Can You Trust an LLM Judge to Evaluate RAG?

Not blindly. Our first rubric flagged honest ‘the docs don’t say’ answers as hallucinations, which produced 83 to 92%. Read a sample of what a judge flags before trusting any score.

It had lumped two different things together: saying something is not documented, and asserting a feature the docs never describe. The new rubric separates them, and we judged everything again.

We also threw out our first batch of unanswerable questions, because only 5 of the 12 were distinct and 2 were actually answerable. The lesson is not to avoid LLM judges. Read a sample of what they flag before you believe a percentage, and remember that ours is still one judge we have not calibrated.

How Each Stack Handles Retries and Errors

With 2 retries configured, plain code and LangChain make 3 attempts and LlamaIndex makes 6. LangChain silently accepts a misspelled collection name, while the other two raise an error.

We configured 2 retries everywhere. Plain code and LangChain made 3 HTTP attempts, as expected. LlamaIndex made 6, because it retries in two layers, and it spent 5.9 seconds on a persistent 500 error against 3.0 and 2.4 for the others.

That rides out a rate limit burst and also doubles the load on an API that is already struggling. Neither is wrong, but choose it on purpose.

Then we misspelled a collection name on purpose. Plain code and LlamaIndex raised a loud error. LangChain did not

# LangChain: a misspelled collection name raises no error
Chroma(collection_name="tech_docs_missing", persist_directory="chroma_db",
       embedding_function=embeddings)  # silently creates an empty collection

# LangChain: a misspelled collection name raises no error
Chroma(collection_name="tech_docs_missing", persist_directory="chroma_db",
       embedding_function=embeddings)  # silently creates an empty collection

A typo in config becomes empty results and no error, which is the worst kind of bug to find in production.

One more, from the load test. At 8 concurrent async requests, LlamaIndex managed 2.4 per second, below its own result at 4. The test used a fake LLM, so it shows framework limits, not OpenAI rate limits.

LangChain vs LlamaIndex vs Custom RAG: Performance and Size

Quality is a tie. Framework overhead is 4 to 24 ms, install size grows by 25 to 84 MB, and about 92% of request time is the LLM call.

About 92% of each request’s time is the LLM call itself, so those milliseconds of overhead vanish. Code size was a wash, and the extra install is small next to the 1.3 GB baseline.

Does Data Quality Matter More Than the RAG Framework?

In our test, yes. Two-thirds of the index was translated copies of the FastAPI docs, and the data moved results more than the framework did.

FastAPI’s retrieval score looked terrible, with the right chunk in the top five only 14% of the time. Yet 86 to 93% of its answers were correct, because the English original was retrieved instead of the translated one we had marked as the answer.

The sneakiest case: we swapped in a different embedding model of the same size. Retrieval overlap fell to 1 chunk in 5, and nothing warned us. The answers still looked fine, because the LLM already knows FastAPI.

If your eval questions are about things the model already knows, your eval cannot see a broken retriever.

Which Should You Choose: LangChain, LlamaIndex or Custom RAG?

Custom code wins when you want control and predictable production behavior. Retries do what you configured, tracebacks are short (7 frames, against 10 for LangChain and 19 to 20 for LlamaIndex), and overhead is lowest. You build your own observability.

LangChain wins when the ecosystem and debugging visibility matter. Its bundled debug output was the best we saw, showing prompt, timings, tokens, and a nested trace. Watch the silent parameter rewrites, the auto-created collections, and that three-sentence prompt.

LlamaIndex wins for retrieval-heavy products. It ranked slightly better by default, and you will replace its defaults anyway. Swap the Q&A prompt, check the double retries, and load test async before you rely on it.

All of this depends on your team, and none of it makes a winner. Our evidence comes with limits: a small synthetic test set, one run on one laptop, and these exact library versions. Treat 0.84 versus 0.86 as a tie, and treat 6 attempts or 2.4 per second as things to verify on your own stack.

Pick the stack your team can debug, then read its default prompt before you ship. The framework rarely changes your answers. What it quietly assumes will.

Repo Link here 🙂