A JavaScript engineer learning to build AI systems · code

Why even do this?

I'd shipped small RAG features before, but always with a nagging sense that I was eyeballing quality rather than measuring it. This was the project where I went through the whole pipeline properly.

The case for taking RAG seriously is an economic one. Frontier context windows keep growing, which makes "just put everything in the prompt" tempting — but inference is being sold below cost right now, and that won't hold. One day API pricing corrects upward, and passing a few GBs of organisational data to answer a single question stops being viable. Latency and compute budgets don't care how big the window gets.

So I built a proper RAG rig — cleaning, chunking, retrieval, reranking, and an eval harness, because a RAG system without evals is just vibe coding. I fed it papers from my own PhD research, since on that material I could act as a reliable human in the loop and actually tell whether the retrieved context was right.

Here are the steps I went through before I had my RAG passing the correct context to my LLM, which meant I got the answer I was looking for.

01 — Cleaning the PDF

It was important to clean whatever I wanted to inject, because the whole purpose of RAG is to pass usable data to the LLM to parse and work with. And the output is only as good as the input, which is why sprucing up the text extracted from the PDF mattered so much. The concept isn't new, it's done across all DS and research domains — but what I did here was individual to my corpus and doesn't apply to all PDFs. That's the bit worth remembering: knowing what you're ingesting matters more than the specific fixes I reached for.

In my case, there were five separate passes run over every paper before it reaches the chunker, each fixing one specific thing the extraction would break:

1dehyphenate          "evalu-\nation"       -> "evaluation"
2strip_references     cuts from the LAST "References" heading onward
3find_repeated_lines  running headers/footers — detected, not hardcoded
4fix_mojibake         "–"   -> "–"
5tidy                 drops <mark> tags, page numbers, "Figure 3" placeholders

dehyphenate rejoins a word the PDF line-wrap split in half, but only when the letter after the break is lowercase — so a capital starting a new sentence right after a hyphen doesn't get glued to whatever came before it.

strip_references cuts from the last "References" heading, not the first, because papers cite the word "references" in body text and some have an appendix after the bibliography — cutting at the first match would have kept the appendix and dropped nothing.

find_repeated_lines doesn't hardcode a journal name or a header format. It counts short lines (under 90 characters) that show up on at least half the pages and drops those — because every journal's running header looks different, and a rule tuned to one paper misses the next.

→ src/parse.py — fix_mojibake(), dehyphenate(), strip_references(), find_repeated_lines(), tidy()

02 — Chunking

A chunk is just a smaller piece of a large body of content. And the whole purpose of RAG is to ensure big pieces of content are broken down into smaller chunks, so that the RAG engine can decide which exact chunks need to be sent alongside a query to the LLM, so that it can answer the question.

There's a mechanical reason too. You can't embed a whole paper — one vector for 12,000 tokens is an average of everything it says, which points at nothing in particular. So the paper gets cut into pieces, and where you cut matters more than how big the pieces are.

125 PDF pages -> 12,774 tokens -> 15 sections -> 41 chunks

Nothing downstream can fix bad chunks. If a chunk is incoherent, no amount of clever retrieval or prompting will rescue it — and nothing in the code will tell you it's incoherent.

So before spending a penny on embeddings, I printed fifty chunks at random to a file and read them.

That sounds unsophisticated and it's the highest-value thing I did all week. If a human can't tell what a chunk is about in isolation, the embedding model can't either. Every parsing bug in this post came from reading actual output, not from a summary statistic.

→ src/ingest.py --dry-run --sample 50

Split on headings first, so a chunk maps to a section of the paper. Then split anything still too big at paragraph boundaries. Both passes use splitters that ship with LangChain — MarkdownHeaderTextSplitter for the first, RecursiveCharacterTextSplitter for the second.

→ src/chunk.py — split_document()

The bit that tripped me up

1RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=75)

That's LangChain's RecursiveCharacterTextSplitter, and it has two settings that matter. chunk_size is the ceiling. chunk_overlap means each chunk carries the last ~75 tokens of the previous one — because a cut lands wherever the size limit falls, not where the meaning ends. Without overlap, a sentence straddling a boundary is severed and neither half makes sense on its own. Carrying a tail forward keeps it retrievable from either side, at the cost of storing some text twice.

That's the theory. On my corpus it did nothing at all — I checked every consecutive pair of chunks and not one shared a single word. The splitter carries overlap by keeping whole trailing splits while they fit inside the budget, and my paragraphs are 200–400 tokens against a 75-token budget. Every candidate was too big, so nothing was ever carried. A setting that looks configured and is inert.

The chunk_size is worse, because it's silently wrong rather than silently absent. length_function defaults to Python's len, so chunk_size=500 means 500 characters, not tokens.

On my prose, 500 tokens is about 370 words. 500 characters is about 95. Nothing warns you — you just get chunks a quarter of the intended size, and worse retrieval that you blame on something else.

The fix is to build it with .from_tiktoken_encoder(). That swaps the length function for one that counts tokens with the same tokeniser the embedding model uses, so chunk_size=500 finally means what you thought it meant — and it's measured in the same units the model will actually see.

→ src/chunk.py — from_tiktoken_encoder()

Why split on headings first

A fixed-size splitter cuts at token 500 whether or not token 500 is the middle of an argument. A heading is different — it's the one place in a document where the author has explicitly said new topic starts here. Splitting there gives you a boundary chosen by meaning rather than by arithmetic, and it costs nothing to find, because the structure is already sitting in the text.

Academic papers are close to the best case for this. Abstract, Method, Results, Discussion, in that order, every time, with sub-headings underneath. A chunk that comes out of that pass maps to a real section of a real paper rather than to a window that happened to land somewhere. A novel would give you almost none of this, which is the next thing I want to test.

The second thing it buys you is the heading path, carried as metadata onto every chunk — including the seven sub-chunks the recursive splitter later carves out of Results:

1paper 6 > Results > Attention checks and exclusions

So a fragment still knows where it came from, which is what I show alongside an answer's citations.

So the division of labour is: the header splitter decides where the cuts should go, a merge pass folds sections too small to stand alone into their neighbour, and only then does the recursive splitter enforce a size ceiling on whatever is still too big. Meaning first, size second.

→ src/chunk.py — merge_small_sections(), contextual_prefix()

Another framework bummer

LangChain's MarkdownHeaderTextSplitter calls .strip() on every line and rejoins them. That flattens code indentation, which I knew about. It also deletes every blank line, which I didn't:

1my parser produced:        103 paragraph breaks
2after the header splitter:   0    <- all gone

To see why that matters, you need to know how the splitter decides where to cut. It has a list of places it is willing to break, in order of preference:

11st choice   a blank line          (between paragraphs)
22nd choice   a single line break
33rd choice   a space               (between words)

It works down that list until the pieces are small enough. Blank lines are the good cut — they fall between complete thoughts.

But the blank lines had just been deleted, so the first choice matched nothing and it dropped to the second: single line breaks. In text pulled out of a PDF, those sit wherever the original page happened to wrap a line — not where a sentence ends, but wherever the column ran out of room for another word.

So my chunk boundaries were being decided by page layout. Half of them ended mid-sentence.

How I fixed it

Two changes. The first is a repair pass that runs after the header splitter and before the recursive one, putting the paragraph breaks back:

1re.compile(r"(?<=[.!?])[ \t]*\n(?=[ \t]*[A-Z(\[])")

A newline that follows sentence-ending punctuation and precedes a capital letter was a paragraph break before .strip() flattened it, so it becomes a blank line again. A newline in the middle of a sentence is a PDF line wrap and gets left alone — joining those would glue words together.

The second is belt and braces on the preference list. I put ". " in as the second choice, above the single line break:

1separators = ["\n\n", ". ", "\n", " ", ""]

So even where the repair pass misses one, the next-best cut is the end of a sentence rather than wherever the page happened to run out of room.

→ src/chunk.py — restore_paragraph_breaks()

03 — Embeddings — turning text into numbers

Embeddings turn human language (our chunks) into a long list of numbers (a vector) so computers can calculate the meaning of words, sentences, or code. Similar meanings get numbers that sit close together on an invisible mathematical map, even if the words are totally different.

I opted to use OpenAIEmbeddings(model="text-embedding-3-small"), wrapped in a function both the indexing script and the query path import.

1EMBED_MODEL = "text-embedding-3-small"
2EMBED_DIMS  = 1536

→ src/embed.py — get_embeddings()

Every chunk from the previous stage goes through this once, at index time, and the result is what actually gets stored — not the text. A quick mental model, if you haven't touched embeddings before: 1,536 numbers position a chunk in space so that chunks about similar things land near each other. Think of the numbers as GPS coordinates for ideas. Words used in similar ways (like "cat" and "kitten") get coordinates close to each other. Nobody chose what the 1,536 axes here mean; the model learned them from enormous amounts of text, and there's no axis you could point to and read off.

A closer look at how cosine distance works for words that don't share the same meaning

I'd assumed — wrongly — that embeddings group words by meaning, so I ran a handful of word pairs through get_embeddings() and looked at the cosine distance between them:

1hot ↔ warm         0.367
2hot ↔ cold         0.445
3hot ↔ freezing     0.650
4hot ↔ democracy    0.855

Important bit — the opposite of "hot" is its second-nearest neighbour — closer than "freezing", which means roughly the same thing. Embeddings capture what a word is about, not what it asserts: hot and cold appear in nearly identical sentences, so they land near each other. I kept this test in mind for the rest of the build — it's the reason a second retriever earns its place in section 06.

04 — Storing the vectors

I picked Postgres with the pgvector extension — one datastore for the vectors and the raw text, which matters later, because keyword search in section 06 needs the words, not the numbers. Setup was one line in schema.sql:

1CREATE EXTENSION IF NOT EXISTS vector;

→ db/schema.sql

I don't write the embed-and-insert code myself either — it's one call, get_vector_store().add_documents(chunks), in the ingestion script. LangChain's PGVector.add_documents() embeds each chunk's text through the shared get_embeddings() object and writes a row per chunk into langchain_pg_embedding: the vector, the raw text, and the metadata dict, all in the same table.

→ src/ingest.py — main()

The read path

05 — Speaking the language of the vectors

Every question I asked got converted into a vector too, before it could grab the correct chunks. I liked to think of it as a translation step — it's not ideal to answer a question in English when the question was actually asked in Polish ;)

This is where the read path crosses the same column the write path did, and it is the one rule you cannot break: the same embedding model must run on both.

It is a serialisation contract. Encode with one model, decode with another, and you get garbage — exactly like writing JSON and parsing it as MessagePack. The difference is that a codec mismatch throws immediately, and this one doesn't. The coordinates are simply unrelated, the results stop meaning anything, and you spend a day blaming your chunking.

That's why get_embeddings() from section 03 is the only place either path reads the model name — both indexing and querying import it. The defence is structural, not a note in a README.

→ src/answer.py — answer()

First things first, was I even getting the right chunks?

Before anything else, embed the question and ask the database for the single nearest chunk. If even the closest thing is far away (cosine distance), stop — return "not found" without ever calling the LLM.

1real questions            distance 0.35 – 0.77
2questions with no answer  distance 0.84 – 0.90
3threshold                          0.75

PS — that threshold isn't a number I liked the look of. It came from putting deliberately unanswerable questions into the evaluation set — medical, CSS, geography — and measuring where they land. There's a clean empty band between the two distributions.

This is the only guard in the system that doesn't depend on the model cooperating. "Answer only from context" is a request; a model handed irrelevant chunks and asked a confident question will often answer from training data anyway. Not making the call is a guarantee. It costs one embedding, at a fraction of a cent, and it happens before a single generation token is spent.

06 — Two searches, not one

Ok, now that we're confident the chunks returned aren't completely irrelevant, we bring in two retrievers.

1. vector_search() — the cosine-similarity search built in sections 03–05, reusing the same embeddings already sitting in pgvector. It's good at meaning, bad at literal tokens: "391.672" and "391" and every other number in the corpus sit close together in embedding space, because a number's exact value isn't what an embedding captures.

2. keyword_search() — Postgres full-text search, which has no concept of meaning at all. It shreds every chunk into stemmed words:

→ src/retrieve.py — vector_search(), keyword_search()

07 — Fusing them by rank

Needed because I now had two ranked lists of 50 chunks each, and their scores couldn't be compared. A cosine distance and a ts_rank are different units.

Easiest way to explain this is to imagine that two people recommend restaurants. Sponge rates out of 10, Bob out of 5 stars — averaging is meaningless. So ignore the ratings and use finishing position, like points in a race:

1              Sponge   Bob       total
2Taco Place     2nd  +  1st   ->  wins
3Sushi Bar      1st  +  3rd
4Pizza Joint    3rd  +  2nd

Taco Place wins — neither person's favourite, but both rate it. Sushi Bar was Sponge's number one and Bob put it last, so it loses.

That's Reciprocal Rank Fusion, and it's the formula I implemented: score = 1 / (60 + rank), summed across both retrievers' result lists. The 60 is there so first place isn't worth double second place — otherwise one retriever could veto the other. Agreement beats enthusiasm.

→ src/retrieve.py — reciprocal_rank_fusion(), hybrid_search()

What I learnt

Keyword search on its own is worse than vector on every metric, and misses seven questions entirely. Fusing it in still improves the result:

1              recall@1  recall@5     MRR   misses
2vector           0.500     0.725   0.606        0
3keyword          0.350     0.575   0.462        7
4both, fused      0.500     0.775   0.618        0

Adding a worse component made the whole better, because the two fail on different questions. That was not intuitive to me before measuring it.

08 — Actually reading the shortlist

Why it's needed: everything so far compared summaries written independently. Nothing has yet read the question against a chunk.

A chunk's 1,536 numbers were computed at index time, before any question existed, and are identical for every question ever asked. One fixed summary serving every possible query.

A cross-encoder reads the question and the chunk together. Two things go in, one number comes out:

1question + "Enjoyment had a stronger effect than Engagement..."   ->   +5.35
2question + "Attitude refers to an individual's evaluation..."     ->  -11.12
3question + "This AR experience featured a dinosaur model..."      ->  -11.41

Do that fifty times (we narrowed chunks down to 50 in RRF), sort, keep five. The model is ms-marco-MiniLM-L-6-v2 — 22 million parameters, 91 MB, running on my laptop (available on huggingface.co/cross-encoder/ms-marco-MiniLM-L6-v2). It was trained on half a million real Bing queries paired with passages, so "does this passage answer this query" is the only thing it does.

→ src/retrieve.py — rerank(), hybrid_reranked()

09 — The prompt, and checking its work

So now I had five chunks that should kinda contain the information the LLM needed to answer my question. And it had to do that without inventing anything.

The five survivors get numbered, each with its source, and separated clearly:

1[1] <paper title> — Results > HTMT matrix
2<the chunk text>
3
4---
5
6[2] ...

The [1] is an address. It's what lets the model cite, and — more importantly — what lets me verify in code afterwards. Five chunks went, so valid citations are 1–5:

1"Enjoyment mattered more [2][3]."   ->  ok
2"The answer is X [7]."              ->  HALLUCINATION, only 5 sent
3"Enjoyment mattered more."          ->  no citation, unverifiable

That middle case should be caught by a regex and a range check. No amount of prompt engineering gives you that guarantee.

The prompt itself is 128 tokens and never changes, so it goes first — stable content before variable content, which is what lets prompt caching work. Total per question: about 1,839 tokens in.

→ src/answer.py — SYSTEM, verify_citations()

Evals! Because RAG without evals is just vibecoding

10 — Building the scoreboard

Forty questions with known answers. For each, search, then record what position the correct chunk came back at. That one number gave me everything.

→ eval/run.py

recall@k

For each question: was the right chunk in the top k? Yes or no, averaged. No partial credit — at recall@5, position 1 and position 5 score the same.

MRR

Read it backwards. Rank — where the correct chunk came back. Reciprocal — one divided by that. Mean — averaged over every question.

1What 3D model was used?              rank 1  ->  1/1 = 1.00
2What threshold did loadings exceed?  rank 1  ->  1/1 = 1.00
3What was the chi-square statistic?   rank 3  ->  1/3 = 0.33
4Which model underpins the study?     rank 5  ->  1/5 = 0.20
5                                                 ─────────
6                                          ÷ 4  =     0.633

Read as "on average, how near the top was the right answer?" The reciprocal makes it steep where it matters: 2nd to 1st gains 0.50, 10th to 9th gains 0.01.

Why recall@5 lied to me

79 chunks, retrieving 5, is 6.3% of everything. The search barely has to try, so recall@5 pinned itself at 1.00 and stayed there.

A metric at its ceiling has nothing to say

1                    recall@1  recall@5     MRR
2before [1,1,3,5]        0.50      1.00   0.633
3after  [1,1,1,2]        0.75      1.00   0.875
4delta                  +0.25     +0.00  +0.242

That system clearly improved. recall@5 reports +0.00. As a headline metric it would have told me a real improvement did nothing.

11 — Why contextual retrieval backfired

This is the failure I learnt most from, because the technique is genuinely well-attested. Prepending a chunk's title is supposed to give it context it otherwise lacks.

With two documents, it makes every chunk in a paper start with the same fifteen tokens. That identical block becomes a large share of all 79 vectors, so chunks become more similar to each other — and almost every question I had required telling chunks apart within a paper, not telling the two papers apart.

→ src/chunk.py — contextual_prefix(), off by default

What I learnt

The technique works. It works on corpora with many documents, where distinguishing documents is the hard part. On mine it paid full price for a benefit I had no use for.

Nothing you read about RAG is true independent of your corpus. That's the lesson, and it cost me the experiment I was most confident about.

12 — Evaluating the evals — sometimes the metrics behave oddly

One category of question sat at an MRR of roughly 0.28 through every experiment. Four different techniques, no movement.

I built a theory. These were questions about findings — relationships between ideas rather than specific numbers — so the answers must be spread across multiple chunks, and no single chunk could ever rank first. Elegant, plausible, wrong.

Academic papers state each finding about four times: abstract, results, discussion, contributions. When I checked how many chunks legitimately answered one such question, the answer was five, at ranks 13, 17, 26 and 34.

My evaluation accepted exactly one of them. Retrieval was returning a completely correct chunk and being scored wrong for it. Worse, some of my questions were ambiguous rather than my labels — both papers ran attention checks, so pinning that question to one paper scored the other's equally-correct answer as zero. I fixed it by letting a label be a list of acceptable phrases instead of one:

1"must_contain": ["ELM", "Elaboration Likelihood Model"]

→ eval/run.py — acceptable_phrases(), is_correct() · eval/golden.json

What I learnt — the biggest finding in the project

1retrieval work    +0.040 MRR   four experiments
2measurement work  +0.146 MRR   zero code changed

Letting a question have several correct answers moved the score more than three times as much as every retrieval improvement combined. The system had always been that good. My ruler was wrong. And it reordered my conclusions — hybrid search had looked like an improvement over plain vector, and with corrected scoring its MRR is slightly lower.

The signal was there the whole time and I misread it. A metric that stays flat across four unrelated changes is not four coincidences. Evaluation is the thing that makes every other decision meaningful, which makes it the thing whose failures are most expensive. Audit it like production code.

13 — Conclusion and final thoughts to self

1. Build the evaluation before the improvements Not because it's virtuous, but because contextual retrieval was the change I was most confident would help, and it turned out to be actively harmful. Without the scoreboard I'd have shipped it and never known.

2. Write your prediction down first Costs one line. Without it, every result feels explicable in hindsight and you learn nothing. With it, being wrong is impossible to miss — and that's where the understanding is.

3. Read fifty chunks before embedding anything Every parsing bug I found came from reading actual output, not from summary statistics. If you can't tell what a chunk is about in isolation, neither can the embedding model.

4. Suspect your ruler Fixing the evaluation was worth more than every retrieval improvement combined, by a factor of three and a half. A metric that won't move is evidence about the metric.

5. Eval the chunk retrieval first, and the LLM second The LLM works with the context you give it. So make sure you eval the chunk retrieval properly before making any changes to the LLM instructions.