Why I built it

For a while, RAG was just a word for me. I knew the idea: find the right documents, give them to the AI, get an answer. But I didn't know what actually happens in between. How do you cut a note into pieces? What is an embedding, really? Why do people add a reranker? And how do you make all of this work across different services?

So I built something to find out.

Cortex is a notes app where you can ask questions. It answers from your own notes and shows which notes it used. There's also an agent that can create, edit and delete notes, but only after you tap approve.

This isn't a tutorial. It's what I tried, what broke, and what I learned on the way.

How the pieces talk to each other

I had built Spring Boot apps before, so the backend itself wasn't new. The new part was splitting the work across three services and making them talk to each other properly.

Cortex has three parts:

• A Flutter app, for writing notes and chatting.
• A Spring Boot API, which owns the notes, sync, attachments and keyword search. It also issues the login token (JWT).
• A Python service (FastAPI), which owns everything AI: chunking, embeddings, search, chat and the agent.

The one rule I kept: the AI side never owns the notes. The Python service has its own Postgres database for vectors, chats, a semantic cache and LLM usage logs. When it needs keyword search, or the agent wants to change a note, it asks the Spring API through /internal/* endpoints that only accept requests with an API key. It never writes to the notes database directly.

The app talks to the Python service directly for chat, and the answer streams back word by word over SSE. The Python service checks the same JWT that Spring issued, so there's only one login.

Saving a note never waits for the AI. Spring puts an embedding job on a Redis queue and records it in an embedding_jobs table. A separate worker process picks the job up, embeds the note, and calls Spring back to say it's done. If no callback comes within 5 minutes, a sweeper in Spring marks the job as failed, so nothing hangs forever. I kept the worker as its own process on purpose: if it crashes, it can't hide behind a healthy /health endpoint on the API.

I was also curious what all this costs. Across 39 real answers I logged between Oct 6 and Nov 3, one chat reply cost about $0.00016 on average (around 1.5 paise), and $0.00028 at the 95th percentile. Agent replies cost about twice that. Embedding calls aren't counted, because Gemini doesn't return usage numbers for them.

The pipeline, step by step

When you ask Cortex a question, six things happen. This is the part I actually wanted to understand.

1. Chunking

A long note is too big to turn into one useful vector, so it's cut into pieces first. I went with the simplest option: fixed windows of 500 tokens, with 50 tokens of overlap so a sentence at the edge isn't lost. Notes of 500 tokens or less stay as one piece.

Gemini can only count tokens through a network call, so I count them locally with tiktoken's cl100k_base. It's not Gemini's exact tokenizer, but it's close enough to size the pieces.

tokens = encoding.encode(text, disallowed_special=())
if len(tokens) <= chunk_size_tokens:
    return [text]

step_size = chunk_size_tokens - chunk_overlap_tokens
for i in range(0, len(tokens), step_size):
    chunks.append(encoding.decode(tokens[i: i + chunk_size_tokens]))
    if i + chunk_size_tokens >= len(tokens):
        break

Update: the title is now added to the top of every chunk. The evals showed why (see the evals post).

2. Embeddings

Each piece becomes a vector, a list of 3,072 numbers, using gemini-embedding-2. You can ask Gemini for shorter vectors, but then you have to normalize them yourself, so I kept the full size.

One thing I didn't know before: Gemini embeds documents and questions slightly differently. Notes are embedded as RETRIEVAL_DOCUMENT and questions as RETRIEVAL_QUERY.

def embed_query(self, text: str) -> list[float]:
    return self.embed([text], task_type="RETRIEVAL_QUERY")[0]

3. Vector search

The vectors live in Postgres with pgvector, next to the rest of the AI data, so I didn't need a separate vector database. A question is embedded the same way, and pgvector finds the pieces closest in meaning (cosine distance). A long note can have many matching pieces, so I keep only the best piece per note.

To make the search fast I wanted an HNSW index, which turned out to need a small trick (more on that below).

4. Keyword search and Reciprocal Rank Fusion

Vector search is good at meaning but can miss exact words, like a name or an ID. So the Python service also asks the Spring API for a normal Postgres full-text search. Each side returns its top 20 notes.

Then the two lists are merged with Reciprocal Rank Fusion (RRF). I liked how simple it is. It ignores the raw scores and looks only at each note's position in each list:

_RRF_K = 60

if v_data is not None:
    score += 1.0 / (_RRF_K + v_rank)
if k_data is not None:
    score += 1.0 / (_RRF_K + k_rank)

A note near the top of both lists wins. A note in only one list still gets a chance.

5. Reranking

RRF gives a merged list of up to 20 notes. A reranker then reads the question and each note together and scores how well they match. I used a small cross-encoder, ms-marco-MiniLM-L-6-v2, that runs locally. It costs nothing per query, and it works differently from the embedding model, which is the whole point of a second opinion.

One small thing I had to handle: a note found only by keyword search has no piece attached, because only vector search returns pieces. At first I gave the reranker that note’s first piece, which failed when the answer was further down. Now all of the note’s pieces go to the reranker, and the note is judged by its best one.

6. The answer

The top 5 go to the AI labelled [Source 1] to [Source 5], with personal data hidden (more on that below). The answer streams back word by word, along with the sources, so the app can show which notes it used.

Two extras I added at the end:

• A semantic cache. If the first question of a new chat is very close in meaning to one asked before, the saved answer comes back without calling the AI. It only applies to the first message, since a follow-up depends on the chat around it.
• Model routing. Questions with 3 or more sources go to a "complex" model setting. Right now both settings point to the same model, so it doesn't change anything yet. The wiring is there for later.

What broke, and what fixed it

This is where I learned the most.

Gemini embed_content returned one embedding for two texts

I passed Gemini's embed_content a plain list of two strings and got back one embedding, not two. No error, no warning. The call looked just like the example in the docs.

Wrapping each string in its own types.Content fixed it:

# Before: two texts in, one embedding out
res = client.models.embed_content(model=model, contents=["note one", "note two"])

# After: one embedding per text
res = client.models.embed_content(
    model=model,
    contents=[types.Content(parts=[types.Part(text=t)]) for t in texts],
)

I only caught it because I was testing in a REPL and checked how many vectors came back. Now I always check the shape of what an SDK returns.

A retired model

My first embedding model, text-embedding-004, returned a 404. It had been retired. Small thing, but it taught me to check a model's status before building on it.

pgvector HNSW index with more than 2,000 dimensions

pgvector's HNSW index supports vectors of up to 2,000 dimensions, and mine have 3,072. The trick is to index a halfvec copy instead, which stores each number in half the space:

CREATE INDEX ix_note_chunks_embedding_hnsw
ON note_chunks USING hnsw ((embedding::halfvec(3072)) halfvec_cosine_ops);

Old pieces stayed after an edit

When I edited a note, the worker saved the new pieces but never removed the old ones. So search could show text that wasn't in the note anymore.

The fix was one line: delete the note's old pieces right before saving the new ones. I put the delete after the embedding call on purpose. If Gemini fails, the old pieces stay, and search still works with slightly old text.

embeddings = embedding_client.embed(chunks)
delete_chunks(message.note_id)
save_chunks(message.note_id, message.owner_id, title=message.title,
            chunks=chunks, embeddingd=embeddings)

One thing I'd still improve: the delete and the save happen in two separate transactions. For a moment in between, that note has no pieces at all. Putting both in one transaction would close that gap.

Redis brpop TimeoutError crashed the worker

The worker waits for jobs with a blocking read on Redis (brpop) and a 5-second timeout. I expected it to return nothing when the queue was empty. Instead, the Redis client raised a timeout error, and the worker crashed every time the queue sat idle for 5 seconds.

The fix was to treat that error as "no job yet" and keep waiting:

try:
    res = redis_client.brpop([settings.redis_queue_key],
                             timeout=settings.redis_brpop_timeout_seconds)
except redis.exceptions.TimeoutError:
    continue

"Can you repeat that?"

This one confused me for a while. I asked Cortex for something from my notes, and it found it. Then I asked, "Can you repeat what you just told me?" It said the information was no longer in the provided text.

Two things were going on. Every message ran a fresh search, and "can you repeat that" has no words worth searching for. Retrieval pulled one unrelated note, which the reranker scored at −11. And my prompt told the AI to answer only from the sources, so it wasn't allowed to use its own previous answer.

The fix was in the prompt, not the search. New facts still need a source from this turn, but the AI can now repeat or refer to what it already said:

Any NEW factual claim must be grounded in the context provided for THIS turn… You may use the earlier turns of this conversation to understand follow-up questions, maintain continuity, and repeat or reference something you already said earlier.

What I took from it: not every chat message is a search query. A chat and a one-time question are different problems.

A note RRF couldn’t rescue

I had a note with my keyword only in its body. Postgres full-text search weights the title higher than the body (setweight A for the title, B for the body), so this note lost to 25 notes with the keyword in their titles. It didn't make the keyword top 20, and it didn't make the vector top 20 either. RRF can only merge what it's given, so the note never had a chance.

I haven't fixed this one yet. Pulling more candidates from each side would probably help, and so would searching titles and bodies separately. If you've solved this differently, I'd like to hear how.

Update: I measured it

Update: I built the test set: 52 notes and 30 questions. Retrieval scored 100%, but the first run of answers failed 5 of 30. A long note was being judged by the wrong chunk, and a phone number was reaching the model. Both are fixed, and 89 of 90 answers now pass. The full story: My RAG evals said 100%. The answers were still wrong.

Personal data and prompt injection

Notes have a lot of personal stuff in them, and with RAG, the notes you search become part of what you send to the AI. That made me think about two things.

Hiding personal data. Before any text goes to the AI, Cortex replaces emails, phone numbers, SSNs, card numbers and IP addresses with placeholders, using regex:

_PATTERNS = {
    "EMAIL": re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"),
    "SSN": re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),
    # PHONE, CREDIT_CARD and IP_ADDRESS follow the same idea
}
for label, pattern in _PATTERNS.items():
    text, n = pattern.subn(f"[REDACTED_{label}]", text)

This happens only in the text sent to the AI, in both the chat prompt and the agent's tool results. The notes themselves don't change, so I can still search for my own phone number, and the app still shows the real values. Every AI call is written to an audit log with a count of what was hidden.

I thought this was working: when I asked for my own email and phone, the AI replied with [REDACTED_EMAIL] and [REDACTED_PHONE]. My evals later showed two bugs. A 7-digit phone number in another note wasn't hidden at all, and hiding values meant I couldn't see my own data. Cortex now swaps each value for a placeholder like [EMAIL_1] before the prompt goes out, and puts the real value back in the answer. The notes are still embedded with Gemini's embedding API, so personal data never reaches the chat model, but it does reach the embedding model.

Prompt injection. The same fact cuts the other way. If a note can reach the AI, a note can also carry instructions. So:

• each note goes to the AI labelled [Source 1], [Source 2] and so on
• the system prompt says source text is untrusted data, and any instructions inside it must be ignored
• the agent can't create, edit or delete anything until I tap approve

My test set now includes a note with an injected instruction. Across 18 answers, the model never followed it.

What I'd do differently

Make the test set before the pipeline. Every choice in the list above would have had a number instead of a feeling.

Sync image attachments from the start. On the phone, images were only saved locally, so they disappeared after a reinstall.

I started this just because I wanted to understand how RAG works. On the way I learned about embeddings, vector search, reranking, prompts, and how to make three services talk to each other without stepping on each other's data. Most of it I learned from things breaking.

Try it

The code is in three public repos:

• cortex, the Flutter app
• cortex_app_be, the Spring Boot API
• cortex-ai, the Python AI service

A one-command Docker setup is coming. If you've built something with RAG, I'd love to hear what broke for you.