CerebroProd · cerebro.aiOperationalOpen CerebroOpen

Engineering

How a note becomes searchable.

A note saved in the app, on mobile, or by an assistant acting for someone moves through six stages before it can be found. Retrieval then runs four passes, combining literal matches, full text, and meaning into one ranking.

← Back to the overview

Ingest

006 stages

  1. 000Write

    A note is saved

    From the app, the mobile client, or an assistant acting for someone. All three land in the same table.

  2. 001Notice

    A database trigger records it

    Not a job that scans for differences later. The database appends to an outbox in the same transaction as the write, so a change cannot be committed and then missed.

  3. 002Drain

    A worker takes the outbox in batches

    Runs continuously and deletes each row as it goes. The queue is a to-do list, not a log.

  4. 003Flatten

    The record becomes one document

    A note is a row plus a dozen joins. The search document carries all of it inline: the meeting, who attended, which companies, who it is shared with.

  5. 004Embed

    The text becomes a vector

    Gemini, 768 dimensions, in batches. Content is hashed first, so unchanged text is never re-embedded and a full re-index costs almost nothing.

  6. 005Index

    Keyword and vector land together

    One upsert into Typesense carries both. A document is never searchable one way but not the other.

From saved to searchable, typically inside a minute. The trigger is the load-bearing part: it makes the write and the intent to index atomic.

What stage 003 actually collapses

In the database

  • note row
  • meeting
  • meeting type
  • attendees
  • organizations
  • author
  • sharing rules
  • group access
  • role access
  • privacy flags
  • draft / archived state

One row, plus joins

One search document

Content

  • title
  • body text
  • note type

Provenance

  • author
  • meeting
  • dates

Relations

  • companies
  • attendees
  • groups

Permissions

  • owner
  • private flag
  • shared with
  • meeting access

Meaning

  • 768-dim vector

The permission fields are the point. They are written into the document, not looked up when someone searches.

How a question becomes an answer

006 steps

  1. 000Ask

    A question in plain language

    In the assistant the person already uses. Nothing new to open.

  2. 001Select

    The assistant picks the right capability

    Each one declares what it does and what it needs, so selection is a decision the model can make rather than a guess.

  3. 002Authorise

    It runs as the person asking

    Their permissions, not the assistant’s. What they cannot see in the app, they cannot see here.

  4. 003Retrieve

    Hybrid search across the record

    Keyword and semantic matching run together and the rankings are fused, so exact names and vague descriptions both land.

  5. 004Answer

    Grounded in what was found

    Sourced from the live record at the moment of asking, not from a snapshot.

  6. 005Log

    Every call recorded

    What ran, for whom, and which record it touched. Writes are audited field by field.

Every step runs as the person asking. The assistant never holds more access than they do.
000One definition, every surface
A capability is defined once — what it does, what it accepts, what it returns. Adapters project that single definition onto each place an assistant can reach it. Adding something once makes it work everywhere, and there is no second implementation to drift.
001The registry is enforced, not encouraged
A lint rule and a continuous-integration check reject any capability that bypasses the shared definition. The consistency is a property of the build rather than a habit people have to maintain.
002Anything that writes must be able to be undone
A capability that asserts a fact has to ship with the means to retract it, and has to declare that it writes. One that claims success without naming what it changed raises an alert rather than passing quietly.
003Retrieval is hybrid on purpose
Keyword search finds the exact company name; vector search finds the meeting nobody remembered the name of. Running both and fusing the rankings is what makes a vague question work as well as a precise one.

Retrieval

004 passes

  1. 000Exact

    Literal match first

    Ask for a company by name and that company is the answer, not the fuzzy neighbourhood around it. This pass short-circuits when it hits.

  2. 001Keyword

    Weighted full text, typo tolerant

    Name fields outrank body text, so someone named in the query beats someone mentioned in passing. Misspellings still land.

  3. 002Vector

    Nearest neighbours on meaning

    The query is embedded with the same model as the documents, then matched by distance. This finds the meeting you can describe but cannot name.

  4. 003Fuse

    One ranking from both

    The two result sets are fused, not concatenated. A result strong in both signals outranks one that is merely excellent in either.

Exact match short-circuits; otherwise keyword and vector both run and their rankings are fused.

Access control

Design note

Filter afterwards

  1. The query runs across everything.
  2. The engine ranks and counts the full set.
  3. Results come back.
  4. Code removes what this person may not see.

The count already told you how much exists. The ranking already moved around documents you cannot see. And any path that skips step four returns all of it.

Constrain the query

  1. The permitted set is expressed as part of the query.
  2. The engine never considers anything outside it.
  3. Ranking and counts are computed over that set alone.
  4. Results come back. There is nothing to remove.

Counts, ranking and results all describe exactly what this person is entitled to. Forgetting the filter is not a code path that exists.

Permissions are part of the query.

Permissions are not a filter applied after search. They are part of the query itself.

Post-filtering leaks. Result counts reveal how much exists. Ranking shifts around documents someone cannot see. Any code path that forgets the filter returns everything.

So permission facts are denormalized into every document at index time: owner, private flag, sharing by person, company, group, or role, and meeting attendees. The query is constrained before it runs, so counts, ranking, and results are computed over exactly the permitted set.

Why it is built this way

004 decisions

000Outbox instead of dual writes
When a note is saved, a database trigger appends to an outbox in the same transaction. A worker drains that outbox in batches, so a change cannot be committed and then missed.
001Hybrid instead of one search mode
Exact literal match runs first and short-circuits when it hits. Weighted full text adds typo tolerance, with name fields outranking body text; vector nearest-neighbour finds the meeting you can describe but cannot name. The two rankings are fused, not concatenated.
002Hash before embedding
Content is hashed before embedding. Unchanged text is never re-embedded, while new text is embedded with Gemini at 768 dimensions in batches.
003Denormalize heavily
A note is a row plus a dozen joins, so the search document carries all of it inline. One upsert into Typesense carries keyword and vector representations together.