Collections & Documents

Concept

A collection is a named container of documents that you can search by meaning. Every collection has an embedding model and a semantic search provider (SSP) chosen at creation time. When you add a document to a collection, Primer splits the document text into overlapping chunks, passes each chunk through the embedding model to produce a dense vector, and stores those vectors in the collection's SSP. At search time, the query is embedded with the same model and the SSP returns the chunks whose vectors are nearest in meaning.

A document is a piece of text you add to a collection, addressed by a path that is unique within the collection (for example runbooks/db-failover.md). Each document has a path, an optional human-readable title (defaults to the path's last segment), free-form metadata (a JSON object), and a body. The body is the source of truth: Primer stores it in a first-class content store keyed by (collection_id, path), separate from the vector index. The body is split into chunks on write; the chunks, not the document as a whole, are what the vector index holds and returns as search results. Search is on in this release, so writing a document both stores its body and (re-)indexes its chunks; the vector index is an optional, derived index over the body rather than the place the body lives.

Two decisions are fixed at create time and cannot be changed afterward:

• Embedder (provider + model): all chunks in a collection must live in the same vector space. Changing the embedder would invalidate the existing index.
• Semantic search provider: chunks are written to the chosen backend. Migrating them to a different backend would require reindexing all documents.

Other collection settings (description, MMR diversification, and cross-encoder reranking) can be updated at any time.

Configuration

Collection fields

MMR diversification

MMR (Maximal Marginal Relevance) diversifies search results so near-duplicate chunks do not all surface together. It is cheap (pure linear algebra over the candidate pool) and adds essentially no latency for pools under a few hundred chunks.

Cross-encoder reranking

A cross-encoder reranker reads the query and each candidate chunk together and produces a fine-grained relevance score. It gives higher accuracy than pure vector similarity at the cost of extra compute per query.

Search pipeline order

When both MMR and a cross-encoder are configured, the pipeline runs in this fixed order:

flowchart LR
    query["Query"] --> embed["Embed query"]
    embed --> store["Vector store\n(top N candidates)"]
    store --> cer["Cross-encoder\n(score + re-sort, top_n)"]
    cer --> mmr["MMR\n(diversify, top k)"]
    mmr --> results["Results"]

The cross-encoder needs a relevance-rich pool, so it runs before MMR. MMR then diversifies the already-relevant pool. Running them in the opposite order would waste cross-encoder compute on diverse-but-irrelevant items.

Walkthrough

Create a collection

1. In the console, open Knowledge and click Collections.
2. Click New collection.
3. Enter an ID (or leave blank to auto-generate) and a description.
4. Choose an Embedding provider and model. Both are fixed after you save.
5. Choose a Semantic search provider. This is also fixed after you save. If you have not registered an SSP, the reserved lance row is always available.
6. Optionally enable MMR and/or a Cross-encoder reranker by toggling their sections and filling in the fields.
7. Click Save.

The collection row is created. The vector index for this collection does not exist yet in the backend; the SSP creates it lazily when the first document is ingested.

Add a document

1. Open the collection and click Documents.
2. Click Add document.
3. Enter a path (e.g. runbooks/db-failover.md). The path is unique within the collection; writing to a path that already exists replaces that document. Optionally set a title (defaults to the path's last segment).
4. Paste the document text directly, or upload a file. Supported upload formats include PDF, DOCX, HTML, and Markdown; Primer converts them to text before splitting.
5. Optionally add metadata as a JSON object (e.g. {"source": "manual", "version": "2"}). Metadata is stored with every chunk and can be used for filtering.
6. Click Save.

Primer stores the document body in the content store under its path, then splits it into overlapping chunks, embeds each chunk, and writes the vectors to the SSP. Progress is synchronous; the save button returns once the body is stored and indexing is complete.

Tip: You can also drop multiple files at once in the upload area to queue them as a batch.

Edit a collection

1. Open the collection and click Edit.
2. Update the description, MMR settings, or cross-encoder settings.
3. Click Save.

The embedding provider, model, and SSP fields are read-only in the edit form. To change them, delete the collection and create a new one with the desired settings, then re-ingest the documents.

Edit a document

1. Open the document and click Edit.
2. Update the title, metadata, or body.
3. Click Save.

Saving a changed body replaces the stored body and triggers a full re-index: Primer deletes the document's existing chunks from the vector store and re-ingests the new text. The collection ID is locked after creation; the path identifies the document and can be changed only by moving it (see below).

Move (rename) a document

Moving a document changes its path. Use it to rename a document or reorganise a collection's hierarchy. The body, title, and metadata are preserved; the move fails if the target path is already taken.

Delete a document

Deleting a document removes its body from the content store and its chunks from the vector store. Both are dropped immediately; there is no soft-delete.

Search a collection

1. Open the collection and click the Search button.
2. Enter a query and optionally adjust k (the number of results, 1-100).
3. Click Search.

Results show the document ID, chunk ID, score, text, and metadata for each hit. If a cross-encoder is configured on the collection, the score column reflects the cross-encoder logit (higher = more relevant); otherwise it reflects the raw vector similarity from the SSP.

Using collections from an agent

The console search above is for operators. An agent reaches collections and documents through tools instead. Add the ones it needs on the agent's Tools tab, then it can:

• Find the right collection: search__search_collections runs a semantic search over your collection definitions and returns the collections whose description best matches a query. It is part of the search toolset, which is only available when internal collections are enabled.
• Search a collection's contents: system__search_collection runs a semantic search over a collection's indexed document contents and returns ranked chunk hits (document_id, chunk_id, score, text, meta), most relevant first. It uses the collection's own embedder and vector store, the same path the operator search bench above uses.
• List documents: system__list_documents(collection_id, prefix?) lists a collection's documents by path. Pass a prefix (e.g. runbooks/) to scope to a subtree. Each entry carries its path, document_id, and size.
• Read a document: system__get_document_content(collection_id, path) returns a document's full body by path.
• Write a document: system__put_document(collection_id, path, content, title?, meta?) creates or replaces the document at path with content as its body, and re-indexes it.
• Move a document: system__move_document(collection_id, from, to) changes a document's path, preserving its body, title, and metadata.

A typical flow is: call search__search_collections to locate a knowledge base, system__search_collection to pull the most relevant chunks (or system__list_documents to browse by path), then call system__get_document_content with the path of each document the task needs.

embedding/embedding-providers

embedding/semantic-search-providers

embedding/cross-encoder-providers

embedding/internal-collections