Search the Screenshots and Voice Memos in Your Notes With a Typed Sentence, Offline — and the Two Limits That Decide What It Finds

"Text-extraction-based indexing pipelines are often lengthy and brittle." — Faysse et al., ColPali: Efficient Document Retrieval with Vision Language Models, ICLR 2025¹

A single open-weights model — running entirely on your own machine — can now map typed sentences, screenshots, and voice recordings into one shared vector space, so that typing "meeting about the rebranding budget" finds both the slide you photographed and the memo you dictated. Two limits govern what it actually retrieves: audio longer than about 327 seconds must be split into chunks, and shrinking the index below 256 dimensions degrades cross-modal retrieval substantially.²³

That is the short version. The rest of this post is the arithmetic you need to make it work — how the shared space is built, how to chunk a 30-minute voice memo without losing matches at the seam, what the numbers say about index compression, and three silent failure traps that produce plausible-looking but wrong results.

This is not a product pitch. EmbeddingGemma 2 (Apache 2.0, Google DeepMind, released 2026-10-06) is the example throughout because it is the first locally runnable model to publish hard token-budget numbers for all four modalities. The limits it reveals are architectural, not brand-specific — any multimodal embedding model faces the same trade-offs.

Why screenshots and voice memos disappear from search

A text-only vector retriever encodes the characters in a file. A screenshot has no characters. A voice memo has no characters. Both are invisible — the retriever returns zero results for either, regardless of what the image shows or what the speaker said.⁴

The workaround most people try first is OCR for images and ASR (automatic speech recognition) for audio. Both introduce a fragile preprocessing step. Faysse et al. observed in their 2025 ICLR work that "text-extraction-based indexing pipelines are often lengthy and brittle"¹ — a screenshot of handwriting, a diagram, or a dark-mode UI fails OCR silently. For audio, Google Research's Ehsan Variani noted that "ASR as a preprocessing step … often bottlenecks performance"⁵ when the goal is semantic retrieval rather than transcription accuracy.

The alternative is to skip the extraction step entirely and embed the raw signal directly — the same way a text query is embedded — so that the typed sentence and the image or audio land near each other in a shared vector space. That requires a model trained across modalities. Until recently, such models either ran in the cloud or required prohibitive local hardware.

How a shared embedding space joins the modalities

A multimodal embedding space is a coordinate system where proximity means semantic relatedness, regardless of input type. A photograph of a whiteboard equation and a typed description of that equation end up close together; a voice memo of a budget discussion and a typed query about budget decisions end up close together.

flowchart LR
    A[Typed query] --> M[EmbeddingGemma 2\nMultimodal Encoder]
    B[Screenshot] --> M
    C[Voice memo chunk] --> M
    M --> V[(Shared 768-d\nvector space)]
    V --> R[Cosine similarity\nranking]

Figure 1: Three input types — a typed query, a screenshot, and a chunked voice memo — pass through the same multimodal encoder into a shared 768-dimensional vector space. Retrieval is a single cosine similarity ranking across all three; no separate pipeline per modality is needed.

The mechanism relies on contrastive training across modality pairs. Radford et al.'s CLIP work (ICML 2021) demonstrated this for images: the model "trained on about 400 million (image, text) pairs" to pull matching pairs toward each other and push mismatches apart.⁶ Girdhar et al.'s ImageBind (CVPR 2023) extended the principle to audio, depth, and IMU by showing that "only image-paired data is sufficient to bind the modalities together"⁷ — audio paired with images during training inherits proximity to the text that describes those images.

The practical consequence is that the query and the document do not need to be the same type. As Wei et al. demonstrated in UniIR (ECCV 2024), a text query can retrieve an image, and an image can retrieve a text document, using the same distance function.⁸ EmbeddingGemma 2 brings this to a single locally-runnable model: its model card states it "maps text (incl. code), images, video, and audio inputs—and combinations thereof—into a single, unified 768-dimensional" space.²

The model ships in four modular sizes: 270M parameters for text and code only, 440M for text plus vision, 570M for text plus audio, and 740M for the full multimodal stack — "all projecting into the same compatible vector space."⁹ You can start with the 440M vision model, index your screenshots, and add the 570M audio encoder later "without recomputing existing vectors."⁹ The earlier version of this model had, according to the Google blog launch post (a vendor claim), "more than 20 million downloads" — a signal that the embedding research has reached production adoption.¹⁰

One directional note matters before going further. The MSEB benchmark score of 69.54 reported on the model card measures a spoken audio query against text documents — audio as the query, text as the document.¹¹ The angle this post explores is the reverse: a typed sentence as the query, a voice memo as the document. Both directions map into the same space, but only the spoken-query direction has a published benchmark number. A typed query against audio documents uses the same shared space and is architecturally supported; the quality in that direction has not been separately benchmarked for this model.

The token budget: what fits in a single embedding pass

Every modality competes for the same 8,192-token context window. The model card states it plainly: "All modalities share a single 8,192-token context window."³ Each modality has a fixed cost per unit of content:

• Images: 280 tokens per image (default resolution) → approximately 29 images per context window.
• Video frames: 140 tokens per frame → approximately 58 frames.
• Audio: 25 tokens per second → approximately 327 seconds (5 minutes 27 seconds) of audio.
• Text: roughly 1 token per 4 characters of English prose.

The audio limit is the operationally significant one. A 30-minute voice memo is 1,800 seconds. At 25 tokens per second, it requires 45,000 tokens — more than five full context windows. The entire memo cannot be embedded as a single vector. This is not a bug; it is an architectural consequence of using a shared token budget. The solution is chunking.

For on-device RAM, the 740M full-multimodal model runs at approximately 567 MB RAM on a recent mobile chip (Pixel 11 Pro figure, per the Google blog launch post — a vendor benchmark).¹⁰ The 570M audio model is smaller. A developer machine will handle either without GPU.

Simon Willison noted after the release: "open weights, so you can run it locally without sending your data to Google."¹² The model vendor describes it as one that "works entirely offline."¹³ Both characterizations are consistent with the Apache 2.0 license.

Chunking long audio: the arithmetic for voice memos

Audio preprocessing requires 16 kHz mono input.³ Most voice memo recorders capture 44.1 kHz stereo; a single ffmpeg conversion handles this before embedding:

ffmpeg -i memo.m4a -ac 1 -ar 16000 memo_16k.wav

Once converted, the memo must be split into segments short enough for the model's context window. The hard limit is 8,192 ÷ 25 = 327.68 seconds per input — call it 327 seconds to stay safely inside the window.

For a 30-minute (1,800-second) memo with 30-second overlap windows:¹⁴

import math

memo_s    = 30 * 60   # 1800 seconds
window_s  = 300       # leave a margin below the 327-second hard limit
overlap_s = 30        # overlap to avoid cutting a sentence in half

n_chunks = math.floor((memo_s - overlap_s) / (window_s - overlap_s)) + 1
print(n_chunks)       # → 7

Seven chunks at 300 seconds each, with a 30-second overlap, cover the full 30 minutes. Each chunk gets its own embedding and its own row in the vector index, tagged with the source filename and the start-offset (e.g. memo_20261005.wav :: 0:00–5:00). When a query returns that chunk, the metadata tells you where to seek in the original file.

The overlap prevents a sentence that straddles a chunk boundary from disappearing from search results. Without overlap, a topic discussed in the last few seconds of chunk 3 and the first few seconds of chunk 4 might miss both, because neither chunk contains enough context for a strong match. A 30-second overlap ensures the boundary content appears in at least one complete chunk.

from pathlib import Path

def chunk_audio_paths(source: Path, window_s=300, overlap_s=30, output_dir=Path("chunks")):
    """
    Yield (start, end, output_path) tuples for ffmpeg chunking.
    Input must already be 16 kHz mono.
    """
    import subprocess, json
    result = subprocess.run(
        ["ffprobe", "-v", "quiet", "-print_format", "json", "-show_format", str(source)],
        capture_output=True, text=True
    )
    duration = float(json.loads(result.stdout)["format"]["duration"])
    step = window_s - overlap_s
    start = 0
    while start < duration:
        end = min(start + window_s, duration)
        name = output_dir / f"{source.stem}_{int(start):05d}_{int(end):05d}.wav"
        yield start, end, name
        if end >= duration:
            break
        start += step

Each output path is the metadata you store alongside the embedding. The index entry format should be {"file": "memo.wav", "start": 0, "end": 300, "vec": [...]} so retrieval returns a playable time range rather than a filename alone.

Shrinking the index: what the numbers say

Storing 768-dimensional float32 vectors costs 3 KB per document. One million documents — roughly a decade of heavy note-taking at one note per day plus attachments — consumes about 1.5 GB. The model supports Matryoshka Representation Learning (MRL), which lets you truncate vectors to a shorter prefix without retraining. The storage trade-off: 256 dimensions (⅓ the size) or 128 dimensions (⅙ the size).

The model card's vendor-reported benchmark table shows what each truncation costs across retrieval tasks (higher is better):

The vendor's developer guide describes 256 dimensions as retaining "about 95% on image, video, and speech retrieval"⁹ relative to the full 768d index, and states: "Quality is close to lossless down to 256 dimensions. 128 dimensions degrades multimodal quality substantially and should be validated against your own workload before adoption."¹⁵

The degradation is asymmetric. From the model card table (own computation):

Single-modality quality — how well an image embedding represents an image, or an audio embedding represents audio — loses relatively little at 128 dimensions. Cross-modal retrieval — finding an image or audio clip with a text query — loses substantially more. The bottom two rows are the ones that matter for the use case this post describes.

The practical recommendation: keep 256 dimensions if the index holds images or audio. According to the developer guide, "1M vectors at 768 dimensions in float32 = 1.5 GB; with 128-dimension MRL, ~250 MB"⁹ — 256 dimensions lands midway in storage while retaining ~95% of cross-modal retrieval quality. 128 dimensions saves negligible additional disk space over 256 dimensions (both are well under 1 GB for 1M documents) and costs 18–22 percentage points of cross-modal retrieval.

The dimension floor has a second consequence. The model card warns: "Queries and documents must share a dimension. A 768-dimensional query cannot be scored against a 128-dimensional corpus."¹⁶ If you build the index at 256 dimensions and later want to query at 768 dimensions, you must re-embed the entire index. As the companion post on model-locked vector indexes explains,¹⁷ the switching cost is an argument for deciding on a dimension budget before you build.

Silent failure traps before you ship

Three failure modes in this pipeline return plausible scores but wrong results — none of them throw an error.

1. Un-normalized truncation. After MRL truncation, vectors must be re-normalized. The model card states: "If vectors are truncated, they must be re-normalized after truncation; failing to re-normalize will cause numerically plausible but semantically incorrect similarity scores."¹⁵ A cosine similarity function produces a number between −1 and 1 regardless. The number looks right; the ranking is not.

import numpy as np

def truncate_and_normalize(vec: np.ndarray, dims: int) -> np.ndarray:
    """Truncate to `dims` and re-normalize. Required after MRL truncation."""
    short = vec[:dims]
    norm  = np.linalg.norm(short)
    if norm == 0:
        return short  # degenerate vector — log and investigate
    return short / norm

2. Float16 NaN propagation. The model card flags "float16 NaN values if the model is run in float16 without attention bias correction."¹⁵ A NaN in a vector produces a NaN cosine distance. Depending on your sort implementation, NaN may rank first, last, or cause a silent crash. If you run inference in float16 for speed, confirm the runtime applies the correction.

3. Cross-modal score inflation. Research on multimodal retrievers has documented that "similarity scores tend to be higher when the query and candidate come from the same modality"⁸ — a text query against a text document will generally score higher than the same query against a semantically equivalent image, because the Modality Gap means same-modality pairs cluster more tightly. Liang et al. (NeurIPS 2022) observed that "the representations of the two modalities are generally 'embedded at arm's length'"¹⁸ even in a shared space. The practical consequence: if your index mixes text notes and image attachments, a text query may rank a closely-matching text note above a more relevant screenshot, not because the screenshot is a worse match in meaning but because its score is deflated by the modality gap. One mitigation is to store the modality of each indexed item and apply a per-modality score normalization before ranking.

A fourth limit, not a failure trap but an honest caveat: dense retrieval underperforms sparse methods on exact-match queries. Sciavolino et al. (EMNLP 2021) found that "dense retrievers drastically underperform sparse methods on entity-rich questions."¹⁹ Typing a person's full name, a phone number, or a specific document reference returns better results from keyword search than from vector search. A hybrid index — BM25 for exact matches, vectors for semantic search, merged by reciprocal rank fusion — handles both.

Trade-offs and when alt text is still the right choice

The multimodal embedding pipeline described here is not the right tool for every note folder. Consider the trade-off clearly:

Multimodal embedding is the better choice when:

• Images contain visual content that cannot be OCR'd (diagrams, handwriting, UI states, whiteboard photos).
• You have many audio recordings and want to search them by meaning, not by transcript keyword.
• You already maintain a vector index for text search and want to extend it to attachments without a separate pipeline.

Alt text or captions remain the better choice when:

• Your notes already have high-quality alt text or captions — the text path is faster to query and free of chunking complexity.
• Images contain mostly typed text that OCR handles cleanly (scanned PDFs, screenshots of articles).
• Storage or compute is the binding constraint — a text-only 270M model uses far less RAM than the full 740M multimodal stack.

As this site's post on alt-texting your images for AI retrieval notes, that approach is "not a substitute for a genuine multimodal pipeline" — but it is faster to set up and sufficient if the visual content is predominantly text.²⁰ The two approaches can coexist: alt-text the images you have already described well; embed the rest.

For the vector index itself — where it should live and how to keep it beside your files — the post on owning the vector index next to your files covers storage location and portability.

Frequently Asked Questions

How do I search my screenshots by what they show without OCR? A multimodal embedding model encodes the visual content of an image directly into a vector, without extracting text first. You embed the screenshot once, store the vector in an index alongside your text note vectors, and query the whole index with a typed sentence. Matches are ranked by cosine distance in the shared 768-dimensional space.²

Can I search voice memos by typing what I said, offline? Yes — with the constraint that each embedding pass covers at most ~327 seconds of audio (8,192 tokens ÷ 25 tokens per second). A 30-minute memo splits into roughly 7 chunks of 300 seconds with 30-second overlaps. Each chunk gets its own vector tagged with its time offset, so search returns the matching time range in the original file.³¹⁴

How many dimensions should I use for an index that includes images and audio? The vendor recommendation is 256 dimensions as the minimum for good cross-modal retrieval quality — approximately 95% of full-quality performance. Below that, 128 dimensions drops cross-modal retrieval by 18–22 percentage points relative to 768d; single-modality embedding quality declines much less (roughly 5–9%). Keep 256 or 768 if the index holds attachments.¹⁵

Does the published MSEB benchmark prove text-to-audio retrieval works well? Not directly. The MSEB retrieval score of 69.54 measures a spoken audio query matched against text documents — audio as the query, not the document. The reverse direction (typed text query, audio document) uses the same shared embedding space and is architecturally supported, but no separate published benchmark covers it for this model.¹¹

What happens if I forget to re-normalize after MRL truncation? Cosine similarity will return scores that look numerically valid — a number between −1 and 1 — but rankings will be semantically incorrect. The model card flags this explicitly: scores are numerically plausible but wrong. Always re-normalize after truncation; the three-line code block in the section above handles it.¹⁵

Can I add audio search to an existing text-only vector index? Yes. All model sizes project into the same compatible vector space, so audio vectors from the 570M or 740M model are compatible with text vectors from the 270M model. Embed the new audio and image attachments with the larger model and append the vectors; existing text vectors are unchanged. The only requirement: the dimension must stay the same across both batches.⁹

When is keyword search still better than vector search for notes? For exact-match lookups — full names, phone numbers, document IDs, specific product codes — dense retrieval underperforms keyword search. A hybrid index (BM25 for exact-match recall, vectors for semantic search, merged by reciprocal rank fusion) handles both well.¹⁹

The files in a notes folder are plain — Markdown, images, audio recordings alongside each other. A model today can index all of them into one searchable space, entirely offline. The two limits — a ~5-minute audio window per embedding pass, and a 256-dimension floor for cross-modal retrieval — are not obstacles so much as specifications. Know them before you build, and the index will do exactly what it promises.

If you keep notes in open, portable files, any future model can re-index them. The embedding changes; the files stay yours.

mnmnote.com is a local-first markdown note editor — your files stay on your own device, ready to index with any tool you choose.