The index comes to you, so your questions stay put

The first three steps happen on our server and need the internet. The last two happen on your computer and don't. Nothing about your searches ever moves from right to left.

The crawler starts from a list of well-known sites and follows links outward. It reads each site's robots.txt and obeys it, waits between requests to the same site, and backs off when a site answers slowly or with errors.

From each page it keeps the title, headings and readable text, and skips menus, headers, footers and sidebars. Exact duplicates are dropped. Pages get revisited: one that changed since last time comes back sooner, one that didn't comes back later, anywhere between a day and a month.

The crawl becomes an inverted index: for every word, the pages it appears in and where. Results are ranked on how well your words match the text, title and headings, whether they appear together as a phrase, how recent the page is, and how many other sites link to it (PageRank).

The full crawl is far bigger than anyone wants to download, so we cut smaller packs from it: the sites the rest of the web links to most, each with its most important pages. Packs are picked by how much a site is linked to, never by topic, so the pack you download doesn't say anything about your interests.

Each index gets a SHA-256 hash, and the hash is signed with our Ed25519 key. If a single byte of the file changes, the signature no longer matches. That's what lets the file travel by any route: our site, a mirror, BitTorrent, a USB stick.

Updates are published as deltas that contain only what changed since the last version, signed the same way.

The first time, you pin our public key. From then on the app refuses any index or update that isn't signed by that key, has been altered, or is older than the one you already have. The check runs again on every search, so a file that gets corrupted later is caught too.

$ index trust provider.pub
$ index install index-v001.bin

Search from the terminal, or start the web interface and open it in your browser. It only listens on localhost, so other computers can't reach it.

# network switched off
$ local-search -n 2 "rust ownership"
1. Rust (programming language) - Wikipedia
   https://en.wikipedia.org/wiki/Rust_(programming_language)
   … The ownership system was in place by 2010. …

2. Introduction - The Rust Programming Language
   https://doc.rust-lang.org/book/ch00-00-introduction.html
   … you'll learn about Rust's ownership system. …

The index has to fit on your disk, so it covers the most-linked parts of the web, not every page that exists. Results are as fresh as your last update, not as fresh as this morning's news. Those are the costs of keeping the search on your side.

Steps 1 to 3 aren't just for us.

The crawler and indexer are open source, and the app accepts packs from any publisher its user chooses to trust. If you can build a better pack than ours, for a language, a field or a set of sites we cover badly, or with your own version of index build, please do.

One thing to keep in mind: a pack about a topic says something about the people who download it. A medical pack tells the network you're interested in medicine. Say so on your download page, and make it easy to get through mirrors or a torrent.

Copy examples/crawl.toml, choose your seed sites and limits, and change user_agent to name your crawler and a page where site owners can reach you.

$ cargo build --release --workspace
$ crawler --config my.toml --state crawl.db --out-dir pages/ --forever

Leave the politeness defaults alone. A pack built by hammering websites isn't better, it just gets crawlers blocked for everyone. crawler-ops gives you a dashboard for a long-running crawl: pages per hour, errors, disk space and the crawler's log.

On a Debian or Ubuntu server, one script sets all of it up: the crawler and image crawler as services, the dashboard behind a password with a button that builds, signs and publishes packs, the app for macOS, Windows and Linux, and this website with your downloads and key fingerprint, served over HTTPS with a one-line install for your users.

$ sudo scripts/setup-server.sh --seed https://example.org/ --domain search.example.org

index sites ranks the crawled sites by how many other sites link to them, and index build keeps the ones you ask for. This is where a pack gets better or worse: try different limits, and search your pack with local-search --debug to see why each result ranked where it did.

$ cat pages/*.jsonl > crawl.jsonl      # the newest record of each URL wins
$ index sites --pages crawl.jsonl -n 50
$ index build --pages crawl.jsonl --version 1 \
    --min-linking-sites 3 --max-sites 5000 --max-pages-per-site 200
$ index keygen --out provider        # provider.key (secret) + provider.pub
$ index sign index-v001.bin --key provider.key

Put the .bin, .sha256 and .sig files and your provider.pub on any static host, and publish the key's fingerprint somewhere people already trust you. Keep provider.key secret: anyone with it can publish packs in your name. For updates, index delta makes a small signed file with only what changed.

index build makes a few decisions for everyone who downloads a pack: which pages get in, how much each one counts, and what is stored about it. Ours are a starting point, not the final word. The app doesn't care how a pack was made, only that the file follows the index format, so you can change any of those decisions and people can search your pack with the same app.

Ours works in five steps, each in its own file in index/src/:

1. Collect (stream.rs). Keeps the newest record of each URL. You could also drop near-duplicates: print versions, mirrors, and pages that differ only by a date.
2. Score (pagerank.rs). PageRank over the links of the whole crawl, scaled from 0 to 1. You could start it from sites you vouch for instead, so links from spam farms count for nothing.
3. Choose (select.rs). The sites most other sites link to, then the best pages of each. You could put small sites first, prefer fresh pages, skip thin ones, or fill a fixed size with the best score per byte.
4. Index (index_page in builder.rs). The words of the text with their positions, plus the title, headings and URL. You could index the description too; ours only shows it in results.
5. Write (stream.rs). The sections, the header and a SHA-256 of the whole file. Leave this step alone: it's what the app reads.

For example, this replaces our rule in select.rs with small sites first: every site's best page goes in before any site gets a second one. On a test crawl where one site had six of ten pages, --max-pages 4 took all four from that site with our rule, and pages from all three sites with this one.

// index/src/select.rs (and add BTreeMap to the use line at the top)
pub(crate) fn select_ranked(
    urls: &[&str],
    sites: &[String],
    _ranks: &[SiteRank],
    authority: &[f32],
    selection: &PackSelection,
) -> Vec<bool> {
    let best_first = |&a: &usize, &b: &usize| {
        authority[b].total_cmp(&authority[a]).then_with(|| urls[a].cmp(urls[b]))
    };
    // A BTreeMap, not a HashMap, so every build picks the same pages.
    let mut by_site: BTreeMap<&str, Vec<usize>> = BTreeMap::new();
    for (i, site) in sites.iter().enumerate() {
        by_site.entry(site).or_default().push(i);
    }
    for pages in by_site.values_mut() {
        pages.sort_by(best_first);
    }

    let budget = selection.max_pages.unwrap_or(usize::MAX);
    let mut keep = vec![false; urls.len()];
    let mut kept = 0;
    for round in 0.. {
        let mut tier: Vec<usize> =
            by_site.values().filter_map(|p| p.get(round).copied()).collect();
        if tier.is_empty() || kept == budget {
            break;
        }
        tier.sort_by(best_first);
        for i in tier.into_iter().take(budget - kept) {
            keep[i] = true;
            kept += 1;
        }
    }
    keep
}

$ cargo build --release --workspace
$ target/release/index build --pages crawl.jsonl --version 1 --max-pages 100000

Keep your version deterministic: no random numbers, and nothing that depends on a HashMap's order. Then the same crawl built with the same --created-at gives the same file, byte for byte, and anyone with your crawl and your code can check that your pack is what you say it is. cargo test --workspace will flag our tests for the old rule; rewrite them to describe yours.

The index format is documented, so your builder can be written in any language. Two things have to match the app exactly:

• How words are split. The app looks up the words of a query in the pack, so the pack must lowercase words, remove accents and reduce them to their stem (English Snowball) just like common/src/tokenize.rs. The header records a tokenizer version, and the app refuses a pack whose version doesn't match its own.
• The order of documents. They're sorted by URL, and a document's place in that order is its ID.

Start by reproducing ours. Build the same crawl with both builders and the same --created-at, and compare the files with cmp. Once they're identical, you know your writer is right and can start changing the decisions. index inspect prints a file's header if you need to see where they differ.

Build both packs from the same crawl, install each in its own folder (see below), and ask both the same 20 or 30 real questions. Note which gives the better first page of results; local-search --debug shows how each result scored. If yours wins, publish the pack, and publish your code with it so people can see why a page is in or out.

The indexer reads JSON Lines, one page per line, so your crawler can be written in any language. If it writes records like this, index build can turn them into a pack:

{"url": "https://example.org/guide",
 "title": "A guide",
 "description": "",
 "lang": "en",
 "headings": ["Getting started"],
 "text": "The readable text of the page…",
 "links": ["https://example.org/", "https://other.example/"],
 "crawled_at": 1760918400,
 "depth": 1}

Shown over several lines here. In the file, each record is a single line.

url, title, text and crawled_at are required; the rest are optional. Links feed PageRank, so include them if you can.

A Burrow store holds one pack at a time. To keep someone else's pack next to ours, give it its own folder:

$ export LOCAL_SEARCH_HOME=~/packs/theirs
$ index trust their-provider.pub
$ index install index-v001.bin
$ local-search "…"

Next: how privacy is enforced →