Engine
Built to query biology at terabyte scale
BioQ answers over 5.74 billion metagenomic proteins and 149 million UniProt records without loading them into a database. These are the techniques behind it, with the numbers we measured.
Most tools look up what is registered. BioQ asks for what meets a condition.
Portals and BLAST-style tools answer “show me X” and “what resembles X”. Discovery questions are conditions across datasets, so the engine joins its layers for you.
Show me UniProt P04637
Sequences similar to mine
Proteins sampled from soil
Assemblies that contain protein A but not protein B
Large soil protein families with no known domain
Every database that knows this ID, and every one that never looked
What it takes to do it yourself
Six real questions, timed both ways. Download times use transfer rates we measured from the public archives; your connection may differ.
Bars use a log scale from 1 millisecond to 10 hours.
Five layers between you and 1.15 TB
Where you work
Same answers everywhereGateway
Who can ask, and how muchEngine
20 of 22 layers liveIndexes we build
≈107 GB, checked against the sourcePublic sources
Read in place, never copied wholeSix indexing decisions that make it fast
The file’s own footer is the index
1.11 TB of parquet already stores min and max IDs for all 46,303 row groups. That 827 MB footer becomes a 2 MB index.
A layout built for many readers
128.7 M representative sequences rewritten into a seekable store: each lookup moves 46 KB instead of up to 242 MB.
Random access inside gzip
Plain gzip must be decompressed from the start. Rewritten as BGZF with a block index, any coordinate is one seek.
One block per question
A 26 GB gzip that could not be seeked became 5,773 independent blocks, with zero records lost and a 404 KB index.
Both directions, packed tight
Pfam protein→domain and domain→protein indexes over 4.62 billion pairs, repacked from 66 GB to 18 GB.
Indexes written with the data
A sparse index bolted on later took over an hour; written during the build it is free. 102,074 small files packed into four.
Homology search that remembers
MMseqs2 with a pre-built k-mer index. Every result is cached under a fingerprint of the sequence, the parameters and the database version, so a repeat returns instantly and still says when it was computed.
first search over the internet
0.04 sthe same search again
Positive controls
| Query | E-value | Identity |
|---|---|---|
| HIV reverse transcriptase | 1.3×10⁻¹³ | 36.7% |
| Human hemoglobin α | 1.1×10⁻¹¹ | 99.0% |
| E. coli β-lactamase | 2.6×10⁻³³ | 50.0% |
Why we index deliberately
A k-mer index runs 20–46× the size of its database. Swiss-Prot (575,748 sequences) is live; larger corpora are being indexed.
One question, every layer, honest answers
/whereis asks nine layers in parallel, each with its own deadline, and reports what each one actually did. Over the internet it answers in 0.12–0.48 s.
GET /api/v1/whereis?id=MGYP000000149140foundClaims need independent evidence
An association needs at least three assemblies from different collection runs. Every answer carries the verdict.
No ratio without a universe
The same overlap reads 1.000 or 212.5 depending on the universe. The engine refuses to guess it.
Faster under load
On the seekable store, median latency improves as concurrency rises, with zero failures.
Queries per second
Median latency
Before it serves, the engine must answer five real questions
- KRAS: top dependent lineage is pancreas
- TP53–MDM2 interaction scores at least 700
- SARS-CoV-2 spike bases come back from RefSeq
- A hemoglobin search finds its homolog
- A negative control proves absence
Silent wrong answers we found, measured and designed out
A windowed search missed 395,688 sequences and answered “not found”.
Every row group is now checked; regression test passes 4 of 4.
With 2 of 19 TrEMBL shards loaded, 89.2% of records would have looked absent.
Coverage is reported; absence is never assumed.
An off-by-one shifted every protein’s domains by a row while all totals matched.
Builders self-check; readers verify lengths at load.
A verifier recorded “match” when both sides were empty.
Verifiers must find something before they compare.
A real 212.5× enrichment was reported as 1.000.
No ratio unless the universe is stated.
AlphaFold “residues” were atom counts, 7.8× too high.
Rebuilt from Cα records; 4 of 4 controls pass.
Five rules every layer follows
- Provenance on every answer.
- An index is checked against its source, never against itself.
- A partial load answers 503 with a reason, never a silent “none”.
- Builds assert their totals.
- Not done without a positive control.
The test suite catches 13 of 13 deliberately broken contracts.
Next: semantic search with protein language models
We embedded all 575,748 Swiss-Prot proteins with ESM-2 (650M) in 47.6 minutes on one GPU and measured it against MMseqs2 on remote homologs.
Pairs MMseqs2 cannot align
Pairs MMseqs2 can align
They complement each other, so the plan is MMseqs2 first, then ESM-2 re-ranking. Not live yet.
See it answer your question
Free accounts get the full engine: every layer, the agent and the API.