VennDB 0.7: a new engine under the same API
venndb is a small append-only, in-memory database for Rust. You put #[derive(VennDB)] on a struct and get a database you can query with simple filters, mostly bool flags. Version 0.7.0 replaces the engine, while the API on top of it stays the same. You can find all details in the changelog.
Why we wrote it
We needed it for rama. Its MemoryProxyDB picks an upstream proxy for every proxied request, filtered on transport and proxy protocol, proxy category, pool, country and city. The data is loaded once and then queried all the time. A Vec scan got slow, and SQLite felt like a lot of database for "give me a random row with these flags". So we wrote venndb, and rama has used it for its proxy selection ever since.
How it works
Every filter value is a set of rows, and a query is the intersection of those sets: a Venn diagram. Bool properties are stored as bitmasks, and values that few rows have as a list of rows. any() picks a random match without looking for all of them first.
It is not meant to replace your real database. Rows cannot be updated or removed, and nothing is persisted.
Usage
#[derive(Debug, VennDB)]pub struct Proxy { #[venndb(key)] id: u32, tcp: bool, socks5: bool, datacenter: bool, #[venndb(filter, any)] country: Country,}let proxy = db .query() .tcp(true) .socks5(true) .country(Country::Be) .any() .expect("a proxy");The docs and examples cover the rest, including set properties and runtime columns for state that changes while you query, such as a proxy that is down.
Alternatives and benchmarks
The closest alternative is a hand-written index of roaring bitmaps. Then there is SQLite in memory, and a plain Vec. On a proxy database shaped like rama's:
roaring gets close once filter maps come in. We are no database experts though, so if you see a flaw in these numbers, do let us know.
Feedback is very welcome, and if you use venndb, we would love to hear about it on Discord.
Source: https://github.com/plabayo/venndb/releases/tag/0.7.0.