Back to blog

VennDB 0.7: a new engine under the same API

VennDB, the in-memory Rust database behind rama's proxy database, gets a new engine in 0.7, with the same API on top of it.

VennDB 0.7: a new engine under the same API

venndb is a small append-only, in-memory database for Rust. You put #[derive(VennDB)] on a struct and get a database you can query with simple filters, mostly bool flags. Version 0.7.0 replaces the engine, while the API on top of it stays the same. You can find all details in the changelog.

Why we wrote it

We needed it for rama. Its MemoryProxyDB picks an upstream proxy for every proxied request, filtered on transport and proxy protocol, proxy category, pool, country and city. The data is loaded once and then queried all the time. A Vec scan got slow, and SQLite felt like a lot of database for "give me a random row with these flags". So we wrote venndb, and rama has used it for its proxy selection ever since.

How it works

Every filter value is a set of rows, and a query is the intersection of those sets: a Venn diagram. Bool properties are stored as bitmasks, and values that few rows have as a list of rows. any() picks a random match without looking for all of them first.

It is not meant to replace your real database. Rows cannot be updated or removed, and nothing is persisted.

Usage

Rustvenndb@0.7.0
#[derive(Debug, VennDB)]pub struct Proxy {    #[venndb(key)]    id: u32,    tcp: bool,    socks5: bool,    datacenter: bool,    #[venndb(filter, any)]    country: Country,}
Rustvenndb@0.7.0
let proxy = db    .query()    .tcp(true)    .socks5(true)    .country(Country::Be)    .any()    .expect("a proxy");

The docs and examples cover the rest, including set properties and runtime columns for state that changes while you query, such as a proxy that is down.

Alternatives and benchmarks

The closest alternative is a hand-written index of roaring bitmaps. Then there is SQLite in memory, and a plain Vec. On a proxy database shaped like rama's:

Bar chart, log scale, of the median time to pick a random matching row from 1 000 000 rows. Three bool filters: venndb 192 ns, roaring 13.8 µs, naive 6.98 ms, sqlite 42.2 ms. One bool filter and a country: venndb 2.18 µs, roaring 16.7 µs, sqlite 6.93 ms, naive 8.35 ms. One bool filter and five cities: venndb 3.26 µs, roaring 49.2 µs, sqlite 7.10 ms, naive 10.2 ms. One bool filter, a pool and three countries: venndb 28.6 µs, roaring 97.1 µs, naive 5.92 ms, sqlite 12.5 ms.
Median query time on 1 000 000 rows, Apple M4 Max. All numbers in the README.

roaring gets close once filter maps come in. We are no database experts though, so if you see a flaw in these numbers, do let us know.

Feedback is very welcome, and if you use venndb, we would love to hear about it on Discord.