BisonDB

A document database with BSON storage, custom B+Trees, and a desktop GUI.

Documentation · Prairie GUI

BisonDB is a document database with a BSON storage engine, inspired by MongoDB. It is written in C++20 and targets Windows as its primary development platform, with Linux support via CI. The bisondb_core library provides a BSON value model, a BSON decoder and encoder, MongoDB Extended JSON v2 reading/writing, an append-only collection store, a custom on-disk B+Tree (without using std::map or third-party storage libraries), a query engine with index-aware planning, and a custom socket layer (Winsock2/POSIX, without Asio). bisond is the network daemon speaking a framed BSON protocol (docs/protocol.md), with a client library and remote mode in the bisonc CLI.

Building

Prerequisites

• CMake 3.21+
• Ninja (recommended) or Visual Studio 2022
• MSVC, GCC, or Clang with C++20 support

Quick start (MSVC / Visual Studio)

cmake --preset msvc
cmake --build --preset msvc-debug
ctest --preset msvc-debug

Quick start (Ninja)

cmake --preset debug
cmake --build --preset debug
ctest --preset debug

MinGW (MSYS2 GCC)

cmake --preset mingw-debug
cmake --build --preset mingw-debug
ctest --preset mingw-debug

bisonc CLI

bisonc converts between BSON files (single documents or concatenated dumps, as produced by mongodump) and JSON.

:: BSON -> JSON Lines (relaxed Extended JSON) on stdout
bisonc to-json dump.bson

:: Canonical (lossless) Extended JSON, written to a file
bisonc to-json dump.bson --canonical -o dump.jsonl

:: Pretty-printed instead of one document per line
bisonc to-json dump.bson --pretty

:: JSON (single document or JSON Lines) -> concatenated BSON
bisonc to-bson dump.jsonl -o dump.bson

:: Document count, total bytes, and per-type value counts
bisonc inspect dump.bson

Errors go to stderr with a non-zero exit code. Canonical mode round-trips losslessly: to-json --canonical followed by to-bson reproduces the original file byte for byte.

Database commands

bisonc db import       data\db zips tests\fixtures\zips.bson
bisonc db find         data\db zips "{\"pop\": {\"$gte\": 40000}}" --limit 5
bisonc db find         data\db zips "{\"pop\": {\"$gte\": 40000}}" --explain
bisonc db create-index data\db zips pop
bisonc db delete-many  data\db zips "{\"state\": \"AK\"}"
bisonc db indexes      data\db zips
bisonc db drop-index   data\db zips pop

--explain prints the chosen plan. The same range query before and after create-index:

{ "plan": "scan",        "docsExamined": 29470, "docsReturned": 1015 }
{ "plan": "index_range", "index": "pop", "docsExamined": 1015, "docsReturned": 1015 }

Filters support {field: literal}, $eq/$ne/$gt/$gte/$lt/$lte/$in, $and/$or, and dotted paths. The planner uses an index for a single equality or range on an indexed field (and _id point lookups); everything else falls back to a full scan. Indexed plans always re-check the complete filter on each fetched document.

bisond server quickstart

⚠️ Authentication is enabled, but there is NO TLS yet. Credentials and data travel over the socket in clear text. Use BisonDB only on loopback or a trusted LAN until the TLS phase ships. See Security for the auth model. bisond binds to 127.0.0.1 by default; binding elsewhere exposes that clear-text traffic to the network.

Terminal 1: start the server. On the first run, seed an admin (provide the password through the environment, never as a CLI arg):

set BISONDB_ADMIN_PASSWORD=choose-a-strong-one
bisond --dir data\db --port 27027 --init-admin admin

(If you omit --init-admin, bisond prints a one-time bootstrap token instead. Use this token once to create the first admin (see Security). Offline alternative: bisonc auth create-admin --dir data\db --username admin.)

The startup banner is followed by structured per-request logs (conn, cmd, durationMs); --quiet suppresses both.

Terminal 2 — talk to it with bisonc (any db subcommand plus --connect host:port):

bisonc ping --connect 127.0.0.1:27027
bisonc db import - zips tests\fixtures\zips.bson --connect 127.0.0.1:27027
bisonc db create-index - zips pop --connect 127.0.0.1:27027
bisonc db find - zips "{\"pop\": {\"$gte\": 100000}}" --explain --connect 127.0.0.1:27027
bisonc status --connect 127.0.0.1:27027

(The <dbdir> positional argument is ignored in remote mode; pass - instead.) The wire protocol is one length-prefixed BSON document per message. The file docs/protocol.md documents the framing, the command set, error codes, and the find-truncation contract, which is complete enough to write a third-party client. find responses are capped at 16 MiB. Larger result sets return truncated: true with a skipNext cursor substitute that the bundled client follows automatically. Ctrl-C performs a graceful shutdown (drain, sync, exit 0). The shutdown command is also available from loopback connections.

Prairie desktop GUI

BisonDB Prairie is a desktop client (Tauri 2 and React) living in the sibling ../Prairie/ folder. It has its own bun and cargo toolchain. Connect to a running bisond or open a local database folder (which spawns a bundled bisond sidecar automatically). It includes a document browser with filters and explain plans, insert, edit, and delete actions with confirmations, index management, and import or export for .bson, .json, and .jsonl files. See ../Prairie/README.md for build steps.

bisonsh: The interactive shell

Terminal 1: bisond --dir data\db. Terminal 2:

> bisonsh
BisonDB 1.0.0 @ 127.0.0.1:27027
type 'help' for the statement grammar
bisondb> db.students.insertMany([{name: 'ada', cgpa: 3.9},
...                              {name: 'bob', cgpa: 2.1},
...                              {name: 'eve', cgpa: 3.7},])
{
  "insertedCount": 3,
  "insertedIds": [ {"$oid": "6a2c389484512c46998d9bf4"}, ... ]
}
bisondb> db.students.find({cgpa: {$gt: 3.5}})
{
  "_id": {"$oid": "6a2c389484512c46998d9bf4"},
  "name": "ada",
  "cgpa": 3.9
}
{
  "_id": {"$oid": "6a2c389484512c46998d9bf6"},
  "name": "eve",
  "cgpa": 3.7
}
returned 2 in 0.3 ms
bisondb> db.students.find({cgpa: {$gt: 3.5}}).explain()
{ "plan": "scan", "docsExamined": 3, "docsReturned": 2 }
scan — examined 3, returned 2
bisondb> db.students.createIndex('cgpa')
{ "built": true, "docsIndexed": 3 }
bisondb> db.students.find({cgpa: {$gt: 3.5}}).explain()
{ "plan": "index_range", "index": "cgpa", "docsExamined": 2, "docsReturned": 2 }
index_range on "cgpa" — examined 2, returned 2
bisondb> exit

JSON arguments are relaxed: unquoted keys ($gt included), single-quoted strings, and trailing commas. Statements with unbalanced brackets continue on ... lines (a blank line cancels). Output is colorized on TTYs (--no-color to disable); long results page after 100 documents. History persists to ~/.bisonsh_history (capped at 1000). Parse errors show a caret diagnostic and never kill the session:

bisondb> db.students.find({cgpa: {$gt 3.5}})
db.students.find({cgpa: {$gt 3.5}})
                             ^ expected ':' after key '$gt'

Scriptable modes exit non-zero on the first error: bisonsh --eval "stmt; stmt", bisonsh -f script.bsh, or piped stdin. --connect host:port picks the server (default 127.0.0.1:27027).

B+Tree internals

Files and recovery

A collection lives in <dbdir>/ as:

File	Contents
<coll>.log	Append-only record log (the primary source of truth)
<coll>._id.idx	Unique B+Tree: encodeKey(_id) → 8-byte log offset
<coll>.<field>.idx	Duplicate-mode B+Tree: encodeKey(field) ‖ 0x00 ‖ _id → (empty)
<coll>.meta.json	Index registry

Log records use the format u8 type (1=PUT, 2=DEL) | u32 len | payload. The log is the source of truth for recovery. Replaying it front to back (where the last record per _id wins) reconstructs the live document set. A torn trailing record is ignored. Index files have a clean flag in their header. The server clears this flag on the first write after opening the database, and sets it only after a full flush. If the server crashes, the flag remains unset, prompting the next startup to discard the index and rebuild it from the log. Indexes function as disposable caches, and the log is never rewritten except during compaction.

Page layout

Every .idx file is an array of fixed-size pages (default 4096 bytes). Page 0 is the header (magic "BSNI", version, pageSize, rootPageId, freeListHead, pageCount, cleanFlag). Nodes use a slotted-page layout:

+--------------------------------- page (4096 B) ----------------------------------+
| type | cnt | freeOff | right |  slot[0] slot[1] ... ->     ...      <- cell  cell |
| u8   | u16 | u16     | u32   |  u16 offsets, sorted by key | free  | data grows  |
+------ 12-byte header --------+-----------------------------+-------+  upward ----+

leaf cell:     keyLen u16 | key | valLen u16 | value          (right = next-leaf link)
internal cell: keyLen u16 | key | childPageId u32             (right = rightmost child)

Lookups binary-search the slot array; range scans walk a leaf's cells then follow the right sibling link. Splits move the upper half of a node to a new page (promoting the right page's first key for leaves, the median for internal nodes) and propagate upward, growing a new root when needed. Deletion uses lazy underflow: pages may stay underfull, and a page is unlinked and freed only when it reaches zero cells (with root collapse). One writer or many readers at a time, via a tree-level shared_mutex.

Key encoding

Index keys are encoded so plain memcmp matches value order. A type-class tag byte gives the cross-type order, then a type-specific payload:

Class	Tag	Payload encoding
Null	0x05	(none)
Numbers (Int32/Int64/Double)	0x10	normalized to double; IEEE bits sign-flipped (all bits when negative), big-endian. -0.0 → +0.0. Integers above 2^53 lose precision. NaN is rejected; indexes skip these documents
String	0x20	UTF-8 with 0x00 escaped as 0x00 0xFF, terminated 0x00 0x00
ObjectId	0x30	12 raw bytes
Bool	0x40	1 byte
DateTime	0x50	int64 ms biased by 2^63, big-endian

Encoded keys cap at 512 bytes. Long keys and missing fields are skipped and counted in the build stats. This differs from MongoDB, which indexes missing fields as null.

Security

BisonDB supports TLS transport encryption and user authentication for single-node setups. It does not support replication. While --tls-insecure and --no-auth flags are available for development, they should not be used in production. These credentials and data are no longer exposed in clear text when TLS is active.

TLS (transport encryption)

✅ Enable TLS with --tls and a certificate. Without --tls the transport is plain TCP (clear text) — fine for loopback dev, not for a network.

# 1. Make a cert/key (self-signed; key file is 0600). Use a real CA cert in production.
bisonc tls gen-cert --out-dir ./tls --cn localhost

# 2. Start the server with TLS + an admin.
BISONDB_ADMIN_PASSWORD=choose-one bisond --dir data/db \
    --tls --tls-cert ./tls/cert.pem --tls-key ./tls/key.pem --init-admin admin

# 3. Connect, trusting the self-signed cert (or pin its fingerprint).
bisonsh --connect localhost:27027 --tls-ca ./tls/cert.pem --username admin

• Library: Mbed-TLS 3.6 (vendored via FetchContent; keeps the binaries DLL-free). TLS 1.2 (ECDHE + AES-GCM); TLS 1.3 is deferred (a config wrinkle in this build).
• Server cert: --tls-cert/--tls-key (operator-provided), or --tls-self-signed to generate one at startup and print its SHA-256 fingerprint for pinning.
• Client verification (secure by default): the default verifies the server cert against the OS trust store and the hostname. --tls-ca <pem> trusts a specific (self-signed) cert; --tls-pin <sha256> pins a fingerprint; --tls-insecure disables verification for development. It prints a warning and marks connections as unverified in the startup banner.
• Private keys are never logged. A mismatched plaintext/TLS pairing fails fast with a message telling you to add or drop --tls.

Authentication (wire protocol v2)

Every connection must authenticate before any data command; the full handshake, state machine, and error codes are in docs/protocol.md. Highlights:

• Users & roles. Three roles: read (find/explain/list/dbStats), readWrite (all data commands), admin (everything + user management + shutdown). Users live in a hidden system file <dbdir>/__auth.bsd, never exposed through listCollections/find.
• Password hashing. Argon2id (memory-hard, via the vetted Monocypher library) with a per-user random salt. Plaintext passwords are never stored or logged.
• Tokens. authenticate returns a 256-bit session token (OS CSPRNG); the server stores only a BLAKE2b-256 hash of it. Tokens are in-memory and expire (default 1h); they are lost on restart. authenticateToken resumes a session; logout revokes it.
• Bootstrap (avoid lockout and insecure defaults). First run with no users either takes --init-admin <user> (password from $BISONDB_ADMIN_PASSWORD, never a CLI arg) or enters setup mode, printing a one-time bootstrap token to stderr that creates exactly one admin. Offline alternative: bisonc auth create-admin --dir <dbdir> --username <u>. There is no anonymous fallback once any user exists.
• Hardening. Generic AuthFailed (no username enumeration), constant-time secret comparison, per-connection failed-login backoff, shutdown requires admin and a loopback peer, and auth events are logged without secrets.
• Dev escape hatch. --no-auth disables auth entirely; it refuses any non-loopback bind and warns loudly on every startup.

In bisonsh, log in with --username (you are prompted for the password, or set $BISONDB_PASSWORD/--token/$BISONDB_TOKEN), then manage accounts with auth login, auth whoami, auth passwd, auth create-user, auth list-users, and auth bootstrap.

Sanitizers

cmake --preset asan (ASan+UBSan) and cmake --preset tsan (ThreadSanitizer, exercises the B+Tree reader/writer test) run in CI on Linux/Clang; MinGW on Windows does not ship these runtimes.

Tests

The Catch2 suite covers unit tests per module, byte-exact round-trips, and the official BSON corpus (fetched copies live in tests/corpus/; re-fetch with tests/corpus/download.ps1 or download.sh). Any .bson files dropped into tests/fixtures/ are automatically round-trip-tested for byte-identical re-encoding.

Code formatting

This project uses clang-format with the configuration in .clang-format (LLVM style, 4-space indent, 100-column limit, left pointer alignment). To check formatting locally:

clang-format --dry-run --Werror $(find src tests -name "*.cpp" -o -name "*.hpp")

CI will fail the lint job if any file is not formatted correctly.

Releases

Latest: v1.2.0 (TLS transport encryption). Prebuilt, statically linked Windows x64 binaries are attached to each GitHub release; the full version history is on the changelog.

License

BisonDB (which includes the bisond engine, bisonsh shell, bisonc CLI, and bisondb_core library) is licensed under the GNU General Public License v3.0. See the LICENSE file for details.

The Prairie desktop GUI is a separate project, also licensed under the GNU General Public License v3.0. The entire project is GPLv3.