Development

Toolchain

Layout

crates/telemetry   event model
crates/storage     WAL, segments, indexes, recovery
crates/query       query language and engine
crates/ingest      native JSON + OTLP parsing
crates/metadata    SQLite metadata store
crates/server      HTTP server and the `vyrtel` binary
tools/loadgen      load generator
dotnet/            .NET benchmark, baseline, shared harness and demo producer
web/               React + TypeScript UI (Vite)
tests/integration  end-to-end Rust tests against a real server
tests/fixtures     shared test data
docs/              documentation (Markdown) and the website that publishes it
docs/brand/        brand guide, logo and palette artwork, asset generator
docs/cla/          individual and corporate Contributor License Agreements
docs/site/         website front page, docs page template, CSS and JS
docs/screenshots/  UI screenshots (regenerated by the Playwright suite)

Running the backend

cargo run -p server -- --data-dir ./data-dev

Without a built UI the server embeds a placeholder page; the API works regardless. Useful flags: --bind 127.0.0.1:8080, --log-level debug, --memory-limit 256MB.

Running the frontend

cd web
npm ci
npm run dev        # http://localhost:5173, proxies /api and /v1 to :8080

Set VYRTEL_URL=http://127.0.0.1:9090 to proxy to a different server.

Running both as one binary

cd web && npm ci && npm run build && cd ..
cargo build --release -p server
./target/release/vyrtel

crates/server/build.rs embeds web/dist when it exists (rebuild the server after rebuilding the UI). In debug builds the files are read from web/dist at runtime, so npm run build is enough to see UI changes.

Test data

# Native logs
curl -X POST localhost:8080/api/v1/events -H 'content-type: application/x-ndjson' \
     --data-binary @tests/fixtures/logs.ndjson

# A realistic stream (logs + a trace per 200 events)
cargo run --release -p loadgen -- --rate 2000 --duration 60s --traces-every 200

The OTLP fixtures contain {{now-…}} placeholders; see tests/fixtures/README.md.

Debugging

• --log-level debug (or VYRTEL_LOG_LEVEL=storage=debug,info) logs sealing, compaction and retention decisions.
• Every query response carries diagnostics; the UI shows them under the results ("▸ N results · 12 ms …").
• GET /api/v1/system/info and /api/v1/system/storage show queue depths, active segment sizes, memory budgets and storage totals.
• Damaged files found at startup are moved to data/quarantine/<signal>/, logged at WARN/ERROR with the reason and listed under signals.<signal>.recovery in /api/v1/system/storage; Settings → Storage warns while any are in quarantine (see Recovery health).
• cargo test -p storage --features crash-points --test crash kills real processes at named points in the write path. To reproduce one by hand, run VYRTEL_CRASH_AT=wal.after_ack cargo run --features storage/crash-points, ingest one event so it aborts, then start it normally (see Testing).
• cargo test -p storage --features fault-injection --test faults injects full-disk, permission, read-only and short-write errors into the WAL and segment paths (see Testing). Current faults are listed under signals.<signal>.faults in /api/v1/system/storage.
• vyrtel config prints the effective configuration.

Formatting and linting

cargo fmt --all
cargo clippy --workspace --all-targets --all-features -- -D warnings
cd web && npm run format && npm run lint && npm run typecheck

The web UI is formatted with Prettier (web/.prettierrc.json: 120 columns to match rustfmt.toml, single quotes, trailing commas). npm run format rewrites files in place; npm run format:check is what CI runs and fails on any unformatted file. Don't hand-align code or fight the formatter; if a line reads badly once wrapped, extract a named local instead. Generated files (src/lib/wordmark.ts) are listed in web/.prettierignore and stay as their generator writes them.

Beyond layout, a few style rules are enforced so they don't come up in review:

• Rust (rustfmt.toml): field-init shorthand (Foo { bar }, not Foo { bar: bar }) and ? instead of try!, on top of the 120-column width.
• TypeScript (web/eslint.config.js): import type for type-only imports, interface rather than type for object shapes (type stays for unions and aliases), T[] rather than Array<T>, === (== null is allowed for null-or-undefined), braces on any if or loop whose body spans lines, object shorthand and template literals instead of string concatenation. npx eslint . --fix fixes all of these.
• Every file (.editorconfig): UTF-8, spaces, final newline, no trailing whitespace; 4-space indents for Rust and C#, 2 for everything else. Most editors pick this up automatically.

Benchmarks and profiling

The .NET projects under dotnet/ drive a running --release server over HTTP and share Vyrtel.Harness (server launcher, deterministic event factory, latency stats). They build the server themselves unless one is already listening (--url/--no-launch).

cargo build --release -p server

# BenchmarkDotNet: ingest throughput (NDJSON, JSON array, OTLP traces, OTLP metrics)
# + query latency; writes docs/benchmarks/results.json
dotnet run --project dotnet/Vyrtel.Benchmark -c Release -- --job short

# Capture a snapshot for a released version (also what the tag CI job does):
dotnet run --project dotnet/Vyrtel.Benchmark -c Release -- --job short --label v0.1.1 --out docs/benchmarks/versions/v0.1.1.json

# Reproducible baseline: repeated runs with the pinned configuration, cold/warm queries with
# diagnostics, accepted/rejected counts, peak RSS and variance-derived regression budgets;
# writes target/baseline/report.{json,md,samples.csv}. --compare checks against an earlier report.
dotnet run --project dotnet/Vyrtel.Baseline -c Release -- --runs 5 --notes "NVMe SSD, AC power"
dotnet run --project dotnet/Vyrtel.Baseline -c Release -- compare old.json target/baseline/report.json

# Unit tests for the baseline tool (no server needed; CI runs them in the .NET job)
dotnet test dotnet/Vyrtel.slnx -c Release

Use the benchmark to see trends; use the baseline (which includes the mixed-load percentile profile) to decide whether a change regressed, because it pins the server configuration, repeats the workload and states how noisy the machine is. See Reproducible baselines.

Both reports record what was built and how big it is on disk: the release binary size, the seeded dataset's data-directory footprint, and the server's own storage counters (bytes stored per event, compression ratio, index overhead) read from GET /api/v1/system/storage before the load starts. The site charts those across v* snapshots, so regressions in binary or dataset size show up next to the latency numbers.

.github/workflows/benchmark.yml runs the benchmark on every main change (and on each v* tag, writing a versions/<tag>.json snapshot) and commits the rollup to the benchmark-results branch, which the site build overlays onto docs/benchmarks/ to render the benchmarks page, including the version-history comparison. The same workflow's baseline job uploads a 3-run baseline as a workflow artifact. dotnet/Vyrtel.LiveDataSim is a sample ASP.NET Core producer (not part of the build).

Release builds

cd web && npm ci && npm run build && cd ..
VYRTEL_REQUIRE_WEB_UI=1 cargo build --release --locked -p server     # target/release/vyrtel
node scripts/release/release.mjs verify-binary target/release/vyrtel # version + embedded UI
docker build -t vyrtel .

Without web/dist, the server embeds a placeholder page telling you to build the UI. VYRTEL_REQUIRE_WEB_UI=1 (set by the release workflow and the Dockerfile) turns that into a build error instead.

Releases are produced by .github/workflows/release.yml when a v* tag is pushed: Linux x86_64/ARM64, macOS ARM64 and Windows x86_64 binaries with SHA-256 checksums, plus a multi-architecture image on GHCR. Nothing is published unless the tag matches the workspace version, the version's compatibility gate and then its performance gates signed off that exact commit before it was tagged, the full CI suite passes and every artifact's version, embedded UI and checksum are verified; prereleases never become latest. Create release tags with node scripts/release/release.mjs pretag --tag vX.Y.Z --create, not git tag. See releasing.md for the gates, the pre-tag sign-off, how to cut or rehearse a release, and what to do when one fails.

Website and docs site

The website (front page plus every doc in this folder, rendered under /docs/) is built from docs/ and published to GitHub Pages by .github/workflows/pages.yml on pushes to main that touch the docs or branding.

cd docs && npm ci
npm run serve      # build to docs/build and preview on http://localhost:4321
npm run brand      # regenerate the outlined wordmark SVGs and web/src/lib/wordmark.ts

Write docs as ordinary Markdown with relative links (configuration.md#docker-and-bind-mounts, ../web/src/App.tsx): the build turns links between docs into site links and links to other repo files into GitHub links. Add a new page to the DOCS list in docs/build.mjs to put it in the sidebar.