Benchmarks

Vyrtel is measured on the same machine on every run, so the numbers stay comparable over time. The .NET benchmark harness launches a release build of the server against a throw-away data directory, seeds a deterministic dataset, then drives it over HTTP with BenchmarkDotNet.

How to read this

• Mean / Median / Min / Max / StdDev are wall-clock latency per request, in milliseconds. Lower is better.
• Throughput is the sustained rate: events per second for ingest, operations per second for queries.
• Executable is the size of the release binary the server under test ran, and Data directory the on-disk footprint of the seeded dataset.
• Storage comes from the server itself: bytes stored per event, compression ratio and index overhead for the seeded dataset.
• Query timings include the cost of serializing the response over the loopback interface, not just server CPU.

Methodology

• Server: cargo build --release, launched with default durability against a temporary data directory on the CI machine.
• Data: a deterministic mix of HTTP request logs across 8 services with a 2% error rate and ~10k distinct customers, plus OTLP traces and metric points. The same EventFactory backs the load generator, the benchmark and the baseline, so the workload stays realistic.
• Coverage: ingest throughput (native NDJSON, native JSON array, OTLP traces, OTLP metrics) and query latency across filters, full-text scans, counts, histograms, facets, trace lookup/search and metric series.
• Job: BenchmarkDotNet ShortRun (--job short). CI uses the same command as a local run, so results are reproducible.
• Footprint: right after seeding, the harness records the release binary size, the data directory on disk and the server's own storage counters. Those are captured before the benchmark traffic starts, so they describe the dataset rather than the load.

Reproduce locally:

cargo build --release -p server
dotnet run --project dotnet/Vyrtel.Benchmark -c Release -- --job short

For a latency breakdown under a mixed read/write load (p50/p90/p95/p99 per operation) and the index hotspots of that load, use the baseline: its mixed-load phase replaces the former Vyrtel.Profiling tool.

The tools sign in as admin with the password they launch the server with (VYRTEL_AUTH_ADMIN_PASSWORD, default vyrtel-harness) and use that session for queries. A session does not authorize ingestion, which a server they launch leaves open (it has no ingest-scoped API keys). Against a server that is already running, set VYRTEL_AUTH_ADMIN_PASSWORD to its admin password, and if it has ingest keys, pass one with --ingest-key <key> (benchmark) or VYRTEL_HARNESS_INGEST_KEY; it is sent on ingest requests only.

Reproducible baselines

The rollup at the end of this page tracks trends on one CI machine. To decide whether a change made Vyrtel slower or heavier, compare like with like: the same seeded workload, the same pinned server configuration and the same host, repeated often enough to know how noisy that host is. dotnet/Vyrtel.Baseline does this and writes a report a reviewer can check.

cargo build --release -p server
dotnet run --project dotnet/Vyrtel.Baseline -c Release -- --runs 5 --label main --notes "NVMe SSD, AC power"

The defaults take about a minute per run on a laptop. Reports go to target/baseline/: report.json (the record that comparisons read), report.md (for review) and report.samples.csv (the latency and HTTP status of every request).

What one run does

Each run is independent: a new temporary data directory and a newly launched release server with the pinned configuration on 127.0.0.1:18480. If anything is already listening on that port the run stops, because it could not know that server's configuration (use --port to choose another).

1. Seed. 100,000 log events (NDJSON, 500 per request), 1,000 OTLP traces and 100 metric points from the same EventFactory as the benchmark (seed 42), sent one request at a time. A 429 or 503 counts as backpressure and is retried after 200 ms, so the dataset always arrives complete. Any other rejection stops the run: an incomplete dataset would make every later number wrong. The run then waits until every queue is empty and checks the dataset: the log events, spans and metric points counted from the fixture payloads must equal both what the server acknowledged and what GET /api/v1/system/info reports storing, per signal. Any shortfall or surplus stops the run, because a server that loses acknowledged events would leave a smaller dataset and every query would look faster. It then records the storage counters and data directory size.
2. Cold and warm queries. For each of the 13 queries below, the server is restarted on the same data directory, the connection is opened with GET /health, and the query is timed once (cold: a new process with an empty index cache and nothing decoded) and then 20 more times (warm). The server's own diagnostics are kept for the cold request and the last warm one. Every one of these requests must succeed: an HTTP error or no response (a timeout, a refused connection) stops the run. A failed request has no latency worth comparing, and timing only the successes would let a broken endpoint record nothing, or 0 ms, and look like an improvement.
3. Mixed load. 4 workers send a weighted mix of NDJSON and JSON ingest (800 events per request) and queries for 20 s; the first 3 s are discarded. Each worker's sequence comes from its own seed, so every run sends the same requests in the same order; only how many fit in the time varies. Every request is classified as accepted, backpressure (429/503), too large (413), other rejection or transport error, per operation, with p50/p90/p95/p99 latency. The server's query statistics (GET /api/v1/system/query-stats) are read when the measured window opens and after it closes; their difference gives the index hotspots of that window alone, without seeding, the cold/warm queries or the warmup. A snapshot that cannot be read leaves the hotspots out of the report with a warning; it does not fail the run.
4. Memory. Server RSS is sampled every 100 ms in every phase, and the OS's own high-water mark (PeakWorkingSet on Windows, VmHWM on Linux) is read before each server stops. The report gives the larger of the two and says which it was.

Options: --runs, --seed, --events, --batch, --warm-iterations, --duration, --warmup, --concurrency, --port, --server, --config, --label, --notes, --out, --compare, --log-level, --keep-data (--help describes each). Changing any workload option makes the report incomparable with earlier ones, and a comparison between them fails.

Pinned configuration

dotnet/Vyrtel.Baseline/baseline.toml writes out every setting that affects throughput, latency or memory, so a change to Vyrtel's defaults cannot silently move a baseline. It matches the defaults except:

The report records the file's SHA-256, the effective configuration from GET /api/v1/system/config and the ingest admission from GET /api/v1/system/info. Editing the file starts a new baseline: re-record it before comparing.

Reading the report

The JSON keeps every run's raw numbers and diagnostics; the CSV keeps every request. Nothing in the report asserts an absolute number: what is fast on one machine says little about another.

Regression budgets

A budget is how much worse than the baseline's mean a new mean may be before it counts as a regression. Budgets are derived from the measured run-to-run variation, not chosen per metric:

budget = max(floor, 3 × CV × k(n), rounded up)
CV     = sample standard deviation ÷ mean, over the baseline's n runs
k(n)   = √((n − 1) ÷ χ²₀.₁₀(n − 1))     3 runs: 3.08   5 runs: 1.94   10 runs: 1.47

• 3 standard deviations. A mean three standard deviations worse than the baseline is unlikely to be noise.
• k(n): an upper bound on the spread. A few runs usually underestimate the true spread, so the CV is first raised to its 90% upper confidence bound (assuming roughly normal noise). Without this, two baselines of the same commit on the reference laptop disagreed: 8 of 74 metrics moved by more than three of the first baseline's 3-run standard deviations, and the comparison reported a regression (latest logs, warm p95, +11.6% against a 10% budget). With both baselines' six runs pooled, none did. More runs give a smaller k(n) and tighter budgets.
• Rounding and floors. Timings round up to a multiple of 5%, sizes to 1%. Within-session runs run back to back, so they cannot see drift between sessions (temperature, background load, power state). The floors cover it, and are set from measurements rather than chosen: across four baselines of the same commit recorded in separate sessions on the reference laptop, timings whose spread-based budget was under 10% drifted by up to 11.2%, so the timing floor is 15%; data-directory size varied 0.55% across all 16 runs and the binary not at all, so the size floor is 2%. On another host, check the floors with a same-commit comparison.
• Enough samples for a percentile. A percentile is gated only when every run had at least 10 samples beyond it: 20 for p50, 200 for p95, 1,000 for p99. A per-run p99 from 400 requests is decided by its top four, and the p95 of 20 warm repetitions is effectively the second largest, so one scheduler stall would decide the verdict. Such metrics are still reported; raise --warm-iterations or --duration to gate them.
• Only successful work is throughput. The gated throughput metrics are successful requests per second and accepted ingest events per second. A server that answers 429 quickly sends more responses per second while doing less work, so attempted requests per second is reported but never gated.
• Failure shares are absolute. The share of requests not accepted (backpressure, rejections and transport errors), for seeding and the mixed load, is budgeted in percentage points: max(1 point, 3 × standard deviation × k(n)). A relative budget would be undefined for a baseline with no failures, which is the normal case. The 1-point floor is a policy choice: the reference laptop rejected nothing in its runs, so there was no spread to measure.
• Not gated when there are fewer than 3 runs (no spread to estimate), the mean is zero, the percentile has too few samples, or the budget would exceed 50% (too noisy to tell a regression from a bad run with that many runs). The report says which, with the arithmetic.

Record a baseline you intend to compare against with --runs 5 or more on a quiet machine (on AC power, other heavy programs closed). Three runs is the minimum: it gates only the steadiest metrics.

Comparing two baselines

Compare a new run against an earlier report, or two existing reports:

dotnet run --project dotnet/Vyrtel.Baseline -c Release -- --runs 3 --compare target/baseline/reference.json
dotnet run --project dotnet/Vyrtel.Baseline -c Release -- compare old/report.json new/report.json

Each metric gets a verdict from the baseline's budget: regression (worse by more than the budget), improvement (better by more than the budget), within budget, not gated or missing. The comparison passes only when all of these hold:

• no metric regressed;
• no metric the baseline gated is missing from the current report (a mixed-load operation with no successful request has no latency, so its metrics go missing);
• at least one metric was gated (a baseline with fewer than 3 runs gates nothing);
• both reports measured the same thing: pinned configuration, workload, CPU, core count, OS, architecture, build profile and report schema. Otherwise a difference says nothing about the code. The number of runs may differ.

The report and console list every reason a comparison failed. Exit status is 0 when it passed, 2 when it did not, and 1 on an error, so a script can act on it.

Checking a host with a same-commit comparison

Before trusting budgets on a new machine, record two baselines of the same commit in separate sessions (for example, an hour apart) and compare them in both directions:

dotnet run --project dotnet/Vyrtel.Baseline -c Release -- --runs 5 --out target/baseline/a.json
# later
dotnet run --project dotnet/Vyrtel.Baseline -c Release -- --runs 3 --out target/baseline/b.json --compare target/baseline/a.json
dotnet run --project dotnet/Vyrtel.Baseline -c Release -- compare target/baseline/b.json target/baseline/a.json

Nothing changed, so any regression is noise the budgets did not cover. If one appears, look at the samples CSV for the run that stands out: a run with fewer mixed-load requests than the others means the whole machine was slower (another program, throttling) and that session should be re-recorded. If it recurs on a quiet machine, the host drifts more than the floors allow: compare against a baseline recorded in the same session as the candidate, or use quieter hardware.

Reference baseline

baselines/2026-10-15-windows-laptop.md (raw data: .json and .samples.csv, kept out of the repository)) is a 5-run baseline of 0.1.1 with the defaults above, on an Intel Core Ultra 7 155H laptop (22 logical cores, 32 GB, NVMe SSD, Windows 11, on AC power with desktop applications open). The server was built from b61cd28; the working tree differed only in this tool, its tests and docs, which the report flags as uncommitted changes. It is evidence of what the tool reports and how noisy that machine is, not a performance promise for other hardware.

These files are report schema 1, recorded before successful-throughput and failure-share metrics, the dataset check and pass/fail comparisons were added. The current tool (schema 2) refuses them rather than reinterpreting them: a schema-1 baseline gated attempted throughput, which counts 429 responses as work. They remain the evidence for the budget rule below. Record a schema-2 reference on your own machine before comparing (see Checking a host).

How many metrics a 5-run baseline can gate on this laptop, by kind:

Cold timings are a single request each and vary too much between runs to gate here: they are reported for review, and their diagnostics show where the cold cost comes from.

Run-to-run evidence

Six baselines of the same commit were recorded on the reference laptop in separate sessions. Nothing changed between them, so every reported regression is a false alarm:

The first two rows are why the rule has k(n) and the tail-sample requirement, the third is where the 15% timing floor comes from, and the last is the check on fresh sessions the rule was not tuned on. TC-02's "identical workloads twice with variance" is rows E and F: both reports hold every run's values, and the comparison tables show the differences.

Two later sessions on the same laptop (with schema 2) show what the gate does when the host itself changes. With the battery at 18% and charging, mixed-load throughput fell from 221 to about 100 successful requests/s and level = Error p50 rose from 20 ms to 100 ms on the same commit. The tool gated only 17–18 of 93 metrics, because within-session spread was too large to trust, and one comparison failed with four regressions of 39–104%. That is a real 2× slowdown of the host, correctly reported; it is also why a reference must be recorded on a quiet machine on AC power, and why a failure on a laptop should be confirmed by re-recording both reports.

The baseline job in .github/workflows/benchmark.yml records a 3-run baseline on each main change and v* tag and uploads it as the baseline-<ref>-<sha> workflow artifact (kept 90 days), with the Markdown in the job summary. Shared runners are noisy, so their budgets describe that hardware; compare two artifacts from the same runner type, not a runner against a laptop.

When a run fails

The tool deletes each run's data directory (%TEMP% or $TMPDIR, named vyrtel-baseline-*) unless --keep-data is given. If it is interrupted (Ctrl+C), a launched vyrtel process and that directory can be left behind: stop the process and delete the directory.

Limitations

• One host at a time. Budgets hold for the host and configuration in the report. There is no universal guarantee across hardware, and comparing reports from different hosts is not meaningful (such a comparison fails as incompatible).
• Cold means a cold server process, not a cold disk. The OS page cache is not dropped (that needs root and differs per OS), so the data files are usually still cached in memory.
• Compaction is off, unlike production defaults; its cost is not measured here.
• Restart time is measured by polling /health every 50 ms. On Windows a refused connection takes about 0.5 s to fail, so restart times there have a 0.5 s floor; treat them as comparable only on the same OS.
• Event timestamps follow the wall clock (the last 30 minutes before the run), so their absolute values differ between runs while everything else in the dataset is identical.
• Peak RSS on macOS comes from 100 ms sampling only, which can miss short spikes.
• Disk type (SSD or HDD, local or network) is not detected; put it in --notes.

CI rollup

The rollup below is regenerated automatically on the CI machine whenever main changes.

.github/workflows/benchmark.yml runs the benchmark on every main change and commits the rollup rendered below to the benchmark-results branch. Pushing a release tag (v*) captures a separate snapshot under versions/<tag>.json on that branch; the Version history section charts those snapshots so improvements between releases are visible. Use --label <tag> --out docs/benchmarks/versions/<tag>.json to produce one locally.

Vyrtel 0.1.1 · short job · 200,000 events · generated 2026-10-23

Measured on Ubuntu 24.04.5 LTS; 4 logical cores; .NET 10.0.13; BenchmarkDotNet 0.15.8.0.

Build and footprint

Executable

10.8 MB (release)

Data directory

21.9 MB on disk after seeding

Events stored

203,200 events in 3 segments

Bytes stored per event

113.17 B

Compression

4.09× (52.8 MB raw → 21.9 MB stored)

Index overhead

7.5% of segment bytes

Ingest

Query

Version history

A snapshot is captured for every released tag (v*), so progress is visible between versions. Older versions are on the left; each column in the table is a full benchmark run.