Rullst CI/CD and Verification Contract

This document describes what the repository’s automation currently executes. It is not evidence that a workflow has passed for a particular commit. A green claim must always point to the GitHub Actions run, commit SHA, logs, and produced artifacts.

Last source-level review: 2026-09-11.

Status language

The distinction matters: Kani, Miri, mutation testing, or a scanner can be very valuable without proving that the entire framework is panic-free, race-free, memory-safe, or compliant with a regulation.

Mainline execution model

The v12 dashboard and its automatic status badges are pinned to main. The continuous workflows accept pushes to main and pull requests targeting it, and expose workflow_dispatch where a safe rerun is useful. Superseded runs of these workflows are cancelled per workflow and ref so rapid development does not spend runner capacity proving an obsolete commit.

ci.yml deliberately treats the expensive operating-system matrix differently. Format and Clippy continue to give feedback on draft pull requests. The complete Linux/macOS/Windows test matrix, blocking line coverage, SemVer fan-out and CodeQL analysis start for a pull request only when it is ready for review, and it can always be requested manually. Each operating system executes eight parallel shards: the non-CLI workspace, ordinary CLI targets, the LMS contract, three public-profile groups and two generated-blueprint groups. The basic/relational/polyglot and foundation/product partitions retain every original case while bounding the longest Windows and macOS jobs. No test is omitted; this changes wall-clock scheduling rather than the assertions being executed. Hosted CI permits two nested compiler jobs for the otherwise serial generated profile/blueprint builds; constrained local runs retain their one-job default. Each CLI shard fetches the locked registry inventory before its generated applications prove that they compile without network access. After that reviewed commit is merged, the automatic main push repeats Linux rather than paying for the same macOS and Windows proof twice. A direct push to main therefore has Linux evidence only until a maintainer explicitly runs ci.yml; release candidates must use the manual full matrix when no successful ready-PR run points to the exact candidate tree.

The SHA-bound quality scorecard is generated only by a ready pull request or a manual full-matrix run. It is deliberately skipped on the Linux-only automatic main run, because that execution cannot honestly award cross-platform verification credit. Run ci.yml manually on a final main candidate to produce the exact-SHA release scorecard. Manual diagnostic runs may select one operating system and one test shard; those deliberately do not produce a full-matrix scorecard and do not replace final-candidate evidence.

GitHub executes schedule events from the repository’s default branch, so scheduled and continuous v12 evidence now share the active main source line. Tag publication remains deliberately unavailable through a manual button.

Manual and periodic execution map

Every verification workflow except the PR-context-only ai-sentinel-pr.yml and tag-only release.yml can now be started from Actions → select workflow → Run workflow. A manual run checks the selected branch’s current SHA; record that SHA and the run URL before treating it as release evidence. The release workflow intentionally has no button because its publication authority begins only with an exact version tag.

The workflows below run only when requested manually:

These workflows are periodic and manually runnable:

All remaining test/build workflows run on the documented push, pull-request or path filters and also expose a manual rerun. For an RC checkpoint, first use the automatic mainline suite, then manually run the five manual-only workflows and any periodic/platform matrix whose latest successful run does not point to the same candidate SHA. Physical devices, store approval, live provider accounts, external security review and human release approval remain outside GitHub Actions.

Required local and release baseline

The contributor baseline from AGENTS.md is:

cargo test --workspace --all-features
cargo clippy --workspace --all-features -- -D warnings
cargo fmt --all

The main CI uses the stricter all-target Clippy form and checks formatting without modifying files:

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-features

Rust CI disables Cargo incremental compilation and uses the pinned sccache Action and binary to store content-addressed compiler outputs in GitHub Actions cache. The current cc build dependency also honors the same Rust compiler wrapper, so compatible bundled DuckDB C++ objects can be reused. Same-repository pull requests can populate only their GitHub-isolated refs/pull/.../merge cache scope, making a failed-job rerun useful without modifying the trusted default-branch namespace. Fork pull requests remain read-only. Pushes and explicit manual runs on main populate entries reusable by later pull requests. Cache contents never substitute for a test result. The tag-only verifier uses the same namespace strictly read-only, so it can reuse an exact trusted compiler output but cannot alter the cache while creating release artifacts; every release command and assertion still runs.

The setup action’s job-scoped Cargo archive is disabled in these compiler-cache jobs, so neither the raw workspace target tree nor duplicate registry bundles compete with compiler objects. CLI integration tests create and remove nested Cargo targets, and archiving whole mutable trees previously produced false missing-directory annotations, duplicated roughly 9.56 GiB across twenty active main caches, and caused eviction churn at GitHub’s default 10 GiB repository limit. Cargo may redownload registry sources on a fresh runner; this small network cost is preferable to storing the same registry/target archive under many job-specific keys. A first run on a new cache namespace is still a cold build; evaluate acceleration using the reported cache hit ratio and a later compatible run, never by weakening or omitting assertions.

The same content-addressed approach accelerates LLVM coverage, benchmark, mutation/fuzz compilation and the scheduled release-mode regression suite. The weekly corpus workflow warms the same fuzz compiler namespace used by the manual campaign. Benchmarks remain sequential on one runner so comparisons do not mix host variance. Coverage deliberately remains one report job because splitting it without a reviewed profile-data merge could change the repository percentage. Its second default-feature pass is limited to ORM, Studio and the public facade: those are the packages with default-SQLite tests excluded by the mutually exclusive all-feature graph, so unrelated workspace tests are not repeated. The release-mode workspace is safe to split because every shard returns an ordinary test result, while the two source locations that actually use proptest! still receive their configured 10,000-case runs. SemVer checks fan out from the machine-readable release order and validate one published API per job, so adding or removing a release package cannot silently drift from the matrix.

The all-feature coverage pass uses pinned cargo-nextest to schedule the same discovered unit and integration tests concurrently. It performs no retry and a flaky retry could not be normalized into success. A repository profile fixes four global slots and permits at most two integration tests that launch nested generated-application Cargo builds, preventing compiler fan-out from trading latency for memory or disk exhaustion. The exact per-test JUnit result and duration record is retained for 30 days. This runner is a coverage scheduler, not a replacement for Rust CI: the required multi-platform shards continue to execute the complete inventory with ordinary cargo test, preserving both traditional libtest process semantics and the materialized-project gates.

CodeQL also remains one analysis job. Its Cargo target cache is disabled so the extractor observes compilation for the exact SHA instead of inheriting a fresh artifact from another run; the analysis database itself is not interchangeable with ordinary test shards. Rust CodeQL’s faster buildless mode is intentionally not used because manual compilation gives the extractor the stronger generated code boundary needed by this release.

ci.yml also compiles and exercises each ORM strict database feature in isolation (PostgreSQL, MySQL, and SQLite), exercises the runtime-only Core and all 45 public umbrella features in isolated additive graphs with automatic manifest-drift detection, runs the portable database matrix on Linux, and tests the complete all-feature workspace in eight parallel shards on Linux, macOS, and Windows. Feature-boundary rows and threat-model negative tests also fan out into four deterministic strict shards each; their matrix job remains a single blocking dependency for the quality scorecard. Each threat-model shard primes the reviewed lockfile before its deliberately offline generated-project checks, so it does not inherit a hidden source-cache dependency from another job. The umbrella’s cfg(doctest) aggregation reads all 52 public tutorial files directly, so that same command discovers the versioned Rust blocks, compiles or executes complete examples, and records explicitly contextual fragments as ignored instead of pretending they are standalone programs. Its pinned live Redis job also proves that scheduled Core jobs are not claimed early, that Core cache inspection returns bounded metadata without values, plus ORM cache hit/TTL/recovery, tenant/table invalidation, rollback preservation, process-local post-commit observers and Scout commit ordering. A separate SQLite outbox contract runs on all three operating systems and covers atomicity, conflicting idempotency keys, claim races, lease expiry, retry and dead-letter; the relational matrix repeats the core outbox lifecycle against PostgreSQL, MySQL, MariaDB and strict SQLite. A dedicated job checks the declared MSRV, Rust 1.96.0. The Linux provider matrix also runs the feature-gated Scout adapter against a digest-pinned Meilisearch image; Algolia and Elasticsearch use bounded local protocol fixtures because no hosted provider account is part of CI. The same matrix runs typed, parameterized L2 and cosine queries against a digest-pinned PostgreSQL + pgvector image. It also runs Nexus’s default Any/SQLite HTTP contract explicitly, because the global all-feature graph intentionally selects a strict database profile and excludes that materialized tenant/audit target. Coverage separately merges the default workspace pass, so those routes contribute real executed-line evidence.

After a ready-PR or manual full-matrix Rust CI run finishes, an observational job emits a SHA-bound per-crate quality scorecard into the workflow summary and a 90-day artifact. The score combines versioned expert-audit ceilings with the actual gate results; a failed/skipped/cancelled gate can remove the dimensions it was meant to prove, while a green gate cannot inflate a crate beyond its audited ceiling. This is engineering-evidence reporting, not capability completion or certification. See the scorecard methodology.

Rows with no feature selected compile every package target. Feature-selected rows compile the isolated library graph; feature-enabled tests, examples, and benchmarks remain covered by the workspace and specialist jobs. This avoids pulling unrelated development dependencies into every boundary while retaining real integration coverage.

The tag-only packaged-distribution gate reads the complete feature set from the extracted rullst package manifest and compiles that crates-only consumer with defaults disabled and every public feature enabled. A partial hand-maintained feature allowlist therefore cannot make a monorepo-only integration appear release-ready.

Recommended main branch-protection profile

Require every job emitted by the following workflows before merging into main: Rust CI, GitHub Actions Lint, Documentation, End-to-End Smoke Tests, Cargo Audit, Security Audit, Cargo Deny, CodeQL, Test Coverage, Cargo Machete, SemVer Checks, Spellcheck, Crate Architecture Policy, TruffleHog, Unsafe Policy, WebAssembly Matrix, Zero Panics, no-std Build, IoT Integration, and PR Security Evidence.

Do not configure a path-filtered, scheduled, manual, deployment, or tag-only workflow as a universal required check: an intentionally skipped workflow may never create the check context. In particular, IoT Cryptography Containment is blocking when relevant paths change, and the Omni desktop, Android and iOS compile workflows are blocking only when the Omni generator boundary changes. Pages, benchmarks, fuzzing, sanitizers, Kani, Miri, mutation testing, udeps, ZAP, Scorecard, and release provenance belong to deeper evidence or release policy. GitHub repository rulesets remain the enforcement source; this document records the recommended profile and does not claim that the hosted setting is already enabled.

Phase 4 release-engineering status

Important evidence boundaries

Zero-panics and unsafe Rust

zero-panics.yml denies Clippy’s unwrap, expect, panic, todo, and unimplemented lints for published runtime libraries, procedural-macro engines, CLI production targets, generated runtime templates, and the Wasm Core path. Tests are excluded where assertion panics are test semantics.

unsafe-policy.yml compiles production libraries and binaries with -Dunsafe-code. The only reviewed file-level exceptions are the Radar OS probe and dynamic-library loader, and the workflow fails if that allowlist changes. This is an enforced boundary, not a claim that all dependencies contain no unsafe Rust.

Coverage

coverage.yml runs LLVM coverage over workspace all-features and default profiles plus the live database matrix, then uploads LCOV to Codecov using GitHub OIDC rather than a long-lived upload secret. It also retains exact JSON and text line summaries for 30 days so a passing upload cannot be confused with the coverage percentage. The all-feature pass uses pinned cargo-nextest only to run the same discovered tests concurrently, with zero retries, bounded nested-Cargo concurrency and a retained JUnit execution record. Rust CI still runs ordinary cargo test across Linux, macOS and Windows. The default-profile pass explicitly includes ORM, Studio, Nexus and the umbrella facade so their real SQLite contracts are not hidden by mutually exclusive all-feature database profiles. Before upload, the workflow independently rejects an LLVM summary below 90% for either the whole repository or the governed framework-library paths. codecov.yml also requires at least 90%, with zero tolerance, for those views and changed lines; failure to upload LCOV also fails the workflow. The report filters examples, benchmarks, auxiliary test support, and separate test files. CLI and proc-macro code therefore remains part of the blocking repository aggregate and is additionally visible as informational components. Their stronger semantic evidence still comes from materialized scaffolds and compile-pass/compile-fail contracts. The README exposes both the public overall badge and the separate framework_libraries badge rather than substituting the higher component result for the repository total.

Formal, dynamic, and stress analysis

• Kani and Miri are manual research evidence scoped to the harnesses/packages that actually execute. Rullst itself remains pinned to stable Rust 1.98.1 and keeps Rust 1.96 as its declared MSRV. Kani builds reviewed upstream revision 8fcd6d90ed07b559e553ca8a92b95f2db69b2c78 with the verifier’s own pinned nightly-2026-08-01 compiler (rustc 1.99.0-nightly) into a bundle and installer, then treats proof failures in twenty isolated harness jobs across ten supported packages as real matrix failures. That revision intentionally predates a compare_bytes compiler crash reproduced with Kani’s first rustc 1.100.0-nightly snapshot. The workflow does not patch manifests, bypass MSRV data, or change Rullst’s stable toolchain. Kani cannot verify the proc-macro-only rullst-macros target. Miri, which only runs on nightly Rust, pins nightly-2026-08-21 (rustc 1.100.0-nightly) and strictly executes 15 named pure-Rust/default-feature scopes. Its matrix excludes native ring, AWS-LC, SQLite, OS-syscall and network/provider execution that Miri cannot interpret; native CI, integration tests and sanitizers remain the applicable evidence for those paths. The rullst umbrella re-export facade and Blog example add no separate interpreter scope. A selected-scope failure fails the run. The Kani Security harnesses prove pure production decisions such as Vault key-ID character policy, DLP buffer admission, ASCII-folded RASP matching, SRI asset limits and bounded Login Guard delay; the IoT matrix also proves the complete CoAP option-component classification. They do not claim that Kani verifies zeroize’s unsupported inline assembly, cryptographic implementations or the entire concurrent middleware implementations.
• Mutation testing is manual, split into 80 lossless shards over the measured 14,380-mutant inventory, and intentionally informational while results are uploaded. Before the expensive matrix starts, a fail-fast --list --json preflight verifies the exact unique candidate set; the final aggregate must match that reviewed list, not merely its count. The hosted command makes --all-features explicit and .cargo/mutants.toml applies the same feature policy locally; the ignored legacy root configuration and its exclusions were not silently activated. Targeted mode accepts exactly one tracked production .rs path so a correction can be retested without restarting the complete workspace campaign. Exit statuses for missed and timed-out mutants remain findings; baseline, usage and internal failures do not get normalized into green jobs. The aggregate also fails closed when an artifact is absent, a shard is incomplete or the reviewed full inventory drifts; its conservative percentage never treats a timeout as caught.
• Fuzzing and corpus maintenance pin nightly-2026-08-21 instead of following a moving nightly alias. This verifier-only toolchain does not change the framework’s stable Rust 1.98.1 toolchain or its Rust 1.96 MSRV. Before a release campaign starts, ten strict preflight jobs compile every target so a stale import or broken fuzz manifest fails in minutes rather than alongside hours of valid campaigns. A five-minute one-target diagnostic is correction feedback only; the evidence-boundary job refuses to call it release evidence. Each of the ten fuzz packages has a checked-in dependency lock; inventory validation requires it, and locked metadata plus a post-command drift check prevents preflight, campaign or corpus maintenance from silently resolving a different dependency graph. The parser campaign restarts its ASan process every 30 minutes while retaining one corpus and the full 5.5-hour target budget, preventing instrumentation RSS accumulation from masquerading as a parser crash.
• Branch coverage, cargo-udeps, TSan and ASan share the reviewed nightly-2026-08-21 analysis snapshot instead of following a moving nightly alias. The first two remain observational/informational; sanitizer failures remain blocking whenever their daily/manual matrix executes.
• cargo-udeps is weekly/manual and explicitly non-blocking.
• TSan and ASan run daily/manual across twelve runtime/domain packages; Messaging runs its integration contract so its concurrent state is actually exercised rather than reporting a zero-test library pass. There is no MSan job in the current sanitizer workflow.
• The manual ZAP workflow materializes, release-builds and migrates a fresh REST API and complete LMS through the real CLI. Both baselines fail on any warning/failure, preserve INFO observations and use no ignored rules. The release blog showcase is scanned separately but remains informational because its documented presentation boundary deliberately uses a relaxed CSP and third-party assets. Its rules retain those external-asset findings for review and reduce only evidenced token/state signals or escaped showcase reflections to INFO; they do not hide findings with IGNORE. These three targets are representative evidence, not coverage of every blueprint, authenticated role, browser, proxy or deployment.
• Property tests and benchmarks are scheduled/manual evidence. The property workflow preserves the complete all-feature release-mode regression suite in eight parallel shards and separately runs the ORM and Connect property contracts with 10,000 generated cases. The eight published benchmark groups, backed by nine Criterion binaries, emit non-blocking alerts at a 20% regression and feed the public benchmark hub; they are not a promise against every nanosecond-level regression.

Fuzzing and OSS-Fuzz

The manual fuzzing.yml matrix covers all 40 declared libFuzzer targets: Core 12, ORM 5, Security 7, Connect 3, Mail 4, AI 3, IoT 3, Capital 1, Nexus 1, and Studio 1. The checked-in .github/fuzz-targets.json is validated against every */fuzz/Cargo.toml, corresponding lockfile and source file before either the manual campaign or weekly corpus job can fan out. Release mode then compiles all targets in ten package-level preflight jobs before starting any long campaign. Both jobs use versioned per-target corpora and one content-addressed compiler-cache namespace; campaign failures retain their exact reproducer, and the weekly job performs a bounded warm-up before minimizing and uploading each actual corpus. A clean run remains evidence only for its exact SHA, target, corpus, toolchain and time budget. Diagnostic mode is limited to one exact inventory target for five minutes and is never counted as the forty-target RC gate.

The oss-fuzz/projects/rullst directory is a local integration draft. It is not proof of upstream acceptance, continuous ClusterFuzz execution, or coverage of all 40 targets; its helper build must be completed and validated against the official OSS-Fuzz repository before submission. The integration is worth finishing, but a “100% first-pass acceptance” promise is not meaningful and should not be made.

Supply chain and release provenance

All direct third-party GitHub Actions references in this repository’s workflow files are pinned to full commit SHAs. A pinned composite action can still carry its own transitive downloads or references, so blocking integrations must also be reviewed for that behavior. RustSec exceptions are limited by deny.toml and documented with owners, controls, and expiry dates in docs/src/security-advisory-exceptions.md.

scorecards.yml runs the pinned OpenSSF Scorecard action on main pushes and weekly, uploads SARIF to GitHub code scanning, and publishes OIDC-authenticated results to the public Scorecard API so the README badge follows the latest completed analysis. The numeric score is supply-chain evidence, not a security certification.

release.yml is tag-only. It verifies source, validates the exact semantic tag against every publishable crate, packages before the first publish, and creates a tag-bound evidence bundle containing the lockfile, Cargo metadata, CycloneDX 1.5, Cargo Audit JSON, dependency policy, bounded compliance evidence, advisory exceptions, commit context, and checksums. The .crate archives and evidence receive a GitHub build-provenance attestation, while the official generic SLSA generator produces release provenance. This does not by itself establish project-wide SLSA Level 3 certification, Sigstore Cosign binary signing, or regulatory compliance.

workflow-lint.yml validates all workflow syntax, GitHub expressions, and embedded shell with Actionlint 1.7.7. Its container is pinned to an immutable linux/amd64 digest, just like third-party GitHub Actions are pinned to full commit SHAs.

architecture.yml is repository-owned and deterministic. It rejects any internal dependency edge or optionality change that is not reflected in the reviewed crate-architecture-policy.json. The earlier TangleGuard integration was removed because its composite action downloaded an unversioned latest binary without a repository-pinned checksum, which was unsuitable for a blocking supply-chain gate.

Workflow inventory (37 definitions)

Durations are intentionally omitted because runner load, cache state, and the dependency graph make static estimates unreliable.

Preserved next-generation roadmap

These ideas remain valuable, but are not current guarantees:

The goal of this roadmap is stronger, reproducible evidence—not a larger number of badges or absolute claims that no finite test suite can establish.