Skip to content

Maintaining and Developing SPRING2

This guide is for developers who want to contribute to SPRING2 or maintain the build system and delivery pipelines.

Project Structure

  • src/: Core C++20 source files and headers.
  • vendor/: Bundled third-party dependencies as .tar.xz archives.
  • docs/user/, docs/dev/, docs/assays/: User, developer, and assay-specific documentation.
  • tools/dev/: Developer workflow entrypoints for linting, smoke checks, and Docker environments.
  • tools/release/: Release-packaging assets for installers, AppImage metadata, signing hooks, and DMG helpers.
  • tools/assay-reference/: Checked-in assay-detection reference assets consumed by code generation and real-data tests.
  • tools/codegen/: Native build-time generators and other source-producing helpers.
  • tools/host/: Vendored host tools such as Ninja and NASM.
  • tests/data/: Checked-in sample inputs and archive fixtures used by unit, integration, smoke, and benchmark coverage.
  • tests/: Unit tests, integration tests, smoke tests, benchmarks, fixtures, and test support headers.

Build System Architecture

SPRING2 uses a modular CMake build system. Each vendor dependency in the vendor/ directory is self-contained:

  • Dependencies are extracted on-the-fly during CMake configuration.
  • Each major dependency (e.g., libbsc, qvz, cloudflare_zlib) has its own internal CMakeLists.txt managing its build logic.
  • All internal libraries are built with POSITION_INDEPENDENT_CODE ON to ensure compatibility with enterprise Linux (PIE) requirements.

Docker Development Environments

SPRING2 provides pre-configured Docker environments that replicate the CI build environments. These are useful for local development without installing dependencies on your host machine.

Available Environments

Three Docker environments are available in tools/dev/docker/:

  • Linux (tools/dev/docker/linux/) - Ubuntu-based build environment with GCC
  • macOS (tools/dev/docker/macos/) - Clang-based environment approximating macOS toolchain (Linux-based)
  • Windows (tools/dev/docker/windows/) - Native Windows Server Core container with MinGW-w64 toolchain

All environments include:

  • CMake 4.2.0
  • Ninja build system
  • NASM assembler
  • Python 3 with pip
  • Compiler cache (ccache for Linux/macOS, sccache for Windows)
  • Fast linkers (mold/lld for Linux, lld for macOS/Windows)

Using Docker Environments

Each Docker environment bind-mounts your host repository into the container at /spring2 (Linux/macOS) or C:\spring2 (Windows), so you work directly in your main checkout.

Build an image (Compose, recommended):

docker compose -f tools/dev/docker/linux/docker-compose.yml build
# or:
docker compose -f tools/dev/docker/macos/docker-compose.yml build
docker compose -f tools/dev/docker/windows/docker-compose.yml build

Build an image (plain Docker):

docker build -f tools/dev/docker/linux/Dockerfile -t spring2:linux .
# or:
docker build -f tools/dev/docker/macos/Dockerfile -t spring2:macos .
docker build -f tools/dev/docker/windows/Dockerfile -t spring2:windows .

Run interactively:

docker compose -f tools/dev/docker/linux/docker-compose.yml run --rm spring2-build-linux
# or:
docker compose -f tools/dev/docker/macos/docker-compose.yml run --rm spring2-build-macos
docker compose -f tools/dev/docker/windows/docker-compose.yml run --rm spring2-build-windows

Build SPRING2 inside the container:

# Linux container
cmake -S /spring2 -B /spring2/out/build-linux -G Ninja
cmake --build /spring2/out/build-linux --parallel
cmake --install /spring2/out/build-linux --prefix /spring2/out

# macOS container
cmake -S /spring2 -B /spring2/out/build-macos -G Ninja
cmake --build /spring2/out/build-macos --parallel
cmake --install /spring2/out/build-macos --prefix /spring2/out

# Windows container
cmake -S C:/spring2 -B C:/spring2/out/build-windows -G Ninja
cmake --build C:/spring2/out/build-windows --parallel
cmake --install C:/spring2/out/build-windows --prefix C:/spring2/out

The same convention applies outside Docker: configure into out/build*, keep editor tooling pointed at out/clangd/compile_commands.json, and install into out/.

Source changes on the host are visible immediately inside the container; rebuild the image only when you change Docker dependencies/tooling.

Notes on macOS Environment

The macOS Docker environment is Linux-based with Clang, not true macOS. It's useful for:

  • Local development matching the macOS CI compiler (Clang)
  • Testing platform-agnostic C++ code
  • Quick iteration without platform-specific behavior

For actual macOS testing, use the existing GitHub Actions CI or a native macOS machine.

Notes on Windows Environment

Windows containers require:

  • Windows 10/11 with Hyper-V or WSL2
  • Docker Desktop for Windows
  • Significantly more resources than Linux images (~2.5GB+)
  • Docker Desktop must be switched to Windows containers mode
  • Working directory is C:\spring2 inside the image

Quality Control

Linting

We use clang-format and several custom lint scripts. The cross-platform lint entrypoint is:

python tools/dev/lint/lint.py

The cross-platform cppcheck entrypoint follows the same pattern:

python tools/dev/lint/cppcheck.py

For Linux-only leak checking, the valgrind smoke entrypoint is:

python tools/dev/smoke/valgrind_smoke.py

Run at least one build before linting so out/clangd/compile_commands.json exists, or copy it from the active build tree if you configured without building.

Compiler Cache (Faster Rebuilds)

For faster incremental development builds, SPRING2 supports compiler cache launchers during CMake configure:

  • Windows: sccache
  • Linux/macOS: ccache (falls back to sccache if ccache is unavailable)

This behavior is enabled by default with -DSPRING_ENABLE_COMPILER_CACHE=ON.

Install suggestions:

  • Linux (Debian/Ubuntu): sudo apt-get install -y ccache
  • macOS (Homebrew): brew install ccache
  • Windows (winget): winget install --id Mozilla.sccache -e

Disable compiler cache explicitly (if needed):

cmake -S . -B out/build -G Ninja -DSPRING_ENABLE_COMPILER_CACHE=OFF

Check cache stats:

  • ccache -s (Linux/macOS)
  • sccache --show-stats (Windows, or any platform using sccache)

Faster Linker (Not on macos)

SPRING2 can also select a faster linker during CMake configure, enabled by default with -DSPRING_ENABLE_FAST_LINKER=ON.

Selection order:

  • Linux: tries mold first (-fuse-ld=mold), then lld (-fuse-ld=lld)
  • Windows (GNU/Clang toolchains): tries lld when supported

Install suggestions:

  • Linux (Debian/Ubuntu): sudo apt-get install -y mold lld
  • Windows (MSYS2/MinGW-w64 UCRT): pacman -S --needed mingw-w64-ucrt-x86_64-lld

If no compatible fast linker is available, the build falls back to the platform default linker automatically.

Disable this behavior explicitly (if needed):

cmake -S . -B out/build -G Ninja -DSPRING_ENABLE_FAST_LINKER=OFF

Precompiled Headers (PCH)

SPRING2 uses precompiled headers to accelerate incremental rebuilds, enabled by default with -DSPRING_ENABLE_PRECOMPILED_HEADERS=ON.

The precompiled header (src/pch.h) includes stable, frequently-used headers:

  • Standard library containers (<vector>, <string>, <array>)
  • I/O utilities (<iostream>, <fstream>, <iomanip>)
  • File system and concurrency (<filesystem>, <mutex>, <thread>, <atomic>)
  • OpenMP (<omp.h>)

When enabled, CMake compiles this header once and caches it, significantly reducing compile time on subsequent rebuilds. The improvement is especially noticeable after touching a few files or after clean configuration, where PCH cache is reused across translation units.

Disable precompiled headers explicitly (if needed):

cmake -S . -B out/build -G Ninja -DSPRING_ENABLE_PRECOMPILED_HEADERS=OFF

Unit and Integration Tests

We use the doctest framework for both granular unit testing and high-level integration testing. You can build and run all tests using standard CMake/CTest workflows:

cmake -S . -B out/build
cmake --build out/build --parallel --target unit-tests integration-tests smoke-tests
ctest --test-dir out/build --output-on-failure

If you already built only the main executable, rebuild the test targets before invoking ctest:

cmake --build out/build --parallel --target unit-tests integration-tests smoke-tests

That same out/build tree is also the default source for out/clangd/compile_commands.json and the smoke/bench helper defaults.

  • Unit Tests: tests/unit/*.cpp (Split by low-level helpers and assay detection behavior).
  • Integration Tests: tests/integration/*.cpp (Split by archive integrity, reader behavior, assay workflows, and grouped archive scenarios).
  • Smoke Tests: tests/smoke/*.cpp (Split C++ CLI behavioral validation executed through the smoke-tests binary).

Archive Metadata Version Compatibility

SPRING2 now treats archive creator versioning as part of the compatibility contract:

  • New archives write a metadata header version and preserve the exact creator version string in cp.bin.
  • Older archives that predate the metadata header are still supported and default to creator version 1.0.0-rc.1 when no explicit version string is present.
  • User-facing preview output should show the effective human-readable archive version and should not expose internal compatibility-routing labels unless you are debugging the implementation.
  • Future decompression-path changes should continue to route by parsed archive version, not by a coarse legacy/current split.

Coverage for this behavior lives in tests/unit/archive_metadata_version_unit_tests.cpp. If you change metadata layout, version parsing, preview version reporting, or decompression routing, update that test first and keep backward compatibility expectations explicit.

Benchmark/Test Scripts

The tests/ directory also includes benchmark and comparison scripts used for manual performance checks:

  • tests/bench/big_bench.py: End-to-end paired-end benchmark on the larger SRR2990433 dataset. Pass --no_debug to suppress SPRING's -v debug output during the run.
  • tests/bench/small_bench.py: Faster benchmark variant for quick local performance checks and assay-specific benchmark coverage.
  • tests/bench/comparison_bench.py: Compares SPRING2 against SPRING1 and Python gzip baselines.
  • tests/bench/big_thread_test.py: Debug-oriented thread-count stress benchmark for CRC/integrity investigations.
  • tests/smoke/*.cpp: Lightweight CLI sanity checks split into themed smoke test units.
  • tests/integration/*.cpp: Archive round-trip and API-level integration coverage split into smaller themed units.
  • tests/unit/*.cpp: Focused tests for low-level helpers and assay detection behavior.

Diagnostic Key Legend

Many debug logs use a normalized key schema for fast triage:

  • block_id: Pipeline stage or thread/block identifier that emitted the log.
  • path: File path or logical input source being checked.
  • expected_bytes: Expected size/count/bound for the check.
  • actual_bytes: Observed size/count/value at runtime.
  • index: Loop position, record number, or block-local offset where the check failed.

Interpretation tips:

  • expected_bytes=1, actual_bytes=0: Usually open/availability failure (resource not usable).
  • actual_bytes < expected_bytes: Typical short read/truncation signal.
  • actual_bytes > expected_bytes: Often out-of-range metadata/value for bound checks.
  • index pinpoints the offending element; combine with block_id to locate the stage.

Release Pipeline

The project uses GitHub Actions for automated multi-architecture releases.

  • Runners: Uses ubuntu-latest (x86_64) and ubuntu-24.04-arm64 (native ARM silicon).
  • Environment: Builds are performed inside AlmaLinux 8 Docker containers to pin the GLIBC baseline to 2.28, ensuring maximum portability across enterprise distributions (RHEL 8+, Ubuntu 20.04+, etc.).
  • Artifacts: The pipeline produces portable AppImages for Linux, universal Mach-O binaries for macOS, native executables for Windows, and architecture-specific Windows installers built with Inno Setup.

Windows Signing

Windows release signing is optional and is only enabled when the GitHub Actions repository secrets below are configured:

  • WINDOWS_SIGNING_CERT_BASE64: Base64-encoded .pfx certificate contents.
  • WINDOWS_SIGNING_CERT_PASSWORD: Password for the .pfx file. Leave unset if the certificate has no password.

To prepare the certificate payload on Windows PowerShell:

[Convert]::ToBase64String([IO.File]::ReadAllBytes('C:\path\to\certificate.pfx'))

The release workflow reads those secrets, uses tools/release/windows/sign-artifact.ps1, signs both the raw spring2.exe binaries and the final setup executables, and skips signing entirely when WINDOWS_SIGNING_CERT_BASE64 is absent.

Windows Installer Tasks

The Windows installer currently supports these optional setup tasks:

  • Add the install directory to the system PATH.
  • Create a desktop shortcut to spring2.exe.

It also creates a Start Menu shortcut to spring2.exe by default.