svaba cloud scatter-gather — launcher, cost & monitor

How it works

① worker image
svaba + reference baked in→ ② N worker VMs
BAM disk attached read-only→ ③ each runs svaba -k <region>→ ④ upload to GCS bucket
+ .done_part* marker→ ⑤ gather
merge + postprocess

The reference/index is local to each worker (in the image). The BAM disk is one read-only PD fanned out to all workers. Sharding is by chromosome groups (few big shards) — far cheaper than many tiny fixed-size chunks, because each shard pays the ~5 GB index load + insert-size learning once.

1 · Configure the run

GCP environment

Project Zone Worker image (svaba+ref baked in) BAM data disk name (RO) Mount point on worker GCS output bucket

Available images — paste `gcloud compute images list --filter=name~svaba --format="value(name)"` to populate the dropdown

Inputs (paths as seen on the worker)

Analysis id Reference fasta (in image) Tumor BAM (on mounted disk) Normal BAM (optional) Blacklist BED (in image) bam-params GCS cache (gs://…)

Sharding

Mode Partitions / chrom groups (N) Shards (N, computed)

Machine & concurrency

Machine type Threads per worker (-p) Max concurrent VMs (0 = all at once) Region vCPU quota Est. wall-time per shard (min) Use --bam-params (learn once) recommended Use Spot/preemptible VMs ~70% cheaper Merge + postprocess (gather) --dump-reads (debug BAMs)

2 · Cost estimate approximate

ON-DEMAND

$0

SPOT (~70% off)

$0

BILLED VM-HOURS

0

≈ WALL-CLOCK

0

Pricing is rough on-demand us-central1 $/hr — verify against the live GCP price list. Cost ≈ N shards × per-shard wall × ($/hr), + a small VM boot tax. --bam-params trims the per-shard insert-learning overhead (~5 CPU-min/shard).

3 · Commands to run

Step -1 — build a worker image from a git commit (optional, when you change svaba)

git ref (hash / branch / tag) new image name (hyphens only)

Boots the builder VM, checks out the commit, rebuilds svaba + bwa + fermi-lite at -O3 -march=native, verifies it compiled, installs it on PATH, then snapshots a new image (data disk untouched). Afterwards, re-run the gcloud compute images list above and paste it to pick the new image. GCP image names can't contain underscores.

Step 0 — precompute insert-size params once (learn-only)

Run this once (any quick region; it always learns genome-wide). Writes the small params TSV every shard then reuses, so no shard repeats the ~14M-read insert sweep. Put the file where the workers can read it (the RO data disk, or fetch from the bucket in the startup script).

Step 1 — launch the scatter (creates N worker VMs)

Step 2 — monitor (poll the done-markers)

Step 3 — gather / merge (if not using `--merge`)

Teardown — delete the worker VMs

4 · Monitor dashboard

paste `gsutil ls <bucket>/.done_part*`

progress

—

Static page can't poll GCS live (no backend). Paste the output of the monitor command (or gcloud compute instances list) and it renders progress. Re-paste to refresh.