BACKBLAZEFIELD GUIDE · BY NIX UNOFFICIAL
SecurityMigrationPartnersFAQGlossaryQuizAboutUnofficial. Not affiliated with Backblaze, Inc.
WHITEBOARD / FILE 11
data flowrestore pathattacker / B2 nodes

11. Whiteboard: AI training data lake with GPU clouds

Start with the lifecycle, not the product. Ingest, prepare, train, checkpoint, serve, retain.

Whiteboard outcome

One S3 namespace on B2 as the system of record, feeding GPU clusters in two or more clouds through $0-egress paths, with a hot tier where it belongs, checkpoints flowing back, and the throughput number written on every arrow.

Before you draw: the five numbers

  1. Corpus size today and in 12 months (PB).
  2. Full-dataset reads per month (the read multiple).
  3. Target aggregate throughput (GB/s) and per-node.
  4. Where the GPUs are (CoreWeave, Vultr, Lambda, Crusoe, on-prem, hyperscaler).
  5. Hot tier in place? (VAST, WEKA, Lustre, local NVMe, none)

Write all five in the top-right corner before drawing anything else.

Draw it in 11 steps

  1. Left: “Raw sources”: web crawl, licensed media, sensors, app logs, partners. Arrow right labeled “ingest (free upload, free PUT)”.
  2. Center-left: big “Backblaze B2: system of record” rectangle with three bucket icons: raw/, curated/, checkpoints/. Say: “One namespace. Every stage writes back here.”
  3. Above B2: “Data prep / curation” box (Spark, Ray, Pixeltable, CVAT labeling). Two-way arrows to raw/ and curated/. Say: “Upload and delete are free, so you can iterate on curation without a delete penalty.”
  4. Right, top: “GPU cloud A: CoreWeave” box. Arrow from curated/ labeled “$0 egress (partner)”.
  5. Right, middle: “GPU cloud B: Vultr or Equinix Metal”. Arrow labeled “$0 egress (partner)”.
  6. Right, bottom: “GPU cloud C: other / on-prem”. Arrow labeled “3x free, then $0.01/GB, or Overdrive: $0 anywhere”.
  7. Between B2 and a GPU box: small box “Hot tier: VAST / WEKA / local NVMe”. Say: “If you need line-rate to every GPU, flash caches the working set. B2 holds everything and refills the cache.”
  8. Arrows from GPU boxes back to checkpoints/: label “checkpoint writes, versioned, Object Lock optional for model provenance”.
  9. Far right: “Inference / serving” with arrow from checkpoints/ labeled “load weights”. Add “RAG docs” bucket feeding it.
  10. Overdrive box around B2 (dashed): “B2 Overdrive: dedicated vaults, 100 Gbps to 1 Tbps, unlimited egress, from $15/TB, multi-PB commit”. Say: “If your read multiple is above about 3.8x or your throughput target is tens of GB/s, this is the tier.”
  11. Agents corner: small robot icon “Agents via B2 MCP Server, scoped key”. Say: “Agents can list, upload, presign, and report egress, but only what the key allows.”
flowchart LR
  SRC[Raw sources] -- ingest: free --> RAW[(B2 raw/)]
  PREP[Prep: Spark / Ray / Pixeltable / CVAT] <--> RAW
  PREP <--> CUR[(B2 curated/)]
  subgraph B2[Backblaze B2 - system of record]
    RAW; CUR; CK[(checkpoints/)]
  end
  CUR -- $0 partner egress --> CW[GPU cloud: CoreWeave]
  CUR -- $0 partner egress --> VU[GPU cloud: Vultr / Equinix Metal]
  CUR -- 3x free or Overdrive --> OT[Other GPU / on-prem]
  CUR --> HOT[Hot tier: VAST / WEKA / NVMe] --> CW
  CW -- checkpoints --> CK
  VU -- checkpoints --> CK
  CK -- load weights --> INF[Inference / serving]
  AG[Agents via B2 MCP Server] -. scoped key .-> B2
Whiteboard diagram (rendered from the source mermaid)

B2 Neo variant (selling to a neocloud)

Replace “GPU cloud A” with the neocloud’s logo and draw B2 inside its boundary with a label “B2 Neo: white-label, your domain, your SSO, Partner API billing”. Draw the hyperscaler outside with a red “egress tax” arrow to show what B2 Neo removes. Reference pattern: CoreWeave uses Backblaze for HDD tiers under its LOTA cache, customers get new tiers with no code changes.

The throughput math

  • 1 Tbps ≈ 125 GB/s. 100 Gbps ≈ 12.5 GB/s.
  • Time to read a 1 PB corpus once at 12.5 GB/s: 1,000,000 GB / 12.5 = 80,000 s ≈ 22.2 hours. At 125 GB/s: ≈ 2.2 hours.
  • GPU idle cost framing: idle_cost = GPUs × $/GPU-hr × hours_waiting. 512 GPUs × $2.50 × 10 hours = $12,800 per stall [use their GPU rate].
  • Hume AI reported close to 10 GB/s uploads from its clusters to B2 (case study).

Discovery traps

  • “Our dataloader uses S3 tagging for splits.” Not supported. Use prefixes or a manifest.
  • “We need IAM roles from our GPU nodes.” Not supported in S3 API docs; use scoped app keys distributed by a secrets manager.
  • “We read everything every epoch from another cloud.” Model the read multiple. Above ~3.8x to non-partner destinations, Overdrive wins on list price.

From the SE chair

Never argue with the flash vendor in the room. Draw them in. “VAST/WEKA for the working set, Backblaze for the full corpus and checkpoints” makes you the architect, not the competitor.