Meshly Community Cloud wiki

1 · Overview

Community Cloud

Relevant source files

Community Cloud is the code behind Meshly, live at meshlycloud.com. It runs other people's apps on machines that people share. Some machines are laptops in a community pool. Others are Linux servers that an organization owns. One small control plane decides what should run where. An agent on every machine makes that happen and reports back.

This wiki explains the code on the dev branch. Each page starts from the problem a part of the system solves, then shows how the code solves it, with diagrams and links to the exact lines. If you are new, read the pages in order: each one builds on the ones before it.

The problem

Most computers sit idle most of the day. Someone who needs to host a small app pays a cloud provider instead.

The idea

Pool the idle capacity. A central service hands out work, and each machine runs it in a container and keeps a public address working.

What it costs

The machines are unreliable and belong to someone else. Almost every part of the code exists to cope with that.

The system in one picture

Everything central runs on one small server. Machines never accept incoming connections: they always dial out, to the control plane for their work and to the Piko gateway to carry visitors' requests to their apps.

flowchart LR
  subgraph people["People"]
    direction TB
    D["Dashboard<br>web/"]
    C["ccloud CLI<br>cmd/ccloud"]
    W["Desktop app<br>cmd/meshly-desktop"]
    V(["Visitors of an app"])
  end
  subgraph server["Main server"]
    direction TB
    N["nginx behind Cloudflare"]
    CP["Control plane<br>internal/cp"]
    PK["Piko gateway"]
    PG[("PostgreSQL")]
    OBS[("VictoriaMetrics<br>VictoriaLogs")]
    RG[("Image registry")]
  end
  subgraph pool["Shared machines"]
    direction TB
    A1["Agent + Podman<br>a laptop"]
    A2["Agent + Podman<br>an org server"]
    B["Builder<br>Docker + Railpack"]
  end
  D --> N
  C --> N
  W --> N
  V --> N
  N --> CP
  N --> PK
  CP --- PG
  CP --- OBS
  CP --- RG
  A1 -- "sync: long poll" --> N
  A2 -- "sync: long poll" --> N
  A1 -- "app tunnel" --> PK
  A2 -- "app tunnel" --> PK
  B -- "image push" --> N
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class CP hl
Arrows show who opens the connection. Nothing connects into a shared machine.

Sources:

What happens when someone deploys an app

Follow one deploy from start to finish. Every later page zooms into one of these steps.

%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
  autonumber
  actor U as Consumer
  participant CP as Control plane
  participant DB as PostgreSQL
  participant A as Agent on a machine
  participant P as Piko gateway
  U->>CP: POST /api/v1/deployments
  CP->>DB: save the workload
  CP->>CP: scheduler picks a machine
  CP->>DB: add a copy, raise the machine's version
  CP-->>A: answer the sync request it was holding
  A->>A: pull the image, start the container
  A->>A: health check passes
  A->>P: open a tunnel for the app
  A->>CP: next sync says running, tunnel up
  CP->>DB: open a lease, start the meter
  Note over P: visitors at https://app.apps-domain now reach the container
A deploy, end to end. The numbered steps match the list below.
  1. The consumer asks for an app: an image (or a GitHub repository), a size and a mode. See Build and deploy.
  2. The control plane saves it as a workload in PostgreSQL. See Data model.
  3. The scheduler scores every eligible machine and picks one. See Scheduler and placement.
  4. It records a copy of the app on that machine and raises the machine's version number.
  5. The machine's agent was waiting on a held sync request. It gets the whole new desired state at once. See Desired state and reconciliation.
  6. The agent pulls the image and starts a container with hard limits. See Running containers.
  7. It checks the app until it answers. See Health checks.
  8. Only then does it open a tunnel to Piko, so visitors never reach a copy that is still starting. See Piko tunnels.
  9. The next sync reports the copy running with its tunnel up.
  10. The control plane opens a lease and bills it by the minute in the community cloud. See Leases and credits.

The numbers that shape behaviour

Most behaviour in this system comes from a handful of timers. When something surprising happens, start here.

TimerValueWhat it doesWhere
Sync hold10 sHow long the control plane holds an agent's request when nothing changed-hold
Scheduler tick5 sPlacement also runs on every change; the tick is the safety netassignLoop
Liveness check2 sHow often silent machines are looked forlivenessLoop
Fast down15 sNo sync and its Piko tunnels gone: the machine is down now-fast-down-after
Offline30 sNo sync: the machine gets no new work-offline-after
Agent fence60 sAn agent with no answer for this long closes its tunnels and pauses its VPSssyncFence, vpsFence
Lost2 minIts copies are marked lost and placed elsewhere-lost-after
Piko token10 minRenewed with under 4 minutes left, so about every 6 minutesinternal/agent/tunnel.go
Meter1 minOpen leases are billed-meter-every
VPS photo1 minA VPS's changed disk blocks go to the copy store (15 min with the write log)-vps-copy-every
Write log batch2 sA VPS's flushed writes go to the hubinternal/agent/logship.go

Sources:

How to read this wiki

How this wiki was made

It was generated from the code on dev at commit d0ca527 and from the team's own records in docs/. The decisions and incidents it quotes are the team's; the explanations are summaries. Where the wiki and the code disagree, the code wins. Findings from the review of 20 November 2026 are marked as such on Known limits.

1.1 · Overview

First principles

Relevant source files

Start from nothing. You want to run a web app on a stranger's laptop and give it a public address. Write down everything that can go wrong. The list is short, and every package in this repository answers one item on it.

The five hard problems

flowchart LR
  G["Run apps on machines<br>that other people share"]
  G --> P1["Machines sit behind<br>home routers"]
  G --> P2["Machines vanish<br>without warning"]
  G --> P3["Machines belong to<br>someone else"]
  G --> P4["Visitors need one<br>stable address"]
  G --> P5["Some apps keep<br>state on disk"]
  P1 --> S1["Machines only dial out:<br>long poll and tunnels"]
  P2 --> S2["Desired state, heartbeats,<br>failover and fencing"]
  P3 --> S3["Owner guard protects the owner,<br>stand-in keys protect the app"]
  P4 --> S4["A gateway routes by app,<br>never by machine"]
  P5 --> S5["Disk copies with generations,<br>and a write log"]
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class G hl
Each problem on the left is a fact about the machines. Each answer on the right is a design rule the code follows everywhere.

1. You cannot reach the machine

A laptop at home sits behind a router that blocks incoming connections. So the control plane can never call a machine. The machine calls the control plane, and keeps one request open so the answer can come back the moment something changes (a long poll). Visitors' traffic uses the same trick: the machine opens an outgoing tunnel to the gateway, and the gateway sends requests down it.

Sources:

2. The machine disappears

The owner closes the lid, the power goes, the Wi-Fi drops. Nothing tells the control plane. So every request from a machine doubles as a heartbeat, and silence past fixed timers means offline, then lost. The control plane always sends the whole desired state, never a diff, so a machine that missed something corrects itself on its next answer. And a machine that loses contact must assume it was replaced: after 60 seconds it stops serving (fencing), so two copies never fight over the same app.

Sources:

3. The machine belongs to someone else

This cuts both ways. The owner must always win: our containers get the lowest CPU weight, the agent keeps a memory reserve, and if the owner needs memory the newest shared container is evicted at once. The app must be protected from the owner too: the owner is root on their own laptop and can read anything a container holds. So API keys never go to community machines. The app gets a stand-in key, and the control plane swaps in the real one when the app calls out.

Sources:

4. Visitors need one address

An app may move between machines several times a day. Its address must not. The gateway (Piko) knows apps by name, not machines by address. Whichever machine currently runs the app holds the tunnel for that name, so moving an app is just a different machine opening the same tunnel.

Sources:

5. Some apps keep state

A stateless web app can restart anywhere. A VPS cannot: its disk is the point. So a VPS gets sealed copies of its disk sent off the machine, a generation number that says which machine may write, and a write log that ships every saved write within seconds. Only the machine holding the current generation can add to the copies.

Sources:

Three rules that carry most of the weight

RuleWhyWhere you will see it
Only dial outHome networks block incoming connections, and machine addresses must never be exposedSync long poll, Piko tunnels, the agent's own VPS tunnel
The control plane holds the truth; machines convergeMachines miss messages, restart and lose state. A full desired state with a version heals all of thatdesired.go, the agent's reconciler, VPS specs
Assume anything can be lost, so version, retry or fence itA late message from a machine that was replaced must never countCopy attempts, report filtering in applySync, VPS generations, the 60 s fence

Check yourself

Why does the control plane send the full desired state on every sync, instead of only what changed?

A machine can miss an answer, restart, or lose its saved state. With the full list and a version number, the next answer always brings it back in line. No message is special, so none can be lost in a way that matters.

Why must a machine stop serving after 60 seconds without an answer?

From the control plane's side, that machine may already be marked lost and its apps started elsewhere. If it kept serving, two copies of one app could answer visitors with different data. Stopping first is the safe guess.

1.2 · Overview

Glossary

Relevant source files

The code uses a small, consistent vocabulary. The dashboard sometimes uses friendlier words than the API and the database; both are listed.

People and places

WordMeaningIn the code
Control planeThe one central service: API, scheduler, liveness, billinginternal/cp
AgentThe program on every shared machine that runs work and reports backinternal/agent
MachineA computer that joined the pool (laptop, server or builder)machines table
ProviderA person who shares a machinemachines.owner_id
ConsumerA person who deploys appsworkloads.created_by
Community cloudThe open pool of anyone's machines; pays in creditsorg_id empty
Organization cloudAn organization's own servers, used only by its people, freeorg_id set
Admin, member, consumerRoles in an organization. Members bring servers; consumers deployadmin, maintainer, member
Platform adminThe first account; the only one who can look at another cloudscopeOf, ?org=
BuilderA trusted machine that turns source code into images and runs no appsmachine role builder

Apps and their copies

WordMeaningIn the code
Deployment, workloadWhat a consumer asked for: image, size, mode, variablesworkloads
Copy, instanceOne run of a workload on one machineinstances
PrimaryA copy that serves visitors through a tunnelrole = 'primary'
StandbyA warm copy with no tunnel, ready to take overrole = 'standby'
Modespot: one copy. standby: a primary and a standby. active: 2 to 5 serving copiesworkloads.mode
AttemptA counter on each new copy, so a late report about an old one is ignoredinstances.attempt
Sizesmall, medium, large (1, 2, 4 units) or custom CPU and memoryappSizes
UnitA quarter of a core and 256 MB; what billing countsunitsOf
DigestThe exact build an image tag points to; every copy runs the same oneworkloads.digest
RolloutReplacing copies one at a time with a new build or new variablesrollout.go
Job, runA container that runs to completion, by hand or on a cron schedulejobs, kind job

Keeping machines in line

WordMeaningIn the code
Capacity promiseA provider's statement of what a machine will share, before it joinscapacity
Join key, server keyOne-time key for a promise; an org member's reusable key for serverskeys, ek_…, sk_…
Desired stateEverything one machine should run, sent whole on every answerSyncResponse
VersionA number per machine that goes up on every change to its desired statemachines.version
SyncThe agent's long-polled request: its report in, desired state outPOST /agent/v1/sync
ReconcileMake what runs match what is desired, with the smallest changereconcileOnce
FencingStopping work on a machine that may have been replacedsyncFence, VPS generations
FailoverMoving an app off a lost machine; every one is recordedfailovers
Owner guardEvicts a shared copy the moment the owner needs memoryguardOwner
LeaseOne copy's use of a machine, from start to stop; what the meter billsleases
CreditsThe community cloud's money, kept as millicreditscredit_ledger

Traffic and secrets

WordMeaningIn the code
PikoThe open-source gateway that carries visitors down machines' outgoing tunnelsgithub.com/andydunstall/piko
EndpointAn app's name at the gateway; a tunnel serves one endpointInstanceSpec.Endpoint
Piko tokenA 10 minute JWT listing the endpoints a machine may servepikoToken
Stand-in keyA fake API key (ccs_…) a community app holds instead of the real onestandIn
EgressThe control plane's proxy that swaps a stand-in for the real key/egress/
Agent tunnelEach agent's own WebSocket to the control plane, for VPS terminals and SSH/agent/v1/tunnel
Custom domainAn app's own name, proven with a TXT recorddomains

VPS words

WordMeaningIn the code
VPSA Linux box with root that stays on one machinevps
GenerationGoes up each time a VPS is placed; only that placement may send copiesvps.generation
Photo, copyA snapshot of a VPS's disk; only changed blocks are sentvps_photos
PackUp to 512 changed 64 KiB blocks, compressed and sealedvps_packs
Write log, batchEvery write, logged; flushed writes sent every 2 secondsinternal/wlog, vps_log
written_genThe placement whose machine sent the newest copy or batchvps.written_gen
Leftover, set asideAn old disk kept after a move, in case it holds the newest datavps_leftovers, .old-g…
DrainA planned move: stop, send a last copy, then movevpsmove.go

1.3 · Overview

Repository map

Relevant source files

One Go module (fyp) builds five programs. A React dashboard lives in web/ and is compiled into the control plane. The team's documentation lives next to the code and changes in the same commits.

Programs and packages

flowchart LR
  subgraph cmds["cmd: programs"]
    cpb["controlplane"]
    agb["agent"]
    cc["ccloud"]
    desk["meshly-desktop"]
    fyp["fyp admin CLI"]
  end
  subgraph pkgs["internal: packages"]
    cp["cp<br>control plane"]
    ag["agent"]
    proto["proto<br>shared types"]
    det["detect"]
    tun["tunnel"]
    obj["objstore"]
    vd["vpsdisk"]
    wl["wlog"]
    nbd["nbd"]
    jwt["jwt"]
  end
  cpb --> cp
  agb --> ag
  agb --> nbd
  cc --> tun
  fyp --> proto
  cp --> proto
  cp --> det
  cp --> tun
  cp --> obj
  cp --> vd
  cp --> jwt
  ag --> proto
  ag --> tun
  ag --> vd
  ag --> wl
  ag --> nbd
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class cp hl
  class ag hl
Imports between the Go packages. proto is the contract: both sides of every wire use its types. The desktop app imports nothing internal: it drives the agent through its local API.

What lives where

PathWhat it isRead it when
internal/cpThe control plane: API, scheduler, liveness, leases, builds, domains, VPS hub (about 14,700 lines)You want to know what the system decides
internal/agentThe agent: sync, reconcile, Podman, tunnels, health, owner guard, VPSs (about 6,200 lines)You want to know what a machine does
internal/protoWire types shared by both sidesYou change the agent protocol
internal/detectReads a repository and predicts its apps, commands, port and variablesImport from GitHub
internal/tunnelyamux over WebSocket, line and frame helpersVPS terminals and SSH
internal/vpsdiskChanged-block detection, manifests, sealed packsVPS copies
internal/wlogA disk's write log and merged batchesThe VPS write log
internal/nbdA userspace NBD server and netlink device setupThe VPS write log
internal/objstoreA folder or S3 (SigV4) behind one interfaceWhere VPS copies are kept
web/The dashboard (React, Vite); its build is committed to internal/cp/uiUI work
contracts/agent-protocol.md and openapi.yaml: how the parts talkBefore changing any wire format
docs/decisions24 decision records: options, choice, costYou ask "why is it like this?"
docs/incidentsWhat broke, why, and what was learnedBefore touching timers or tunnels
docs/operations.mdHow the live platform runs, updates and recoversDeploying
scripts/Local stack, builds, packaging, the worm guard, server setup and deploysRunning anything

Run it locally

The local stack needs WSL or Linux as root with Podman. Start PostgreSQL, VictoriaMetrics, VictoriaLogs and a registry in containers (the commands are in the README), then:

  1. bash scripts/dev-up.sh starts Piko and a control plane on port 9080 with an empty database. The first account to sign up becomes the platform admin.
  2. In the dashboard, promise a machine (Provide, Add a machine, A Linux server) to get a join key.
  3. bash scripts/dev-up.sh agent lab-1 KEY starts a local agent that joins with it, as a real machine would.
  4. go test ./... uses the same databases and skips tests whose database is not running.

Before any npm command

Run bash scripts/ci-guard.sh first. This repository was hit twice by a worm that hid code in .vscode/, npm scripts and build configs. The guard fails on those signs and on lock-file packages from outside the npm registry. CI runs it before anything else.

Sources:

2 · Architecture

Architecture

Relevant source files

The architecture is a hub and many spokes. The hub is one server that holds every decision and every record. The spokes are machines that only ever call the hub. This page shows the pieces, where they run, who connects to whom, and who trusts whom.

The problem

Hundreds of unreliable machines must act as one cloud, on a small budget, run by students.

The idea

Put all state and all decisions in one control plane with one database. Keep machines simple: they run what they are told and report what they see.

What it costs

One server is a single point of failure, and everything competes for about 1.9 GB of memory.

The pieces and where they run

flowchart TB
  subgraph edge["Internet edge"]
    CF["Cloudflare<br>TLS, 100 MB body cap"]
  end
  subgraph main["Main server: AWS t3.small, Sydney"]
    NG["nginx<br>ports 80 and 443"]
    CP["meshly-cp<br>control plane, 127.0.0.1:8080"]
    PK["meshly-piko<br>proxy :8100, upstream :8001"]
    PG[("postgresql<br>records")]
    VM[("meshly-vm<br>metrics")]
    VL[("meshly-vl<br>logs, 7 days")]
    RG[("meshly-registry<br>images, 127.0.0.1:5000")]
  end
  BLD["meshly-builder<br>same AWS zone, stopped when idle"]
  subgraph pool["Shared machines"]
    LAP["Laptops in the community<br>desktop app, agent in WSL"]
    SRV["Organization servers<br>ccloud join, systemd"]
  end
  CF --> NG
  NG --> CP
  NG --> PK
  CP --> PG
  CP --> VM
  CP --> VL
  CP --> RG
  LAP --> CF
  SRV --> CF
  BLD -- "private network" --> NG
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class CP hl
Every service on the main server listens on 127.0.0.1. Only nginx faces the internet, and only through Cloudflare.

Two more pieces sit outside this picture. Community capacity during the pilot comes from two Oracle Cloud Always Free machines in Mumbai. The builder is a separate AWS machine that costs about $0.11 an hour, so it stays stopped when nobody is building.

Sources:

Who connects to whom

Read this table as the system's wiring. Notice that every row starts at a machine or a person, never at the server.

FromToWhatWhy this way
AgentPOST /agent/v1/syncLong poll: report in, desired state outWorks behind NAT; doubles as heartbeat
AgentPiko upstream, /piko/v1/upstream/<app>One WebSocket per app it servesVisitors reach the app with no open port on the machine
AgentGET /agent/v1/tunnelIts own WebSocket with yamux, while it keeps a VPSTerminals and SSH that must outlive Piko's token renewals
Agent/v2/ registry gatePulls images of apps it runsAccess follows what the control plane already knows
Builder/v2/ over AWS's private networkPushes a built imageCloudflare refuses bodies over 100 MB
Browser/api/v1/…The dashboard's API, session cookieOne API for every client
Visitorhttps://<app>.<apps domain>nginx sends it to Piko's proxy, Piko down a tunnelRouting by app name, not by machine
Community app/egress/…API calls with a stand-in keyThe real key never leaves the control plane

Sources:

Where data lives

Three kinds of data, three stores, all on the main server's disk.

DataStoreWhy that store
Records and events: accounts, apps, copies, leases, VPSsPostgreSQL 16Few rows, many joins, transactions for placement and fencing
Numbers over time: CPU and memory samplesVictoriaMetrics (SQLite when absent)Compact, fast range queries, little memory
Text: app log linesVictoriaLogs, 7 days, at most 2 GiBFull-text search; the latest 200 lines per copy also stay in Postgres
Built imagesCNCF Distribution registryStandard /v2/ protocol behind the control plane's gate
VPS disk copiesA folder or an S3 bucket (VPS_COPIES)Sealed packs; the maps that make sense of them stay in Postgres

Single disk

Everything above shares one disk on one machine, and the server has no scheduled backups yet (roadmap, Phase 0). See Known limits.

Sources:

Who trusts whom

Security in this system is mostly a question of trust boundaries. Keep this table in mind when you read any code that hands out a secret or accepts a report.

PartyTrusted withNot trusted with
Control plane and buildersEverything: they decide and build what runs everywhere
An organization's serversIts own apps' real secretsOther organizations' apps (placement keeps them apart)
A community machineRunning a copy, reading what is in itAPI keys (it gets stand-ins), a VPS's copy store, any other machine's data
An agent's reportWhat it sees on its own machineAnything about copies it does not hold: reports about older attempts or generations are ignored

Check yourself

Why does the builder push over AWS's private network instead of the public name?

Cloudflare, in front of the public name, refuses request bodies over 100 MB, and image layers are often larger. nginx accepts uploads only from the private range, so the public path stays read-only.

What would break first if the main server went down for ten minutes?

Everything new: no deploys, no failovers, no billing, and new visitors could not reach apps because Piko runs there too. Running containers keep running on their machines, but their tunnels go through the server, so apps stop answering visitors.

2.1 · Architecture

Desired state and reconciliation

Relevant source files

This is the heart of the system. The control plane keeps, for every machine, a list of what should run there and a version number. The agent keeps asking for that list and makes its machine match it. Every other feature plugs into this loop.

The problem

You cannot push commands to a machine you cannot reach, and a machine that misses one command is now wrong forever.

The idea

Never send commands. Send the whole desired state with a version, and let the machine close the gap itself, as often as needed.

What it costs

One open request per machine, held in memory, so only one control plane can run today.

Commands versus state

Imagine telling a laptop "start app A", then "stop app B". If the laptop misses the second message, it runs B forever and nobody notices. Now imagine telling it instead "you should run exactly A and C, version 42". If it misses version 42, it gets version 43 later, which again lists everything. Nothing is ever lost for good. This is called level-triggered control, and it is why a rebooted agent needs no special recovery.

flowchart TB
  subgraph cp["Control plane"]
    direction LR
    W["Any change:<br>deploy, failover, redeploy"] --> BV["bumpVersion<br>machines.version + 1"]
    BV --> WK["waits.wake(machine)"]
    DS["desired(machine)<br>copies, VPSs, builds, Piko token"]
  end
  subgraph ag["Agent"]
    direction LR
    SY["syncOnce<br>long poll"] --> AP["apply<br>save state.json"]
    AP --> RC["reconcileOnce<br>smallest change"]
    RC --> PD["Podman containers<br>and tunnels"]
  end
  WK -. "ends the held request" .-> DS
  DS -- "full desired state" --> SY
  PD -- "report on the next sync" --> W
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class BV hl
  class RC hl
The loop. Changes on the left raise a version and wake the machine; the agent applies the full state and reports what it sees.

The long poll, step by step

%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
  participant A as Agent
  participant H as handleSync
  participant N as notifier
  participant DB as PostgreSQL
  A->>H: POST /agent/v1/sync (have_version 41, report)
  H->>N: wait(machine): take the channel first
  H->>DB: applySync: heartbeat, copy states, samples
  H->>DB: SELECT version
  alt version is still 41
    H->>H: hold up to 10 s
    Note over N: the scheduler commits a change and calls wake(machine)
    N-->>H: channel closed
  end
  H->>DB: desired(machine)
  H-->>A: full desired state, version 42
  A->>A: apply, then reconcile
  A->>H: next sync at once (have_version 42)
The request is both the heartbeat and the delivery channel. With nothing to say, it returns after the hold, and the agent asks again.

One detail in handleSync is worth slowing down for. The handler takes the wake channel before it reads the version. If it read the version first, a change could land in the gap between the read and the wait, and the request would sleep for 10 seconds on news that already arrived. Taking the channel first closes that gap: any later wake closes the channel the handler already holds.

Sources:

How long a request is held

Normally 10 seconds (-hold). While someone follows an app's logs, the hold drops to 1 second, so new lines arrive every second. An earlier design had the agent cut its own held request every 2 seconds to send lines; it never received an answer, went 60 seconds without a successful sync, and fenced itself: the app went down for as long as anyone watched its logs. That incident is why the server, not the agent, now shortens the hold.

Sources:

What the desired state contains

FieldWhat it says
versionThe machine's version; the agent sends it back as have_version
instances[]Every copy to run: image pinned to a digest, command, port, CPU and memory, role (primary or standby), variables and secrets as the cloud allows, health check
piko_tokenA fresh 10 minute JWT listing the endpoints this machine may serve
vps[], vps_remove[]The VPSs this machine keeps, with their generation and copy settings, and the ones to delete for good
registry, builds[]The image registry to log in to; for builder machines, the builds to run

Sources:

The agent's side

When an answer arrives, apply stores it, saves it to state.json if the version changed, notes the time of the last good sync (used for fencing) and signals the reconciler. reconcileOnce then compares the list with the containers Podman reports and makes the smallest change: start what is missing, remove what is no longer wanted, and open or close tunnels. Running it twice does no harm, which is the property that makes the whole loop safe.

SituationWhat the reconciler does
Desired copy has no containerStart it, unless it is starting, failed to start, or was evicted by the owner guard
Container not in the desired listClose its tunnel with a short drain, then remove it
Primary is running and passes its health checkOpen its tunnel
No successful sync for 60 sClose every tunnel (fencing) until the control plane answers again
Owner paused sharingRemove shared containers

Sources:

Reports can arrive late

The other direction needs the same care. A report about a copy can arrive after that copy was replaced. applySync ignores any report about a copy that is no longer active, or about an older attempt, so a late "failed" can never kill the new copy and a late "running" can never bill twice. The VPS code uses generations for the same reason.

Sources:

Check yourself

A laptop sleeps for an hour and wakes up. What does it need to do to catch up?

Nothing special. Its next sync carries an old have_version, so the control plane answers at once with the full current state, and the reconciler removes whatever moved away while it slept and starts whatever is now assigned to it.

Why can only one control plane run at a time today?

The wake channels live in one process's memory. A change committed by a second process could not wake a request held by the first. Running several would need a notification shared between processes. Scaling the control plane is the roadmap's Phase 4.

2.2 · Architecture

Two clouds, one platform

Relevant source files

Community Cloud started as two student projects: one for anyone's laptops, one for an organization's servers. They turned out to need the same scheduler, tunnels, agent and billing. So there is one platform, and a single column decides which cloud a row belongs to: org_id.

The problem

Two codebases would build scheduling, tunnels, billing and the agent twice, for a difference that comes down to whose hardware it is.

The idea

One control plane, one API, one database. org_id empty means the community cloud; set means an organization.

What it costs

Every query must be scoped correctly, or one organization sees another's data.

What differs, and where

Community cloudOrganization cloudCode
org_idemptysetevery main table
MachinesAnyone's laptops, joined with the desktop appThe organization's Linux servers, joined with ccloud join and a member's server keyregistry.go
PlacementcommunityStrategy: machines with no organizationorgStrategy: only the organization's own machinesscheduler.go
SecretsAPI keys stay on the control plane; the app gets a stand-inEvery secret goes into the containersecrets.go
BillingConsumers pay credits per unit-minute; providers earn themFree; leases still record usageleases.go
Who sees whatYour own apps and machinesMembers see their servers; the admin sees everything in the organizationorgs.go

One helper decides the scope

Every handler that lists or changes data asks scopeOf which cloud the request may see. Everyone is pinned to their own: the community cloud or their organization. Only the platform admin (the first account) may pick another one with ?org=. Keeping this in one place means a future split into two products changes one function.

flowchart TB
  R["Request from a signed-in person"] --> S{"scopeOf"}
  S -- "platform admin with ?org=X" --> X["Cloud X"]
  S -- "member of an organization" --> O["Their organization only"]
  S -- "everyone else" --> C["Community cloud only"]
  X --> Q["Queries filtered by org_id"]
  O --> Q
  C --> Q
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class S hl
Scoping is one decision, made once per request.

Sources:

Check yourself

Can an organization's app ever run on a community laptop?

No. orgStrategy.Eligible accepts only machines whose org_id equals the workload's. Community machines have none, so they are never eligible.

Why does an organization's app get its real API keys, when a community app does not?

An organization runs its apps on its own servers, which it already trusts. A community app runs on a stranger's laptop, where the owner is root and could read any key in the container.

3 · Control plane

Control plane

Relevant source files

The control plane is one Go process (internal/cp) with one PostgreSQL database. It does two kinds of work: it answers HTTP requests from people, agents and apps, and it runs a set of background loops that keep the world moving even when nobody asks.

The problem

Decisions about many machines must be consistent: two schedulers must never put the same copy in two places.

The idea

One process, one database, and every decision made inside a Postgres transaction.

What it costs

It scales as far as one process and one database go, and its in-memory wake-ups pin it to one copy.

Inside the process

flowchart TB
  subgraph http["HTTP surfaces: Handler and webRoutes"]
    AG["/agent/v1/<br>enroll, sync, tunnel, VPS copies"]
    API["/api/v1/<br>dashboard, CLI, desktop"]
    V1["/v1/<br>admin-token CLI"]
    REG["/v2/<br>registry gate"]
    EG["/egress/<br>API key proxy"]
    DL["/download/<br>agent and CLI files"]
  end
  subgraph loops["Background loops"]
    L1["assignLoop: 5 s and on every change<br>placement, builds, VPS moves"]
    L2["livenessLoop: 2 s<br>offline, lost, fast down"]
    L3["meterLoop: 1 min<br>billing"]
    L4["cronLoop: 5 s<br>scheduled job runs"]
    L5["copiesLoop: 30 s<br>VPS checkpoints, clean-up"]
    L6["domainLoop: 1 min<br>custom domain checks"]
  end
  DB[("PostgreSQL")]
  http --> DB
  loops --> DB
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class DB hl
Requests and loops never talk to each other directly. They meet in the database, and the scheduler is woken through a channel (kickAssigner).

Every loop uses one helper, s.loop(every, wake, once). It runs a round on a timer or when woken, and on shutdown it lets the current round finish and starts no other, so a round of scheduling or billing is never cut halfway.

Sources:

The HTTP surfaces

PrefixWho calls itHow it authenticatesMain handlers
/agent/v1/AgentsMachine token (Bearer); only its SHA-256 is storedhandleEnroll, handleSync, handleAgentTunnel, VPS copy and log endpoints
/api/v1/Dashboard, ccloud, desktop appSession cookie or session tokenDeployments, capacity, organizations, jobs, VPSs, domains, credits
/v1/The fyp admin CLIAdmin or member tokenKeys, machines, workloads, routes
/v2/Machines and buildersHTTP Basic: machine token or a build's one-time tokenhandleRegistry
/egress/Community appsSigned stand-in keyhandleEgress

Sources:

Starting and stopping

New opens the database (running the idempotent schema in store.go) and reads the configuration from flags and environment. Run starts the loops and the listeners. On SIGTERM the server stops taking connections, answers every held agent sync at once (so agents reach the next control plane quickly), lets loops finish their round, and gives requests in flight up to -drain (10 s) to finish. A second signal stops it at once.

Sources:

Found in review, 20 Nov 2026

After the control plane itself was down for a while, liveness counts that downtime against every machine. See Liveness and failover.

3.1 · Control plane

Data model

Relevant source files

The whole schema lives in one string in store.go. It runs on every start: CREATE TABLE IF NOT EXISTS and ALTER TABLE … ADD COLUMN IF NOT EXISTS, so an old database upgrades itself in place. Times are milliseconds since 1970 in BIGINT columns.

The problem

The system must remember who owns what, what runs where, and what was billed, and survive restarts at any moment.

The idea

A small relational model: people, machines, workloads and their copies, with leases joining copies to money.

What it costs

No migration tool: the schema only grows. Renaming or dropping a column needs care by hand.

Apps, machines and money

erDiagram
  ORGS ||--o{ USERS : "has"
  USERS ||--o{ CAPACITY : "promises"
  CAPACITY |o--o| MACHINES : "fulfilled by"
  USERS ||--o{ MACHINES : "provides"
  USERS ||--o{ WORKLOADS : "deploys"
  WORKLOADS ||--o{ INSTANCES : "runs as copies"
  MACHINES ||--o{ INSTANCES : "hosts"
  INSTANCES ||--o{ LEASES : "billed through"
  LEASES ||--o{ CREDIT_LEDGER : "two rows a minute"
  WORKLOADS ||--o{ SECRETS : "holds"
  WORKLOADS ||--o{ BUILDS : "built by"
  WORKLOADS ||--o{ DOMAINS : "served at"
  WORKLOADS ||--o{ FAILOVERS : "recorded in"
  JOBS ||--o{ WORKLOADS : "each run is one"
  MACHINES {
    text id PK
    text owner_id
    text org_id
    text status
    bigint version
    bigint last_seen
  }
  WORKLOADS {
    text id PK
    text name
    text mode
    text digest
    text org_id
    text state
  }
  INSTANCES {
    text id PK
    text workload_id FK
    text machine_id FK
    text role
    int attempt
    text state
  }
  LEASES {
    text id PK
    text instance_id FK
    int units
    text role
    bigint billed_until
  }
The core tables. org_id on users, machines, workloads and leases is what separates the two clouds.

Sources:

What each table is for

TableOne row isNotes
orgs, users, sessionsAn organization, a person, a signed-in sessionPasswords are PBKDF2-SHA256 with a salt; sessions store only a token hash
capacity, keysA promise to share a machine, and the join keys for itA key has uses left and an expiry
machinesA joined computerIts token hash, offer, live host figures, status, version, time in the pool
workloadsAn app or a job rundigest pins the build; rollout_* columns track a redeploy
instancesOne copy on one machineActive states are assigned, starting, running; the last log line time prevents duplicates
leases, credit_ledgerA copy's use of a machine; a credit movementbilled_until makes billing safe to repeat
failoversOne app leaving a lost copyFrom, to, how, when it was down, detected and serving again
secretsOne sealed secret of an appIts host for egress, and a use counter
buildsOne source buildIts status, digest and the hash of its one-time registry token
jobsA job definition with an optional scheduleEach run is a workloads row with job_id
domains, certsA custom-domain claim; a cached certificateA partial unique index allows one verified owner per host
vps and vps_*A VPS and its events, packs, photos, log batches, leftoversSee VPS
ssh_keys, github_connectionsA person's SSH public keys; their GitHub tokens, sealed

VPS tables

erDiagram
  USERS ||--o{ VPS : "owns"
  MACHINES ||--o{ VPS : "keeps"
  VPS ||--o{ VPS_EVENTS : "history"
  VPS ||--o{ VPS_PHOTOS : "copies, as block maps"
  VPS ||--o{ VPS_PACKS : "sealed changed blocks"
  VPS ||--o{ VPS_LOG : "write log batches"
  VPS ||--o{ VPS_LEFTOVERS : "old disks on old machines"
  USERS ||--o{ SSH_KEYS : "log in with"
  VPS {
    text id PK
    text machine_id
    bigint generation
    bigint written_gen
    bigint copy_seq
    bigint log_seq
  }
The S3 store holds only sealed bytes. Which bytes belong where (vps_photos.manifest), the chain of copies and every VPS's key live in these tables.

Sources:

A thin layer over database/sql

db.go wraps the Postgres driver so the code can write ? placeholders, which it rewrites to $1, $2… (rebind). isUnique recognises a unique violation, which several places use as a lock: for example, two accounts racing to verify the same custom domain.

Sources:

Check yourself

Why is a lease its own table instead of a column on the copy?

A copy can change role (a standby promoted to primary) and the price changes with it. Billing follows the role: the standby lease ends and a primary lease starts. Separate rows also make "what did this provider earn" a simple sum.

3.2 · Control plane

Machines and capacity

Relevant source files

Before a machine can run anything, its owner promises how much to share, the machine joins with a key, and from then on it reports what is really free. This page follows a machine from promise to retirement.

The problem

A machine must prove it was invited, and must never be given more work than it can hold, even when its owner starts using it.

The idea

A promise with a one-time key, an offer capped at the hardware, and a live free figure that leaves a reserve for the owner.

What it costs

Server keys in organizations are kept in plain text so their owner can see them again; replacing one is a click.

From promise to machine

%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
  actor O as Owner
  participant D as Dashboard or desktop app
  participant CP as Control plane
  participant A as Agent
  O->>D: share 2 cores and 2 GB
  D->>CP: POST /api/v1/capacity (a promise)
  CP-->>D: join key ek_ (1 use, 1 hour)
  O->>A: ccloud join, or the desktop app
  A->>CP: POST /agent/v1/enroll with the key, offer and hardware
  CP->>CP: cap the offer at the hardware, create the machine
  CP-->>A: machine_id and token
  A->>A: save identity.json (0600)
  A->>CP: first sync, the machine goes online
The server keeps only the SHA-256 of the machine token. In an organization, a member's reusable server key replaces the one-time key.

Sources:

How much a machine may offer

An offer is capped twice. At join, capOffer limits it to the hardware: CPU less a tenth, memory less the larger of 512 MB and a tenth. While running, the agent's host reporter measures what is actually free and keeps a reserve for the owner (512 MB or 10% of memory). The scheduler uses the smaller of the offer and the live free figure.

MachineHardwareLargest offerOwner reserve
Laptop, 4 cores, 8 GB4000 m, 8192 MB3600 m, 7372 MB819 MB
Server, 8 cores, 16 GB8000 m, 16384 MB7200 m, 14745 MB1638 MB
Small VM, 1 core, 1 GB1000 m, 1024 MB900 m, 512 MB512 MB

Sources:

Units and sizes

People choose sizes; billing counts units. One unit is a quarter of a core and 256 MB. Small, medium and large apps take 1, 2 and 4 units. A custom size is billed by whichever resource needs more units. A machine's capacity in units is the smaller of its CPU and memory counts, so a machine with lots of CPU and little memory is honest about what it can hold.

Sources:

The life of a promise

stateDiagram-v2
  [*] --> promised: owner promises capacity
  promised --> provisioned: a machine joins with the key
  provisioned --> usable: first sync, healthy, not paused
  usable --> used: a copy runs on it
  used --> usable: the last copy leaves
  usable --> provisioned: paused or runtime unhealthy
  used --> offline: silent, left or removed
  usable --> offline: silent, left or removed
  offline --> usable: syncs again
What the dashboard shows for each promise. used means credits are being earned.

Sources:

Every sync is a check-in

recordCheckIn stores the machine's latest host figures, its public IP, any runtime error and the time, and tracks how long it has been online (online_since, pool_ms). Those two numbers give the machine's reliability: the share of its life it has spent online. The scheduler prefers machines with a high one.

Sources:

Leaving

Joining again with a new key for the same promise retires the old machine. A machine that sends leaving (pause, shutdown) goes offline at once and its apps move at once. An admin can remove a machine; its token stops working on its next check.

Sources:

Check yourself

A provider offers 4 GB, but the laptop has only 1.5 GB free right now. What does the scheduler see?

The smaller figure. The agent reports free memory beyond the reserve, and the scheduler uses the smaller of that and the offer. So the laptop gets no new work that needs more than what is really free.

3.3 · Control plane

Scheduler and placement

Relevant source files

The scheduler answers one question many times a second: for this copy of this app, which machine? It runs on every change and every 5 seconds, places one copy per database transaction, and wakes the chosen machine.

The problem

Pick a machine that is likely to stay online, has room, is allowed to run the app, and does not put two copies in the same place.

The idea

Filter the machines with hard rules, then rank the rest with a simple score you can compute by hand.

What it costs

It is greedy: one copy at a time, best machine now. It never moves running copies to make a better fit.

One round

flowchart TB
  T["assignOnce<br>every change, every 5 s"] --> B["assignBuilds and placeVPS"]
  B --> W["for each active or queued workload"]
  W --> R{"rolloutStep<br>a redeploy step?"}
  R -- "yes" --> W
  R -- "no" --> P["placeOne"]
  P --> N{"what is missing?"}
  N -- "a primary" --> F["record a failover,<br>promote the standby if any"]
  N -- "a standby" --> SB["standbyStrategy"]
  N -- "nothing" --> W
  F --> PK["pick: filter, then score"]
  SB --> PK
  PK --> C["addCopy, bumpVersion, commit"]
  C --> WK["wake the machine"]
  WK --> P
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class PK hl
placeOne repeats until a workload has every copy its mode wants, or no machine has room.

Sources:

How many copies

ModePrimariesStandbysWhen a machine is lost
spot10A new copy starts elsewhere: pull and start
standby11The standby is promoted; measured about 17 s from an abrupt drop to serving
active2 to 50Piko spreads requests over the others; a replacement starts

Sources:

Hard rules: which machines may take the copy

pick starts from online machines that run apps (builders run none) and keeps a machine only if all of these hold:

  1. The strategy allows it: communityStrategy takes machines with no organization; orgStrategy takes only the workload's own organization.
  2. It runs no other active copy of this workload. Copies of one app never share a machine.
  3. No copy of this same build failed there in the last 10 minutes (failedRecently). A broken machine does not take the same app again and again.
  4. It has enough free CPU and memory for the copy.

Soft rules: the score

The machines that pass are sorted first by how many copies of this app were lost or evicted there before (fewer is better), then by the strategy's score:

TermRangeMeaning
100 × reliability0 to 100Share of its life the machine has been online. The strongest term by far
packing0 to 10How full it already is: fill busy machines before waking idle ones
apart−30, −20 eachFor every existing copy on the same provider, and on the same public IP

A standby uses a different score: the same reliability term, plus free units (prefer the emptiest machine), and twice the spreading penalty (−60 and −40), because a standby on the same Wi-Fi as its primary dies with it.

Sources:

Worked example

An app in the community cloud already has one copy on Ali's laptop at home. Where does the second copy of an active app go?

MachineReliabilityPackingApartScore
Sara's desktop, 4 of 8 units used9550100
Ali's second laptop, same home, 2 of 8 used902.5−5042.5
Bilal's laptop, idle600060

Sara's desktop wins. Ali's second laptop is almost as reliable, but it shares a provider and a network with the first copy, so one power cut would take both.

After the choice

In the same transaction, addCopy inserts the copy with the next attempt number, a pending failover record gets its destination, and bumpVersion raises the machine's version. After the commit, waits.wake ends that machine's held sync, so the agent hears within milliseconds.

Sources:

When an app itself is broken

If a workload's copies fail on 2 different machines, or 3 times in all, the problem is the app, not the machines. The workload is marked failed and the scheduler stops trying until the owner redeploys.

Sources:

Check yourself

Why does reliability weigh ten times more than packing?

On volunteer machines the main risk is the machine going away. A copy on a machine that is usually there saves a failover later; packing only saves some idle machines from waking up.

Why is the spreading rule soft and not a hard filter?

A demo or a small organization may have all its machines in one building. A hard rule would leave the app with nowhere to run; a penalty still prefers spreading when it can.

3.4 · Control plane

Liveness and failover

Relevant source files

A machine that vanishes sends no message. The control plane notices only silence, and silence has to be read carefully: a slow sync looks the same as a dead laptop for a while. Liveness turns silence into decisions with a few timers.

The problem

Decide when a silent machine is gone, fast enough that visitors barely notice, and never so fast that a slow machine loses its apps for nothing.

The idea

Escalate with time: no new work at 30 s, apps moved at 2 minutes, sooner if Piko also lost every tunnel from it.

What it costs

A spot app can be down for up to two minutes plus a restart. Standby mode exists to cut that.

What happens when a laptop vanishes

flowchart LR
  T0["0 s<br>lid closes"] --> T15["15 s<br>no sync and its Piko<br>tunnels gone:<br>down now"]
  T15 --> T17["about 17 s<br>standby promoted,<br>serving again"]
  T0 --> T30["30 s<br>no sync: offline,<br>no new work"]
  T30 --> T120["2 min<br>copies marked lost,<br>placed elsewhere"]
  T0 --> F60["60 s on the laptop<br>if it is still on:<br>it closes its tunnels"]
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class T15 hl
  class F60 warn
Two clocks run at once: the control plane's, on top, and the agent's own fence, below.

The three timers

RuleDefaultEffectCode
No sync for OfflineAfter30 sStatus offline; the scheduler gives it no new worklivenessOnce
No sync for FastDownAfter and all its Piko tunnels gone15 smachineDown now: Piko's view confirms the machine is really gonepikoHealth
Offline for LostAfter2 minmachineDown: active copies become lost, the scheduler replaces themlivenessOnce
Agent: no successful sync for 60 s60 sThe agent closes its tunnels until the control plane answersreconcileOnce

Sources:

Failover, recorded

When a workload is short of a primary, placeOne first opens a failovers row (from which copy and machine, why, when it went down and when that was noticed). If the app has a standby, promoteStandby turns it into the primary and its agent opens the tunnel as soon as it hears. Otherwise a new copy is placed. When the new primary serves, the row gets its serving_at time, and the dashboard shows how many seconds the app was away.

%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
  participant L as livenessLoop
  participant S as Scheduler
  participant DB as PostgreSQL
  participant B as Standby's agent
  participant P as Piko
  L->>DB: machineDown: primary copy lost, version bumped
  S->>DB: openFailover (from, reason, down_at)
  S->>DB: promoteStandby: role primary, leases switched
  S-->>B: wake the held sync
  B->>B: primary and healthy, so open the tunnel
  B->>P: tunnel for the app
  B->>S: next sync: tunnel up
  S->>DB: failover serving_at
With a warm standby, recovery is one role change and one tunnel. No image pull, no container start.

Sources:

Fencing on the machine's side

The agent cannot know whether it was declared lost, so it assumes the worst. With no successful sync for 60 seconds, it closes every tunnel and serves nothing until the control plane confirms its work again. A laptop that slept through a failover therefore never serves an old copy next to the new one.

Sources:

Found in review, 20 Nov 2026

No grace period after a control plane restart. Liveness compares each machine's last_seen with now. If the control plane itself was down for more than 2 minutes, every machine looks silent on its first round, before agents have reconnected. Agents back off up to 60 seconds plus jitter, so many reconnect late. Their copies are marked lost and placed on whichever machines reconnected first; VPSs move too once past move_after_min. A fix is to measure silence from the later of last_seen and the control plane's start.

The team's own open incident is related: an agent restart is still counted as leaving the pool (Known limits).

Check yourself

Why look at Piko's tunnels before declaring a machine down at 15 s?

A late sync alone is ambiguous: the control plane might be slow, or the network briefly bad. If Piko has also lost every tunnel from that machine, two independent views agree it is gone, so moving its apps early is safe.

3.5 · Control plane

Redeploys and config changes

Relevant source files

A running app changes in two ways: a new build, or new variables. Both are handled the same way: start new copies, stop old ones only once the new ones serve, and go back if anything fails.

The problem

Replace every copy of a live app without a gap for visitors, and without ending up half old and half broken.

The idea

Pin the exact build (a digest), replace copies one at a time, make before break, and keep what to go back to.

What it costs

A single spot copy with no room elsewhere restarts in place: a short gap. A request can fail at the moment an old tunnel closes.

Pinning the build

A tag like alice/api:latest can point to a different build tomorrow. So at deploy time the control plane asks the image's registry what the tag points to now (imageDigest: a HEAD request, an anonymous token if needed, public addresses only) and stores the digest. Machines run image@sha256:…, so every copy runs the same build and a stale image on disk is never reused. Redeploy asks the registry again.

Sources:

Replacing copies one at a time

%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
  participant S as rolloutStep
  participant N as New copy's agent
  participant O as Old copy's agent
  participant P as Piko
  S->>N: add a copy at the new digest
  N->>N: pull, start, health check passes
  N->>P: open the tunnel (both copies serve)
  N->>S: sync: running, tunnel up
  S->>O: remove the old copy
  O->>P: drain, then shut the tunnel
  O->>O: remove the container
  Note over S: repeat until no copy is old, then commit the redeploy
For a moment old and new serve the same address side by side, so visitors never meet a gap.
ModeOrder of replacement
spotA new copy on another machine, then the old one stops. With no room elsewhere, it restarts in place
standbyThe standby is replaced first, promoted beside the old primary, then the old primary stops and a fresh standby follows
activeOne copy at a time; the others keep serving

If a new copy fails, or the rollout makes no progress for 15 minutes (rolloutStuck), rollBack returns the app to the build it ran and the page says why.

Sources:

Changing variables

A container keeps the values it started with, so changing a variable is also a rollout. Every change gets a number (config_rev) and every copy records the number it started with. A copy is old if its build or its number differs. The values to go back to are kept until the change is done.

Some values are built into the app, such as NEXT_PUBLIC_… or VITE_…. Changing one needs a new build of the commit the app already runs. Until that build succeeds, nothing about the app changes.

Sources:

Check yourself

Why not just restart the copies with the new image?

A restart has a gap, and if the new build is broken the app is down. Starting the new copy first and stopping the old one only once the new one serves keeps the app up, and a failure leaves the old copies untouched.

3.6 · Control plane

Accounts and organizations

Relevant source files

People sign up into one of two worlds. In the community cloud they provide or consume capacity. In an organization they have a role that decides what they see and whether they bring servers.

The problem

The same API serves strangers in a public pool and colleagues in a private one, and neither may see the other's data.

The idea

Every account has at most one organization and a role in it; every request is pinned to one cloud by scopeOf.

What it costs

One organization per person, for now.

How an organization works

flowchart TB
  SU["Sign up, then New organization"] --> AD["Admin"]
  AD -- "Members page invite link" --> MB["Members<br>org_role maintainer<br>bring servers"]
  AD -- "Consumers page invite link" --> CO["Consumers<br>org_role member<br>deploy"]
  AD --> K1["Admin's server key"]
  MB --> K2["Each member's own server key"]
  K1 --> J["sudo ccloud join --key sk_..."]
  K2 --> J
  J --> SV["Servers in the organization's pool"]
  CO --> DP["Apps on the organization's servers"]
  SV --> DP
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class AD hl
The dashboard says members and consumers; the API and database say maintainer and member.
  • Any number of servers can join with one server key, until its owner replaces it. A server belongs to the person whose key it joined with.
  • Members see their own servers. The admin sees every server, person, app and everyone's usage, changes roles, and can remove a person: their apps stop and their servers leave the pool.
  • Nobody outside sees the organization. Only the platform admin can look in, with ?org=.

Sources:

Sessions and sign-in

Passwords are stored as PBKDF2-SHA256 with a random salt. A session is a random token; the database keeps only its hash. Over HTTPS the browser cookie is named __Host-meshly_session, which an app on a sibling name cannot overwrite. Further browser protections (security headers, a cross-origin check, apps on a separate domain) are listed as Phase 0 work in the roadmap.

Sources:

Check yourself

Why does each member get their own server key, rather than one key for the organization?

So a server always belongs to a person, the admin can see whose it is, and one person's key can be replaced without touching anyone else's servers.

3.7 · Control plane

Leases and credits

Relevant source files

In the community cloud, people who run apps pay credits and people who share machines earn them. The unit of record is a lease: one copy holding some units of one machine, from start to stop.

The problem

Bill fairly by the minute, survive crashes mid-billing, and never charge a minute twice.

The idea

Each lease remembers how far it was billed. The meter bills only the time since, inside the same transaction as the ledger rows.

What it costs

A machine could claim a copy runs when it does not; crediting only copies whose tunnel Piko confirms is on the roadmap.

The rates

WhatPrice
One unit, primary, per minute1 credit (1000 millicredits)
One unit, warm standby, per minute0.25 credit
Welcome balance for a new account500 credits
Inside an organizationfree, usage still recorded

Example: a medium app (2 units) in standby mode for one hour costs 2 × 60 = 120 credits for the primary and 2 × 60 × 0.25 = 30 for the standby. The providers of the two machines earn the same amounts. A job's run is rounded up to whole minutes when it ends: a 61 second run pays for 2 minutes.

Sources:

The meter

%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
  participant M as meterLoop (every minute)
  participant DB as PostgreSQL
  M->>DB: begin, open leases with their billed_until
  loop each lease
    M->>M: owed = units x rate x time since start
    M->>DB: consumer row: minus (owed now - owed before)
    M->>DB: provider row: plus the same
    M->>DB: billed_until = now
  end
  M->>DB: commit
Bills are computed from the lease's start each time, so the sum of all bills always equals the whole time, with no rounding drift.

A lease opens when a copy starts running (openLease, which does nothing if one is already open) and closes when it stops or changes role. A balance is simply the sum of a person's ledger rows.

Sources:

Check yourself

The control plane crashes halfway through a billing round. Can a minute be billed twice?

No. The ledger rows and the new billed_until are written in one transaction. A crash rolls back the whole round, and the next round bills from the old billed_until.

4 · Agent

Agent

Relevant source files

The agent is one Go program on every shared machine. It joins once, then spends its life in one loop: ask the control plane what to run, make the machine match, report back. Around that loop run a few helpers, each a goroutine with one job.

The problem

A program on a stranger's laptop must keep many things in step (containers, tunnels, health, metrics, disks) while its network comes and goes.

The idea

One sync loop owns the conversation with the control plane. Every other job runs in its own goroutine and is nudged through small signal channels.

What it costs

Several loops share state under one mutex, and their timers interact. Incidents in docs/incidents show where.

What runs inside an agent

flowchart LR
  RUN["Run<br>enroll or load identity,<br>read the host"]
  RUN --> SYNC["sync loop<br>syncOnce, apply, backoff"]
  RUN --> WATCH["watchLoop<br>ends a held sync early"]
  RUN --> MET["metricsLoop<br>samples every 5 s"]
  RUN --> HEA["healthLoop<br>checks each service copy"]
  RUN --> COP["copyLoop and logLoop<br>VPS copies and write log"]
  RUN --> TUN["tunnelLoop<br>own WebSocket for VPS sessions"]
  RUN --> LOC["serveLocal<br>127.0.0.1:7070"]
  SYNC --> REC["reconcileLoop<br>containers and Piko tunnels"]
  SYNC --> VPS["vpsLoop<br>VPS containers and disks"]
  WATCH --> SYNC
  HEA --> REC
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class SYNC hl
The goroutines Run starts. Arrows between loops are small signal channels: the sync loop wakes the reconcilers, the watch loop and health checks wake the others. Only the sync loop talks to the control plane's sync endpoint.

Sources:

The sync loop

  1. Ask. syncOnce sends the report (every copy's state, host figures, samples, log lines, VPS reports) and waits up to 35 seconds for an answer.
  2. Apply. apply stores the desired state, saves it to state.json if the version changed, records the time of this successful sync, and signals the reconcilers.
  3. Repeat at once. The next request goes immediately; the control plane holds it until something changes or 10 seconds pass.
  4. On failure, back off. Wait 1 second, doubling up to 60 seconds, plus random jitter of up to the same again, and keep running current work meanwhile.
  5. On stop, say goodbye. A clean stop sends leaving, so the control plane moves this machine's apps at once instead of waiting for silence.

Sources:

Reporting early

While a sync is held, things change on the machine: a container exits, the runtime breaks, the owner starts a big program. watchLoop compares a key of the report with the last one sent and, when it differs, ends the held request so the control plane hears now. Free capacity counts at most every 10 seconds, and figures that move on their own (a VPS's lag and upload rate) are left out of the key: an earlier version included them, cut every held sync short, and the agent fenced itself after 60 seconds.

Sources:

Files the agent keeps

FileWhat
identity.json (0600)Machine id and token, written once at enroll
state.jsonThe last desired state, so a restart knows what it was running
env/Each container's variables file, deleted once the container is created
vps-store/ or vps-store.xfsVPS disk images, in a filesystem that can clone

Sources:

The local API

The agent listens on 127.0.0.1:7070 for programs on the same machine. The desktop app and ccloud status, limits, pause and resume use it. The agent enforces every limit itself, so closing the desktop app changes nothing about what the laptop shares.

Sources:

Check yourself

Why the random jitter in the backoff?

If the control plane restarts, hundreds of agents fail at the same moment. Without jitter they would all retry at the same moments too, in waves. Jitter spreads them out.

4.1 · Agent

Running containers

Relevant source files

Every app copy is a Podman container started with hard limits. The reconciler decides what to start and stop; a small driver turns those decisions into Podman (or Docker) commands.

The problem

Run strangers' code on someone's laptop with firm limits, without a heavy daemon eating the owner's memory.

The idea

Podman, which needs no daemon, driven through its CLI, with every limit on the command line and nothing exposed beyond 127.0.0.1.

What it costs

Containers share the host's kernel. In practice Podman runs as root. There is no disk limit or per-customer network for apps yet.

How a copy is started

Podman.Start builds one podman run command. Each flag answers a concrete risk:

FlagWhy
--cpus, --memory, --memory-swap equal to memoryThe copy gets exactly its size and cannot spill into swap
--pids-limit 256A fork bomb stays inside its box
--security-opt no-new-privilegesSetuid programs inside cannot gain privileges
--cpu-shares 2, --oom-score-adj 1000 (community machines)The owner's programs win the CPU, and the kernel kills our container first
-p 127.0.0.1:<host port>:<app port>The app is reachable only from the machine itself, where the tunnel picks it up
--restart on-failure:3A crash is retried locally before anyone is told
--env-file (0600, deleted after create)Values never appear in the process list, where any program could read them
labels fyp.machine, fyp.instanceThe agent recognises its own containers and never touches others

Sources:

The reconcile round

flowchart TB
  A["desired copies<br>from the last answer"] --> D{"compare"}
  B["containers<br>from podman ps, by label"] --> D
  D -- "missing" --> S["start it<br>unless starting, failed or evicted"]
  D -- "not wanted, or paused" --> G["close its tunnel with a short drain,<br>then remove it"]
  D -- "running primary" --> H{"healthy and<br>not fenced?"}
  H -- "yes" --> T["open or keep its tunnel"]
  H -- "no" --> C["no tunnel"]
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class D hl
Running the round twice changes nothing the second time. That is what makes it safe to run on every signal and every few seconds.

Sources:

Logs

For each copy, the agent reads podman logs --timestamps from where its cursor is and sends the new lines with the next sync: at most 1000 lines and 128 KB per copy per sync, newest kept. The cursor moves only when an answer comes back, and the control plane drops any line at or before the newest it already has, so each line counts once even across restarts.

Sources:

Docker for builders

The driver speaks to any Docker-compatible CLI. Builder machines use Docker, because Amazon Linux 2023 has no Podman package and Railpack needs BuildKit anyway. The few differences (for example --config versus --authfile for registry logins) are handled in place.

Sources:

Check yourself

Why publish the app's port on 127.0.0.1 instead of all addresses?

Visitors arrive through the agent's tunnel, which connects locally. Publishing on all addresses would let anyone on the owner's network reach the app directly, bypassing the gateway and its health gating.

4.2 · Agent

Protecting the owner

Relevant source files

The person who shares a laptop must never regret it. If they open a video editor while an app runs, the editor must win at once. Five layers make sure of that, from gentle to brutal.

The problem

The owner can start anything at any moment, and nobody will wait for a scheduler to notice.

The idea

Let the kernel favour the owner for CPU, keep a memory reserve, and evict our newest copy the moment that reserve is touched.

What it costs

An evicted copy restarts elsewhere: a short outage for a spot app. Standby mode hides it.

Five layers

flowchart TB
  O["The owner starts a heavy program"] --> L1["1. CPU: our containers have weight 2,<br>the kernel gives the owner the CPU"]
  O --> L2["2. Memory, slow: free memory is reported<br>beyond a reserve, so no new work comes"]
  O --> L3["3. Memory, fast: below the reserve,<br>the newest shared copy is evicted now"]
  L3 --> L4["4. Last resort: oom-score-adj 1000,<br>the kernel kills our container first"]
  L5["5. Hard caps on every container:<br>CPU, memory, processes"]
  L3 --> M["The control plane places the copy elsewhere"]
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class L3 hl
Layers 1 and 5 hold all the time; 2 to 4 react to the owner's load.

The reserve

The reserve is the larger of 512 MB and a tenth of the machine's memory. The host reporter reads /proc and reports free CPU and memory after the reserve, so the control plane stops sending new work as soon as the owner's usage grows. On Windows the agent runs inside WSL and sees only that virtual machine, so the desktop app also hands in the whole computer's numbers, and free capacity respects both.

Sources:

Eviction

guardOwner runs on each host reading. If available memory is below the reserve and the last eviction was more than 10 seconds ago, it picks the newest running shared copy, removes its container, marks it evicted with the reason, and kicks a sync so the control plane hears now. The scheduler then treats the copy like any lost one and places it on another machine.

Sources:

Check yourself

Why evict the newest copy rather than the biggest one?

The newest copy has served the least, so moving it loses the least warm state, and it is the one the system most recently judged "fits here". It is also a simple, predictable rule.

4.3 · Agent

Health checks

Relevant source files

A container that runs is not always an app that works. It can hang, or answer every request with an error, while the machine looks fine. So the agent checks each service copy itself, and only a healthy primary gets a tunnel.

The problem

Send visitors only to copies that really answer, and replace copies that stop answering.

The idea

Check from the machine, where the copy is, on a schedule the app's owner can tune; restart a few times, then give up and let the control plane move it.

What it costs

A check adds a little load to every app, and an app that is slow to load needs a longer grace time.

States

stateDiagram-v2
  [*] --> starting: container started or restarted
  starting --> healthy: a check passes
  healthy --> unhealthy: failures in a row
  starting --> unhealthy: failures after the grace time
  unhealthy --> starting: restart (at most 3)
  unhealthy --> failed: still failing after the 3rd restart
  failed --> [*]: the control plane moves the copy
Only healthy opens a primary's tunnel. Standbys are checked too, so a promotion never lands on a broken copy.

Sources:

The two kinds of check

TypePasses whenDetail
httpGET path answers 2xx or 3xx within the timeoutNo redirects followed, no keep-alive
tcpThe port accepts a connection and keeps it open for 300 ms, or sends somethingConnecting alone is not enough: Docker's port proxy accepts every connection and closes it at once when nothing listens inside

Sources:

Defaults

SettingDefaultMeaning
interval_s10 sTime between checks (1 to 300)
timeout_s2 sA check slower than this fails
failures3Failures in a row before unhealthy
grace_s60 sBefore the first pass, failures in this window do not count; the app is checked every second meanwhile
Restarts before failed3Forgotten once the copy has been healthy for 10 minutes

A copy that fails is treated like any failed copy: another machine (not this one for 10 minutes), a standby promoted first, and the app given up after failing on 2 machines or 3 times. During a redeploy, a new copy that never passes keeps its tunnel closed, so the redeploy rolls back.

Sources:

Check yourself

Why are standbys health-checked even though they serve nobody?

A standby exists to take over. If it had quietly broken, the failover would promote a broken copy exactly when the app most needs one that works.

5 · Networking

Networking

Relevant source files

Four kinds of traffic cross this system: agents asking for work, visitors reaching apps, people opening terminals into VPSs, and apps calling outside APIs. All of them enter through the same front door, and none of them ever needs a port open on a shared machine.

The problem

Visitors must reach a container on a laptop behind a home router, whose address must stay private and which may change several times a day.

The idea

Reverse every connection: machines dial out and keep tunnels open; the gateway sends requests down them by app name.

What it costs

Every byte passes through one server: an extra hop, a bandwidth bill, and a single point of failure until Piko runs as a cluster.

All the paths

flowchart TB
  VIS(["Visitor"]) --> CF["Cloudflare, then nginx :443"]
  AG["Agent on a shared machine"] -- "sync and agent tunnel" --> CF
  AG -- "app tunnels" --> CF
  APP["App container<br>127.0.0.1 only"] -. "egress calls" .-> CF
  AG --- APP
  CF -- "dashboard, API,<br>sync, agent tunnel" --> CP["Control plane :8080"]
  CF -- "*.apps names" --> PX["Piko proxy :8100"]
  CF -- "tunnel upstream" --> PU["Piko upstream :8001"]
  PX --> PU
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class AG hl
Every arrow from the machine points outwards. Piko's upstream path is where tunnels arrive; its proxy path is where visitors' requests leave.

Sources:

Timeouts along the way

Each hop has its own idea of how long a request may take. Most surprises in this area came from one of these disagreeing with another.

WhereLimitWhy it matters
Cloudflare Free, proxied request100 s, 100 MB bodyLarge uploads (image layers) cannot go through it
nginx, agent sync90 sLonger than the 35 s the agent waits and the 10 s hold
nginx, Piko tunnel WebSocket1 hTunnels are renewed every 6 minutes anyway
Piko, reading an app request10 minPiko's default of 10 s cut uploads
Piko, answering an app request1 hLong requests and streams

Sources:

The pages in this section

5.1 · Networking

Piko tunnels

Relevant source files

Piko is an open-source reverse proxy built for upstreams that dial out. The agent embeds Piko's Go client: for every primary copy it serves, it keeps one outgoing WebSocket to Piko, and Piko sends visitors' requests down it as streams.

The problem

A machine that left the pool, or ignores the control plane, must not keep serving an app. But visitors must never see a gap while tunnels change.

The idea

Short tokens (10 minutes) that list the apps a machine may serve, and a make-before-break renewal well before each token expires.

What it costs

A renewal every 6 minutes per app, and a request still running when a drain ends is cut.

A visitor's request

%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
  participant V as Visitor
  participant N as nginx
  participant P as Piko
  participant A as Agent
  participant C as Container
  V->>N: GET https://shop.apps-domain/cart
  N->>P: proxy port, endpoint shop
  P->>A: new stream on the tunnel for shop
  A->>C: forward to 127.0.0.1 host port
  C-->>A: response
  A-->>P: response on the stream
  P-->>V: response
The machine's address never appears. If several machines serve shop (active mode, or a rollout), Piko spreads requests over their tunnels.

Sources:

Who may serve an app

Every sync answer carries a fresh Piko token: a JWT signed with the secret Piko shares with the control plane, listing the endpoints this machine may serve, valid for 10 minutes. Piko checks it when a tunnel connects. A machine whose copies moved away simply stops receiving tokens that name those apps.

Sources:

Renewal, make before break

Piko checks a token only when a tunnel connects, and closes the connection itself when that token expires. So a tunnel cannot be refreshed in place; it must be replaced. With less than 4 minutes left on its token, the agent replaces it:

%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
  participant A as Agent
  participant P as Piko
  Note over A,P: old tunnel serving, token has 4 minutes left
  A->>P: connect a new tunnel with the fresh token
  Note over P: requests now alternate between both tunnels
  A->>A: old tunnel: wait until no request is in flight (30 s at most)
  A->>P: Shutdown the old connection
  Note over P: old connection leaves the list about 0.25 s later
No gap and no failed request: 196 requests across renewals in production all succeeded.

Why Shutdown, not Close

The first version used Piko's Close. It turned out to be only a yamux GoAway: the connection stayed on Piko's list, and the next request sent to it failed with a 502, once per app per renewal. The fix was to take streams with Upstream.Listen and forward them in the agent's own code, so it can call the listener's Shutdown after a drain. A copy that is no longer wanted gets the same treatment with a shorter drain (1.5 s), shorter than the 2 s the reconciler waits before removing its container.

Sources:

When tunnels open and close

EventTunnel
Primary copy running and its health check passesOpens
Standby copyNone: warm, but invisible to visitors
Copy unhealthy, or no longer wantedCloses after a short drain
No successful sync for 60 sAll close: the machine may have been replaced
Piko loses every tunnel of a silent machine for 15 sThe control plane declares the machine down at once

Sources:

Check yourself

Why not use long-lived Piko tokens and avoid renewals altogether?

The short token is what stops a machine that has left the pool, or that ignores the control plane, from serving an app for long afterwards. Longer tokens trade fewer renewals for a longer window of stale serving.

5.2 · Networking

Custom domains

Relevant source files

An app can be served at its owner's own name, such as shop.example.com. The owner proves they control the name with a DNS record, points it at the platform, and the control plane serves it over HTTPS with its own Let's Encrypt certificate.

The problem

Anyone can type any domain into a form. The platform must serve a name only for the person who controls it, and only one person at a time.

The idea

A claim per name, proven by a TXT record only the name's owner can create, with one verified owner enforced by a unique index.

What it costs

The control plane needs its own HTTPS listener. Off unless DOMAINS_TARGET is set; not switched on in production yet.

Proving a name

%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
  actor O as App owner
  participant CP as Control plane
  participant DNS as Their DNS provider
  O->>CP: POST /api/v1/deployments/shop/domains (shop.example.com)
  CP-->>O: token, and the records to add
  O->>DNS: TXT _meshly-verify.shop.example.com = token
  O->>DNS: CNAME shop.example.com to the platform's target
  loop domainLoop, every minute
    CP->>DNS: look up the TXT record
  end
  CP->>CP: proof found: status verified (unique per host)
A root domain cannot have a CNAME, so the page shows A records with the target's addresses instead.

Sources:

Claim states

stateDiagram-v2
  [*] --> pending: claim added
  pending --> verified: TXT proof found
  pending --> taken: another account verified first
  verified --> pending: 3 definite failures in a row
  taken --> pending: the verified claim was removed
A pending claim never blocks anyone. Only one claim per host can be verified, because of the partial unique index domains_verified_host.
ClaimChecked
pending, younger than an hourevery minute
pending, younger than a dayevery 10 minutes
pending, olderhourly
verified or takenevery 6 hours

Only a definite answer counts against a verified claim (the record is not found, or does not match). Timeouts and temporary DNS errors change nothing.

Sources:

Serving a verified name

flowchart TB
  V(["Visitor: https://shop.example.com"]) --> L["Control plane's own HTTPS listener<br>DOMAINS_HTTPS"]
  L --> C{"certPolicy:<br>verified host of a live service?"}
  C -- "no" --> X["refused: no certificate is ever issued"]
  C -- "yes" --> R["domainRoute: host to app<br>cached 30 s"]
  R --> P["Piko proxy<br>header x-piko-endpoint: shop"]
  P --> T["the app's tunnel, as for any visitor"]
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class C hl
Certificates come from Let's Encrypt through autocert and are cached in the certs table, sealed with SECRETS_KEY. A plain HTTP listener answers HTTP-01 challenges and redirects everything else.

The policy check matters: without it, anyone could point a random name at the platform and make it request certificates until Let's Encrypt's rate limits blocked the real ones.

Sources:

Check yourself

Why a TXT record at a special name, rather than just checking that the CNAME points at us?

Pointing a name at the platform proves nothing about who asked for it: an old CNAME could still point here after a domain changed hands. The TXT token is unique to one claim, so only the person who made that claim, and controls the DNS, can create it.

5.3 · Networking

Secrets and egress

Relevant source files

On a stranger's laptop, the owner is root and can read anything a container holds. So in the community cloud, an app's API keys never leave the control plane. The app gets a stand-in, and the control plane swaps in the real key on the way out.

The problem

An app needs its API key to call an outside service, and the machine it runs on cannot be trusted with that key.

The idea

Give the app a stand-in key that works only through our proxy, only for its one host, only while its copy runs.

What it costs

Only keys sent in an HTTP header or query work this way. Signing keys and database passwords still reach the machine, with a warning.

Where each secret goes

Organization cloudCommunity cloud
Plain variablein the containerin the container
API key (a secret with a host)in the containerstays in the control plane; the container gets ccs_…
Secret the app uses itself (signing key, database password)in the containerin the container, and the dashboard warns that the machine's owner can read it

Every secret is sealed at rest with AES-256-GCM under SECRETS_KEY and is write-only: the dashboard shows its name and where it goes, never its value. Secrets never go to a build, because an image keeps whatever it was built with.

Sources:

A call through egress

%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
  participant App as Community app
  participant E as handleEgress
  participant DB as PostgreSQL
  participant API as api.openai.com
  App->>E: POST /egress/v1/chat, Bearer ccs_ stand-in
  E->>E: check the signature and the rate (10 calls a second per copy)
  E->>DB: is this copy still running? which secret and host?
  E->>E: swap the stand-in for the real key
  E->>API: POST https://api.openai.com/v1/chat, Bearer sk-
  API-->>E: answer, streamed
  E-->>App: answer, streamed
The host comes from the secret, never from the request, and the dialer refuses any address that is not public.
  • The stand-in can sit wherever the key would: a Bearer token, Basic auth, a header of the app's own, or the query string.
  • A secret can name the variable that holds the API's address (for example OPENAI_BASE_URL). In the community cloud it is set to the egress address, so SDKs that read it need no code change.
  • The connection refuses localhost, private networks and the cloud's metadata service, so the proxy cannot be turned against the platform's own servers.
  • A key kept in the control plane can be changed in place, with no restart of the app.

Sources:

Check yourself

The laptop's owner copies the stand-in key out of the container. What can they do with it?

Very little. It works only through the platform's proxy, only for that one host, only at 10 calls a second, and only while that copy runs. When the copy stops or moves, the stand-in dies. They never see the real key.

5.4 · Networking

Agent tunnel and sessions

Relevant source files

A VPS needs a way in: a terminal in the browser and real SSH from a laptop. Piko tunnels cannot carry that, because they are replaced every 6 minutes and every SSH session would be cut. So each agent that keeps a VPS holds a second, longer-lived connection: its own tunnel to the control plane.

The problem

Interactive sessions last hours, the machine is behind NAT, and Cloudflare only proxies HTTPS ports.

The idea

One outgoing WebSocket per agent with yamux on top, authenticated by the machine's own token, which never expires; the control plane opens a stream per session.

What it costs

An agent reconnect, or a control plane restart, ends open sessions. People reconnect; tmux helps.

Two kinds of session

flowchart TB
  BR["Browser terminal<br>xterm.js"] -- "WebSocket<br>/api/v1/vps/NAME/terminal" --> CP["Control plane"]
  SSH["ssh, scp, VS Code<br>via ccloud ssh ProxyCommand"] -- "WebSocket<br>/api/v1/vps/NAME/ssh" --> CP
  CP -- "new yamux stream<br>op pty or ssh" --> AG["Agent<br>GET /agent/v1/tunnel"]
  AG -- "pty: podman exec on a terminal" --> V["VPS container"]
  AG -- "ssh: agent is the SSH server,<br>then podman exec" --> V
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class AG hl
The VPS image needs no SSH server: the agent is the SSH server, and every session is podman exec into the container.

Sources:

The tunnel itself

  • The agent dials GET /agent/v1/tunnel with its machine token while it keeps a VPS and is not paused, and reconnects with backoff from 1 second to 1 minute.
  • The control plane accepts it only with no Origin header, so a web page cannot open it with someone's cookie. A machine taken out of the pool is cut off within a minute.
  • yamux pings every 20 seconds and gives a stuck write 30 seconds; its 10 second default would end the session during a burst on a home uplink.
  • Each stream starts with one JSON line: {"op": "pty" or "ssh", "vps": id}. The agent answers {"ok": true} or an error within 20 seconds.

Sources:

An SSH session, end to end

%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
  participant U as ssh on a laptop
  participant C as ccloud ssh
  participant CP as Control plane
  participant A as Agent
  participant V as VPS container
  U->>C: ProxyCommand: ccloud ssh NAME
  C->>CP: WebSocket /api/v1/vps/NAME/ssh (session)
  CP->>A: stream on the agent tunnel, op ssh
  A-->>C: SSH handshake, the VPS's own host key (ccloud pins it)
  U->>A: public key login: only the owner's SSH keys
  A->>V: podman exec, with a wrapper that reports the exit code on fd 3
  V-->>U: shell, command, scp or sftp
SSH runs end to end between the laptop and the agent; the control plane only relays bytes. No port forwarding is allowed.

The stdin lesson

podman exec does not return while its stdin is open, even after the program inside has ended, and SSH clients and browsers keep stdin open. Sessions never ended. Now every program runs under a small wrapper that writes its exit code to file descriptor 3 the moment it ends; the agent then closes stdin or hangs up the terminal itself. When a session ends, its shell inside the container gets SIGHUP, as sshd does, so closed browser tabs leave no shells behind.

Sources:

Check yourself

Why is a WebSocket over 443 used for SSH instead of a normal SSH port?

Cloudflare Free proxies only 443 and a few HTTPS ports, and exposing a raw port would mean revealing the server's address. Wrapping SSH in a WebSocket lets it ride the same front door as everything else.

6 · Build and deploy

Build and deploy

Relevant source files

There are three ways to get an app running, and they all end in the same place: a workload pinned to one image digest, placed by the scheduler. The difference is how much of the work the platform does before that.

The problem

People arrive with a ready image, with one repository, or with a repository that holds several apps that must find each other.

The idea

Turn every path into the same thing as early as possible: an image digest the scheduler can place.

What it costs

Building from source needs a trusted builder machine to be running.

Three paths, one destination

flowchart TB
  I1["Deploy an image<br>docker.io/alice/api:1.4"] --> PIN["imageDigest: ask its registry<br>which build the tag points to"]
  I2["Deploy from a repository<br>source: repo, branch, folder"] --> Q["queue a build of the newest commit"]
  I3["Import a repository<br>several apps at once"] --> DET["detect apps, then POST /api/v1/imports<br>all or nothing"]
  DET --> Q
  Q --> BLD["a builder builds and pushes<br>to the platform's registry"]
  BLD --> PIN2["the image's digest"]
  PIN --> W["workload pinned to a digest"]
  PIN2 --> W
  W --> S["scheduler places copies"]
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class W hl
Whatever the path, machines only ever see image@sha256:….

Sources:

What a deploy request says

FieldMeaning
image or sourceA container image, or {repo, branch, folder, build_cmd, start_cmd}
sizesmall, medium, large, or custom CPU and memory
mode, replicasspot, standby, or active with 2 to 5 copies
portWhere the app listens; $PORT is 8080 unless set
variables, secretsPlain values, and secrets that go where the cloud allows (Secrets and egress)
healthAn HTTP path or a TCP check, with timings (Health checks)

Sources:

6.1 · Build and deploy

Import and detection

Relevant source files

Before anything is built, the import page reads the repository and shows the apps in it: framework, commands, port, the variables they read, what they depend on, and what would go wrong. Nothing in the repository is ever run to find this out.

The problem

The deploy form used to ask for folder, commands and port by hand, and a wrong guess was discovered only after a build ran and failed.

The idea

Fixed rules, run on the files, that predict what the real builder (Railpack) will do. Read-only, cached by commit.

What it costs

A second source of truth beside Railpack, which can disagree with it, and a rule set to re-check when Railpack changes.

How a repository is read

flowchart TB
  U["GET /api/v1/repositories/inspect"] --> C{"result cached<br>for this commit and rules version?"}
  C -- "yes" --> OUT["apps, frameworks, commands,<br>port, variables, warnings"]
  C -- "no" --> L["list the commit's files: one API call"]
  L --> M["read manifests first,<br>then each app's source"]
  M --> R["rules in Railpack's order of languages<br>group by framework, not folder name"]
  R --> OUT
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class R hl
Up to 60 files per app and 400 per repository, config-like names first. If time runs out, apps are still shown and the result is marked incomplete.
  • A variable's value never leaves internal/detect, only its name. From a committed .env only names are kept, with a warning.
  • An address between two apps is filled in only when the code shows what shape it takes, because a wrong value would be baked in by the build.
  • TestMatchesRailpack holds what Railpack actually chose for six generated apps, so the rules cannot drift from the builder unnoticed.
  • Limits keep one person from exhausting GitHub's allowance: 12 looks a minute per person after a burst of 10, 4 repositories read at once, 25 seconds per look.

Sources:

Private repositories: the GitHub App

People connect GitHub through a GitHub App with read-only access to the repositories they pick. What they may see is always GitHub's answer to their own token at that moment; the platform stores no list of who can read what. A builder never gets a token at all, only a link to one commit's archive that lasts 5 minutes.

Not connectedConnected
Reads withThe server's token, public repositories onlyTheir installation, for private repositories
GetsPublic repositories by pasting the addressA repository list, private repositories, deploy on push

Sources:

Importing several apps together

POST /api/v1/imports checks every app first, then saves them all in one transaction: if any part fails, nothing is created, not a workload, a secret or a build. Apps from one branch are pinned to the same commit, each app's address is filled into the others' variables where the code shows its shape, and all get the same project label.

Sources:

Check yourself

Why are predicted commands not sent to the build unless the owner edits them?

Railpack is the one that really builds. If our prediction were sent as an override and was slightly wrong, it would break a build Railpack would have got right. Predictions are for showing; Railpack decides.

6.2 · Build and deploy

Builders and the registry

Relevant source files

Source code becomes an image on a builder: a machine the platform admin trusts as much as the platform itself, which runs no apps. The image goes to the platform's own registry, behind a gate in the control plane that decides every push and pull.

The problem

Building on the machines that run apps fails for slow downloads, small memory, inconsistent results and trust: a machine could build anything and claim it was the app.

The idea

Dedicated builders, one build at a time, and short-lived credentials that work for one app's repository only while its build runs.

What it costs

No builder running means no new deploys from source. Every image byte passes through the control plane and sits on its disk, and old images are not cleaned up yet.

A build, end to end

%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
  participant CP as Control plane
  participant B as Builder's agent
  participant GH as GitHub archive
  participant R as Registry gate /v2/
  CP->>CP: queueBuild: pin the branch's newest commit
  CP-->>B: sync answer lists the build, with a one-time token
  B->>GH: download the commit's .tar.gz
  B->>B: Dockerfile? build --pull. Otherwise railpack build on BuildKit
  B->>R: push apps/workload-id, user build, its token
  R->>R: only this app's repository, only while the build runs
  B->>CP: sync: done, with the digest
  CP->>CP: rollOutBuild: copies move to the new digest one at a time
Builds time out after 15 minutes. A lost builder's build goes to another. Secrets never go to a build; plain variables do, as build arguments.

Sources:

Why Railpack

Most repositories have no Dockerfile. The team built seven fresh generator apps (Express, NestJS, Next.js, Vite, Django, Rails, Laravel) with three builders and default settings only. Railpack ran 5 of 7 out of the box with the smallest images (a Vite app at 154 MB); Heroku's builder ran 4; Paketo ran 2, with images around 1 GB. Small images matter when laptops download them over home internet. An app's own Dockerfile always wins.

Sources:

The registry gate

The registry program (CNCF Distribution) listens on 127.0.0.1 and checks nobody. The control plane stands in front of it at /v2/ and decides every request from what it already knows:

WhoUserPasswordMay
A machinemachineIts machine tokenPull the apps it has a copy of (assigned, starting or running)
A buildbuildThe build's one-time tokenPush and pull its own app's repository, while the build runs

Nobody lists the registry or deletes from it, and a push's request to copy a layer from another app's repository becomes a plain upload, so no token can reach into another app. The build token is an HMAC of the build; only its hash is stored.

Sources:

Builders are special machines

The platform admin adds a builder in Admin, Builders: a one-time join key, and the machine that joins with it (sudo ccloud join --builder --key …) has the role builder from its first sync. The scheduler skips builders for apps, so no app can tamper with a build, and builders are nobody's shared capacity.

Sources:

Check yourself

Why is a cheating builder so dangerous, and what limits the damage?

What a builder produces runs on every machine the app moves to, so a cheating builder could plant code in other people's apps. That is why builders are trusted like the platform. Each build's token still works only for its own app and only while it runs, so a leaked token is worth little.

6.3 · Build and deploy

Jobs and cron

Relevant source files

A job is a container that runs to completion: a backup, a report, a batch. Jobs suit volunteer machines well, because a run whose machine goes away can simply run again somewhere else.

The problem

Scheduled work must run on time, exactly as often as intended, on machines that may vanish mid-run.

The idea

Each run is an ordinary workload of kind job. The scheduler, liveness and the meter already handle it; jobs only add the schedule.

What it costs

A run that loses its machine starts again from the beginning, so jobs should be safe to repeat.

Definition and runs

flowchart TB
  J["Job definition<br>image, command, size, schedule, rules"] --> CL["cronLoop every 5 s:<br>runs that are due"]
  J --> BH["Run now, by hand"]
  CL --> RUN["a workload of kind job,<br>job_id set"]
  BH --> RUN
  RUN --> SCH["scheduler places it like any copy"]
  SCH --> END{"exit code"}
  END -- "0" --> OK["done, lease rounded up to a whole minute"]
  END -- "error, or machine lost" --> RT["runs again, up to the job's retries"]
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class RUN hl
One code path for every container: a job's run is placed, watched and billed exactly like an app's copy.

Sources:

The rules a schedule needs

RuleChoices
ScheduleA cron expression in a chosen time zone
OverlapWhat a due run does while an earlier one is still going: skip it, allow both, or replace the old one
RetriesHow many times a failed run, or one whose machine was lost, runs again
Time limitRuns past it are ended
Missed slotsIf the control plane was down when runs were due, the next tick starts one run, not one per missed time

Sources:

Check yourself

A nightly backup job missed three nights because the control plane was down. How many runs start when it comes back?

One. A backup that missed three slots needs one backup now, not three at once.

7 · VPS

VPS

Relevant source files

A VPS is a Linux machine with root (Ubuntu 24.04 or Debian 12) on one of the pool's machines. Unlike an app, it cannot simply restart somewhere else: its disk is the point. Most of the VPS code exists to keep that disk safe while its host comes and goes.

The problem

Give people a box with root and a lasting disk, on machines that sleep, lose power and leave the platform.

The idea

A system container placed once and kept on its machine, with its own disk image that is copied off the machine all the time, and a generation number that says who may write.

What it costs

On a community machine, the owner can look inside a running VPS. A container is not a VM; a microVM tier for machines with KVM is the next step.

A VPS on its machine

flowchart TB
  subgraph m["The machine"]
    AG["Agent: vpsManager"]
    subgraph store["VPS store: XFS with reflink"]
      IMG["disk.img<br>ext4, 10, 20 or 40 GB"]
      LOG["write log"]
    end
    DEV["loop device, or NBD served by the agent"]
    CT["Podman system container<br>fyp-vps-id, its own network"]
  end
  HUB["Control plane: the hub"]
  OBJ[("Folder or S3 bucket<br>sealed packs and batches")]
  IMG --> DEV
  DEV --> CT
  AG --> CT
  AG -- "photos and log batches, sealed" --> HUB
  HUB --> OBJ
  USR(["Owner: browser terminal or ssh"]) --> HUB
  HUB -- "agent tunnel" --> AG
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class HUB hl
The disk image is the container's root filesystem (podman create --rootfs). Its size is also its quota.

Sources:

How the container is made

  • The same limits as apps: CPU, memory equal to swap, --cpu-shares 2, --oom-score-adj 1000, no-new-privileges, and 4096 processes.
  • A network of its own (fyp-vpsnet-<id>, isolated) and, where the host allows, --userns=auto, so root inside is nobody outside. If the runtime refuses that, the agent drops it from then on and logs why.
  • Labelled fyp.vps, never fyp.machine, so the app reconciler, the owner guard and leaving never touch it.

Sources:

Three rules the agent keeps

  1. It removes a VPS (container, disk, network, host key) only when the control plane lists it in vps_remove. Never because it is missing, never on an unknown state.
  2. A VPS the desired state no longer lists is stopped, not removed, and only once a fresh desired state came from the control plane since the agent started.
  3. While the owner pauses sharing, VPSs are stopped; they start again on resume.

And one safety reflex: with no successful sync for 60 seconds, the agent pauses its VPSs, so a machine cut off from the platform stops changing a disk that may be restarting elsewhere.

Found in review, 20 Nov 2026

That pause only counts from the last successful sync. After an agent restart there has been none (lastSyncOK is 0 and is not saved), so an agent that restarts while it cannot reach the control plane starts its VPSs from state.json and never pauses them. If the VPS was moved meanwhile, two copies run at once. The hub stays consistent thanks to generations, but programs inside run twice. A fix: count the fence from the agent's start when there was no sync yet, and start no VPS from saved state before the first fresh answer.

Sources:

Placement, power and billing

A VPS is its own resource in the vps table, not a workload: the scheduler and liveness never treat it as a replaceable copy. placeOneVPS places it once, on a machine in its cloud that reports a VPS store with room. Its generation goes up each time it is placed. A running VPS is billed per minute like a primary copy; a stopped one is free.

Sources:

The pages in this section

7.1 · VPS

Disk copies and fencing

Relevant source files

Every minute or so, a VPS's machine takes a photo of its disk and sends only the blocks that changed since the last photo the hub accepted, sealed with the VPS's own key. If the machine disappears, the VPS can restart elsewhere from the newest photo.

The problem

A disk must outlive its machine, but home upload is slow (5 to 30 Mbps) and must never slow the VPS down.

The idea

Run at local disk speed, and send changes in the background. A slow link only makes copies older; it never slows the disk.

What it costs

Without the write log, up to about a minute of changes can be lost when a machine dies. Machines need xfsprogs.

Taking a photo

%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
  participant A as Agent (takeCopy)
  participant FS as ext4 in disk.img
  participant H as Hub (control plane)
  participant O as Object store
  A->>FS: freeze (FIFREEZE)
  A->>A: clone disk.img to next.img (FICLONE, tens of ms)
  A->>FS: thaw
  A->>A: compare next.img with base.img: extents, then 64 KiB blocks
  loop changed blocks, 512 per pack
    A->>A: flate, then AES-256-GCM with the VPS key
    A->>H: PUT /agent/v1/vps/id/packs/seq-n?generation=g
    H->>O: store the sealed pack
  end
  A->>H: POST /agent/v1/vps/id/copies (generation, seq, base, packs)
  H->>H: Postgres transaction: right generation? builds on the newest?
  H-->>A: copy_seq, or 409 fenced or base
  A->>A: only now next.img becomes base.img
With the write log on, photos are taken between two writes without a freeze, and go every 15 minutes instead of every minute.

The last step is the subtle one. base.img is replaced only after the hub accepted the copy, so a failure never skips a change: the next round compares with the same base again.

Sources:

Postgres decides which copy counts

The hub accepts a copy only if both checks pass, inside one transaction:

CheckRefused withWhy
The machine holds the VPS at this generation409 fencedAn old machine that comes back must never add to the chain
base is the hub's newest copy and seq the next one409 baseCopies form a chain; a gap would corrupt the restore

On fenced the agent stops copying that VPS. On base, or when it has no base, it sends a full copy (every non-zero block), which starts a new chain. Because the check lives in Postgres and not in the store, any S3-compatible bucket works.

Sources:

Keeping the chain short

Every 20 copies, checkpoint merges the maps into one full map, so a restore reads a few maps rather than hundreds. Packs that no copy needs any more are deleted after 6 hours, so a restore already in progress still finds them. Objects are stored as vps/<id>/<generation>/<copy>-<n>.pack.

Sources:

Restoring

A machine given a VPS with copy_seq > 0 and no disk for it restores: it reads the map, makes a sparse image, writes every block from its pack, keeps a clone as its new base, replays any write-log batches after the copy, and then sends only what changes. A failed restore is tried again; the VPS is never created empty.

Sources:

Found in review, 20 Nov 2026

The bucket alone cannot restore anything. S3 holds only sealed block bytes. Which block goes where (vps_photos.manifest), the copy chain and every VPS's key (sealed with SECRETS_KEY) live only in Postgres on the main server, which has no backups yet. Off-server backups of the database and the key are what make "your data is safe" true.

Packs are bound only to their VPS. The sealing's additional data is meshly-vps:<id>, while write-log batches also bind their range. Someone who can write the bucket could swap one pack of a VPS for another of the same size undetected. Binding the generation, copy and pack number closes that.

The hub buffers whole uploads. Packs (up to 40 MiB) and batches (up to 48 MB) are read fully into memory, with no limit on how many at once, on a 2 GB server.

Sources:

Check yourself

Why freeze the filesystem before cloning, if the clone takes only milliseconds?

Without a freeze the clone could catch ext4 halfway through a change, with its journal and data out of step. A freeze makes the filesystem consistent on disk for that instant; cloning keeps the freeze short.

7.2 · VPS

The write log

Relevant source files

Photos every minute can lose a minute of work. People asked for no data loss. On home internet the honest version is: lose only the last few seconds of saved writes, at no cost to disk speed. The write log does that by putting the agent itself in the disk's path.

The problem

Capture every write in order, on WSL2's kernel too, where DRBD, ublk and dm-era are not available.

The idea

The agent serves the VPS's disk as a network block device (NBD) to its own kernel, logs every write before applying it, and ships what each flush covered every 2 seconds.

What it costs

Our process is in the disk's path, the image is written twice (about 8% of random-write speed), and batches double the upload for data changed between photos.

The path of one write

flowchart TB
  P["A program in the VPS writes and calls fsync"] --> K["Kernel: /dev/nbdX<br>backend meshly-vps:id"]
  K --> S["Agent's NBD server<br>internal/nbd"]
  S --> L["1. append a record to the write log<br>CRC32C, in arrival order"]
  L --> I["2. write the bytes to disk.img"]
  S --> F["Flush: fsync disk.img<br>the write is now safe on the laptop"]
  F --> B["every 2 s or 4 MiB:<br>records the flush covered, merged per 4 KiB block"]
  B --> H["Hub: POST /agent/v1/vps/id/log<br>sealed batch, in order"]
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class S hl
Only records a completed flush covered are sent, so the hub never holds a write the laptop could still lose. The laptop's disk always wins.

Sources:

The pieces

PieceWhat it does
internal/nbdA userspace NBD server, and device setup over generic netlink. The kernel holds I/O for 10 minutes if the server goes away (dead_conn_timeout) and resends unanswered requests after 90 s
internal/wlogThe log: 64 MiB segments, 8-byte aligned records with CRC32C. A record the file ends inside is a torn tail and is cut; a damaged whole record is never skipped
wlog.CoalesceTurns records into a batch: zero ranges for trims, then each block's newest bytes, so a block written ten times goes once
wlog sealingAES-256-GCM with the VPS key, bound to the VPS and the batch's record range
cp/vpslog.goThe hub takes a batch only from the current generation, only on top of a copy, and only if it follows its newest record; a retry is harmless

Sources:

When the log cannot go on

There is exactly one fallback: start the log over and send a photo marked log_reset. The hub then forgets any batch past that photo in the same transaction. It happens when:

  • the log passed its ceiling (the smaller of 8 GiB and a tenth of the store's free room) or its oldest record is 6 hours old;
  • the log is damaged, or a log write failed;
  • the computer restarted without stopping the VPS cleanly (a new boot id): the log was not fsynced, so after a power cut it may lack records the image has;
  • the hub misses part of the log (gap).

Sources:

When the upload cannot keep up

Each VPS has a lag budget: 30 seconds ("protect my data"), 300 seconds by default, or 0 ("keep me fast"). Once the backlog would take more than 70% of the budget to send, the disk's writes are slowed to 80% of what the upload carries. With 0 it is never slowed, and falls back to photos at the ceiling. The dashboard shows each VPS as Live or how far behind it is.

Sources:

Surviving an agent restart

When the agent stops, it leaves the NBD devices attached. The kernel holds their I/O, and the next agent takes them over (NBD_CMD_RECONFIGURE). The VPS never notices. The agent marks itself PR_SET_IO_FLUSHER and caps each device's dirty pages at 256 MB, against the deadlock of a disk served from the same machine. A server that stops on its own is taken over again; after three failures the VPS is stopped rather than left on a disk nobody serves.

Sources:

Check yourself

Why is the log itself not fsynced on every flush?

It measured 2.4 ms rising to 8.8 ms per fsync. Instead the image is fsynced on every flush (that makes an acknowledged write safe on the laptop), and an unclean restart throws the log away and sends a photo. Safety holds; speed is kept.

7.3 · VPS

Moving a VPS

Relevant source files

Moving a VPS is risky in a way moving an app is not: the old machine's disk may hold writes nobody else has. The rules on this page all answer one question: when two disks of the same VPS exist, which one is the newest?

The problem

An early version moved a VPS off an offline laptop to nowhere, then would not let it go back. A naive fix could start from an old disk and silently lose newer work.

The idea

Move only to a machine with room, in one step. Remember which placement saved last (written_gen), and let only that disk continue.

What it costs

With no other machine, the VPS is down until its machine returns. Once moved, it stays, even on a machine that is often offline.

When a VPS moves

flowchart TB
  O["Its machine is offline past move_after_min<br>default 5, 0 = wait"] --> R{"another online machine in its cloud<br>with room for disk, CPU and memory?"}
  R -- "no" --> H["held: it waits, and continues there<br>from its own disk, losing nothing"]
  R -- "yes" --> MV["moveVPS: new generation,<br>placed on that machine in one step"]
  MV --> RS["the new machine restores the newest copy,<br>then the log batches after it"]
  MV --> LO["the old disk becomes a leftover"]
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class R hl
A move is decided together with its destination. The incident of 2026-11-20 is why: a single-machine test moved a VPS to nowhere.

Sources:

The newest disk wins

written_gen is the placement whose machine sent the newest accepted copy or log batch. It changes only when the hub accepts one, and only the current placement can send, so it only ever goes up. The disk of that placement holds everything the copies hold, and maybe more.

When a machine finds a disk for a VPS it is told to run, diskVerdict decides:

The disk belongs toVerdict
This placementUsed, as always
The placement named adopt_gen (that is, written_gen)Continued from: it becomes this placement's, and its copies and log go on
Any other placement, or unknown while copies existSet aside as <id>.old-g<gen>-<time> for a day, and the newest copy is restored
What happenedwritten_genLaptop 1's old diskWhen the VPS returns to laptop 1
Laptop 1 saved last, then went offline; laptop 2 restored but saved nothing1the newest, plus what it had not sentcontinues from its own disk: nothing lost
Laptop 2 ran it and saved its work2older than the copiesset aside; restored from the newest copy with laptop 2's work

Copies and batches go to the hub only from the placement's own disk, so an older disk can never be sent as the newest. A leftover's one-day clock starts only once a newer placement has saved, because until then it may be the newest disk.

Sources:

Found in review, 20 Nov 2026

Decision 0023 says a restore "can lose at most the seconds the old placement never sent". It is whatever that placement never sent: seconds while its upload keeps up, up to its lag budget (300 s by default) plus up to 60 s before the fence when it does not, and longer with a budget of 0.

A planned move: drain first

An owner can move a VPS whose machine is online, for example off a laptop that is online only 62% of the time. Nothing is lost, because the old machine finishes first:

%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
  actor U as Owner
  participant CP as Control plane
  participant A1 as Old machine
  participant A2 as New machine
  U->>CP: POST /api/v1/vps/NAME/move (to laptop 2)
  CP-->>A1: spec says drain
  A1->>A1: stop the VPS, close its disk cleanly
  A1->>CP: one last copy (starts the hub's log over)
  A1->>CP: report drained, with that copy's number
  CP->>CP: still the newest? then move: new generation on laptop 2
  CP-->>A2: spec with the VPS
  A2->>CP: restore that copy, start the VPS
The VPS is down for the stop, the last copy and the restore: a few minutes for a small disk. A move not done within an hour, or cancelled, starts the VPS again where it was.

The page suggests a move when the VPS's machine has been online less than 80% of its life and another machine with room has been online at least 15 points more.

Sources:

Check yourself

Why not let a VPS go back to its old machine whenever that machine returns?

If another machine ran the VPS in between and saved work, the old disk is older than that work. Starting from it would lose the work without anyone noticing. Only the disk of written_gen may continue.

Why does a planned move send one last copy after stopping the VPS?

A stopped disk holds every write. A copy taken then holds everything the VPS ever saved, so the new machine starts from exactly the same disk.

8 · Observability

Observability

Relevant source files

Metrics and logs ride on the same sync request as everything else. Nothing on a machine opens another connection to send them, so they work wherever the machine can reach the control plane.

The problem

People need graphs and logs of apps that run on machines they cannot reach, kept cheaply on a small server.

The idea

Piggyback samples and log lines on every sync, store numbers and text in two small purpose-built stores, and keep a short tail in Postgres as a fallback.

What it costs

Seven days of searchable logs, at most 2 GiB. A machine that cannot sync cannot send its last words until it can.

Where each signal goes

flowchart TB
  subgraph agent["Agent"]
    S["metricsLoop: a sample every 5 s<br>device, platform share, each container"]
    LG["log lines since the last answered sync<br>podman logs --timestamps"]
  end
  SY["POST /agent/v1/sync"]
  S --> SY
  LG --> SY
  SY --> MV["metrics.add: VictoriaMetrics<br>SQLite when it is absent"]
  SY --> FL["freshLines: drop lines at or before<br>the newest already kept"]
  FL --> TL["Postgres: each copy's latest lines<br>the live view"]
  FL --> VL["VictoriaLogs: every line, 7 days<br>the searchable history"]
  MV --> DASH["Dashboard graphs and trends"]
  TL --> DASH
  VL --> DASH
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class SY hl
The control plane also records platform-wide figures once a minute (platformMetricsLoop).

Sources:

Following logs live

When someone opens an app's logs, the control plane marks the workload (logs_until) and holds that machine's syncs for 1 second instead of 10. Each answer brings the next sync a second later, carrying the new lines. Lines in VictoriaLogs carry the app's id, not its name, because a deleted app's name can be taken again but its id never is.

Sources:

What the dashboard shows

ViewEndpointData
App metricsGET /api/v1/deployments/{name}/metricsCPU and memory of each copy over a time window
App logs, live and history…/logs, …/logs/historyPostgres tail; VictoriaLogs search
Machine metricsGET /api/v1/machines/{id}/metricsThe whole device and the platform's share of it
FailoversGET /api/v1/failoversFrom, to, how, and seconds until serving again
Admin overview and trendsGET /api/v1/admin/overview, …/trendsMachines, apps, usage across the platform

Sources:

9 · Clients

Clients

Relevant source files

Four programs talk to the platform for people: the web dashboard for everyone, the desktop app for laptop owners, ccloud for servers and VPS logins, and fyp for admins. They all use the same API; none holds logic the control plane does not also enforce.

The problem

Very different people (consumers, laptop owners, server admins, platform admins) need different tools for the same platform.

The idea

Thin clients over one API, with every rule enforced on the server or by the agent.

What it costs

The dashboard's build is committed into the Go tree, so a change to web/ must be rebuilt and committed with it.

Who uses what

flowchart LR
  C(["Consumer"]) --> D["Dashboard<br>web/, built into internal/cp/ui"]
  PO(["Laptop owner"]) --> W["Desktop app, Windows<br>agent inside WSL"]
  SA(["Server admin"]) --> CC["ccloud join, status,<br>limits, pause, leave"]
  VO(["VPS owner"]) --> CS["ccloud login, ssh"]
  PA(["Platform admin"]) --> F["fyp CLI<br>admin token"]
  D --> API["/api/v1"]
  W --> LA["agent's local API<br>127.0.0.1:7070"]
  W --> API
  CC --> LA
  CC --> API
  CS --> API
  F --> V1["/v1"]
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class API hl
The desktop app and ccloud drive the agent through its local API; the agent enforces every limit itself.

The dashboard

A React app built with Vite in web/. Its build is embedded in the control plane binary (//go:embed all:ui) and served as a single-page app, so the platform ships as one file. Pages: landing, sign-in, deploy, import, provide (machines), organization, jobs, VPS, domains, credits, builders and admin. CI warns when the committed build is not the build of web/.

Sources:

ccloud

CommandWhat it does
ccloud join --server URL --key KEYAsks how much CPU and memory to share and installs the agent as a systemd service
ccloud join --builder --key KEYJoins a builder: checks Docker and Railpack, starts BuildKit
ccloud status, limits, pause, resume, leave, logsManage the local agent
ccloud login --server URL --email …Save a session, on Linux, macOS or Windows
ccloud ssh NAME [COMMAND]Log in to a VPS over the control plane (Agent tunnel)
ccloud ssh config NAMEA ~/.ssh/config block, so plain ssh NAME.meshly, scp, sftp and VS Code work

The control plane serves the agent and CLI builds at /download/{file}, from an allow-list of file names.

Sources:

The desktop app

A native window (WebView2 on Windows) with three views: This laptop (set it up once, share, set limits, pause, live graphs of the laptop and the platform's part), Earnings (credits earned, time in the pool), and Settings. The agent runs inside WSL. The app is only a front end: closing it changes nothing about what the laptop shares.

Sources:

fyp

The original admin CLI, from the first MVP: enrollment keys, machines, accounts, services and jobs, through /v1 with a fixed admin token. The roadmap lists closing /v1 as Phase 0 work.

Sources:

10 · Operations

Operations

Relevant source files

How the live platform is put together, how code reaches it, and how the team learns from what breaks. The authoritative runbook is docs/operations.md; this page explains its shape.

The problem

A student team must run a public service on a small cloud budget, with several people pushing code and one person deploying.

The idea

One server, every service on localhost behind nginx and Cloudflare, CI on every push, and deploys that the server pulls only after CI passed.

What it costs

A single point of failure, a single disk, and no scheduled backups yet.

What runs where

ServiceRoleListens on
meshly-cpThe control plane127.0.0.1:8080
meshly-pikoThe app gateway: proxy, upstream, admin127.0.0.1:8100, :8001, :8002
postgresqlRecords127.0.0.1:5432
meshly-vm, meshly-vlMetrics; logs (7 days, 2 GiB)127.0.0.1:8428, :9428
meshly-registryBuilt images127.0.0.1:5000
nginxTLS behind Cloudflare, routing by name80, 443

A t3.small in Sydney (the AWS organization allows EC2 in no other region tested), a builder that sleeps when idle, and two free Oracle machines in Mumbai for community capacity.

Sources:

From a push to production

flowchart TB
  PR["Pull request or push to dev or main"] --> CI["CI: worm guard first, dashboard build check,<br>gofmt, go vet, go test -race with Postgres,<br>VictoriaMetrics and VictoriaLogs, builds"]
  CI -- "push, every job passed" --> REL["release job: meshly-release bundle<br>files, checksums, commit"]
  REL --> TM["on the server: meshly-deploy timer, every minute<br>read-only token, nothing pushed in"]
  TM --> CK{"newest push run passed,<br>for the branch tip,<br>digest and checksums match?"}
  CK -- "no" --> W["wait"]
  CK -- "yes" --> DP["dump the database, keep the previous build,<br>install, restart meshly-cp"]
  DP --> HC{"/login answers within 30 s?"}
  HC -- "yes" --> OK["live"]
  HC -- "no" --> RB["put the previous build back"]
  classDef hl stroke-width:1.5px
  classDef warn stroke-width:1.5px
  class CK hl
Automatic deploys are on dev and start paused; they are not switched on yet. Until then, deploys are done by hand with update.sh, by one person.

The server pulls rather than GitHub pushing, for two technical reasons: SSH to the server is open only to listed addresses (GitHub's runners use thousands), and a server key stored in GitHub could be read by every collaborator with write access.

Sources:

The worm guard

The repository was hit twice by a worm that hid code in .vscode/, npm scripts and build configs, and force-pushed forged copies of branches. scripts/ci-guard.sh fails on those signs and on lock-file packages from outside the npm registry or with new install scripts. CI runs it before any npm command, and so should everyone locally. It refuses to run without a real python3, after it once passed on Windows with none of its checks.

Sources:

Learning from incidents

Anything that broke and taught something gets a file in docs/incidents, with what the team believed and why it was wrong. The lessons so far, in one line each:

IncidentLesson
Following logs closed tunnelsTimers interact: the server, not the agent, should shorten the hold
A failed request every tunnel renewalKnow exactly what a library's Close does before relying on it
App port mismatchTraffic must go to the port the app really listens on: an owner's PORT now decides both
Restarting an agent moved its apps (open)A restart is not leaving; only ccloud leave should say goodbye
podman exec waits for stdinDetect a program's end yourself; do not wait for the tool
Write-log figures stopped every syncNumbers that move on their own must never trigger early reports
A VPS moved to nowhereDecide a move together with its destination; test with a single machine
Turning off Cloudflare's proxy broke HTTPSThe origin certificate only works behind the proxy

Sources:

11 · Known limits

Known limits

Relevant source files

A system built on other people's machines has limits that no code removes, limits the team has listed and plans to fix, and a few found in a code review. They are kept apart here, so you know whose finding each one is.

Limits no code removes

On a machine the platform does not control, the owner has root. They can read an app's files, environment and traffic on that machine, and the platform cannot reliably detect it, because the owner also controls the agent that reports. The real fix is confidential computing (AMD SEV-SNP, Intel TDX), which volunteers' laptops do not have. So machines receive built images, never repositories, and never secrets inside an image; but for Node, Python, PHP and Ruby the image still holds the source files. Code and data that must stay private belong on an organization's own servers. Machines behind NAT cannot offer a public IP.

Sources:

The team's Phase 0: safe for strangers

The roadmap lists what must be fixed before the community cloud opens to strangers. In short:

AreaOpen problem
BrowserApps share the dashboard's domain, so an app could plant cookies; framing and cross-site protections are incomplete
BillingA machine can claim running with no tunnel and earn credits
ReportsA machine can write metric samples for another machine's copy; a forged X-Forwarded-For can be recorded as its IP
KeysA leaked organization server key gets real secrets with no approval step
APINo rate limits outside /egress; /v1 is public behind one fixed token; names like login are not reserved
MachinesApps on one machine share the Podman network; nothing caps an app's writable layer; an agent restart counts as leaving the pool
OperationsNo database backups; no branch protection on GitHub

Sources:

Found in a code review, 20 Nov 2026

These come from a review of the VPS records against the code on dev at d0ca527. They are not yet in the team's documents.

SeverityFindingWhere
HighAn agent that restarts while it cannot reach the control plane starts its VPSs from state.json and never pauses them, because the 60 s fence counts only from a successful sync. Two copies of a moved VPS can run at once
HighNo grace period after a control plane restart: after an outage over 2 minutes, liveness marks machines lost before their agents reconnect (backoff up to 2 minutes), so apps and VPSs move needlessly
HighVPS copies in S3 cannot be read without Postgres (manifests, chain, keys sealed with SECRETS_KEY), and the server has no backups, so "your data is safe" depends on backups that do not exist yet
MediumThe hub reads whole packs and batches into memory with no limit on uploads at once, on a 2 GB server; all VPS traffic passes through it twice
MediumDecision 0023 promises a restore loses "at most the seconds" never sent; with a slow upload it is up to the lag budget plus 60 s, and more with a budget of 0
LowCopy packs are sealed with only the VPS id as additional data, so packs of one VPS could be swapped in the bucket undetected

Docs that lag the code

The roadmap's status table, last updated before mid-November, still lists health checks as having no HTTP path and custom domains as not started. On dev, the agent runs HTTP and TCP health checks (4.3) and custom domains are built and off by default (5.2).

Sources: