Community Cloud is the code behind Meshly, live at meshlycloud.com. It runs other people's apps on machines that people share. Some machines are laptops in a community pool. Others are Linux servers that an organization owns. One small control plane decides what should run where. An agent on every machine makes that happen and reports back.
This wiki explains the code on the dev branch. Each page starts from the problem a part of the system solves, then shows how the code solves it, with diagrams and links to the exact lines. If you are new, read the pages in order: each one builds on the ones before it.
The problem
Most computers sit idle most of the day. Someone who needs to host a small app pays a cloud provider instead.
The idea
Pool the idle capacity. A central service hands out work, and each machine runs it in a container and keeps a public address working.
What it costs
The machines are unreliable and belong to someone else. Almost every part of the code exists to cope with that.
The system in one picture
Everything central runs on one small server. Machines never accept incoming connections: they always dial out, to the control plane for their work and to the Piko gateway to carry visitors' requests to their apps.
flowchart LR
subgraph people["People"]
direction TB
D["Dashboard<br>web/"]
C["ccloud CLI<br>cmd/ccloud"]
W["Desktop app<br>cmd/meshly-desktop"]
V(["Visitors of an app"])
end
subgraph server["Main server"]
direction TB
N["nginx behind Cloudflare"]
CP["Control plane<br>internal/cp"]
PK["Piko gateway"]
PG[("PostgreSQL")]
OBS[("VictoriaMetrics<br>VictoriaLogs")]
RG[("Image registry")]
end
subgraph pool["Shared machines"]
direction TB
A1["Agent + Podman<br>a laptop"]
A2["Agent + Podman<br>an org server"]
B["Builder<br>Docker + Railpack"]
end
D --> N
C --> N
W --> N
V --> N
N --> CP
N --> PK
CP --- PG
CP --- OBS
CP --- RG
A1 -- "sync: long poll" --> N
A2 -- "sync: long poll" --> N
A1 -- "app tunnel" --> PK
A2 -- "app tunnel" --> PK
B -- "image push" --> N
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class CP hl
Arrows show who opens the connection. Nothing connects into a shared machine.
Sources:
What happens when someone deploys an app
Follow one deploy from start to finish. Every later page zooms into one of these steps.
%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
autonumber
actor U as Consumer
participant CP as Control plane
participant DB as PostgreSQL
participant A as Agent on a machine
participant P as Piko gateway
U->>CP: POST /api/v1/deployments
CP->>DB: save the workload
CP->>CP: scheduler picks a machine
CP->>DB: add a copy, raise the machine's version
CP-->>A: answer the sync request it was holding
A->>A: pull the image, start the container
A->>A: health check passes
A->>P: open a tunnel for the app
A->>CP: next sync says running, tunnel up
CP->>DB: open a lease, start the meter
Note over P: visitors at https://app.apps-domain now reach the container
A deploy, end to end. The numbered steps match the list below.
The consumer asks for an app: an image (or a GitHub repository), a size and a mode. See Build and deploy.
The control plane saves it as a workload in PostgreSQL. See Data model.
It was generated from the code on dev at commit d0ca527 and from the team's own records in docs/. The decisions and incidents it quotes are the team's; the explanations are summaries. Where the wiki and the code disagree, the code wins. Findings from the review of 20 November 2026 are marked as such on Known limits.
1.1 · Overview
First principles
Relevant source files
Start from nothing. You want to run a web app on a stranger's laptop and give it a public address. Write down everything that can go wrong. The list is short, and every package in this repository answers one item on it.
The five hard problems
flowchart LR
G["Run apps on machines<br>that other people share"]
G --> P1["Machines sit behind<br>home routers"]
G --> P2["Machines vanish<br>without warning"]
G --> P3["Machines belong to<br>someone else"]
G --> P4["Visitors need one<br>stable address"]
G --> P5["Some apps keep<br>state on disk"]
P1 --> S1["Machines only dial out:<br>long poll and tunnels"]
P2 --> S2["Desired state, heartbeats,<br>failover and fencing"]
P3 --> S3["Owner guard protects the owner,<br>stand-in keys protect the app"]
P4 --> S4["A gateway routes by app,<br>never by machine"]
P5 --> S5["Disk copies with generations,<br>and a write log"]
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class G hl
Each problem on the left is a fact about the machines. Each answer on the right is a design rule the code follows everywhere.
1. You cannot reach the machine
A laptop at home sits behind a router that blocks incoming connections. So the control plane can never call a machine. The machine calls the control plane, and keeps one request open so the answer can come back the moment something changes (a long poll). Visitors' traffic uses the same trick: the machine opens an outgoing tunnel to the gateway, and the gateway sends requests down it.
Sources:
2. The machine disappears
The owner closes the lid, the power goes, the Wi-Fi drops. Nothing tells the control plane. So every request from a machine doubles as a heartbeat, and silence past fixed timers means offline, then lost. The control plane always sends the whole desired state, never a diff, so a machine that missed something corrects itself on its next answer. And a machine that loses contact must assume it was replaced: after 60 seconds it stops serving (fencing), so two copies never fight over the same app.
Sources:
3. The machine belongs to someone else
This cuts both ways. The owner must always win: our containers get the lowest CPU weight, the agent keeps a memory reserve, and if the owner needs memory the newest shared container is evicted at once. The app must be protected from the owner too: the owner is root on their own laptop and can read anything a container holds. So API keys never go to community machines. The app gets a stand-in key, and the control plane swaps in the real one when the app calls out.
Sources:
4. Visitors need one address
An app may move between machines several times a day. Its address must not. The gateway (Piko) knows apps by name, not machines by address. Whichever machine currently runs the app holds the tunnel for that name, so moving an app is just a different machine opening the same tunnel.
Sources:
5. Some apps keep state
A stateless web app can restart anywhere. A VPS cannot: its disk is the point. So a VPS gets sealed copies of its disk sent off the machine, a generation number that says which machine may write, and a write log that ships every saved write within seconds. Only the machine holding the current generation can add to the copies.
Sources:
Three rules that carry most of the weight
Rule
Why
Where you will see it
Only dial out
Home networks block incoming connections, and machine addresses must never be exposed
Sync long poll, Piko tunnels, the agent's own VPS tunnel
The control plane holds the truth; machines converge
Machines miss messages, restart and lose state. A full desired state with a version heals all of that
desired.go, the agent's reconciler, VPS specs
Assume anything can be lost, so version, retry or fence it
A late message from a machine that was replaced must never count
Copy attempts, report filtering in applySync, VPS generations, the 60 s fence
Check yourself
Why does the control plane send the full desired state on every sync, instead of only what changed?
A machine can miss an answer, restart, or lose its saved state. With the full list and a version number, the next answer always brings it back in line. No message is special, so none can be lost in a way that matters.
Why must a machine stop serving after 60 seconds without an answer?
From the control plane's side, that machine may already be marked lost and its apps started elsewhere. If it kept serving, two copies of one app could answer visitors with different data. Stopping first is the safe guess.
1.2 · Overview
Glossary
Relevant source files
The code uses a small, consistent vocabulary. The dashboard sometimes uses friendlier words than the API and the database; both are listed.
People and places
Word
Meaning
In the code
Control plane
The one central service: API, scheduler, liveness, billing
internal/cp
Agent
The program on every shared machine that runs work and reports back
internal/agent
Machine
A computer that joined the pool (laptop, server or builder)
machines table
Provider
A person who shares a machine
machines.owner_id
Consumer
A person who deploys apps
workloads.created_by
Community cloud
The open pool of anyone's machines; pays in credits
org_id empty
Organization cloud
An organization's own servers, used only by its people, free
org_id set
Admin, member, consumer
Roles in an organization. Members bring servers; consumers deploy
admin, maintainer, member
Platform admin
The first account; the only one who can look at another cloud
scopeOf, ?org=
Builder
A trusted machine that turns source code into images and runs no apps
machine role builder
Apps and their copies
Word
Meaning
In the code
Deployment, workload
What a consumer asked for: image, size, mode, variables
workloads
Copy, instance
One run of a workload on one machine
instances
Primary
A copy that serves visitors through a tunnel
role = 'primary'
Standby
A warm copy with no tunnel, ready to take over
role = 'standby'
Mode
spot: one copy. standby: a primary and a standby. active: 2 to 5 serving copies
workloads.mode
Attempt
A counter on each new copy, so a late report about an old one is ignored
instances.attempt
Size
small, medium, large (1, 2, 4 units) or custom CPU and memory
appSizes
Unit
A quarter of a core and 256 MB; what billing counts
unitsOf
Digest
The exact build an image tag points to; every copy runs the same one
workloads.digest
Rollout
Replacing copies one at a time with a new build or new variables
rollout.go
Job, run
A container that runs to completion, by hand or on a cron schedule
jobs, kind job
Keeping machines in line
Word
Meaning
In the code
Capacity promise
A provider's statement of what a machine will share, before it joins
capacity
Join key, server key
One-time key for a promise; an org member's reusable key for servers
keys, ek_…, sk_…
Desired state
Everything one machine should run, sent whole on every answer
SyncResponse
Version
A number per machine that goes up on every change to its desired state
machines.version
Sync
The agent's long-polled request: its report in, desired state out
POST /agent/v1/sync
Reconcile
Make what runs match what is desired, with the smallest change
reconcileOnce
Fencing
Stopping work on a machine that may have been replaced
syncFence, VPS generations
Failover
Moving an app off a lost machine; every one is recorded
failovers
Owner guard
Evicts a shared copy the moment the owner needs memory
guardOwner
Lease
One copy's use of a machine, from start to stop; what the meter bills
leases
Credits
The community cloud's money, kept as millicredits
credit_ledger
Traffic and secrets
Word
Meaning
In the code
Piko
The open-source gateway that carries visitors down machines' outgoing tunnels
github.com/andydunstall/piko
Endpoint
An app's name at the gateway; a tunnel serves one endpoint
InstanceSpec.Endpoint
Piko token
A 10 minute JWT listing the endpoints a machine may serve
pikoToken
Stand-in key
A fake API key (ccs_…) a community app holds instead of the real one
standIn
Egress
The control plane's proxy that swaps a stand-in for the real key
/egress/
Agent tunnel
Each agent's own WebSocket to the control plane, for VPS terminals and SSH
/agent/v1/tunnel
Custom domain
An app's own name, proven with a TXT record
domains
VPS words
Word
Meaning
In the code
VPS
A Linux box with root that stays on one machine
vps
Generation
Goes up each time a VPS is placed; only that placement may send copies
vps.generation
Photo, copy
A snapshot of a VPS's disk; only changed blocks are sent
vps_photos
Pack
Up to 512 changed 64 KiB blocks, compressed and sealed
vps_packs
Write log, batch
Every write, logged; flushed writes sent every 2 seconds
internal/wlog, vps_log
written_gen
The placement whose machine sent the newest copy or batch
vps.written_gen
Leftover, set aside
An old disk kept after a move, in case it holds the newest data
vps_leftovers, .old-g…
Drain
A planned move: stop, send a last copy, then move
vpsmove.go
1.3 · Overview
Repository map
Relevant source files
One Go module (fyp) builds five programs. A React dashboard lives in web/ and is compiled into the control plane. The team's documentation lives next to the code and changes in the same commits.
Programs and packages
flowchart LR
subgraph cmds["cmd: programs"]
cpb["controlplane"]
agb["agent"]
cc["ccloud"]
desk["meshly-desktop"]
fyp["fyp admin CLI"]
end
subgraph pkgs["internal: packages"]
cp["cp<br>control plane"]
ag["agent"]
proto["proto<br>shared types"]
det["detect"]
tun["tunnel"]
obj["objstore"]
vd["vpsdisk"]
wl["wlog"]
nbd["nbd"]
jwt["jwt"]
end
cpb --> cp
agb --> ag
agb --> nbd
cc --> tun
fyp --> proto
cp --> proto
cp --> det
cp --> tun
cp --> obj
cp --> vd
cp --> jwt
ag --> proto
ag --> tun
ag --> vd
ag --> wl
ag --> nbd
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class cp hl
class ag hl
Imports between the Go packages. proto is the contract: both sides of every wire use its types. The desktop app imports nothing internal: it drives the agent through its local API.
What lives where
Path
What it is
Read it when
internal/cp
The control plane: API, scheduler, liveness, leases, builds, domains, VPS hub (about 14,700 lines)
Reads a repository and predicts its apps, commands, port and variables
Import from GitHub
internal/tunnel
yamux over WebSocket, line and frame helpers
VPS terminals and SSH
internal/vpsdisk
Changed-block detection, manifests, sealed packs
VPS copies
internal/wlog
A disk's write log and merged batches
The VPS write log
internal/nbd
A userspace NBD server and netlink device setup
The VPS write log
internal/objstore
A folder or S3 (SigV4) behind one interface
Where VPS copies are kept
web/
The dashboard (React, Vite); its build is committed to internal/cp/ui
UI work
contracts/
agent-protocol.md and openapi.yaml: how the parts talk
Before changing any wire format
docs/decisions
24 decision records: options, choice, cost
You ask "why is it like this?"
docs/incidents
What broke, why, and what was learned
Before touching timers or tunnels
docs/operations.md
How the live platform runs, updates and recovers
Deploying
scripts/
Local stack, builds, packaging, the worm guard, server setup and deploys
Running anything
Run it locally
The local stack needs WSL or Linux as root with Podman. Start PostgreSQL, VictoriaMetrics, VictoriaLogs and a registry in containers (the commands are in the README), then:
bash scripts/dev-up.sh starts Piko and a control plane on port 9080 with an empty database. The first account to sign up becomes the platform admin.
In the dashboard, promise a machine (Provide, Add a machine, A Linux server) to get a join key.
bash scripts/dev-up.sh agent lab-1 KEY starts a local agent that joins with it, as a real machine would.
go test ./... uses the same databases and skips tests whose database is not running.
Before any npm command
Run bash scripts/ci-guard.sh first. This repository was hit twice by a worm that hid code in .vscode/, npm scripts and build configs. The guard fails on those signs and on lock-file packages from outside the npm registry. CI runs it before anything else.
Sources:
2 · Architecture
Architecture
Relevant source files
The architecture is a hub and many spokes. The hub is one server that holds every decision and every record. The spokes are machines that only ever call the hub. This page shows the pieces, where they run, who connects to whom, and who trusts whom.
The problem
Hundreds of unreliable machines must act as one cloud, on a small budget, run by students.
The idea
Put all state and all decisions in one control plane with one database. Keep machines simple: they run what they are told and report what they see.
What it costs
One server is a single point of failure, and everything competes for about 1.9 GB of memory.
The pieces and where they run
flowchart TB
subgraph edge["Internet edge"]
CF["Cloudflare<br>TLS, 100 MB body cap"]
end
subgraph main["Main server: AWS t3.small, Sydney"]
NG["nginx<br>ports 80 and 443"]
CP["meshly-cp<br>control plane, 127.0.0.1:8080"]
PK["meshly-piko<br>proxy :8100, upstream :8001"]
PG[("postgresql<br>records")]
VM[("meshly-vm<br>metrics")]
VL[("meshly-vl<br>logs, 7 days")]
RG[("meshly-registry<br>images, 127.0.0.1:5000")]
end
BLD["meshly-builder<br>same AWS zone, stopped when idle"]
subgraph pool["Shared machines"]
LAP["Laptops in the community<br>desktop app, agent in WSL"]
SRV["Organization servers<br>ccloud join, systemd"]
end
CF --> NG
NG --> CP
NG --> PK
CP --> PG
CP --> VM
CP --> VL
CP --> RG
LAP --> CF
SRV --> CF
BLD -- "private network" --> NG
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class CP hl
Every service on the main server listens on 127.0.0.1. Only nginx faces the internet, and only through Cloudflare.
Two more pieces sit outside this picture. Community capacity during the pilot comes from two Oracle Cloud Always Free machines in Mumbai. The builder is a separate AWS machine that costs about $0.11 an hour, so it stays stopped when nobody is building.
Sources:
Who connects to whom
Read this table as the system's wiring. Notice that every row starts at a machine or a person, never at the server.
From
To
What
Why this way
Agent
POST /agent/v1/sync
Long poll: report in, desired state out
Works behind NAT; doubles as heartbeat
Agent
Piko upstream, /piko/v1/upstream/<app>
One WebSocket per app it serves
Visitors reach the app with no open port on the machine
Agent
GET /agent/v1/tunnel
Its own WebSocket with yamux, while it keeps a VPS
Terminals and SSH that must outlive Piko's token renewals
Agent
/v2/ registry gate
Pulls images of apps it runs
Access follows what the control plane already knows
Builder
/v2/ over AWS's private network
Pushes a built image
Cloudflare refuses bodies over 100 MB
Browser
/api/v1/…
The dashboard's API, session cookie
One API for every client
Visitor
https://<app>.<apps domain>
nginx sends it to Piko's proxy, Piko down a tunnel
Routing by app name, not by machine
Community app
/egress/…
API calls with a stand-in key
The real key never leaves the control plane
Sources:
Where data lives
Three kinds of data, three stores, all on the main server's disk.
Data
Store
Why that store
Records and events: accounts, apps, copies, leases, VPSs
PostgreSQL 16
Few rows, many joins, transactions for placement and fencing
Numbers over time: CPU and memory samples
VictoriaMetrics (SQLite when absent)
Compact, fast range queries, little memory
Text: app log lines
VictoriaLogs, 7 days, at most 2 GiB
Full-text search; the latest 200 lines per copy also stay in Postgres
Built images
CNCF Distribution registry
Standard /v2/ protocol behind the control plane's gate
VPS disk copies
A folder or an S3 bucket (VPS_COPIES)
Sealed packs; the maps that make sense of them stay in Postgres
Single disk
Everything above shares one disk on one machine, and the server has no scheduled backups yet (roadmap, Phase 0). See Known limits.
Sources:
Who trusts whom
Security in this system is mostly a question of trust boundaries. Keep this table in mind when you read any code that hands out a secret or accepts a report.
Party
Trusted with
Not trusted with
Control plane and builders
Everything: they decide and build what runs everywhere
An organization's servers
Its own apps' real secrets
Other organizations' apps (placement keeps them apart)
A community machine
Running a copy, reading what is in it
API keys (it gets stand-ins), a VPS's copy store, any other machine's data
An agent's report
What it sees on its own machine
Anything about copies it does not hold: reports about older attempts or generations are ignored
Check yourself
Why does the builder push over AWS's private network instead of the public name?
Cloudflare, in front of the public name, refuses request bodies over 100 MB, and image layers are often larger. nginx accepts uploads only from the private range, so the public path stays read-only.
What would break first if the main server went down for ten minutes?
Everything new: no deploys, no failovers, no billing, and new visitors could not reach apps because Piko runs there too. Running containers keep running on their machines, but their tunnels go through the server, so apps stop answering visitors.
2.1 · Architecture
Desired state and reconciliation
Relevant source files
This is the heart of the system. The control plane keeps, for every machine, a list of what should run there and a version number. The agent keeps asking for that list and makes its machine match it. Every other feature plugs into this loop.
The problem
You cannot push commands to a machine you cannot reach, and a machine that misses one command is now wrong forever.
The idea
Never send commands. Send the whole desired state with a version, and let the machine close the gap itself, as often as needed.
What it costs
One open request per machine, held in memory, so only one control plane can run today.
Commands versus state
Imagine telling a laptop "start app A", then "stop app B". If the laptop misses the second message, it runs B forever and nobody notices. Now imagine telling it instead "you should run exactly A and C, version 42". If it misses version 42, it gets version 43 later, which again lists everything. Nothing is ever lost for good. This is called level-triggered control, and it is why a rebooted agent needs no special recovery.
flowchart TB
subgraph cp["Control plane"]
direction LR
W["Any change:<br>deploy, failover, redeploy"] --> BV["bumpVersion<br>machines.version + 1"]
BV --> WK["waits.wake(machine)"]
DS["desired(machine)<br>copies, VPSs, builds, Piko token"]
end
subgraph ag["Agent"]
direction LR
SY["syncOnce<br>long poll"] --> AP["apply<br>save state.json"]
AP --> RC["reconcileOnce<br>smallest change"]
RC --> PD["Podman containers<br>and tunnels"]
end
WK -. "ends the held request" .-> DS
DS -- "full desired state" --> SY
PD -- "report on the next sync" --> W
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class BV hl
class RC hl
The loop. Changes on the left raise a version and wake the machine; the agent applies the full state and reports what it sees.
The long poll, step by step
%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
participant A as Agent
participant H as handleSync
participant N as notifier
participant DB as PostgreSQL
A->>H: POST /agent/v1/sync (have_version 41, report)
H->>N: wait(machine): take the channel first
H->>DB: applySync: heartbeat, copy states, samples
H->>DB: SELECT version
alt version is still 41
H->>H: hold up to 10 s
Note over N: the scheduler commits a change and calls wake(machine)
N-->>H: channel closed
end
H->>DB: desired(machine)
H-->>A: full desired state, version 42
A->>A: apply, then reconcile
A->>H: next sync at once (have_version 42)
The request is both the heartbeat and the delivery channel. With nothing to say, it returns after the hold, and the agent asks again.
One detail in handleSync is worth slowing down for. The handler takes the wake channel before it reads the version. If it read the version first, a change could land in the gap between the read and the wait, and the request would sleep for 10 seconds on news that already arrived. Taking the channel first closes that gap: any later wake closes the channel the handler already holds.
Sources:
How long a request is held
Normally 10 seconds (-hold). While someone follows an app's logs, the hold drops to 1 second, so new lines arrive every second. An earlier design had the agent cut its own held request every 2 seconds to send lines; it never received an answer, went 60 seconds without a successful sync, and fenced itself: the app went down for as long as anyone watched its logs. That incident is why the server, not the agent, now shortens the hold.
Sources:
What the desired state contains
Field
What it says
version
The machine's version; the agent sends it back as have_version
instances[]
Every copy to run: image pinned to a digest, command, port, CPU and memory, role (primary or standby), variables and secrets as the cloud allows, health check
piko_token
A fresh 10 minute JWT listing the endpoints this machine may serve
vps[], vps_remove[]
The VPSs this machine keeps, with their generation and copy settings, and the ones to delete for good
registry, builds[]
The image registry to log in to; for builder machines, the builds to run
Sources:
The agent's side
When an answer arrives, apply stores it, saves it to state.json if the version changed, notes the time of the last good sync (used for fencing) and signals the reconciler. reconcileOnce then compares the list with the containers Podman reports and makes the smallest change: start what is missing, remove what is no longer wanted, and open or close tunnels. Running it twice does no harm, which is the property that makes the whole loop safe.
Situation
What the reconciler does
Desired copy has no container
Start it, unless it is starting, failed to start, or was evicted by the owner guard
Container not in the desired list
Close its tunnel with a short drain, then remove it
Primary is running and passes its health check
Open its tunnel
No successful sync for 60 s
Close every tunnel (fencing) until the control plane answers again
Owner paused sharing
Remove shared containers
Sources:
Reports can arrive late
The other direction needs the same care. A report about a copy can arrive after that copy was replaced. applySync ignores any report about a copy that is no longer active, or about an older attempt, so a late "failed" can never kill the new copy and a late "running" can never bill twice. The VPS code uses generations for the same reason.
Sources:
Check yourself
A laptop sleeps for an hour and wakes up. What does it need to do to catch up?
Nothing special. Its next sync carries an old have_version, so the control plane answers at once with the full current state, and the reconciler removes whatever moved away while it slept and starts whatever is now assigned to it.
Why can only one control plane run at a time today?
The wake channels live in one process's memory. A change committed by a second process could not wake a request held by the first. Running several would need a notification shared between processes. Scaling the control plane is the roadmap's Phase 4.
2.2 · Architecture
Two clouds, one platform
Relevant source files
Community Cloud started as two student projects: one for anyone's laptops, one for an organization's servers. They turned out to need the same scheduler, tunnels, agent and billing. So there is one platform, and a single column decides which cloud a row belongs to: org_id.
The problem
Two codebases would build scheduling, tunnels, billing and the agent twice, for a difference that comes down to whose hardware it is.
The idea
One control plane, one API, one database. org_id empty means the community cloud; set means an organization.
What it costs
Every query must be scoped correctly, or one organization sees another's data.
What differs, and where
Community cloud
Organization cloud
Code
org_id
empty
set
every main table
Machines
Anyone's laptops, joined with the desktop app
The organization's Linux servers, joined with ccloud join and a member's server key
registry.go
Placement
communityStrategy: machines with no organization
orgStrategy: only the organization's own machines
scheduler.go
Secrets
API keys stay on the control plane; the app gets a stand-in
Every secret goes into the container
secrets.go
Billing
Consumers pay credits per unit-minute; providers earn them
Free; leases still record usage
leases.go
Who sees what
Your own apps and machines
Members see their servers; the admin sees everything in the organization
orgs.go
One helper decides the scope
Every handler that lists or changes data asks scopeOf which cloud the request may see. Everyone is pinned to their own: the community cloud or their organization. Only the platform admin (the first account) may pick another one with ?org=. Keeping this in one place means a future split into two products changes one function.
flowchart TB
R["Request from a signed-in person"] --> S{"scopeOf"}
S -- "platform admin with ?org=X" --> X["Cloud X"]
S -- "member of an organization" --> O["Their organization only"]
S -- "everyone else" --> C["Community cloud only"]
X --> Q["Queries filtered by org_id"]
O --> Q
C --> Q
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class S hl
Scoping is one decision, made once per request.
Sources:
Check yourself
Can an organization's app ever run on a community laptop?
No. orgStrategy.Eligible accepts only machines whose org_id equals the workload's. Community machines have none, so they are never eligible.
Why does an organization's app get its real API keys, when a community app does not?
An organization runs its apps on its own servers, which it already trusts. A community app runs on a stranger's laptop, where the owner is root and could read any key in the container.
3 · Control plane
Control plane
Relevant source files
The control plane is one Go process (internal/cp) with one PostgreSQL database. It does two kinds of work: it answers HTTP requests from people, agents and apps, and it runs a set of background loops that keep the world moving even when nobody asks.
The problem
Decisions about many machines must be consistent: two schedulers must never put the same copy in two places.
The idea
One process, one database, and every decision made inside a Postgres transaction.
What it costs
It scales as far as one process and one database go, and its in-memory wake-ups pin it to one copy.
Inside the process
flowchart TB
subgraph http["HTTP surfaces: Handler and webRoutes"]
AG["/agent/v1/<br>enroll, sync, tunnel, VPS copies"]
API["/api/v1/<br>dashboard, CLI, desktop"]
V1["/v1/<br>admin-token CLI"]
REG["/v2/<br>registry gate"]
EG["/egress/<br>API key proxy"]
DL["/download/<br>agent and CLI files"]
end
subgraph loops["Background loops"]
L1["assignLoop: 5 s and on every change<br>placement, builds, VPS moves"]
L2["livenessLoop: 2 s<br>offline, lost, fast down"]
L3["meterLoop: 1 min<br>billing"]
L4["cronLoop: 5 s<br>scheduled job runs"]
L5["copiesLoop: 30 s<br>VPS checkpoints, clean-up"]
L6["domainLoop: 1 min<br>custom domain checks"]
end
DB[("PostgreSQL")]
http --> DB
loops --> DB
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class DB hl
Requests and loops never talk to each other directly. They meet in the database, and the scheduler is woken through a channel (kickAssigner).
Every loop uses one helper, s.loop(every, wake, once). It runs a round on a timer or when woken, and on shutdown it lets the current round finish and starts no other, so a round of scheduling or billing is never cut halfway.
Sources:
The HTTP surfaces
Prefix
Who calls it
How it authenticates
Main handlers
/agent/v1/
Agents
Machine token (Bearer); only its SHA-256 is stored
handleEnroll, handleSync, handleAgentTunnel, VPS copy and log endpoints
HTTP Basic: machine token or a build's one-time token
handleRegistry
/egress/
Community apps
Signed stand-in key
handleEgress
Sources:
Starting and stopping
New opens the database (running the idempotent schema in store.go) and reads the configuration from flags and environment. Run starts the loops and the listeners. On SIGTERM the server stops taking connections, answers every held agent sync at once (so agents reach the next control plane quickly), lets loops finish their round, and gives requests in flight up to -drain (10 s) to finish. A second signal stops it at once.
Sources:
Found in review, 20 Nov 2026
After the control plane itself was down for a while, liveness counts that downtime against every machine. See Liveness and failover.
3.1 · Control plane
Data model
Relevant source files
The whole schema lives in one string in store.go. It runs on every start: CREATE TABLE IF NOT EXISTS and ALTER TABLE … ADD COLUMN IF NOT EXISTS, so an old database upgrades itself in place. Times are milliseconds since 1970 in BIGINT columns.
The problem
The system must remember who owns what, what runs where, and what was billed, and survive restarts at any moment.
The idea
A small relational model: people, machines, workloads and their copies, with leases joining copies to money.
What it costs
No migration tool: the schema only grows. Renaming or dropping a column needs care by hand.
Apps, machines and money
erDiagram
ORGS ||--o{ USERS : "has"
USERS ||--o{ CAPACITY : "promises"
CAPACITY |o--o| MACHINES : "fulfilled by"
USERS ||--o{ MACHINES : "provides"
USERS ||--o{ WORKLOADS : "deploys"
WORKLOADS ||--o{ INSTANCES : "runs as copies"
MACHINES ||--o{ INSTANCES : "hosts"
INSTANCES ||--o{ LEASES : "billed through"
LEASES ||--o{ CREDIT_LEDGER : "two rows a minute"
WORKLOADS ||--o{ SECRETS : "holds"
WORKLOADS ||--o{ BUILDS : "built by"
WORKLOADS ||--o{ DOMAINS : "served at"
WORKLOADS ||--o{ FAILOVERS : "recorded in"
JOBS ||--o{ WORKLOADS : "each run is one"
MACHINES {
text id PK
text owner_id
text org_id
text status
bigint version
bigint last_seen
}
WORKLOADS {
text id PK
text name
text mode
text digest
text org_id
text state
}
INSTANCES {
text id PK
text workload_id FK
text machine_id FK
text role
int attempt
text state
}
LEASES {
text id PK
text instance_id FK
int units
text role
bigint billed_until
}
The core tables. org_id on users, machines, workloads and leases is what separates the two clouds.
Sources:
What each table is for
Table
One row is
Notes
orgs, users, sessions
An organization, a person, a signed-in session
Passwords are PBKDF2-SHA256 with a salt; sessions store only a token hash
capacity, keys
A promise to share a machine, and the join keys for it
A key has uses left and an expiry
machines
A joined computer
Its token hash, offer, live host figures, status, version, time in the pool
workloads
An app or a job run
digest pins the build; rollout_* columns track a redeploy
instances
One copy on one machine
Active states are assigned, starting, running; the last log line time prevents duplicates
leases, credit_ledger
A copy's use of a machine; a credit movement
billed_until makes billing safe to repeat
failovers
One app leaving a lost copy
From, to, how, when it was down, detected and serving again
secrets
One sealed secret of an app
Its host for egress, and a use counter
builds
One source build
Its status, digest and the hash of its one-time registry token
jobs
A job definition with an optional schedule
Each run is a workloads row with job_id
domains, certs
A custom-domain claim; a cached certificate
A partial unique index allows one verified owner per host
vps and vps_*
A VPS and its events, packs, photos, log batches, leftovers
A person's SSH public keys; their GitHub tokens, sealed
VPS tables
erDiagram
USERS ||--o{ VPS : "owns"
MACHINES ||--o{ VPS : "keeps"
VPS ||--o{ VPS_EVENTS : "history"
VPS ||--o{ VPS_PHOTOS : "copies, as block maps"
VPS ||--o{ VPS_PACKS : "sealed changed blocks"
VPS ||--o{ VPS_LOG : "write log batches"
VPS ||--o{ VPS_LEFTOVERS : "old disks on old machines"
USERS ||--o{ SSH_KEYS : "log in with"
VPS {
text id PK
text machine_id
bigint generation
bigint written_gen
bigint copy_seq
bigint log_seq
}
The S3 store holds only sealed bytes. Which bytes belong where (vps_photos.manifest), the chain of copies and every VPS's key live in these tables.
Sources:
A thin layer over database/sql
db.go wraps the Postgres driver so the code can write ? placeholders, which it rewrites to $1, $2… (rebind). isUnique recognises a unique violation, which several places use as a lock: for example, two accounts racing to verify the same custom domain.
Sources:
Check yourself
Why is a lease its own table instead of a column on the copy?
A copy can change role (a standby promoted to primary) and the price changes with it. Billing follows the role: the standby lease ends and a primary lease starts. Separate rows also make "what did this provider earn" a simple sum.
3.2 · Control plane
Machines and capacity
Relevant source files
Before a machine can run anything, its owner promises how much to share, the machine joins with a key, and from then on it reports what is really free. This page follows a machine from promise to retirement.
The problem
A machine must prove it was invited, and must never be given more work than it can hold, even when its owner starts using it.
The idea
A promise with a one-time key, an offer capped at the hardware, and a live free figure that leaves a reserve for the owner.
What it costs
Server keys in organizations are kept in plain text so their owner can see them again; replacing one is a click.
From promise to machine
%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
actor O as Owner
participant D as Dashboard or desktop app
participant CP as Control plane
participant A as Agent
O->>D: share 2 cores and 2 GB
D->>CP: POST /api/v1/capacity (a promise)
CP-->>D: join key ek_ (1 use, 1 hour)
O->>A: ccloud join, or the desktop app
A->>CP: POST /agent/v1/enroll with the key, offer and hardware
CP->>CP: cap the offer at the hardware, create the machine
CP-->>A: machine_id and token
A->>A: save identity.json (0600)
A->>CP: first sync, the machine goes online
The server keeps only the SHA-256 of the machine token. In an organization, a member's reusable server key replaces the one-time key.
Sources:
How much a machine may offer
An offer is capped twice. At join, capOffer limits it to the hardware: CPU less a tenth, memory less the larger of 512 MB and a tenth. While running, the agent's host reporter measures what is actually free and keeps a reserve for the owner (512 MB or 10% of memory). The scheduler uses the smaller of the offer and the live free figure.
Machine
Hardware
Largest offer
Owner reserve
Laptop, 4 cores, 8 GB
4000 m, 8192 MB
3600 m, 7372 MB
819 MB
Server, 8 cores, 16 GB
8000 m, 16384 MB
7200 m, 14745 MB
1638 MB
Small VM, 1 core, 1 GB
1000 m, 1024 MB
900 m, 512 MB
512 MB
Sources:
Units and sizes
People choose sizes; billing counts units. One unit is a quarter of a core and 256 MB. Small, medium and large apps take 1, 2 and 4 units. A custom size is billed by whichever resource needs more units. A machine's capacity in units is the smaller of its CPU and memory counts, so a machine with lots of CPU and little memory is honest about what it can hold.
Sources:
The life of a promise
stateDiagram-v2
[*] --> promised: owner promises capacity
promised --> provisioned: a machine joins with the key
provisioned --> usable: first sync, healthy, not paused
usable --> used: a copy runs on it
used --> usable: the last copy leaves
usable --> provisioned: paused or runtime unhealthy
used --> offline: silent, left or removed
usable --> offline: silent, left or removed
offline --> usable: syncs again
What the dashboard shows for each promise. used means credits are being earned.
Sources:
Every sync is a check-in
recordCheckIn stores the machine's latest host figures, its public IP, any runtime error and the time, and tracks how long it has been online (online_since, pool_ms). Those two numbers give the machine's reliability: the share of its life it has spent online. The scheduler prefers machines with a high one.
Sources:
Leaving
Joining again with a new key for the same promise retires the old machine. A machine that sends leaving (pause, shutdown) goes offline at once and its apps move at once. An admin can remove a machine; its token stops working on its next check.
Sources:
Check yourself
A provider offers 4 GB, but the laptop has only 1.5 GB free right now. What does the scheduler see?
The smaller figure. The agent reports free memory beyond the reserve, and the scheduler uses the smaller of that and the offer. So the laptop gets no new work that needs more than what is really free.
3.3 · Control plane
Scheduler and placement
Relevant source files
The scheduler answers one question many times a second: for this copy of this app, which machine? It runs on every change and every 5 seconds, places one copy per database transaction, and wakes the chosen machine.
The problem
Pick a machine that is likely to stay online, has room, is allowed to run the app, and does not put two copies in the same place.
The idea
Filter the machines with hard rules, then rank the rest with a simple score you can compute by hand.
What it costs
It is greedy: one copy at a time, best machine now. It never moves running copies to make a better fit.
One round
flowchart TB
T["assignOnce<br>every change, every 5 s"] --> B["assignBuilds and placeVPS"]
B --> W["for each active or queued workload"]
W --> R{"rolloutStep<br>a redeploy step?"}
R -- "yes" --> W
R -- "no" --> P["placeOne"]
P --> N{"what is missing?"}
N -- "a primary" --> F["record a failover,<br>promote the standby if any"]
N -- "a standby" --> SB["standbyStrategy"]
N -- "nothing" --> W
F --> PK["pick: filter, then score"]
SB --> PK
PK --> C["addCopy, bumpVersion, commit"]
C --> WK["wake the machine"]
WK --> P
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class PK hl
placeOne repeats until a workload has every copy its mode wants, or no machine has room.
Sources:
How many copies
Mode
Primaries
Standbys
When a machine is lost
spot
1
0
A new copy starts elsewhere: pull and start
standby
1
1
The standby is promoted; measured about 17 s from an abrupt drop to serving
active
2 to 5
0
Piko spreads requests over the others; a replacement starts
Sources:
Hard rules: which machines may take the copy
pick starts from online machines that run apps (builders run none) and keeps a machine only if all of these hold:
The strategy allows it: communityStrategy takes machines with no organization; orgStrategy takes only the workload's own organization.
It runs no other active copy of this workload. Copies of one app never share a machine.
No copy of this same build failed there in the last 10 minutes (failedRecently). A broken machine does not take the same app again and again.
It has enough free CPU and memory for the copy.
Soft rules: the score
The machines that pass are sorted first by how many copies of this app were lost or evicted there before (fewer is better), then by the strategy's score:
Term
Range
Meaning
100 × reliability
0 to 100
Share of its life the machine has been online. The strongest term by far
packing
0 to 10
How full it already is: fill busy machines before waking idle ones
apart
−30, −20 each
For every existing copy on the same provider, and on the same public IP
A standby uses a different score: the same reliability term, plus free units (prefer the emptiest machine), and twice the spreading penalty (−60 and −40), because a standby on the same Wi-Fi as its primary dies with it.
Sources:
Worked example
An app in the community cloud already has one copy on Ali's laptop at home. Where does the second copy of an active app go?
Machine
Reliability
Packing
Apart
Score
Sara's desktop, 4 of 8 units used
95
5
0
100
Ali's second laptop, same home, 2 of 8 used
90
2.5
−50
42.5
Bilal's laptop, idle
60
0
0
60
Sara's desktop wins. Ali's second laptop is almost as reliable, but it shares a provider and a network with the first copy, so one power cut would take both.
After the choice
In the same transaction, addCopy inserts the copy with the next attempt number, a pending failover record gets its destination, and bumpVersion raises the machine's version. After the commit, waits.wake ends that machine's held sync, so the agent hears within milliseconds.
Sources:
When an app itself is broken
If a workload's copies fail on 2 different machines, or 3 times in all, the problem is the app, not the machines. The workload is marked failed and the scheduler stops trying until the owner redeploys.
Sources:
Check yourself
Why does reliability weigh ten times more than packing?
On volunteer machines the main risk is the machine going away. A copy on a machine that is usually there saves a failover later; packing only saves some idle machines from waking up.
Why is the spreading rule soft and not a hard filter?
A demo or a small organization may have all its machines in one building. A hard rule would leave the app with nowhere to run; a penalty still prefers spreading when it can.
3.4 · Control plane
Liveness and failover
Relevant source files
A machine that vanishes sends no message. The control plane notices only silence, and silence has to be read carefully: a slow sync looks the same as a dead laptop for a while. Liveness turns silence into decisions with a few timers.
The problem
Decide when a silent machine is gone, fast enough that visitors barely notice, and never so fast that a slow machine loses its apps for nothing.
The idea
Escalate with time: no new work at 30 s, apps moved at 2 minutes, sooner if Piko also lost every tunnel from it.
What it costs
A spot app can be down for up to two minutes plus a restart. Standby mode exists to cut that.
What happens when a laptop vanishes
flowchart LR
T0["0 s<br>lid closes"] --> T15["15 s<br>no sync and its Piko<br>tunnels gone:<br>down now"]
T15 --> T17["about 17 s<br>standby promoted,<br>serving again"]
T0 --> T30["30 s<br>no sync: offline,<br>no new work"]
T30 --> T120["2 min<br>copies marked lost,<br>placed elsewhere"]
T0 --> F60["60 s on the laptop<br>if it is still on:<br>it closes its tunnels"]
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class T15 hl
class F60 warn
Two clocks run at once: the control plane's, on top, and the agent's own fence, below.
The three timers
Rule
Default
Effect
Code
No sync for OfflineAfter
30 s
Status offline; the scheduler gives it no new work
livenessOnce
No sync for FastDownAfter and all its Piko tunnels gone
15 s
machineDown now: Piko's view confirms the machine is really gone
pikoHealth
Offline for LostAfter
2 min
machineDown: active copies become lost, the scheduler replaces them
livenessOnce
Agent: no successful sync for 60 s
60 s
The agent closes its tunnels until the control plane answers
reconcileOnce
Sources:
Failover, recorded
When a workload is short of a primary, placeOne first opens a failovers row (from which copy and machine, why, when it went down and when that was noticed). If the app has a standby, promoteStandby turns it into the primary and its agent opens the tunnel as soon as it hears. Otherwise a new copy is placed. When the new primary serves, the row gets its serving_at time, and the dashboard shows how many seconds the app was away.
%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
participant L as livenessLoop
participant S as Scheduler
participant DB as PostgreSQL
participant B as Standby's agent
participant P as Piko
L->>DB: machineDown: primary copy lost, version bumped
S->>DB: openFailover (from, reason, down_at)
S->>DB: promoteStandby: role primary, leases switched
S-->>B: wake the held sync
B->>B: primary and healthy, so open the tunnel
B->>P: tunnel for the app
B->>S: next sync: tunnel up
S->>DB: failover serving_at
With a warm standby, recovery is one role change and one tunnel. No image pull, no container start.
Sources:
Fencing on the machine's side
The agent cannot know whether it was declared lost, so it assumes the worst. With no successful sync for 60 seconds, it closes every tunnel and serves nothing until the control plane confirms its work again. A laptop that slept through a failover therefore never serves an old copy next to the new one.
Sources:
Found in review, 20 Nov 2026
No grace period after a control plane restart. Liveness compares each machine's last_seen with now. If the control plane itself was down for more than 2 minutes, every machine looks silent on its first round, before agents have reconnected. Agents back off up to 60 seconds plus jitter, so many reconnect late. Their copies are marked lost and placed on whichever machines reconnected first; VPSs move too once past move_after_min. A fix is to measure silence from the later of last_seen and the control plane's start.
The team's own open incident is related: an agent restart is still counted as leaving the pool (Known limits).
Check yourself
Why look at Piko's tunnels before declaring a machine down at 15 s?
A late sync alone is ambiguous: the control plane might be slow, or the network briefly bad. If Piko has also lost every tunnel from that machine, two independent views agree it is gone, so moving its apps early is safe.
3.5 · Control plane
Redeploys and config changes
Relevant source files
A running app changes in two ways: a new build, or new variables. Both are handled the same way: start new copies, stop old ones only once the new ones serve, and go back if anything fails.
The problem
Replace every copy of a live app without a gap for visitors, and without ending up half old and half broken.
The idea
Pin the exact build (a digest), replace copies one at a time, make before break, and keep what to go back to.
What it costs
A single spot copy with no room elsewhere restarts in place: a short gap. A request can fail at the moment an old tunnel closes.
Pinning the build
A tag like alice/api:latest can point to a different build tomorrow. So at deploy time the control plane asks the image's registry what the tag points to now (imageDigest: a HEAD request, an anonymous token if needed, public addresses only) and stores the digest. Machines run image@sha256:…, so every copy runs the same build and a stale image on disk is never reused. Redeploy asks the registry again.
Sources:
Replacing copies one at a time
%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
participant S as rolloutStep
participant N as New copy's agent
participant O as Old copy's agent
participant P as Piko
S->>N: add a copy at the new digest
N->>N: pull, start, health check passes
N->>P: open the tunnel (both copies serve)
N->>S: sync: running, tunnel up
S->>O: remove the old copy
O->>P: drain, then shut the tunnel
O->>O: remove the container
Note over S: repeat until no copy is old, then commit the redeploy
For a moment old and new serve the same address side by side, so visitors never meet a gap.
Mode
Order of replacement
spot
A new copy on another machine, then the old one stops. With no room elsewhere, it restarts in place
standby
The standby is replaced first, promoted beside the old primary, then the old primary stops and a fresh standby follows
active
One copy at a time; the others keep serving
If a new copy fails, or the rollout makes no progress for 15 minutes (rolloutStuck), rollBack returns the app to the build it ran and the page says why.
Sources:
Changing variables
A container keeps the values it started with, so changing a variable is also a rollout. Every change gets a number (config_rev) and every copy records the number it started with. A copy is old if its build or its number differs. The values to go back to are kept until the change is done.
Some values are built into the app, such as NEXT_PUBLIC_… or VITE_…. Changing one needs a new build of the commit the app already runs. Until that build succeeds, nothing about the app changes.
Sources:
Check yourself
Why not just restart the copies with the new image?
A restart has a gap, and if the new build is broken the app is down. Starting the new copy first and stopping the old one only once the new one serves keeps the app up, and a failure leaves the old copies untouched.
3.6 · Control plane
Accounts and organizations
Relevant source files
People sign up into one of two worlds. In the community cloud they provide or consume capacity. In an organization they have a role that decides what they see and whether they bring servers.
The problem
The same API serves strangers in a public pool and colleagues in a private one, and neither may see the other's data.
The idea
Every account has at most one organization and a role in it; every request is pinned to one cloud by scopeOf.
What it costs
One organization per person, for now.
How an organization works
flowchart TB
SU["Sign up, then New organization"] --> AD["Admin"]
AD -- "Members page invite link" --> MB["Members<br>org_role maintainer<br>bring servers"]
AD -- "Consumers page invite link" --> CO["Consumers<br>org_role member<br>deploy"]
AD --> K1["Admin's server key"]
MB --> K2["Each member's own server key"]
K1 --> J["sudo ccloud join --key sk_..."]
K2 --> J
J --> SV["Servers in the organization's pool"]
CO --> DP["Apps on the organization's servers"]
SV --> DP
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class AD hl
The dashboard says members and consumers; the API and database say maintainer and member.
Any number of servers can join with one server key, until its owner replaces it. A server belongs to the person whose key it joined with.
Members see their own servers. The admin sees every server, person, app and everyone's usage, changes roles, and can remove a person: their apps stop and their servers leave the pool.
Nobody outside sees the organization. Only the platform admin can look in, with ?org=.
Sources:
Sessions and sign-in
Passwords are stored as PBKDF2-SHA256 with a random salt. A session is a random token; the database keeps only its hash. Over HTTPS the browser cookie is named __Host-meshly_session, which an app on a sibling name cannot overwrite. Further browser protections (security headers, a cross-origin check, apps on a separate domain) are listed as Phase 0 work in the roadmap.
Sources:
Check yourself
Why does each member get their own server key, rather than one key for the organization?
So a server always belongs to a person, the admin can see whose it is, and one person's key can be replaced without touching anyone else's servers.
3.7 · Control plane
Leases and credits
Relevant source files
In the community cloud, people who run apps pay credits and people who share machines earn them. The unit of record is a lease: one copy holding some units of one machine, from start to stop.
The problem
Bill fairly by the minute, survive crashes mid-billing, and never charge a minute twice.
The idea
Each lease remembers how far it was billed. The meter bills only the time since, inside the same transaction as the ledger rows.
What it costs
A machine could claim a copy runs when it does not; crediting only copies whose tunnel Piko confirms is on the roadmap.
The rates
What
Price
One unit, primary, per minute
1 credit (1000 millicredits)
One unit, warm standby, per minute
0.25 credit
Welcome balance for a new account
500 credits
Inside an organization
free, usage still recorded
Example: a medium app (2 units) in standby mode for one hour costs 2 × 60 = 120 credits for the primary and 2 × 60 × 0.25 = 30 for the standby. The providers of the two machines earn the same amounts. A job's run is rounded up to whole minutes when it ends: a 61 second run pays for 2 minutes.
Sources:
The meter
%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
participant M as meterLoop (every minute)
participant DB as PostgreSQL
M->>DB: begin, open leases with their billed_until
loop each lease
M->>M: owed = units x rate x time since start
M->>DB: consumer row: minus (owed now - owed before)
M->>DB: provider row: plus the same
M->>DB: billed_until = now
end
M->>DB: commit
Bills are computed from the lease's start each time, so the sum of all bills always equals the whole time, with no rounding drift.
A lease opens when a copy starts running (openLease, which does nothing if one is already open) and closes when it stops or changes role. A balance is simply the sum of a person's ledger rows.
Sources:
Check yourself
The control plane crashes halfway through a billing round. Can a minute be billed twice?
No. The ledger rows and the new billed_until are written in one transaction. A crash rolls back the whole round, and the next round bills from the old billed_until.
4 · Agent
Agent
Relevant source files
The agent is one Go program on every shared machine. It joins once, then spends its life in one loop: ask the control plane what to run, make the machine match, report back. Around that loop run a few helpers, each a goroutine with one job.
The problem
A program on a stranger's laptop must keep many things in step (containers, tunnels, health, metrics, disks) while its network comes and goes.
The idea
One sync loop owns the conversation with the control plane. Every other job runs in its own goroutine and is nudged through small signal channels.
What it costs
Several loops share state under one mutex, and their timers interact. Incidents in docs/incidents show where.
What runs inside an agent
flowchart LR
RUN["Run<br>enroll or load identity,<br>read the host"]
RUN --> SYNC["sync loop<br>syncOnce, apply, backoff"]
RUN --> WATCH["watchLoop<br>ends a held sync early"]
RUN --> MET["metricsLoop<br>samples every 5 s"]
RUN --> HEA["healthLoop<br>checks each service copy"]
RUN --> COP["copyLoop and logLoop<br>VPS copies and write log"]
RUN --> TUN["tunnelLoop<br>own WebSocket for VPS sessions"]
RUN --> LOC["serveLocal<br>127.0.0.1:7070"]
SYNC --> REC["reconcileLoop<br>containers and Piko tunnels"]
SYNC --> VPS["vpsLoop<br>VPS containers and disks"]
WATCH --> SYNC
HEA --> REC
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class SYNC hl
The goroutines Run starts. Arrows between loops are small signal channels: the sync loop wakes the reconcilers, the watch loop and health checks wake the others. Only the sync loop talks to the control plane's sync endpoint.
Sources:
The sync loop
Ask.syncOnce sends the report (every copy's state, host figures, samples, log lines, VPS reports) and waits up to 35 seconds for an answer.
Apply.apply stores the desired state, saves it to state.json if the version changed, records the time of this successful sync, and signals the reconcilers.
Repeat at once. The next request goes immediately; the control plane holds it until something changes or 10 seconds pass.
On failure, back off. Wait 1 second, doubling up to 60 seconds, plus random jitter of up to the same again, and keep running current work meanwhile.
On stop, say goodbye. A clean stop sends leaving, so the control plane moves this machine's apps at once instead of waiting for silence.
Sources:
Reporting early
While a sync is held, things change on the machine: a container exits, the runtime breaks, the owner starts a big program. watchLoop compares a key of the report with the last one sent and, when it differs, ends the held request so the control plane hears now. Free capacity counts at most every 10 seconds, and figures that move on their own (a VPS's lag and upload rate) are left out of the key: an earlier version included them, cut every held sync short, and the agent fenced itself after 60 seconds.
Sources:
Files the agent keeps
File
What
identity.json (0600)
Machine id and token, written once at enroll
state.json
The last desired state, so a restart knows what it was running
env/
Each container's variables file, deleted once the container is created
vps-store/ or vps-store.xfs
VPS disk images, in a filesystem that can clone
Sources:
The local API
The agent listens on 127.0.0.1:7070 for programs on the same machine. The desktop app and ccloud status, limits, pause and resume use it. The agent enforces every limit itself, so closing the desktop app changes nothing about what the laptop shares.
Sources:
Check yourself
Why the random jitter in the backoff?
If the control plane restarts, hundreds of agents fail at the same moment. Without jitter they would all retry at the same moments too, in waves. Jitter spreads them out.
4.1 · Agent
Running containers
Relevant source files
Every app copy is a Podman container started with hard limits. The reconciler decides what to start and stop; a small driver turns those decisions into Podman (or Docker) commands.
The problem
Run strangers' code on someone's laptop with firm limits, without a heavy daemon eating the owner's memory.
The idea
Podman, which needs no daemon, driven through its CLI, with every limit on the command line and nothing exposed beyond 127.0.0.1.
What it costs
Containers share the host's kernel. In practice Podman runs as root. There is no disk limit or per-customer network for apps yet.
How a copy is started
Podman.Start builds one podman run command. Each flag answers a concrete risk:
Flag
Why
--cpus, --memory, --memory-swap equal to memory
The copy gets exactly its size and cannot spill into swap
The owner's programs win the CPU, and the kernel kills our container first
-p 127.0.0.1:<host port>:<app port>
The app is reachable only from the machine itself, where the tunnel picks it up
--restart on-failure:3
A crash is retried locally before anyone is told
--env-file (0600, deleted after create)
Values never appear in the process list, where any program could read them
labels fyp.machine, fyp.instance
The agent recognises its own containers and never touches others
Sources:
The reconcile round
flowchart TB
A["desired copies<br>from the last answer"] --> D{"compare"}
B["containers<br>from podman ps, by label"] --> D
D -- "missing" --> S["start it<br>unless starting, failed or evicted"]
D -- "not wanted, or paused" --> G["close its tunnel with a short drain,<br>then remove it"]
D -- "running primary" --> H{"healthy and<br>not fenced?"}
H -- "yes" --> T["open or keep its tunnel"]
H -- "no" --> C["no tunnel"]
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class D hl
Running the round twice changes nothing the second time. That is what makes it safe to run on every signal and every few seconds.
Sources:
Logs
For each copy, the agent reads podman logs --timestamps from where its cursor is and sends the new lines with the next sync: at most 1000 lines and 128 KB per copy per sync, newest kept. The cursor moves only when an answer comes back, and the control plane drops any line at or before the newest it already has, so each line counts once even across restarts.
Sources:
Docker for builders
The driver speaks to any Docker-compatible CLI. Builder machines use Docker, because Amazon Linux 2023 has no Podman package and Railpack needs BuildKit anyway. The few differences (for example --config versus --authfile for registry logins) are handled in place.
Sources:
Check yourself
Why publish the app's port on 127.0.0.1 instead of all addresses?
Visitors arrive through the agent's tunnel, which connects locally. Publishing on all addresses would let anyone on the owner's network reach the app directly, bypassing the gateway and its health gating.
4.2 · Agent
Protecting the owner
Relevant source files
The person who shares a laptop must never regret it. If they open a video editor while an app runs, the editor must win at once. Five layers make sure of that, from gentle to brutal.
The problem
The owner can start anything at any moment, and nobody will wait for a scheduler to notice.
The idea
Let the kernel favour the owner for CPU, keep a memory reserve, and evict our newest copy the moment that reserve is touched.
What it costs
An evicted copy restarts elsewhere: a short outage for a spot app. Standby mode hides it.
Five layers
flowchart TB
O["The owner starts a heavy program"] --> L1["1. CPU: our containers have weight 2,<br>the kernel gives the owner the CPU"]
O --> L2["2. Memory, slow: free memory is reported<br>beyond a reserve, so no new work comes"]
O --> L3["3. Memory, fast: below the reserve,<br>the newest shared copy is evicted now"]
L3 --> L4["4. Last resort: oom-score-adj 1000,<br>the kernel kills our container first"]
L5["5. Hard caps on every container:<br>CPU, memory, processes"]
L3 --> M["The control plane places the copy elsewhere"]
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class L3 hl
Layers 1 and 5 hold all the time; 2 to 4 react to the owner's load.
The reserve
The reserve is the larger of 512 MB and a tenth of the machine's memory. The host reporter reads /proc and reports free CPU and memory after the reserve, so the control plane stops sending new work as soon as the owner's usage grows. On Windows the agent runs inside WSL and sees only that virtual machine, so the desktop app also hands in the whole computer's numbers, and free capacity respects both.
Sources:
Eviction
guardOwner runs on each host reading. If available memory is below the reserve and the last eviction was more than 10 seconds ago, it picks the newest running shared copy, removes its container, marks it evicted with the reason, and kicks a sync so the control plane hears now. The scheduler then treats the copy like any lost one and places it on another machine.
Sources:
Check yourself
Why evict the newest copy rather than the biggest one?
The newest copy has served the least, so moving it loses the least warm state, and it is the one the system most recently judged "fits here". It is also a simple, predictable rule.
4.3 · Agent
Health checks
Relevant source files
A container that runs is not always an app that works. It can hang, or answer every request with an error, while the machine looks fine. So the agent checks each service copy itself, and only a healthy primary gets a tunnel.
The problem
Send visitors only to copies that really answer, and replace copies that stop answering.
The idea
Check from the machine, where the copy is, on a schedule the app's owner can tune; restart a few times, then give up and let the control plane move it.
What it costs
A check adds a little load to every app, and an app that is slow to load needs a longer grace time.
States
stateDiagram-v2
[*] --> starting: container started or restarted
starting --> healthy: a check passes
healthy --> unhealthy: failures in a row
starting --> unhealthy: failures after the grace time
unhealthy --> starting: restart (at most 3)
unhealthy --> failed: still failing after the 3rd restart
failed --> [*]: the control plane moves the copy
Only healthy opens a primary's tunnel. Standbys are checked too, so a promotion never lands on a broken copy.
Sources:
The two kinds of check
Type
Passes when
Detail
http
GET path answers 2xx or 3xx within the timeout
No redirects followed, no keep-alive
tcp
The port accepts a connection and keeps it open for 300 ms, or sends something
Connecting alone is not enough: Docker's port proxy accepts every connection and closes it at once when nothing listens inside
Sources:
Defaults
Setting
Default
Meaning
interval_s
10 s
Time between checks (1 to 300)
timeout_s
2 s
A check slower than this fails
failures
3
Failures in a row before unhealthy
grace_s
60 s
Before the first pass, failures in this window do not count; the app is checked every second meanwhile
Restarts before failed
3
Forgotten once the copy has been healthy for 10 minutes
A copy that fails is treated like any failed copy: another machine (not this one for 10 minutes), a standby promoted first, and the app given up after failing on 2 machines or 3 times. During a redeploy, a new copy that never passes keeps its tunnel closed, so the redeploy rolls back.
Sources:
Check yourself
Why are standbys health-checked even though they serve nobody?
A standby exists to take over. If it had quietly broken, the failover would promote a broken copy exactly when the app most needs one that works.
5 · Networking
Networking
Relevant source files
Four kinds of traffic cross this system: agents asking for work, visitors reaching apps, people opening terminals into VPSs, and apps calling outside APIs. All of them enter through the same front door, and none of them ever needs a port open on a shared machine.
The problem
Visitors must reach a container on a laptop behind a home router, whose address must stay private and which may change several times a day.
The idea
Reverse every connection: machines dial out and keep tunnels open; the gateway sends requests down them by app name.
What it costs
Every byte passes through one server: an extra hop, a bandwidth bill, and a single point of failure until Piko runs as a cluster.
All the paths
flowchart TB
VIS(["Visitor"]) --> CF["Cloudflare, then nginx :443"]
AG["Agent on a shared machine"] -- "sync and agent tunnel" --> CF
AG -- "app tunnels" --> CF
APP["App container<br>127.0.0.1 only"] -. "egress calls" .-> CF
AG --- APP
CF -- "dashboard, API,<br>sync, agent tunnel" --> CP["Control plane :8080"]
CF -- "*.apps names" --> PX["Piko proxy :8100"]
CF -- "tunnel upstream" --> PU["Piko upstream :8001"]
PX --> PU
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class AG hl
Every arrow from the machine points outwards. Piko's upstream path is where tunnels arrive; its proxy path is where visitors' requests leave.
Sources:
Timeouts along the way
Each hop has its own idea of how long a request may take. Most surprises in this area came from one of these disagreeing with another.
Where
Limit
Why it matters
Cloudflare Free, proxied request
100 s, 100 MB body
Large uploads (image layers) cannot go through it
nginx, agent sync
90 s
Longer than the 35 s the agent waits and the 10 s hold
Piko is an open-source reverse proxy built for upstreams that dial out. The agent embeds Piko's Go client: for every primary copy it serves, it keeps one outgoing WebSocket to Piko, and Piko sends visitors' requests down it as streams.
The problem
A machine that left the pool, or ignores the control plane, must not keep serving an app. But visitors must never see a gap while tunnels change.
The idea
Short tokens (10 minutes) that list the apps a machine may serve, and a make-before-break renewal well before each token expires.
What it costs
A renewal every 6 minutes per app, and a request still running when a drain ends is cut.
A visitor's request
%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
participant V as Visitor
participant N as nginx
participant P as Piko
participant A as Agent
participant C as Container
V->>N: GET https://shop.apps-domain/cart
N->>P: proxy port, endpoint shop
P->>A: new stream on the tunnel for shop
A->>C: forward to 127.0.0.1 host port
C-->>A: response
A-->>P: response on the stream
P-->>V: response
The machine's address never appears. If several machines serve shop (active mode, or a rollout), Piko spreads requests over their tunnels.
Sources:
Who may serve an app
Every sync answer carries a fresh Piko token: a JWT signed with the secret Piko shares with the control plane, listing the endpoints this machine may serve, valid for 10 minutes. Piko checks it when a tunnel connects. A machine whose copies moved away simply stops receiving tokens that name those apps.
Sources:
Renewal, make before break
Piko checks a token only when a tunnel connects, and closes the connection itself when that token expires. So a tunnel cannot be refreshed in place; it must be replaced. With less than 4 minutes left on its token, the agent replaces it:
%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
participant A as Agent
participant P as Piko
Note over A,P: old tunnel serving, token has 4 minutes left
A->>P: connect a new tunnel with the fresh token
Note over P: requests now alternate between both tunnels
A->>A: old tunnel: wait until no request is in flight (30 s at most)
A->>P: Shutdown the old connection
Note over P: old connection leaves the list about 0.25 s later
No gap and no failed request: 196 requests across renewals in production all succeeded.
Why Shutdown, not Close
The first version used Piko's Close. It turned out to be only a yamux GoAway: the connection stayed on Piko's list, and the next request sent to it failed with a 502, once per app per renewal. The fix was to take streams with Upstream.Listen and forward them in the agent's own code, so it can call the listener's Shutdown after a drain. A copy that is no longer wanted gets the same treatment with a shorter drain (1.5 s), shorter than the 2 s the reconciler waits before removing its container.
Sources:
When tunnels open and close
Event
Tunnel
Primary copy running and its health check passes
Opens
Standby copy
None: warm, but invisible to visitors
Copy unhealthy, or no longer wanted
Closes after a short drain
No successful sync for 60 s
All close: the machine may have been replaced
Piko loses every tunnel of a silent machine for 15 s
The control plane declares the machine down at once
Sources:
Check yourself
Why not use long-lived Piko tokens and avoid renewals altogether?
The short token is what stops a machine that has left the pool, or that ignores the control plane, from serving an app for long afterwards. Longer tokens trade fewer renewals for a longer window of stale serving.
5.2 · Networking
Custom domains
Relevant source files
An app can be served at its owner's own name, such as shop.example.com. The owner proves they control the name with a DNS record, points it at the platform, and the control plane serves it over HTTPS with its own Let's Encrypt certificate.
The problem
Anyone can type any domain into a form. The platform must serve a name only for the person who controls it, and only one person at a time.
The idea
A claim per name, proven by a TXT record only the name's owner can create, with one verified owner enforced by a unique index.
What it costs
The control plane needs its own HTTPS listener. Off unless DOMAINS_TARGET is set; not switched on in production yet.
Proving a name
%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
actor O as App owner
participant CP as Control plane
participant DNS as Their DNS provider
O->>CP: POST /api/v1/deployments/shop/domains (shop.example.com)
CP-->>O: token, and the records to add
O->>DNS: TXT _meshly-verify.shop.example.com = token
O->>DNS: CNAME shop.example.com to the platform's target
loop domainLoop, every minute
CP->>DNS: look up the TXT record
end
CP->>CP: proof found: status verified (unique per host)
A root domain cannot have a CNAME, so the page shows A records with the target's addresses instead.
Sources:
Claim states
stateDiagram-v2
[*] --> pending: claim added
pending --> verified: TXT proof found
pending --> taken: another account verified first
verified --> pending: 3 definite failures in a row
taken --> pending: the verified claim was removed
A pending claim never blocks anyone. Only one claim per host can be verified, because of the partial unique index domains_verified_host.
Claim
Checked
pending, younger than an hour
every minute
pending, younger than a day
every 10 minutes
pending, older
hourly
verified or taken
every 6 hours
Only a definite answer counts against a verified claim (the record is not found, or does not match). Timeouts and temporary DNS errors change nothing.
Sources:
Serving a verified name
flowchart TB
V(["Visitor: https://shop.example.com"]) --> L["Control plane's own HTTPS listener<br>DOMAINS_HTTPS"]
L --> C{"certPolicy:<br>verified host of a live service?"}
C -- "no" --> X["refused: no certificate is ever issued"]
C -- "yes" --> R["domainRoute: host to app<br>cached 30 s"]
R --> P["Piko proxy<br>header x-piko-endpoint: shop"]
P --> T["the app's tunnel, as for any visitor"]
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class C hl
Certificates come from Let's Encrypt through autocert and are cached in the certs table, sealed with SECRETS_KEY. A plain HTTP listener answers HTTP-01 challenges and redirects everything else.
The policy check matters: without it, anyone could point a random name at the platform and make it request certificates until Let's Encrypt's rate limits blocked the real ones.
Sources:
Check yourself
Why a TXT record at a special name, rather than just checking that the CNAME points at us?
Pointing a name at the platform proves nothing about who asked for it: an old CNAME could still point here after a domain changed hands. The TXT token is unique to one claim, so only the person who made that claim, and controls the DNS, can create it.
5.3 · Networking
Secrets and egress
Relevant source files
On a stranger's laptop, the owner is root and can read anything a container holds. So in the community cloud, an app's API keys never leave the control plane. The app gets a stand-in, and the control plane swaps in the real key on the way out.
The problem
An app needs its API key to call an outside service, and the machine it runs on cannot be trusted with that key.
The idea
Give the app a stand-in key that works only through our proxy, only for its one host, only while its copy runs.
What it costs
Only keys sent in an HTTP header or query work this way. Signing keys and database passwords still reach the machine, with a warning.
Where each secret goes
Organization cloud
Community cloud
Plain variable
in the container
in the container
API key (a secret with a host)
in the container
stays in the control plane; the container gets ccs_…
Secret the app uses itself (signing key, database password)
in the container
in the container, and the dashboard warns that the machine's owner can read it
Every secret is sealed at rest with AES-256-GCM under SECRETS_KEY and is write-only: the dashboard shows its name and where it goes, never its value. Secrets never go to a build, because an image keeps whatever it was built with.
Sources:
A call through egress
%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
participant App as Community app
participant E as handleEgress
participant DB as PostgreSQL
participant API as api.openai.com
App->>E: POST /egress/v1/chat, Bearer ccs_ stand-in
E->>E: check the signature and the rate (10 calls a second per copy)
E->>DB: is this copy still running? which secret and host?
E->>E: swap the stand-in for the real key
E->>API: POST https://api.openai.com/v1/chat, Bearer sk-
API-->>E: answer, streamed
E-->>App: answer, streamed
The host comes from the secret, never from the request, and the dialer refuses any address that is not public.
The stand-in can sit wherever the key would: a Bearer token, Basic auth, a header of the app's own, or the query string.
A secret can name the variable that holds the API's address (for example OPENAI_BASE_URL). In the community cloud it is set to the egress address, so SDKs that read it need no code change.
The connection refuses localhost, private networks and the cloud's metadata service, so the proxy cannot be turned against the platform's own servers.
A key kept in the control plane can be changed in place, with no restart of the app.
Sources:
Check yourself
The laptop's owner copies the stand-in key out of the container. What can they do with it?
Very little. It works only through the platform's proxy, only for that one host, only at 10 calls a second, and only while that copy runs. When the copy stops or moves, the stand-in dies. They never see the real key.
5.4 · Networking
Agent tunnel and sessions
Relevant source files
A VPS needs a way in: a terminal in the browser and real SSH from a laptop. Piko tunnels cannot carry that, because they are replaced every 6 minutes and every SSH session would be cut. So each agent that keeps a VPS holds a second, longer-lived connection: its own tunnel to the control plane.
The problem
Interactive sessions last hours, the machine is behind NAT, and Cloudflare only proxies HTTPS ports.
The idea
One outgoing WebSocket per agent with yamux on top, authenticated by the machine's own token, which never expires; the control plane opens a stream per session.
What it costs
An agent reconnect, or a control plane restart, ends open sessions. People reconnect; tmux helps.
Two kinds of session
flowchart TB
BR["Browser terminal<br>xterm.js"] -- "WebSocket<br>/api/v1/vps/NAME/terminal" --> CP["Control plane"]
SSH["ssh, scp, VS Code<br>via ccloud ssh ProxyCommand"] -- "WebSocket<br>/api/v1/vps/NAME/ssh" --> CP
CP -- "new yamux stream<br>op pty or ssh" --> AG["Agent<br>GET /agent/v1/tunnel"]
AG -- "pty: podman exec on a terminal" --> V["VPS container"]
AG -- "ssh: agent is the SSH server,<br>then podman exec" --> V
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class AG hl
The VPS image needs no SSH server: the agent is the SSH server, and every session is podman exec into the container.
Sources:
The tunnel itself
The agent dials GET /agent/v1/tunnel with its machine token while it keeps a VPS and is not paused, and reconnects with backoff from 1 second to 1 minute.
The control plane accepts it only with no Origin header, so a web page cannot open it with someone's cookie. A machine taken out of the pool is cut off within a minute.
yamux pings every 20 seconds and gives a stuck write 30 seconds; its 10 second default would end the session during a burst on a home uplink.
Each stream starts with one JSON line: {"op": "pty" or "ssh", "vps": id}. The agent answers {"ok": true} or an error within 20 seconds.
Sources:
An SSH session, end to end
%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
participant U as ssh on a laptop
participant C as ccloud ssh
participant CP as Control plane
participant A as Agent
participant V as VPS container
U->>C: ProxyCommand: ccloud ssh NAME
C->>CP: WebSocket /api/v1/vps/NAME/ssh (session)
CP->>A: stream on the agent tunnel, op ssh
A-->>C: SSH handshake, the VPS's own host key (ccloud pins it)
U->>A: public key login: only the owner's SSH keys
A->>V: podman exec, with a wrapper that reports the exit code on fd 3
V-->>U: shell, command, scp or sftp
SSH runs end to end between the laptop and the agent; the control plane only relays bytes. No port forwarding is allowed.
The stdin lesson
podman exec does not return while its stdin is open, even after the program inside has ended, and SSH clients and browsers keep stdin open. Sessions never ended. Now every program runs under a small wrapper that writes its exit code to file descriptor 3 the moment it ends; the agent then closes stdin or hangs up the terminal itself. When a session ends, its shell inside the container gets SIGHUP, as sshd does, so closed browser tabs leave no shells behind.
Sources:
Check yourself
Why is a WebSocket over 443 used for SSH instead of a normal SSH port?
Cloudflare Free proxies only 443 and a few HTTPS ports, and exposing a raw port would mean revealing the server's address. Wrapping SSH in a WebSocket lets it ride the same front door as everything else.
6 · Build and deploy
Build and deploy
Relevant source files
There are three ways to get an app running, and they all end in the same place: a workload pinned to one image digest, placed by the scheduler. The difference is how much of the work the platform does before that.
The problem
People arrive with a ready image, with one repository, or with a repository that holds several apps that must find each other.
The idea
Turn every path into the same thing as early as possible: an image digest the scheduler can place.
What it costs
Building from source needs a trusted builder machine to be running.
Three paths, one destination
flowchart TB
I1["Deploy an image<br>docker.io/alice/api:1.4"] --> PIN["imageDigest: ask its registry<br>which build the tag points to"]
I2["Deploy from a repository<br>source: repo, branch, folder"] --> Q["queue a build of the newest commit"]
I3["Import a repository<br>several apps at once"] --> DET["detect apps, then POST /api/v1/imports<br>all or nothing"]
DET --> Q
Q --> BLD["a builder builds and pushes<br>to the platform's registry"]
BLD --> PIN2["the image's digest"]
PIN --> W["workload pinned to a digest"]
PIN2 --> W
W --> S["scheduler places copies"]
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class W hl
Whatever the path, machines only ever see image@sha256:….
Sources:
What a deploy request says
Field
Meaning
image or source
A container image, or {repo, branch, folder, build_cmd, start_cmd}
size
small, medium, large, or custom CPU and memory
mode, replicas
spot, standby, or active with 2 to 5 copies
port
Where the app listens; $PORT is 8080 unless set
variables, secrets
Plain values, and secrets that go where the cloud allows (Secrets and egress)
health
An HTTP path or a TCP check, with timings (Health checks)
Before anything is built, the import page reads the repository and shows the apps in it: framework, commands, port, the variables they read, what they depend on, and what would go wrong. Nothing in the repository is ever run to find this out.
The problem
The deploy form used to ask for folder, commands and port by hand, and a wrong guess was discovered only after a build ran and failed.
The idea
Fixed rules, run on the files, that predict what the real builder (Railpack) will do. Read-only, cached by commit.
What it costs
A second source of truth beside Railpack, which can disagree with it, and a rule set to re-check when Railpack changes.
How a repository is read
flowchart TB
U["GET /api/v1/repositories/inspect"] --> C{"result cached<br>for this commit and rules version?"}
C -- "yes" --> OUT["apps, frameworks, commands,<br>port, variables, warnings"]
C -- "no" --> L["list the commit's files: one API call"]
L --> M["read manifests first,<br>then each app's source"]
M --> R["rules in Railpack's order of languages<br>group by framework, not folder name"]
R --> OUT
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class R hl
Up to 60 files per app and 400 per repository, config-like names first. If time runs out, apps are still shown and the result is marked incomplete.
A variable's value never leaves internal/detect, only its name. From a committed .env only names are kept, with a warning.
An address between two apps is filled in only when the code shows what shape it takes, because a wrong value would be baked in by the build.
TestMatchesRailpack holds what Railpack actually chose for six generated apps, so the rules cannot drift from the builder unnoticed.
Limits keep one person from exhausting GitHub's allowance: 12 looks a minute per person after a burst of 10, 4 repositories read at once, 25 seconds per look.
Sources:
Private repositories: the GitHub App
People connect GitHub through a GitHub App with read-only access to the repositories they pick. What they may see is always GitHub's answer to their own token at that moment; the platform stores no list of who can read what. A builder never gets a token at all, only a link to one commit's archive that lasts 5 minutes.
Not connected
Connected
Reads with
The server's token, public repositories only
Their installation, for private repositories
Gets
Public repositories by pasting the address
A repository list, private repositories, deploy on push
Sources:
Importing several apps together
POST /api/v1/imports checks every app first, then saves them all in one transaction: if any part fails, nothing is created, not a workload, a secret or a build. Apps from one branch are pinned to the same commit, each app's address is filled into the others' variables where the code shows its shape, and all get the same project label.
Sources:
Check yourself
Why are predicted commands not sent to the build unless the owner edits them?
Railpack is the one that really builds. If our prediction were sent as an override and was slightly wrong, it would break a build Railpack would have got right. Predictions are for showing; Railpack decides.
6.2 · Build and deploy
Builders and the registry
Relevant source files
Source code becomes an image on a builder: a machine the platform admin trusts as much as the platform itself, which runs no apps. The image goes to the platform's own registry, behind a gate in the control plane that decides every push and pull.
The problem
Building on the machines that run apps fails for slow downloads, small memory, inconsistent results and trust: a machine could build anything and claim it was the app.
The idea
Dedicated builders, one build at a time, and short-lived credentials that work for one app's repository only while its build runs.
What it costs
No builder running means no new deploys from source. Every image byte passes through the control plane and sits on its disk, and old images are not cleaned up yet.
A build, end to end
%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
participant CP as Control plane
participant B as Builder's agent
participant GH as GitHub archive
participant R as Registry gate /v2/
CP->>CP: queueBuild: pin the branch's newest commit
CP-->>B: sync answer lists the build, with a one-time token
B->>GH: download the commit's .tar.gz
B->>B: Dockerfile? build --pull. Otherwise railpack build on BuildKit
B->>R: push apps/workload-id, user build, its token
R->>R: only this app's repository, only while the build runs
B->>CP: sync: done, with the digest
CP->>CP: rollOutBuild: copies move to the new digest one at a time
Builds time out after 15 minutes. A lost builder's build goes to another. Secrets never go to a build; plain variables do, as build arguments.
Sources:
Why Railpack
Most repositories have no Dockerfile. The team built seven fresh generator apps (Express, NestJS, Next.js, Vite, Django, Rails, Laravel) with three builders and default settings only. Railpack ran 5 of 7 out of the box with the smallest images (a Vite app at 154 MB); Heroku's builder ran 4; Paketo ran 2, with images around 1 GB. Small images matter when laptops download them over home internet. An app's own Dockerfile always wins.
Sources:
The registry gate
The registry program (CNCF Distribution) listens on 127.0.0.1 and checks nobody. The control plane stands in front of it at /v2/ and decides every request from what it already knows:
Who
User
Password
May
A machine
machine
Its machine token
Pull the apps it has a copy of (assigned, starting or running)
A build
build
The build's one-time token
Push and pull its own app's repository, while the build runs
Nobody lists the registry or deletes from it, and a push's request to copy a layer from another app's repository becomes a plain upload, so no token can reach into another app. The build token is an HMAC of the build; only its hash is stored.
Sources:
Builders are special machines
The platform admin adds a builder in Admin, Builders: a one-time join key, and the machine that joins with it (sudo ccloud join --builder --key …) has the role builder from its first sync. The scheduler skips builders for apps, so no app can tamper with a build, and builders are nobody's shared capacity.
Sources:
Check yourself
Why is a cheating builder so dangerous, and what limits the damage?
What a builder produces runs on every machine the app moves to, so a cheating builder could plant code in other people's apps. That is why builders are trusted like the platform. Each build's token still works only for its own app and only while it runs, so a leaked token is worth little.
6.3 · Build and deploy
Jobs and cron
Relevant source files
A job is a container that runs to completion: a backup, a report, a batch. Jobs suit volunteer machines well, because a run whose machine goes away can simply run again somewhere else.
The problem
Scheduled work must run on time, exactly as often as intended, on machines that may vanish mid-run.
The idea
Each run is an ordinary workload of kind job. The scheduler, liveness and the meter already handle it; jobs only add the schedule.
What it costs
A run that loses its machine starts again from the beginning, so jobs should be safe to repeat.
Definition and runs
flowchart TB
J["Job definition<br>image, command, size, schedule, rules"] --> CL["cronLoop every 5 s:<br>runs that are due"]
J --> BH["Run now, by hand"]
CL --> RUN["a workload of kind job,<br>job_id set"]
BH --> RUN
RUN --> SCH["scheduler places it like any copy"]
SCH --> END{"exit code"}
END -- "0" --> OK["done, lease rounded up to a whole minute"]
END -- "error, or machine lost" --> RT["runs again, up to the job's retries"]
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class RUN hl
One code path for every container: a job's run is placed, watched and billed exactly like an app's copy.
Sources:
The rules a schedule needs
Rule
Choices
Schedule
A cron expression in a chosen time zone
Overlap
What a due run does while an earlier one is still going: skip it, allow both, or replace the old one
Retries
How many times a failed run, or one whose machine was lost, runs again
Time limit
Runs past it are ended
Missed slots
If the control plane was down when runs were due, the next tick starts one run, not one per missed time
Sources:
Check yourself
A nightly backup job missed three nights because the control plane was down. How many runs start when it comes back?
One. A backup that missed three slots needs one backup now, not three at once.
7 · VPS
VPS
Relevant source files
A VPS is a Linux machine with root (Ubuntu 24.04 or Debian 12) on one of the pool's machines. Unlike an app, it cannot simply restart somewhere else: its disk is the point. Most of the VPS code exists to keep that disk safe while its host comes and goes.
The problem
Give people a box with root and a lasting disk, on machines that sleep, lose power and leave the platform.
The idea
A system container placed once and kept on its machine, with its own disk image that is copied off the machine all the time, and a generation number that says who may write.
What it costs
On a community machine, the owner can look inside a running VPS. A container is not a VM; a microVM tier for machines with KVM is the next step.
A VPS on its machine
flowchart TB
subgraph m["The machine"]
AG["Agent: vpsManager"]
subgraph store["VPS store: XFS with reflink"]
IMG["disk.img<br>ext4, 10, 20 or 40 GB"]
LOG["write log"]
end
DEV["loop device, or NBD served by the agent"]
CT["Podman system container<br>fyp-vps-id, its own network"]
end
HUB["Control plane: the hub"]
OBJ[("Folder or S3 bucket<br>sealed packs and batches")]
IMG --> DEV
DEV --> CT
AG --> CT
AG -- "photos and log batches, sealed" --> HUB
HUB --> OBJ
USR(["Owner: browser terminal or ssh"]) --> HUB
HUB -- "agent tunnel" --> AG
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class HUB hl
The disk image is the container's root filesystem (podman create --rootfs). Its size is also its quota.
Sources:
How the container is made
The same limits as apps: CPU, memory equal to swap, --cpu-shares 2, --oom-score-adj 1000, no-new-privileges, and 4096 processes.
A network of its own (fyp-vpsnet-<id>, isolated) and, where the host allows, --userns=auto, so root inside is nobody outside. If the runtime refuses that, the agent drops it from then on and logs why.
Labelled fyp.vps, never fyp.machine, so the app reconciler, the owner guard and leaving never touch it.
Sources:
Three rules the agent keeps
It removes a VPS (container, disk, network, host key) only when the control plane lists it in vps_remove. Never because it is missing, never on an unknown state.
A VPS the desired state no longer lists is stopped, not removed, and only once a fresh desired state came from the control plane since the agent started.
While the owner pauses sharing, VPSs are stopped; they start again on resume.
And one safety reflex: with no successful sync for 60 seconds, the agent pauses its VPSs, so a machine cut off from the platform stops changing a disk that may be restarting elsewhere.
Found in review, 20 Nov 2026
That pause only counts from the last successful sync. After an agent restart there has been none (lastSyncOK is 0 and is not saved), so an agent that restarts while it cannot reach the control plane starts its VPSs from state.json and never pauses them. If the VPS was moved meanwhile, two copies run at once. The hub stays consistent thanks to generations, but programs inside run twice. A fix: count the fence from the agent's start when there was no sync yet, and start no VPS from saved state before the first fresh answer.
Sources:
Placement, power and billing
A VPS is its own resource in the vps table, not a workload: the scheduler and liveness never treat it as a replaceable copy. placeOneVPS places it once, on a machine in its cloud that reports a VPS store with room. Its generation goes up each time it is placed. A running VPS is billed per minute like a primary copy; a stopped one is free.
Every minute or so, a VPS's machine takes a photo of its disk and sends only the blocks that changed since the last photo the hub accepted, sealed with the VPS's own key. If the machine disappears, the VPS can restart elsewhere from the newest photo.
The problem
A disk must outlive its machine, but home upload is slow (5 to 30 Mbps) and must never slow the VPS down.
The idea
Run at local disk speed, and send changes in the background. A slow link only makes copies older; it never slows the disk.
What it costs
Without the write log, up to about a minute of changes can be lost when a machine dies. Machines need xfsprogs.
Taking a photo
%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
participant A as Agent (takeCopy)
participant FS as ext4 in disk.img
participant H as Hub (control plane)
participant O as Object store
A->>FS: freeze (FIFREEZE)
A->>A: clone disk.img to next.img (FICLONE, tens of ms)
A->>FS: thaw
A->>A: compare next.img with base.img: extents, then 64 KiB blocks
loop changed blocks, 512 per pack
A->>A: flate, then AES-256-GCM with the VPS key
A->>H: PUT /agent/v1/vps/id/packs/seq-n?generation=g
H->>O: store the sealed pack
end
A->>H: POST /agent/v1/vps/id/copies (generation, seq, base, packs)
H->>H: Postgres transaction: right generation? builds on the newest?
H-->>A: copy_seq, or 409 fenced or base
A->>A: only now next.img becomes base.img
With the write log on, photos are taken between two writes without a freeze, and go every 15 minutes instead of every minute.
The last step is the subtle one. base.img is replaced only after the hub accepted the copy, so a failure never skips a change: the next round compares with the same base again.
Sources:
Postgres decides which copy counts
The hub accepts a copy only if both checks pass, inside one transaction:
Check
Refused with
Why
The machine holds the VPS at this generation
409 fenced
An old machine that comes back must never add to the chain
base is the hub's newest copy and seq the next one
409 base
Copies form a chain; a gap would corrupt the restore
On fenced the agent stops copying that VPS. On base, or when it has no base, it sends a full copy (every non-zero block), which starts a new chain. Because the check lives in Postgres and not in the store, any S3-compatible bucket works.
Sources:
Keeping the chain short
Every 20 copies, checkpoint merges the maps into one full map, so a restore reads a few maps rather than hundreds. Packs that no copy needs any more are deleted after 6 hours, so a restore already in progress still finds them. Objects are stored as vps/<id>/<generation>/<copy>-<n>.pack.
Sources:
Restoring
A machine given a VPS with copy_seq > 0 and no disk for it restores: it reads the map, makes a sparse image, writes every block from its pack, keeps a clone as its new base, replays any write-log batches after the copy, and then sends only what changes. A failed restore is tried again; the VPS is never created empty.
Sources:
Found in review, 20 Nov 2026
The bucket alone cannot restore anything. S3 holds only sealed block bytes. Which block goes where (vps_photos.manifest), the copy chain and every VPS's key (sealed with SECRETS_KEY) live only in Postgres on the main server, which has no backups yet. Off-server backups of the database and the key are what make "your data is safe" true.
Packs are bound only to their VPS. The sealing's additional data is meshly-vps:<id>, while write-log batches also bind their range. Someone who can write the bucket could swap one pack of a VPS for another of the same size undetected. Binding the generation, copy and pack number closes that.
The hub buffers whole uploads. Packs (up to 40 MiB) and batches (up to 48 MB) are read fully into memory, with no limit on how many at once, on a 2 GB server.
Sources:
Check yourself
Why freeze the filesystem before cloning, if the clone takes only milliseconds?
Without a freeze the clone could catch ext4 halfway through a change, with its journal and data out of step. A freeze makes the filesystem consistent on disk for that instant; cloning keeps the freeze short.
7.2 · VPS
The write log
Relevant source files
Photos every minute can lose a minute of work. People asked for no data loss. On home internet the honest version is: lose only the last few seconds of saved writes, at no cost to disk speed. The write log does that by putting the agent itself in the disk's path.
The problem
Capture every write in order, on WSL2's kernel too, where DRBD, ublk and dm-era are not available.
The idea
The agent serves the VPS's disk as a network block device (NBD) to its own kernel, logs every write before applying it, and ships what each flush covered every 2 seconds.
What it costs
Our process is in the disk's path, the image is written twice (about 8% of random-write speed), and batches double the upload for data changed between photos.
The path of one write
flowchart TB
P["A program in the VPS writes and calls fsync"] --> K["Kernel: /dev/nbdX<br>backend meshly-vps:id"]
K --> S["Agent's NBD server<br>internal/nbd"]
S --> L["1. append a record to the write log<br>CRC32C, in arrival order"]
L --> I["2. write the bytes to disk.img"]
S --> F["Flush: fsync disk.img<br>the write is now safe on the laptop"]
F --> B["every 2 s or 4 MiB:<br>records the flush covered, merged per 4 KiB block"]
B --> H["Hub: POST /agent/v1/vps/id/log<br>sealed batch, in order"]
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class S hl
Only records a completed flush covered are sent, so the hub never holds a write the laptop could still lose. The laptop's disk always wins.
Sources:
The pieces
Piece
What it does
internal/nbd
A userspace NBD server, and device setup over generic netlink. The kernel holds I/O for 10 minutes if the server goes away (dead_conn_timeout) and resends unanswered requests after 90 s
internal/wlog
The log: 64 MiB segments, 8-byte aligned records with CRC32C. A record the file ends inside is a torn tail and is cut; a damaged whole record is never skipped
wlog.Coalesce
Turns records into a batch: zero ranges for trims, then each block's newest bytes, so a block written ten times goes once
wlog sealing
AES-256-GCM with the VPS key, bound to the VPS and the batch's record range
cp/vpslog.go
The hub takes a batch only from the current generation, only on top of a copy, and only if it follows its newest record; a retry is harmless
Sources:
When the log cannot go on
There is exactly one fallback: start the log over and send a photo marked log_reset. The hub then forgets any batch past that photo in the same transaction. It happens when:
the log passed its ceiling (the smaller of 8 GiB and a tenth of the store's free room) or its oldest record is 6 hours old;
the log is damaged, or a log write failed;
the computer restarted without stopping the VPS cleanly (a new boot id): the log was not fsynced, so after a power cut it may lack records the image has;
the hub misses part of the log (gap).
Sources:
When the upload cannot keep up
Each VPS has a lag budget: 30 seconds ("protect my data"), 300 seconds by default, or 0 ("keep me fast"). Once the backlog would take more than 70% of the budget to send, the disk's writes are slowed to 80% of what the upload carries. With 0 it is never slowed, and falls back to photos at the ceiling. The dashboard shows each VPS as Live or how far behind it is.
Sources:
Surviving an agent restart
When the agent stops, it leaves the NBD devices attached. The kernel holds their I/O, and the next agent takes them over (NBD_CMD_RECONFIGURE). The VPS never notices. The agent marks itself PR_SET_IO_FLUSHER and caps each device's dirty pages at 256 MB, against the deadlock of a disk served from the same machine. A server that stops on its own is taken over again; after three failures the VPS is stopped rather than left on a disk nobody serves.
Sources:
Check yourself
Why is the log itself not fsynced on every flush?
It measured 2.4 ms rising to 8.8 ms per fsync. Instead the image is fsynced on every flush (that makes an acknowledged write safe on the laptop), and an unclean restart throws the log away and sends a photo. Safety holds; speed is kept.
7.3 · VPS
Moving a VPS
Relevant source files
Moving a VPS is risky in a way moving an app is not: the old machine's disk may hold writes nobody else has. The rules on this page all answer one question: when two disks of the same VPS exist, which one is the newest?
The problem
An early version moved a VPS off an offline laptop to nowhere, then would not let it go back. A naive fix could start from an old disk and silently lose newer work.
The idea
Move only to a machine with room, in one step. Remember which placement saved last (written_gen), and let only that disk continue.
What it costs
With no other machine, the VPS is down until its machine returns. Once moved, it stays, even on a machine that is often offline.
When a VPS moves
flowchart TB
O["Its machine is offline past move_after_min<br>default 5, 0 = wait"] --> R{"another online machine in its cloud<br>with room for disk, CPU and memory?"}
R -- "no" --> H["held: it waits, and continues there<br>from its own disk, losing nothing"]
R -- "yes" --> MV["moveVPS: new generation,<br>placed on that machine in one step"]
MV --> RS["the new machine restores the newest copy,<br>then the log batches after it"]
MV --> LO["the old disk becomes a leftover"]
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class R hl
A move is decided together with its destination. The incident of 2026-11-20 is why: a single-machine test moved a VPS to nowhere.
Sources:
The newest disk wins
written_gen is the placement whose machine sent the newest accepted copy or log batch. It changes only when the hub accepts one, and only the current placement can send, so it only ever goes up. The disk of that placement holds everything the copies hold, and maybe more.
When a machine finds a disk for a VPS it is told to run, diskVerdict decides:
The disk belongs to
Verdict
This placement
Used, as always
The placement named adopt_gen (that is, written_gen)
Continued from: it becomes this placement's, and its copies and log go on
Any other placement, or unknown while copies exist
Set aside as <id>.old-g<gen>-<time> for a day, and the newest copy is restored
What happened
written_gen
Laptop 1's old disk
When the VPS returns to laptop 1
Laptop 1 saved last, then went offline; laptop 2 restored but saved nothing
1
the newest, plus what it had not sent
continues from its own disk: nothing lost
Laptop 2 ran it and saved its work
2
older than the copies
set aside; restored from the newest copy with laptop 2's work
Copies and batches go to the hub only from the placement's own disk, so an older disk can never be sent as the newest. A leftover's one-day clock starts only once a newer placement has saved, because until then it may be the newest disk.
Sources:
Found in review, 20 Nov 2026
Decision 0023 says a restore "can lose at most the seconds the old placement never sent". It is whatever that placement never sent: seconds while its upload keeps up, up to its lag budget (300 s by default) plus up to 60 s before the fence when it does not, and longer with a budget of 0.
A planned move: drain first
An owner can move a VPS whose machine is online, for example off a laptop that is online only 62% of the time. Nothing is lost, because the old machine finishes first:
%%{init: {"sequence": {"wrap": true, "width": 160, "actorMargin": 36, "messageMargin": 30, "noteMargin": 8, "boxMargin": 8}}}%%
sequenceDiagram
actor U as Owner
participant CP as Control plane
participant A1 as Old machine
participant A2 as New machine
U->>CP: POST /api/v1/vps/NAME/move (to laptop 2)
CP-->>A1: spec says drain
A1->>A1: stop the VPS, close its disk cleanly
A1->>CP: one last copy (starts the hub's log over)
A1->>CP: report drained, with that copy's number
CP->>CP: still the newest? then move: new generation on laptop 2
CP-->>A2: spec with the VPS
A2->>CP: restore that copy, start the VPS
The VPS is down for the stop, the last copy and the restore: a few minutes for a small disk. A move not done within an hour, or cancelled, starts the VPS again where it was.
The page suggests a move when the VPS's machine has been online less than 80% of its life and another machine with room has been online at least 15 points more.
Sources:
Check yourself
Why not let a VPS go back to its old machine whenever that machine returns?
If another machine ran the VPS in between and saved work, the old disk is older than that work. Starting from it would lose the work without anyone noticing. Only the disk of written_gen may continue.
Why does a planned move send one last copy after stopping the VPS?
A stopped disk holds every write. A copy taken then holds everything the VPS ever saved, so the new machine starts from exactly the same disk.
8 · Observability
Observability
Relevant source files
Metrics and logs ride on the same sync request as everything else. Nothing on a machine opens another connection to send them, so they work wherever the machine can reach the control plane.
The problem
People need graphs and logs of apps that run on machines they cannot reach, kept cheaply on a small server.
The idea
Piggyback samples and log lines on every sync, store numbers and text in two small purpose-built stores, and keep a short tail in Postgres as a fallback.
What it costs
Seven days of searchable logs, at most 2 GiB. A machine that cannot sync cannot send its last words until it can.
Where each signal goes
flowchart TB
subgraph agent["Agent"]
S["metricsLoop: a sample every 5 s<br>device, platform share, each container"]
LG["log lines since the last answered sync<br>podman logs --timestamps"]
end
SY["POST /agent/v1/sync"]
S --> SY
LG --> SY
SY --> MV["metrics.add: VictoriaMetrics<br>SQLite when it is absent"]
SY --> FL["freshLines: drop lines at or before<br>the newest already kept"]
FL --> TL["Postgres: each copy's latest lines<br>the live view"]
FL --> VL["VictoriaLogs: every line, 7 days<br>the searchable history"]
MV --> DASH["Dashboard graphs and trends"]
TL --> DASH
VL --> DASH
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class SY hl
The control plane also records platform-wide figures once a minute (platformMetricsLoop).
Sources:
Following logs live
When someone opens an app's logs, the control plane marks the workload (logs_until) and holds that machine's syncs for 1 second instead of 10. Each answer brings the next sync a second later, carrying the new lines. Lines in VictoriaLogs carry the app's id, not its name, because a deleted app's name can be taken again but its id never is.
Sources:
What the dashboard shows
View
Endpoint
Data
App metrics
GET /api/v1/deployments/{name}/metrics
CPU and memory of each copy over a time window
App logs, live and history
…/logs, …/logs/history
Postgres tail; VictoriaLogs search
Machine metrics
GET /api/v1/machines/{id}/metrics
The whole device and the platform's share of it
Failovers
GET /api/v1/failovers
From, to, how, and seconds until serving again
Admin overview and trends
GET /api/v1/admin/overview, …/trends
Machines, apps, usage across the platform
Sources:
9 · Clients
Clients
Relevant source files
Four programs talk to the platform for people: the web dashboard for everyone, the desktop app for laptop owners, ccloud for servers and VPS logins, and fyp for admins. They all use the same API; none holds logic the control plane does not also enforce.
The problem
Very different people (consumers, laptop owners, server admins, platform admins) need different tools for the same platform.
The idea
Thin clients over one API, with every rule enforced on the server or by the agent.
What it costs
The dashboard's build is committed into the Go tree, so a change to web/ must be rebuilt and committed with it.
Who uses what
flowchart LR
C(["Consumer"]) --> D["Dashboard<br>web/, built into internal/cp/ui"]
PO(["Laptop owner"]) --> W["Desktop app, Windows<br>agent inside WSL"]
SA(["Server admin"]) --> CC["ccloud join, status,<br>limits, pause, leave"]
VO(["VPS owner"]) --> CS["ccloud login, ssh"]
PA(["Platform admin"]) --> F["fyp CLI<br>admin token"]
D --> API["/api/v1"]
W --> LA["agent's local API<br>127.0.0.1:7070"]
W --> API
CC --> LA
CC --> API
CS --> API
F --> V1["/v1"]
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class API hl
The desktop app and ccloud drive the agent through its local API; the agent enforces every limit itself.
The dashboard
A React app built with Vite in web/. Its build is embedded in the control plane binary (//go:embed all:ui) and served as a single-page app, so the platform ships as one file. Pages: landing, sign-in, deploy, import, provide (machines), organization, jobs, VPS, domains, credits, builders and admin. CI warns when the committed build is not the build of web/.
Sources:
ccloud
Command
What it does
ccloud join --server URL --key KEY
Asks how much CPU and memory to share and installs the agent as a systemd service
ccloud join --builder --key KEY
Joins a builder: checks Docker and Railpack, starts BuildKit
ccloud status, limits, pause, resume, leave, logs
Manage the local agent
ccloud login --server URL --email …
Save a session, on Linux, macOS or Windows
ccloud ssh NAME [COMMAND]
Log in to a VPS over the control plane (Agent tunnel)
ccloud ssh config NAME
A ~/.ssh/config block, so plain ssh NAME.meshly, scp, sftp and VS Code work
The control plane serves the agent and CLI builds at /download/{file}, from an allow-list of file names.
Sources:
The desktop app
A native window (WebView2 on Windows) with three views: This laptop (set it up once, share, set limits, pause, live graphs of the laptop and the platform's part), Earnings (credits earned, time in the pool), and Settings. The agent runs inside WSL. The app is only a front end: closing it changes nothing about what the laptop shares.
Sources:
fyp
The original admin CLI, from the first MVP: enrollment keys, machines, accounts, services and jobs, through /v1 with a fixed admin token. The roadmap lists closing /v1 as Phase 0 work.
Sources:
10 · Operations
Operations
Relevant source files
How the live platform is put together, how code reaches it, and how the team learns from what breaks. The authoritative runbook is docs/operations.md; this page explains its shape.
The problem
A student team must run a public service on a small cloud budget, with several people pushing code and one person deploying.
The idea
One server, every service on localhost behind nginx and Cloudflare, CI on every push, and deploys that the server pulls only after CI passed.
What it costs
A single point of failure, a single disk, and no scheduled backups yet.
What runs where
Service
Role
Listens on
meshly-cp
The control plane
127.0.0.1:8080
meshly-piko
The app gateway: proxy, upstream, admin
127.0.0.1:8100, :8001, :8002
postgresql
Records
127.0.0.1:5432
meshly-vm, meshly-vl
Metrics; logs (7 days, 2 GiB)
127.0.0.1:8428, :9428
meshly-registry
Built images
127.0.0.1:5000
nginx
TLS behind Cloudflare, routing by name
80, 443
A t3.small in Sydney (the AWS organization allows EC2 in no other region tested), a builder that sleeps when idle, and two free Oracle machines in Mumbai for community capacity.
Sources:
From a push to production
flowchart TB
PR["Pull request or push to dev or main"] --> CI["CI: worm guard first, dashboard build check,<br>gofmt, go vet, go test -race with Postgres,<br>VictoriaMetrics and VictoriaLogs, builds"]
CI -- "push, every job passed" --> REL["release job: meshly-release bundle<br>files, checksums, commit"]
REL --> TM["on the server: meshly-deploy timer, every minute<br>read-only token, nothing pushed in"]
TM --> CK{"newest push run passed,<br>for the branch tip,<br>digest and checksums match?"}
CK -- "no" --> W["wait"]
CK -- "yes" --> DP["dump the database, keep the previous build,<br>install, restart meshly-cp"]
DP --> HC{"/login answers within 30 s?"}
HC -- "yes" --> OK["live"]
HC -- "no" --> RB["put the previous build back"]
classDef hl stroke-width:1.5px
classDef warn stroke-width:1.5px
class CK hl
Automatic deploys are on dev and start paused; they are not switched on yet. Until then, deploys are done by hand with update.sh, by one person.
The server pulls rather than GitHub pushing, for two technical reasons: SSH to the server is open only to listed addresses (GitHub's runners use thousands), and a server key stored in GitHub could be read by every collaborator with write access.
Sources:
The worm guard
The repository was hit twice by a worm that hid code in .vscode/, npm scripts and build configs, and force-pushed forged copies of branches. scripts/ci-guard.sh fails on those signs and on lock-file packages from outside the npm registry or with new install scripts. CI runs it before any npm command, and so should everyone locally. It refuses to run without a real python3, after it once passed on Windows with none of its checks.
Sources:
Learning from incidents
Anything that broke and taught something gets a file in docs/incidents, with what the team believed and why it was wrong. The lessons so far, in one line each:
Incident
Lesson
Following logs closed tunnels
Timers interact: the server, not the agent, should shorten the hold
A failed request every tunnel renewal
Know exactly what a library's Close does before relying on it
App port mismatch
Traffic must go to the port the app really listens on: an owner's PORT now decides both
Restarting an agent moved its apps (open)
A restart is not leaving; only ccloud leave should say goodbye
podman exec waits for stdin
Detect a program's end yourself; do not wait for the tool
Write-log figures stopped every sync
Numbers that move on their own must never trigger early reports
A VPS moved to nowhere
Decide a move together with its destination; test with a single machine
Turning off Cloudflare's proxy broke HTTPS
The origin certificate only works behind the proxy
Sources:
11 · Known limits
Known limits
Relevant source files
A system built on other people's machines has limits that no code removes, limits the team has listed and plans to fix, and a few found in a code review. They are kept apart here, so you know whose finding each one is.
Limits no code removes
On a machine the platform does not control, the owner has root. They can read an app's files, environment and traffic on that machine, and the platform cannot reliably detect it, because the owner also controls the agent that reports. The real fix is confidential computing (AMD SEV-SNP, Intel TDX), which volunteers' laptops do not have. So machines receive built images, never repositories, and never secrets inside an image; but for Node, Python, PHP and Ruby the image still holds the source files. Code and data that must stay private belong on an organization's own servers. Machines behind NAT cannot offer a public IP.
Sources:
The team's Phase 0: safe for strangers
The roadmap lists what must be fixed before the community cloud opens to strangers. In short:
Area
Open problem
Browser
Apps share the dashboard's domain, so an app could plant cookies; framing and cross-site protections are incomplete
Billing
A machine can claim running with no tunnel and earn credits
Reports
A machine can write metric samples for another machine's copy; a forged X-Forwarded-For can be recorded as its IP
Keys
A leaked organization server key gets real secrets with no approval step
API
No rate limits outside /egress; /v1 is public behind one fixed token; names like login are not reserved
Machines
Apps on one machine share the Podman network; nothing caps an app's writable layer; an agent restart counts as leaving the pool
Operations
No database backups; no branch protection on GitHub
Sources:
Found in a code review, 20 Nov 2026
These come from a review of the VPS records against the code on dev at d0ca527. They are not yet in the team's documents.
Severity
Finding
Where
High
An agent that restarts while it cannot reach the control plane starts its VPSs from state.json and never pauses them, because the 60 s fence counts only from a successful sync. Two copies of a moved VPS can run at once
High
No grace period after a control plane restart: after an outage over 2 minutes, liveness marks machines lost before their agents reconnect (backoff up to 2 minutes), so apps and VPSs move needlessly
High
VPS copies in S3 cannot be read without Postgres (manifests, chain, keys sealed with SECRETS_KEY), and the server has no backups, so "your data is safe" depends on backups that do not exist yet
Medium
The hub reads whole packs and batches into memory with no limit on uploads at once, on a 2 GB server; all VPS traffic passes through it twice
Medium
Decision 0023 promises a restore loses "at most the seconds" never sent; with a slow upload it is up to the lag budget plus 60 s, and more with a budget of 0
Low
Copy packs are sealed with only the VPS id as additional data, so packs of one VPS could be swapped in the bucket undetected
Docs that lag the code
The roadmap's status table, last updated before mid-November, still lists health checks as having no HTTP path and custom domains as not started. On dev, the agent runs HTTP and TCP health checks (4.3) and custom domains are built and off by default (5.2).