Making Pyo highly available

High availability covers two things: your services, and Pyo itself. Your services matter most, and they don't depend on Pyo to keep running.

If Pyo goes downCopy Anchor Link

Pyo is the control planeThe part that manages everything and applies your changes. Your services don't need it to keep serving.. If it stops, your services keep serving traffic, because they run on your clusters and not inside Pyo. You can't deploy or change anything until it's back.

Your servicesCopy Anchor Link

To keep a service up through a server failure:

1. Add at least two more servers to the cluster.
2. Run two or more replicas, or turn on autoscaling with a minimum of two.
3. Keep services statelessA stateless service keeps nothing on its own disk or in memory between requests, like uploads or sessions. Any copy of it can answer, so it can move freely. where you can. Replicas spread across servers, and stateless services move to a healthy server if one fails.
4. Point your domain at more than one server, through a CDNA network of servers, like Cloudflare, that sits between visitors and your servers to speed up and protect your sites. or a load balancer. If DNS points at one server's address, traffic stops when that server does.

Volumes stay on the server that created them, so a service with a volume can't move to another server. For data that has to survive losing a server, use one of Pyo's databases with high availability, or your database's own replication.

What Pyo runs todayCopy Anchor Link

Updates don't cause downtime, because a new replica is ready before the old one stops. The limits above are about losing a whole server.

Make Pyo more resilientCopy Anchor Link

These steps remove the biggest single points of failure. Try them on a spare cluster first.

1. Run three k3s serversCopy Anchor Link

By default, a cluster has one k3s server. Three keep the cluster's control plane running if one fails. Use an odd number, and at least three.

On the first server, turn on the embedded etcdThe database that stores a Kubernetes cluster's state. With it embedded, several k3s servers share that state. datastore. Add this to /etc/rancher/k3s/config.yaml, then restart k3s:

cluster-init: true

sudo systemctl restart k3s

Read the join token from the first server:

sudo cat /var/lib/rancher/k3s/server/token

Then add two more servers, using the same settings as the first, including secrets-encryption:

curl -sfL https://get.k3s.io | K3S_TOKEN=<token> sh -s - server \
  --server https://<first-server>:6443

The command Pyo gives you for adding a server adds a worker, not a k3s server, so run these by hand. See the k3s documentation for details.

2. Use a hosted PostgreSQLCopy Anchor Link

Pyo keeps everything that matters in PostgreSQL, so this is the most important step. Move it to a hosted database with its own failoverSwitching to a standby copy automatically when the main one fails.:

1. Create the database, then take a final copy of Pyo's database:

   Copy Text

   sudo k3s kubectl -n pyo-system exec statefulset/pyo-postgres -- \
     pg_dump -U pyo pyo > pyo.sql

2. Restore pyo.sql into the new database.

3. Point Pyo at it and restart:

   Copy Text

   sudo k3s kubectl -n pyo-system patch secret pyo-postgres --type merge \
     -p '{"stringData":{"url":"postgres://user:password@host:5432/pyo"}}'
   sudo k3s kubectl -n pyo-system rollout restart deployment/pyo deployment/pyo-worker

Don't deploy while you do this, because changes made after the copy aren't moved. Updating Pyo keeps this setting.

3. Keep backups off the clusterCopy Anchor Link

Back up Pyo itself to a bucket outside your cluster, and keep your master key somewhere safe. With both, you can rebuild Pyo on new servers even if you lose the whole cluster.

What isn't highly available yetCopy Anchor Link

Even with these steps, some parts still depend on one server:

• Image registry: if its server is down, new builds can't be stored. Servers can't pull images they don't already have.
• ClickHouse: there's no replica. Losing its server loses observability history, but nothing else.

For most teams, this is a good trade-off. Your services stay up, and Pyo can be rebuilt from a backup.