Skip to content
shardDocs

The data directory

The data directory is where the daemon keeps everything it manages: the sandboxes, the images, the snapshots, the secrets and the network policies. It is /var/lib/shard, and shard setup creates it. To use another directory, start the daemon with shard --root <dir> daemon and pass the same --root to every command.

  • The daemon is its only writer. Every command goes to the daemon, which changes the files. Nothing else writes there.
  • One daemon per data directory. The daemon holds a lock on daemon.lock, so a second daemon on the same directory refuses to start.
  • One provider per data directory. Each sandbox record names the provider that made it. A daemon refuses to start on a data directory that holds another provider’s sandboxes, because it cannot run them.
  • Commands find the daemon through it. A command on the host connects to the socket shard.sock in the data directory.

What it holds

Section titled “What it holds”
/var/lib/shard/
├── shard.sock          the API socket
├── daemon.lock         keeps a second daemon off this directory
├── daemon.pid          the daemon's process ID
├── auth/               the signing key and token ledger of shard serve
├── .reserve            64 MiB held back, so a full disk can still free space
├── sandboxes/<id>/     one per sandbox: its record, logs and runtime files
├── names/              a link per sandbox name
├── checkpoints/        the saved memory of each paused sandbox
├── network/leases/     a file per IP address a sandbox holds
├── images/
│   ├── layout/         every pulled image, as an OCI image layout
│   ├── rootfs/         each image unpacked, once per digest
│   ├── erofs/          Firecracker: each image as a read-only EROFS file
│   └── disks/          vz: each image as an ext4 disk
├── snapshots/<id>/     one per snapshot: its record and the files it copied
├── snapshot-names/     a link per snapshot name
├── secrets/            a file per secret, mode 0600
├── policies/           the network policies
├── proxy/              the certificate authority of the egress proxy
├── egress.cursor       where the daemon left off in the kernel log
├── kernel/             Firecracker and vz: the guest kernel
├── runsc/              gVisor: the runtime's state per sandbox
├── runc/               runc: the runtime's state per sandbox
├── sysbox-runc/        Sysbox: the runtime's state per sandbox
├── exec/               gVisor, runc and Sysbox: each exec's scratch files
├── firecracker/        Firecracker: each microVM's files
├── jail/               Firecracker: the jail each microVM starts in
└── vz/                 vz: the VM helper and the guest's shard-init
  • Secrets stay in one place. secrets/ is the only place a secret value is written.
  • Only the provider in use makes its own paths. The daemon downloads kernel/ when it first starts, and installs vz/ from its own binary.
  • Each sandbox and snapshot has a record. sandboxes/<id>/sandbox.json holds a sandbox’s name, state, provider, image and bounds, and snapshots/<id>/snapshot.json holds a snapshot’s.
  • An image unpacks once per digest, so two tags of one image unpack once.
  • A restart catches up. egress.cursor lets a new daemon still log the packets the host dropped while it was down, and a new daemon clears the exec/ files an earlier one left.

Outside the data directory

Section titled “Outside the data directory”

shard also puts a few things elsewhere on a sandbox host:

  • /usr/local/bin/shard, and on Linux /usr/local/bin/shard-init: the binaries that setup installs.
  • The service: /etc/systemd/system/shard.service on Linux, or /Library/LaunchDaemons/shard.daemon.plist on a Mac.
  • The API key: /etc/shard/api-key on Linux, when setup serves the HTTP API.
  • /var/log/shard: the daemon’s logs on a Mac. On Linux, the logs go to the journal.
  • /var/run/shard/host.lock: on Linux, the lock that keeps a second daemon off the host, because every data directory shares one sandbox network.
  • The sandbox network: on Linux, the bridge shard0, one veth pair per running sandbox, and the nft tables inet shard and bridge shard. On a host that runs firewalld, a shard zone holds the bridge.
  • /var/lib/shard.xfs: on Firecracker, when the filesystem under /var/lib/shard cannot clone files, an XFS image mounted over /var/lib/shard, with its line in /etc/fstab.

A machine that only connects to a remote server keeps the CLI at ~/.local/bin/shard and the saved connection in ~/.config/shard/config.json.

For what shard setup installs, step by step, see What setup does.