Troubleshooting

Start with two commands that show which side is wrong:

shard version
shard capabilities

shard version prints the version of the CLI, then the version of the daemon or server it reaches. If nothing answers, the second line says why. On a machine that has not been set up:

client v0.1.2
shard: cannot connect to the shard daemon at /var/lib/shard/shard.sock: is it set up? shard setup

shard capabilities lists the nine lifecycle verbs and swap, and marks each one true or false for the provider the server runs.

The CLI prefixes returned errors with shard:. Setup also prints failures beside the step that failed. Find the line you got in the sections below.

The installer

The installer prints error: and a reason, and exits with code 1. Two messages are notices instead: the install still succeeded.

~/.local/bin is not on your PATH. This is a notice. The installer put shard in ~/.local/bin, but your shell does not look there. The installer prints the commands that add it, both for new shells and for the shell you are in. Run them.

has no build for linux-arm64. shard ships Linux builds for x86-64 only, and the error lists the builds that the release does have.

the GitHub releases API rate limit for this IP address is spent. The installer asks the GitHub API for the newest release without a token, and GitHub allows 60 such calls an hour for each IP address. The error says when the limit resets. Run the installer again after that.

could not reach the GitHub releases API at api.github.com. Check the network, and run the installer again.

the checksum of ... is ..., and SHA256SUMS ... says .... The download does not match the release, so the installer installed nothing. Run it again.

An Intel Mac can connect to a remote shard server. It cannot host sandboxes itself. This is also a notice. Run shard setup and choose a remote server.

Setup

Setup stops at the first check or step that fails. It marks that step, says why it failed, and makes no further changes:

✗ Download and verify required tools
  Could not download runsc: connection timed out.

Setup stopped. Earlier completed steps remain in place.
Run shard setup again to retry.

Every step is safe to run again. Fix the cause and run shard setup again, and it picks up where it stopped. If shard is not on your PATH yet, the last line names the full path of the binary instead. A run without a terminal also repeats its options there.

While a download runs, its step shows how much has arrived. A download fails if it receives no data for 30 seconds, and its line then ends no data for 30s; setup stops a download that sends nothing for that long. A slow download that is still making progress runs to the end.

If a check fails before setup changes anything, the output says No installation changes were made.

Setup needs administrator access, and sudo does not allow this user: .... Setup uses sudo for the steps that need root. Ask an administrator to give your user sudo access, or run shard setup as root.

Setup needs administrator access, and sudo needs a password it has no terminal to ask for. Run shard setup in a terminal, as root, or as a user with passwordless sudo.

Firecracker shows Requires access to /dev/kvm. The host has no hardware virtualization that shard can use, and the line under it says why, for example This machine does not provide it. Choose gVisor, or use a host with /dev/kvm: bare metal, or a VM with nested virtualization.

no terminal to ask ...: pass .... Setup is running without a terminal, as in a script or over ssh with no TTY, so a question has no answer. The error names the option that answers the question, and setup never picks one for you. A local run names every missing option in one error. For the API key, the error is no terminal to read the API key: set SHARD_API_KEY.

setup was cancelled; nothing changed. You declined the confirmation, and setup exits with code 1.

Setup exit codes lists the codes that a script can check.

The daemon

is it running?

When a command cannot reach the daemon, its error ends with a question and the command that answers it. How the line ends depends on how this host runs the daemon:

The line ends with	The daemon runs as	Its log
is the background service running? shard setup repairs it	the service that shard setup installed	sudo journalctl -u shard, or /var/log/shard/daemon.log on a Mac
is it running? systemctl status shard	a systemd service you installed by hand	sudo journalctl -u shard
is it running? launchctl print system/shard.daemon	a launchd service you installed by hand on a Mac	/var/log/shard/daemon.log
is it running? sudo shard daemon --provider gvisor	a process you start	the terminal that runs it
is it set up? shard setup	nothing yet

For the service that setup installed, run shard setup and choose Check or repair the installation. The repair starts the service if it stopped. It also checks the tools, the files that setup installed and the provider’s files in /var/lib/shard. It shows each change before making it, and it keeps your settings and your data.

To restart a service yourself:

sudo systemctl restart shard                      # Linux
sudo launchctl kickstart -k system/shard.daemon   # macOS

A daemon you started yourself stops when its terminal closes. Start it again in its own terminal.

permission denied; run the command again with sudo

On Linux the API socket belongs to root, so run local commands with sudo:

sudo shard list

On a Mac the daemon runs as your user, so local sandbox commands need no sudo. Setup and service commands can still need administrator access.

another shard daemon already holds ...

A root can have only one daemon, so a second shard daemon on the same root refuses to start. This happens when you start a daemon by hand while the service is already running one. Use the service, or stop it first.

another shard daemon, over the root ..., already serves this host's sandbox bridge and nft table

Every root on a Linux host shares one sandbox bridge and one nft table, so a host runs one daemon. A daemon on a second root refuses to start while another one runs, and the message names the root of the one that runs. This happens when you start a daemon by hand with --root while the service is running. Stop the service with sudo systemctl stop shard, or use its root.

task proxy failed, restart in ...

The daemon log repeats this line, and the same line for dns, while every shard create fails. The host has no nft. Setup installs it, so this happens after an install by hand. Install the nftables package. The next restart of each task succeeds, and the daemon itself needs no restart.

A sandbox is stopped after a reboot

A reboot ends every sandbox process. When the daemon comes back, it marks each sandbox that was running as stopped. shard list leaves stopped sandboxes out but says how many there are:

1 stopped sandbox; shard list --all

shard list --all shows the reason next to the state: stopped (daemon restarted and found no process). shard start <name> brings the sandbox back with its files.

An ordinary daemon restart leaves already running sandboxes alive, and the next daemon adopts them. An interrupted create, fork or restore can require cleanup instead.

Sandboxes

provider sysbox does not support fork on this host; use a server that supports fork. shard refuses a verb that the provider cannot do, and it never falls back to a weaker one. shard capabilities lists what the server supports. The verb and the provider in the message vary.

ran out of memory in shard list. The sandbox went past its memory bound, and the host’s OOM killer ended it. On gVisor, a sandbox that uses its whole bound is slowed down first, and only one that keeps growing past it is killed. The daemon starts it again on its own with its files: within 5 seconds the first time, then after a wait that starts at 10 seconds and doubles up to 5 minutes while the kills come within 10 minutes of a start. shard list counts the kills and shows a start still due, for example stopped (ran out of memory 3 times, starts again in 20s). shard stop <name> calls that start off. The bound is fixed when the sandbox is created, so to get more memory, create a new sandbox with a larger --memory.

stopped (the sandbox process died). The host or a crash ended the sandbox. shard start <name> brings it back.

exited with 7 in shard ps. The process exited with code 7, but the sandbox is still up. A sandbox outlives its processes, so you can still exec into it, or shard run the process again, until shard stop ends it. shard logs <sandbox> <name> shows what the process printed.

An outgoing request from the sandbox times out. A firewall rule of your own on the host can drop the request before shard sees it. shard lets the bridge shard0 through ufw and firewalld itself, so look for another rule that drops traffic on shard0:

sudo iptables -S
sudo nft list ruleset

A name does not resolve, or a request gets a 403, in a sandbox with a network policy. Check the policy and shard’s private-address restrictions. shard policy logs <name> shows each decision and the rule or restriction that made it. A policy with only address rules needs --allow dns before names resolve. The egress reference lists every rule.

An HTTPS client fails on a certificate in a sandbox with a secret. shard’s proxy inserts the key on the host, so it presents its own certificate to the client. Most clients trust it without any change. Java and some Rust builds need an extra step, and a client that pins a certificate cannot use a secret. The secrets reference has the steps.

A request to an internal server gets a 502 that says the upstream's certificate is signed by an authority the host does not trust. In a sandbox with a policy or a secret, the proxy checks the server’s certificate against the host’s trust store, not the sandbox’s. Add the authority that signed it to the host’s store, then restart the daemon. The egress reference has the commands for each system.

A remote server

A client command goes to the server that --remote names. Without --remote it uses a nonempty SHARD_REMOTE, then the saved connection. An explicit --remote "" selects the local socket. The API key comes from a nonempty SHARD_API_KEY, or from the saved connection when it names the same server.

cannot connect to the shard server at ...: connection refused; check the URL, and that shard serve or the proxy in front of it runs. Nothing answers at that address. The cause in the line can also be no such host or connection timed out. Check the URL, and check that shard serve is running on the server.

the server certificate is not trusted (...); for a private certificate authority, set SHARD_CA_FILE to its PEM file. For HTTPS, shard checks the certificate and has no option to skip the check. If a private authority signed the server’s certificate, point SHARD_CA_FILE at that authority’s certificate:

export SHARD_CA_FILE=/path/to/ca.pem

the server at ... did not accept the API key in SHARD_API_KEY. The server refused the key. A key in SHARD_API_KEY takes precedence over the saved one, so unset the variable to use the saved key. If the line names the saved key instead, run shard setup to replace it.

the saved API key is for ..., so shard does not send it to ...; set SHARD_API_KEY, or run shard setup. You named a server other than the saved one, and shard sends a saved key only to the server it was saved for. Set SHARD_API_KEY for this server.

the token lacks sandbox:write; use a token with sandbox:write. The key works, but its scopes do not cover the command. Ask for a key that has that scope. shard tokens scopes lists every scope.

shard info runs on the daemon host only and cannot reach .... Some commands, such as shard info and shard daemon status, act only on this machine and refuse a remote server. If you have a saved connection, add --remote "" to run one of them here.

Still stuck

Read the daemon’s log. On Linux, run sudo journalctl -u shard. On a Mac, the log is /var/log/shard/daemon.log. A daemon you started yourself writes its log to its terminal. When the daemon keeps the cause of an error in its log, the error line says where to look.

On the daemon host, shard daemon status (with sudo on Linux) shows the running daemon’s version, provider, process and background tasks.