Arena troubleshooting and known issues

Run diagnostics in trusted host Bash as the rootless deployment user, from the trusted tooling checkout outside execution mounts. Start with python3 -B tools/ai/arena.py status --arena shyft --detail below; substitute your recorded arena/member names and state paths. Status is inspection, not external-provider acceptance. The arena.py ssh examples below launch commands as ci inside the untrusted member; they are not host management sessions.

Logs and status/configuration output can contain task contents, personal paths, URLs or credentials; inspect/redact before sharing. Do not dump container environment, secret values, adapted credential-bearing Caddy JSON or authorization headers. Stop at failed preconditions. A failed operation may have partly changed resources: preserve its records and investigate before retrying or continuing. Read Known issues and assurance limits early; historical observations are not current-host tests. This chapter explains current diagnostics under accepted policy, not new recovery or provisioning features.

Symptom → first check → next procedure

Observable symptom	First supported check on the trusted host	Next reference / stop condition
Service missing or failed	Arena status detail, user-manager state, scoped units/journal below.	Service missing or failed; Host prerequisites and local readiness. Do not substitute another runtime or alter shared host wait services.
Image/artifact preparation or provenance mismatch	build.sh status and the reported operation evidence.	Image or artifact preparation fails; Selection, details and refresh. Do not bypass signature/package checks or overwrite conflicting inputs.
SSH identity or permission error	Saved endpoint/image in status; local pinned-config inspection below.	SSH mismatch or permission denied; Advanced prepared inputs and mounts. Changed host key or unexplained mapping/ownership means stop and inspect.
Refresh/add/remove conflict or incomplete phase	Status detail and the matching managed --dry-run plan.	Refresh/add/remove conflict or interrupted operation; Refresh, retry and retained recovery. Unprovable ownership blocks recovery/removal.
SSH works, inference or agent task fails	Local gateway observations and selected agent versions below.	Inference or agent fails while SSH works; Known issues and assurance limits. Distinguish gateway, provider/model and actual file-tool results.
Optional IDE/client/tunnel/GUI problem	Confirm the CLI SSH baseline and reviewed backend/seed/client inputs.	Optional IDE/client or tunnel problem; Optional IDE provisioning and named members. GUI/license/tunnel acceptance is separate; ordinary add does not install an IDE.

Follow the failing layer: host readiness and prepared inputs first, then owned service state and pinned SSH, then local gateways, upstream authorization and actual agent files/tests. Missing or unhealthy controls are not denial evidence; a green local result does not establish the next layer. Use Classify evidence before accepting a result for bounded evidence status, not a general network scan. Correct the reported prerequisite and reconcile retained state before retry; no reset/prune, guessed JSON edits or blind restart.

Service missing or failed

These queries are scoped to the example arena’s units and recent user journal; select the exact recorded unit names for yours. systemctl status and journalctl can include sensitive log lines: inspect locally and redact before sharing. Keep the time/unit scope narrow, not a whole-host log dump.

python3 -B tools/ai/arena.py status --arena shyft --detail
systemctl --user is-system-running
systemctl --user status ai-shyft-dev.service ai-shyft-api.service ai-shyft-egress.service
journalctl --user -u ai-shyft-dev.service -u ai-shyft-api.service \
  -u ai-shyft-egress.service --since '-10 min' --no-pager

Check the reported failed phase and generated unit/source identity. A missing user manager is a real prerequisite failure; do not substitute another runtime. Missing services can indicate an unsupported Quadlet key or generator failure. Fix the reported prerequisite, then retry the identical managed command only when recorded configuration/ownership and completed phases are consistent. Use Refresh, retry and retained recovery for partial state; do not blindly replay. Do not manually overwrite unrelated source Quadlets or enable generated services. Inactive host network-online.target is not a startup blocker. Do not mask, reset, override or start shared host wait services to recover an arena.

Image or artifact preparation fails

bash tools/ai/build.sh status
bash tools/ai/build.sh prepare --help
python3 -B tools/ai/agents/artifacts.py prepare --help

Use the operation log path reported by the tool; logs and provenance may contain private paths or input details, so review/redact before sharing. For non-default image state, use the same build.sh ... --state PATH as preparation. Signature/package trust failures must be fixed in trusted inputs, not bypassed. See Image preparation. Changed/missing prepared output or bundle bytes are conflicts. Refresh prepares a new image; it does not overwrite existing selections silently. A late recording failure after a helper committed an image can require a deliberate new output name or preparation refresh. No arena command pulls a missing image.

SSH mismatch or permission denied

ssh -G reads the saved client configuration locally, including private paths; it does not connect. Keep its output private. Confirm the endpoint and identity first; the final wrapper command is a remote ci identity observation, not a repair or a host-shell command. These paths/names assume the example default root.

python3 -B tools/ai/arena.py status --arena shyft --detail
ssh -G -F "$HOME/.local/share/dev-environment/ssh/ai-shyft-dev/config" ai-shyft-dev
python3 -B tools/ai/arena.py ssh --arena shyft --member dev -- id

Check the saved loopback port, dedicated key/pin and live selected image. A changed host key requires inspection; do not delete known_hosts or disable strict host checking. The wrapper forwards no authentication agent. Custom roots change the SSH file location. Workspace ownership uses the image’s actual ci UID/GID via keep-id. Review dedicated source ownership and narrow SELinux labels; do not recursively chown/chmod/relabel a broad home tree or grant ci administrative privileges. Do not allow passwords/root login or forward SSH/GPG agents to solve connection failures. See Advanced prepared inputs and mounts for mapping/mount rules.

Refresh/add/remove conflict or interrupted operation

Use --dry-run and status detail. Running refresh targets require --stop-running for a changed image; stop independent host writers as needed. Incompatible profile/user-mapping/mount changes require a separate migration. Resource-name reuse, changed sources/drop-ins, unexpected network attachments or mismatching secret IDs are ownership conflicts, not permission to force deletion. Keep recorded snapshots, keys and state; resolve only the proven offending resource. See Arena administration reference for safe retry and retained recreation. Removal preserves data and secrets by default. Image rollback changes retained image identities, not workspace/data, runtime migrations, cache, processes or unsaved buffers. Preserve config, keys, snapshots and evidence; do not delete JSON or prune storage to force a successful retry.

Inference or agent fails while SSH works

The version commands execute in the member through pinned SSH; they make no inference call and do not test file tools. Treat their output and status paths as private diagnostic evidence. Do not print a key or adapted broker configuration to diagnose authorization.

python3 -B tools/ai/arena.py status --arena shyft --detail
python3 -B tools/ai/arena.py ssh --arena shyft --member dev -- opencode --version
python3 -B tools/ai/arena.py ssh --arena shyft --member dev -- codex --version

Local gateway health checks listener readiness and a denied unknown route, not upstream authentication/model availability. An unavailable upstream or invalid secret can block inference while local work/SSH remains usable.

The broker permits GET /v1/models and POST /v1/responses, fixes the approved upstream and replaces client authorization with its broker-held credential. Squid permits only broker-source TLS CONNECT to the selected upstream host on port 443; it does not inspect decrypted request bodies. Client launcher/model defaults do not enforce model selection against an untrusted member submitting a different model identifier through a permitted Responses request. No local broker-side model-ID allowlist or per-task spending limit was identified in the audited chain. Provider-side model validation and entitlements were not checked in that audit; neither acceptance nor denial of another model is established. Permitted inference remains a task-data export and budget-use path.

Direct external access and other proxy destinations are intentionally denied. Do not broaden egress, install a vendor login or switch to Chat Completions to avoid a failure. Unsupported agent versions need reviewed profile support, not launcher bypass. A text-only response does not prove file tools ran; inspect actual files and execute the task’s test inside the execution member, as in CLI-first arena workflow. Real-provider agent execution is a separate deliberate operator action, not an automatic diagnostic. See Maintenance and manual boundaries for acceptance boundaries; do not disable agent safeguards.

Optional IDE/client or tunnel problem

Confirm the pinned CLI SSH baseline before attributing a failure to IDE state. Keep host management, execution-member commands and the client workstation context distinct. Review matching backend/client/JBR, sanitized seed and private runtime records in Optional IDE provisioning and named members; ordinary managed arena.py add does not supply backend/seed provisioning. Client tunnel/listening-address, reconnect, licensing and GUI/RDP acceptance remain manual. Do not expose a client tunnel or artifact service broadly, enable vendor egress, recopy a seed over initialized runtime state or treat an IDE connection as stronger containment.

Known issues and assurance limits

• Codex file tools fail on the tested Arch/Fedora runtime with bwrap: Can't mount proc on /proc: Operation not permitted. Text/SSE success does not resolve this. Do not disable its sandbox or add SYS_ADMIN. The supported OpenCode file/exec path was demonstrated with a local mock; real provider execution is a separate operator check.

• Ordinary peers reached owned bridge-gateway listeners in Podman’s shared rootless network namespace. Strict gateway-service denial remains failed/unresolved despite bounded cross-arena container/broker denials. These listeners were not in the physical host’s main namespace; other bounded passes do not establish universal host-service isolation. Avoid sensitive services on those bridge addresses; use separate VMs/hosts if that stronger boundary is required. See Bounded security verification for exact scope.

• A previously tested Arch Valgrind 3.25.1 failed mandatory memcmp resolution in stripped ld-linux-x86-64.so.2 from glibc 2.44+r50+g1848099f063e-1 before task execution. Supply trusted matching loader debug symbols/input and retest separately; no unapproved runtime download or privilege exception is justified. Prior Fedora development Memcheck clean/invalid-fixture evidence is distinct and tied to its selected image; scripting developer-debug checks are not applicable. See Privilege and debugging evidence limits.

• Persistent state is not a backup. Rollback changes images, not data migrations. Conflict checks cannot see all future VPN routes, aliases, independent writers or other state roots/contexts. User-session lifetime, deployment-specific SELinux, external-provider readiness and real IDE/client/licensing checks remain deployment responsibilities.

Lifecycle regression results and manual user checks do not establish independent security review, full containment or immunity to shared-kernel exploits. Scoped verification belongs in Bounded security verification; optional GUI/cache issues are covered by Optional IDE provisioning and named members and Arena administration reference.

Return to CLI-first arena workflow for ordinary operation, Image preparation for trusted inputs, or Arena administration reference for ownership/recovery. Use Bounded security verification for separately authorized bounded verification; read-only diagnosis does not authorize fixtures, live probes or disruptive tests.