Arena administration reference¶
This lookup reference is for the environment operator on the trusted host.
Use CLI-first arena workflow for prepare/create/SSH, refresh, add and removal; Arena troubleshooting and known issues
for diagnosis; AI-assisted work retains authority/review policy. Image acquisition is in
Image preparation; optional UI provisioning is in
Optional IDE provisioning and named members. No command here grants new host or agent authority.
The agent’s SSH session as ci is untrusted execution, not a management
shell. Only the compiler-check block below runs inside a member; all other
snippets are host-side. This chapter describes current implementation under
accepted policy, not proposed provisioning or recovery features.
Run host snippets from non-traced Bash and stop at failed preconditions/commands;
do not copy subsequent blocks over partial state or failed acceptance.
Need |
Go to |
|---|---|
Understand state and authority |
State, ownership and execution boundaries and Configuration, state and provenance |
Check the host without secrets |
|
Inspect ownership or secret handling |
|
Refresh, remove, recreate or recover |
|
Select advanced images or mounts |
|
Inspect persistent compiler cache |
|
Migrate an existing legacy installation |
Legacy staged refresh compatibility, Unregistered legacy cache adoption or size/disable changes and Legacy registry adoption and interrupted-operation recovery; these are separate legacy-only branches |
Diagnose or verify a boundary |
Arena troubleshooting and known issues and Bounded security verification |
State, ownership and execution boundaries¶
Desired configuration describes the recorded arena. Observed deployment records track progress and resource identities; live inspection determines current service/container state. Provenance describes selected inputs and operations. None of these records alone proves health, provider readiness or containment.
---
config:
flowchart:
nodeSpacing: 20
rankSpacing: 35
subGraphTitleMargin:
bottom: 24
top: 8
---
flowchart TD
O["Trusted operator + tooling checkout<br/>build.sh / arena.py; outside task mounts"]
O -.->|"generate / validate"| D["Private desired records<br/>arenas/NAME/config.json + manifest.json"]
O -.->|"record observed progress"| V["Private deployment/NAME.json<br/>observed progress + ownership / operations"]
I["Separate image-preparation state<br/>named selections + provenance"] -.->|"selected ID"| O
P["Local image store<br/>immutable image identities"] -.->|"inspect; never pull at deployment"| O
D -.-> E["Host control<br/>rootless Podman + user-systemd / Quadlet"]
V -.->|"reconcile live identities"| E
E -.->|"manage owned resources"| M
E -.-> G
subgraph ARENA["Named arena"]
M["Untrusted members / internal network<br/>same-arena peers mutually reachable"] -->|"permitted inference"| G["Gateway guards<br/>ai-api broker + egress policy"]
end
W["Agent-accessible host files<br/>workspace / data: reviewed RO or RW<br/>dedicated runtime / cache: generated RW<br/>writable contents mutable by agent"] -->|"selected mounts only"| M
S["Host secret + SSH administration<br/>Podman secret; pinned client identity"] -.->|"provider key to broker only"| G
S -.->|"loopback SSH; no agent forwarding"| M
classDef untrusted fill:#fbe1dc,stroke:#a74432,color:#54251e
classDef guard fill:#e4effb,stroke:#35669a,color:#193d62
classDef record fill:#f8fafc,stroke:#777777,color:#333333
class M,W untrusted
class O,E,G,S guard
class D,V,I,P record
style ARENA fill:#f8fafc,stroke:#777777
Host administration and deliberately exposed member state; these scopes describe authority, not proven isolation.¶
Dashed arrows denote trusted management/inspection; solid arrows denote permitted member access. Red/orange labels identify mutable untrusted execution/state; blue identifies control, gray identifies records/resources. The operator controls mount selection; private runtime/cache is private to the deployment user and member, not protected from that member’s agent. Read-only inputs remain disclosable through permitted inference.
The default private root is $HOME/.local/share/dev-environment. Its recorded
configuration/deployment, generated sources, SSH identities and operation evidence
have different roles from the deliberately mounted runtime/cache subdirectories.
Image-preparation metadata defaults to a separate directory under that root;
local images reside in the container image store, not in those JSON records.
The path and flag distinctions are in Configuration, state and provenance.
Upstream credentials belong to Podman secret administration and the broker; they are not development-member mounts. SSH client identity and pinned host-key configuration remain host-controlled; only generated server/settings files are exposed as required. Members share one arena trust domain. Separate arenas on one host share kernel and administrative authority: containers are not VM-level isolation. See AI-assisted work and Known issues and assurance limits for the unresolved bridge-gateway host-service finding and Codex file-tool limitation.
Host prerequisites and local readiness¶
Use rootless Podman/Buildah with subordinate UID/GID mappings, cgroup v2, a usable user-systemd manager and the installed Quadlet generator. Host tools include Bash, Python 3.11+, jq, Git, iproute2, OpenSSH and OpenSSL; public image acquisition needs Skopeo. Diagnose a degraded manager rather than assuming all units work. No automatic linger, host-policy change or vendor/distcc egress is provided. Inspect host prerequisites without exposing credentials:
podman info --format '{{.Host.Security.Rootless}} {{.Host.CgroupsVersion}}'
systemctl --user is-system-running
ip -j -4 route show table all
ss -ltn
Arena source networks/containers use [Quadlet] DefaultDependencies=false to
omit the implicit host network-online wait. This is not systemd’s
[Unit] DefaultDependencies=false. Generated network requirements and explicit
gateway ordering remain. Images are pinned locally with no pulls. Local SSH and
broker default-deny health do not establish provider/model/entitlement readiness.
Configuration, state and provenance¶
Ordinary commands generate JSON; it is not handwritten operator input. Default
private management root is $HOME/.local/share/dev-environment. Desired
configuration is arenas/NAME/config.json with an integrity manifest;
deployment/NAME.json separately records observed progress, provenance,
ownership and operation history. Private runtime/, cache/ and ssh/
state is separate from task workspaces. Image selection/provenance/log state is
image-preparation/; see Image preparation.
Use arena.py --root /absolute/private/path COMMAND ... for an alternate
arena root. Image preparation uses build.sh ... --state PATH and arena
commands use --image-state PATH to consume that store; changing arena root
does not implicitly change the image store. Keep management records and
provider/client credentials outside execution-member mounts. Only explicitly
generated per-member runtime/cache and read-only server/tool settings are
exposed; never mount the root itself. Paths must be absolute, without spaces,
symlink components, hardlinks where checked, or shell/Quadlet metacharacters.
Arena/member names start with a lowercase letter, then use lowercase letters,
digits and hyphens, up to 24 characters. Checks cannot detect all
filesystem aliases or inventories under other roots/Podman contexts/users.
Route checks describe current inventory, not future VPN changes or a reservation
against independent host/container writers. An unavailable check is a failure,
not a successful conflict clearance.
The manifest checks retained configuration/source integrity; deployment progress can be incomplete or stale. Inspect desired versus observed state with:
python3 -B tools/ai/arena.py status --arena shyft --detail
python3 -B tools/ai/arena.py status --arena shyft --json
bash tools/ai/build.sh status
Completion reports SSH commands and detail paths. Preparation resolves public references automatically, reuses immutable acquired inputs when appropriate, and records IDs/provenance. Local references never gain invented registry provenance. Changed local source/output identities or incompatible recorded selection inputs are conflicts requiring deliberate preparation refresh. Identical preparation reruns do not re-resolve a floating registry tag; use preparation refresh to acquire a newer public source. Retain previous images; there is no automatic pruning.
Secrets and resource ownership¶
First interactive create asks for a hidden key only after non-secret checks.
It records the exact arena-created secret ID and ownership labels, never its
value. create ... --secret NAME references an externally managed Podman
secret; noninteractive initial create requires that supported mechanism.
Never put the value in arguments, JSON, shell history or logs. Never dump
container environment or adapted secret-bearing Caddy configuration.
Removal validates recorded source hashes, generated-unit source identity and
live resource identities/policy. Names alone do not establish ownership.
Unexpected attached resources, changed sources/drop-ins or unprovable ownership
block removal. Work/data, private state, keys, images, records and secrets remain.
Only remove --arena NAME --remove-owned-secret may delete an exact
arena-created secret after ownership and other known-reference checks; external
secrets cannot be deleted through this option. Mounted and environment-secret
references are checked without reading values.
Uninspectable references in newer API-created containers block deletion.
References in independent uninstalled configurations/other state roots cannot
be exhaustively indexed.
After deliberate owned-secret deletion, retained recreation requires an
interactive hidden prompt; externally managed secrets must retain their identity.
Advanced prepared inputs and mounts¶
Create/add/member refresh normally consume --images SELECTION. Advanced local
prepared-image use requires --image LOCAL_REF --image-profile PROFILE;
gateway overrides are --squid-image LOCAL_REF --caddy-image LOCAL_REF.
These inspect/pin local identities, never pull. Gateway selection names can be
chosen with --squid-images NAME --caddy-images NAME. Caddy must be acquired,
not rebuilt. The only current gateway policy is openai-responses-v1.
Create offers --subnet, --address and --port; add offers member
--address/--port. Defaults allocate a free supported subnet, addresses
starting at .10 and loopback ports. Unavailable route/conflict checks are errors.
Gateway .2/.3 addresses remain reserved. --data-ro/--data-rw add dedicated
mounts. --shared-path explicitly acknowledges supported shared paths; it
does not isolate members sharing the same arena. Never expose the tooling
checkout, broad host home, management root, personal credentials, broker secret,
client private key, host/container management sockets or SSH/GPG agents. Writable
mounts expose host files for immediate agent modification; --shared-path is
an acknowledgement of sharing, not a new isolation boundary.
Development defaults to a separate private ccache; scripting disables it.
--ccache enabled|disabled --ccache-size 10GiB select supported policy for
new members. --selinux private applies narrow labels to dedicated binds;
do not relabel broad trees or disable host enforcement. Exact options:
python3 -B tools/ai/arena.py create --help
python3 -B tools/ai/arena.py add --help
python3 -B tools/ai/arena.py refresh --help
Refresh, retry and retained recovery¶
Use CLI-first arena workflow for ordinary command sequences; this section explains
state transitions and failure boundaries. --dry-run provides an inspectable
plan for managed create/add/refresh/remove; it does not waive validation or prove
that a later deployment will succeed.
---
config:
flowchart:
nodeSpacing: 22
rankSpacing: 30
---
flowchart TD
I["Prepared local selection<br/>build.sh refresh prepares a new ID"] --> C["Explicit create / add<br/>validate local identities + ownership"]
C --> A["Recorded managed deployment<br/>inspect live services separately"]
I --> F["Explicit arena.py refresh<br/>validate before --stop-running interruption"]
F -->|"changed compatible ID"| A
F -.-> N["No-op<br/>unchanged ID; services untouched"]
F -.-> Q["Refusal<br/>incompatible or unprovable inputs"]
A --> R["Explicit remove<br/>proven owned deployment only"]
R --> T["Retained configuration + state<br/>work, runtime/cache, keys, images, records<br/>secrets retained by default"]
T --> K["create NAME --retained<br/>validate saved IDs; refuse missing inputs"]
K --> A
A -.-> B["Explicit --rollback<br/>prior deployed image ID only; no data rollback"]
B --> A
classDef guard fill:#e4effb,stroke:#35669a,color:#193d62
classDef record fill:#f8fafc,stroke:#777777,color:#333333
class C,F,R,K,B guard
class I,A,N,Q,T record
Human-authorized managed transitions; interruption and retry are not atomic, and retained state is not a backup.¶
Arrows require operator-invoked commands; a build or registry update is not a service transition. No-op is an unchanged compatible identity; refusal is a separate failed precondition, not successful reconciliation. Retained recreation uses saved identities/configuration and valid keys, not latest tags, new seed copies or restored processes. Repeated removal preserves retained state after ownership checks. Ordinary removal stops/removes owned services, containers, networks and source Quadlets and reloads user-systemd; it is not a data purge.
Refresh validates profile, user mapping and mounts before stopping anything;
incompatible profile changes are refused, not migrated. --stop-running
authorizes interruption of affected managed services, not independent host
writers. Initialized seeds, workspace/data, keys, addresses, agent/IDE state and
applicable ccache remain. Image rollback restores the last target’s retained
local identities with refresh --arena NAME --member MEMBER --rollback
--stop-running (or --gateways --rollback --stop-running); it does not undo
data changes. Private-state snapshots exclude workspace/data and ccache.
Gateway refresh preserves policy/secret references and interrupts all members’
inference while local members remain usable.
External deployment operations are not atomic. Failed steps retain diagnostic
state and completed phases. Fix the reported prerequisite and retry the identical
command; do not delete JSON, keys or user files. Interrupted writes or externally
changed identities may require stopped-writer inspection against recorded source
hashes/snapshots before retry; unprovable ownership is deliberately refused.
create NAME --retained recreates all saved members of a removed arena without
new image selection or seed copying. Missing retained images/secrets must be
restored deliberately; tools never silently fetch substitutes. See
Secrets and resource ownership for the explicitly deleted owned-secret exception.
Persistent work, runtime and cache are not image layers or independent backups;
image rollback does not restore private runtime/cache or data migrations.
A failed operation may already have stopped services or changed external resources. Preserve its journal/logs, recorded before/after identities and snapshots; reconcile ownership and completed phases before retrying the same command. Missing or changed source/manifest/metadata is a blocker, not permission for forced cleanup, pruning, key regeneration or a blind restart. No general one-command recovery from arbitrary external mutation is promised.
Legacy staged refresh compatibility¶
refresh --arena NAME --member MEMBER --image REF alone retains its old staged
plan meaning. Its --apply --acknowledge-migration belong only to that legacy
path. Use --images SELECTION for managed deployment, or include
--image-profile PROFILE with an advanced local image override. Mixing staged
flags and new deployment options is rejected. Legacy prepare/adopt/add-environment
interfaces remain; managed operations require managed ownership records and do
not silently adopt old installations. See Legacy registry adoption and interrupted-operation recovery.
Optional IDE backend/seed provisioning is also a distinct advanced branch:
managed arena.py add does not expose those inputs. Its legacy
add-environment path stages files without establishing managed deployment
ownership; use Add a separate named IDE member rather than mixing it into normal add.
Persistent local compiler cache¶
Only Arch/Fedora development supports managed preinstalled ccache. Scripting
rejects enabled caching; its inherited executables are not a required developer
toolchain. Cache setup adds no packages, distcc, remote cache or network path.
ROOT/cache/ENVIRONMENT/ccache is private, separate from IDE/agent runtime and
replacement snapshots, mounted at /var/cache/shyft-ccache. Ownership, allocation
conflicts, symlinks and hardlinks are checked. Private SELinux labeling applies
only to dedicated mutable binds, not unrelated host trees; deployment-host label
enforcement remains manual. Never import personal/release caches.
Read-only host-controlled ssh/ENVIRONMENT/ccache.conf fixes path, selected
size and local-only operation. Empty remote/prefix defaults are checked through
--show-config, not empty serialized assignments (tested ccache 4.14 treats
those as line continuations). Managed SSH SetEnv supplies
CCACHE_CONFIGPATH/CCACHE_DIR to actual SSH/IDE sessions as well as container
processes. These are defaults, not a boundary against an agent changing/disabling
caching. Correctness defaults, compiler checking and hash-directory behavior are
not relaxed. Metadata records version/settings; inspect effective values.
Shyft CMakePresets.json (shyft-ci and inherited Linux presets), local
CMakeUserPresets.json and tools/cmake/CMakeExamplePresets.json already use
CMAKE_C_COMPILER_LAUNCHER=ccache and CMAKE_CXX_COMPILER_LAUNCHER=ccache.
Keep the selected compiler; do not add wrapper directories to PATH, wrap CC/CXX
again or overwrite user presets. Other projects need their own reviewed launcher
configuration, not a global replacement compiler.
Backend Bash shell as SSH ci in an already reviewed running development
member. Prerequisite: supplied checkout/dependencies support shyft-ci;
otherwise select the already reviewed compiler-selecting preset/build directory.
Stop at any failed command:
printf '%s\n' "$CCACHE_DIR" "$CCACHE_CONFIGPATH"
command -v ccache gcc g++ cmake
ccache --version
ccache --show-config
ccache --show-stats
cd /home/ci/workspace
cmake --preset shyft-ci
cmake --build build/ci --verbose
sed -n '/^CMAKE_\(C\|CXX\)_COMPILER\(:\|_LAUNCHER:\)/p' build/ci/CMakeCache.txt
ccache --show-stats
Expected: compiler paths remain GCC/Clang, launchers say ccache and each verbose compile invokes it once; local path/limit and empty remote backend match policy. Stats are cumulative. An incremental build doing no compilation is not a hit; changed tools/options/paths/headers may legitimately miss. Never clear/reset an existing user’s cache/stats for acceptance. Version/empty-directory checks alone are not integration evidence. Distinct environments have distinct mount paths; namespaces alone do not establish cache access control, and arena peers are still mutually network-reachable.
Unregistered legacy cache adoption or size/disable changes¶
This branch is only for an existing setup.py runtime not registered in an
arena, before legacy arena adoption below. Named/adopted arena cache policy is
recorded in immutable member configuration; arena.py refresh currently accepts
image changes, not cache-setting changes. Do not mutate arena-managed runtime
metadata through this lower-level branch or hand-edit registry files. A new
member may select its cache policy; a changed existing arena policy needs a
separately reviewed migration, not changed prepare flags or a guessed API.
Host Bash, ordinary rootless deployment user. Set the following from the actual legacy runtime inventory, not the named workflow’s defaults. Its installed source must be owned by this setup. Stop agents/IDEs, disconnect clients and keep all host/container writers/restart paths stopped. Back up workspace/data/cache independently; replacement snapshots omit them.
: "${TOOLS:?absolute trusted tools/ai path}" "${ROOT:?actual private canonical root}"
: "${UNITS:?actual user Quadlet source directory}"
: "${LEGACY_ENVIRONMENT:?existing unregistered setup runtime identity}"
: "${CCACHE_MODE:?enabled or disabled}" "${CCACHE_SIZE:?approved local size, e.g. 10GiB}"
systemctl --user stop "$LEGACY_ENVIRONMENT.service"
test "$(systemctl --user show "$LEGACY_ENVIRONMENT.service" -p ActiveState --value)" = inactive
CURRENT_IMAGE=$(jq -er .image_id "$ROOT/runtime/$LEGACY_ENVIRONMENT/metadata.json")
python3 -B "$TOOLS/setup.py" replace-image --root "$ROOT" --name "$LEGACY_ENVIRONMENT" \
--image "$CURRENT_IMAGE" --acknowledge-migration \
--ccache "$CCACHE_MODE" --ccache-size "$CCACHE_SIZE"
: "${CACHE_SNAPSHOT:?exact snapshot path just printed, not an older snapshot}"
(
set -eu
test ! -L "$UNITS/$LEGACY_ENVIRONMENT.container"
cmp "$UNITS/$LEGACY_ENVIRONMENT.container" "$CACHE_SNAPSHOT/quadlet.container"
updated=$(mktemp "$UNITS/.$LEGACY_ENVIRONMENT-cache.XXXXXX")
cat "$ROOT/quadlet/$LEGACY_ENVIRONMENT.container" > "$updated"
mv -T -- "$updated" "$UNITS/$LEGACY_ENVIRONMENT.container"
)
jq .ccache "$ROOT/runtime/$LEGACY_ENVIRONMENT/metadata.json"
Only after successful source comparison, reviewed generated mounts/settings and
operator authorization should the operator reload/start the owned service and
inspect ccache over its pinned SSH configuration. Failed comparison means stop,
not overwrite. This is not a lock against other installers; coordinate writes.
Same-image replacement snapshots private runtime state/metadata/Quadlet/managed
SSH settings, retaining keys/cache without clearing it. Disabling removes the
mount and sets CCACHE_DISABLE=1 to avoid fallback into IDE cache; existing
launcher presets still compile. Cache may later be re-enabled. Omitting options
preserves legacy unmanaged image/XDG behavior, not automatic adoption.
Legacy registry adoption and interrupted-operation recovery¶
Adoption retains existing network/gateway/container/SSH names, ports, keys and
state. It records identities: legacy and checks exact existing generator
policy/units/runtime metadata. It does not install, rename or restart resources.
This is a migration reference for existing installations, not fixed-name
initial setup; use the main named workflow for new installations.
Host Bash with TOOLS/ROOT selected as above. Supply LEGACY_JSON from
an operator-reviewed inventory using schema version 1, actual arena name/subnet/
state_root, identities: legacy, actual member names, image/profile, mounts,
seed/backend/cache/SELinux/autostart settings, exact gateway image references,
policy: openai-responses-v1 and secret: openai-api-key. Supply
LEGACY_GATEWAYS as the retained old staged directory outside every dev mount,
with its images.json, policy and source units intact. Do not substitute a new
named gateway directory or use baseline NAME/STAGE variables.
: "${LEGACY_JSON:?absolute reviewed actual legacy arena JSON}"
: "${LEGACY_GATEWAYS:?absolute retained legacy gateway directory}"
python3 -B "$TOOLS/arena.py" --root "$ROOT" adopt \
--config "$LEGACY_JSON" --gateway-directory "$LEGACY_GATEWAYS"
Expected: plan only. Inspect it, stop every listed legacy service/container and all writers/restart paths, preserve evidence, then explicitly authorize apply:
python3 -B "$TOOLS/arena.py" --root "$ROOT" adopt \
--config "$LEGACY_JSON" --gateway-directory "$LEGACY_GATEWAYS" --apply
LEGACY_ARENA=$(jq -er .name "$LEGACY_JSON")
python3 -B "$TOOLS/arena.py" --root "$ROOT" prepare \
--config "$ROOT/arenas/$LEGACY_ARENA/config.json"
Apply refuses running/paused/restarting legacy resources and unknown policy or ownership differences. Only registry metadata/policy evidence is added. Recovery is to keep using the untouched legacy sources/state. Full renaming is not automated: inventory stopped identities/mounts and review a separate migration rather than generating new keys or mixing named and retained aliases.
Failed ordinary preparation rolls back only newly owned files. Abrupt termination
can leave an incomplete arena or pending.json; these are explicitly invalid,
not initialized state. Stop writers and preserve config.json, manifest.json,
runtime metadata, SSH/source-unit evidence, last-refresh.json and retained
snapshot. Exact-config partial preparation may be retried only if existing members
validate. Interrupted refresh retains old config/proposed IDs in its journal:
reconcile runtime/source/manifest against the exact retained snapshot before
resuming. Do not delete state, regenerate keys or restart blindly.
Replacement preserves saved mounts, not processes, unsaved buffers or writable container-layer files. Private runtime snapshots exclude workspace/data and compiler cache. Image rollback cannot undo migrated IDE/agent/data state; retain old images and compatible independent backups for application-specific recovery. Use main named refresh for registered members, not direct lower-level replacement.
Inspection, verification and acceptance¶
Status/detail and the host inspections above describe configuration, ownership and local service state. They do not establish real-provider inference, agent file-tool success, GUI/licensing acceptance or whole-boundary containment. Keep records/logs private and avoid secret-bearing environment/configuration output; use Arena troubleshooting and known issues for safe diagnosis.
Standalone verification commands and evidence boundary lists operator-invoked standalone tests and their prerequisites in the security chapter. Disposable fixtures, hosted deployment checks and manual acceptance are distinct evidence; none is independent certification. Those tests were not run for this documentation edit. Review the scoped Bounded security verification procedures and Known issues and assurance limits before claiming acceptance. Human review, signing, publishing and integration remain outside agent authority under AI-assisted work.