Operations

Deploying a systemd user service

This is an operator template, not an installed unit. Adapt the executable path to your Shyft installation; here %h/.local/shyft is the installation prefix. Prepare ~/.local/state/shyft and, if needed, the protected token file described in Bearer-token security. Put this unit in ~/.config/systemd/user/shyft-met-no.service:

[Unit]
Description=Shyft MET Norway integration service
After=network-online.target

[Service]
Type=simple
ExecStart=%h/.local/shyft/bin/shyft-integration-met-no --db-path %h/.local/state/shyft/met-no --listen 127.0.0.1:20080 --reconcile-interval-seconds 300 --log-level info
Restart=on-failure
RestartSec=5
UMask=0077

[Install]
WantedBy=default.target

The process stays foregrounded: do not add & or a wrapper that detaches it. For authenticated MET requests, append --bearer-token-file %h/.config/shyft/met-no.token to ExecStart. The user unit does not itself guarantee network availability or that DTSS is running. Arrange DTSS startup before activating agents. User-manager persistence after logout depends on the host’s user-service/linger policy.

systemctl --user daemon-reload
systemctl --user enable --now shyft-met-no.service
systemctl --user status shyft-met-no.service
systemctl --user restart shyft-met-no.service
journalctl --user -u shyft-met-no.service
journalctl --user -u shyft-met-no.service -f

Service health and agent status

Use the installed Python client against the running service. A running manager does not mean all agents are ready or all forecasts are stored.

from shyft.integration import met_no

client = met_no.Client(host_port='127.0.0.1:20080', timeout_ms=1000,
                       operation_timeout_ms=2000)
try:
    status = client.get_status()
    print('Service running:', status.running)
    for agent in status.agents:
        print(agent.name, agent.mid, agent.state.name)
        print('Targets:', agent.fc_db, agent.cc_db)
        print('Last started:', agent.last_reconcile_started)
        print('Last completed:', agent.last_reconcile_completed)
        print('Next:', agent.next_reconcile)
        print('Diagnostic:', agent.last_diagnostic)
        result = agent.last_reconcile_result
        if result is not None:
            print('Cycles:', agent.completed_cycles, 'cycle time:', result.now)
            print('Discovery:', result.discovered, result.discovery_issues)
            print('Already complete:', result.complete_before)
            print('Acquisition:', result.planned_acquire, result.acquired,
                  result.acquisition_failed, result.unavailable, result.deferred)
            print('Concat:', result.concat_attempted, result.concat_stored,
                  result.concat_failed, 'cancelled:', result.cancelled)
            for issue in result.issues:
                print(issue.stage.name, issue.forecast, issue.period,
                      issue.http_status, issue.diagnostic)
finally:
    client.close()

ServiceStatus.agents is ordered by logical configuration name. get_agent_status(name) returns one snapshot, or None if no agent exists. The active mid and name come from that runtime agent, not from choosing the newest ModelInfo. Read its configuration with read_model(agent.mid). For the persistent selection independently of runtime state, use get_active_version(name) or the service-wide get_active_versions() mapping. get_versions(name) lists stored metadata ordered by created, then id; that order does not select the active version. See Configuration for switching and clearing selections without deleting versions or data. Snapshots are read-only; modifying copied lists does not affect the service.

The Python AgentState members are:

• RESOLVING: connecting to DTSS and validating target metadata;

• READY: resolved and waiting/scheduled for reconciliation;

• RECONCILING: a cycle is in progress;

• BLOCKED: structural target resolution failed; inspect last_diagnostic;

• STOPPING: cancellation/shutdown requested;

• STOPPED: agent is not running.

no_utctime in scheduling timestamps means no timestamp/schedule is available (for example, no next automatic cycle in manual mode). last_reconcile_result is None until a cycle completes. Its counters and issues describe the latest completed cycle, not lifetime acquisition totals; completed_cycles counts cycles for this runtime agent. Issue stages are ReconcileStage.DISCOVERY, ACQUISITION, and CONCAT.

Request reconciliation now

For an existing active agent (replace the logical name as needed):

from shyft.integration import met_no

client = met_no.Client(host_port='127.0.0.1:20080', timeout_ms=1000)
try:
    name = 'site-weather'
    before = client.get_agent_status(name=name)
    if (before is not None and before.state in
            (met_no.AgentState.READY, met_no.AgentState.RECONCILING)):
        client.request_reconcile(name=name)
        print(client.get_agent_status(name=name))
    else:
        print('No ready/running agent:', before)
finally:
    client.close()

This is asynchronous/coalesced: the request wakes/schedules the agent and returns without waiting for cycle completion. Multiple requests can coalesce; they do not guarantee one cycle per request. Inspect status afterwards, watching completed_cycles, completion time, and the latest result. A snapshot immediately after acknowledgement may still show the earlier cycle. Requests do not bypass cache freshness, acquisition retry_after, or upstream cooldown. Unknown, blocked, stopped, and stopping agents cannot be reconciled through this operation; a request neither activates a version nor fixes a blocked target.

Troubleshooting a missing forecast

1. Inspect service/agent status and the active mid. Check the configured product and forecast-reference period; confirm the service is using the intended version, not merely a stored inactive version.

2. Inspect acquisition bookkeeping for that mid and ForecastId.

3. Inspect the DTSS fc archive if needed; its forecast presence is authoritative, not the local record alone.

4. Request reconciliation for a ready/reconciling agent.

5. Recheck status and the acquisition record after a completed cycle; inspect discovery or concat issues separately from acquisition failures.

For example, using a known active agent’s product and ID:

from shyft.integration import met_no
from shyft.time_series import time

client = met_no.Client(host_port='127.0.0.1:20080', timeout_ms=1000)
try:
    agent = client.get_agent_status('site-weather')
    if agent is not None:
        cfg = client.read_model(mid=agent.mid)
        forecast = met_no.ForecastId(
            product=cfg.product.name,
            reference_time=time('2026-06-16T03:00:00Z'),
            member=None)  # Use a member index for an ensemble-member product.
        record = client.get_acquisition_state(mid=agent.mid, id=forecast)
        if record is None:
            print('No local bookkeeping; inspect fc and the catalog day.')
        else:
            print(record.status.name, record.attempts, record.last_attempt,
                  record.updated, record.retry_after,
                  record.http_status, record.diagnostic)
finally:
    client.close()

The current AcquisitionStatus members mean:

UNKNOWN

No confirmed local outcome; if absent from fc, reconciliation plans acquisition. An absent record (None) similarly does not prove missing data.

STORED

Acquisition previously stored and verified the forecast in fc. The current fc inventory still wins if data has since been removed.

PERMANENTLY_UNAVAILABLE

Local state marks this forecast unavailable; it is not planned for acquisition while absent from fc.

RETRYABLE_FAILURE

A failed attempt that can be retried. A valid future retry_after defers it; when due (or unset as no_utctime), it is eligible again.

Not every enum outcome is generated by current acquisition: caught acquisition errors, including individual HTTP failures, are recorded as RETRYABLE_FAILURE with no per-record retry deadline. Upstream throttling is also handled by the service-wide controller’s pacing/cooldown. Do not infer permanent forecast unavailability from a single HTTP error.

Local state versus fc

fc is authoritative. If local state says STORED but fc no longer contains the forecast, reconciliation plans reacquisition when that forecast is discovered in the active period. Conversely, a complete forecast in fc proves completion even without a local stored record; it is not downloaded just to recreate bookkeeping. Inventory checks the target’s configured variables, forecast reference times, and ensemble representation.

Catalog-day diagnostics

The cache is shared by product/day across configurations in the service DB. get_catalog_day reads a cached UTC day without triggering discovery:

from shyft.integration import met_no
from shyft.time_series import time

client = met_no.Client(host_port='127.0.0.1:20080', timeout_ms=1000)
try:
    agent = client.get_agent_status('site-weather')
    if agent is not None:
        product = client.read_model(agent.mid).product
        day = client.get_catalog_day(
            product=product, day_start=time('2026-06-16T00:00:00Z'))
        if day is None:
            print('This product/day has not been cached.')
        else:
            print(day.status.name, day.fetched_at, day.retry_after,
                  day.http_status, day.diagnostic)
            for forecast in day.forecasts:
                print(forecast.id)
            for issue in day.issues:
                print(issue.period, issue.http_status, issue.diagnostic)
finally:
    client.close()

CatalogDayStatus.AVAILABLE means the catalog was readable (possibly with no matching runs, or with identity-parse issues). Stable historical available days remain cached to avoid repeated archive scans. Available recent days can refresh after the configured interval because new runs may appear.

RETRYABLE_FAILURE means day discovery failed but may be retried after retry_after. Failed refreshes retain any previous cached forecasts. PERMANENTLY_UNAVAILABLE applies to historical catalog days returning HTTP 404/410; these are not refreshed. The same HTTP outcomes for a recent day remain retryable. Catalog status and acquisition status are different records/enums.

Discovery uses actual catalog entries. It does not synthesize expected/missing forecast cycles. An unlisted forecast therefore may have no acquisition record; check catalog diagnostics rather than assuming it was downloaded and lost.

Blocked agents

Typical resolution failures include empty/unreachable DTSS endpoints, missing fc, missing cc when concat is enabled, incompatible fc/cc grid/CRS or ensemble layout, or cc without exactly one anchor. Inspect last_diagnostic and repair the target/configuration.

A blocked agent does not automatically repeat target resolution, and request_reconcile is rejected. After repairing the external target, use clear_active_version(name) followed by set_active_version(name, mid) to recreate the agent with its retained version, or select a different valid stored version directly. Selecting the same already-active mid without clearing does not recreate it. Merely storing a new passive version does not fix it; see Configuration for the Python activation workflow. Restarting the service through its supervisor also recreates persisted active agents.

An anchor later than an incoming reference time is a per-forecast concat failure, not necessarily a blocked agent: inspect concat_failed and ReconcileStage.CONCAT issues. Other target/variable/source incompatibilities may be reported during acquisition. Ordinary individual HTTP/acquisition and concat failures do not normally block the whole agent; subsequent cycles can continue processing other forecasts.

Historical backfill

1. Pre-create authoritative fc with the desired CRS, grid, variables, and ensemble layout.

2. Optionally pre-create/seed cc with an anchor no later than the earliest forecast reference time; see The cc anchor precondition.

3. Construct a finite Config.period of forecast reference times.

4. Store the configuration version through Python, then call client.set_active_version(name, mid) with its logical name and stored ID.

5. Inspect service/agent status and verify the intended name/mid is active.

6. Allow automatic reconciliation to acquire historical catalog entries, or request a cycle if the service runs in manual mode.

7. Use cycle results, acquisition state, and catalog-day diagnostics for failures.

The shared persistent catalog cache reduces repeated archive catalog traffic across configurations and process restarts. A finite period does not force MET to retain a dataset or supply runs that are absent from its catalog.

Ongoing operation

Use period = [desired_start, max_utctime) as shown in Configuration, then store/activate that version. With automatic scheduling enabled, agents periodically reconcile the observable part of the interval as new forecasts appear. Catalog caching suppresses unnecessary scans; valid future retry_after values suppress premature record retries; forecasts already complete in fc are not unnecessarily downloaded again. Upstream concurrency, pacing, and cooldown are shared across service agents.

Concat recovery and backups

Concat reads forecasts already verified in fc; it does not redownload MET data. Each normal reconciliation can retry concat for discovered complete forecasts, including after a restart. Repeating concat is safe under the normal DTSS merge/concat conditions. Temporary cc failures leave successful fc acquisition independent and intact.

Recovery still requires suitable fc contents, catalog discovery in the active period, compatible targets, a valid cc anchor, member selection, and an aligned slice within each forecast. The service will not automatically rebase cc or scan arbitrary fc data outside that workflow. Re-create/seed a damaged derived target through your DTSS provisioning process, then restart/reconcile as needed.

The backup hierarchy is:

• fc: authoritative acquired forecast archive;

• MET service RocksDB: configuration/version metadata, active versions, acquisition bookkeeping, and catalog cache;

• cc: derived concatenated output, regenerable from appropriate fc contents when the normal concat conditions hold.

Preserve both fc and the configuration DB according to your deployment’s retention/recovery requirements. Do not treat the entire service DB as a cache that can be discarded: losing it also loses configuration versions and active selection. Protect the separately provisioned credential file separately.

Do not blindly copy an open RocksDB directory. Stop the service before a filesystem-level backup unless using a supported RocksDB backup/checkpoint mechanism. Coordinate DTSS archive backups with that service’s supported backup procedure too. This executable exposes no backup/checkpoint command.

Upgrade and restart

1. Stop the service: systemctl --user stop shyft-met-no.service.

2. Preserve required data/backups and upgrade/install Shyft.

3. Start it again: systemctl --user start shyft-met-no.service.

4. Inspect journal and Python get_status().

5. Verify that the persisted active names/mid values have agents and that target resolution succeeds; then check subsequent reconciliation results.

Runtime status, timestamps, and cycle results are reconstructed on process start, not persisted as a stale snapshot. Configuration versions, active selection, bookkeeping, and catalog cache survive in the service DB. Restarting restores active agents, not inactive stored versions. No schema-migration CLI is provided.