DTSS – Distributed Time Series System

DTSS is a high-performance time-series computation and storage engine. It provides a reusable core (the dtss::kernel) that can be embedded in applications, exposed as a service, or orchestrated from Python.

The system is designed around three principles:

  • Efficient time-series storage and evaluation

  • Dual communication interfaces (wire and web)

  • Optional primary–replica deployment for scalable computation

Core Architecture

At the heart of DTSS is the dtss::kernel. The kernel:

  • Manages time-series containers and storage backends

  • Evaluates symbolic time-series expressions

  • Maintains caching and subscription state

  • Optionally operates in replica mode for distributed deployments

The kernel can be used:

  • Directly in C++ applications

  • Through Python bindings

  • Via network servers (wire and web interfaces)

Network servers attach to the kernel and expose its functionality without altering the core execution model.

Dual Interface Model

DTSS services expose two complementary interfaces:

Wire Interface

The wire interface is a versioned, high-performance binary protocol intended for service-to-service communication and controlled deployments.

It provides:

  • Efficient request/response handling

  • Transfer of complete domain objects (e.g. full models or time-series)

  • Minimal serialization overhead

  • Strong typing via the C++ and Python client APIs

The wire protocol is designed to support both tightly coupled deployments and cross-version interoperability.

Protocol Versioning

The wire protocol is strictly versioned.

Each released protocol version is immutable: once defined, it never changes, (but it might be deprecated and go out of support). This guarantees that a client and server selecting the same

protocol version can communicate safely and predictably.

This enables:

  • Deployment of clients and services in different environments

  • Safe cooperation between different Shyft binary versions

  • Long-term stability for replication and data exchange scenarios

When connecting, client and server can negotiate a mutually supported protocol version. Automatic negotiation is not yet implemented, but is already possible using python or c++ api. A practical approach is just setting max level on the consumer/client side, based on current deployment on server.

Although protocol versions are immutable, support for very old versions may eventually be discontinued. Deprecation and removal of protocol versions follow normal release procedures, allowing controlled migration to newer versions.

Stable Versioned Protocols

Stable protocol versions are intended for interoperability across different Shyft releases.

These protocols focus on functionality that is common and broadly useful for distributed communication, such as:

  • Exchange of concrete time-series

  • Replica primary–replica synchronization

  • Queue-bridge data exchange

  • Structured information transfer between services

Because these protocol versions never change once released, a replica node does not need to be upgraded simultaneously with the primary, and vice versa.

Internal Protocol Version

In addition to the stable versioned protocols, the wire interface also provides an internal version.

The internal version:

  • Requires client and server to run the same semantic Shyft version (major.minor.patch).

  • Exposes the full set of protocol features.

  • Uses the most efficient communication techniques available.

  • Is ideal for tightly coupled internal application communication.

The internal protocol supports advanced operations such as:

  • Symbolic expression evaluation

  • Full model transfer

  • Extended feature sets beyond the stable interoperability subset

Shyft follows semantic versioning for its binaries. When using the internal protocol version, client and server must match at the semantic version level to ensure compatibility.

Python users typically interact with DTSS through the synchronous, type-safe DtsClient built on the wire interface.

Web Interface

The web interface is websocket-based and optimized for interactive applications and front-end integration.

It provides:

  • JSON-based request and response messages

  • Asynchronous interaction model

  • Subscription mechanisms with server push

  • Incremental data retrieval (projections / views)

Unlike the wire interface, the web interface is designed for interactive UI-driven workflows and progressive data loading.

Protocol Evolution and Compatibility

The web protocol is designed to be backward compatible between minor versions.

New versions evolve additively:

  • New message types may be introduced

  • New optional fields may be added to existing messages

  • Existing fields are not removed or semantically altered

This allows newer servers to interoperate with older front-end clients, provided the client only relies on supported features.

High-Performance JSON Processing

Although JSON is used as the message format, the implementation is optimized for performance:

  • Request parsing uses embedded native parsers built with Boost.Spirit and Boost.Hana.

  • Response generation uses high-performance generators (Boost.Karma and fmt).

  • Parsing and generation are designed to avoid unnecessary allocations and copies.

This allows the web interface to maintain high throughput while preserving flexibility and readability for front-end applications.

Model-Based Services

DTSS is often embedded in higher-level, model-based services, like the DStm.

In these services:

  • Time-series expressions are attached to domain models

  • Models define structure, relationships, and computation logic

  • Time-series serve as decorated attributes on those models, with dynamic inputs and outputs

Like DTSS itself, such services typically expose both wire and web interfaces and are orchestrated and extended using Python.

Customization and Extensibility

The DTSS server can be customized with Python hooks to:

  • Integrate external time-series databases

  • Extend storage backends

  • Add custom computation logic

  • Attach domain-specific services

This makes DTSS suitable both as a standalone service and as a computation core within larger systems.

Primary–Replica Deployment

DTSS supports primary–replica configurations.

In this mode:

  • The primary node owns authoritative time-series storage

  • Replica nodes perform computation and maintain local caches

  • Subscriptions ensure cache coherency between replica and primary

  • Computation and memory cache can scale horizontally while storage remains centralized

Replica nodes forward terminal I/O operations to the primary while evaluating expressions and serving clients locally. Subscriptions and polling mechanisms ensure that cached data remains consistent with the authoritative primary state.

This architecture enables distributed computation with strong consistency guarantees for stored time-series.