***************** shyft.time_series ***************** This package contains the following classes and functions to use by end-users. The namespace itself contains more classes and functions, but these are used internally. .. note:: **Vector types** Because the actual code is written in C++ which is strongly typed, Shyft Python code uses the concept of "vector" classes which are basically lists of the named type e.g. *TsBindInfo* and *TsBindInfoVector*. These are not specifically documented here. However, some vector types like *TsVector* are documented, because they provide more functionality than a simple list. Time #### Elements in this category deal with date/time. Function utctime_now ==================== .. autoclass:: shyft.time_series.utctime_now :members: Function deltahours =================== .. autoclass:: shyft.time_series.deltahours :members: Function deltaminutes ===================== .. autoclass:: shyft.time_series.deltaminutes :members: Class time ========== .. autoclass:: shyft.time_series.time :members: Class YMDhms ============ .. autoclass:: shyft.time_series.YMDhms :members: Class YWdhms ============ .. autoclass:: shyft.time_series.YWdhms :members: Class TzInfo ============ .. autoclass:: shyft.time_series.TzInfo :members: Class Calendar ============== .. autoclass:: shyft.time_series.Calendar :members: Class UtcPeriod =============== .. autoclass:: shyft.time_series.UtcPeriod :members: Class UtcTimeVector =================== .. autoclass:: shyft.time_series.UtcTimeVector :members: Time series ########### Elements in this category are the actual time series. Class TimeAxis ============== .. autoclass:: shyft.time_series.TimeAxis :members: Class TimeSeries ================ .. autoclass:: shyft.time_series.TimeSeries :members: Class TsVector ============== .. autoclass:: shyft.time_series.TsVector :members: Time series expressions ####################### The elements in this category implement the time series expressions solution. Class TsBindInfo ================ .. autoclass:: shyft.time_series.TsBindInfo :members: DTSS - The Distributed Time series System ######################################### The elements in this category implements the the DTSS. The DTSS provides ready to use services and components, which is useful in itself. In addition, the services are extensible by python hooks, callbacks, that allow the user to extend/adapt the functionality to cover other time-series data base backends and services. Note that DTSS is not a data-base as such, but do have a built in high performance time-series db. The DTSS is better viewed as **computing component/service**, that are capable of evaluating time-series expressions, extracting the wanted information, and sending it back to the clients. One of the important properties of the DTSS is that we can bring the heavy computations to where the data is located. In addition it as a specialized advanced caching system that allows evaluations to run on memory(utilizing multi-core evaluations). The DTSS contains a high performance in memory **queue** for messages that consists of collections of time-series. The queue mechanism also provide end-to-end handshake, so that producer can know that consumer have processed the queue message. The **Transfer** service built into the DTSS also allows for efficient direct replication, to other DTSS instances for timeseries that matches regular expression, and even with regular expression translation before pushing to remote instance.a The transfer mechanism is resilient to network and service interruptions, and can propagate changes to large sets of time-series in a few millisecond (limited by network/storage bandwidth). The open design allows it to utilize any existing legacy ts-databases/services through customization points. Class DtsServer =============== .. autoclass:: shyft.time_series.DtsServer :members: .. autoclass:: shyft.time_series.DtssCfg :members: Class DtsClient =============== .. autoclass:: shyft.time_series.DtsClient :members: Class StorePolicy ----------------- .. autoclass:: shyft.time_series.StorePolicy :members: DTSS Exchange ############# The **Exchange** section focuses on tools for transferring and managing time series across dtss services through advanced time-series replication mechanism and also bridging queues. All exchange functions are network fault tolerant, self repairing and best provide best effort features to ensure data flows to the designated site. The exchange services are also designed and tested for high volumes, an large data-sets. The exchange functions of the DTSS also provides extensive set of monitoring, statistics counting to detect errors in a way that can be observed. The Exchange Transfer and QueueBridge is created so that communication initiative goes from the local site to a remote site, to ensure the mechanism fits well into a highly secured site. Typical the communication between the sites should be established using ssh-tunnel, or vpn connections. DTSS Transfer ############# The current features includes pushing a specified set of time-series (regular expression) to a remote site. The time-range, from ..to , as well as incremental real-time push of changes is supported. Time-series names can also be translated so that they appear different on the remote site. Class TransferConfiguration --------------------------- .. autoclass:: shyft.time_series.TransferConfiguration :members: Class TransferStatus -------------------- .. autoclass:: shyft.time_series.TransferStatus :members: DTSS QueueBridge ################ The QueueBridge feature provides a way to link two instances of DTSS with queues. The mechanism ensure unique mapping of 'msg_id', so that other producers, or queue bridges can be linked to the same queue. Extended monitoring and statistics is provided, with optional logging of events. The pattern found useful, and implemented is .. mermaid:: graph TB subgraph Remote_Site ["Remote Site"] Producer["Producer"] R_Queue["Queue: R"] Producer --> R_Queue end subgraph Local_Site ["Local Site"] QueueBridge["QueueBridge (Fetcher & Ack Worker)"] Q_Queue["Queue: Q"] Consumer["Consumer"] R_Queue -->|Fetch Messages| QueueBridge --> Q_Queue Q_Queue -->|Fetch message| Consumer -->|Ack message| Q_Queue Q_Queue -->|Send Acks| QueueBridge -->|Route Acks| R_Queue end classDef queue fill:#fff3b1,stroke:#333,stroke-width:2px; classDef worker fill:#87ceeb,stroke:#333,stroke-width:2px; classDef producer_consumer fill:#e0e0e0,stroke:#333,stroke-width:2px; class R_Queue,Q_Queue queue; class QueueBridge worker; class Producer,Consumer producer_consumer; Methods for for start,status,stop and remove QueueBridges - :func:`~shyft.time_series.DtsClient.start_q_bridge` - :func:`~shyft.time_series.DtsClient.get_q_bridge_status` - :func:`~shyft.time_series.DtsClient.get_q_bridges` - :func:`~shyft.time_series.DtsClient.stop_q_bridge` In the current version, configured and started queue bridges are not persisted across of DTSS restarts. This means that an external monitor/control using a python script and a DTSS client is needed to control the bridge. The creation of the bridge, can be done during the bootstrap of the DTSS and since the implementation is robust for remote network connection problems, and even remote DTSS configuration problems, it will start to work when a connection to a remote configured DTSS is available. Class QueueBridgeConfiguration --------------------------- .. autoclass:: shyft.time_series.QueueBridgeConfiguration :members: Class QueueBridgeStatus -------------------- .. autoclass:: shyft.time_series.QueueBridgeStatus :members: DTSS Queue for time-series ########################## The queue mechanism provides a producer-consumer pattern that also involves consumer get-message followed by a consumer-ack message. The latter provides semantic information from the consumer, such as that the message was received, interpreted and applied, and also to provide diagnostics, incase it failed. .. mermaid:: graph TB Producer["Producer Node"]:::producer_consumer Consumer["Consumer Node"]:::producer_consumer DTSS_Queue["DTSS node with Queue"]:::queue Producer -->|Put Message| DTSS_Queue DTSS_Queue -->|Ack Info| Producer DTSS_Queue -->|Fetch Message| Consumer Consumer -->|Ack Message| DTSS_Queue classDef queue fill:#fff3b1,stroke:#333,stroke-width:2px; classDef producer_consumer fill:#e0e0e0,stroke:#333,stroke-width:2px; The `DtsClient` class provides a methods to work with queues. **Maintenance and configuration** - :func:`~shyft.time_series.DtsClient.q_list` - provides a list of queues configured - :func:`~shyft.time_series.DtsClient.q_add` - Adds a named queue to the list of queues - :func:`~shyft.time_series.DtsClient.q_remove` - Remove a named queue from the list of queues - :func:`~shyft.time_series.DtsClient.q_maintain` - Prune out 'done' messages in a queue **Message activities** - :func:`~shyft.time_series.DtsClient.q_put`. - a producer puts next message to the queue - :func:`~shyft.time_series.DtsClient.q_get`. - a consumer reads next message from the queue - :func:`~shyft.time_series.DtsClient.q_ack`. - a consumer completes its read cycle with ack and diagnostics/feedback **Query and information** - :func:`~shyft.time_series.DtsClient.q_infos` - Get all message infos contained in a queue - :func:`~shyft.time_series.DtsClient.q_info` - Get info for a specific message - :func:`~shyft.time_series.DtsClient.q_find` - Check/find a specific message Notice that by default, when a message is fetched from the queue, the associated TsVector payload is removed. This is to ensure memory is kept at a minimum. There is no automatic pruning of messages, so a separate client/controller needs to be active and call :func:`~shyft.time_series.DtsClient.q_maintain` at regular intervals to prune out messages done where ttl has expired. Also note that the current version does not persist the configured queues, nor the messages across restart of the DTSS. Usual techniques is to add the queue as part of the python bootstrap code, so that the queue is part of the configuration/startup of the DTSS. Persistence and other features are possible and easy to implement if needed. Class QueueMessage ------------------ .. autoclass:: shyft.time_series.QueueMessage :members: Class QueueMessageInfo ---------------------- .. autoclass:: shyft.time_series.QueueMessageInfo :members: Class TsInfo ============ .. autoclass:: shyft.time_series.TsInfo :members: Class CacheStats ================ .. autoclass:: shyft.time_series.CacheStats :members: Geo-location Time series ######################## The elements in this section integrate the generic time series concepts above with a geo-spatial co-ordinate system. This functionality extends to co-ordinate based queries in the time series storage. Class GeoPoint ============== .. autoclass:: shyft.time_series.GeoPoint :members: Class GeoTimeSeries =================== .. autoclass:: shyft.time_series.GeoTimeSeries :members: Class GeoTimeSeriesVector ========================= .. autoclass:: shyft.time_series.GeoTimeSeriesVector :members: Class GeoQuery ============== .. autoclass:: shyft.time_series.GeoQuery :members: Class GeoSlice ============== .. autoclass:: shyft.time_series.GeoSlice :members: Class GeoTsMatrix ================= .. autoclass:: shyft.time_series.GeoTsMatrix :members: Class GeoMatrixShape ==================== .. autoclass:: shyft.time_series.GeoMatrixShape :members: Class GeoGridSpec ================= .. autoclass:: shyft.time_series.GeoGridSpec :members: Class GeoEvalArgs ================= .. autoclass:: shyft.time_series.GeoEvalArgs :members: Class GeoTimeSeriesConfiguration ================================ .. autoclass:: shyft.time_series.GeoTimeSeriesConfiguration :members: Class GeoPointVector ================================ .. autoclass:: shyft.time_series.GeoPointVector :members: Class GeoPointVectorVector ================================ .. autoclass:: shyft.time_series.GeoPointVectorVector :members: Working with time series ######################## The elements in this section define how code shall behave or are actual tools dealing with time series. Policies ======== The elements in this section describe how time series are interpreted. Class convolve_policy --------------------- .. autoclass:: shyft.time_series.convolve_policy :members: Class derivative_method ----------------------- .. autoclass:: shyft.time_series.derivative_method :members: Class extend_fill_policy ------------------------ .. autoclass:: shyft.time_series.extend_fill_policy :members: Class extend_split_policy ------------------------- .. autoclass:: shyft.time_series.extend_split_policy :members: Class interpolation_scheme -------------------------- .. autoclass:: shyft.time_series.interpolation_scheme :members: Class point_interpretation_policy --------------------------------- .. autoclass:: shyft.time_series.point_interpretation_policy :members: Class trim_policy ----------------- .. autoclass:: shyft.time_series.trim_policy :members: Tools ===== The elements in this section work with time series. Class KrlsRbfPredictor ---------------------- .. autoclass:: shyft.time_series.KrlsRbfPredictor :members: Class QacParameter ------------------ Qac = Quality Assurance Controls .. autoclass:: shyft.time_series.QacParameter :members: Hydrology ========= The elements in this section are hydrology specific. Class IcePackingParameters -------------------------- .. autoclass:: shyft.time_series.IcePackingParameters :members: Class IcePackingRecessionParameters ----------------------------------- .. autoclass:: shyft.time_series.IcePackingRecessionParameters :members: Class ice_packing_temperature_policy ------------------------------------ .. autoclass:: shyft.time_series.ice_packing_temperature_policy :members: Class RatingCurveFunction ------------------------- .. autoclass:: shyft.time_series.RatingCurveFunction :members: Class RatingCurveParameters --------------------------- .. autoclass:: shyft.time_series.RatingCurveParameters :members: Class RatingCurveSegment ------------------------ .. autoclass:: shyft.time_series.RatingCurveSegment :members: Class RatingCurveSegments ------------------------- .. autoclass:: shyft.time_series.RatingCurveSegments :members: Class RatingCurveTimeFunction ----------------------------- .. autoclass:: shyft.time_series.RatingCurveTimeFunction :members: Class RatingCurveTimeFunctions ------------------------------ .. autoclass:: shyft.time_series.RatingCurveTimeFunctions :members: