shyft.time_series ¶

Notes

ref. to related functions .diff_units(…) and .trim(..)

calendar_units(self: shyft.time_series.Calendar, t: shyft.time_series.time) → shyft.time_series.YMDhms¶

returns YMDhms (.year,.month,.day,.hour,.minute..) for specified t, in the time-zone as given by the calendar

Parameters:: t (time) – timestamp utc seconds since epoch
Returns:: calendar_units. calendar units as in year-month-day hour-minute-second
Return type:: YMDhms

calendar_week_units(self: shyft.time_series.Calendar, t: shyft.time_series.time) → shyft.time_series.YWdhms¶

returns iso YWdhms, with properties (.iso_year,.iso_week,.week_day,.hour,..) for specified t, in the time-zone as given by the calendar

Parameters:: t (time) – timestamp utc seconds since epoch
Returns:: calendar_week_units. calendar units as in iso year-week-week_day hour-minute-second
Return type:: YWdms

day_of_year(self: shyft.time_series.Calendar, t: shyft.time_series.time) → int¶

returns the day of year for the specified time

Parameters:: t (time) – time to use for computation
Returns:: day_of_year. in range 1..366
Return type:: int

diff_units(self: shyft.time_series.Calendar, t1: shyft.time_series.time, t2: shyft.time_series.time, delta_t: shyft.time_series.time, trim_policy: shyft.time_series.trim_policy = <trim_policy.TRIM_IN: 0>) → int¶

calculate the distance t1..t2 in specified units, taking dst into account if observed The function takes calendar semantics when delta_t is calendar::DAY,WEEK,MONTH,YEAR, and in addition also dst if observed. e.g. the diff_units of calendar::DAY over summer->winter shift is 1, even if the number of hours during those days are 23 and 25 summer and winter transition respectively. It computes the calendar semantics of (t2-t1)/deltaT, where deltaT could be calendar units DAY,WEEK,MONTH,YEAR

Parameters:

t1 (time) – timestamp utc seconds since epoch
t2 (time) – timestamp utc seconds since epoch
delta_t (time) – timestep in seconds, with semantic interpretation of DAY,WEEK,MONTH,YEAR
trim_policy (trim_policy) – Default TRIM_IN, could be TRIM_OUT or TRIM_ROUND.

Returns:

n_units. number of units, so that t2 = c.add(t1,delta_t,n) + remainder(discarded). Depending on the trim_policy, the remainder results will add/subtract one unit to the result.

Return type:

int

Notes

ref. to related functions .add(…) and .trim(…)

quarter(self: shyft.time_series.Calendar, t: shyft.time_series.time) → int¶

returns the quarter of the specified t, -1 if invalid t

Parameters:: t (int) – timestamp utc seconds since epoch
Returns:: quarter. in range[1..4], -1 if invalid time
Return type:: int

static region_id_list() → list[str]¶

Returns a list over predefined Olson time-zone identifiers

Notes

the list is currently static and reflects tz-rules approx as of 2014

time(*args, **kwargs)¶

Overloaded function.

time(self: shyft.time_series.Calendar, YMDhms: shyft.time_series.YMDhms) -> shyft.time_series.time

convert calendar coordinates into time using the calendar time-zone

Parameters:: YMDhms (YMDhms) – calendar cooordinate structure containg year,month,day, hour,minute,second
Returns:: timestamp. timestamp as in seconds utc since epoch
Return type:: int

time(self: shyft.time_series.Calendar, YWdhms: shyft.time_series.YWdhms) -> shyft.time_series.time

convert calendar iso week coordinates structure into time using the calendar time-zone

Parameters:: YWdhms (YWdhms) – structure containg iso specification calendar coordinates
Returns:: timestamp. timestamp as in seconds utc since epoch
Return type:: int

time(self: shyft.time_series.Calendar, Y: typing.SupportsInt | typing.SupportsIndex, M: typing.SupportsInt | typing.SupportsIndex = 1, D: typing.SupportsInt | typing.SupportsIndex = 1, h: typing.SupportsInt | typing.SupportsIndex = 0, m: typing.SupportsInt | typing.SupportsIndex = 0, s: typing.SupportsInt | typing.SupportsIndex = 0, us: typing.SupportsInt | typing.SupportsIndex = 0) -> shyft.time_series.time

convert calendar coordinates into time using the calendar time-zone

Parameters:

Y (int) – Year
M (int) – Month [1..12], default=1
D (int) – Day [1..31], default=1
h (int) – hour [0..23], default=0
m (int) – minute [0..59], default=0
s (int) – second [0..59], default=0
us (int) – micro second[0..999999], default=0

Returns:

timestamp. timestamp as in seconds utc since epoch

Return type:

convert calendar iso week coordinates into time using the calendar time-zone

Parameters:

Y (int) – ISO Year
W (int) – ISO Week [1..54], default=1
wd (int) – week_day [1..7]=[mo..su], default=1
h (int) – hour [0..23], default=0
m (int) – minute [0..59], default=0
s (int) – second [0..59], default=0
us (int) – micro second[0..999999], default=0

Returns:

timestamp. timestamp as in seconds utc since epoch

Return type:

to_string(*args, **kwargs)¶

Overloaded function.

to_string(self: shyft.time_series.Calendar, time: shyft.time_series.time) -> str

convert time t to readable iso standard string taking: the current calendar properties, including timezone into account

Parameters:: utctime (time) – seconds utc since epoch
Returns:: iso time string. iso standard formatted string,including tz info
Return type:: str

to_string(self: shyft.time_series.Calendar, utcperiod: shyft.time_series.UtcPeriod) -> str

convert utcperiod p to readable string taking current calendar properties, including timezone into account

Parameters:: utcperiod (UtcPeriod) – An UtcPeriod object
Returns:: period-string. [start..end), iso standard formatted string,including tz info
Return type:: str

trim(self: shyft.time_series.Calendar, t: shyft.time_series.time, delta_t: shyft.time_series.time) → shyft.time_series.time¶

Round time t down to the nearest calendar time-unit delta_t, taking the calendar time-zone and dst into account.

Parameters:

t (time) – timestamp utc seconds since epoch
delta_t (time) – timestep in seconds, with semantic interpretation of Calendar.{DAY|WEEK|MONTH|YEAR}

Returns:

new trimmed timestamp, seconds utc since epoch

Return type:

Notes

ref to related functions .add(t,delta_t,n),.diff_units(t1,t2,delta_t)

property tz_info¶

keeping the time-zone name, utc-offset and DST rules (if any)

Type:: TzInfo

Class UtcPeriod¶

class shyft.time_series.UtcPeriod(*args, **kwargs)¶

Bases: pybind11_object

UtcPeriod defines the open utc-time range [start..end) where end is required to be equal or greater than start

Overloaded function.

__init__(self: shyft.time_series.UtcPeriod) -> None
__init__(self: shyft.time_series.UtcPeriod, start: shyft.time_series.time, end: shyft.time_series.time) -> None

Create utcperiod given start and end

__init__(self: shyft.time_series.UtcPeriod, start: typing.SupportsInt | typing.SupportsIndex, end: typing.SupportsInt | typing.SupportsIndex) -> None

Create utcperiod given start and end

__init__(self: shyft.time_series.UtcPeriod, start: typing.SupportsFloat | typing.SupportsIndex, end: typing.SupportsFloat | typing.SupportsIndex) -> None

Create utcperiod given start and end

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.UtcPeriod) -> None
__init__(self: shyft.time_series.UtcPeriod, start: shyft.time_series.time, end: shyft.time_series.time) -> None

Create utcperiod given start and end

__init__(self: shyft.time_series.UtcPeriod, start: typing.SupportsInt | typing.SupportsIndex, end: typing.SupportsInt | typing.SupportsIndex) -> None

Create utcperiod given start and end

__init__(self: shyft.time_series.UtcPeriod, start: typing.SupportsFloat | typing.SupportsIndex, end: typing.SupportsFloat | typing.SupportsIndex) -> None

Create utcperiod given start and end

contains(*args, **kwargs)¶

Overloaded function.

contains(self: shyft.time_series.UtcPeriod, t: shyft.time_series.time) -> bool

returns true if time t is contained in this utcperiod

contains(self: shyft.time_series.UtcPeriod, t: typing.SupportsInt | typing.SupportsIndex) -> bool

returns true if time t is contained in this utcperiod

contains(self: shyft.time_series.UtcPeriod, p: shyft.time_series.UtcPeriod) -> bool

returns true if utcperiod p is contained in this utcperiod

diff_units(*args, **kwargs)¶

Overloaded function.

diff_units(self: shyft.time_series.UtcPeriod, calendar: shyft.time_series.Calendar, delta_t: shyft.time_series.time) -> int

Calculate the distance from start to end of UtcPeriod in specified units, taking dst into account if observed The function takes calendar semantics when delta_t is calendar::DAY,WEEK,MONTH,YEAR, and in addition also dst if observed. e.g. the diff_units of calendar::DAY over summer->winter shift is 1, even if the number of hours during those days are 23 and 25 summer and winter transition respectively

Parameters:

calendar (calendar) – shyft calendar
delta_t (time) – timestep in seconds, with semantic interpretation of DAY,WEEK,MONTH,YEAR

Returns:

n_units. number of units in UtcPeriod

Return type:

int

diff_units(self: shyft.time_series.UtcPeriod, calendar: shyft.time_series.Calendar, delta_t: typing.SupportsInt | typing.SupportsIndex) -> int

Parameters:

calendar (calendar) – shyft calendar
delta_t (int) – timestep in seconds, with semantic interpretation of DAY,WEEK,MONTH,YEAR

Returns:

n_units. number of units in UtcPeriod

Return type:

int

property end¶

Defines the end of the period, not inclusive

Type:: time

intersection(self: shyft.time_series.UtcPeriod, rhs: shyft.time_series.UtcPeriod) → shyft.time_series.UtcPeriod¶

Returns the intersection of two periods if there is an intersection, the resulting period will be .valid() and .timespan()>0 If there is no intersection, an empty not .valid() period is returned

Parameters:

self (UtcPeriod) – 1st UtcPeriod argument
rhs (UtcPeriod) – 2nd UtcPeriod argument

Returns:

intersection. The computed intersection, or an empty not .valid() UtcPeriod

Return type:

UtcPeriod

overlaps(self: shyft.time_series.UtcPeriod, p: shyft.time_series.UtcPeriod) → bool¶: returns true if period p overlaps this utcperiod

property start¶

Defines the start of the period, inclusive

Type:: time

timespan(self: shyft.time_series.UtcPeriod) → shyft.time_series.time¶: returns end-start, the timespan of the period

to_string(self: shyft.time_series.UtcPeriod) → str¶: A readable representation in UTC

trim(*args, **kwargs)¶

Overloaded function.

trim(self: shyft.time_series.UtcPeriod, calendar: shyft.time_series.Calendar, delta_t: shyft.time_series.time, trim_policy: shyft.time_series.trim_policy = <trim_policy.TRIM_IN: 0>) -> shyft.time_series.UtcPeriod

Round UtcPeriod up or down to the nearest calendar time-unit delta_t taking the calendar time-zone and dst into account

Parameters:

calendar (calendar) – shyft calendar
delta_t (time) – timestep in seconds, with semantic interpretation of Calendar.(DAY,WEEK,MONTH,YEAR)
trim_policy (trim_policy) – TRIM_IN if rounding period inwards, else rounding outwards

Returns:

trimmed_UtcPeriod. new trimmed UtcPeriod

Return type:

UtcPeriod

trim(self: shyft.time_series.UtcPeriod, calendar: shyft.time_series.Calendar, delta_t: typing.SupportsInt | typing.SupportsIndex, trim_policy: shyft.time_series.trim_policy = <trim_policy.TRIM_IN: 0>) -> shyft.time_series.UtcPeriod

Round UtcPeriod up or down to the nearest calendar time-unit delta_t taking the calendar time-zone and dst into account

Parameters:

calendar (calendar) – shyft calendar
delta_t (int) – timestep in seconds, with semantic interpretation of Calendar.(DAY,WEEK,MONTH,YEAR)
trim_policy (trim_policy) – TRIM_IN if rounding period inwards, else rounding outwards

Returns:

trimmed_UtcPeriod. new trimmed UtcPeriod

Return type:

UtcPeriod

valid(self: shyft.time_series.UtcPeriod) → bool¶: returns true if start<=end otherwise false

Class UtcTimeVector¶

shyft.time_series.UtcTimeVector¶: alias of list

Time series¶

Elements in this category are the actual time series.

Class TimeAxis¶

class shyft.time_series.TimeAxis(*args, **kwargs)¶

Bases: pybind11_object

A time-axis is a set of ordered non-overlapping periods, and TimeAxis provides the most generic implementation of this. The internal representation is selected based on provided parameters to the constructor. The internal representation is one of TimeAxis FixedDeltaT CalendarDelataT or ByPoints. The internal representation type and corresponding realizations are available as properties.

Notes

The internal representation can be one of TimeAxisCalendarDeltaT,TimeAxisFixedDeltaT,TimeAxisByPoints

Overloaded function.

__init__(self: shyft.time_series.TimeAxis) -> None
__init__(self: shyft.time_series.TimeAxis, calendar_dt: shyft.time_series.TimeAxisCalendarDeltaT) -> None

create a time-axis from a calendar time-axis

Parameters:: calendar_dt (TimeAxisCalendarDeltaT) – existing calendar time-axis

__init__(self: shyft.time_series.TimeAxis, fixed_dt: shyft.time_series.TimeAxisFixedDeltaT) -> None

create a time-axis from a a fixed delta-t time-axis

Parameters:: fixed_dt (TimeAxisFixedDeltaT) – existing fixed delta-t time-axis

__init__(self: shyft.time_series.TimeAxis, point_dt: shyft.time_series.TimeAxisByPoints) -> None

create a time-axis from a a by points time-axis

Parameters:: point_dt (TimeAxisByPoints) – existing by points time-axis

__init__(self: shyft.time_series.TimeAxis, start: shyft.time_series.time, delta_t: shyft.time_series.time, n: typing.SupportsInt | typing.SupportsIndex) -> None

creates a time-axis with n intervals, fixed delta_t, starting at start

Parameters:

start (utctime) – utc-time 1970 utc based
delta_t (utctime) – number of seconds delta-t, length of periods in the time-axis
n (int) – number of periods in the time-axis

__init__(self: shyft.time_series.TimeAxis, calendar: shyft.time_series.Calendar, start: shyft.time_series.time, delta_t: shyft.time_series.time, n: typing.SupportsInt | typing.SupportsIndex) -> None

creates a calendar time-axis

Parameters:

calendar (Calendar) – specifies the calendar to be used, keeps the time-zone and dst-arithmetic rules
start (utctime) – utc-time 1970 utc based
delta_t (utctime) – number of seconds delta-t, length of periods in the time-axis
n (int) – number of periods in the time-axis

__init__(self: shyft.time_series.TimeAxis, time_points: typing.Annotated[numpy.typing.ArrayLike, numpy.float64], t_end: shyft.time_series.time) -> None

creates a time-axis by specifying the time_points and t-end of the last interval

Parameters:

time_points (numpy.array[float|int]) – ordered set of unique utc-time points, the start of each consecutive period
t_end (time) – the end of the last period in time-axis, utc-time 1970 utc based, must be > time_points[-1]

__init__(self: shyft.time_series.TimeAxis, time_points: collections.abc.Sequence[shyft.time_series.time], t_end: shyft.time_series.time) -> None

creates a time-axis by specifying the time_points and t-end of the last interval

Parameters:

time_points (UtcTimeVector) – ordered set of unique utc-time points, the start of each consecutive period
t_end (time) – the end of the last period in time-axis, utc-time 1970 utc based, must be > time_points[-1]

__init__(self: shyft.time_series.TimeAxis, time_points: typing.Annotated[numpy.typing.ArrayLike, numpy.float64]) -> None

create a time-axis supplying n+1 points to define n intervals

Parameters:: time_points (numpy.array[float|int]) – ordered set of unique utc-time points, 0..n-2:the start of each consecutive period,n-1: end of last

__init__(self: shyft.time_series.TimeAxis, time_points: collections.abc.Sequence[shyft.time_series.time]) -> None

create a time-axis supplying n+1 points to define n intervals

Parameters:: time_points (UtcTimeVector) – ordered set of unique utc-time points, 0..n-2:the start of each consecutive period,n-1: end of last period

__call__(self: shyft.time_series.TimeAxis, i: SupportsInt | SupportsIndex) → shyft.time_series.UtcPeriod¶

Parameters:: i (int) – the i’th period, 0..n-1
Returns:: period. the i’th period of the time-axis
Return type:: UtcPeriod

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.TimeAxis) -> None
__init__(self: shyft.time_series.TimeAxis, calendar_dt: shyft.time_series.TimeAxisCalendarDeltaT) -> None

create a time-axis from a calendar time-axis

Parameters:: calendar_dt (TimeAxisCalendarDeltaT) – existing calendar time-axis

__init__(self: shyft.time_series.TimeAxis, fixed_dt: shyft.time_series.TimeAxisFixedDeltaT) -> None

create a time-axis from a a fixed delta-t time-axis

Parameters:: fixed_dt (TimeAxisFixedDeltaT) – existing fixed delta-t time-axis

__init__(self: shyft.time_series.TimeAxis, point_dt: shyft.time_series.TimeAxisByPoints) -> None

create a time-axis from a a by points time-axis

Parameters:: point_dt (TimeAxisByPoints) – existing by points time-axis

__init__(self: shyft.time_series.TimeAxis, start: shyft.time_series.time, delta_t: shyft.time_series.time, n: typing.SupportsInt | typing.SupportsIndex) -> None

creates a time-axis with n intervals, fixed delta_t, starting at start

Parameters:

start (utctime) – utc-time 1970 utc based
delta_t (utctime) – number of seconds delta-t, length of periods in the time-axis
n (int) – number of periods in the time-axis

__init__(self: shyft.time_series.TimeAxis, calendar: shyft.time_series.Calendar, start: shyft.time_series.time, delta_t: shyft.time_series.time, n: typing.SupportsInt | typing.SupportsIndex) -> None

creates a calendar time-axis

Parameters:

calendar (Calendar) – specifies the calendar to be used, keeps the time-zone and dst-arithmetic rules
start (utctime) – utc-time 1970 utc based
delta_t (utctime) – number of seconds delta-t, length of periods in the time-axis
n (int) – number of periods in the time-axis

__init__(self: shyft.time_series.TimeAxis, time_points: typing.Annotated[numpy.typing.ArrayLike, numpy.float64], t_end: shyft.time_series.time) -> None

creates a time-axis by specifying the time_points and t-end of the last interval

Parameters:

time_points (numpy.array[float|int]) – ordered set of unique utc-time points, the start of each consecutive period
t_end (time) – the end of the last period in time-axis, utc-time 1970 utc based, must be > time_points[-1]

__init__(self: shyft.time_series.TimeAxis, time_points: collections.abc.Sequence[shyft.time_series.time], t_end: shyft.time_series.time) -> None

creates a time-axis by specifying the time_points and t-end of the last interval

Parameters:

time_points (UtcTimeVector) – ordered set of unique utc-time points, the start of each consecutive period
t_end (time) – the end of the last period in time-axis, utc-time 1970 utc based, must be > time_points[-1]

__init__(self: shyft.time_series.TimeAxis, time_points: typing.Annotated[numpy.typing.ArrayLike, numpy.float64]) -> None

create a time-axis supplying n+1 points to define n intervals

Parameters:: time_points (numpy.array[float|int]) – ordered set of unique utc-time points, 0..n-2:the start of each consecutive period,n-1: end of last

__init__(self: shyft.time_series.TimeAxis, time_points: collections.abc.Sequence[shyft.time_series.time]) -> None

create a time-axis supplying n+1 points to define n intervals

Parameters:: time_points (UtcTimeVector) – ordered set of unique utc-time points, 0..n-2:the start of each consecutive period,n-1: end of last period

property calendar_dt¶

The calendar dt representation(if active)

Type:: TimeAxisCalendarDeltaT

empty(self: shyft.time_series.TimeAxis) → bool¶

true if empty time-axis

Returns:: empty. true if empty time-axis
Return type:: bool

property fixed_dt¶

The fixed dt representation (if active)

Type:: TimeAxisFixedDeltaT

index_of(self: shyft.time_series.TimeAxis, t: shyft.time_series.time, ix_hint: SupportsInt | SupportsIndex = 18446744073709551615) → int¶

Parameters:: t (utcime) – utctime in seconds 1970.01.01
Returns:: index. the index of the time-axis period that contains t, npos if outside range
Return type:: int

merge(self: shyft.time_series.TimeAxis, other: shyft.time_series.TimeAxis) → shyft.time_series.TimeAxis¶

Returns a new time-axis that contains the union of time-points/periods of the two time-axis. If there is a gap between, it is filled merge with empty time-axis results into the other time-axis

Parameters:: other (TimeAxis) – The other time-axis to merge with
Returns:: merge_result. the resulting merged time-axis
Return type:: TimeAxis

open_range_index_of(self: shyft.time_series.TimeAxis, t: shyft.time_series.time, ix_hint: SupportsInt | SupportsIndex = 18446744073709551615) → int¶

returns the index that contains t, or is before t

Parameters:

t (int) – utctime in seconds 1970.01.01
ix_hint (int) – index-hint to make search in point-time-axis faster

Returns:

index. the index the time-axis period that contains t, npos if before first period n-1, if t is after last period

Return type:

int

period(self: shyft.time_series.TimeAxis, i: SupportsInt | SupportsIndex) → shyft.time_series.UtcPeriod¶

Parameters:: i (int) – the i’th period, 0..n-1
Returns:: period. the i’th period of the time-axis
Return type:: UtcPeriod

property point_dt¶

point_dt representation(if active)

Type:: TimeAxisByPoints

size(self: shyft.time_series.TimeAxis) → int¶

Returns:

number of periods in time-axis

Return type:

int

slice(self: shyft.time_series.TimeAxis, start: SupportsInt | SupportsIndex, n: SupportsInt | SupportsIndex) → shyft.time_series.TimeAxis¶

returns slice of time-axis as a new time-axis

Parameters:

start (int) – first interval to include
n (int) – number of intervals to include

Returns:

time-axis. A new time-axis with the specified slice

Return type:

TimeAxis

time(self: shyft.time_series.TimeAxis, i: SupportsInt | SupportsIndex) → shyft.time_series.time¶

Parameters:: i (int) – the i’th period, 0..n-1
Returns:: utctime. the start(utctime) of the i’th period of the time-axis
Return type:: int

property time_points¶

property time_points_double¶

property timeaxis_type¶

describes what time-axis representation type this is,e.g (fixed|calendar|point)_dt

Type:: TimeAxisType

total_period(self: shyft.time_series.TimeAxis) → shyft.time_series.UtcPeriod¶

Returns:: total_period. the period that covers the entire time-axis
Return type:: UtcPeriod

Class TimeSeries¶

class shyft.time_series.TimeSeries(*args, **kwargs)¶

Bases: pybind11_object

A time-series providing mathematical and statistical operations and functionality.

A time-series can be an expression, or a concrete point time-series. All time-series do have a time-axis, values, and a point fx policy. The value, f(t) outside the time-axis is nan Operations between time-series, e.g. a+b, respects the mathematical nan op something equals nan

The time-series can provide a value for all the intervals, and the point_fx policy defines how the values should be interpreted:

POINT_INSTANT_VALUE(linear):

the point value is valid at the start of the period, linear between points -or flat extended value if the next point is nan. typical for state-variables, like water-level, temperature measured at 12:00 etc.

POINT_AVERAGE_VALUE(stair-case):

the point represents an average or constant value over the period typical for model-input and results, precipitation mm/h, discharge m^3/s

Examples:

>>> import numpy as np
>>> from shyft.time_series import Calendar,deltahours,TimeAxis,TimeSeries,POINT_AVERAGE_VALUE as fx_avg
>>>
>>> utc = Calendar()  # ensure easy consistent explicit handling of calendar and time
>>> ta = TimeAxis(utc.time(2016, 9, 1, 8, 0, 0), deltahours(1), 10)  # create a time-axis to use
>>> a = TimeSeries(ta, np.linspace(0, 10, num=len(ta)), fx_avg)
>>> b = TimeSeries(ta, np.linspace(0,  1, num=len(ta)), fx_avg)
>>> c = a + b*3.0  # c is now an expression, time-axis is the overlap of a and b, lazy evaluation
>>> c_values = c.values  # compute and extract the values, as numpy array
>>> c_evaluated=c.evaluate() # computes the expression, return a new concrete point-ts equal to the expression
>>>
>>> # Calculate data for new time-points
>>> value_1 = a(utc.time(2016, 9, 1, 8, 30)) # calculates value at a given time
>>> ta_target = TimeAxis(utc.time(2016, 9, 1, 7, 30), deltahours(1), 12)  # create a target time_axis
>>> ts_new = a.average(ta_target) # new time-series with values on ta_target
>>>

TimeSeries can also be symbolic, that is, have urls, that is resolved later, serverside using the DtsServer The TimeSeries functionality includes:

construction: TimeSeries(time-axis,values,point_interpretation), TimeSeries(ts_url), TimeSeries(ts_url,ts_fragment)

mutating points: set() fill() scale_by() merge_points()

combining/extending: extend() use_time_axis_from()

resampling: average() accumulate() time_shift() use_time_axis_from() use_time_axis()

f(x): transform() abs() derivative() integral() min() max() pow() log()

boolean f(x): inside() , creates mask(1.0,0.0 series) you can use for math/filtering expressions

statistics: statistics() kling_gupta() nash_sutcliffe()

filtering: convolve_w() krls_interpolation()

quality and correction: quality_and_self_correction() quality_and_ts_correction() , min-max limits, replace by surrounding points or replacement ts

bit-encoded: decode()

stacking and percentiles: stack()

n-ary operations: TsVector.sum() TsVector.forecast_merge() , operations on TsVector that results in a TimeSeries

hydrology domain: rating_curve() bucket_to_hourly() ice_packing() ice_packing_recession() shyft.time_series.create_glacier_melt_ts_m3s()

Other useful classes to look at: TimeAxis , Calendar , TsVector , point_interpretation_policy

Please check the extensive test suite, notebooks, examples and time_series for usage.

Overloaded function.

__init__(self: shyft.time_series.TimeSeries) -> None
__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxis, fill_value: typing.SupportsFloat | typing.SupportsIndex, point_fx: shyft.time_series.point_interpretation_policy) -> None

construct a time-series with time-axis ta, specified fill-value, and point interpretation policy point_fx

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxis, values: typing.Annotated[numpy.typing.ArrayLike, numpy.float64], point_fx: shyft.time_series.point_interpretation_policy) -> None

Construct a TimeSeries object from a time axis, NumPy array of values, and point interpretation policy

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxis, values: collections.abc.Sequence[typing.SupportsFloat | typing.SupportsIndex], point_fx: shyft.time_series.point_interpretation_policy) -> None

construct a timeseries timeaxis ta with corresponding values, and point interpretation policy point_fx

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxisFixedDeltaT, fill_value: typing.SupportsFloat | typing.SupportsIndex, point_fx: shyft.time_series.point_interpretation_policy) -> None

construct a time-series with time-axis ta, specified fill-value, and point interpretation policy point_fx

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxisFixedDeltaT, values: typing.Annotated[numpy.typing.ArrayLike, numpy.float64], point_fx: shyft.time_series.point_interpretation_policy) -> None

Construct a TimeSeries object from a time axis, NumPy array of values, and point interpretation policy

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxisFixedDeltaT, values: collections.abc.Sequence[typing.SupportsFloat | typing.SupportsIndex], point_fx: shyft.time_series.point_interpretation_policy) -> None

construct a timeseries timeaxis ta with corresponding values, and point interpretation policy point_fx

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxisByPoints, fill_value: typing.SupportsFloat | typing.SupportsIndex, point_fx: shyft.time_series.point_interpretation_policy) -> None

construct a time-series with time-axis ta, specified fill-value, and point interpretation policy point_fx

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxisByPoints, values: typing.Annotated[numpy.typing.ArrayLike, numpy.float64], point_fx: shyft.time_series.point_interpretation_policy) -> None

Construct a TimeSeries object from a time axis, NumPy array of values, and point interpretation policy

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxisByPoints, values: collections.abc.Sequence[typing.SupportsFloat | typing.SupportsIndex], point_fx: shyft.time_series.point_interpretation_policy) -> None

construct a timeseries timeaxis ta with corresponding values, and point interpretation policy point_fx

__init__(self: shyft.time_series.TimeSeries, core_result_ts: shyft.time_series.TsFixed) -> None

construct a time-series from a shyft core time-series, to ease working with core-time-series in user-interface/scripting

__init__(self: shyft.time_series.TimeSeries, pattern: collections.abc.Sequence[typing.SupportsFloat | typing.SupportsIndex], dt: shyft.time_series.time, ta: shyft.time_series.TimeAxis) -> None

construct a repeated pattern time-series given a equally spaced dt pattern and a time-axis ta

Parameters:

pattern (DoubleVector) – a list of numbers giving the pattern
dt (int) – number of seconds between each of the pattern-values
ta (TimeAxis) – time-axis that forms the resulting time-series time-axis

__init__(self: shyft.time_series.TimeSeries, pattern: collections.abc.Sequence[typing.SupportsFloat | typing.SupportsIndex], dt: shyft.time_series.time, t0: shyft.time_series.time, ta: shyft.time_series.TimeAxis) -> None

construct a time-series given a equally spaced dt pattern, starting at t0, and a time-axis ta

__init__(self: shyft.time_series.TimeSeries, ts_id: str) -> None

constructs a bind-able ts, providing a symbolic possibly unique id that at a later time can be bound, using the .bind(ts) method to concrete values if the ts is used as ts, like size(),.value(),time() before it is bound, then a runtime-exception is raised

Parameters:: ts_id (str) – url-like identifier for the time-series,notice that shyft://<container>/<path> is for shyft-internal store

__init__(self: shyft.time_series.TimeSeries, arg0: shyft.time_series.TimeSeries) -> None
__init__(self: shyft.time_series.TimeSeries, ts_id: str, bts: shyft.time_series.TimeSeries) -> None

constructs a ready bound ts, providing a symbolic possibly unique id that at a later time can be used to correlate with back-end store

Parameters:

ts_id (str) – url-type of id, notice that shyft://<container>/<path> is for shyft-internal store
bts (TimeSeries) – A time-series, that is either a concrete ts, or an expression that can be evaluated to form a concrete ts

__call__(self: shyft.time_series.TimeSeries, t: shyft.time_series.time) → float¶: return the f(t) value for the time-series

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.TimeSeries) -> None
__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxis, fill_value: typing.SupportsFloat | typing.SupportsIndex, point_fx: shyft.time_series.point_interpretation_policy) -> None

construct a time-series with time-axis ta, specified fill-value, and point interpretation policy point_fx

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxis, values: typing.Annotated[numpy.typing.ArrayLike, numpy.float64], point_fx: shyft.time_series.point_interpretation_policy) -> None

Construct a TimeSeries object from a time axis, NumPy array of values, and point interpretation policy

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxis, values: collections.abc.Sequence[typing.SupportsFloat | typing.SupportsIndex], point_fx: shyft.time_series.point_interpretation_policy) -> None

construct a timeseries timeaxis ta with corresponding values, and point interpretation policy point_fx

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxisFixedDeltaT, fill_value: typing.SupportsFloat | typing.SupportsIndex, point_fx: shyft.time_series.point_interpretation_policy) -> None

construct a time-series with time-axis ta, specified fill-value, and point interpretation policy point_fx

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxisFixedDeltaT, values: typing.Annotated[numpy.typing.ArrayLike, numpy.float64], point_fx: shyft.time_series.point_interpretation_policy) -> None

Construct a TimeSeries object from a time axis, NumPy array of values, and point interpretation policy

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxisFixedDeltaT, values: collections.abc.Sequence[typing.SupportsFloat | typing.SupportsIndex], point_fx: shyft.time_series.point_interpretation_policy) -> None

construct a timeseries timeaxis ta with corresponding values, and point interpretation policy point_fx

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxisByPoints, fill_value: typing.SupportsFloat | typing.SupportsIndex, point_fx: shyft.time_series.point_interpretation_policy) -> None

construct a time-series with time-axis ta, specified fill-value, and point interpretation policy point_fx

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxisByPoints, values: typing.Annotated[numpy.typing.ArrayLike, numpy.float64], point_fx: shyft.time_series.point_interpretation_policy) -> None

Construct a TimeSeries object from a time axis, NumPy array of values, and point interpretation policy

__init__(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxisByPoints, values: collections.abc.Sequence[typing.SupportsFloat | typing.SupportsIndex], point_fx: shyft.time_series.point_interpretation_policy) -> None

construct a timeseries timeaxis ta with corresponding values, and point interpretation policy point_fx

__init__(self: shyft.time_series.TimeSeries, core_result_ts: shyft.time_series.TsFixed) -> None

construct a time-series from a shyft core time-series, to ease working with core-time-series in user-interface/scripting

__init__(self: shyft.time_series.TimeSeries, pattern: collections.abc.Sequence[typing.SupportsFloat | typing.SupportsIndex], dt: shyft.time_series.time, ta: shyft.time_series.TimeAxis) -> None

construct a repeated pattern time-series given a equally spaced dt pattern and a time-axis ta

Parameters:

pattern (DoubleVector) – a list of numbers giving the pattern
dt (int) – number of seconds between each of the pattern-values
ta (TimeAxis) – time-axis that forms the resulting time-series time-axis

__init__(self: shyft.time_series.TimeSeries, pattern: collections.abc.Sequence[typing.SupportsFloat | typing.SupportsIndex], dt: shyft.time_series.time, t0: shyft.time_series.time, ta: shyft.time_series.TimeAxis) -> None

construct a time-series given a equally spaced dt pattern, starting at t0, and a time-axis ta

__init__(self: shyft.time_series.TimeSeries, ts_id: str) -> None

Parameters:: ts_id (str) – url-like identifier for the time-series,notice that shyft://<container>/<path> is for shyft-internal store

__init__(self: shyft.time_series.TimeSeries, arg0: shyft.time_series.TimeSeries) -> None
__init__(self: shyft.time_series.TimeSeries, ts_id: str, bts: shyft.time_series.TimeSeries) -> None

constructs a ready bound ts, providing a symbolic possibly unique id that at a later time can be used to correlate with back-end store

Parameters:

ts_id (str) – url-type of id, notice that shyft://<container>/<path> is for shyft-internal store
bts (TimeSeries) – A time-series, that is either a concrete ts, or an expression that can be evaluated to form a concrete ts

abs(self: shyft.time_series.TimeSeries) → shyft.time_series.TimeSeries¶

create a new ts, abs(py::self

Returns:: ts. a new time-series expression, that will provide the abs-values of self.values
Return type:: TimeSeries

accumulate(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxis) → shyft.time_series.TimeSeries¶

create a new ts where each i-th value :: | integral f(t)*dt, from t0..ti

given the specified time-axis ta, and the point interpretation.

Parameters:: ta (TimeAxis) – time-axis that specifies the periods where accumulated integral is applied
Returns:: ts. a new time-series expression, that will provide the accumulated values when requested
Return type:: TimeSeries

Notes: In contrast to integral() , accumulate has a point-instant interpretation. As values() gives the start values of each interval, see TimeSeries , accumulate(ta).values provides the accumulation over the intervals [t0..t0, t0..t1, t0..t2, …], thus values[0] is always 0.)

average(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxis) → shyft.time_series.TimeSeries¶

create a new ts that is the true average of self over the specified time-axis ta. Notice that same definition as for integral applies; non-nan parts goes into the average

Parameters:: ta (TimeAxis) – time-axis that specifies the periods where true-average is applied
Returns:: ts. a new time-series expression, that will provide the true-average when requested
Return type:: TimeSeries

Notes

the self point interpretation policy is used when calculating the true average

bind(self: shyft.time_series.TimeSeries, bts: shyft.time_series.TimeSeries) → None¶

given that this ts,self, is a bind-able ts (aref_ts) and that bts is a concrete point TimeSeries, or something that can be evaluated to one, use it as representation for the values of this ts. Other related functions are find_ts_bind_info,TimeSeries(‘a-ref-string’)

Parameters:: bts (TimeSeries) – a concrete point ts, or ready-to-evaluate expression, with time-axis, values and fx_policy

Notes

raises runtime_error if any of preconditions is not true

bind_done(self: shyft.time_series.TimeSeries, skip_check: bool = False) → None¶

after bind operations on unbound time-series of an expression is done, call bind_done() to prepare the expression for use Other related methods are .bind(), .find_ts_bind_info() and needs_bind().

Parameters:: skip_check (bool) – If set true this function assumes all siblings are bound, as pr. standard usage pattern for the mentioned functions

Notes

Usually this is done automatically by the dtss framework, but if not using dtss

this function is needed after the symbolic ts’s are bound

bucket_to_hourly(self: shyft.time_series.TimeSeries, start_hour_utc: SupportsInt | SupportsIndex, bucket_emptying_limit: SupportsFloat | SupportsIndex) → shyft.time_series.TimeSeries¶

Precipitation bucket measurements have a lot of tweaks that needs to be resolved, including negative variations over the day due to faulty temperature-dependent volume/weight sensors attached.

A precipitation bucket accumulates precipitation, so the readings should be strictly increasing by time, until the bucket is emptied (full, or as part of maintenance).

The goal for the bucket_to_hourly algorithm is to provide hourly precipitation, based on some input signal that usually is hourly(averaging is used if not hourly).

The main strategy is to use 24 hour differences (typically at hours in a day where the temperature is low, like early in the morning.), to adjust the hourly volume.

Differences in periods of 24hour are distributed on all positive hourly events, the negative derivatives are zeroed out, so that the hourly result for each 24 hour is steady increasing, and equal to the difference of the 24hour area.

The derivative is then used to compute the hourly precipitation rate in mm/h

Parameters:

start_hour_utc (int) – valid range [0..24], usually set to early morning(low-stable temperature)
bucket_emptying_limit (float) – a negative number, range[-oo..0), limit of when to detect an emptying of a bucket in the unit of the measurements series

Returns:

ts. a new hourly rate ts, that transforms the accumulated series, compensated for the described defects

Return type:

clone_expression(self: shyft.time_series.TimeSeries) → shyft.time_series.TimeSeries¶

create a copy of the ts-expressions, except for the bound payload of the reference ts. For the reference terminals, those with ts_id, only the ts_id is copied. Thus, to re-evaluate the expression, those have to be bound.

Notes

this function is only useful in context where multiple bind/rebind while keeping the expression is needed.

Returns:: semantic_clone. returns a copy of the ts, except for the payload at reference/symbolic terminals, where only ts_id is copied
Return type:: TsVector

compress(self: shyft.time_series.TimeSeries, accuracy: SupportsFloat | SupportsIndex) → shyft.time_series.TimeSeries¶

Compress by reducing number of points sufficient to represent the same f(t) within accuracy. The returned ts is a new ts with break-point/variable interval representation. note: lazy binding expressions(server-side eval) is not yet supported.

Parameters:: () (accuracy) – if v[i]-v[i+1] <accuracy the v[i+1] is dropped
Returns:: compressed_ts. a new compressed within accuracy time-series
Return type:: TimeSeries

compress_size(self: shyft.time_series.TimeSeries, accuracy: SupportsFloat | SupportsIndex) → int¶

Compute number of points this time-series could be reduced to if calling ts.compress(accuracy). note: lazy binding expressions(server-side eval) is not yet supported.

Parameters:: () (accuracy) – if v[i]-v[i+1] <accuracy the v[i+1] is dropped
Returns:: compressed_size. number of distinct point needed to represent the time-series
Return type:: int

convolve_w(self: shyft.time_series.TimeSeries, weights: collections.abc.Sequence[SupportsFloat | SupportsIndex], policy: shyft.time_series.convolve_policy) → shyft.time_series.TimeSeries¶

create a new ts that is the convolved ts with the given weights list

Parameters:

weights (DoubleVector) – the weights profile, use DoubleVector.from_numpy(…) to create these. It’s the callers responsibility to ensure the sum of weights are 1.0
policy (convolve_policy) – (USE_NEAREST|USE_ZERO|USE_NAN + BACKWARD|FORWARD|CENTER). Specifies how to handle boundary values

Returns:

ts. a new time-series that is evaluated on request to the convolution of self

Return type:

decode(self: shyft.time_series.TimeSeries, start_bit: SupportsInt | SupportsIndex, n_bits: SupportsInt | SupportsIndex) → shyft.time_series.TimeSeries¶

Create an time-series that decodes the source using provided specification start_bit and n_bits. This function can typically be used to decode status-signals from sensors stored as binary encoded bits, using integer representation The floating point format allows up to 52 bits to be precisely stored as integer - thus there are restrictions to start_bit and n_bits accordingly. Practical sensors quality signals have like 32 bits of status information encoded If the value in source time-series is:

negative

nan

larger than 52 bits

Then nan is returned for those values

ts.decode(start_bit=1,n_bits=1) will return values [0,1,nan] similar: ts.decode(start_bit=1,n_bits=2) will return values [0,1,2,3,nan] etc..

Parameters:

start_bit (int) – where in the n-bits integer the value is stored, range[0..51]
n_bits (int) – how many bits are encoded, range[0..51], but start_bit +n_bits < 51

Returns:

decode_ts. Evaluated on demand decoded time-series

Return type:

derivative(self: shyft.time_series.TimeSeries, method: shyft.time_series.derivative_method = <derivative_method.DEFAULT: 0>) → shyft.time_series.TimeSeries¶

Compute the derivative of the ts, according to the method specified. For linear(POINT_INSTANT_VALUE), it is always the derivative of the straight line between points, - using nan for the interval starting at the last point until end of time-axis. Default for stair-case(POINT_AVERAGE_VALUE) is the average derivative over each time-step, - using 0 as rise for the first/last half of the intervals at the boundaries. here you can influence the method used, selecting .forward_diff, .backward_diff

Parameters:: method (derivative_method) – default value gives center/average derivative .(DEFAULT|FORWARD|BACKWARD|CENTER)
Returns:: derivative. The derivative ts
Return type:: TimeSeries

static deserialize(blob: shyft.time_series.ByteVector) → shyft.time_series.TimeSeries¶: convert a blob, as returned by .serialize() into a Timeseries

evaluate(self: shyft.time_series.TimeSeries) → shyft.time_series.TimeSeries¶

Forces evaluation of the expression, returns a new concrete time-series that is detached from the expression.

Returns:: ts. the evaluated copy of the expression that self represents
Return type:: TimeSeries

extend(self: shyft.time_series.TimeSeries, ts: shyft.time_series.TimeSeries, split_policy: shyft.time_series.extend_split_policy = <extend_split_policy.LHS_LAST: 0>, fill_policy: shyft.time_series.extend_fill_policy = <extend_fill_policy.FILL_NAN: 0>, split_at: shyft.time_series.time = time(0), fill_value: SupportsFloat | SupportsIndex = nan) → shyft.time_series.TimeSeries¶

create a new time-series that is self extended with ts

Parameters:

ts (TimeSeries) – time-series to extend self with, only values after both the start of self, and split_at is used
split_policy (extend_split_policy) – policy determining where to split between self and ts
fill_policy (extend_fill_policy) – policy determining how to fill any gap between self and ts
split_at (utctime) – time at which to split if split_policy == EPS_VALUE
fill_value (float) – value to fill any gap with if fill_policy == EPF_FILL

Returns:

extended_ts. a new time-series that is the extension of self with ts

Return type:

fill(self: shyft.time_series.TimeSeries, v: SupportsFloat | SupportsIndex) → None¶: fill all values with v

find_ts_bind_info(self: shyft.time_series.TimeSeries) → list[shyft.time_series.TsBindInfo]¶

recursive search through the expression that this ts represents, and return a list of TsBindInfo that can be used to inspect and possibly ‘bind’ to ts-values. see also related function bind()

Returns:: bind_info. A list of BindInfo where each entry contains a symbolic-ref and a ts that needs binding
Return type:: TsBindInfoVector

get(self: shyft.time_series.TimeSeries, i: SupportsInt | SupportsIndex) → shyft.time_series.Point¶: returns i-th point(t,v)

get_krls_predictor(self: shyft.time_series.TimeSeries, dt: shyft.time_series.time, gamma: SupportsFloat | SupportsIndex = 0.001, tolerance: SupportsFloat | SupportsIndex = 0.01, size: SupportsInt | SupportsIndex = 1000000) → shyft.time_series.KrlsRbfPredictor¶

Get a KRLS predictor trained on this time-series.

If you only want a interpolation of self use krls_interpolation instead, this method return the underlying predictor instance that can be used to generate mean-squared error estimates, or can be further trained on more data.

Notes

A predictor can only be generated for a bound time-series.

Parameters:

dt (float) – The time-step in seconds the underlying predictor is specified for. Note that this does not put a limit on time-axes used, but for best results it should be approximatly equal to the time-step of time-axes used with the predictor. In addition it should not be to long, else you will get poor results. Try to keep the dt less than a day, 3-8 hours is usually fine.
gamma (float (optional)) – Determines the width of the radial basis functions for the KRLS algorithm. Lower values mean wider basis functions, wider basis functions means faster computation but lower accuracy. Note that the tolerance parameter also affects speed and accurcy. A large value is around 1E-2, and a small value depends on the time step. By using values larger than 1E-2 the computation will probably take to long. Testing have reveled that 1E-3 works great for a time-step of 3 hours, while a gamma of 1E-2 takes a few minutes to compute. Use 1E-4 for a fast and tolerably accurate prediction. Defaults to 1E-3
tolerance (float (optional)) – The krls training tolerance. Lower values makes the prediction more accurate, but slower. This typically have less effect than gamma, but is usefull for tuning. Usually it should be either 0.01 or 0.001. Defaults to 0.01
size (int (optional)) – The size of the “memory” of the underlying predictor. The default value is usually enough. Defaults to 1000000.

Examples:

>>> import numpy as np
>>> import scipy.stats as stat
>>> from shyft.time_series import (
...     Calendar, utctime_now, deltahours,
...     TimeAxis, TimeSeries
... )
>>>
>>> cal = Calendar()
>>> t0 = utctime_now()
>>> dt = deltahours(1)
>>> n = 365*24  # one year
>>>
>>> # generate random bell-shaped data
>>> norm = stat.norm()
>>> data = np.linspace(0, 20, n)
>>> data = stat.norm(10).pdf(data) + norm.pdf(np.random.rand(*data.shape))
>>> # -----
>>> ta = TimeAxis(cal, t0, dt, n)
>>> ts = TimeSeries(ta, data)
>>>
>>> # create a predictor
>>> pred = ts.get_krls_predictor()
>>> total_mse = pred.predictor_mse(ts)  # compute mse relative to ts
>>> krls_ts = pred.predict(ta)  # generate a prediction, this is the result from ts.krls_interpolation
>>> krls_mse_ts = pred.mse_ts(ts, points=6)  # compute a mse time-series using 6 points around each sample

Returns:: krls_predictor. A KRLS predictor pre-trained once on self.
Return type:: KrlsRbfPredictor

Other related methods are:: shyft.time_series.TimeSeries.krls_interpolation()

get_time_axis(self: shyft.time_series.TimeSeries) → shyft.time_series.TimeAxis¶

ice_packing(self: shyft.time_series.TimeSeries, ip_params: shyft.time_series.IcePackingParameters, ipt_policy: shyft.time_series.ice_packing_temperature_policy) → shyft.time_series.TimeSeries¶

Create a binary time-series indicating whether ice-packing is occuring or not.

Note

self is interpreted and assumed to be a temperature time-series.

The ice packing detection is based on the mean temperature in a predetermined time window before the time-point of interrest (see IcePackingParameters.window. The algorithm determines there to be ice packing when the mean temperature is below a given threshold temperature (see IcePackingParameters.threshold_temp).

Parameters:

ip_param (IcePackingParameters) – Parameter container controlling the ice packing detection.
ipt_policy (ice_packing_temperature_policy) – Policy flags for determining how to deal with missing temperature values.

Returns:

ice_packing_ts. A time-series indicating wheter ice packing occurs or not

Return type:

Example:

>>> import numpy as np
>>> from shyft.time_series import (
...     IcePackingParameters, ice_packing_temperature_policy,
...     TimeAxis, TimeSeries, point_interpretation_policy, DoubleVector,
...     utctime_now, deltahours, deltaminutes,
... )
>>>
>>> t0 = utctime_now()
>>> dt = deltaminutes(15)
>>> n = 100
>>>
>>> # generate jittery data
>>> # - first descending from +5 to -5 then ascending back to +5
>>> # - include a NaN hole at the bottom of the V
>>> n_ = n if (n//2)*2 == n else n+1  # assure even
>>> data = np.concatenate((
...     np.linspace(5, -5, n_//2), np.linspace(-5, 5, n_//2)
... )) + np.random.uniform(-0.75, 0.75, n_)  # add uniform noise
>>> data[n_//2 - 1:n_//2 + 2] = float('nan')  # add some missing data
>>>
>>> # create Shyft data structures
>>> ta = TimeAxis(t0, dt, n_)
>>> temperature_ts = TimeSeries(ta, DoubleVector.from_numpy(data),
...                             point_interpretation_policy.POINT_AVERAGE_VALUE)
>>>
>>> # do the ice packing detection
>>> ip_param = IcePackingParameters(
...     threshold_window=deltahours(5),
...     threshold_temperature=-1.0)
>>> # try all the different temperature policies
>>> ice_packing_ts_disallow = temperature_ts.ice_packing(ip_param, ice_packing_temperature_policy.DISALLOW_MISSING)
>>> ice_packing_ts_initial = temperature_ts.ice_packing(ip_param, ice_packing_temperature_policy.ALLOW_INITIAL_MISSING)
>>> ice_packing_ts_any = temperature_ts.ice_packing(ip_param, ice_packing_temperature_policy.ALLOW_ANY_MISSING)
>>>
>>> # plotting
>>> from matplotlib import pyplot as plt
>>> from shyft.time_series import time_axis_extract_time_points
>>>
>>> # NOTE: The offsets below are added solely to be able to distinguish between the different time-axes
>>>
>>> plt.plot(time_axis_extract_time_points(ta)[:-1], temperature_ts.values, label='Temperature')
>>> plt.plot(time_axis_extract_time_points(ta)[:-1], ice_packing_ts_disallow.values + 1,
...          label='Ice packing? [DISALLOW_MISSING]')
>>> plt.plot(time_axis_extract_time_points(ta)[:-1], ice_packing_ts_initial.values - 1,
...          label='Ice packing? [ALLOW_INITIAL_MISSING]')
>>> plt.plot(time_axis_extract_time_points(ta)[:-1], ice_packing_ts_any.values - 3,
...          label='Ice packing? [ALLOW_ANY_MISSING]')
>>> plt.legend()
>>> plt.show()

ice_packing_recession(self: shyft.time_series.TimeSeries, ip_ts: shyft.time_series.TimeSeries, ipr_params: shyft.time_series.IcePackingRecessionParameters) → shyft.time_series.TimeSeries¶

Create a new time series where segments are replaced by recession curves.

Note

The total period (TimeSeries.total_period) of self needs to be equal to, or contained in the total period of ip_ts.

Parameters:

ip_ts (TimeSeries) – A binary time-series indicating if ice packing occurring. See TimeSeries.ice_packing.
ip_param (IcePackingParameters) – Parameter container controlling the ice packing recession curve.

Returns:

ice_packing_recession_ts. A time-series where sections in self is replaced by recession curves as indicated by ip_ts.

Return type:

Example:

>>> import numpy as np
>>> from shyft.time_series import (
...     IcePackingParameters, IcePackingRecessionParameters, ice_packing_temperature_policy,
...     TimeAxis, TimeSeries, point_interpretation_policy, DoubleVector,
...     utctime_now, deltahours, deltaminutes,
... )
>>>
>>> t0 = utctime_now()
>>> dt = deltaminutes(15)
>>> n = 100
>>>
>>> # generate jittery temperature data
>>> # - first descending from +5 to -5 then ascending back to +5
>>> # - include a NaN hole at the bottom of the V
>>> n_ = n if (n//2)*2 == n else n+1  # assure even
>>> temperature_data = np.concatenate((
...     np.linspace(5, -5, n_//2), np.linspace(-5, 5, n_//2)
... )) + np.random.uniform(-0.75, 0.75, n_)  # add uniform noise
>>> temperature_data[n_ // 2 - 1:n_ // 2 + 2] = float('nan')  # add some missing data
>>>
>>> # create Shyft data structures for temperature
>>> ta = TimeAxis(t0, dt, n_)
>>> temperature_ts = TimeSeries(ta, DoubleVector.from_numpy(temperature_data),
...                             point_interpretation_policy.POINT_AVERAGE_VALUE)
>>>
>>> # generate jittery waterflow data
>>> # - an upwards curving parabola
>>> x0 = ta.total_period().start
>>> x1 = ta.total_period().end
>>> x = np.linspace(x0, x1, n_)
>>> flow_data = -0.0000000015*(x - x0)*(x - x1) + 1 + np.random.uniform(-0.5, 0.5, n_)
>>> del x0, x1, x
>>>
>>> # create Shyft data structures for temperature
>>> flow_ts = TimeSeries(ta, DoubleVector.from_numpy(flow_data),
...                      point_interpretation_policy.POINT_AVERAGE_VALUE)
>>>
>>> # do the ice packing detection
>>> ip_param = IcePackingParameters(
...     threshold_window=deltahours(5),
...     threshold_temperature=-1.0)
>>> # compute the detection time-series
>>> # ice_packing_ts = temperature_ts.ice_packing(ip_param, ice_packing_temperature_policy.DISALLOW_MISSING)
>>> # ice_packing_ts = temperature_ts.ice_packing(ip_param, ice_packing_temperature_policy.ALLOW_INITIAL_MISSING)
>>> ice_packing_ts = temperature_ts.ice_packing(ip_param, ice_packing_temperature_policy.ALLOW_ANY_MISSING)
>>>
>>> # setup for the recession curve
>>> ipr_param = IcePackingRecessionParameters(
...     alpha=0.00009,
...     recession_minimum=2.)
>>> # compute a recession curve based on the ice packing ts
>>> ice_packing_recession_ts_initial = flow_ts.ice_packing_recession(ice_packing_ts, ipr_param)
>>>
>>> # plotting
>>> from matplotlib import pyplot as plt
>>> from shyft.time_series import time_axis_extract_time_points
>>>
>>> plt.plot(time_axis_extract_time_points(ta)[:-1], temperature_ts.values, label='Temperature')
>>> plt.plot(time_axis_extract_time_points(ta)[:-1], flow_ts.values, label='Flow')
>>> plt.plot(time_axis_extract_time_points(ta)[:-1], ice_packing_ts.values,
...          label='Ice packing?')
>>> plt.plot(time_axis_extract_time_points(ta)[:-1], ice_packing_recession_ts_initial.values,
...          label='Recession curve')
>>> plt.legend()
>>> plt.show()

index_of(self: shyft.time_series.TimeSeries, t: shyft.time_series.time) → int¶: return the index of the intervall that contains t, or npos if not found

inside(self: shyft.time_series.TimeSeries, min_v: SupportsFloat | SupportsIndex, max_v: SupportsFloat | SupportsIndex, nan_v: SupportsFloat | SupportsIndex = nan, inside_v: SupportsFloat | SupportsIndex = 1.0, outside_v: SupportsFloat | SupportsIndex = 0.0) → shyft.time_series.TimeSeries¶

Create an inside min-max range ts, that transforms the point-values that falls into the half open range [min_v .. max_v ) to the value of inside_v(default=1.0), or outside_v(default=0.0), and if the value considered is nan, then that value is represented as nan_v(default=nan) You would typically use this function to form a true/false series (inside=true, outside=false)

Parameters:

min_v (float) – minimum range, values < min_v are not inside min_v==NaN means no lower limit
max_v (float) – maximum range, values >= max_v are not inside. max_v==NaN means no upper limit
nan_v (float) – value to return if the value is nan
inside_v (float) – value to return if the ts value is inside the specified range
outside_v (float) – value to return if the ts value is outside the specified range

Returns:

inside_ts. Evaluated on demand inside time-series

Return type:

integral(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxis) → shyft.time_series.TimeSeries¶

create a new ts that is the true integral of self over the specified time-axis ta. defined as integral of the non-nan part of each time-axis interval

Parameters:: ta (TimeAxis) – time-axis that specifies the periods where true-integral is applied
Returns:: ts. a new time-series expression, that will provide the true-integral when requested
Return type:: TimeSeries

Notes

the self point interpretation policy is used when calculating the true average

kling_gupta(self: shyft.time_series.TimeSeries, other_ts: shyft.time_series.TimeSeries, s_r: SupportsFloat | SupportsIndex = 1.0, s_a: SupportsFloat | SupportsIndex = 1.0, s_b: SupportsFloat | SupportsIndex = 1.0) → float¶

Computes the kling-gupta KGEs correlation for self and supplied model_ts over the specified time_axis

Parameters:

other_ts (TimeSeries) – the predicted/calculated time-series to correlate
s_r (float) – the kling gupta scale r factor(weight the correlation of goal function)
s_a (float) – the kling gupta scale a factor(weight the relative average of the goal function)
s_b (float) – the kling gupta scale b factor(weight the relative standard deviation of the goal function)

Returns:

KGEs. The KGEs= 1-EDs that have a maximum at 1.0

Return type:

float

krls_interpolation(self: shyft.time_series.TimeSeries, dt: shyft.time_series.time, gamma: SupportsFloat | SupportsIndex = 0.001, tolerance: SupportsFloat | SupportsIndex = 0.01, size: SupportsInt | SupportsIndex = 1000000) → shyft.time_series.TimeSeries¶

Compute a new TS that is a krls interpolation of self.

The KRLS algorithm is a kernel regression algorithm for aproximating data, the implementation used here is from DLib: http://dlib.net/ml.html#krls The new time-series has the same time-axis as self, and the values vector contain no nan entries.

If you also want the mean-squared error of the interpolation use get_krls_predictor instead, and use the predictor api to generate a interpolation and a mse time-series. Other related functions are TimeSeries.get_krls_predictor, KrlsRbfPredictor

Parameters:

dt (float) – The time-step in seconds the underlying predictor is specified for. Note that this does not put a limit on time-axes used, but for best results it should be approximatly equal to the time-step of time-axes used with the predictor. In addition it should not be to long, else you will get poor results. Try to keep the dt less than a day, 3-8 hours is usually fine.
gamma (float (optional)) – Determines the width of the radial basis functions for the KRLS algorithm. Lower values mean wider basis functions, wider basis functions means faster computation but lower accuracy. Note that the tolerance parameter also affects speed and accurcy. A large value is around 1E-2, and a small value depends on the time step. By using values larger than 1E-2 the computation will probably take to long. Testing have reveled that 1E-3 works great for a time-step of 3 hours, while a gamma of 1E-2 takes a few minutes to compute. Use 1E-4 for a fast and tolerably accurate prediction. Defaults to 1E-3
tolerance (float (optional)) – The krls training tolerance. Lower values makes the prediction more accurate, but slower. This typically have less effect than gamma, but is usefull for tuning. Usually it should be either 0.01 or 0.001. Defaults to 0.01
size (int (optional)) – The size of the “memory” of the underlying predictor. The default value is usually enough. Defaults to 1000000.

Examples:

>>> import numpy as np
>>> import scipy.stats as stat
>>> from shyft.time_series import (
...     Calendar, utctime_now, deltahours,
...     TimeAxis, TimeSeries
... )
>>>
>>> cal = Calendar()
>>> t0 = utctime_now()
>>> dt = deltahours(1)
>>> n = 365*24  # one year
>>>
>>> # generate random bell-shaped data
>>> norm = stat.norm()
>>> data = np.linspace(0, 20, n)
>>> data = stat.norm(10).pdf(data) + norm.pdf(np.random.rand(*data.shape))
>>> # -----
>>> ta = TimeAxis(cal, t0, dt, n)
>>> ts = TimeSeries(ta, data)
>>>
>>> # compute the interpolation
>>> ts_ipol = ts.krls_interpolation(deltahours(3))

Returns:: krls_ts. A new time series being the KRLS interpolation of self.
Return type:: TimeSeries

log(self: shyft.time_series.TimeSeries) → shyft.time_series.TimeSeries¶: create a new ts that contains log(py::self)

lower_half(self: shyft.time_series.TimeSeries) → shyft.time_series.TimeSeries¶

Create a ts that contains non-negative values only.

Returns:: lower_half_ts. Evaluated on demand inside time-series
Return type:: TimeSeries

lower_half_mask(self: shyft.time_series.TimeSeries) → shyft.time_series.TimeSeries¶

Create a ts that contains 1.0 in place of non-positive values, and 0.0 in case of positive values.

Returns:: lower_half_mask_ts. Evaluated on demand inside time-series
Return type:: TimeSeries

max(*args, **kwargs)¶

Overloaded function.

max(self: shyft.time_series.TimeSeries, number: typing.SupportsFloat | typing.SupportsIndex) -> shyft.time_series.TimeSeries

create a new ts that contains the max of self and number for each time-step

max(self: shyft.time_series.TimeSeries, ts_other: shyft.time_series.TimeSeries) -> shyft.time_series.TimeSeries

create a new ts that contains the max of self and ts_other

merge_points(self: shyft.time_series.TimeSeries, ts: shyft.time_series.TimeSeries) → shyft.time_series.TimeSeries¶

Given that self is a concrete point-ts(not an expression), or empty ts, this function modifies the point-set of self, with points, (time,value) from other ts The result of the merge operation is the distinct set of time-points from self and other ts where values from other ts overwrites values of self if they happen to be at the same time-point

Parameters:: ts (TimeSeries) – time-series to merge the time,value points from
Returns:: self. self modified with the merged points from other ts
Return type:: TimeSeries

min(*args, **kwargs)¶

Overloaded function.

min(self: shyft.time_series.TimeSeries, number: typing.SupportsFloat | typing.SupportsIndex) -> shyft.time_series.TimeSeries

create a new ts that contains the min of self and number for each time-step

min(self: shyft.time_series.TimeSeries, ts_other: shyft.time_series.TimeSeries) -> shyft.time_series.TimeSeries

create a new ts that contains the min of self and ts_other

min_max_check_linear_fill(*args, **kwargs)¶

Overloaded function.

min_max_check_linear_fill(self: shyft.time_series.TimeSeries, v_min: typing.SupportsFloat | typing.SupportsIndex, v_max: typing.SupportsFloat | typing.SupportsIndex, dt_max: typing.SupportsInt | typing.SupportsIndex = time.max) -> shyft.time_series.TimeSeries

Create a min-max range checked ts with fill-values if value is NaN or outside range If the underlying time-series is point-instant, then fill-values are linear-interpolation, otherwise, the previous value, if available is used as fill-value. A similar function with more features is quality_and_self_correction()

Parameters:

v_min (float) – minimum range, values < v_min are considered NaN. v_min==NaN means no lower limit
v_max (float) – maximum range, values > v_max are considered NaN. v_max==NaN means no upper limit
dt_max (int) – maximum time-range in seconds allowed for interpolating/extending values, default= max_utctime

Returns:

min_max_check_linear_fill. Evaluated on demand time-series with NaN, out of range values filled in

Return type:

min_max_check_linear_fill(self: shyft.time_series.TimeSeries, v_min: typing.SupportsFloat | typing.SupportsIndex, v_max: typing.SupportsFloat | typing.SupportsIndex, dt_max: shyft.time_series.time = time.max) -> shyft.time_series.TimeSeries

Create a min-max range checked ts with fill-values if value is NaN or outside range If the underlying time-series is point-instant, then fill-values are linear-interpolation, otherwise, the previous value, if available is used as fill-value. Similar and more parameterized function is quality_and_self_correction()

Parameters:

v_min (float) – minimum range, values < v_min are considered NaN. v_min==NaN means no lower limit
v_max (float) – maximum range, values > v_max are considered NaN. v_max==NaN means no upper limit
dt_max (int) – maximum time-range in seconds allowed for interpolating/extending values, default= max_utctime

Returns:

min_max_check_linear_fill. Evaluated on demand time-series with NaN, out of range values filled in

Return type:

min_max_check_ts_fill(*args, **kwargs)¶

Overloaded function.

min_max_check_ts_fill(self: shyft.time_series.TimeSeries, v_min: typing.SupportsFloat | typing.SupportsIndex, v_max: typing.SupportsFloat | typing.SupportsIndex, dt_max: typing.SupportsInt | typing.SupportsIndex, cts: shyft.time_series.TimeSeries) -> shyft.time_series.TimeSeries

Create a min-max range checked ts with cts-filled-in-values if value is NaN or outside range

Parameters:

v_min (float) – minimum range, values < v_min are considered NaN. v_min==NaN means no lower limit
v_max (float) – maximum range, values > v_max are considered NaN. v_max==NaN means no upper limit
dt_max (int) – maximum time-range in seconds allowed for interpolating values
cts (TimeSeries) – time-series that keeps the values to be filled in at points that are NaN or outside min-max-limits

Returns:

min_max_check_ts_fill. Evaluated on demand time-series with NaN, out of range values filled in

Return type:

min_max_check_ts_fill(self: shyft.time_series.TimeSeries, v_min: typing.SupportsFloat | typing.SupportsIndex, v_max: typing.SupportsFloat | typing.SupportsIndex, dt_max: shyft.time_series.time, cts: shyft.time_series.TimeSeries) -> shyft.time_series.TimeSeries

Create a min-max range checked ts with cts-filled-in-values if value is NaN or outside range

Parameters:

v_min (float) – minimum range, values < v_min are considered NaN. v_min==NaN means no lower limit
v_max (float) – maximum range, values > v_max are considered NaN. v_max==NaN means no upper limit
dt_max (int) – maximum time-range in seconds allowed for interpolating values
cts (TimeSeries) – time-series that keeps the values to be filled in at points that are NaN or outside min-max-limits

Returns:

min_max_check_ts_fill. Evaluated on demand time-series with NaN, out of range values filled in

Return type:

Ref: http://en.wikipedia.org/wiki/Nash%E2%80%93Sutcliffe_model_efficiency_coefficient

nash_sutcliffe(self: shyft.time_series.TimeSeries, other_ts: shyft.time_series.TimeSeries) → float¶

Computes the Nash-Sutcliffe model effiency coefficient (n.s) for the two time-series over the specified self(observed) time_axis

Parameters:: other_ts (TimeSeries) – the time-series that is the model simulated / calculated ts
Returns:: ns. The n.s performance, that have a maximum at 1.0
Return type:: float

needs_bind(self: shyft.time_series.TimeSeries) → bool¶: returns true if there are any unbound time-series in the expression this time-series represent These functions also supports symbolic time-series handling: .find_ts_bind_info(),bind() and bind_done()

operands(self: shyft.time_series.TimeSeries) → list[shyft.time_series.TimeSeries]¶: Returns the operands of a timeseries expression

partition_by(self: shyft.time_series.TimeSeries, calendar: shyft.time_series.Calendar, t: shyft.time_series.time, partition_interval: shyft.time_series.time, n_partitions: SupportsInt | SupportsIndex, common_t0: shyft.time_series.time) → shyft.time_series.TsVector¶

DEPRECATED(replaced by .stack ) : from a time-series, construct a TsVector of n time-series partitions. The partitions are simply specified by calendar, delta_t(could be symbolic, like YEAR : MONTH:DAY) and n. To make yearly partitions, just pass Calendar.YEAR as partition_interval. The t - parameter set the start - time point in the source-time-series, e.g. like 1930.09.01 The common_t0 - parameter set the common start - time of the new partitions, e.g. 2017.09.01

The typical usage will be to use this function to partition years into a vector with 80 years, where we can do statistics, percentiles to compare and see the different effects of yearly season variations. Note that the function is more general, allowing any periodic partition, like daily, weekly, monthly etc. that allows you to study any pattern or statistics that might be periodic by the partition pattern. Other related methods are time_shift,average,TsVector.

Parameters:

calendar (Calendar) – The calendar to use, typically utc
t (utctime) – specifies where to pick the first partition
partition_interval (utctimespan) – the length of each partition, Calendar.YEAR,Calendar.DAY etc.
n_partitions (int) – number of partitions
common_t0 (utctime) – specifies the time to correlate all the partitions

Returns:

ts-partitions. with length n_partitions, each ts is time-shifted to common_t0 expressions

Return type:

point_interpretation(self: shyft.time_series.TimeSeries) → shyft.time_series.point_interpretation_policy¶: returns the point interpretation policy

pow(*args, **kwargs)¶

Overloaded function.

pow(self: shyft.time_series.TimeSeries, number: typing.SupportsFloat | typing.SupportsIndex) -> shyft.time_series.TimeSeries

create a new ts that contains pow(py::self,number)

pow(self: shyft.time_series.TimeSeries, ts_other: shyft.time_series.TimeSeries) -> shyft.time_series.TimeSeries

create a new ts that contains pow(py::self,ts_other)

quality_and_self_correction(self: shyft.time_series.TimeSeries, parameters: shyft.time_series.QacParameter) → shyft.time_series.TimeSeries¶

returns a new time-series that applies quality checks accoring to parameters and fills in values according to rules specified in parameters.

Parameters:: parameter (QacParameter) – Parameter with rules for quality and corrections
Returns:: ts. a new time-series where the values are subject to quality and correction as specified
Return type:: TimeSeries

quality_and_ts_correction(self: shyft.time_series.TimeSeries, parameters: shyft.time_series.QacParameter, cts: shyft.time_series.TimeSeries) → shyft.time_series.TimeSeries¶

returns a new time-series that applies quality checks accoring to parameters and fills in values from the cts, according to rules specified in parameters.

Parameters:

parameter (QacParameter) – Parameter with rules for quality and corrections
cts (TimeSeries) – is used to fill in correct values, as f(t) for values that fails quality-checks

Returns:

ts. a new time-series where the values are subject to quality and correction as specified

Return type:

rating_curve(self: shyft.time_series.TimeSeries, rc_param: shyft.time_series.RatingCurveParameters) → shyft.time_series.TimeSeries¶

Create a new TimeSeries that is computed using a RatingCurveParameter instance.

Examples:

>>> import numpy as np
>>> from shyft.time_series import (
...     utctime_now, deltaminutes,
...     TimeAxis, TimeSeries,
...     RatingCurveFunction, RatingCurveParameters
... )
>>>
>>> # parameters
>>> t0 = utctime_now()
>>> dt = deltaminutes(30)
>>> n = 48*2
>>>
>>> # make rating function, each with two segments
>>> rcf_1 = RatingCurveFunction()
>>> rcf_1.add_segment(0, 2, 0, 1)    # add segment from level 0, computing f(h) = 2*(h - 0)**1
>>> rcf_1.add_segment(5.3, 1, 1, 1.4)  # add segment from level 5.3, computing f(h) = 1.3*(h - 1)**1.4
>>> rcf_2 = RatingCurveFunction()
>>> rcf_2.add_segment(0, 1, 1, 1)    # add segment from level 0, computing f(h) = 1*(h - 1)**1
>>> rcf_2.add_segment(8.0, 0.5, 0, 2)  # add segment from level 8.0, computing f(h) = 0.5*(h - 0)**2
>>>
>>> # add rating curves to a parameter pack
>>> rcp = RatingCurveParameters()
>>> rcp.add_curve(t0, rcf_1)  # rcf_1 is active from t0
>>> rcp.add_curve(t0+dt*n//2, rcf_2)  # rcf_2 takes over from t0 + dt*n/2
>>>
>>> # create a time-axis/-series
>>> ta = TimeAxis(t0, dt, n)
>>> ts = TimeSeries(ta, np.linspace(0, 12, n))
>>> rc_ts = ts.rating_curve(rcp)  # create a new time series computed using the rating curve functions
>>>

Parameters:: rc_param (RatingCurveParameter) – RatingCurveParameter instance.
Returns:: rcts. A new TimeSeries computed using self and rc_param.
Return type:: TimeSeries

repeat(self: shyft.time_series.TimeSeries, repeat_time_axis: shyft.time_series.TimeAxis) → shyft.time_series.TimeSeries¶

Repeat all time-series over the given repeat_time_axis periods

Parameters:: repeat_time_axis (TimeAxis) – A time-axis that have the coarse repeat interval, like YEAR or similar
Returns:: repeated_ts. time-series where pattern of self is repeated throughout the period of repeat_time_axis
Return type:: TimeSeries

scale_by(self: shyft.time_series.TimeSeries, v: SupportsFloat | SupportsIndex) → None¶: scale all values by the specified factor v

serialize(self: shyft.time_series.TimeSeries) → shyft.time_series.ByteVector¶: convert ts (expression) into a binary blob

set(self: shyft.time_series.TimeSeries, i: SupportsInt | SupportsIndex, v: SupportsFloat | SupportsIndex) → None¶: set the i-th value

set_point_interpretation(self: shyft.time_series.TimeSeries, policy: shyft.time_series.point_interpretation_policy) → None¶: set new policy

set_ts_id(self: shyft.time_series.TimeSeries, ts_id: str) → None¶: Set a new ts_id of symbolic ts, requires unbound ts. To create symbolic time-series use TimeSeries(‘url://like/id’) or with payload: TimeSeries(‘url://like/id’,ts_with_values)

size(self: shyft.time_series.TimeSeries) → int¶: returns number of points

slice(self: shyft.time_series.TimeSeries, i0: SupportsInt | SupportsIndex, n: SupportsInt | SupportsIndex) → shyft.time_series.TimeSeries¶

Given that self is a concrete point-ts(not an expression), or empty ts, return a new TimeSeries containing the n values starting from index i0.

Parameters:

i0 (int) – Index of first element to include in the slice
n (int) – Number of elements to include in the slice

stack(self: shyft.time_series.TimeSeries, calendar: shyft.time_series.Calendar, t0: shyft.time_series.time, n_dt: SupportsInt | SupportsIndex, dt: shyft.time_series.time, n_partitions: SupportsInt | SupportsIndex, target_t0: shyft.time_series.time, dt_snap: shyft.time_series.time) → shyft.time_series.TsVector¶

stack time-series into a TsVector of n_partitions time-series, each with semantic calendar length n_dt x dt. The partitions are simply specified by calendar, n_dt x dt(could be symbolic, like YEAR : MONTH:DAY) and n. To make yearly partitions, just pass 1, Calendar.YEAR as n_dt and dt respectively. The t0 - parameter set the start - time point in the source-time-series, e.g. like 1930.09.01 The target_t0 - parameter set the common start-time of the stack, e.g. 2017.09.01 The dt_snap - parameter is useful to ensure that if target_to is a monday, then each partition is adjusted to neares monday. The snap mechanism could be useful if you would like to stack something like consumption, that would follow a weekly pattern.

Parameters:

calendar (Calendar) – The calendar to use, typically utc
t0 (utctime) – specifies where to pick the first partition, e.g. 1930.09.01
n_dt (int) – number of calendar units for the length of the stride
dt (utctimespan) – the basic calendar length unit, Calendar.YEAR,Calendar.DAY
n_partitions (int) – number of partitions,e.g. length of the resulting TsVector
target_t0 (utctime) – specifies the common target time for the stack, e.g. 2017.09.01
dt_snap (utctimespan) – default 0, if set to WEEK, each stacked partition will be week-aligned.

Returns:

stacked_ts. with length n_partitions, each ts is time-shifted (calendar n_dt x n) to common_t0 expressions

Return type:

statistics(self: shyft.time_series.TimeSeries, ta: shyft.time_series.TimeAxis, p: SupportsInt | SupportsIndex) → shyft.time_series.TimeSeries¶

Create a new ts that extract the specified statistics from self over the specified time-axis ta Statistics are created for the point values of the time-series that falls within each time-period of the time-axis. If there are no points within the period, nan will be the result. Tip: use ts.average(ta_hourly_resolution).statistics(ta_weekly,p=50) to get the functional true hourly average statistics.

Parameters:

ta (TimeAxis) – time-axis for the statistics
p (int) – percentile range [0..100], or statistical_property.AVERAGE|MIN_EXTREME|MAX_EXTREME

Returns:

ts. a new time-series expression, will provide the statistics when requested

Return type:

stringify(self: shyft.time_series.TimeSeries) → str¶: return human-readable string of ts or expression

time(self: shyft.time_series.TimeSeries, i: SupportsInt | SupportsIndex) → shyft.time_series.time¶: returns the time at the i-th point

property time_axis¶

time_shift(self: shyft.time_series.TimeSeries, delta_t: shyft.time_series.time) → shyft.time_series.TimeSeries¶

create a new ts that is a the time-shift’ed version of self

Parameters:: delta_t (int) – number of seconds to time-shift, positive values moves forward
Returns:: ts. a new time-series, that appears as time-shifted version of self
Return type:: TimeSeries

total_period(self: shyft.time_series.TimeSeries) → shyft.time_series.UtcPeriod¶: returns the total period covered by the time-axis of this time-series

transform(self: shyft.time_series.TimeSeries, points: Annotated[numpy.typing.ArrayLike, numpy.float64], method: shyft.time_series.interpolation_scheme) → shyft.time_series.TimeSeries¶

Create a transformed time-series, having values taken from pointwise function evaluation. Function values are determined by interpolating the given points, using the specified method. Valid method arguments are ‘polynomial’, ‘linear’ and ‘catmull-rom’.

Returns:: transform_ts. New TimeSeries where each element is an evaluated-on-demand transformed time-series.
Return type:: TimeSeries

ts_id(self: shyft.time_series.TimeSeries) → str¶

returns ts_id of symbolic ts, or empty string if not symbolic ts To create symbolic time-series use TimeSeries(‘url://like/id’) or with payload: TimeSeries(‘url://like/id’,ts_with_values)

Returns:: ts_id. url-like ts_id as passed to constructor or empty if the ts is not a ts with ts_id
Return type:: str

unbind(self: shyft.time_series.TimeSeries) → None¶: Reset the ts-expression to unbound state, discarding bound symbol references. For time-series, or expressions, that does not have symbolic references, no effect, see also .find_ts_bind_info(),bind() and bind_done()

upper_half(self: shyft.time_series.TimeSeries) → shyft.time_series.TimeSeries¶

Create a ts that contains non-negative values only.

Returns:: upper_half_ts. Evaluated on demand inside time-series
Return type:: TimeSeries

upper_half_mask(self: shyft.time_series.TimeSeries) → shyft.time_series.TimeSeries¶

Create a ts that contains 1.0 in place of non-negative values, and 0.0 in case of negative values.

Returns:: upper_half_mask_ts. Evaluated on demand inside time-series
Return type:: TimeSeries

use_time_axis(self: shyft.time_series.TimeSeries, time_axis: shyft.time_series.TimeAxis) → shyft.time_series.TimeSeries¶

Create a new ts that have the same values as self, but filtered to the time-axis points from from the supplied time-axis. This function migth be useful for making new time-series, that exactly matches the time-axis of another series. Values of the resulting time-series is like like: [self(t) for t in time_axis.time_points[:-1]

Parameters:: time_axis (TimeAxis) – the wanted time-axis
Returns:: ts. a new time-series, that appears as resampled values of self
Return type:: TimeSeries

use_time_axis_from(self: shyft.time_series.TimeSeries, other: shyft.time_series.TimeSeries) → shyft.time_series.TimeSeries¶

Create a new ts that have the same values as self, but filtered to the time-axis points from from the other supplied time-series. This function migth be useful for making new time-series, that exactly matches the time-axis of another series. Values of the resulting time-series is like like: [self(t) for t in other.time_axis.time_points[:-1] Notice that the other time-series can be an unbound (expression) in this case.

Parameters:: other (TimeSeries) – time-series that provides the wanted time-axis
Returns:: ts. a new time-series, that appears as resampled values of self
Return type:: TimeSeries

property v¶

the values values (possibly calculated on the fly)

Type:: DoubleVector

value(self: shyft.time_series.TimeSeries, i: SupportsInt | SupportsIndex) → float¶: returns the value at the i-th time point

property values¶

the values values (possibly calculated on the fly)

Type:: DoubleVector

Class TsVector¶

class shyft.time_series.TsVector(*args, **kwargs)¶

Bases: pybind11_object

A vector, as in strongly typed list, array, of time-series that supports ts-math operations. You can create a TsVector from a list, or list generator of type TimeSeries. TsVector is to TimeSeries that a numpy array is to numbers, see also TimeSeries

Math operations and their types transformations:

number bin_op ts_vector -> ts_vector

ts_vector bin_op ts_vector -> ts_vector

ts bin_op ts_vector -> ts_vector

where bin_op is any of (*,/,+,-) and explicit forms of binary functions like pow,log,min,max.

In addition these are also available: average() integral() accumulate() time_shift() percentiles()

All operation return a new object, usually a ts-vector, containing the resulting expressions

Examples:

>>> import numpy as np
>>> from shyft.time_series import TsVector,Calendar,deltahours,TimeAxis,TimeSeries,POINT_AVERAGE_VALUE as fx_avg
>>>
>>> utc = Calendar()  # ensure easy consistent explicit handling of calendar and time
>>> ta1 = TimeAxis(utc.time(2016, 9, 1, 8, 0, 0), deltahours(1), 10)  # create a time-axis for ts1
>>> ts1 = TimeSeries(ta1, np.linspace(0, 10, num=len(ta)), fx_avg)
>>> ta2 = TimeAxis(utc.time(2016, 9, 1, 8, 30, 0), deltahours(1), 5)  # create a time-axis to ts2
>>> ts2 = TimeSeries(ta2, np.linspace(0,  1, num=len(ta)), fx_avg)
>>> tsv = TsVector([ts1, ts2]) # create ts vector from list of time-series
>>> c = tsv + tsv*3.0  # c is now an expression, time-axis is the overlap of a and b, lazy evaluation
>>> c_values = c[0].values  # compute and extract the values of the ith (here: 0) time-series, as numpy array
>>>
>>> # Calculate data for new time-points
>>> value_1 = tsv(utc.time(2016, 9, 1, 8, 30)) # calculates value at a given time
>>> ta_target = TimeAxis(utc.time(2016, 9, 1, 7, 30), deltahours(1), 12)  # create a target time_axis
>>> tsv_new = tsv.average(ta_target) # new ts-vector with values on target time_axis
>>> ts0_val = tsv_new[0].values # access values of the ith (here: 0) time-series as a numpy array
>>>

Overloaded function.

__init__(self: shyft.time_series.TsVector) -> None
__init__(self: shyft.time_series.TsVector, arg0: shyft.time_series.TsVector) -> None

Copy constructor

__init__(self: shyft.time_series.TsVector, arg0: collections.abc.Iterable) -> None

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.TsVector) -> None
__init__(self: shyft.time_series.TsVector, arg0: shyft.time_series.TsVector) -> None

Copy constructor

__init__(self: shyft.time_series.TsVector, arg0: collections.abc.Iterable) -> None

abs(self: shyft.time_series.TsVector) → shyft.time_series.TsVector¶

create a new ts-vector, with all members equal to abs(py::self

Returns:: tsv. a new TsVector expression, that will provide the abs-values of self.values
Return type:: TsVector

accumulate(self: shyft.time_series.TsVector, ta: shyft.time_series.TimeAxis) → shyft.time_series.TsVector¶

create a new vector of time-series where the vaue of each i-th element is computed as: integral f(t) *dt, from t0..ti given the specified time-axis ta and point interpretation.

Parameters:: ta (TimeAxis) – time-axis that specifies the periods where accumulated integral is applied
Returns:: tsv. a new time-series expression, that will provide the accumulated values when requested
Return type:: TsVector

Notes

Has a point-instant interpretation, see also note in TimeSeries.accumulate() for possible consequences

append(self: shyft.time_series.TsVector, x: shyft.time_series.TimeSeries) → None¶: Add an item to the end of the list

average(self: shyft.time_series.TsVector, ta: shyft.time_series.TimeAxis) → shyft.time_series.TsVector¶

create a new vector of ts that is the true average of self over the specified time-axis ta.

Parameters:: ta (TimeAxis) – time-axis that specifies the periods where true-average is applied
Returns:: tsv. a new time-series expression, that will provide the true-average when requested
Return type:: TsVector

Notes

the self point interpretation policy is used when calculating the true average

average_slice(self: shyft.time_series.TsVector, lead_time: shyft.time_series.time, delta_t: shyft.time_series.time, n: SupportsInt | SupportsIndex) → shyft.time_series.TsVector¶

Returns a ts-vector with the average time-series of the specified slice The slice for each ts is specified by the lead_time, delta_t and n parameters. See also nash_sutcliffe,forecast_merge

Parameters:

lead_time (int) – number of seconds lead-time offset from each ts .time(0)
delta_t (int) – delta-time seconds to average as basis for n.s. simulation and observation values
n (int) – number of time-steps of length delta_t to slice out of each forecast/simulation ts

Returns:

ts_vector_sliced. a ts-vector with average ts of each slice specified.

Return type:

clear(self: shyft.time_series.TsVector) → None¶: Clear the contents

clone_expression(self: shyft.time_series.TsVector) → shyft.time_series.TsVector¶

Notes

this function is only useful in context where multiple bind/rebind while keeping the expression is needed.

Returns:: semantic_clone. returns a copy of the ts, except for the payload at reference/symbolic terminals, where only ts_id is copied
Return type:: TsVector

derivative(self: shyft.time_series.TsVector, method: shyft.time_series.derivative_method = <derivative_method.DEFAULT: 0>) → shyft.time_series.TsVector¶

create a new vector of ts where each i-th element is the derivative of f(t)

Parameters:: method (derivative_method) – what derivative_method variant to use
Returns:: tsv. where each member is the derivative of the source
Return type:: TsVector

evaluate(self: shyft.time_series.TsVector) → shyft.time_series.TsVector¶

Evaluates the expressions in TsVector multithreaded, and returns the resulting TsVector, where all items now are concrete terminals, that is, not expressions anymore. Useful client-side if you have complex large expressions where all time-series are bound (not symbols)

Returns:: evaluated_clone. returns the computed result as a new ts-vector
Return type:: TsVector

extend(*args, **kwargs)¶

Overloaded function.

extend(self: shyft.time_series.TsVector, L: shyft.time_series.TsVector) -> None

Extend the list by appending all the items in the given list

extend(self: shyft.time_series.TsVector, L: collections.abc.Iterable) -> None

Extend the list by appending all the items in the given list

extend_ts(*args, **kwargs)¶

Overloaded function.

extend_ts(self: shyft.time_series.TsVector, ts: shyft.time_series.TimeSeries, split_policy: shyft.time_series.extend_split_policy = <extend_split_policy.LHS_LAST: 0>, fill_policy: shyft.time_series.extend_fill_policy = <extend_fill_policy.FILL_NAN: 0>, split_at: shyft.time_series.time = time(0), fill_value: typing.SupportsFloat | typing.SupportsIndex = nan) -> shyft.time_series.TsVector

create a new dd::ats_vector where all time-series are extended by ts

Parameters:

ts (TimeSeries) – time-series to extend each time-series in self with
split_policy (extend_ts_split_policy) – policy determining where to split between self and ts
fill_policy (extend_ts_fill_policy) – policy determining how to fill any gap between self and ts
split_at (utctime) – time at which to split if split_policy == EPS_VALUE
fill_value (float) – value to fill any gap with if fill_policy == EPF_FILL

Returns:

new_ts_vec. a new time-series vector where all time-series in self have been extended by ts

Return type:

extend_ts(self: shyft.time_series.TsVector, ts: shyft.time_series.TsVector, split_policy: shyft.time_series.extend_split_policy = <extend_split_policy.LHS_LAST: 0>, fill_policy: shyft.time_series.extend_fill_policy = <extend_fill_policy.FILL_NAN: 0>, split_at: shyft.time_series.time = time(0), fill_value: typing.SupportsFloat | typing.SupportsIndex = nan) -> shyft.time_series.TsVector

create a new dd::ats_vector where all ts’ are extended by the matching ts from ts_vec

Parameters:

ts_vec (TsVector) – time-series vector to extend time-series in self with
split_policy (extend_ts_split_policy) – policy determining where to split between self and ts
fill_policy (extend_ts_fill_policy) – policy determining how to fill any gap between self and ts
split_at (utctime) – time at which to split if split_policy == EPS_VALUE
fill_value (float) – value to fill any gap with if fill_policy == EPF_FILL

Returns:

new_ts_vec. a new time-series vector where all time-series in self have been extended by the corresponding time-series in ts_vec

Return type:

extract_as_table(self: shyft.time_series.TsVector, cal: shyft.time_series.Calendar, time_scale: SupportsFloat | SupportsIndex) → numpy.ndarray¶

Extract values in the ts-vector as a table, where columns: | [0] is the distinct union of all time_scale*(time-points i, + cal.tz_offset(i)) | [1..n] is the value contribution of the i-th ts, nan if no contribution at that time-point This function primary usage is within visual-layer of the shyft.dashboard package to speed up processing. The semantics and parameters reflects this.

Parameters:

cal (Calendar) – Calendar to use for tz-offset of each time-point (to resolve bokeh lack of tz-handling)
time_scale (float) – time-scale to multiply the time from si-unit [s] to any scaled unit, typically ms

Returns:

table. A 2d array where [0] contains time, [1..n] the values

Return type:

array

forecast_merge(self: shyft.time_series.TsVector, lead_time: shyft.time_series.time, fc_interval: shyft.time_series.time) → shyft.time_series.TimeSeries¶

merge the forecasts in this vector into a time-series that is constructed taking a slice of length fc_interval starting lead_time into each of the forecasts of this time-series vector. The content of the vector should be ordered in forecast-time, each entry at least fc_interval separated from the previous. If there is missing forecasts (larger than fc_interval between two forecasts) this is automagically repaired using extended slices from the existing forecasts

Parameters:

lead_time (int) – start slice number of seconds from t0 of each forecast
fc_interval (int) – length of each slice in seconds, and thus also gives the forecast-interval separation

Returns:

merged time-series. A merged forecast time-series

Return type:

insert(self: shyft.time_series.TsVector, i: SupportsInt | SupportsIndex, x: shyft.time_series.TimeSeries) → None¶: Insert an item at a given position.

inside(self: shyft.time_series.TsVector, min_v: SupportsFloat | SupportsIndex, max_v: SupportsFloat | SupportsIndex, nan_v: SupportsFloat | SupportsIndex = nan, inside_v: SupportsFloat | SupportsIndex = 1.0, outside_v: SupportsFloat | SupportsIndex = 0.0) → shyft.time_series.TsVector¶

Create an inside min-max range ts-vector, that transforms the point-values that falls into the half open range [min_v .. max_v > to the value of inside_v(default=1.0), or outside_v(default=0.0), and if the value considered is nan, then that value is represented as nan_v(default=nan) You would typically use this function to form a true/false series (inside=true, outside=false)

Parameters:

min_v (float) – minimum range, values < min_v are not inside min_v==NaN means no lower limit
max_v (float) – maximum range, values >= max_v are not inside. max_v==NaN means no upper limit
nan_v (float) – value to return if the value is nan
inside_v (float) – value to return if the ts value is inside the specified range
outside_v (float) – value to return if the ts value is outside the specified range

Returns:

inside_tsv. New TsVector where each element is an evaluated-on-demand inside time-series

Return type:

integral(self: shyft.time_series.TsVector, ta: shyft.time_series.TimeAxis) → shyft.time_series.TsVector¶

create a new vector of ts that is the true integral of self over the specified time-axis ta. defined as integral of the non-nan part of each time-axis interval

Parameters:: ta (TimeAxis) – time-axis that specifies the periods where true-integral is applied
Returns:: tsv. a new time-series expression, that will provide the true-integral when requested
Return type:: TsVector

Notes

the self point interpretation policy is used when calculating the true average

log(self: shyft.time_series.TsVector) → shyft.time_series.TsVector¶

max(*args, **kwargs)¶

Overloaded function.

max(self: shyft.time_series.TsVector, number: typing.SupportsFloat | typing.SupportsIndex) -> shyft.time_series.TsVector

returns max of vector and a number

max(self: shyft.time_series.TsVector, ts: shyft.time_series.TimeSeries) -> shyft.time_series.TsVector

returns max of ts-vector and a ts

max(self: shyft.time_series.TsVector, tsv: shyft.time_series.TsVector) -> shyft.time_series.TsVector

returns max of ts-vector and another ts-vector

min(*args, **kwargs)¶

Overloaded function.

min(self: shyft.time_series.TsVector, number: typing.SupportsFloat | typing.SupportsIndex) -> shyft.time_series.TsVector

returns min of vector and a number

min(self: shyft.time_series.TsVector, ts: shyft.time_series.TimeSeries) -> shyft.time_series.TsVector

returns min of ts-vector and a ts

min(self: shyft.time_series.TsVector, tsv: shyft.time_series.TsVector) -> shyft.time_series.TsVector

returns min of ts-vector and another ts-vector

nash_sutcliffe(self: shyft.time_series.TsVector, observation_ts: shyft.time_series.TimeSeries, lead_time: shyft.time_series.time, delta_t: shyft.time_series.time, n: SupportsInt | SupportsIndex) → float¶

Computes the nash-sutcliffe (wiki nash-sutcliffe) criteria between the observation_ts over the slice of each time-series in the vector. The slice for each ts is specified by the lead_time, delta_t and n parameters. The function is provided to ease evaluation of forecast performance for different lead-time periods into each forecast. The returned value range is 1.0 for perfect match -oo for no match, or nan if observations is constant or data missing. See also nash_sutcliffe_goal_function

Parameters:

observation_ts (TimeSeries) – the observation time-series
lead_time (int) – number of seconds lead-time offset from each ts .time(0)
delta_t (int) – delta-time seconds to average as basis for n.s. simulation and observation values
n (int) – number of time-steps of length delta_t to slice out of each forecast/simulation ts

Returns:

nash-sutcliffe value. the nash-sutcliffe criteria evaluated over all time-series in the TsVector for the specified lead-time, delta_t and number of elements

Return type:

double

percentiles(*args, **kwargs)¶

Overloaded function.

percentiles(self: shyft.time_series.TsVector, time_axis: shyft.time_series.TimeAxis, percentiles: collections.abc.Sequence[typing.SupportsInt | typing.SupportsIndex]) -> shyft.time_series.TsVector

Calculate the percentiles of all timeseries. over the specified time-axis. The definition is equal to e.g. NIST R7, excel, and in R. The time-series point_fx interpretation is used when performing the true-average over the time_axis periods. This functions works on bound expressions, for unbound expressions, use the DtsClient.percentiles.

See also DtsClient.percentiles() if you want to evaluate percentiles of an unbound expression.

Parameters:

percentiles (IntVector) – A list of numbers,like [ 0, 25,50,-1,75,100] will return 6 time-series. Number with special sematics are: -1 -> arithmetic average, -1000 -> min extreme value +1000 -> max extreme value
time_axis (TimeAxis) – The time-axis used when applying true-average to the time-series

Returns:

calculated_percentiles. Time-series list with evaluated percentile results, same length as input

Return type:

percentiles(self: shyft.time_series.TsVector, time_axis: shyft.time_series.TimeAxisFixedDeltaT, percentiles: collections.abc.Sequence[typing.SupportsInt | typing.SupportsIndex]) -> shyft.time_series.TsVector

Calculate the percentiles of the timeseries. over the specified time-axis. The definition is equal to e.g. NIST R7, excel, and in R. The time-series point_fx interpretation is used when performing the true-average over the time_axis periods. This functions works on bound expressions, for unbound expressions, use the DtsClient.percentiles.

See also DtsClient.percentiles() if you want to evaluate percentiles of an unbound expression.

Parameters:

percentiles (IntVector) – A list of numbers,[ 0, 25,50,-1,75,100] will return 6 time-series,`-1 -> arithmetic average`, -1000 -> min extreme value, ` +1000 max extreme value`
time_axis (TimeAxisFixedDeltaT) – The time-axis used when applying true-average to the time-series

Returns:

calculated_percentiles. Time-series list with evaluated percentile results, same length as input

Return type:

pop(*args, **kwargs)¶

Overloaded function.

pop(self: shyft.time_series.TsVector) -> shyft.time_series.TimeSeries

Remove and return the last item

pop(self: shyft.time_series.TsVector, i: typing.SupportsInt | typing.SupportsIndex) -> shyft.time_series.TimeSeries

Remove and return the item at index i

pow(*args, **kwargs)¶

Overloaded function.

pow(self: shyft.time_series.TsVector, number: typing.SupportsFloat | typing.SupportsIndex) -> shyft.time_series.TsVector

returns TsVector pow(py::self,number)

pow(self: shyft.time_series.TsVector, ts: shyft.time_series.TimeSeries) -> shyft.time_series.TsVector

returns TsVector pow(py::self,ts)

pow(self: shyft.time_series.TsVector, tsv: shyft.time_series.TsVector) -> shyft.time_series.TsVector

returns TsVector pow(py::self,tsv)

repeat(self: shyft.time_series.TsVector, repeat_time_axis: shyft.time_series.TimeAxis) → shyft.time_series.TsVector¶

Repeat all time-series over the given repeat_time_axis periods

Parameters:: repeat_time_axis (TimeAxis) – A time-axis that have the coarse repeat interval, like YEAR or similar
Returns:: tsv. time-series vector, where each element is repeated according to parameter
Return type:: TsVector

size(self: shyft.time_series.TsVector) → int¶

slice(self: shyft.time_series.TsVector, indexes: collections.abc.Sequence[SupportsInt | SupportsIndex]) → shyft.time_series.TsVector¶

returns a slice of self, specified by indexes

Parameters:: indexes (IntVector) – the indicies to pick out from self, if indexes is empty, then all is returned
Returns:: slice. a new TsVector, with content according to indexes specified
Return type:: TsVector

statistics(self: shyft.time_series.TsVector, ta: shyft.time_series.TimeAxis, p: SupportsInt | SupportsIndex) → shyft.time_series.TsVector¶

create a new vector of ts where each element is ts.statistics(ta,p)

Parameters:

ta (TimeAxis) – time-axis for the statistics
p (int) – percentile range [0..100], or statistical_property.AVERAGE|MIN_EXTREME|MAX_EXTREME

Returns:

tsv. a new time-series expression, will provide the statistics when requested

Return type:

sum(self: shyft.time_series.TsVector) → shyft.time_series.TimeSeries¶: returns sum of all ts in TsVector as ts as in reduce(add,..))

time_shift(self: shyft.time_series.TsVector, delta_t: shyft.time_series.time) → shyft.time_series.TsVector¶

create a new vector of ts that is a the time-shifted version of self

Parameters:: delta_t (time) – number of seconds to time-shift, positive values moves forward
Returns:: tsv. a new time-series, that appears as time-shifted version of self
Return type:: TsVector

transform(self: shyft.time_series.TsVector, points: Annotated[numpy.typing.ArrayLike, numpy.float64], method: shyft.time_series.interpolation_scheme) → shyft.time_series.TsVector¶

Create a transformed ts-vector, having values taken from pointwise function evaluation. Function values are determined by interpolating the given points, using the specified method. Valid method arguments are ‘polynomial’, ‘linear’ and ‘catmull-rom’.

Returns:: transform_tsv. New TsVector where each element is an evaluated-on-demand transformed time-series.
Return type:: TsVector

use_time_axis(self: shyft.time_series.TsVector, time_axis: shyft.time_series.TimeAxis) → shyft.time_series.TsVector¶

Create a new ts-vector applying TimeSeries.use_time_axis() on each member, e.g. resampling instant values at specified time-points.

Parameters:: time_axis (TimeAxis) – time-axis used to resample values from orignal ts
Returns:: tsv. time-series vector, where each element have resampled time-axis values
Return type:: TsVector

use_time_axis_from(self: shyft.time_series.TsVector, other: shyft.time_series.TimeSeries) → shyft.time_series.TsVector¶

Create a new ts-vector applying TimeSeries.use_time_axis_from() on each member

Parameters:: other (TimeSeries) – time-series that provides the wanted time-axis
Returns:: tsv. time-series vector, where each element have time-axis from other
Return type:: TsVector

value_range(self: shyft.time_series.TsVector, p: shyft.time_series.UtcPeriod) → list[float]¶

Computes min and max of all non-nan values in the period for bound expressions.

Parameters:: p (UtcPeriod)
Returns:: values. Resulting [min_value, max_value]. If all values are equal, min = max = the_value
Return type:: DoubleVectorVector

values_at(self: shyft.time_series.TsVector, t: shyft.time_series.time) → numpy.ndarray¶

Computes the value at specified time t for all time-series

Parameters:: t (utctime) – seconds since epoch 1970 UTC

Time series expressions¶

The elements in this category implement the time series expressions solution.

Class TsBindInfo¶

class shyft.time_series.TsBindInfo(self: shyft.time_series.TsBindInfo)¶

Bases: pybind11_object

TsBindInfo gives information about the time-series and it’s binding represented by encoded string reference Given that you have a concrete ts, you can bind that the bind_info.ts using bind_info.ts.bind() see also Timeseries.find_ts_bind_info() and Timeseries.bind()

__init__(self: shyft.time_series.TsBindInfo) → None¶

property id¶

a unique id/url that identifies a time-series in a ts-database/file-store/service

Type:: str

property ts¶

the ts, provides .bind(another_ts) to set the concrete values

Type:: TimeSeries

DTSS - The Distributed Time series System¶

The elements in this category implements the the DTSS. The DTSS provides ready to use services and components, which is useful in itself.

In addition, the services are extensible by python hooks, callbacks, that allow the user to extend/adapt the functionality to cover other time-series data base backends and services.

Note that DTSS is not a data-base as such, but do have a built in high performance time-series db. The DTSS is better viewed as computing component/service, that are capable of evaluating time-series expressions, extracting the wanted information, and sending it back to the clients. One of the important properties of the DTSS is that we can bring the heavy computations to where the data is located. In addition it as a specialized advanced caching system that allows evaluations to run on memory(utilizing multi-core evaluations).

The DTSS contains a high performance in memory queue for messages that consists of collections of time-series. The queue mechanism also provide end-to-end handshake, so that producer can know that consumer have processed the queue message.

The Transfer service built into the DTSS also allows for efficient direct replication, to other DTSS instances for timeseries that matches regular expression, and even with regular expression translation before pushing to remote instance.a

The transfer mechanism is resilient to network and service interruptions, and can propagate changes to large sets of time-series in a few millisecond (limited by network/storage bandwidth).

The open design allows it to utilize any existing legacy ts-databases/services through customization points.

Class DtsServer¶

class shyft.time_series.DtsServer(self: shyft.time_series.DtsServer, config: shyft.time_series.ServerConfig = shyft.time_series.ServerConfig(stale_duration='1970-01-01T01:00:00Z', stale_sweep_interval='1970-01-01T00:00:00.100000Z', log=shyft.time_series.LogConfig(file='', level=200)))¶

Bases: pybind11_object

A distributed time-series server.

The server part of the Shyft Distributed TimeSeries System(DTSS). Capable of processing time-series messages and responding accordingly.

It has dual service interfaces:

raw-socket boost serialized binary, use DtsClient

web-api, web-socket(https/wss w. auth supported) using boost.beast, boost.spirit to process/emit messages. This also supports ts-change subscriptions.

python customization and extension capability

The user can setup callback to python to handle unbound symbolic time-series references, ts-urls. This means that you can use your own ts database backend if you have one that can beat the shyft-internal ts-db.

The DtsServer then resolves symbolic references reading time-series from a service or storage for the specified period. The server object will then compute the resulting time-series vector, and respond back to clients with the results multi-node considerations:

firewall/routing: ensure that the port you are using are open for ip-traffic(use ssh-tunnel if you need ssl/tls)

we strongly recommend using linux for performance and long term stability

The Dts also support replica mode, that allows scaling out cache and computations to several Dtss instances. It also provide a read-only mode for the replica nodes. see configure_replica

backend storage

There are several internal back-ends, and customization for external storage as well. Internal storage containers:

rocksdb - by facebook, configurable specifying ts_rdb in the set_container method

filedb - fast zero overhead, and simple internal binary formats, configurable specifying ts_db in the set_container method

The kind of backend storage for the backing store ts-containers is specified in the set_container method, for explicit creation of ts-containers. Notice that for remotely client created containers for geo time-series storage, the default_geo_db_type applies, set to ts_rdb.

External storage can be setup by supplying python callbacks for the find,`read`,`store` and remove_container hooks.

Then specifying something else than the shyft:// pre-fix for the ts-urls, allows any external storage to be used.

To ensure that containers are (remotely) found and configured after reboot/restart, ensure to provide a dtss configuration file where this information is stored.

HPC setup, configure linux os user limits

For high-performance environments, the ulimit, especially memory, number of files open, needs to be set to higher values than the defaults, usually nofiles is 1024, which is to low for HPC apps. We recommend 4096, or 8192 or even higher for demanding databases. To control heap allocations and virtual size, use gnu malloc tunable parameters. The env. MALLOC_ARENA_MAX=1, helps controlling the max heap size of the dtss to the cache-memory target. It has been found to be useful in raw-iron, or even laptop setups, and required in k8s setups where memory limits are enforced.

For tuning rocksdb, read tuning guides for those libraries, -we provide some basic parameters for tuning, but more can be added if needed.

See also

DtsClient

__init__(self: shyft.time_series.DtsServer, config: shyft.time_series.ServerConfig = shyft.time_series.ServerConfig(stale_duration='1970-01-01T01:00:00Z', stale_sweep_interval='1970-01-01T00:00:00.100000Z', log=shyft.time_series.LogConfig(file='', level=200))) → None¶

add_auth_tokens(self: shyft.time_series.DtsServer, tokens: collections.abc.Sequence[str]) → None¶

Adds auth tokens, and activate authentication. The tokens is compared exactly to the autorization token passed in the request. Authorization should onlye be used for the https/wss, unless other measures(vpn/ssh tunnels etc.) are used to protect auth tokens on the wire Important! Ensure to start_web_api with tls_only=True when using auth!

Parameters:: () (tokens) – list of tokens, where each token is like Basic dXNlcjpwd2Q=, e.g: base64 user:pwd

property alive_connections¶

returns currently alive connections to the server

Type:: int

property auth_needed¶

returns true if the server is setup with auth-tokens, requires web-api clients to pass a valid token

Type:: bool

auth_tokens(self: shyft.time_series.DtsServer) → list[str]¶: returns the registered authentication tokens.

cache(self: shyft.time_series.DtsServer, ts_ids: collections.abc.Sequence[str], ts_vector: shyft.time_series.TsVector) → None¶

add/update specified ts_ids with corresponding ts to cache please notice that there is no validation of the tds_ids, they are threated as identifiers,not verified against any existing containers etc. Requests that follow, will use the cached item as long as it satisfies the identifier and the coverage period requested

Parameters:

ts_ids (StringVector) – a list of time-series ids
ts_vector (TsVector) – a list of corresponding time-series

property cache_max_items¶

cache_max_items is the maximum number of time-series identities that are kept in memory. Elements exceeding this capacity is elided using the least-recently-used algorithm. Notice that assigning a lower value than the existing value will also flush out time-series from cache in the least recently used order.

Type:: int

property cache_memory_target¶

The memory max target in number of bytes. If not set directly the following equation is used: cache_memory_target = cache_ts_initial_size_estimate * cache_max_items When setting the target directly, the number of items in the cache is set so that real memory usage is less than the specified target. The setter could cause elements to be flushed out of the cache.

Type:: int

property cache_stats¶

the current cache statistics

Type:: CacheStats

property cache_ts_initial_size_estimate¶

The initial time-series size estimate in bytes for the cache mechanism. memory-target = cache_ts_initial_size_estimate * cache_max_items algorithm. Notice that assigning a lower value than the existing value will also flush out time-series from cache in the least recently used order.

Type:: int

property can_modify¶

true if this instance can modify main dtss server items, time-series, containers etc.

Type:: bool

property cb¶

callback for binding unresolved time-series references to concrete time-series. Called if the incoming messages contains unbound time-series. The signature of the callback function should be TsVector cb(Vector,utcperiod)

Examples:

>>> from shyft import time_series as sa

>>> def resolve_and_read_ts(ts_ids,read_period):
>>>     print('ts_ids:', len(ts_ids), ', read period=', str(read_period))
>>>     ta = sa.TimeAxis(read_period.start, sa.deltahours(1), read_period.timespan()//sa.deltahours(1))
>>>     x_value = 1.0
>>>     r = sa.TsVector()
>>>     for ts_id in ts_ids :
>>>         r.append(sa.TimeSeries(ta, fill_value = x_value))
>>>         x_value = x_value + 1
>>>     return r
>>> # and then bind the function to the callback
>>> dtss=sa.DtsServer()
>>> dtss.cb=resolve_and_read_ts
>>> dtss.set_listening_port(20000)
>>> dtss.process_messages(60000)

Type:: Callable[[StringVector,UtcPeriod],TsVector]

clear(self: shyft.time_series.DtsServer) → None¶: stop serving connections, gracefully.

See also

cb, process_messages(msec),start_server()

clear_cache_stats(self: shyft.time_series.DtsServer) → None¶: clear accumulated cache_stats

close(self: shyft.time_series.DtsServer) → None¶: stop serving connections, gracefully.

See also

cb, process_messages(msec),start_server()

property configuration_file¶

configuration file to enable persistent container configurations over coldstarts

Type:: str

configure_replica(self: shyft.time_series.DtsServer, ip: str, port: SupportsInt | SupportsIndex, poll_interval: SupportsFloat | SupportsIndex, unsubscribe_threshold: SupportsInt | SupportsIndex, unsubscribe_max_delay: SupportsFloat | SupportsIndex, protocol_version: SupportsInt | SupportsIndex = 0, can_modify: bool = True, subscription_consistency_interval: SupportsFloat | SupportsIndex = 0.0) → None¶

Set replica mode, redirecting all IO calls on this dtss to the primary ip:port dtss. This instance of the dtss is kept in sync with changes done on the primary using subscription to changes on the primary Calculations, and caches are still done locally unloading the computational efforts from the primary.

Parameters:

ip (str) – The ip address where the primary dtss is running
port (int) – The port number for the primary dtss
poll_interval (time) – [s] max time between each update from primary, typicall 0.1 s is ok
unsubscribe_threshold (int) – minimum number of unsubscribed time-series before also unsubscribing from the primary
unsubscribe_max_delay (int) – maximum time to delay unsubscriptions, regardless number
protocol_version (int) – version to run the sync over, defaults to internal
can_modify (bool) – if false, inhibit any modifying operations to primary/main dtss
subscription_consistency_interval (float) – if >0.0, then enable costly consistency check and fix functionality. It should be > 10 minutes or more due to high cost

property default_geo_db_config¶

Default parameters for geo db created by clients

Type:: GeoTimeSeriesConfiguration

property default_geo_db_type¶

default container type for geo db created by clients,(ts_rdb,`ts_db`), defaults set to ts_rdb

Type:: str

property failed_connections¶

returns count of connections closed due to bad requests

Type:: int

find(self: shyft.time_series.DtsServer, search_expression: str) → list[shyft.time_series.TsInfo]¶

Find ts information that fully matches the regular search-expression. For the shyft file based backend, take care to specify path elements precisely, so that the directories visited are minimised. e.g:a/.*/my.ts Will prune out any top level directory not starting with a, but will match any subdirectories below that level. Refer to python test-suites for a wide range of examples using find. Notice that the regexp search algorithm respects the case. Please be aware that custom backend by python extension might have different rules.

Parameters:: search_expression (str) – regular search-expression, to be interpreted by the back-end tss server
Returns:: ts_info_vector. The search result, as vector of TsInfo objects
Return type:: TsInfoVector

See also

TsInfo,TsInfoVector

property find_cb¶

callback for finding time-series using a search-expression. Called everytime the .find() method is called. The signature of the callback function should be fcb(search_expr: str)->TsInfoVector

Examples:

>>> from shyft import time_series as sa

>>> def find_ts(search_expr: str)->sa.TsInfoVector:
>>>     print('find:',search_expr)
>>>     r = sa.TsInfoVector()
>>>     tsi = sa.TsInfo()
>>>     tsi.name = 'some_test'
>>>     r.append(tsi)
>>>     return r
>>> # and then bind the function to the callback
>>> dtss=sa.DtsServer()
>>> dtss.find_cb=find_ts
>>> dtss.set_listening_port(20000)
>>> # more code to invoce .find etc.

Type:: Callable[[str],TsInfoVector]

flush_cache(self: shyft.time_series.DtsServer, ts_ids: collections.abc.Sequence[str]) → None¶

flushes the specified ts_ids from the cache Has only effect for ts-ids that are in cache, non-existing items are ignored

Parameters:: ts_ids (StringVector) – a list of time-series ids to flush out

flush_cache_all(self: shyft.time_series.DtsServer) → None¶: flushes all items out of the cache (cache_stats remain un-touched)

property geo_ts_read_cb¶

Callback for reading geo_ts db. Called everytime there is a need for geo_ts not stored in cached. The signature of the callback function should be grcb(cfg:GeoTimeSeriesConfiguration, slice:GeoSlice)->GeoMatrix

Type:: Callable[[GeoTimeSeriesConfiguration,GeoSlice],GeoMatrix]

property geo_ts_store_cb¶

callback for storing to geo_ts db. Called everytime the client.store_geo_ts() method is called. The signature of the callback function should be gscb(cfg:GeoTimeSeriesConfiguration, tsm:GeoMatrix, replace:bool)->None

Type:: Callable[[GeoTimeSeriesConfiguration,GeoMatrix,bool],None]

get_container_names(self: shyft.time_series.DtsServer) → list[str]¶: Return names of containers available on the server

get_geo_db_ts_info(self: shyft.time_series.DtsServer) → list[shyft.time_series.GeoTimeSeriesConfiguration]¶

Returns the configured geo-ts data-bases on the server, so queries can be specified and formulated

Returns:

A strongly typed list of GeoTimeseriesConfiguration

Return type:

GeoTimeseriesConfigurationVector

See also

.geo_evaluate()

get_listening_ip(self: shyft.time_series.DtsServer) → str¶

Get the current ip listen address

Returns:: listening ip. note that 0.0.0.0 means listening for all interfaces

get_listening_port(self: shyft.time_series.DtsServer) → int¶: returns the port number it’s listening at for serving incoming request

get_max_connections(self: shyft.time_series.DtsServer) → int¶: returns the maximum number of connections to be served concurrently

property graceful_close_timeout_ms¶

how long to let a connection linger after messages is processed to allow for flushing out reply to the client. Ref to dlib.net dlib.net/dlib/server/server_kernel_abstract.h.html

Type:: int

is_running(self: shyft.time_series.DtsServer) → bool¶: true if server is listening and running

See also

start_server(),process_messages(msec)

maintain_backend_container(self: shyft.time_series.DtsServer, name: str, info_db: bool, data_db: bool) → None¶

maintain backend container store, e.g. rocksdb. The info_db and data_db controls which part of backend are maintained. The info part is the key-value store for ts-name to ts-info The data part is the key-value store for the data values of the time-series. The rocksdb is to a large degree self-maintained and auto-tuned, but introduction of certain filters (bloom filter for the info db, and prefix-transform filter for data db), will require maintenance of existing databases to take full performance effect of those filter. Notice that the maintenance is not needed unless there is performance issues that can be addressed by the introduced filter-configuration. The info part are usually a small data-base, so the operation goes in few seconds The data part can however span several terra-bytes, and would involve reading and writing the entire amount stored. We recommend running this on realistic copy of prod on test/preprod system, and verify time-usage and performance improvements. For detailed technical description refer to rocksdb CompactRange function.

Parameters:

name (str) – Name of the container as
info_db (bool) – If true, to force maintenance for the info-part of db
data_db (bool) – If true, to force maintenance for the data-part of db

Notes

This functions can be called on a running db, but it will impact performance for the length of the call, - so ensure to compute size and time estimates before starting

process_messages(self: shyft.time_series.DtsServer, msec: SupportsInt | SupportsIndex) → None¶

wait and process messages for specified number of msec before returning the dtss-server is started if not already running

Parameters:: msec (int) – number of millisecond to process messages

Notes

this method releases GIL so that callbacks are not blocked when the

dtss-threads perform the callback

See also

cb,start_server(),is_running,clear()

q_ack(self: shyft.time_series.DtsServer, name: str, msg_id: str, diagnostics: str) → None¶

After q_get, q_ack confirms that the message is ok/handled back to the process that called q_put.

Parameters:

() (diagnostics) – the name of the queue
() – the msg_id, required to be unique within current messages keept by the queue
() – the freetext diagnostics to but along with the message, we recommend json formatted

q_add(self: shyft.time_series.DtsServer, name: str) → None¶

Add a a named queue to the dtss server

Parameters:: () (name) – the name of the new queue, required to be unique

q_find(self: shyft.time_series.DtsServer, name: str, msg_id: str) → shyft.time_series.QueueMessage¶

Find a historical message with the given id

Parameters:

() (msg_id) – the name of the queue
() – the id of the message

Returns:

q_msg. The historical message, or None if the message dosen’t exist

q_get(self: shyft.time_series.DtsServer, name: str, max_wait: shyft.time_series.time) → shyft.time_series.QueueMessage¶

Get a message out from the named queue, waiting max_wait time for it if it’s not already there.

Parameters:

() (max_wait) – the name of the queue
() – max_time to wait for message to arrive

Returns:

q_msg. A queue message consisting of .info describing the message, and the time-series vector .tsv

q_list(self: shyft.time_series.DtsServer) → list[str]¶: returns a list of defined queues on the dtss server

q_maintain(self: shyft.time_series.DtsServer, name: str, keep_ttl_items: bool, flush_all: bool = False) → None¶

Maintains, removes items that has passed through the queue, and are marked as done. To flush absolutely all items, pass flush_all=True.

Parameters:

() (flush_all) – the name of the queue
() – If true, the ttl set for the done messages are respected, and they are not removed until the create+ttl has expired
() – removes all items in the queue and kept by the queue, the queue is emptied

q_msg_info(self: shyft.time_series.DtsServer, name: str, msg_id: str) → shyft.time_series.QueueMessageInfo¶

From the specified queue, fetch info about specified msg_id. By inspecting the provided information, one can see when the messaage is created, fetched, and done with.

Parameters:

() (msg_id) – the name of the queue
() – the msg_id

Returns:

msg_info. the information/state of the identified message

q_msg_infos(self: shyft.time_series.DtsServer, name: str) → list[shyft.time_series.QueueMessageInfo]¶

Returns all message informations from a queue, including not yet pruned fetched/done messages

Parameters:: () (name) – the name of the queue
Returns:: msg_infos. the list of information keept in the named queue

q_put(self: shyft.time_series.DtsServer, name: str, msg_id: str, description: str, ttl: shyft.time_series.time, tsv: shyft.time_series.TsVector) → None¶

Put a message, as specified with the supplied parameters, into the specified named queue.

Parameters:

() (tsv) – the name of the queue
() – the msg_id, required to be unique within current messages keept by the queue
() – the freetext description to but along with the message, we recommend json formatted
() – time-to-live for the message after done, if specified, the q_maintain process can be asked to keep done messages that have ttl
() – time-series vector, with the wanted payload of time-series

q_remove(self: shyft.time_series.DtsServer, name: str) → None¶

Removes a named queue from dtss server, including all data in flight on the queue

Parameters:: () (name) – the name of the queue

q_size(self: shyft.time_series.DtsServer, name: str) → int¶

Returns number of queue messages waiting to be read by q_get.

Parameters:: () (name) – the name of the queue
Returns:: unread count. number of elements queued up

read(self: shyft.time_series.DtsServer, ts_ids: collections.abc.Sequence[str], read_period: shyft.time_series.UtcPeriod, use_ts_cached_read: bool = True, update_ts_cache: bool = True) → shyft.time_series.TsVector¶

Reads from the db-backend/cache the specified ts_ids for covering read_period. NOTE: That the ts-backing-store, either cached or by read, will return data for:

at least the period needed to evaluate the read_period

In case of cached result, this will currently involve the entire matching cached time-series segment.

Parameters:

ts_ids (StringVector) – a list of shyft-urrls, like shyft://abc/def
read_period (UtcPeriod) – the valid non-zero length period that the binding service should read from the backing ts-store/ts-service
use_ts_cached_read (bool) – use of server-side ts-cache
update_ts_cache (bool) – when reading time-series, also update the cache with the data

Returns:

tsvector. an evaluated list of point time-series in the same order as the input list

Return type:

See also

DtsServer

remove_auth_tokens(self: shyft.time_series.DtsServer, tokens: collections.abc.Sequence[str]) → None¶

removes auth tokens, if it matches all available tokens, then deactivate auth requirement for clients

Parameters:: () (tokens) – list of tokens, where each token is like Basic dXNlcjpwd2Q=, e.g: base64 user:pwd

remove_container(self: shyft.time_series.DtsServer, container_url: str, delete_from_disk: bool = False) → None¶

remove an internal shyft store container or an external container from the dtss-server. container_url on the form shyft://<container>/ will remove internal containers all other urls with be forwarded to the remove_external_cb callback on the server removal of containers can take a long time to finish

Parameters:

container_url (str) – url of the container as pr. url definition above
delete_from_disk (bool) – Flag to indicate if the container should be deleted from disk

property remove_container_cb¶

callback for removing external containers. Called when the .remove_container() method is called with a non-shyft container url. The signature of the callback function should be rcb(container_url: string, remove_from_disk: bool)->None

Type:: Callable[[str, bool],None]

set_auto_cache(self: shyft.time_series.DtsServer, active: bool) → None¶

set auto caching all reads active or passive. Default is off, and caching must be done through explicit calls to .cache(ts_ids,ts_vector)

Parameters:: active (bool) – if set True, all reads will be put into cache

set_can_remove(self: shyft.time_series.DtsServer, can_remove: bool) → None¶

Set whether the DtsServer supports removing time-series The default setting is false, su unless this method is called with true as argument the server will not allow removing data using DtsClient.remove.

Parameters:: can_remove (bool) – true if the server should allow removing data. false otherwise

set_container(self: shyft.time_series.DtsServer, name: str, root_dir: str, container_type: str = '', cfg: shyft.time_series.DtssCfg = DtssCfg(ppf=1024, compress=DtssCfg.CompressOptions(algo=no_compression, bottom_algo=no_compression, index_compression=true), max_file_size=104857600, write_buffer_size=67108864, log_level=200, test_mode=0, ix_cache=0, ts_cache=0)) → None¶

sets, or replaces, an internal shyft store container to the dtss-server. All ts-urls with shyft://<container>/ will resolve to this internal time-series storage for find/read/store operations

Parameters:

name (str) – Name of the container as pr. url definition above
root_dir (str) – A valid directory root for the container
container_type (str) – one of (‘ts_rdb’, ‘ts_db’), container type to add.

Notes

currently this call should only be used when the server is not processing messages, - before starting, or after stopping listening operations

set_geo_ts_db(self: shyft.time_series.DtsServer, geo_ts_cfg: shyft.time_series.GeoTimeSeriesConfiguration) → None¶

This add/replace a geo-ts database to the server, so that geo-related requests can be resolved by means of this configuation and the geo-related callbacks.

Parameters:: geo_ts_cfg (GeoTimeseriesConfiguration) – The configuration for the new geo-ts data-base

set_listening_ip(self: shyft.time_series.DtsServer, ip: str) → None¶

Set the ip address to specific interface ip. Must be called prior to the start server method

Parameters:: () (ip) – ip address, like 127.0.0.1 for local host only interface

set_listening_port(self: shyft.time_series.DtsServer, port_no: SupportsInt | SupportsIndex) → None¶

set the listening port for the service

Parameters:

port_no (int) – a valid and available tcp-ip port number to listen on.
20000 (typically it could be)

Returns:

nothing.

Return type:

None

set_master_slave_mode(self: shyft.time_series.DtsServer, ip: str, port: SupportsInt | SupportsIndex, master_poll_time: SupportsFloat | SupportsIndex, unsubscribe_threshold: SupportsInt | SupportsIndex, unsubscribe_max_delay: SupportsFloat | SupportsIndex, protocol_version: SupportsInt | SupportsIndex = 0, can_modify: bool = True, subscription_consistency_interval: SupportsFloat | SupportsIndex = 0.0) → None¶

Deprecated(use configure_replica)

Parameters:

ip (str) – The ip address where the primary dtss is running
port (int) – The port number for the primary dtss
master_poll_time (time) – [s] max time between each update from primary, typicall 0.1 s is ok
unsubscribe_threshold (int) – minimum number of unsubscribed time-series before also unsubscribing from the primary
unsubscribe_max_delay (int) – maximum time to delay unsubscriptions, regardless number
protocol_version (int) – version to run the sync over, defaults to internal
can_modify (bool) – if false, inhibit any modifying operations to primary/main dtss

set_max_connections(self: shyft.time_series.DtsServer, max_connect: SupportsInt | SupportsIndex) → None¶

limits simultaneous connections to the server (it’s multithreaded, and uses on thread pr. connect)

Parameters:: max_connect (int) – maximum number of connections before denying more connections

See also

get_max_connections()

property stale_connection_close_count¶

returns count of connection closed due to stale/no communication activity

Type:: int

start_async(self: shyft.time_series.DtsServer) → int¶

(deprecated, use start_server) start server listening in background, and processing messages

See also

set_listening_port(port_no),set_listening_ip,is_running,cb,process_messages(msec)

Returns:: port_no. the port used for listening operations, either the value as by set_listening_port, or if it was unspecified, a new available port
Return type:: in

Notes

you should have setup up the callback, cb before calling start_async

Also notice that processing will acquire the GIL

-so you need to release the GIL to allow for processing messages

See also

process_messages(msec)

start_server(self: shyft.time_series.DtsServer) → int¶

start server listening in background, and processing messages

See also

set_listening_port(port_no),set_listening_ip,is_running,cb,process_messages(msec)

Returns:: port_no. the port used for listening operations, either the value as by set_listening_port, or if it was unspecified, a new available port
Return type:: in

Notes

you should have setup up the callback, cb before calling start_server

Also notice that processing will acquire the GIL

-so you need to release the GIL to allow for processing messages

See also

process_messages(msec)

start_web_api(self: shyft.time_series.DtsServer, host_ip: str, port: SupportsInt | SupportsIndex, doc_root: str, fg_threads: SupportsInt | SupportsIndex = 2, bg_threads: SupportsInt | SupportsIndex = 4, tls_only: bool = False) → int¶

starts the dtss web-api on the specified host_ip, port, doc_root and number of threads

Parameters:

host_ip (str) – 0.0.0.0 for any interface, 127.0.0.1 for local only etc.
port (int) – port number to serve the web_api on, ensure it’s available!
doc_root (str) – directory from which we will serve http/https documents, like index.html etc.
fg_threads (int) – number of web-api foreground threads, typical 1-4 depending on load
bg_threads (int) – number of long running background threads workers to serve dtss-request etc.
tls_only (bool) – default false, set to true to enforce tls sessions only.

Returns:

port. real port number used, if 0 is passed as port it is auto-allocated

Return type:

int

stop_server(self: shyft.time_series.DtsServer, timeout: SupportsInt | SupportsIndex = 1000) → None¶: stop serving connections, gracefully.

See also

start_server()

stop_web_api(self: shyft.time_series.DtsServer) → None¶: Stops any ongoing web-api service

store(self: shyft.time_series.DtsServer, tsv: shyft.time_series.TsVector, store_policy: shyft.time_series.StorePolicy) → None¶

Store the time-series in the ts-vector in the dtss backend. Stores the time-series fragments data passed to the backend. If store_policy.strict == True: It is semantically stored as if

first erasing the existing stored points in the range of ts.time_axis().total_period()

then inserting the points of the ts.

Thus, only modifying the parts of time-series that are covered by the ts-fragment passed. If there is no previously existing time-series, its merely stores the ts-fragment as the initial content and definition of the time-series.

When creating time-series 1st time, pay attention to the time-axis, and point-interpretation as this remains the properties of the newly created time-series. Storing 15min data to a time-series defined initially as hourly series will raise exception. On the other hand, the variable interval time-series are generic and will accept any time-resolution to be stored

If store_policy.strict ==False The passed time-series fragment is interpreted as a f(t), and projected to the time-axis time-points/intervals of the target time-series If the target time-series is a stair-case type (POINT_AVERAGE_VALUE), then the true average of the passed time-series fragment is used to align with the target. If the target time-series is a linear type (POINT_INSTANT_VALUE), then the f(t) of the passed time-series fragment for the time-points of the target-series is used. The store_policy.recreate == True, is used to replace the entire definition of any previously stored time-series. This is semantically as if erasing the previously stored time-series and replacing its entire content and definition, starting fresh with the newly passed time-series. The store_policy.best_effort == True or False, controls how logical errors are handled. If best_effort is set to True, then all time-series are attempted stored, and if any failed, the returned value of the function will be non-empty list of diagnostics identifying those that failed and the diagnostics. If best_effort is set to False, then exception is raised on the first item that fails, the remaining items are not stored. The time-series should be created like this, with url and a concrete point-ts:

>>>   a=sa.TimeSeries(ts_url,ts_points)
>>>   tsv.append(a)

Parameters:

tsv (TsVector) – ts-vector with time-series, url-reference and values to be stored at dtss server
store_policy (StorePolicy) – Determines how to project the passed time-series fragments to the backend stored time-series

Returns:

diagnostics. For any failed items, normally empty

Return type:

TsDiagnosticsItemList

See also

TsVector

property store_ts_cb¶

callback for storing time-series. Called everytime the .store_ts() method is called and non-shyft urls are passed. The signature of the callback function should be scb(tsv: TsVector)->None

Examples:

>>> from shyft import time_series as sa

>>> def store_ts(tsv:sa.TsVector)->None:
>>>     print('store:',len(tsv))
>>>     # each member is a bound ref_ts with an url
>>>     # extract the url, decode and store
>>>     #
>>>     #
>>>     return
>>> # and then bind the function to the callback
>>> dtss=sa.DtsServer()
>>> dtss.store_ts_cb=store_ts
>>> dtss.set_listening_port(20000)
>>> # more code to invoce .store_ts etc.

Type:: Callable[[TsVector],None]

swap_container(self: shyft.time_series.DtsServer, container_name_a: str, container_name_b: str = False) → None¶

Swap the backend storage for container a and b. The content of a and b should be equal prior to the call to ensure wanted semantics, as well as cache correctness. This is the case if a is immutable, and copied to b prior to the operation. If a is not permanently immutable, it has to be ensured at least for the time where the copy/swap operation is done. The intended purpose is to support migration and moving ts-db backends. When swap is done, the remove_container can be used for the container that is redunant. A typical operation is copy a->`a_tmp`, then swap(a,`a_tmp`), then remove(shyft://a_tmp,True)

Parameters:

container_name_a (str) – Name of container a
container_name_b (str) – Name of container b

class shyft.time_series.DtssCfg(*args, **kwargs)¶

Bases: pybind11_object

Configuration for google level db specific parameters.

Each parameter has reasonable defaults, have a look at rocksdb documentation for the effect of max_file_size, write_buffer_size and compression. The ppf remains constant once db is created (any changes will be ignored). Notice that when storing ts, or geo-db, individual ppf pr. ts can be set. The other can be changed on persisted/existing databases.

About compression: Turns out although very effective for a lot of time-series, it has a single thread performance cost of 2..3x native read/write performance due to compression/decompression.

However, for geo dtss we are using multithreaded writes, so performance is limited to the io-capacity, so it should be set to true for those kind of scenarios. Using compression might also reduce real IO operations to the ratio of compression, and used together with ix_cache, rocksdb can reduce IO operation and increase performance.

Overloaded function.

__init__(self: shyft.time_series.DtssCfg) -> None
__init__(self: shyft.time_series.DtssCfg, ppf: typing.SupportsInt | typing.SupportsIndex, compress: shyft.time_series.DtssCfg.CompressOptions, max_file_size: typing.SupportsInt | typing.SupportsIndex, write_buffer_size: typing.SupportsInt | typing.SupportsIndex, log_level: typing.SupportsInt | typing.SupportsIndex = 200, test_mode: typing.SupportsInt | typing.SupportsIndex = 0, ix_cache: typing.SupportsInt | typing.SupportsIndex = 1000000000, ts_cache: typing.SupportsInt | typing.SupportsIndex = 1000000000) -> None

construct a DtssCfg with all values specified

__init__(self: shyft.time_series.DtssCfg, ppf: typing.SupportsInt | typing.SupportsIndex = 1024, compress: bool = False, max_file_size: typing.SupportsInt | typing.SupportsIndex = 104857600, write_buffer_size: typing.SupportsInt | typing.SupportsIndex = 67108864, log_level: typing.SupportsInt | typing.SupportsIndex = 200, test_mode: typing.SupportsInt | typing.SupportsIndex = 0, ix_cache: typing.SupportsInt | typing.SupportsIndex = 1000000000, ts_cache: typing.SupportsInt | typing.SupportsIndex = 1000000000) -> None

Legacy constructor: construct a DtssCfg using bool for compression (maps to snappy if true, no_compression if false)

class CompressAlgorithm(self: shyft.time_series.DtssCfg.CompressAlgorithm, value: SupportsInt | SupportsIndex)¶

Bases: pybind11_object

CompressAlgorithm to use.

Members:

NO_COMPRESSION

SNAPPY

LZ4

ZSTD

LZ4 = <CompressAlgorithm.LZ4: 2>¶

NO_COMPRESSION = <CompressAlgorithm.NO_COMPRESSION: 0>¶

SNAPPY = <CompressAlgorithm.SNAPPY: 1>¶

ZSTD = <CompressAlgorithm.ZSTD: 3>¶

__init__(self: shyft.time_series.DtssCfg.CompressAlgorithm, value: SupportsInt | SupportsIndex) → None¶

DtssCfg.CompressAlgorithm.name -> str

property value¶

class CompressOptions(*args, **kwargs)¶

Bases: pybind11_object

Configuration for compression.

Overloaded function.

__init__(self: shyft.time_series.DtssCfg.CompressOptions) -> None
__init__(self: shyft.time_series.DtssCfg.CompressOptions, algo: shyft.time_series.DtssCfg.CompressAlgorithm, bottom_algo: shyft.time_series.DtssCfg.CompressAlgorithm, index_compression: bool) -> None

Construct a compress_options with specified compression algorithms and index compression flag

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.DtssCfg.CompressOptions) -> None
__init__(self: shyft.time_series.DtssCfg.CompressOptions, algo: shyft.time_series.DtssCfg.CompressAlgorithm, bottom_algo: shyft.time_series.DtssCfg.CompressAlgorithm, index_compression: bool) -> None

Construct a compress_options with specified compression algorithms and index compression flag

property algo¶

Compression algorithm to use.

Type:: CompressionAlgorithm

property bottom_algo¶

Compression algorithm to use for bottom level and blob-files(if enabled).

Type:: CompressionAlgorithm

property index_compression¶

Compression for index data.

Type:: bool

property ts_info_compression¶

compress ts-info.

Type:: bool

property zstd_level¶

zstd compression level.

Type:: int

property zstd_threads¶

zstd compression threads.

Type:: int

LZ4 = <CompressAlgorithm.LZ4: 2>¶

NO_COMPRESSION = <CompressAlgorithm.NO_COMPRESSION: 0>¶

SNAPPY = <CompressAlgorithm.SNAPPY: 1>¶

ZSTD = <CompressAlgorithm.ZSTD: 3>¶

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.DtssCfg) -> None
__init__(self: shyft.time_series.DtssCfg, ppf: typing.SupportsInt | typing.SupportsIndex, compress: shyft.time_series.DtssCfg.CompressOptions, max_file_size: typing.SupportsInt | typing.SupportsIndex, write_buffer_size: typing.SupportsInt | typing.SupportsIndex, log_level: typing.SupportsInt | typing.SupportsIndex = 200, test_mode: typing.SupportsInt | typing.SupportsIndex = 0, ix_cache: typing.SupportsInt | typing.SupportsIndex = 1000000000, ts_cache: typing.SupportsInt | typing.SupportsIndex = 1000000000) -> None

construct a DtssCfg with all values specified

__init__(self: shyft.time_series.DtssCfg, ppf: typing.SupportsInt | typing.SupportsIndex = 1024, compress: bool = False, max_file_size: typing.SupportsInt | typing.SupportsIndex = 104857600, write_buffer_size: typing.SupportsInt | typing.SupportsIndex = 67108864, log_level: typing.SupportsInt | typing.SupportsIndex = 200, test_mode: typing.SupportsInt | typing.SupportsIndex = 0, ix_cache: typing.SupportsInt | typing.SupportsIndex = 1000000000, ts_cache: typing.SupportsInt | typing.SupportsIndex = 1000000000) -> None

Legacy constructor: construct a DtssCfg using bool for compression (maps to snappy if true, no_compression if false)

property avoid_flush_during_recovery¶

skip memtable flush on open for faster startup; rocksdb default is false, shyft default is true

Type:: bool

property avoid_flush_during_shutdown¶

set true to skip memtable flush on close (faster shutdown); set false to flush cleanly and avoid compaction burst on next open

Type:: bool

property blob_cache_size¶

(default 0 = share ts_cache) dedicated cache size in bytes for blob values; set to a value covering your hot working set (e.g. n_ts * fragment_bytes) to avoid Ceph I/O on tangling-fragment reads after a pod restart

Type:: int

property blob_file_size¶

target size for blob files

Type:: int

property bytes_pr_sync¶

rocksdb bytes_pr_sync, typically 4M

Type:: int

property compaction_readahead_size¶

readahead for compaction reads (bytes); 0=rocksdb default (2MB). Recommend 16-32MB for Ceph/network storage

Type:: int

property compression¶

configured compress options

Type:: CompressOptions

property enable_blob_files¶

enable blob-files for keeping ts-frag values

Type:: bool

property ix_cache¶

low-level rocksdb index-cache, important performance tuning for cases with many time-series

Type:: int

property level0_file_num_compaction_trigger¶

(default 8) number of L0 files that triggers background compaction; lower values keep L0 smaller at the cost of more frequent compactions

Type:: int

property level0_slowdown_writes_trigger¶

(default 100) number of L0 files at which writes are throttled to let compaction catch up; raise if you see spurious write stalls on bursty ingest workloads

Type:: int

property level0_stop_writes_trigger¶

(default 200) number of L0 files at which writes are hard-stopped; keep at roughly 2x slowdown_trigger to give compaction a chance to drain before a full stall

Type:: int

property log_level¶

default warn(200), trace(-1000),debug(0),info(100),error(300),fatal(400)

Type:: int

property max_bytes_for_level_base¶

256 GB base -> 2.56 TB next level -> 25.6 TB etc. Size this to comfortably hold your expected total dataset at the base level or one level above. For the 10-TB / Arome case 256 GB is a good starting point; raise to 1 TB for 28 TB+ databases. Set to 0 for legacy auto (10*max_file_size, uncapped). Note: if blob files are used, most of data is stored in blob files, and only ref to blob goes in sst. Set the size accordingly to the value blob reference like 16 bytes pr entry.

Type:: int
Type:: (default 256 GB) target size in bytes for the base compaction level. With ts_frag_dynamic_level_bytes=True, upper levels are derived proportionally (10x each)

property max_file_opening_threads¶

rocksdb max_file_opening_threads, 4..16

Type:: int

property max_file_size¶

(default 100Mega), target size of sst files for ts-fragments, for larger DB consider 1G and more

Type:: int

property max_open_files¶

limit number of files opened at the same time, related to ulimit -n for the process

Type:: int

property max_subcompactions¶

rocksdb max_subcompactions, increase level workers 2-8

Type:: int

property max_total_wal_size¶

max size of WAL before forced flush

Type:: int

property max_ts_info_file_size¶

rocksdb max_ts_info_file_size, target file-sizes for ts-info files, typical 100M

Type:: int

property max_write_buffer_number¶

rocksdb max_write_buffer_number, typically 4..8

Type:: int

property min_blob_size¶

threshold for blob values, use 8*ppf to make full fragments become blobs

Type:: int

property min_write_buffer_number_to_merge¶

rocksdb min_write_buffer_number_to_merge, 1 or 2 maybe 4

Type:: int

property parallel_threads¶

argument to rocksdb option.IncreaseParallelism, 8-number of cores

Type:: int

property ppf¶

(default 1024) ts-points per fragment(e.g.key/value), determine the ts-fragment size, read/write operations to key-value storage are in fragment sizes.

Type:: int

property rate_limiter_auto_tuned¶

if True, RocksDB dynamically adjusts rate limiter budget to avoid starving high-priority flush I/O

Type:: bool

property rate_limiter_bytes_per_sec¶

I/O rate limit for compaction (bytes/s); 0=unlimited. Set to ~60-70% of Ceph sustained write throughput, e.g. 200*1024*1024

Type:: int

property rate_limiter_fairness¶

ratio of high-priority (flush) to low-priority (compaction) I/O tokens; default 10

Type:: int

property report_interval_sec¶

rocksdb report_interval_sec, 0 no reports

Type:: int

property test_mode¶

for internal use only, should always be set to 0(the default)

Type:: int

property ts_cache¶

low-level data-cache, could be useful in case of very large compressed databases

Type:: int

property ts_frag_block_size¶

rocksdb ts_frag.block_size, 64..256kb

Type:: int

property ts_frag_dynamic_level_bytes¶

pushes data to the deepest level, reducing write and space amplification for large, growing databases. Requires max_bytes_for_level_base large enough to absorb database growth without cascading compaction on every flush. False: static level targets used for legacy deployments or network storage where compaction cascades cause I/O saturation.

Type:: bool
Type:: (default True) RocksDB level_compaction_dynamic_level_bytes for the ts-fragment CF. True (modern RocksDB default)

property ts_frag_universal_style¶

use universal style compact for ts-fragments

Type:: bool

property ts_info_block_size¶

rocksdb ts_info.block_size, 4..16kb

Type:: int

property ts_info_max_bytes_for_level_base¶

10 GB base -> 100 GB next level -> 1 TB etc. Size this to comfortably hold your expected total dataset at the base level or one level above. ts_info name(avg.tsname length),value(116bytes) gives estimate for each ts-item, and then compute the estimated target size as n_ts x (avg.tsname length + 116). Set to 0 for legacy auto (10*max_file_size, uncapped).

Type:: int
Type:: (default 10 GB) target size in bytes for the base compaction level. With ts_info_dynamic_level_bytes=True, upper levels are derived proportionally (10x each)

property wal_compression¶

(default no compression) compress WAL to reduce write operations to disk.

Type:: CompressAlgorithm

property writable_file_max_buffer_size¶

write buffer for SST writes (bytes); 0=rocksdb default (1MB). Recommend 8MB for Ceph/network storage

Type:: int

property write_buffer_size¶

(default 10Mega), to balance write io-activity.

Type:: int

Class DtsClient¶

class shyft.time_series.DtsClient(*args, **kwargs)¶

Bases: pybind11_object

The client side part of the distributed time series system(DTSS).

The DtsClient communicate with the DtsServer using an efficient raw socket protocol using boost binary serialization. Provide a protocol version in the constructor to guarantee compatibility with dtss servers over several shyft releases. A typical operation would be that the DtsClient forwards TsVector that represents lists and structures of time-series expressions) to the DtsServer(s), that takes care of binding unbound symbolic time-series, evaluate and return the results back to the DtsClient. This class is closely related to the DtsServer and useful reference is also TsVector .

Best practice for client/server is to use cache following two simple rules(the default):

Always caching writes (because then consumers get it fresh and fast).

Always use caching reads(utilize and maintain the adaptive cache).

The only two known very rare and special scenarios where uncached writes can be useful are when loading large initial content of time-series db. Another special scenario, where caching reads should be turned off is when using the 3rd party dtss backend extension, where the 3rd party db is written/modified outside the control of dtss Also note that the caching works with the ts-terminals, not the result of the expressions. When reading time-series expressions, such as ts = ts1 - ts2, implementation of the cache is such that it contains the ts-terminals (here, ts1 and ts2), not the expression itself (ts). The .cache_stats, provides cache statistics for the server. The cache can be flushed, useful for some special cases of loading data outside cache.

Overloaded function.

__init__(self: shyft.time_series.DtsClient, host_port: str, auto_connect: bool = True, timeout_ms: typing.SupportsInt | typing.SupportsIndex = 1000, protocol_version: typing.SupportsInt | typing.SupportsIndex = 0) -> None

Constructs a dts-client with the specifed host_port parameter. A connection is immediately done to the server at specified port. If no such connection can be made, it raises a RuntimeError. If protocol_version is provided the client uses versioned protocols (see get_supported_protocols to get the supported protocols on the server)

host_port (str): a string of the format ‘host:portnumber’, e.g. ‘localhost:20000’

auto_connect (bool): default True, connection pr. call. if false, connection last lifetime of object unless explicitely closed/reopened

timeout_ms (int): default 1000ms, used for timeout connect/reconnect/close operations

protocol_version (int): default to internal, set protocol version on the client (see get_valid_versions to see what the client supports)

__init__(self: shyft.time_series.DtsClient, host_ports: collections.abc.Sequence[str], auto_connect: bool, timeout_ms: typing.SupportsInt | typing.SupportsIndex, protocol_version: typing.SupportsInt | typing.SupportsIndex = 0) -> None

Constructs a dts-client with the specifed host_ports parameters. A connection is immediately done to the server at specified port. If no such connection can be made, it raises a RuntimeError. If several servers are passed, the .evaluate and .percentile function will partition the ts-vector between the provided servers and scale out the computation If protocol_version is provided the client uses versioned protocols (see get_supported_protocols to get the supported protocols on the server)

host_ports (StringVector): a a list of string of the format ‘host:portnumber’, e.g. ‘localhost:20000’

auto_connect (bool): default True, connection pr. call. if false, connection last lifetime of object unless explicitly closed/reopened

timeout_ms (int): default 1000ms, used for timeout connect/reconnect/close operations

protocol_version (int): default to internal, set protocol version on the client (see get_valid_versions to see what the client supports)

__init__(self: shyft.time_series.DtsClient, host_port: str, timeout_ms: typing.SupportsInt | typing.SupportsIndex = 1000, *, operation_timeout_ms: typing.SupportsInt | typing.SupportsIndex = 0, protocol_version: typing.SupportsInt | typing.SupportsIndex = 0) -> None

host_port (str): a string of the format ‘host:portnumber’, e.g. ‘localhost:20000’

timeout_ms (int): default 1000ms, used for timeout connect/reconnect/close operations

operation_timeout_ms (int): default 0, if set, terminate operations that takes longer then specified limit

protocol_version (int): default to internal, set protocol version on the client (see get_valid_versions to see what the client supports)

__init__(self: shyft.time_series.DtsClient, host_ports: collections.abc.Sequence[str], timeout_ms: typing.SupportsInt | typing.SupportsIndex, *, operation_timeout_ms: typing.SupportsInt | typing.SupportsIndex = 0, protocol_version: typing.SupportsInt | typing.SupportsIndex = 0) -> None

host_ports (StringVector): a a list of string of the format ‘host:portnumber’, e.g. ‘localhost:20000’

timeout_ms (int): default 1000ms, used for timeout connect/reconnect/close operations

operation_timeout_ms (int): default 0, if set, terminate operations that takes longer then specified limit

protocol_version (int): default to internal, set protocol version on the client (see get_valid_versions to see what the client supports)

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.DtsClient, host_port: str, auto_connect: bool = True, timeout_ms: typing.SupportsInt | typing.SupportsIndex = 1000, protocol_version: typing.SupportsInt | typing.SupportsIndex = 0) -> None

host_port (str): a string of the format ‘host:portnumber’, e.g. ‘localhost:20000’

auto_connect (bool): default True, connection pr. call. if false, connection last lifetime of object unless explicitely closed/reopened

timeout_ms (int): default 1000ms, used for timeout connect/reconnect/close operations

protocol_version (int): default to internal, set protocol version on the client (see get_valid_versions to see what the client supports)

__init__(self: shyft.time_series.DtsClient, host_ports: collections.abc.Sequence[str], auto_connect: bool, timeout_ms: typing.SupportsInt | typing.SupportsIndex, protocol_version: typing.SupportsInt | typing.SupportsIndex = 0) -> None

host_ports (StringVector): a a list of string of the format ‘host:portnumber’, e.g. ‘localhost:20000’

auto_connect (bool): default True, connection pr. call. if false, connection last lifetime of object unless explicitly closed/reopened

timeout_ms (int): default 1000ms, used for timeout connect/reconnect/close operations

protocol_version (int): default to internal, set protocol version on the client (see get_valid_versions to see what the client supports)

__init__(self: shyft.time_series.DtsClient, host_port: str, timeout_ms: typing.SupportsInt | typing.SupportsIndex = 1000, *, operation_timeout_ms: typing.SupportsInt | typing.SupportsIndex = 0, protocol_version: typing.SupportsInt | typing.SupportsIndex = 0) -> None

host_port (str): a string of the format ‘host:portnumber’, e.g. ‘localhost:20000’

timeout_ms (int): default 1000ms, used for timeout connect/reconnect/close operations

operation_timeout_ms (int): default 0, if set, terminate operations that takes longer then specified limit

protocol_version (int): default to internal, set protocol version on the client (see get_valid_versions to see what the client supports)

__init__(self: shyft.time_series.DtsClient, host_ports: collections.abc.Sequence[str], timeout_ms: typing.SupportsInt | typing.SupportsIndex, *, operation_timeout_ms: typing.SupportsInt | typing.SupportsIndex = 0, protocol_version: typing.SupportsInt | typing.SupportsIndex = 0) -> None

host_ports (StringVector): a a list of string of the format ‘host:portnumber’, e.g. ‘localhost:20000’

timeout_ms (int): default 1000ms, used for timeout connect/reconnect/close operations

operation_timeout_ms (int): default 0, if set, terminate operations that takes longer then specified limit

protocol_version (int): default to internal, set protocol version on the client (see get_valid_versions to see what the client supports)

add_geo_ts_db(self: shyft.time_series.DtsClient, geo_cfg: shyft.time_series.GeoTimeSeriesConfiguration) → None¶: Adds a new, or replaces/updates existing, geo time-series database to the dtss-server with the given specifications

geo_cfg (GeoTimeSeriesConfiguration): the configuration to be added to the server specifying the dimensionality etc.

See also

.get_geo_db_ts_info()

property auto_connect¶

If connections are made as needed, and kept short, otherwise externally managed.

Type:: bool

cache_flush(self: shyft.time_series.DtsClient) → None¶: Flush the cache (including statistics) on the server. This can be useful in scenario when cache_on_write=False in the store operations.

property cache_stats¶

Get the cache_stats (including statistics) on the server.

Type:: CacheStats

close(self: shyft.time_series.DtsClient, timeout_ms: SupportsInt | SupportsIndex = 1000) → None¶: Close the connection. If auto_connect is enabled it will automatically reopen if needed.

property compress_expressions¶

If True, the expressions are compressed before sending to the server. For expressions of any size, like 100 elements, with expression depth 100 (e.g. nested sums), this can speed up the transmission by a factor or 3.

Type:: bool

property connections¶

Get remote server connections.

Type:: int

evaluate(self: shyft.time_series.DtsClient, ts_vector: shyft.time_series.TsVector, utcperiod: shyft.time_series.UtcPeriod, use_ts_cached_read: bool = True, update_ts_cache: bool = True, clip_result: shyft.time_series.UtcPeriod = UtcPeriod()) → shyft.time_series.TsVector¶

Evaluates the expressions in the ts_vector. If the expression includes unbound symbolic references to time-series, these time-series will be passed to the binding service callback on the serverside, passing on the specifed utcperiod.

NOTE: That the ts-backing-store, either cached or by read, will return data for:

at least the period needed to evaluate the utcperiod
In case of cached result, this will currently involve the entire matching cached time-series segment.

In particular, this means that the returned result could be larger than the specified utcperiod, unless you specify clip_result

Other available methods, such as the expression (x.average(ta)), including time-axis, can be used to exactly control the returned result size. Also note that the semantics of utcperiod is to ensure that enough data is read from the backend, so that it can evaluate the expressions. Use clip_result argument to clip the time-range of the resulting time-series to fit your need if needed - this will typically be in scenarios where you have not supplied time-axis operations (unbounded eval), and you also are using caching.

See also

DtsClient.percentiles() if you want to evaluate percentiles of an expression.

Parameters:

ts_vector (TsVector) – a list of time-series (expressions), including unresolved symbolic references
utcperiod (UtcPeriod) – the valid non-zero length period that the binding service should read from the backing ts-store/ts-service
use_ts_cached_read (bool) – use of server-side ts-cache
update_ts_cache (bool) – when reading time-series, also update the cache with the data
clip_result (UtcPeriod) – If supplied, clip the time-range of the resulting time-series to cover evaluation f(t) over this period only

Returns:

tsvector. an evaluated list of point time-series in the same order as the input list

Return type:

See also

DtsServer

extend_geo_db(self: shyft.time_series.DtsClient, geo_ts_db_name: str, grid_extension: shyft.time_series.GeoGridSpec, tsm: shyft.time_series.GeoMatrix, cache: bool = True) → None¶

Extend an existing geo-db with new geo-points-or-polygons. To ensure the new version of the geo-db is complete, the tsm matrix must contain all time-series for the new extension. This includes all forecasts (t0 dimension), ensembles, variables, for each of the new points or polygons added. To ensure successful execution, get the current geo-db configuration during a time where no updates occurs. Then use that geo-db information to create the correct tsm dimensions, tsm=TsMatrix(n_t0=len(g., n_v= , n_e= ,n_g=extra_points) Then fill in using tsm.set_ts(..). Notice that the grid_extension must be compatible with the existing geo-db grid, in terms of just points, or points/polygon. The new points or polygons should be unique to the existing geo-db grid.

Parameters:

geo_ts_db_name (str) – The name of the geo_ts_db, e.g. arome, ec, arome_cc ec_cc etc.
grid_extension (GeoGridSpec) – GeoGridSpec for the extension (epsg ignored)
tsm (TsMatrix) – A dense matrix with complete dimensionality for all variables, ensembles, the new geo-points,time-dimension 1..n.
cache (bool) – Also put values to the cache

See also

.get_geo_ts_db_info(),.geo_evaluate

find(self: shyft.time_series.DtsClient, search_expression: str) → list[shyft.time_series.TsInfo]¶

Find ts information that fully matches the regular search-expression. For the shyft file based backend, take care to specify path elements precisely, so that the directories visited is minimised. e.g:a/.*/my.ts Will prune out any top level directory not starting with a, but will match any subdirectories below that level. Refer to python test-suites for a wide range of examples using find. Notice that the regexp search algoritm uses ignore case. Please be aware that custom backend by python extension might have different rules.

Parameters:: search_expression (str) – regular search-expression, to be interpreted by the back-end tss server
Returns:: ts_info_vector. The search result, as vector of TsInfo objects
Return type:: TsInfoVector

See also

TsInfo,TsInfoVector

geo_evaluate(*args, **kwargs)¶

Overloaded function.

geo_evaluate(self: shyft.time_series.DtsClient, geo_ts_db_name: str, variables: collections.abc.Sequence[str], ensembles: collections.abc.Sequence[typing.SupportsInt | typing.SupportsIndex], time_axis: shyft.time_series.TimeAxis, ts_dt: shyft.time_series.time, geo_range: shyft.time_series.GeoQuery, concat: bool, cc_dt0: shyft.time_series.time, use_cache: bool = True, update_cache: bool = True) -> shyft.time_series.GeoTsMatrix

Evaluates a geo-temporal query on the server, and return the results

Parameters:

geo_ts_db_name (str) – The name of the geo_ts_db, e.g. arome, ec, arome_cc ec_cc etc.
variables (StringVector) – list of variables, like ‘temperature’,’precipitation’. If empty, return data for all available variables
ensembles (IntVector) – list of ensembles to read, if empty return all available
time_axis (TimeAxis) – return geo_ts where t0 matches time-points of this time-axis. If concat, the ta.total_period().end determines how long to extend latest forecast
ts_dt (time) – specifies the length of the time-slice to read from each time-series
geo_range (GeoQuery) – Specify polygon to include, empty means all
concat (bool) – If true, the geo_ts for each ensemble/point is joined together to form one singe time-series, concatenating a slice from each of the forecasts
cc_dt0 (time) – concat delta time to skip from the beginning of each geo_ts, so you can specify 3h, then select [+3h..slice-end) from each forecast
use_cache (bool) – use cache if available(speedup)
update_cache (bool) – if reading data from back-end, also stash it to the cache for faster evaluations

Returns:

A matrix where the elements are GeoTimeSeries, accessible using the indicies time,variable, ensemble, t0

Return type:

GeoMatrix

See also

.get_geo_ts_db_info()

geo_evaluate(self: shyft.time_series.DtsClient, eval_args: shyft.time_series.GeoEvalArgs, use_cache: bool = True, update_cache: bool = True) -> shyft.time_series.GeoTsMatrix

Evaluates a geo-temporal query on the server, and return the results

Parameters:

eval_args (GeoEvalArgs) – complete set of arguments for geo-evaluation, including geo-db, scope for variables, ensembles, time and geo-range
use_cache (bool) – use cache if available(speedup)
update_cache (bool) – if reading data from backend, also stash it to the cache for faster evaluations

Returns:

A matrix where the elements are GeoTimeSeries, accessible using indicies time,variable, ensemble, t0

Return type:

GeoMatrix

See also

.get_geo_ts_db_info()

geo_store(self: shyft.time_series.DtsClient, geo_ts_db_name: str, tsm: shyft.time_series.GeoMatrix, replace: bool, cache: bool = True) → None¶

Store a ts-matrix with needed dimensions and data to the specified geo-ts-db

Parameters:

geo_ts_db_name (str) – The name of the geo_ts_db, e.g. arome, ec, arome_cc ec_cc etc.
tsm (TsMatrix) – A dense matrix with dimensionality complete for variables, ensembles and geo-points,flexible time-dimension 1..n
replace (bool) – Replace existing geo time-series with the new ones, does not extend existing ts, replaces them!
cache (bool) – Also put values to the cache

See also

.get_geo_ts_db_info(),.geo_evaluate

get_client_supported_protocols(self: shyft.time_series.DtsClient) → Annotated[list[int], 'FixedSize(2)']¶

Returns:: versions. the versioned protocols the client supports (min, max)
Return type:: List

get_container_names(self: shyft.time_series.DtsClient) → list[str]¶: Return a list of the names of containers available on the server

get_geo_db_ts_info(self: shyft.time_series.DtsClient) → list[shyft.time_series.GeoTimeSeriesConfiguration]¶

Returns the configured geo-ts data-bases on the server, so queries can be specified and formulated

Returns:

A strongly typed list of GeoTimeseriesConfiguration

Return type:

GeoTimeseriesConfigurationVector

See also

.geo_evaluate()

get_q_bridge_status(self: shyft.time_series.DtsClient, name: str, clear_status: bool) → shyft.time_series.QueueBridgeStatus¶

Get status of specified configured queue bridge, if clear_status is True, also clear it.

Parameters:

() (clear_status) – the name of the queue bridge configuration
() – if true, also clear status at server

Returns:

status. The status as collected for the queue bridge

get_q_bridges(self: shyft.time_series.DtsClient) → list[shyft.time_series.QueueBridgeConfiguration]¶

returns configured active queue bridges.

Returns:: configurations. A list of configured queue bridges

get_server_supported_protocols(self: shyft.time_series.DtsClient) → Annotated[list[int], 'FixedSize(2)']¶

Returns:: versions. the versioned protocols the server supports (min, max)
Return type:: List

get_server_version(self: shyft.time_series.DtsClient) → str¶: Returns the server version major.minor.patch string, if multiple servers, the version of the first is returned

get_transfer_status(self: shyft.time_series.DtsClient, name: str, clear_status: bool) → shyft.time_series.TransferStatus¶

Get status of specified status, if clear_status is True, also clear it.

Parameters:

() (clear_status) – the name of the transfer
() – if true, also clear status at server

Returns:

transfer_status. The TransferStatus

get_transfers(self: shyft.time_series.DtsClient) → list[shyft.time_series.TransferConfiguration]¶

returns configured active transfers.

Returns:: transfer_configurations. A list of configured Transfers

get_ts_info(self: shyft.time_series.DtsClient, ts_url: str) → shyft.time_series.TsInfo¶

Get ts information for a time-series from the backend

Parameters:: ts_url (str) – Time-series url to lookup ts info for
Returns:: ts_info. A TsInfo object
Return type:: TsInfo

See also

TsInfo

get_ts_infos(self: shyft.time_series.DtsClient, container_name: str, ts_names: collections.abc.Sequence[str]) → list[shyft.time_series.TsInfo]¶

Get ts information for specified list of ts-names within a specified ts-container from the backend. The returned result have same size as the list of names, but order is not guaranteed. Items that does not exists will have their property set so that ‘.exists’ is False. note that only shyft:// containers are supported.

Parameters:

container_name (str) – plain name of the container
ts_names (StringVector) – A list of names to lookup to provide ts-info for.

Returns:

ts_infos. A list of TsInfo

Return type:

TsInfoVector

See also

TsInfo

merge_store_ts_points(self: shyft.time_series.DtsClient, tsv: shyft.time_series.TsVector, cache_on_write: bool = True) → None¶

Merge the ts-points supplied in the tsv into the existing time-series on the server side. The effect of each ts is similar to as if:

read ts.total_period() from ts point store

in memory appy the TimeSeries.merge_points(ts) on the read-ts

write the resulting merge-result back to the ts-store

This function is suitable for typical data-collection tasks where the points collected is from an external source, appears as batches, that should just be added to the existing point-set

Parameters:

tsv (TsVector) – ts-vector with time-series, url-reference and values to be stored at dtss server
cache_on_write (bool) – updates the cache with the result of the merge operation, if set to False, this is skipped, notice that this is only useful for very special use-cases.

Returns:

None.

See also

TsVector

property operation_timeout_ms¶

Operation timeout for remote server operations, in number milliseconds.

Type:: int

percentiles(self: shyft.time_series.DtsClient, ts_vector: shyft.time_series.TsVector, utcperiod: shyft.time_series.UtcPeriod, time_axis: shyft.time_series.TimeAxis, percentile_list: collections.abc.Sequence[SupportsInt | SupportsIndex], use_ts_cached_read: bool = True, update_ts_cache: bool = True) → shyft.time_series.TsVector¶

Evaluates the expressions in the ts_vector for the specified utcperiod. If the expression includes unbound symbolic references to time-series, these time-series will be passed to the binding service callback on the serverside.

Parameters:

ts_vector (TsVector) – a list of time-series (expressions), including unresolved symbolic references
utcperiod (UtcPeriod) – the valid non-zero length period that the binding service should read from the backing ts-store/ts-service
time_axis (TimeAxis) – the time_axis for the percentiles, e.g. a weekly time_axis
percentile_list (IntVector) – a list of percentiles, where -1 means true average, 25=25percentile etc
use_ts_cached_read (bool) – utilize server-side cached results
update_ts_cache (bool) – when reading time-series, also update the server-side cache

Returns:

tsvector. an evaluated list of percentile time-series in the same order as the percentile input list

Return type:

See also

.evaluate(), DtsServer

q_ack(self: shyft.time_series.DtsClient, name: str, msg_id: str, diagnostics: str) → None¶

After q_get, q_ack confirms that the message is ok/handled back to the process that called q_put.

Parameters:

() (diagnostics) – the name of the queue
() – the msg_id, required to be unique within current messages keept by the queue
() – the freetext diagnostics to but along with the message, we recommend json formatted

q_add(self: shyft.time_series.DtsClient, name: str) → None¶

Add a a named queue to the dtss server

Parameters:: () (name) – the name of the new queue, required to be unique

q_find(self: shyft.time_series.DtsClient, name: str, msg_id: str) → shyft.time_series.QueueMessage¶

Find a historical message with the given id

Parameters:

() (msg_id) – the name of the queue
() – the id of the message

Returns:

q_msg. The historical message, or None if the message dos not exist

q_get(self: shyft.time_series.DtsClient, name: str, max_wait: shyft.time_series.time) → shyft.time_series.QueueMessage¶

Get a message out from the named queue, waiting max_wait time for it if it’s not already there.

Parameters:

() (max_wait) – the name of the queue
() – max_time to wait for the message to arrive

Returns:

q_msg. A queue message consisting of .info describing the message, and the time-series vector .tsv

q_list(self: shyft.time_series.DtsClient) → list[str]¶: returns a list of defined queues on the dtss server

q_maintain(self: shyft.time_series.DtsClient, name: str, keep_ttl_items: bool, flush_all: bool = False) → None¶

Maintains, removes items that have passed through the queue, and are marked as done. To flush absolutely all items, pass flush_all=True.

Parameters:

() (flush_all) – the name of the queue
() – If true, the ttl set for the done messages are respected, and they are not removed until the create+ttl has expired
() – removes all items in the queue and kept by the queue, the queue is emptied

q_msg_info(self: shyft.time_series.DtsClient, name: str, msg_id: str) → shyft.time_series.QueueMessageInfo¶

From the specified queue, fetch info about specified msg_id. By inspecting the provided information, one can see when the message is created, fetched, and done with.

Parameters:

() (msg_id) – the name of the queue
() – the msg_id

Returns:

msg_info. the information/state of the identified message

q_msg_infos(self: shyft.time_series.DtsClient, name: str) → list[shyft.time_series.QueueMessageInfo]¶

Returns all the message information from a queue, including not yet pruned fetched/done messages

Parameters:: () (name) – the name of the queue
Returns:: msg_infos. the list of information kept in the named queue

q_put(self: shyft.time_series.DtsClient, name: str, msg_id: str, description: str, ttl: shyft.time_series.time, tsv: shyft.time_series.TsVector) → None¶

Put a message, as specified with the supplied parameters, into the specified named queue.

Parameters:

() (tsv) – the name of the queue
() – the msg_id, required to be unique within current messages keept by the queue
() – the freetext description to but along with the message, we recommend json formatted
() – time-to-live for the message after done, if specified, the q_maintain process can be asked to keep done messages that have ttl
() – time-series vector, with the wanted payload of time-series

q_remove(self: shyft.time_series.DtsClient, name: str) → None¶

Removes a named queue from dtss server, including all data in flight on the queue

Parameters:: () (name) – the name of the queue

q_size(self: shyft.time_series.DtsClient, name: str) → int¶

Returns number of queue messages waiting to be read by q_get.

Parameters:: () (name) – the name of the queue
Returns:: unread count. number of elements queued up

remove(self: shyft.time_series.DtsClient, ts_url: str) → None¶

Remove a time-series from the dtss backend The time-series referenced by ts_url is removed from the backend DtsServer. Important: Ensure to remove all references to the time-series prior to removing it, including references in expressions or stm model bindings on dstm server instances. Failure to adhere to the rule will cause references to either fail evaluation, or even update. dstm servers with references to removed time-series in the expression/subscription list, might need cache-flush to detect that the expressions with dangling references to removed time-series. A re-evaluation of an expression that have references to missing time-series will fail.

Note that the DtsServer setting may prohibit removing time-series.

Parameters:: ts_url (str) – shyft url referencing a time series

remove_container(self: shyft.time_series.DtsClient, container_url: str, delete_from_disk: bool = False) → None¶

Parameters:

container_url (str) – url of the container as pr. url definition above
delete_from_disk (bool) – Flag to indicate if the container should be deleted from disk

remove_geo_ts_db(self: shyft.time_series.DtsClient, geo_ts_db_name: str) → None¶: Remove the specified geo time-series database from dtss-server

geo_ts_db_name (str): the name of the geo-ts-database to be removed

See also

.get_geo_db_ts_info(),add_geo_ts_db()

remove_ts(self: shyft.time_series.DtsClient, urls: collections.abc.Sequence[str], transactional: bool = True) → None¶

Remove multiple time-series from the dtss backend Provides a remove function, that takes a list of urls.

Parameters:

urls (StringVector) – list of shyft urls referencing time series to be removed
transactional (bool) – if true (default), ensure all are removed in one batch otherwise they are removed in batches.

reopen(self: shyft.time_series.DtsClient, timeout_ms: SupportsInt | SupportsIndex = 1000) → None¶: (Re)open a connection after close or server restart.

set_container(self: shyft.time_series.DtsClient, name: str, relative_path: str, container_type: str = 'ts_db', cfg: shyft.time_series.DtssCfg = DtssCfg(ppf=1024, compress=DtssCfg.CompressOptions(algo=no_compression, bottom_algo=no_compression, index_compression=true), max_file_size=104857600, write_buffer_size=67108864, log_level=200, test_mode=0, ix_cache=0, ts_cache=0)) → None¶

create an internal shyft store container to the dtss-server with a root relative path. All ts-urls with shyft://<container>/ will resolve to this internal time-series storage for find/read/store operations will not replace existing containers that have the same name

Parameters:

name (str) – Name of the container as pr. url definition above
relative_path (str) – A valid directory for the container relative to the root path of the server.
container_type (str) – one of (‘ts_rdb’, ‘ts_db’), container type to add.

start_q_bridge(self: shyft.time_series.DtsClient, cfg: shyft.time_series.QueueBridgeConfiguration) → None¶

Starts a transfer on the server using the provided configuration.

Parameters:: () (cfg) – the configuration for the transfer

start_transfer(self: shyft.time_series.DtsClient, cfg: shyft.time_series.TransferConfiguration) → None¶

Starts a transfer on the server using the provided TransferConfiguration.

Parameters:: () (cfg) – the configuration for the transfer

stop_q_bridge(self: shyft.time_series.DtsClient, name: str, max_wait: shyft.time_series.time) → None¶

Stop,cancel, removes a named queue bridge.

Parameters:

() (max_wait) – the name of the transfer to remove
() – to let existing transfers gracefully finish

stop_transfer(self: shyft.time_series.DtsClient, name: str, max_wait: shyft.time_series.time) → None¶

Stop,cancel, removes a named transfer.

Parameters:

() (max_wait) – the name of the transfer to remove
() – to let existing transfers gracefully finish

store(self: shyft.time_series.DtsClient, tsv: shyft.time_series.TsVector, store_policy: shyft.time_series.StorePolicy) → list[shyft.time_series.TsDiagnosticsItem] | None¶

Store the time-series in the ts-vector in the dtss backend. Stores the time-series fragments data passed to the backend. If store_policy.strict == True: It is semantically stored as if

first erasing the existing stored points in the range of ts.time_axis().total_period()

then inserting the points of the ts.

>>>   a=sa.TimeSeries(ts_url,ts_points)
>>>   tsv.append(a)

Parameters:

tsv (TsVector) – ts-vector with time-series, url-reference and values to be stored at dtss server
store_policy (StorePolicy) – Determines how to project the passed time-series fragments to the backend stored time-series

Returns:

diagnostics. For any failed items, normally empty

Return type:

TsDiagnosticsItemList

See also

TsVector

store_ts(self: shyft.time_series.DtsClient, tsv: shyft.time_series.TsVector, overwrite_on_write: bool = False, cache_on_write: bool = True) → None¶

Store the time-series in the ts-vector in the dtss backend. Stores the time-series fragments data passed to the backend. It is semantically stored as if

first erasing the existing stored points in the range of ts.time_axis().total_period()

then inserting the points of the ts.

When creating time-series 1st time, pay attention to the time-axis, and point-interpretation as this remains the properties of the newly created time-series. Storing 15min data to a time-series defined initially as hourly series will raise exception. On the otherhand, the variable interval time-series are generic and will accept any time-resolution to be stored

The overwrite_on_write = True, is used to replace the entire definition of any previously stored time-series. This is semantically as if erasing the previously stored time-series and replacing its entire content and definition, starting fresh with the newly passed time-series. The time-series should be created like this, with url and a concrete point-ts:

>>>   a=sa.TimeSeries(ts_url,ts_points)
>>>   tsv.append(a)

Parameters:

tsv (TsVector) – ts-vector with time-series, url-reference and values to be stored at dtss server
overwrite_on_write (bool) – When True the backend replaces the entire content and definition of any existing time-series with the passed time-series
cache_on_write (bool) – defaults True, if set to False, the cache is not updated, and should only be considered used in very special use-cases.

Returns:

None.

See also

TsVector

swap_container(self: shyft.time_series.DtsClient, container_name_a: str, container_name_b: str = False) → None¶

Parameters:

container_name_a (str) – Name of container a
container_name_b (str) – Name of container b

total_clients = 0¶

update_geo_ts_db_info(self: shyft.time_series.DtsClient, geo_ts_db_name: str, description: str, json: str, origin_proj4: str) → None¶

Update info fields of the geo ts db configuration to the supplied parameters

Parameters:

geo_ts_db_name (str) – The name of the geo_ts_db, e.g. arome, ec, arome_cc ec_cc etc.
description (str) – The description field of the database
json (str) – The user specified json like string
origin_proj4 (str) – The origin proj4 field update

See also

.get_geo_ts_db_info()

Class DtsWebClient¶

class shyft.time_series.DtsWebClient(self: shyft.time_series.DtsWebClient, config: shyft.time_series.WebClientConfig)¶

Bases: pybind11_object

The web client side part of the distributed time series system(DTSS).

The DtsWebClient communicate with the DtsServer using an efficient web socket protocol using boost spirit qi and karma parser/emitter. The web-protocol is json based, and stable with optional-as-used fields for upgrades. This class is closely related to the DtsServer and useful reference is also TsVector .

Best practice for client/server is to use cache following two simple rules(the default):

Always caching writes (because then consumers get it fresh and fast).

Always use caching reads(utilize and maintain the adaptive cache).

Constructs a dts-web-client with the configuration. A connection deferred until first use.

config (WebClientConfig): Configuration to use for the connection

__init__(self: shyft.time_series.DtsWebClient, config: shyft.time_series.WebClientConfig) → None¶: Constructs a dts-web-client with the configuration. A connection deferred until first use.

config (WebClientConfig): Configuration to use for the connection

find(self: shyft.time_series.DtsWebClient, search_expression: str) → list[shyft.time_series.TsInfo]¶

Find ts information that fully matches the regular search-expression. For the shyft file based backend, take care to specify path elements precisely, so that the directories visited is minimised. e.g:a/.*/my.ts Will prune out any top level directory not starting with a, but will match any subdirectories below that level. Refer to python test-suites for a wide range of examples using find. Notice that the regexp search algoritm uses ignore case. Please be aware that custom backend by python extension might have different rules.

Parameters:: search_expression (str) – regular search-expression, to be interpreted by the back-end tss server
Returns:: ts_info_vector. The search result, as vector of TsInfo objects
Return type:: TsInfoVector

See also

TsInfo,TsInfoVector

get_ts_infos(self: shyft.time_series.DtsWebClient, container_name: str, ts_names: collections.abc.Sequence[str]) → list[shyft.time_series.TsInfo]¶

Parameters:

container_name (str) – plain name of the container
ts_names (StringVector) – A list of names to lookup to provide ts-info for.

Returns:

ts_infos. A list of TsInfo

Return type:

TsInfoVector

See also

TsInfo

q_ack(self: shyft.time_series.DtsWebClient, name: str, msg_id: str, diagnostics: str) → None¶

After q_get, q_ack confirms that the message is ok/handled back to the process that called q_put.

Parameters:

() (diagnostics) – the name of the queue
() – the msg_id, required to be unique within current messages keept by the queue
() – the freetext diagnostics to but along with the message, we recommend json formatted

q_get(self: shyft.time_series.DtsWebClient, name: str, max_wait: shyft.time_series.time) → shyft.time_series.QueueMessage¶

Get a message out from the named queue, waiting max_wait time for it if it’s not already there.

Parameters:

() (max_wait) – the name of the queue
() – max_time to wait for the message to arrive

Returns:

q_msg. A queue message consisting of .info describing the message, and the time-series vector .tsv

q_list(self: shyft.time_series.DtsWebClient) → list[str]¶: returns a list of defined queues on the dtss server

q_maintain(self: shyft.time_series.DtsWebClient, name: str, keep_ttl_items: bool, flush_all: bool = False) → None¶

Maintains, removes items that have passed through the queue, and are marked as done. To flush absolutely all items, pass flush_all=True.

Parameters:

() (flush_all) – the name of the queue
() – If true, the ttl set for the done messages are respected, and they are not removed until the create+ttl has expired
() – removes all items in the queue and kept by the queue, the queue is emptied

q_msg_info(self: shyft.time_series.DtsWebClient, name: str, msg_id: str) → shyft.time_series.QueueMessageInfo¶

From the specified queue, fetch info about specified msg_id. By inspecting the provided information, one can see when the message is created, fetched, and done with.

Parameters:

() (msg_id) – the name of the queue
() – the msg_id

Returns:

msg_info. the information/state of the identified message

q_msg_infos(self: shyft.time_series.DtsWebClient, name: str) → list[shyft.time_series.QueueMessageInfo]¶

Returns all the message information from a queue, including not yet pruned fetched/done messages

Parameters:: () (name) – the name of the queue
Returns:: msg_infos. the list of information kept in the named queue

q_put(self: shyft.time_series.DtsWebClient, name: str, msg_id: str, description: str, ttl: shyft.time_series.time, tsv: shyft.time_series.TsVector) → None¶

Put a message, as specified with the supplied parameters, into the specified named queue.

Parameters:

() (tsv) – the name of the queue
() – the msg_id, required to be unique within current messages keept by the queue
() – the freetext description to but along with the message, we recommend json formatted
() – time-to-live for the message after done, if specified, the q_maintain process can be asked to keep done messages that have ttl
() – time-series vector, with the wanted payload of time-series

q_size(self: shyft.time_series.DtsWebClient, name: str) → int¶

Returns number of queue messages waiting to be read by q_get.

Parameters:: () (name) – the name of the queue
Returns:: unread count. number of elements queued up

read_ts(self: shyft.time_series.DtsWebClient, ts_urls: collections.abc.Sequence[str], read_period: shyft.time_series.UtcPeriod, *, use_ts_cached_read: bool = True, clip_result: shyft.time_series.UtcPeriod = UtcPeriod()) → shyft.time_series.TsVector¶

Read list of ts-urls for specified period

Parameters:

ts_urls (List[str]) – a list of time-series urls
read_period (UtcPeriod) – The minimum period to read and cover, returned result might be more, see ‘clip_period’ to limit.
use_ts_cached_read (bool) – use of server-side ts-cache
clip_result (UtcPeriod) – Clip response exactly to this period

Returns:

tsvector. an evaluated list of point time-series in the same order as the input list

Return type:

See also

DtsServer

store(self: shyft.time_series.DtsWebClient, tsv: shyft.time_series.TsVector, store_policy: shyft.time_series.StorePolicy) → str | None¶

Store the time-series in the ts-vector in the dtss backend. Stores the time-series fragments data passed to the backend. If store_policy.strict == True: It is semantically stored as if

first erasing the existing stored points in the range of ts.time_axis().total_period()

then inserting the points of the ts.

>>>   a=sa.TimeSeries(ts_url,ts_points)
>>>   tsv.append(a)

Parameters:

tsv (TsVector) – ts-vector with time-series, url-reference and values to be stored at dtss server
store_policy (StorePolicy) – Determines how to project the passed time-series fragments to the backend stored time-series

Returns:

diagnostics. For any failed items, normally empty

Return type:

TsDiagnosticsItemList

See also

TsVector

Class WebClientConfig¶

class shyft.time_series.WebClientConfig(self: shyft.time_series.WebClientConfig, host: str, port: SupportsInt | SupportsIndex, *, target: str = '/', use_tls: bool = True, verify_cert: bool = True, authorization: str = '', max_message_size: SupportsInt | SupportsIndex = 100000000, establish_timeout: shyft.time_series.time = time(30))¶

Bases: pybind11_object

The client configuration for the web client.

Constructs a dts-web-client with the configuration. A connection deferred until first use.

config (WebClientConfig): Configuration to use for the connection

__init__(self: shyft.time_series.WebClientConfig, host: str, port: SupportsInt | SupportsIndex, *, target: str = '/', use_tls: bool = True, verify_cert: bool = True, authorization: str = '', max_message_size: SupportsInt | SupportsIndex = 100000000, establish_timeout: shyft.time_series.time = time(30)) → None¶: Constructs a dts-web-client with the configuration. A connection deferred until first use.

config (WebClientConfig): Configuration to use for the connection

property authorization¶

like ‘Basic base64-usr-pwd’

Type:: str

property establish_timeout¶

establish dns, tls, and ws handshake timeouts

Type:: time

property host¶

resolveable host or ip

Type:: str

property max_message_size¶

allow ws messages up to this size

Type:: int

property port¶

port for host service

Type:: int

property target¶

url part for target, like ‘/’ or ‘/api/ws’

Type:: str

property use_tls¶

use tls (ref. shyft web-api for configuration of certs)

Type:: bool

property verify_cert¶

verify certificate chain also

Type:: bool

Class StorePolicy¶

class shyft.time_series.StorePolicy(self: shyft.time_series.StorePolicy, recreate: bool, strict: bool, cache: bool, best_effort: bool = False, ppf: SupportsInt | SupportsIndex = 0, compress: bool = False, accuracy: SupportsFloat | SupportsIndex = 0.0)¶

Bases: pybind11_object

Determine how DTSS stores time-series, used in context of the DtsClient.

construct object with specified parameters

__init__(self: shyft.time_series.StorePolicy, recreate: bool, strict: bool, cache: bool, best_effort: bool = False, ppf: SupportsInt | SupportsIndex = 0, compress: bool = False, accuracy: SupportsFloat | SupportsIndex = 0.0) → None¶: construct object with specified parameters

property accuracy¶

default 0.0, if compress is True,- for break-point time-series, points with values equal within supplied accuracy.

Type:: float

property best_effort¶

try to store all, return diagnostics for logical errors instead of raise exception

Type:: bool

property cache¶

update cache with new values

Type:: bool

property compress¶

default False, When writing break-point series, minimize points by compressing points with values equal within supplied accuracy.

Type:: bool

property ppf¶

if (re)creating a time-series, then time points-per-fragment override the configured ppf for the backend storage. Default 0 means using backend configured ppf

Type:: int

property recreate¶

recreate time-series if it existed, with entirely new definition

Type:: bool

property strict¶

use strict requirement for aligment of timepoints. If True(default), then require perfect matching time-axis on the incoming ts fragment, and transfer those points to the target. Notice that if the target is a break-point/flexible interval time-series, then all time-axis passed are a perfect match. If False, use a functional mapping approach, resampling or averaging the passed time-series fragment to align with the target time-series. If target is linear time-series, the new ts fragment is evaluated f(t) for the covered time-points in the target time-series. If target is stair-case time-series, then the true average of the new ts fragment, evaluated over the covering/touched period of the target time-series.

Type:: bool

DTSS Exchange¶

The Exchange section focuses on tools for transferring and managing time series across dtss services through advanced time-series replication mechanism and also bridging queues.

All exchange functions are network fault tolerant, self repairing and best provide best effort features to ensure data flows to the designated site.

The exchange services are also designed and tested for high volumes, an large data-sets.

The exchange functions of the DTSS also provides extensive set of monitoring, statistics counting to detect errors in a way that can be observed.

The Exchange Transfer and QueueBridge is created so that communication initiative goes from the local site to a remote site, to ensure the mechanism fits well into a highly secured site.

Typical the communication between the sites should be established using ssh-tunnel, or vpn connections.

DTSS Transfer¶

The current features includes pushing a specified set of time-series (regular expression) to a remote site.

The time-range, from ..to, relative time-now, as well as incremental real-time push or pull changes to or from remote site is supported.

Time-series names can also be translated so that they appear different on the remote site.

class shyft.time_series.TransferConfiguration(self: shyft.time_series.TransferConfiguration)¶

Bases: pybind11_object

The transfer configuration describes what time-series to transfer, when, how and where to transfer.

class PeriodSpec¶

Bases: pybind11_object

__init__(*args, **kwargs)¶

property now_relative¶

if >0, then time.now() trimmed to now_relative

Type:: time

property spec¶

period range to transfer,either now_releative, or absolute

Type:: UtcPeriod

class RemoteSpec¶

Bases: pybind11_object

__init__(*args, **kwargs)¶

property host¶

host ip or name

Type:: str

property port¶

host port number

Type:: str

class TimeSeriesSpec¶

Bases: pybind11_object

The TimeSeriesSpec controls what to transfer, using regular expressions.

The search_pattern is used to search for time-series or related records stored in the backend using a regex-based query. The search efficiency depends on the structure of the pattern provided, particularly when interacting with the underlying RocksDB or File backend.

The form of the search_pattern is typically shyft://container/someprefix-regexp

The regexp starts on the first character that are deemed to be a startof regular expression character.

‘\’, ‘.’, ‘(’, ‘[’, ‘^’, ‘$’, ‘*’, ‘?’, ‘+’, ‘{’, ‘|’

Performance considerations:

Effect of Prefix (`someprefix`):
- Using a constant string prefix as part of the pattern significantly reduces the server-side computational load.
- RocksDB can prune out irrelevant entries efficiently, avoiding full enumeration of all time-series meta-data records and testing only those that match the prefix.
- Examples of efficient patterns: - shyft://container/sensor-123.* - shyft://container/temperature-data.*
Performance Bottlenecks Without Prefix:
- If the pattern lacks a constant prefix (e.g., shyft://container/.* or shyft://container/[a-z].*), the backend must perform full enumeration of all time-series meta-data records, which involves:
- Enumerating the entire database ts-info records.
- Performing regular expression matching for every entry in the database on the server side.
Wildcards and Immediate Regex Start:
- Patterns starting with wildcards or regex constructs (e.g., .*, [a-z].*) are costly due to the lack of pruning hints for filtering irrelevant entries.
- Examples of inefficient patterns: - shyft://container/.*: Forces full enumeration of all stored time-series meta-data records. - shyft://container/[a-z].*: Scans the full database and matches regex for entries starting with any lower-case letter.

Best practices for optimal performance:

Always use specific constant prefixes in the pattern to narrow the search scope. For example, prefer shyft://container/sensor/WindSpeed-123.* instead of shyft://container/.*.
Avoid using complex or overly broad regex patterns unless absolutely necessary.
Ensure to set poll-interval timers large enough to make searching activities minimal.

Examples:

Efficient pattern:

shyft://container/sensor-123.* - RocksDB can directly filter irrelevant entries based on the “sensor-123” prefix, avoiding unnecessary enumeration.
Inefficient pattern:

shyft://container/.* - Forces full enumeration of all time-series meta-data records and applies regex matching on each record.
Example with wildcards:

shyft://container/[a-z].* - Requires scanning the database and matching records starting with [a-z], which is more costly.

Always use constant string prefixes to achieve efficient, scalable search operations, especially for large time-series datasets.

__init__(*args, **kwargs)¶

property replace_pattern¶

reg-ex replace pattern

Type:: str

property search_pattern¶

reg-ex search pattern, ref class doc for performance hints

Type:: str

class WhenSpec¶

Bases: pybind11_object

__init__(*args, **kwargs)¶

property change_linger_time¶

max time to wait before transfer detected changes

Type:: time

property changed¶

if true, use subscription to transfer when changed

Type:: bool

property empty_match_poll_interval¶

max time to wait before retry an empty pattern match during initial phase

Type:: time

property poll_interval¶

how often to check for changes

Type:: time

property re_match_poll_interval¶

if >0 then max time to wait before re-check pattern match during transfer phase

Type:: time

property ta¶

if non-empty, transfer according to time-axis

Type:: TimeAxis

class XferSpec¶

Bases: pybind11_object

__init__(*args, **kwargs)¶

property partition_size¶

maximum number of ts to transfer in one batch

Type:: int

property retries¶

retries on connect/re-send for remote

Type:: int

property sleep_before_retry¶

when stuff fail, how long to sleep before retry connect/send message to remote

Type:: time

__init__(self: shyft.time_series.TransferConfiguration) → None¶

property how¶

retries and delays for remote connection

Type:: XferSpec

property json¶

a user specified json info that could be useful for extensions

Type:: str

property name¶

name of the transfer, used for reference to active transfers

Type:: str

property protocol_version¶

protocol version to run the transfer over

Type:: int

property read_remote¶

transfer-direction,if true, pull from remote,otherwise push

Type:: bool

property read_updates_cache¶

if true, also update cache while reading

Type:: bool

property store_policy¶

used for writing time-series

Type:: StorePolicy

property what¶

timeseries to transfer,reg-expr

Type:: TimeSeriesSpec

property when¶

oneshot, subscription,continuous, scheduled

Type:: WhenSpec

property where¶

specification for the remote, host, port

Type:: RemoteSpec

property xfer_period¶

specification of the time-period to transfer

Type:: PeriodSpec

class shyft.time_series.TransferStatus¶

Bases: pybind11_object

Transfer status gives insight into the current state of a transfer.

class ReadError¶

Bases: pybind11_object

__init__(*args, **kwargs)¶

property code¶

read failure code

Type:: LookupError

property ts_url¶

ts-url

Type:: str

class WriteError¶

Bases: pybind11_object

__init__(*args, **kwargs)¶

property code¶

write failure code

Type:: TsDiagnostics

property ts_url¶

ts-url

Type:: str

__init__(*args, **kwargs)¶

property last_activity¶

last activity registered time

Type:: time

property n_read_errors¶

number of read errors

Type:: int

property n_remote_errors¶

number of remote errors

Type:: int

property n_ts_found¶

number of ts found for transfer, if it is zero an readers alive the reader part will continue search for time-series until they appear

Type:: int

property n_write_errors¶

number of write errors

Type:: int

property read_errors¶

read errors

Type:: ReadErrorList

property read_speed¶

points/seconds

Type:: float

property reader_alive¶

true if reader is still working/monitoring

Type:: bool

property remote_errors¶

remote connection errors

Type:: StringVector

property total_transferred¶

number of points transferred

Type:: int

property write_errors¶

write errors

Type:: WriteErrorList

property write_speed¶

points/seconds

Type:: float

property writer_alive¶

true if writer is still working/monitoring

Type:: bool

DTSS QueueBridge¶

The QueueBridge feature provides a way to link two instances of DTSS with queues. The mechanism ensure unique mapping of ‘msg_id’, so that other producers, or queue bridges can be linked to the same queue.

Extended monitoring and statistics is provided, with optional logging of events.

The pattern found useful, and implemented is

        graph TB
    subgraph Remote_Site ["Remote Site"]
        Producer["Producer"]
        R_Queue["Queue: R"]
        Producer --> R_Queue
    end

    subgraph Local_Site ["Local Site"]
        QueueBridge["QueueBridge (Fetcher & Ack Worker)"]
        Q_Queue["Queue: Q"]
        Consumer["Consumer"]
        R_Queue -->|Fetch Messages| QueueBridge --> Q_Queue
        Q_Queue -->|Fetch message| Consumer -->|Ack message| Q_Queue
        Q_Queue -->|Send Acks| QueueBridge -->|Route Acks| R_Queue
    end

    classDef queue fill:#fff3b1,stroke:#333,stroke-width:2px;
    classDef worker fill:#87ceeb,stroke:#333,stroke-width:2px;
    classDef producer_consumer fill:#e0e0e0,stroke:#333,stroke-width:2px;

    class R_Queue,Q_Queue queue;
    class QueueBridge worker;
    class Producer,Consumer producer_consumer;

Methods for for start,status,stop and remove QueueBridges

start_q_bridge()
get_q_bridge_status()
get_q_bridges()
stop_q_bridge()

In the current version, configured and started queue bridges are not persisted across of DTSS restarts. This means that an external monitor/control using a python script and a DTSS client is needed to control the bridge.

The creation of the bridge, can be done during the bootstrap of the DTSS and since the implementation is robust for remote network connection problems, and even remote DTSS configuration problems, it will start to work when a connection to a remote configured DTSS is available.

class shyft.time_series.QueueBridgeConfiguration(self: shyft.time_series.QueueBridgeConfiguration)¶

Bases: pybind11_object

The bridge configuration describes what queues to bridge between remote and local, and how tight the timing should be.

class QueueSpec¶

Bases: pybind11_object

__init__(*args, **kwargs)¶

property ack_poll_interval¶

how often to check for ack messages in the local queue

Type:: time

property queue_name¶

the name of the local queue

Type:: str

class RemoteSpec¶

Bases: pybind11_object

__init__(*args, **kwargs)¶

property host¶

host ip or name

Type:: str

property port¶

host port number

Type:: str

property queue_name¶

name of the remote queue

Type:: str

class XferSpec¶

Bases: pybind11_object

This class specifies how to deal with remote communication errors, and how long to wait in io-operation on the remote queue. The latter influences how fast the dtss server that hosts the exchange/q_bridge will react to a shut-down/restart.

The io_operation_timeout parameter controls for how long an io operation is allowed to take before it is timed out, the connection closed and the event reported, then reconnection cycle started.

When waiting for queue messages, the queue_pick_up_max_wait timeouts are added to compensate for known server time.

The minimum value would be the time it takes to:

send the message

server to compute the answer

send the answer back

Due to the nature of network, a large slack should be added. Typical values could be 1..10 seconds.

__init__(*args, **kwargs)¶

property io_operation_timeout¶

timeout remote operations that are stuck in network/no reply

Type:: time

property queue_pick_up_max_wait¶

time to try read message or ack from queues

Type:: time

property retries¶

retries on connect for remote

Type:: int

property sleep_before_retry¶

when stuff fail, how long to sleep before retry.

Type:: time

__init__(self: shyft.time_series.QueueBridgeConfiguration) → None¶

property how¶

time-out and retry parameters

Type:: XferSpec

property json¶

a user specified json info that could be useful for extensions

Type:: str

property name¶

name of the queue bridge, used for reference to active bridges

Type:: str

property protocol_version¶

protocol version to run the transfer over

Type:: int

property what¶

queue to bridge,local and remote

Type:: QueueSpec

property where¶

specification for the remote, host, port

Type:: RemoteSpec

class shyft.time_series.QueueBridgeStatus¶

Bases: pybind11_object

Status of the queue-bridge operations.

__init__(*args, **kwargs)¶

property ack_errors¶

number of ack related errors

Type:: int

property completed¶

number of completed message/ack exchanges over the bridge

Type:: int

property fetch_errors¶

number of fetch errors to remote queue

Type:: int

property fetched¶

number of messages fetched from remote and written to the local queue

Type:: int

property last_ack_error¶

last ack related error

Type:: QueueBridgeErrorEntry

property last_activity¶

when any msg fetch or ack workers had activity

Type:: time

property last_fetch_error¶

last fetch error

Type:: QueueBridgeErrorEntry

property name¶

the name of the queue bridge

Type:: str

DTSS Queue for time-series¶

The queue mechanism provides a producer-consumer pattern that also involves consumer get-message followed by a consumer-ack message. The latter provides semantic information from the consumer, such as that the message was received, interpreted and applied, and also to provide diagnostics, incase it failed.

        graph TB
    Producer["Producer Node"]:::producer_consumer
    Consumer["Consumer Node"]:::producer_consumer
    DTSS_Queue["DTSS node with Queue"]:::queue

    Producer -->|Put Message| DTSS_Queue
    DTSS_Queue -->|Ack Info| Producer
    DTSS_Queue -->|Fetch Message| Consumer
    Consumer -->|Ack Message| DTSS_Queue

    classDef queue fill:#fff3b1,stroke:#333,stroke-width:2px;
    classDef producer_consumer fill:#e0e0e0,stroke:#333,stroke-width:2px;

The DtsClient class provides a methods to work with queues.

Maintenance and configuration

q_list() - provides a list of queues configured
q_add() - Adds a named queue to the list of queues
q_remove() - Remove a named queue from the list of queues
q_maintain() - Prune out ‘done’ messages in a queue

Message activities

q_put(). - a producer puts next message to the queue
q_get(). - a consumer reads next message from the queue
q_ack(). - a consumer completes its read cycle with ack and diagnostics/feedback

Query and information

q_infos() - Get all message infos contained in a queue
q_info() - Get info for a specific message
q_find() - Check/find a specific message

Notice that by default, when a message is fetched from the queue, the associated TsVector payload is removed. This is to ensure memory is kept at a minimum.

There is no automatic pruning of messages, so a separate client/controller needs to be active and call q_maintain() at regular intervals to prune out messages done where ttl has expired.

Also note that the current version does not persist the configured queues, nor the messages across restart of the DTSS.

Usual techniques is to add the queue as part of the python bootstrap code, so that the queue is part of the configuration/startup of the DTSS.

Persistence and other features are possible and easy to implement if needed.

class shyft.time_series.QueueMessage(self: shyft.time_series.QueueMessage, msg_id: str, desc: str, ttl: shyft.time_series.time, tsv: shyft.time_series.TsVector)¶

Bases: pybind11_object

A QueueMessage as returned from the DtsClient.q_get(..) consist of the .info part and the payload time-series vector .tsv

constructs a QueueMessage

Parameters:

msg_id (str) – unique identifier for the message
desc (str) – custom description
ttl (time) – time the message should live on the queue
tsv (TsVector) – timeseries payload

__init__(self: shyft.time_series.QueueMessage, msg_id: str, desc: str, ttl: shyft.time_series.time, tsv: shyft.time_series.TsVector) → None¶

constructs a QueueMessage

Parameters:

msg_id (str) – unique identifier for the message
desc (str) – custom description
ttl (time) – time the message should live on the queue
tsv (TsVector) – timeseries payload

property info¶

The information about the message

Type:: QueueMessageInfo

property tsv¶

The time-series vector payload part of the message

Type:: TsVector

class shyft.time_series.QueueMessageInfo¶

Bases: pybind11_object

Information about the queue item, such as the state of the item, in-queue,fetched, done. This element is never to be created by the python user, but is a return type from the dtss queue message info related calls.

__init__(*args, **kwargs)¶

property created¶

Time when the message was put into the queue

Type:: time

property description¶

A user specified description, we recommend json format

Type:: str

property diagnostics¶

Time when the message acknowledged done from the receiver(end-to-end ack)

Type:: str

property done¶

Time when the message acknowledged done from the receiver(end-to-end ack)

Type:: time

property fetched¶

Time when the message was fetched from the queue

Type:: time

property msg_id¶

The unique id for this message in the live-queue

Type:: str

property ttl¶

Time to live set for this message, used to prune out old messages

Type:: time

Class TsInfo¶

class shyft.time_series.TsInfo(*args, **kwargs)¶

Bases: pybind11_object

Gives some information from the backend ts data-store about the stored time-series, that could be useful in some contexts

Overloaded function.

__init__(self: shyft.time_series.TsInfo) -> None
__init__(self: shyft.time_series.TsInfo, name: str, point_fx: shyft.time_series.point_interpretation_policy, delta_t: shyft.time_series.time, olson_tz_id: str, data_period: shyft.time_series.UtcPeriod, created: shyft.time_series.time, modified: shyft.time_series.time) -> None

construct a TsInfo with all values specified

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.TsInfo) -> None
__init__(self: shyft.time_series.TsInfo, name: str, point_fx: shyft.time_series.point_interpretation_policy, delta_t: shyft.time_series.time, olson_tz_id: str, data_period: shyft.time_series.UtcPeriod, created: shyft.time_series.time, modified: shyft.time_series.time) -> None

construct a TsInfo with all values specified

property created¶

when time-series was created, seconds 1970s utc

Type:: time

property data_period¶

the period for data-stored, if applicable

Type:: UtcPeriod

property delta_t¶

time-axis steps, in seconds, 0 if irregular time-steps

Type:: time

property exists¶

true if time-series exists in backend

Type:: bool

property modified¶

when time-series was last modified, seconds 1970 utc

Type:: time

property name¶

the unique name

Type:: str

property olson_tz_id¶

empty or time-axis calendar for calendar,t0,delta_t type time-axis

Type:: str

property point_fx¶

how to interpret the points, instant value, or average over period

Type:: point_interpretation_policy

Class CacheStats¶

class shyft.time_series.CacheStats(self: shyft.time_series.CacheStats)¶

Bases: pybind11_object

Cache statistics for the DtsServer.

__init__(self: shyft.time_series.CacheStats) → None¶

property coverage_misses¶

number of misses where we did find the time-series id, but the period coverage was insufficient

Type:: int

property fragment_count¶

number of time-series fragments in the cache, (greater or equal to id_count)

Type:: int

property hits¶

number of hits by time-series id

Type:: int

property id_count¶

number of unique time-series identities in cache

Type:: int

property misses¶

number of misses by time-series id

Type:: int

property point_count¶

total number of time-series points in the cache

Type:: int

Geo-location Time series¶

The elements in this section integrate the generic time series concepts above with a geo-spatial co-ordinate system. This functionality extends to co-ordinate based queries in the time series storage.

Class GeoPoint¶

class shyft.time_series.GeoPoint(*args, **kwargs)¶

Bases: pybind11_object

GeoPoint commonly used in the shyft::core for representing a 3D point in the terrain model. The primary usage is in geo-located time-series and the interpolation routines

Absolutely a primitive point model aiming for efficiency and simplicity.

Units of x,y,z are metric, z positive upwards to sky, represents elevation x is east-west axis y is south-north axis

Overloaded function.

__init__(self: shyft.time_series.GeoPoint) -> None
__init__(self: shyft.time_series.GeoPoint, x: typing.SupportsFloat | typing.SupportsIndex, y: typing.SupportsFloat | typing.SupportsIndex, z: typing.SupportsFloat | typing.SupportsIndex) -> None

construct a geo_point with x,y,z

Parameters:

x (float) – meter units
y (float) – meter units
z (float) – meter units

__init__(self: shyft.time_series.GeoPoint, clone: shyft.time_series.GeoPoint) -> None

create a copy

Parameters:: clone (GeoPoint) – the object to clone

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.GeoPoint) -> None
__init__(self: shyft.time_series.GeoPoint, x: typing.SupportsFloat | typing.SupportsIndex, y: typing.SupportsFloat | typing.SupportsIndex, z: typing.SupportsFloat | typing.SupportsIndex) -> None

construct a geo_point with x,y,z

Parameters:

x (float) – meter units
y (float) – meter units
z (float) – meter units

__init__(self: shyft.time_series.GeoPoint, clone: shyft.time_series.GeoPoint) -> None

create a copy

Parameters:: clone (GeoPoint) – the object to clone

static difference(a: shyft.time_series.GeoPoint, b: shyft.time_series.GeoPoint) → shyft.time_series.GeoPoint¶: returns GeoPoint(a.x - b.x, a.y - b.y, a.z - b.z)

static distance2(a: shyft.time_series.GeoPoint, b: shyft.time_series.GeoPoint) → float¶: returns the euclidian distance^2

static distance_measure(a: shyft.time_series.GeoPoint, b: shyft.time_series.GeoPoint, p: SupportsFloat | SupportsIndex, zscale: SupportsFloat | SupportsIndex = 0.0) → float¶: return sum(a-b)^p

transform(*args, **kwargs)¶

Overloaded function.

transform(self: shyft.time_series.GeoPoint, from_epsg: typing.SupportsInt | typing.SupportsIndex, to_epsg: typing.SupportsInt | typing.SupportsIndex) -> shyft.time_series.GeoPoint

compute transformed point from_epsg to to_epsg coordinate

Parameters:

from_epsg (int) – interpret the current point as epsg coordinate
to_epsg (int) – the returned points epsg coordinate system

Returns:

new point. The new point in the specified cartesian epsg coordinate system

Return type:

GeoPoint

Notes

Limitations: The geographical or cartesian classification is based on well known epsg for geographical coordinates(4326,4226,4269,4258), others assumed cartesian.

transform(self: shyft.time_series.GeoPoint, from_proj4: str, to_proj4: str) -> shyft.time_series.GeoPoint

compute transformed point from_epsg to to_epsg coordinate

Parameters:

from_proj4 (str) – interpret the current point as this proj4 specification
to_proj4 (str) – the returned points proj4 specified coordinate system

Returns:

new point. The new point in the specified proj4 coordinate system

Return type:

GeoPoint

property x¶

east->west

Type:: float

static xy_distance(a: shyft.time_series.GeoPoint, b: shyft.time_series.GeoPoint) → float¶: returns sqrt((a.x - b.x)*(a.x - b.x) + (a.y - b.y)*(a.y - b.y))

property y¶

south->north

Type:: float

property z¶

ground->upwards

Type:: float

static zscaled_distance(a: shyft.time_series.GeoPoint, b: shyft.time_series.GeoPoint, zscale: SupportsFloat | SupportsIndex) → float¶: sqrt( (a.x - b.x)*(a.x - b.x) + (a.y - b.y)*(a.y - b.y) + (a.z - b.z)*(a.z - b.z)*zscale*zscale)

Class GeoTimeSeries¶

class shyft.time_series.GeoTimeSeries(*args, **kwargs)¶

Bases: pybind11_object

A minimal geo-located time-series, a time-series plus a representative 3d mid_point

Overloaded function.

__init__(self: shyft.time_series.GeoTimeSeries) -> None
__init__(self: shyft.time_series.GeoTimeSeries, mid_point: shyft.time_series.GeoPoint, ts: shyft.time_series.TimeSeries) -> None

Construct a GeoTimeSeries

Parameters:

mid_point (GeoPoint) – The 3d location representative for ts
ts (TimeSeries) – Any kind of TimeSeries

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.GeoTimeSeries) -> None
__init__(self: shyft.time_series.GeoTimeSeries, mid_point: shyft.time_series.GeoPoint, ts: shyft.time_series.TimeSeries) -> None

Construct a GeoTimeSeries

Parameters:

mid_point (GeoPoint) – The 3d location representative for ts
ts (TimeSeries) – Any kind of TimeSeries

property mid_point¶

the mid-point(of an area) for which the assigned time-series is valid

Type:: GeoPoint

property ts¶

the assigned time-series

Type:: TimeSeries

Class GeoTimeSeriesVector¶

class shyft.time_series.GeoTimeSeriesVector(*args, **kwargs)¶

Bases: pybind11_object

Overloaded function.

__init__(self: shyft.time_series.GeoTimeSeriesVector) -> None
__init__(self: shyft.time_series.GeoTimeSeriesVector, arg0: shyft.time_series.GeoTimeSeriesVector) -> None

Copy constructor

__init__(self: shyft.time_series.GeoTimeSeriesVector, arg0: collections.abc.Iterable) -> None
__init__(self: shyft.time_series.GeoTimeSeriesVector, time_axis: shyft.time_series.TimeAxis, geo_points: shyft.time_series.GeoPointVector, np_array: typing.Annotated[numpy.typing.ArrayLike, numpy.float64], point_fx: shyft.time_series.point_interpretation_policy) -> None

Create a GeoTimeSeriesVector from time-axis,geo-points,2d-numpy-array and point-interpretation

Parameters:

time_axis (TimeAxis) – time-axis that matches in length to 2nd dim of np_array
geo_points (GeoPointVector) – the geo-positions for the time-series, should be of length n_ts
np_array (np.ndarray) – numpy array of dtype=np.float64, and shape(n_ts,n_points)
point_fx (point interpretation) – one of POINT_AVERAGE_VALUE|POINT_INSTANT_VALUE

Returns:

GeoTimeSeriesVector. a GeoTimeSeriesVector of length first np_array dim, n_ts, each with geo-point and time-series with time-axis, values and point_fx

Return type:

GeoTimeSeriesVector

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.GeoTimeSeriesVector) -> None
__init__(self: shyft.time_series.GeoTimeSeriesVector, arg0: shyft.time_series.GeoTimeSeriesVector) -> None

Copy constructor

__init__(self: shyft.time_series.GeoTimeSeriesVector, arg0: collections.abc.Iterable) -> None
__init__(self: shyft.time_series.GeoTimeSeriesVector, time_axis: shyft.time_series.TimeAxis, geo_points: shyft.time_series.GeoPointVector, np_array: typing.Annotated[numpy.typing.ArrayLike, numpy.float64], point_fx: shyft.time_series.point_interpretation_policy) -> None

Create a GeoTimeSeriesVector from time-axis,geo-points,2d-numpy-array and point-interpretation

Parameters:

time_axis (TimeAxis) – time-axis that matches in length to 2nd dim of np_array
geo_points (GeoPointVector) – the geo-positions for the time-series, should be of length n_ts
np_array (np.ndarray) – numpy array of dtype=np.float64, and shape(n_ts,n_points)
point_fx (point interpretation) – one of POINT_AVERAGE_VALUE|POINT_INSTANT_VALUE

Returns:

GeoTimeSeriesVector. a GeoTimeSeriesVector of length first np_array dim, n_ts, each with geo-point and time-series with time-axis, values and point_fx

Return type:

GeoTimeSeriesVector

append(self: shyft.time_series.GeoTimeSeriesVector, x: shyft.time_series.GeoTimeSeries) → None¶: Add an item to the end of the list

clear(self: shyft.time_series.GeoTimeSeriesVector) → None¶: Clear the contents

extend(*args, **kwargs)¶

Overloaded function.

extend(self: shyft.time_series.GeoTimeSeriesVector, L: shyft.time_series.GeoTimeSeriesVector) -> None

Extend the list by appending all the items in the given list

extend(self: shyft.time_series.GeoTimeSeriesVector, L: collections.abc.Iterable) -> None

Extend the list by appending all the items in the given list

extract_ts_vector(self: shyft.time_series.GeoTimeSeriesVector) → shyft.time_series.TsVector¶

Provides a TsVector of the time-series part of GeoTimeSeries

Returns:: ts-vector. A TsVector(shallow copy) of the time-series part of GeoTsVector
Return type:: TsVector

insert(self: shyft.time_series.GeoTimeSeriesVector, i: SupportsInt | SupportsIndex, x: shyft.time_series.GeoTimeSeries) → None¶: Insert an item at a given position.

pop(*args, **kwargs)¶

Overloaded function.

pop(self: shyft.time_series.GeoTimeSeriesVector) -> shyft.time_series.GeoTimeSeries

Remove and return the last item

pop(self: shyft.time_series.GeoTimeSeriesVector, i: typing.SupportsInt | typing.SupportsIndex) -> shyft.time_series.GeoTimeSeries

Remove and return the item at index i

size(self: shyft.time_series.GeoTimeSeriesVector) → int¶

values_at_time(self: shyft.time_series.GeoTimeSeriesVector, t: shyft.time_series.time) → numpy.ndarray¶

The values at specified time as a an numpy array This function can be suitable if you are doing area-animated (birds-view) presentations

Parameters:: t (time) – the time that should be used for getting each value
Returns:: values. The evaluated geo.ts(t) for all items in the vector
Return type:: array

Class GeoQuery¶

class shyft.time_series.GeoQuery(*args, **kwargs)¶

Bases: pybind11_object

A query as a polygon with specified geo epsg coordinate system

Overloaded function.

__init__(self: shyft.time_series.GeoQuery) -> None
__init__(self: shyft.time_series.GeoQuery, epsg: typing.SupportsInt | typing.SupportsIndex, points: shyft.time_series.GeoPointVector) -> None

Construct a GeoQuery from specified parameters

Parameters:

epsg (int) – A valid epsg for the polygon, and also wanted coordinate system
points (GeoPointVector) – 3 or more points forming a polygon that is the spatial scope

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.GeoQuery) -> None
__init__(self: shyft.time_series.GeoQuery, epsg: typing.SupportsInt | typing.SupportsIndex, points: shyft.time_series.GeoPointVector) -> None

Construct a GeoQuery from specified parameters

Parameters:

epsg (int) – A valid epsg for the polygon, and also wanted coordinate system
points (GeoPointVector) – 3 or more points forming a polygon that is the spatial scope

property epsg¶

the epsg coordinate system

Type:: int

property polygon¶

the polygon giving the spatial scope

Type:: GeoPointVector

Class GeoSlice¶

class shyft.time_series.GeoSlice(*args, **kwargs)¶

Bases: pybind11_object

Keeps data that describes as slice into the t0-variable-ensemble-geo, (t,v,e,g), space. It is the result-type of GeoTimeSeriesConfiguration.compute(GeoEvalArgs) and is passed to the geo-db-read callback to specify wanted time-series to read. Note that the content of a GeoSlice can only be interpreted in terms of the GeoTimeSeriesConfiguration it is derived from. The indices and values of the slice, strongly relates to the definition of it’s geo-tsdb.

Overloaded function.

__init__(self: shyft.time_series.GeoSlice) -> None
__init__(self: shyft.time_series.GeoSlice, v: collections.abc.Sequence[typing.SupportsInt | typing.SupportsIndex], g: collections.abc.Sequence[typing.SupportsInt | typing.SupportsIndex], e: collections.abc.Sequence[typing.SupportsInt | typing.SupportsIndex], t: collections.abc.Sequence[shyft.time_series.time], ts_dt: shyft.time_series.time, skip_dt: shyft.time_series.time = time(0)) -> None

Construct a GeoSlice from supplied vectors.

Parameters:

v (IntVector) – list of variables idx, each defined by GeoTimeSeriesConfiguration.variables[i]
g (IntVector) – list of geo-point idx, each defined by GeoTimeSeriesConfiguration.grid.points[i]
e (IntVector) – list of ensembles, each in range 0..GeoTimeSeriesConfiguration.n_ensembles-1
t (UtcTimeVector) – list of t0-time points, each of them should exist in GeoTimeSeriesConfiguration.t0_times
ts_dt (time) – time-length to read from each time-series, we read from [t0+skip_dt.. t0+skip_dt+ts_dt)
skip_dt (time) – time-length to skip from start each time-series, we read from [t0+skip_dt.. t0+skip_dt+ts_dt)

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.GeoSlice) -> None
__init__(self: shyft.time_series.GeoSlice, v: collections.abc.Sequence[typing.SupportsInt | typing.SupportsIndex], g: collections.abc.Sequence[typing.SupportsInt | typing.SupportsIndex], e: collections.abc.Sequence[typing.SupportsInt | typing.SupportsIndex], t: collections.abc.Sequence[shyft.time_series.time], ts_dt: shyft.time_series.time, skip_dt: shyft.time_series.time = time(0)) -> None

Construct a GeoSlice from supplied vectors.

Parameters:

v (IntVector) – list of variables idx, each defined by GeoTimeSeriesConfiguration.variables[i]
g (IntVector) – list of geo-point idx, each defined by GeoTimeSeriesConfiguration.grid.points[i]
e (IntVector) – list of ensembles, each in range 0..GeoTimeSeriesConfiguration.n_ensembles-1
t (UtcTimeVector) – list of t0-time points, each of them should exist in GeoTimeSeriesConfiguration.t0_times
ts_dt (time) – time-length to read from each time-series, we read from [t0+skip_dt.. t0+skip_dt+ts_dt)
skip_dt (time) – time-length to skip from start each time-series, we read from [t0+skip_dt.. t0+skip_dt+ts_dt)

property e¶

list of ensembles, each in range 0..GeoTimeSeriesConfiguration.n_ensembles-1

Type:: IntVector

property g¶

list of geo-point idx, each defined by GeoTimeSeriesConfiguration.grid.points[i]

Type:: IntVector

property period¶

slice period as [skip_dt..skip_dt+ts_dt) from each ts relative t0

Type:: UtcPeriod

property skip_dt¶

time length to skip from start of each time-series, [t0+skip_dt.. t0+skip_dt+ts_dt)

Type:: time

property t¶

list of t0-time points, each of them should exist in GeoTimeSeriesConfiguration.t0_times

Type:: UtcTimeVector

property ts_dt¶

time length to read from each time-series, [t0+skip_dt.. t0+skip_dt+ts_dt)

Type:: time

property v¶

list of variables idx, each defined by GeoTimeSeriesConfiguration.variables[i]

Type:: IntVector

Class GeoTsMatrix¶

class shyft.time_series.GeoTsMatrix(self: shyft.time_series.GeoTsMatrix, n_t0: SupportsInt | SupportsIndex, n_v: SupportsInt | SupportsIndex, n_e: SupportsInt | SupportsIndex, n_g: SupportsInt | SupportsIndex)¶

Bases: pybind11_object

GeoTsMatrix is 4d matrix,index dimensions (t0,variable,ensemble,geo_point) to be understood as a slice of a geo-ts-db (slice could be the entire db) The element type of the matrix is GeoTimeSeries This type is typically the result of a dtss geo-evaluation, and note that GeoTimeSeries also provides the position.

create GeoTsMatrix with specified t0,variables,ensemble and geo-point dimensions

__init__(self: shyft.time_series.GeoTsMatrix, n_t0: SupportsInt | SupportsIndex, n_v: SupportsInt | SupportsIndex, n_e: SupportsInt | SupportsIndex, n_g: SupportsInt | SupportsIndex) → None¶: create GeoTsMatrix with specified t0,variables,ensemble and geo-point dimensions

concatenate(self: shyft.time_series.GeoTsMatrix, cc_dt0: shyft.time_series.time, concat_interval: shyft.time_series.time) → shyft.time_series.GeoTsMatrix¶

Concatenate all the forecasts in the GeoTsMatrix using supplied parameters

Parameters:

cc_dt0 (time) – skip first period of length cc_dt0 each forecast
concat_interval (time) – the nominal length between each ts.time(0) of all time-series

Returns:

tsm. A new concatenated geo-ts-matrix

Return type:

GeoTsMatrix

evaluate(self: shyft.time_series.GeoTsMatrix) → shyft.time_series.GeoTsMatrix¶

Apply the expression to each time-series of the specified variable.

Args: :returns: GeoTsMatrix. A new GeoTsMatrix, where all time-series is evaluated :rtype: GeoTsMatrix

extract_geo_ts_vector(self: shyft.time_series.GeoTsMatrix, t: SupportsInt | SupportsIndex, v: SupportsInt | SupportsIndex, e: SupportsInt | SupportsIndex) → shyft.time_series.GeoTimeSeriesVector¶

Given given arguments, return the GeoTimeSeriesVector suitable for constructing GeoPointSource for hydrology region-environments forcing data

Parameters:

t (int) – the forecast index, e.g. selects specific forecast, in case of several (t0)
v (int) – the variable index, e.g. selects temperature,precipitation etc.
e (int) – the ensemble index, in case of many ensembles, select specific ensemble

Returns:

GeoTimeSeriesVector. The GeoTsVector for selected forcast time t, variable and ensemble

Return type:

GeoTimeSeriesVector

get_geo_point(self: shyft.time_series.GeoTsMatrix, t: SupportsInt | SupportsIndex, v: SupportsInt | SupportsIndex, e: SupportsInt | SupportsIndex, g: SupportsInt | SupportsIndex) → shyft.time_series.GeoPoint¶: return self[t,v,e,g].mid_point of type GeoPoint

get_ts(self: shyft.time_series.GeoTsMatrix, t: SupportsInt | SupportsIndex, v: SupportsInt | SupportsIndex, e: SupportsInt | SupportsIndex, g: SupportsInt | SupportsIndex) → shyft.time_series.TimeSeries¶: return self[t,v,e,g] of type TimeSeries

set_geo_point(self: shyft.time_series.GeoTsMatrix, t: SupportsInt | SupportsIndex, v: SupportsInt | SupportsIndex, e: SupportsInt | SupportsIndex, g: SupportsInt | SupportsIndex, point: shyft.time_series.GeoPoint) → None¶: performs self[t,v,e,g].mid_point= point

set_ts(self: shyft.time_series.GeoTsMatrix, t: SupportsInt | SupportsIndex, v: SupportsInt | SupportsIndex, e: SupportsInt | SupportsIndex, g: SupportsInt | SupportsIndex, ts: shyft.time_series.TimeSeries) → None¶: performs self[t,v,e,g].ts= ts

property shape¶

The shape of a GeoMatrix in terms of forecasts(n_t0),variables(n_v),ensembles(n_e) and geopoints(n_g)

Type:: GeoMatrixShape

transform(*args, **kwargs)¶

Overloaded function.

transform(self: shyft.time_series.GeoTsMatrix, variable: typing.SupportsInt | typing.SupportsIndex, expression: shyft.time_series.TimeSeries) -> shyft.time_series.GeoTsMatrix

Apply the expression to each time-series of the specified variable.

Parameters:

() (expr) – the variable index to select the specific variable, use -1 to apply to all
() – ts expression, like 2.0*TimeSeries(‘x’), where x will be substituted with the variable, notice that its a required to be just one unbound time-series with the reference name ‘x’.

Returns:

GeoTsMatrix. A new GeoTsMatrix, where the time-series for the specified variable is transformed

Return type:

GeoTsMatrix

transform(self: shyft.time_series.GeoTsMatrix, expr_vector: shyft.time_series.TsVector) -> shyft.time_series.GeoTsMatrix

Apply the expression to each time-series of the specified variable.

Parameters:: () (expr_vector) – ts expressions, like 2.0*TimeSeries(‘0’), where 0 will be substituted with the corresponding variable, notice that its a required to be just one unbound time-series with the reference name ‘0’.
Returns:: GeoTsMatrix. A new GeoTsMatrix, where the time-series for the specified variable is transformed
Return type:: GeoTsMatrix

Class GeoMatrixShape¶

class shyft.time_series.GeoMatrixShape(self: shyft.time_series.GeoMatrixShape, n_t0: SupportsInt | SupportsIndex, n_v: SupportsInt | SupportsIndex, n_e: SupportsInt | SupportsIndex, n_g: SupportsInt | SupportsIndex)¶

Bases: pybind11_object

Create with specified dimensionality

__init__(self: shyft.time_series.GeoMatrixShape, n_t0: SupportsInt | SupportsIndex, n_v: SupportsInt | SupportsIndex, n_e: SupportsInt | SupportsIndex, n_g: SupportsInt | SupportsIndex) → None¶: Create with specified dimensionality

property n_e¶

number of ensembles

Type:: int

property n_g¶

number of geo points

Type:: int

property n_t0¶

number of t0, e.g forecasts

Type:: int

property n_v¶

number of variables

Type:: int

Class GeoGridSpec¶

class shyft.time_series.GeoGridSpec(*args, **kwargs)¶

Bases: pybind11_object

A point set for a geo-grid, but does not have to be a regular grid. It serves the role of defining the spatial representative mid-points for a typical spatial grid, e.g as for arome, or ec forecasts, where the origin shape usually is a box. To support general grid-spec, the optional, then equally sized, shapes provides the polygon shape for each individual mid-point.

Overloaded function.

__init__(self: shyft.time_series.GeoGridSpec) -> None
__init__(self: shyft.time_series.GeoGridSpec, epsg: typing.SupportsInt | typing.SupportsIndex, points: shyft.time_series.GeoPointVector) -> None

Construct a GeoQuery from specified parameters

Parameters:

epsg (int) – A valid epsg for the spatial points
points (GeoPointVector) – 0 or more representative points for the spatial properties of the grid

__init__(self: shyft.time_series.GeoGridSpec, epsg: typing.SupportsInt | typing.SupportsIndex, polygons: shyft.time_series.GeoPointVectorVector) -> None

Construct a GeoQuery from specified parameters

Parameters:

epsg (int) – A valid epsg for the spatial points
polygons (GeoPointVectorVector) – 0 or more representative shapes as polygons, the mid-points are computed based on shapes

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.GeoGridSpec) -> None
__init__(self: shyft.time_series.GeoGridSpec, epsg: typing.SupportsInt | typing.SupportsIndex, points: shyft.time_series.GeoPointVector) -> None

Construct a GeoQuery from specified parameters

Parameters:

epsg (int) – A valid epsg for the spatial points
points (GeoPointVector) – 0 or more representative points for the spatial properties of the grid

__init__(self: shyft.time_series.GeoGridSpec, epsg: typing.SupportsInt | typing.SupportsIndex, polygons: shyft.time_series.GeoPointVectorVector) -> None

Construct a GeoQuery from specified parameters

Parameters:

epsg (int) – A valid epsg for the spatial points
polygons (GeoPointVectorVector) – 0 or more representative shapes as polygons, the mid-points are computed based on shapes

property epsg¶

the epsg coordinate system

Type:: int

find_geo_match(self: shyft.time_series.GeoGridSpec, geo_query: shyft.time_series.GeoQuery) → list[int]¶

finds the points int the grid that is covered by the polygon of the geo_query note: that currently we only consider the horizontal dimension when matching points

Parameters:: geo_query (GeoQuery) – A polygon giving an area to capture
Returns:: matches. a list of all points that is inside, or on the border of the specified polygon, in guaranteed ascending point index order
Return type:: IntVector

property points¶

the representative mid-points of the spatial grid

Type:: GeoPointVector

property polygons¶

the polygons describing the grid, mid-points are centroids of the polygons

Type:: GeoPointVectorVector

Class GeoEvalArgs¶

class shyft.time_series.GeoEvalArgs(*args, **kwargs)¶

Bases: pybind11_object

GeoEvalArgs is used for the geo-evaluate functions.

It describes scope for the geo-evaluate function, in terms of:

the geo-ts database identifier

variables to extract, by names

ensemble members (list of ints)

temporal, using t0 from specified time-axis, + ts_dt for time-range

spatial, using points for a polygon

and optionally:

the concat postprocessing with parameters

Overloaded function.

__init__(self: shyft.time_series.GeoEvalArgs) -> None
__init__(self: shyft.time_series.GeoEvalArgs, geo_ts_db_id: str, variables: collections.abc.Sequence[str], ensembles: collections.abc.Sequence[typing.SupportsInt | typing.SupportsIndex], time_axis: shyft.time_series.TimeAxis, ts_dt: shyft.time_series.time, geo_range: shyft.time_series.GeoQuery, concat: bool, cc_dt0: shyft.time_series.time) -> None

Construct GeoEvalArgs from specified parameters

Parameters:

geo_ts_db_id (str) – identifies the geo-ts-db, short, as ‘arome’, ‘ec’, as specified with server.add_geo_ts_db(cfg)
variables (StringVector) – names of the wanted variables, if empty, return all variables configured
ensembles (IntVector) – List of ensembles, if empty, return all ensembles configured
time_axis (TimeAxis) – specifies the t0, and .total_period().end is used as concatenation open-end fill-in length
ts_dt (time) – specifies the time-length to read from each time-series,t0.. t0+ts_dt, and .total_period().end is used as concatenation open-end fill-in length
geo_range (GeoQuery) – the spatial scope of the query, if empty, return all configured
concat (bool) – postprocess using concatenated forecast, returns ‘one’ concatenated forecast from many.
cc_dt0 (time) – concat lead-time, skip cc_dt0 of each forecast (offsets the slice you selects)

__init__(self: shyft.time_series.GeoEvalArgs, geo_ts_db_id: str, ensembles: collections.abc.Sequence[typing.SupportsInt | typing.SupportsIndex], time_axis: shyft.time_series.TimeAxis, ts_dt: shyft.time_series.time, geo_range: shyft.time_series.GeoQuery, concat: bool, cc_dt0: shyft.time_series.time, ts_expressions: shyft.time_series.TsVector) -> None

Construct GeoEvalArgs from specified parameters

Parameters:

geo_ts_db_id (str) – identifies the geo-ts-db, short, as ‘arome’, ‘ec’, as specified with server.add_geo_ts_db(cfg)
ensembles (IntVector) – List of ensembles, if empty, return all ensembles configured
time_axis (TimeAxis) – specifies the t0, and .total_period().end is used as concatenation open-end fill-in length
ts_dt (time) – specifies the time-length to read from each time-series,t0.. t0+ts_dt, and .total_period().end is used as concatenation open-end fill-in length
geo_range (GeoQuery) – the spatial scope of the query, if empty, return all configured
concat (bool) – postprocess using concatenated forecast, returns ‘one’ concatenated forecast from many.
cc_dt0 (time) – concat lead-time, skip cc_dt0 of each forecast (offsets the slice you selects)
ts_expressions (TsVector) – expressions to evaluate, where the existing variables are referred to by index number as a string, ex. TimeSeries(‘0’)

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.GeoEvalArgs) -> None
__init__(self: shyft.time_series.GeoEvalArgs, geo_ts_db_id: str, variables: collections.abc.Sequence[str], ensembles: collections.abc.Sequence[typing.SupportsInt | typing.SupportsIndex], time_axis: shyft.time_series.TimeAxis, ts_dt: shyft.time_series.time, geo_range: shyft.time_series.GeoQuery, concat: bool, cc_dt0: shyft.time_series.time) -> None

Construct GeoEvalArgs from specified parameters

Parameters:

geo_ts_db_id (str) – identifies the geo-ts-db, short, as ‘arome’, ‘ec’, as specified with server.add_geo_ts_db(cfg)
variables (StringVector) – names of the wanted variables, if empty, return all variables configured
ensembles (IntVector) – List of ensembles, if empty, return all ensembles configured
time_axis (TimeAxis) – specifies the t0, and .total_period().end is used as concatenation open-end fill-in length
ts_dt (time) – specifies the time-length to read from each time-series,t0.. t0+ts_dt, and .total_period().end is used as concatenation open-end fill-in length
geo_range (GeoQuery) – the spatial scope of the query, if empty, return all configured
concat (bool) – postprocess using concatenated forecast, returns ‘one’ concatenated forecast from many.
cc_dt0 (time) – concat lead-time, skip cc_dt0 of each forecast (offsets the slice you selects)

__init__(self: shyft.time_series.GeoEvalArgs, geo_ts_db_id: str, ensembles: collections.abc.Sequence[typing.SupportsInt | typing.SupportsIndex], time_axis: shyft.time_series.TimeAxis, ts_dt: shyft.time_series.time, geo_range: shyft.time_series.GeoQuery, concat: bool, cc_dt0: shyft.time_series.time, ts_expressions: shyft.time_series.TsVector) -> None

Construct GeoEvalArgs from specified parameters

Parameters:

geo_ts_db_id (str) – identifies the geo-ts-db, short, as ‘arome’, ‘ec’, as specified with server.add_geo_ts_db(cfg)
ensembles (IntVector) – List of ensembles, if empty, return all ensembles configured
time_axis (TimeAxis) – specifies the t0, and .total_period().end is used as concatenation open-end fill-in length
ts_dt (time) – specifies the time-length to read from each time-series,t0.. t0+ts_dt, and .total_period().end is used as concatenation open-end fill-in length
geo_range (GeoQuery) – the spatial scope of the query, if empty, return all configured
concat (bool) – postprocess using concatenated forecast, returns ‘one’ concatenated forecast from many.
cc_dt0 (time) – concat lead-time, skip cc_dt0 of each forecast (offsets the slice you selects)
ts_expressions (TsVector) – expressions to evaluate, where the existing variables are referred to by index number as a string, ex. TimeSeries(‘0’)

property cc_dt0¶

concat lead-time

Type:: time

property concat¶

postprocess using concatenated forecast, returns ‘one’ concatenated forecast from many

Type:: bool

property ens¶

list of ensembles to return, empty=all, if specified >0

Type:: IntVector

property geo_range¶

the spatial scope, as simple polygon

Type:: GeoQuery

property geo_ts_db_id¶

the name for the config (keep it minimal)

Type:: str

property t0_time_axis¶

specifies the t0, and .total_period().end is used as concatenation open-end fill-in length

Type:: TimeAxis

property ts_dt¶

specifies the time-length to read from each time-series,t0.. t0+ts_dt,

Type:: time

property ts_expressions¶

time series expressions to evaluate instead of variables

Type:: TsVector

property variables¶

the human readable description of this geo ts db

Type:: StringVector

Class GeoTimeSeriesConfiguration¶

class shyft.time_series.GeoTimeSeriesConfiguration(*args, **kwargs)¶

Bases: pybind11_object

Contain minimal description to efficiently work with arome/ec forecast data. It defines the spatial, temporal and ensemble dimensions available, and provides means of mapping a GeoQuery to a set of ts_urls that serves as keys for manipulating and assembling forcing input data for example to the shyft hydrology region-models.

Overloaded function.

__init__(self: shyft.time_series.GeoTimeSeriesConfiguration) -> None
__init__(self: shyft.time_series.GeoTimeSeriesConfiguration, prefix: str, name: str, description: str, grid: shyft.time_series.GeoGridSpec, t0_times: collections.abc.Sequence[shyft.time_series.time], dt: shyft.time_series.time, n_ensembles: typing.SupportsInt | typing.SupportsIndex, variables: collections.abc.Sequence[str], json: str = ‘’, origin_proj4: str = ‘’, group_by_grid: bool = False, ppf: typing.SupportsInt | typing.SupportsIndex = 0) -> None

Construct a GeoQuery from specified parameters

Parameters:

prefix (str) – ts-url prefix, like shyft://container/ for internally stored ts, or geo:// for externally stored parts.Keep the container name short, like one letter, and let it point to a rocksdb configured container,e.g.: shyft://r/ is a minimal path. Keeping urls short matters when reaching many millons of items.
name (str) – A shortest possible unique name of the configuration
description (str) – a human readable description of the configuration
grid (GeoGridSpec) – specification of the spatial grid
t0_times (UtcTimeVector) – List of time where we have register time-series,e.g forecast times, first time-point
dt (time) – the (max) length of each geo-ts, so geo_ts total_period is [t0..t0+dt)
n_ensembles (int) – number of ensembles available, must be >0, 1 if no ensembles
variables (StringVector) – list of minimal keys, representing temperature, precipitation etc
json (str) – A user specified json string
origin_proj4 (str) – The proj4 string for the origin transform of this dataset
group_by_grid (bool) – If true, make dense time-series storage grouping grid-points
ppf (int) – Allow for time points-pr-fragment specification for this geo db

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.GeoTimeSeriesConfiguration) -> None
__init__(self: shyft.time_series.GeoTimeSeriesConfiguration, prefix: str, name: str, description: str, grid: shyft.time_series.GeoGridSpec, t0_times: collections.abc.Sequence[shyft.time_series.time], dt: shyft.time_series.time, n_ensembles: typing.SupportsInt | typing.SupportsIndex, variables: collections.abc.Sequence[str], json: str = ‘’, origin_proj4: str = ‘’, group_by_grid: bool = False, ppf: typing.SupportsInt | typing.SupportsIndex = 0) -> None

Construct a GeoQuery from specified parameters

Parameters:

prefix (str) – ts-url prefix, like shyft://container/ for internally stored ts, or geo:// for externally stored parts.Keep the container name short, like one letter, and let it point to a rocksdb configured container,e.g.: shyft://r/ is a minimal path. Keeping urls short matters when reaching many millons of items.
name (str) – A shortest possible unique name of the configuration
description (str) – a human readable description of the configuration
grid (GeoGridSpec) – specification of the spatial grid
t0_times (UtcTimeVector) – List of time where we have register time-series,e.g forecast times, first time-point
dt (time) – the (max) length of each geo-ts, so geo_ts total_period is [t0..t0+dt)
n_ensembles (int) – number of ensembles available, must be >0, 1 if no ensembles
variables (StringVector) – list of minimal keys, representing temperature, precipitation etc
json (str) – A user specified json string
origin_proj4 (str) – The proj4 string for the origin transform of this dataset
group_by_grid (bool) – If true, make dense time-series storage grouping grid-points
ppf (int) – Allow for time points-pr-fragment specification for this geo db

bounding_box(self: shyft.time_series.GeoTimeSeriesConfiguration, slice: shyft.time_series.GeoSlice) → shyft.time_series.GeoPointVector¶

Compute the 3D bounding_box, as two GeoPoints containing the min-max of x,y,z of points in the GeoSlice Could be handy when generating queries to externally stored geo-ts databases like netcdf etc. See also convex_hull().

Parameters:: slice (GeoSlice) – a geo-slice with specified dimensions in terms of t0, variables, ensembles,geo-points
Returns:: bbox. with two GeoPoints, [0] keeping the minimum x,y,z, and [1] the maximum x,y,z
Return type:: GeoPointVector

compute(self: shyft.time_series.GeoTimeSeriesConfiguration, eval_args: shyft.time_series.GeoEvalArgs) → shyft.time_series.GeoSlice¶

Compute the GeoSlice from evaluation arguments

Parameters:: eval_args (GeoEvalArgs) – Specification to evaluate
Returns:: geo_slice. A geo-slice describing (t0,v,e,g) computed
Return type:: GeoSlice

convex_hull(self: shyft.time_series.GeoTimeSeriesConfiguration, slice: shyft.time_series.GeoSlice) → shyft.time_series.GeoPointVector¶

Compute the 2D convex hull, as a list of GeoPoints describing the smallest convex planar polygon containing all points in the slice wrt. x,y. The returned point sequence is ‘closed’, i.e the first and last point in the sequence are equal. See also bounding_box().

Parameters:: slice (GeoSlice) – a geo-slice with specified dimensions in terms of t0, variables, ensembles,geo-points
Returns:: hull. containing the sequence of points of the convex hull polygon.
Return type:: GeoPointVector

create_geo_ts_matrix(self: shyft.time_series.GeoTimeSeriesConfiguration, slice: shyft.time_series.GeoSlice) → shyft.time_series.GeoTsMatrix¶

Creates a GeoTsMatrix(element type is GeoTimeSeries) to hold the values according to dimensionality of GeoSlice

Parameters:: slice (GeoSlice) – a geo-slice with specified dimensions in terms of t0, variables, ensembles,geo-points
Returns:: geo_ts_matrix. ready to be filled in with points and time-series
Return type:: GeoTsMatrix

create_ts_matrix(self: shyft.time_series.GeoTimeSeriesConfiguration, slice: shyft.time_series.GeoSlice, with_urls: bool = False) → shyft.time_series.GeoMatrix¶

Creates a GeoMatrix (element type is TimeSeries only) to hold the values according to dimensionality of GeoSlice

Parameters:

slice (GeoSlice) – a geo-slice with specified dimensions in terms of t0, variables, ensembles,geo-points
with_urls (bool) – if true, fill in ts-urls for the time-series, so that they can be used for direct evaluation/expressions

Returns:

ts_matrix. ready to be filled in time-series(they are all empty/null), or if with_urls, a symbolic unbound ts with correct url.

Return type:

GeoMatrix

property description¶

the human readable description of this geo ts db

Type:: str

property dt¶

the (max) length of each geo-ts, so geo_ts total_period is [t0..t0+dt)

Type:: time

find_geo_match_ix(self: shyft.time_series.GeoTimeSeriesConfiguration, geo_query: shyft.time_series.GeoQuery) → list[int]¶

Returns the indices to the points that matches the geo_query (polygon)

Parameters:: geo_query (GeoQuery) – The query, polygon that matches the spatial scope
Returns:: point_indexes. The list of indices that matches the geo_query
Return type:: IntVector

property grid¶

the spatial grid definition

Type:: GeoGridSpec

property group_by_grid¶

if true, make dense time-series storage grouping grid-points

Type:: bool

property json¶

a json formatted string with custom data as needed

Type:: str

property n_ensembles¶

number of ensembles available, range 1..n

Type:: int

property name¶

the name for the config (keep it minimal)

Type:: str

property origin_proj4¶

informative only, if not empty, specifies the origin proj4 of this dataset

Type:: str

property ppf¶

allow for time points-pr-fragment specification for this geo db

Type:: int

property prefix¶

//r/` for internally stored ts, or geo:// for externally stored parts

Type:: str
Type:: ts-url prefix, like `shyft

property t0_time_axis¶

t0 time-points as time-axis

Type:: TimeAxis

property t0_times¶

list of time-points, where there are registered/available time-series

Type:: UtcTimeVector

property variables¶

the list of available properties, like short keys for precipitation,temperature etc

Type:: StringList

Class GeoPointVector¶

class shyft.time_series.GeoPointVector(*args, **kwargs)¶

Bases: pybind11_object

A vector, list, of GeoPoints

Overloaded function.

__init__(self: shyft.time_series.GeoPointVector) -> None
__init__(self: shyft.time_series.GeoPointVector, arg0: shyft.time_series.GeoPointVector) -> None

Copy constructor

__init__(self: shyft.time_series.GeoPointVector, arg0: collections.abc.Iterable) -> None

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.GeoPointVector) -> None
__init__(self: shyft.time_series.GeoPointVector, arg0: shyft.time_series.GeoPointVector) -> None

Copy constructor

__init__(self: shyft.time_series.GeoPointVector, arg0: collections.abc.Iterable) -> None

append(self: shyft.time_series.GeoPointVector, x: shyft.time_series.GeoPoint) → None¶: Add an item to the end of the list

area(self: shyft.time_series.GeoPointVector) → float¶

compute polygon area, if less than 3 points NAN is returned, otherwise boost::geometry::area(polygon)

Returns:: area. the area in m2 (or according to coordinate projection)
Return type:: float

clear(self: shyft.time_series.GeoPointVector) → None¶: Clear the contents

static create_from_x_y_z(x: collections.abc.Sequence[SupportsFloat | SupportsIndex], y: collections.abc.Sequence[SupportsFloat | SupportsIndex], z: collections.abc.Sequence[SupportsFloat | SupportsIndex]) → shyft.time_series.GeoPointVector¶: Create a GeoPointVector from x,y and z DoubleVectors of equal length

extend(*args, **kwargs)¶

Overloaded function.

extend(self: shyft.time_series.GeoPointVector, L: shyft.time_series.GeoPointVector) -> None

Extend the list by appending all the items in the given list

extend(self: shyft.time_series.GeoPointVector, L: collections.abc.Iterable) -> None

Extend the list by appending all the items in the given list

insert(self: shyft.time_series.GeoPointVector, i: SupportsInt | SupportsIndex, x: shyft.time_series.GeoPoint) → None¶: Insert an item at a given position.

mid_point(self: shyft.time_series.GeoPointVector) → shyft.time_series.GeoPoint¶

compute polygon mid point x,y as boost::geometry::centroid, and z as average of points.z(leaving out the closing point).

Returns:: mid_point. The computed mid point, (nan,nan,nan) if there are no points
Return type:: GeoPoint

nearest_neighbours(self: shyft.time_series.GeoPointVector, p: shyft.time_series.GeoPoint, n: SupportsInt | SupportsIndex, z_scale: SupportsFloat | SupportsIndex = 0.0) → shyft.time_series.GeoPointVector¶

Return the n nearest neighbours of point p using z_scaled distance for z direction

Parameters:

p (GeoPoint) – Finds nearest neighbours to this point
n (int) – Number of neighbours to find
z_scale (float) – Default 0.0, e.g. only x,y direction counts, if 1.0, z counts equal etc.

Returns:

neighbours. The neighbours found

Return type:

GeoPointVector

polygon_buffer(self: shyft.time_series.GeoPointVector, distance: SupportsFloat | SupportsIndex) → shyft.time_series.GeoPointVector¶

Computes the buffer polygon for the supplied points using the distance parameter. boost.geometry.buffer algorithm is used, and if it results in multiple polygons, then boost.convex_hull is used to reduce it from many to one polygon envelope. If number of points is large, currently > 50, a reduction algorithm using convex_hull is applied as first step. On the return, the point z coordinate is set to the average of the polygon points z.

Parameters:: distance (float) – The buffer distance to nearest point in self
Returns:: polygon. A closed buffer polygon surrounding the passed points
Return type:: GeoPointVector

pop(*args, **kwargs)¶

Overloaded function.

pop(self: shyft.time_series.GeoPointVector) -> shyft.time_series.GeoPoint

Remove and return the last item

pop(self: shyft.time_series.GeoPointVector, i: typing.SupportsInt | typing.SupportsIndex) -> shyft.time_series.GeoPoint

Remove and return the item at index i

size(self: shyft.time_series.GeoPointVector) → int¶

transform(*args, **kwargs)¶

Overloaded function.

transform(self: shyft.time_series.GeoPointVector, from_epsg: typing.SupportsInt | typing.SupportsIndex, to_epsg: typing.SupportsInt | typing.SupportsIndex) -> shyft.time_series.GeoPointVector

compute transformed point from_epsg to to_epsg coordinate

Parameters:

from_epsg (int) – interpret the current point as cartesian epsg coordinate
to_epsg (int) – the returned points cartesian epsg coordinate system

Returns:

new point. The new points in the specified cartesian epsg coordinate system

Return type:

GeoPointVector

Notes

Limitations: The geographical or cartesian classification is based on well known epsg for geographical coordinates(4326,4226,4269,4258), others assumed cartesian.

transform(self: shyft.time_series.GeoPointVector, from_proj4: str, to_proj4: str) -> shyft.time_series.GeoPointVector

compute transformed point from_epsg to to_epsg coordinate

Parameters:

from_proj4 (str) – interpret the current point as this proj4 specification
to_proj4 (str) – the returned points proj4 specified coordinate system

Returns:

new GeoPointVector. The new points in the specified proj4 coordinate system

Return type:

GeoPointVector

Class GeoPointVectorVector¶

class shyft.time_series.GeoPointVectorVector(*args, **kwargs)¶

Bases: pybind11_object

A strongly typed list, of GeoPointVectors

Overloaded function.

__init__(self: shyft.time_series.GeoPointVectorVector) -> None
__init__(self: shyft.time_series.GeoPointVectorVector, arg0: shyft.time_series.GeoPointVectorVector) -> None

Copy constructor

__init__(self: shyft.time_series.GeoPointVectorVector, arg0: collections.abc.Iterable) -> None

__init__(*args, **kwargs)¶

Overloaded function.

__init__(self: shyft.time_series.GeoPointVectorVector) -> None
__init__(self: shyft.time_series.GeoPointVectorVector, arg0: shyft.time_series.GeoPointVectorVector) -> None

Copy constructor

__init__(self: shyft.time_series.GeoPointVectorVector, arg0: collections.abc.Iterable) -> None

append(self: shyft.time_series.GeoPointVectorVector, x: shyft.time_series.GeoPointVector) → None¶: Add an item to the end of the list

areas(self: shyft.time_series.GeoPointVectorVector) → list[float]¶

Returns the polygon area for each polygon in the list

Args: :returns: areas. The computed polygon areas :rtype: DoubleVector

clear(self: shyft.time_series.GeoPointVectorVector) → None¶: Clear the contents

extend(*args, **kwargs)¶

Overloaded function.

extend(self: shyft.time_series.GeoPointVectorVector, L: shyft.time_series.GeoPointVectorVector) -> None

Extend the list by appending all the items in the given list

extend(self: shyft.time_series.GeoPointVectorVector, L: collections.abc.Iterable) -> None

Extend the list by appending all the items in the given list

insert(self: shyft.time_series.GeoPointVectorVector, i: SupportsInt | SupportsIndex, x: shyft.time_series.GeoPointVector) → None¶: Insert an item at a given position.

intersection_areas(self: shyft.time_series.GeoPointVectorVector, polygon: shyft.time_series.GeoPointVector, n_max_neighbours: SupportsInt | SupportsIndex = 8) → list[float]¶

Returns the intersection area with polygon, each polygon in the list Notice that the algo first compute the 8 nearest neighbours, then computes intersections of those.

Parameters:

polygon (GeoPointVector) – The polygon to compute the intersection with
n_max_neighbours (int) – Limits the intersection work considering only the nearest neighbours, default 8

Returns:

areas. The computed intersection area for each polygon in polygons, -the same order

Return type:

DoubleVector

mid_points(self: shyft.time_series.GeoPointVectorVector) → shyft.time_series.GeoPointVector¶

Returns the polygon mid_point area for each polygon in the list

Args: :returns: mid_points. The computed mid-points :rtype: GeoPointVector

pop(*args, **kwargs)¶

Overloaded function.

pop(self: shyft.time_series.GeoPointVectorVector) -> shyft.time_series.GeoPointVector

Remove and return the last item

pop(self: shyft.time_series.GeoPointVectorVector, i: typing.SupportsInt | typing.SupportsIndex) -> shyft.time_series.GeoPointVector

Remove and return the item at index i

size(self: shyft.time_series.GeoPointVectorVector) → int¶

Working with time series¶

The elements in this section define how code shall behave or are actual tools dealing with time series.

Policies¶

The elements in this section describe how time series are interpreted.

Class convolve_policy¶

class shyft.time_series.convolve_policy(self: shyft.time_series.convolve_policy, value: SupportsInt | SupportsIndex)¶

Bases: pybind11_object

Ref TimeSeries.convolve_w function, this policy determine how to handle initial conditions USE_NEAREST: value(0) is used for all values before value(0), and value(n-1) is used for all values after value(n-1) == ‘mass preserving’ USE_ZERO : use zero for all values before value(0) or after value(n-1) == ‘shape preserving’ USE_NAN : nan is used for all values outside the ts BACKWARD : filter is ‘backward looking’ == boundary handling in the beginning of ts FORWARD : filter is ‘forward looking’ == boundary handling in the end of ts CENTER : filter is centered == boundary handling in both ends

Members:

USE_NEAREST

USE_ZERO

USE_NAN

BACKWARD

FORWARD

CENTER

BACKWARD = <convolve_policy.BACKWARD: 64>¶

CENTER = <convolve_policy.CENTER: 32>¶

FORWARD = <convolve_policy.FORWARD: 16>¶

USE_NAN = <convolve_policy.USE_NAN: 4>¶

USE_NEAREST = <convolve_policy.USE_NEAREST: 1>¶

USE_ZERO = <convolve_policy.USE_ZERO: 2>¶

__init__(self: shyft.time_series.convolve_policy, value: SupportsInt | SupportsIndex) → None¶

convolve_policy.name -> str

property value¶

Class derivative_method¶

class shyft.time_series.derivative_method(self: shyft.time_series.derivative_method, value: SupportsInt | SupportsIndex)¶

Bases: pybind11_object

Ref. the .derivative time-series function, this defines how to compute the derivative of a given time-series

Members:

DEFAULT

FORWARD

BACKWARD

CENTER

BACKWARD = <derivative_method.BACKWARD: 2>¶

CENTER = <derivative_method.CENTER: 3>¶

DEFAULT = <derivative_method.DEFAULT: 0>¶

FORWARD = <derivative_method.FORWARD: 1>¶

__init__(self: shyft.time_series.derivative_method, value: SupportsInt | SupportsIndex) → None¶

derivative_method.name -> str

property value¶

Class extend_fill_policy¶

class shyft.time_series.extend_fill_policy(self: shyft.time_series.extend_fill_policy, value: SupportsInt | SupportsIndex)¶

Bases: pybind11_object

Ref TimeSeries.extend function, this policy determines how to represent values in a gap EPF_NAN : use nan values in the gap EPF_LAST: use the last value before the gap EPF_FILL: use a supplied value in the gap

Members:

FILL_NAN

USE_LAST

FILL_VALUE

FILL_NAN = <extend_fill_policy.FILL_NAN: 0>¶

FILL_VALUE = <extend_fill_policy.FILL_VALUE: 2>¶

USE_LAST = <extend_fill_policy.USE_LAST: 1>¶

__init__(self: shyft.time_series.extend_fill_policy, value: SupportsInt | SupportsIndex) → None¶

extend_fill_policy.name -> str

property value¶

Class extend_split_policy¶

class shyft.time_series.extend_split_policy(self: shyft.time_series.extend_split_policy, value: SupportsInt | SupportsIndex)¶

Bases: pybind11_object

Ref TimeSeries.extend function, this policy determines where to split/shift from one ts to the other EPS_LHS_LAST : use nan values in the gap EPS_RHS_FIRST: use the last value before the gap EPS_VALUE : use a supplied value in the gap

Members:

LHS_LAST

RHS_FIRST

AT_VALUE

AT_VALUE = <extend_split_policy.AT_VALUE: 2>¶

LHS_LAST = <extend_split_policy.LHS_LAST: 0>¶

RHS_FIRST = <extend_split_policy.RHS_FIRST: 1>¶

__init__(self: shyft.time_series.extend_split_policy, value: SupportsInt | SupportsIndex) → None¶

extend_split_policy.name -> str

property value¶

Class interpolation_scheme¶

class shyft.time_series.interpolation_scheme(self: shyft.time_series.interpolation_scheme, value: SupportsInt | SupportsIndex)¶

Bases: pybind11_object

Interpolation methods used by TimeSeries.transform

Members:

SCHEME_POLYNOMIAL

SCHEME_LINEAR

SCHEME_CATMULL_ROM

SCHEME_CATMULL_ROM = <interpolation_scheme.SCHEME_CATMULL_ROM: 2>¶

SCHEME_LINEAR = <interpolation_scheme.SCHEME_LINEAR: 0>¶

SCHEME_POLYNOMIAL = <interpolation_scheme.SCHEME_POLYNOMIAL: 1>¶

__init__(self: shyft.time_series.interpolation_scheme, value: SupportsInt | SupportsIndex) → None¶

interpolation_scheme.name -> str

property value¶

Class point_interpretation_policy¶

class shyft.time_series.point_interpretation_policy(self: shyft.time_series.point_interpretation_policy, value: SupportsInt | SupportsIndex)¶

Bases: pybind11_object

Determines how to interpret the points in a timeseries when interpreted as a function of time, f(t)

Members:

POINT_INSTANT_VALUE

POINT_AVERAGE_VALUE

POINT_AVERAGE_VALUE = <point_interpretation_policy.POINT_AVERAGE_VALUE: 1>¶

POINT_INSTANT_VALUE = <point_interpretation_policy.POINT_INSTANT_VALUE: 0>¶

__init__(self: shyft.time_series.point_interpretation_policy, value: SupportsInt | SupportsIndex) → None¶

point_interpretation_policy.name -> str

property value¶

Class trim_policy¶

class shyft.time_series.trim_policy(self: shyft.time_series.trim_policy, value: SupportsInt | SupportsIndex)¶

Bases: pybind11_object

Enum to decide if to trim inwards or outwards where TRIM_IN means inwards, TRIM_ROUND rounds halfway cases away from zero.

Members:

TRIM_IN

TRIM_OUT

TRIM_ROUND

TRIM_IN = <trim_policy.TRIM_IN: 0>¶

TRIM_OUT = <trim_policy.TRIM_OUT: 1>¶

TRIM_ROUND = <trim_policy.TRIM_ROUND: 2>¶

__init__(self: shyft.time_series.trim_policy, value: SupportsInt | SupportsIndex) → None¶

trim_policy.name -> str

property value¶

Tools¶

The elements in this section work with time series.

Class KrlsRbfPredictor¶

class shyft.time_series.KrlsRbfPredictor(self: shyft.time_series.KrlsRbfPredictor, dt: shyft.time_series.time, gamma: SupportsFloat | SupportsIndex = 0.001, tolerance: SupportsFloat | SupportsIndex = 0.01, size: SupportsInt | SupportsIndex = 1000000)¶

Bases: pybind11_object

Time-series predictor using the KRLS algorithm with radial basis functions.

The KRLS (Kernel Recursive Least-Squares) algorithm is a kernel regression algorithm for aproximating data, the implementation used here is from:

This predictor uses KRLS with radial basis functions (RBF). Other related shyft.time_series.TimeSeries.krls_interpolation() shyft.time_series.TimeSeries.TimeSeries.get_krls_predictor()

Examples:

>>>
>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> from shyft.time_series import (
...     Calendar, utctime_now, deltahours,
...     TimeAxis, TimeSeries,
...     KrlsRbfPredictor
... )
>>>
>>> # setup
>>> cal = Calendar()
>>> t0 = utctime_now()
>>> dt = deltahours(3)
>>> n = 365*8  # one year
>>>
>>> # ready plot
>>> fig, ax = plt.subplots()
>>>
>>> # shyft objects
>>> ta = TimeAxis(t0, dt, n)
>>> pred = KrlsRbfPredictor(
...     dt=deltahours(8),
...     gamma=1e-5,  # NOTE: this should be 1e-3 for real data
...     tolerance=0.001
... )
>>>
>>> # generate data
>>> total_series = 4
>>> data_range = np.linspace(0, 2*np.pi, n)
>>> ts = None  # to store the final data-ts
>>> # -----
>>> for i in range(total_series):
>>>     data = np.sin(data_range) + (np.random.random(data_range.shape) - 0.5)/5
>>>     ts = TimeSeries(ta, data)
>>>     # -----
>>>     training_mse = pred.train(ts)  # train the predictor
>>>     # -----
>>>     print(f'training step {i+1:2d}: mse={training_mse}')
>>>     ax.plot(ta.time_points[:-1], ts.values, 'bx')  # plot data
>>>
>>> # prediction
>>> ts_pred = pred.predict(ta)
>>> ts_mse = pred.mse_ts(ts, points=3)  # mse using 7 point wide filter
>>>                                     # (3 points before/after)
>>>
>>> # plot interpolation/predicton on top of results
>>> ax.plot(ta.time_points[:-1], ts_mse.values, '0.6', label='mse')
>>> ax.plot(ta.time_points[:-1], ts_pred.values, 'r-', label='prediction')
>>> ax.legend()
>>> plt.show()

Construct a new predictor.

Parameters:

dt (float) – The time-step in seconds the predictor is specified for.
used (Note that this does not put a limit on time-axes)
be (but for best results it should)
it (approximatly equal to the time-step of time-axes used with the predictor. In addition)
long (should not be to)
day (else you will get poor results. Try to keep the dt less than a)

:param : :param 3-8 hours is usually fine.: :param gamma: Determines the width of the radial basis functions for :type gamma: float (optional) :param the KRLS algorithm. Lower values mean wider basis functions: :param wider basis functions means faster: :param computation but lower accuracy. Note that the tolerance parameter also affects speed and accurcy.: :param A large value is around 1E-2: :param and a small value depends on the time step. By using values larger: :param than 1E-2 the computation will probably take to long. Testing have reveled that 1E-3 works great: :param for a time-step of 3 hours: :param while a gamma of 1E-2 takes a few minutes to compute. Use 1E-4 for a: :param fast and tolerably accurate prediction.: :param Defaults to 1E-3: :param tolerance: The krls training tolerance. Lower values makes the prediction more accurate, :type tolerance: float (optional) :param but slower. This typically have less effect than gamma: :param but is usefull for tuning. Usually it should be: :param either 0.01 or 0.001.: :param Defaults to 0.01: :param size: The size of the “memory” of the predictor. The default value is :type size: int (optional) :param usually enough. Defaults to 1000000.:

__init__(self: shyft.time_series.KrlsRbfPredictor, dt: shyft.time_series.time, gamma: SupportsFloat | SupportsIndex = 0.001, tolerance: SupportsFloat | SupportsIndex = 0.01, size: SupportsInt | SupportsIndex = 1000000) → None¶

Construct a new predictor.

Parameters:

dt (float) – The time-step in seconds the predictor is specified for.
used (Note that this does not put a limit on time-axes)
be (but for best results it should)
it (approximatly equal to the time-step of time-axes used with the predictor. In addition)
long (should not be to)
day (else you will get poor results. Try to keep the dt less than a)

clear(self: shyft.time_series.KrlsRbfPredictor) → None¶: Clear all training data from the predictor.

mse_ts(self: shyft.time_series.KrlsRbfPredictor, ts: shyft.time_series.TimeSeries, points: SupportsInt | SupportsIndex = 0) → shyft.time_series.TimeSeries¶

Compute a mean-squared error time-series of the predictor relative to the supplied ts.

Parameters:

ts (TimeSeries) – Time-series to compute mse against.
points (int (optional)) – Positive number of extra points around each point to use for mse.
0. (Defaults to)

Returns:

mse_ts. Time-series with mean-squared error values.

Return type: