What is Shyft?
Shyft is a cross-platform, open source LGPL V.3 toolbox developed at Statkraft , in cooperation with the University of Oslo’s Department of Geosciences.
The goal of Shyft is to facilitate collaboration among system providers, users, and research communities solving energy market-related problems.
We perceive that the industry today is fragmented and inflexible, with specialized communities that do not collaborate closely enough and with systems that are not compatible.
We believe that the entire industry would benefit from increased transparency and standardization concerning data management, data types, and system interfaces.
By sharing fundamental components, we believe it is possible to achieve faster development, improved solutions, and closer collaboration among professional and academic communities in the power market.
Shyft is an open source toolbox that includes data types, interfaces, HPC services, and methods for data management that we believe can contribute to more standardization and collaboration around energy market-related issues, if other actors wish to use them and contribute to their improvement.
Shyft is in active maintenance and development, used in 24x7 operations at Statkraft and forms the basis for much of the data management and analysis work in Statkraft’s Energy Management division.
Shyft has rolling-releases - improvements are shipped as soon as testing proves stability and quality.
The overall goal for the toolbox is to provide python-enabled high performance components with operational quality.
This includes (but is not limited to) the features listed below:
Well-designed components and algorithms aiming at speed, robustness and scalability.
Using well known, high performance, open-source, 3rd party libraries.
C++ is the core language, with python as an interface for interaction and orchestration.
The toolbox comes with Unit-tests, integration tests, examples, and demo data, with high test coverage.
Continuous build-test-deploy on many platforms (linux, windows, even raspberry pi/arm is possible).
Integrated issue tracking and merge-requests, allowing everyone to contribute.
Transparent and openly available source code.
Supports in-house build-systems for those that would like to extend and integrate with in-house software products.
Some of our tools and libraries will work nice for other domains as well, like the Package shyft.time-series.
Features
Highly efficient C++ core
Highly modular MicroService architecture
Easy to use Python API to work with services, algorithms and orchestration.
Timeseries:
The TimeSeries module contains methods to handle time series data and time series calculations at high performance.
Use time series to read, store, and manipulate large amounts of time series data.
Create mathematical expressions and evaluate them when needed
Time-series, including expressions, can be attached to models as attributes
Perform distributed calculation for scalable high performance backend, with caching
Geo-location aware time-series database for working with geo-time-series
DTSS with advanced web-socket based api with subscription, for direct integration with web-front-ends technologies
Hydrology:
The Hydrology module provides a flexible hydrological forecasting toolbox built for operational environments.
Enable computationally efficient calculations of hydrological response at the regional scale
Allow for using the multiple working hypothesis to quantify forecast uncertainties
Provide the ability to conduct hydrological simulations with multiple forcing configurations
Foster rapid implementation into operational modeling improvements identified through research activities
Energy Market:
The Energy Market module uses models and algorithms to help energy companies plan, bid, and optimize their operations, with a focus on hydropower and the ability to handle data efficiently.
Provides tools for building, storing, and maintaining energy-market models.
Adopts a model-driven approach which integrates fundamental models with various algorithms for business-driven development.
Incorporates a Python-enabled architecture for both server and client-side operations.
Offers a comprehensive hydropower model suitable for daily planning, bidding processes, optimization, and daily operations.
The detailed hydropower model can also estimate inflow from catchments surrounding the system.
Encourages experts and institutions to focus on algorithm development while allowing energy companies to manage their data.
Dashboard:
The dashboard module is a tool that lets you easily create visual web displays using Bokeh, centralizing methods for cleaner code and flexible design.
Designed primarily to facilitate building of web dashboards using Bokeh.
Centralizes common methods to avoid repeated code, streamline app/widget description, ensure convenient refactoring, and promote code re-use.
Philosophy centers around treating Bokeh as a visualization engine in a model-presenter-view paradigm, aiming for eventual flexibility in replacing Bokeh with other engines.
Encourages pushing code to shyft.dashboard when beneficial for multiple projects.
Distinguishes roles: Users (app developers for projects) and Developers (shyft.dashboard developers).
History
Shyft was initially created to replace external software solutions that was difficult to modify and slow to update. Open source has been a primary strategy for Shyft:
Closer cooperation with research institutions
Shortest possible pathway from R&D to Operation
Increased knowledgebase
Initially, Shyft features was developed to solve hydrological problems. The goal for Statkraft is to create as much value as possible from the water that is available. This requires numerical answers for the following questions for all its locations:
How much water do we currently have in our repositories?
When will new water arrive in our systems?
What will the weather be like?
Some estimations to handle these questions include:
How snow melts
How ground water flow
How the ground dries up
40-50% of energy consumption in Norway arrives as snow in any given year. How you estimate how snow melts have a huge impact on your forecast.
Since e.g. snow melting plays such a large role, it was essential to not be locked in by external software, which contained limited sets of snow algorithms.
These problems involved being able to solve equations very efficiently, and being able to handle large amounts of geographically distributed time series data in a fast and consistent way. These base components of Shyft turned out so robust, efficient, and scalable, that they are now part of a variety of Statkraft systems.
One of the components in Shyft is the time-series components, which has proven to be useful in many other applications as well. Therefore, it has been distributed as a separate installable package: shyft.time_series
More tools are continuously being added, and more problems are being solved in the Shyft toolbox.
The current scope of the toolbox now covers the energy market area, as illustrated:
Figure crafted by Christine Schei Liland
Design principles
Shyft is tailored for regional-scale hydropower inflow forecasting, emphasizing the exploration of multiple model realizations for forecast uncertainty insight. While focusing on operational requirements, Shyft ensures top-notch code quality and efficiency for the hydropower market through the following principles:
2.1 Enterprise-Level Software: Shyft adheres to the latest code standards, ensuring secure, tested, and well-documented open-source software. With tracking and extensive testing Shyft has over 90% coverage for C++ and around 60% for Python.
2.2 Direct Connection to Data Stores: Believing that “Data should live at the source!”, Shyft minimizes the use of intermediate data files and emphasizes direct connections to original data sources. This ensures timely forecasting while accommodating ongoing quality checks and potential data updates.
2.3 Efficient Integration of New Knowledge: Recognizing the importance of Research & Development, Shyft offers a solid hydrological modelling platform with flexibility to incorporate new algorithms. This ensures that research findings can be efficiently integrated into the operational framework.
2.4 Flexible Method Application: Shyft champions exploratory research flexibility, providing tools to quantify uncertainty. Users can delve into various hydrological algorithms and select alternative datasets, understanding the resulting variability.
2.5 Hot Service: Shyft aims to allow on-the-spot model computations, moving away from storing results for later analysis. By focusing on rapid, efficient algorithms, it envisions real-time model adjustments and computations.
Overall, Shyft’s design principles ensure a software that is consistent, easily integrated, and upholds high standards of code quality and security.