Concepts

Shyft provides a fully working operationally used descriptive electricity energy-market model framework.

The main physical components that constitute the essential parts are represented, such as the power-grid, producers, consumers, and also details relevant for production planning, catchments(runoff), reservoirs, tunnel and river network, wind farms, and generic power modules for describing other power sources (nuclear, coal, gas etc.). The detail level is developed so that we can cover common energy-management processes, spanning from intra-day up to long term market simulations, and ‘what if’ scenarios for renewable energy sources, maintenance planning etc.

In this context, the descriptive model try to capture the essential components, as they are physically, like river, tunnel, unit, bus-bar, transmission-line, or for logical items, like contracts, quantity, price etc.

The descriptive model allows us to use common terms, and makes room for the ‘algorithmic models’, often created by harvesting information from the descriptive model, and translate it into data-structures, like matrices, networks, or combination, that can be passed to solvers to find optimum solutions, or practical/realistic simulation results.

Note

The following content is produced iteratively using questions to open AI

Main component categories

        classDiagram
    direction LR
    System -- Network :networks
    System -- HydroPowerSystem : hydro_power_systems
    System -- WindFarm : wind_farms
    System -- PowerModule : power_modules
    System -- MarketArea : market_areas
    System -- UnitGroup : unit_groups
    System -- Contract : contracts
    System -- ContractPortfolio : contract_portfolios
    

The top-level object is energy market System. It keeps together the main components that describes the energy-market model.

Note

The current python class name for System is StmSystem but will be renamed in coming releases

Network, the power grid

The power grid is a complex system responsible for delivering electricity from the power generation units to end consumers. It primarily consists of generating stations, transmission lines, and distribution lines, all interconnected to ensure a continuous and reliable flow of electricity. In this context, two critical components of the power grid are the bus-bar and transmission line.

        classDiagram
    direction TB
    Network -- Busbar: busbars
    Network -- TransmissionLine : transmission_lines
    TransmissionLine --> Busbar: to
    TransmissionLine --> Busbar: from
    

Bus-bar

A bus-bar is a key node in the power grid where multiple paths converge. It’s a thick strip made from a highly conductive material such as copper or aluminum that helps in the distribution of electricity. Its main function is to effectively distribute electric power with high efficiency while also reducing the chance of power losses. Bus-bars are usually found in electrical substations, where different electrical inputs are effectively managed and dispatched. It symbolically represents the point in a system where multiple elements can supply power or have power drawn from.

Transmission line

Transmission lines constitute the network that connects power generation units with local distribution systems. These high-capacity carriers efficiently deliver electrical power over long distances. Made from highly conductive materials like aluminum or copper, these lines are designed to minimize losses during transmission. They usually operate at high voltages, in order to reduce the dissipation of energy as heat, and are supported high above the ground on transmission towers. A crucial part of the grid, transmission lines ensure that generated power is reliably delivered to substations, which then distribute the power to end consumers.

Hydro power system

The Shyft Hydro Power System is a comprehensive network designed to harness the potential of water for electricity generation. The fundamental components of the system include catchments, reservoirs or rivers, a tunnel network, and power generation units comprising turbines and generators.

        classDiagram
    direction LR
    HydroPowerSystem -- Catchment: catchments
    HydroPowerSystem -- Reservoir: reservoirs
    HydroPowerSystem -- Waterway: waterways
    HydroPowerSystem -- Gate: gates
    Waterway --> Gate: gates
    HydroPowerSystem -- PowerPlant: power_plants
    HydroPowerSystem -- Unit: units
    PowerPlant --> Unit: units
    

Catchments

The system begins with catchment areas, also known as drainage basins. These are areas of land where precipitation collects and drains off into a common outlet. In the context of the hydro power system, the outlet is typically a reservoir or a river. The role of the catchment is to provide a consistent source of water, known as discharge or runoff, to fuel the power generation process.

Reservoirs and Rivers

Upon leaving the catchment, the runoff flows into reservoirs or rivers. These bodies of water serve as natural storage systems for the collected water. In some cases, rivers provide an added input of water to the system through lateral inflow. These large volumes of stored water carry significant potential energy due to their elevation. Rivers float into creek-inlets, or reservoirs, that keeps the water.

Tunnel Network

The water from the reservoirs or creek-inlets is then channeled through a tunnel network to the power generation units. These tunnels, also known as penstocks in hydroelectric power systems, are designed to guide the water flux towards the turbines effectively. The force of the flowing water through these tunnels helps generating the high mechanical energy used for power generation. The complexity of the tunnel network depends on power-plant design. The most complex systems do have multiple reservoir upstreams, with pressure-balanced flow, as well as similar downstream reservoirs. The turbines might be sharing tunnels, penstocks and draft in any configuration as designed. Also note that turbines might be reversible, working as pumps. Depending on physical design a unit can be both generating and pumping, but not at the same time, or just one of them.

Power Generation Units

At the heart of the system are the power generation units, each consisting of a turbine and a generator. As water rushes down the tunnels, it strikes the blades of the turbine, causing it to spin. This kinetic energy gets converted into mechanical energy. The rotating turbine is connected to a generator, where this mechanical energy gets further converted into electrical energy. This is the fundamental process through which the hydro power system generates electricity.

Wind farms

A wind farm, is a group of wind turbines in the same location used for the production of electric power. A large wind farm may consist of several hundred individual wind turbines spread over an extensive area.

Wind Turbines

Wind turbines are the critical components of a wind park. They convert the kinetic energy in the wind into mechanical power. This mechanical power can be used for specific tasks, or a generator can convert it into electricity to power homes, businesses, schools, etc. Each wind turbine consists of a tower, rotor, and nacelle. The rotor has multiple blades (usually three) that capture wind’s energy. The nacelle houses a drive train which includes a gearbox and generator.

Substations(planned)

Substations play a crucial role within a wind park. They are intermediary points in the power grid that transform the electrical output from wind turbines to a higher voltage so that it can be transmitted efficiently over long distances. Substations also control and protect the electrical power flows, respond to faults, provide points for grid connection and disconnection, and serve as hubs to redistribute the power to multiple directions.

Grid Connection

The electricity produced by a wind park needs to be transmitted to where it is needed. After the electricity is generated by the turbines and stepped up to a high voltage by the substations, the power is fed into the electric grid. Here, it is combined with electricity from other power plants and distributed to end consumers. The process for connecting a wind park to the grid is complex and involves extensive planning to ensure the stability of the grid while making the most of the fluctuating and location-dependent nature of wind energy. Grid operators need to balance the supply from the wind park with the demand from consumers while also ensuring the quality of the electricity supply (e.g., maintaining frequency and reliability).

Power modules

The PowerModule in Shyft is a versatile component that can simulate various entities within a power system. It can represent either power production units like power plants or power consumption entities like towns or cities. The functionality of a PowerModule is dictated by the properties assigned to it.

PowerModule as a Power Producer

When configured to act like a power plant, the PowerModule can be given properties to simulate any type of power plant, each with its own unique energy conversion methods.

• Gas/Coal-Fired, Combined Heat and Power: To replicate a fossil fuel power plant, the PowerModule can be given properties such as the type of fuel used, the efficiency of the plant at converting that fuel to electricity, the capacity of the plant, and emission factors.

• Nuclear : When functioning as a nuclear power plant, the PowerModule may have properties pertaining to the type of nuclear reactor, the level of efficiency in converting thermal energy to electricity, the plant’s output capacity, and the associated waste and emission factors. Other energy production methods, like wind, hydro, and solar, can also be simulated by appropriately setting the PowerModule’s properties.

PowerModule as a Power Consumer

The PowerModule can also be configured to mimic a power consumer, such as an urban city or a small town. In this scenario, the PowerModule is given properties to represent characteristics of electricity usage. For example, it can be assigned features that signify peak load times, average electricity use, factors impacting power use like weather or time of day, population size, and the type of users (residential, commercial, industrial). Price/temperature sensitivity and seasonal patterns are also common to use for these parts.

Summary

In essence, the PowerModule in Shyft is a powerful tool for simulating diverse elements within a power grid, be they power producers or consumers. The versatility of this module allows for accurate modeling and forecasting in power systems, thereby enhancing the optimization and management of the power grid.

Market areas

The MarketArea class in Shyft plays a crucial role in simulating and managing power markets within a power grid. The boundaries of a MarketArea are defined by a unique set of bus-bars that it contains and the power transmission lines that interconnect one instance of the MarketArea with others.

Bus-Bars and Power Transmission Lines

Bus-bars are conductor rails that distribute power from generators to the load. In the context of MarketArea, they serve as the nodes that define the spatial extent of a market area. The interconnected bus-bars form a network that allows power to be transmitted and distributed across different regions within the MarketArea. Power transmission lines are crucial for interconnecting different market areas. These high-voltage lines carry electricity over long distances from power plants to load centers. In MarketArea, these lines allow for the transfer of electricity between different instances of the market, enabling a more efficient distribution of power resources based on supply and demand conditions in different areas.

Multiple Market Types

A key feature of the MarketArea class is the support for multiple market types. Depending on the Transmission System Operator’s (TSO’s) requirements for operational stability and redundancy, different delineations can be defined within a MarketArea. For example, there could be wholesale electricity markets, real-time markets, day-ahead markets, and ancillary services markets within a MarketArea, each serving specific purposes and enabling different types of transactions. These markets can be structured and designated according to the unique operational and redundancy needs of the TSO. To summarize, the MarketArea class in Shyft provides a comprehensive tool for representing and operating various aspects of power markets within a power grid. The establishment of different market types and the interconnection of market areas via transmission lines offer flexible and robust ways to ensure a stable, efficient, and reliable power grid.

Contracts and Contract portfolios

The ‘Contract’ in Shyft is a fundamental object that represents an agreement between two or more parties(actors) to buy or sell a specific product at a predetermined price and time. The properties of the ‘Contract’ encapsulate the following essential information:

• Time,t: The time element represents the period during which the product will be delivered. This could be a specific point in time or a defined time-window depending on the type of product and market rules.

• Price(t): This represents the cost per unit of the product that has been agreed upon by the parties involved.

• Quantity(t): This represents the volume or amount of the product that will be delivered under the agreement.

• Constraints and Penalties(t): Various constraints with corresponding penalties applicable to the contract

Actors

The actors represents the entities that operates in the energy-market, producers, consumers, grid-owners, transmission system operators(TSO).

The are related to contracts, as well as energy-market data providers or observers. They can also be co-owners in assets of some form.

Contract and Hydro Power-Plant

Contracts can optionally form an association with a hydro power plant. This feature ensures that a certain volume of energy supplied under the contract is generated using renewable energy from the hydro power-plant. By including this option, Shyft allows for agreements that support renewable energy initiatives and follow environmental compliance rules. Contract Semantic Relations In addition to physical relationships, contracts can also have semantic relations with other contracts. These relations can be governed by market rules applicable to the product. Depending on the defined market rules, this could be:

• Chained Contracts: In this type of relationship, the contract is considered applicable only if the entire chain of contracts is chosen. This is often seen in situations where delivery of the product involves multiple stages or when contracts are used to hedge risks.

• Exclusive Contracts: Here, the semantic relation terms specify that the contract is exclusive, in the sense that one contract can be chosen out of two or more, but not simultaneously.

ContractPortfolio

A ‘ContractPortfolio’ in Shyft is a collection of ‘Contract’ objects. It offers a consolidated view of multiple contracts for efficient management and risk assessment. The ‘ContractPortfolio’ allows for the analysis of aggregated risks, costs, and revenues from the portfolio of contracts. It aids in decision-making, helps manage complexities associated with multiple contracts, and plays a key role in strategizing and planning. Overall, the ‘Contract’ and ‘ContractPortfolio’ classes in Shyft represent a comprehensive system for managing energy contracts, enabling efficient and flexible market operations, promoting renewable energy, and allow for flexible modelling of market rules.

UnitGroup

The UnitGroup class in Shyft is a key component representing a group of active hydro power units that collectively contribute to the production of a specific product or generation capability.

The unit group members, are hydro-power units, which in turn are connected to the power-grid via bus-bars. The membership time-frame can also be controlled explicit by a mask time-series (1 or 0), so that the contribution also follows the mask time-series.

Obligations and Delivery

The UnitGroup has an obligation, typically defined by contracts or regulatory requirements, that should be met. This can be viewed as the performance metric or output that is expected from the group of units. For example, it can be the total amount of electricity that the unit group is obligated to generate in a specific timeframe. One of the main functionalities provided by the UnitGroup class is the computed delivery of the product that the unit group provides. This refers to the capability of the UnitGroup to calculate and report the cumulative product or energy output delivered by the group of units at any given time.

In practice, there is often a schedule, a plan for the delivery, then there is the optimized/simulated delivery, and most important, the realised delivery, that indicates the historical fact from observed measurements.

Nordic/European Energy Market operational processes and data-flows

This section describes the operational energy-managment process and data-flows for typical de-regulated energy markets, and also indicates how Shyft, and popular tools/techniques are applied.

The scope and terms are currently addressing the Nordic/European market, as this has been the primary use-cases for Shyft, and the development of Shyft as such.

Similar markets for electricity and energy management exists around the world, and the tools, models and micro-services approach could be useful in those as well.

Hopefully this help to establish the context for Shyft model-driven micro service architecture, so that it can be utilized in efficient manner.

Birdsview of the energy-managment process and the essential data-flows

Legend and Color Usage

The colors and backgrounds used in the visualizations are chosen to emphasize specific, non-functional aspects of the energy market model.

Red (data-flow)

• Indicates values that directly control production-system setpoints.

• These systems and flows are subject to strict national laws and regulations.

• Requirements include end-to-end encryption, both:

  • Technical level: compliance with cryptographic standards required.

  • Juridical level: exclusive ownership and control of endpoint systems.

  • This involves implementing robust physical and cyber defences, ensuring data soveregnty, and maintaining autonomy over the digital and physical infrastructure vital for a nation’s functioning.

Blue (data-flow)

• Represents observations from the process-control (SCADA) system.

• These values are required by the intraday execution and balance control system to generate valid plans and send proper control signals back to the process-control system.

Light Green (sub-process background)

• Used for intraday execution and balance processes, as well as the process-control (SCADA).

• These systems are subject to national laws and regulations to ensure a robust energy infrastructure.

• They include software, data, and algorithms that directly control asset production levels and gate flows.

• Related ‘red data-flows’ feed into these processes.

• National requirements typically demand that the responsible legal entity has exclusive logical and physical control over equipment, networks, and the data centers.

• Further general reading about Critical Infrastructure Protection and Resilience:

  • EU Critical infrastructure resilience https://home-affairs.ec.europa.eu/policies/internal-security/counter-terrorism-and-radicalisation/protection/critical-infrastructure-resilience-eu-level_en

• Norway specific guides can be found here (norwegian)

  • General: https://veiledere.nve.no/kraftberedskapsforskriften

  • Specific: https://veiledere.nve.no/kraftberedskapsforskriften/kapittel-7-beskyttelse-av-driftskontrollsystem/

• Swedish guides can be found here

  • TSO(SvK) in general: https://www.svk.se/om-kraftsystemet/legalt-ramverk/svenska-kraftnats-foreskrifter/

  • PMFS 2022:1 : https://sakerhetspolisen.se/sakerhetsskydd/lag-forordning-och-foreskrifter.html

• With the introduction of Nordic balancing model(NBM), https://nordicbalancingmodel.net, mFRR, aFRR, FCR, in which the TSO(Svenska Kraftnät,Statnett,Fingrid,Energinet) directly control production levels, end-to-end communication—both technical and juridical—must follow the ENTSO-E standard: https://www.entsoe.eu/ecco-sp , ECP/EDX protocol.

• The NBM implementation guides https://nordicbalancingmodel.net/implementation-guides , provides further details and specific message-flows.

Blue / Light Blue (sub-process background)

• Refers to Nordic/EU-level financial and corporate scope.

• These processes aim to determine an optimal economic strategy at a regional level.

• They do not directly control production; instead, they generate strategic plan proposals for the intraday execution process.

Pink

• Denotes financial or contract-related items and data-flows.

(1) External Data

The sources of external data and api’s relevant for the energy-management processes are multiple, and can be separated into a few categories as indicated, that feeds into the different processes.

(2) Weather processing and inflow forecasting

Ingest external weather data, slice and transform relevant parts into usable datasets that are used to feed the inflow forecast engine, as well as scenario consistent weather inputs to the other operative processes.

(3) Long and medium term forecasting and strategy

These process have models for the entire energy-market scoped, it takes forcing data covering the geographic, temporal and scenario scope, simulate the market for 1..n years, and further break down those strategy into seasonal(1..months) and even weeks-wise strategies. These are formulated as water-values, price,production,consumption,exchange, reservoir levels.

Each level in this process provides highly useful signals that are needed for the other processes;

4. Day-ahead markets (Elspot and balancing)

9. Trade, hedge using futures/forwards

(4) Day-ahead market and bidding

Taking inputs from current state of the production system, the plans until next bidding period, combined with water-values, price forecasts, generate elpsot-bids that reflects the seasonal-weekly disposition goals.

When elspot bids are accepted, work out detailed execution plan proposals, that also allows for placing bids into the balancing market products, FCR, FRR. When these are all finished, the detailed commitment is available for the intraday execution and balance control process.

(5) Intraday execution and balance control

The inputs, forcing data, such as obligation and commitment from the upstream day-ahead process, real time updated system state from the SCADA, the current and close future inflow and weather.

Detailed and optimized(using SHOP) plans are created ahead, and forwarded to the SCADA/Autopilot function. Local operational conditions for the water-courses are accounted for and plans are adjusted accordingly.

The deliveries of the products are kept in balance with the obligations, and for the mFRR market, energy-bids realizing the day-ahead mFRR commitment promises are generated and sent to TCO.

TCO mFRR energy activations via the dedicated ECP/EDX endpoint are checked for correctness, and forwarded to the SCADA for activation.

The Intraday execution and control process is subject to high security and availability, so basically all communication initiatives with the upstream processes, needs to be initiated from this process.

Shyft supports that scenario, utilizing the dtss exchange mechanisms that a provides push/pull of time-series and also bridging queue that contains packets of time-series in similar manner.

(6) Process control, SCADA, Autopilot

The process control, SCADA, with Autopilot for automated plan executions.

Shyft provides the basic mechanism in its c++ library for implementing these features.

Another important feature of Shyft in this regards, is the stability of the time-series protocols and api’s that allows minimal, stable and safe data-exchange from the upstream intraday exececution process.

(7) Nordic balancing market

The Nordic balancing market, NBM, is a collaboration between the Nordic TSOs (Statnett, Svenska kraftnät, Fingrid, Energinet) to develop and harmonize balancing markets (mFRR, aFRR, FCR). It complements the Nord Pool’s markets by securing real-time balance in the grid area.

(8) Nord Pool Exchange

Nord Pool is the leading power exchange in Europe, operating the day-ahead and intraday electricity markets. It covers the Nordic and Baltic countries, and extends into several EU countries through market coupling. Nord Pool’s scope is therefore regional and European, as its markets are integrated with the wider European power system via cross-border transmission lines and capacity allocation mechanisms.

(9) Trade and hedging

The trade and hedging process uses financial instruments such as forwards and futures to manage the risks associated with participation in the physical Nord Pool market. The main goals of this process are:

• Risk management: reduce exposure to volatile spot prices by locking in prices for future delivery periods.

• Revenue stability: secure predictable income streams to support investment planning and financial performance.

• Portfolio optimization: balance market positions across time horizons and trading partners.

Trading is carried out on the Nord Pool market place, but can also involve bilateral agreements (OTC contracts) between market participants. These financial positions complement physical market participation by providing a hedge against price fluctuations in day-ahead and intraday markets.

(10) Contract clearing and settlement

Contract clearing and settlement ensures that the financial and physical commitments from trading activities (Day-Ahead, Intraday, and bilateral trade) are correctly executed. This process handles the verification of traded volumes, the calculation of imbalances, and the financial settlement between market participants. It provides the final link that turns market transactions into binding financial and operational outcomes.

Detailed processes and data-flows for the operative energy-managment processes

The details further leverage the overall energy-market operational data-flow, with essential internal processes and data-flows.

Detail information

The color usage is indicating semantically the same as for the overview drawing. In this diagram, we also indicate the internal important component in each sub-process.

It is structurally the same diagram, but now with more details for each sub-process.

(2) Weather processing and hydrological forecasting

Weather processing and hydrological forecasting sub-process details includes collectors that pulls data from public and private providers of weather data and prognosis. Quality assurance algorithms for the received data-sets are applied, and if accepted, then these are sliced, transformed and appended to Shyft dtss/grid time-series database.

Typical, these gridded forecasts do have multiple scenarios(50..100), and the using concatenation of the best forecast front period, e.g hour 1..6 of arome, yields the ‘historical’ best estimate for ‘observed’ gridded history.

When providing LTM weather and inflow forecasts, these are used to provide ‘synthetic’ forecasts for the LTM simulation horizon (e.g. 3..5 years).

Shyft using the shyft.hydrology, with micro-services and orchestration, along with the Shyft DTSS, provides a complete set of components for all tasks related to this process.

(3) Long and medium term strategy

The Long and medium term strategy (LTM) process is exemplified using the Sintef software portfolio tools, that split up the process into using EMPS for the multi-year simulation, then using EOPS, or PRODRISK for breaking down the long term dispositions into season-weekly strategic signals.

Planned in Shyft

Statkraft has decided to open-source its basic Shyft based tools to work with these Sintef models for analysis purposes. Work are in progress to provide this, ETA is end of 2026, or 2026 Q3.

(4) and (5) Day-ahead and intraday execution and balance control

The details here indicates usage of the Sintef SHOP as a tool to compute the optimal solutions, given the market-model constraints, the hydro-power system etc.

Shyft provides the DStm and Compute(SHOP) micro-services that are the components to use for these parts.

(6) Process control, Autopilot and SCADA (OT domain)

The Shyft C++ library, together with the TLS-secured web API and Shyft DTSS queue mechanisms, provides the foundation for secure data exchange and controlled execution of operational plans.

This part of the system belongs to the Operational Technology (OT) domain, including SCADA and Autopilot components, as well as intraday planning systems that interface with external market platforms (e.g. mFRR/aFRR via ECP/EDX).

Within OT, systems are typically organized in security layers, where the most sensitive components (e.g. SCADA and control systems) are isolated from less trusted zones.

Shyft supports this layered model by:

• using DTSS queues as controlled interfaces between layers

• ensuring that communication is initiated from the inside (trusted zone) towards outer or less trusted zones

• avoiding the need for inbound connections or firewall openings into sensitive environments

Plan packages are received through these controlled channels and executed by Autopilot components, ensuring safe, auditable, and deterministic operation.

This design aligns with OT security principles and supports deployment in restricted or air-gapped environments.

Data-flow, Security, compliance and SLA, related notes

Notice that for the Day-ahead and the intraday sub-processes, the short term weather and inflow forecasts are an embedded part of the process implementation and context.

This is due to both the functional requirements, the uptime (SLA), and for the intraday process also the national laws and regulations. The latter require that the intraday system should continue working, regardless the state of ‘external systems’ as view from the intraday execution and control process.

Another detail to notice is the separation of trade/bid that occurs in the day-ahead process, that are of strategic or financial nature, vs. the detailed planning, FRR bid follow ups in the intraday process.

It is essential for the electric system stability that the balancing market bids/obligations for execution are kept as a logical unit.

E.g. if a producer promise 100 MW mFRR delivery 12:00->12:15, then its vital for the electric network stability that it gets activated when asked for by the TSO.

The TSO and the ENTSO-E ‘ECP/EDX’ protocol end-points are also highlighted, located within the processes, with text color red to indicate its significance. Again, the ‘pink’ color is used for the Day-ahead end-point, since this ECP/EDX end-point is used to relay the day-ahead capacity bids/closure information.

Shyft architecture to provide secured energy IT-infrastructure

Shyft Dtss provides the data exchange concept and functions that provides push/pull of sets of time-series between the process boundaries. It also provides Shyft Dtss queues with bridge from one site to another capabilities.

The design of the shyft time-series data flow also provides version stable protocols that allows different deployment scenario and pace for the different business processes.

Another important feature of Shyft data-flows is that it provides both push and pull, where the initiative direction of establishing communication is from the safer security zone. That feature allows firewall setup to close all incoming ports, and make the critical parts of our energy infrastructure be more resilient for attacks.