Installation

There are several different methods to get up and running with Shyft depending on your intended use. We categorize users into several categories:

  • users are those interested in using Shyft for hydrologic analysis. First time users should probably start with Binary installation with the sigbjorn conda channel to at least gain familiarity of the functionality of the framework.
  • contributers are those who will explore the Python api and may contribute back to the project. You may want to modify Python source code, so you should follow the Source Installation.
  • developers are those who are interested in the C++ core, and are interested in creating their own algorithms. Advance programming skills and familiarity is required. Developers will want to see both the Source Installation and Build Instructions documentation.

The reason to categorize users relates to the installation requirements. For users it is not necessary to build/compile the C++ core, and one can simply use available distributions. Contributors also don’t need to build the C++ core, but will want to clone the repository to have access to the Python source code – mostly for making modifications to orchestration.

Shyft is developed for both Unix-like and Windows operating systems, though we have a strong preference for linux.

System Requirements

Shyft is developed with Python and C++. Our choice of Python is Anaconda and we recommend using the conda package management system. It is fine to use Miniconda if you don’t want to install the full Anaconda Distribution. You can follow our Python Installation Guide for Shyft.

Environment Variables

There are a few environment variables we use with Shyft. If you do not plan to do a Source Installation of Shyft, these are not strictly required. If you know what you are doing, you can modify these to your liking, but this is our recommendation:

SHYFT_WORKSPACE
The main folder where you will in parallel clone the repositories and setup Shyft. You don’t need to work here, but you will at likely point your PYTHONPATH here, along with some other runtime variables. You should use this as your main build directory.
SHYFT_DEPENDENCIES_DIR
Optional, only if you know what you are doing, and would have a different working tree strucutre than the recommented relative-path oriented standard approach. Build Instructions for Shyft.
SHYFT_DATA
If you install prebuilt shyft, conda install -c sigbjorn shyft, -then you need to set this variable to run the demo-notebooks that resides in the shyft-doc/notebooks directory. It should point to the shyft-data repository in order to run the tests. Default values in the shyft code uses the shyft-data directory parallel to the shyft source code directory.

Minimal ‘Run’ Requirements

If you are not planning on building Shyft, then the requirements are:

  • 64-bit computer.
  • Windows or Linux.
    • Windows: ms c++ vs 2017 redist.
    • Linux: gcc-7 runtime libraries (e.g. libgcc), BLAS and LAPACK library runtimes
  • Python >= 3.6
  • Python (minimum) libraries to access shyft-core-api, time-series, dtss:
    • numpy , compatible with the shyft build/dependency requrement, like numpy 1.13, 1.14
  • Python (maximum) libraries for remote-access data-sources, netcdf, demo and notebooks:
    • pyyaml
    • netcdf4
    • gdal
    • matplotlib
    • requests
    • nose
    • coverage
    • pip
    • shapely
    • pyproj
    • jupyter

NOTE: You do not need administrative or root permissions to install Shyft if you select a user-writable install location.

Build Requirements

Significantly greater requirements exist if you intend to build the C++ core of Shyft. In addition to the Minimal ‘Run’ Requirements there are some basic requirements which we leave to the user to be sure are met, as these are quite typical:

  • A C++11 compiler (linux - gcc-7 or higher, windows -Visual Studio 2017 with updates)
  • The BLAS and LAPACK libraries (development packages)
  • A Python 3 (3.6 or higher) interpreter
  • The NumPy package (>= 1.8.0)
  • The netCDF4 package (>= 1.2.1)
  • The CMake building tool (3.8 or higher)
  • git version control

When shyft is checked out, the shyft/build_support directory contains build_dependencies.sh and win_build_dependencies.sh scripts that when executed, will download, compile and install all requirements locally into directory ~/projects/shyft_dependencies.

The linux version also downloads and install miniconda into the ~/projects/miniconda directory. Similar for the windows-version, but on windows it’s recommended to install miniconda/anaconda as a user/system maintained package.

CMake support used for building will search the relative path ../shyft_dependencies and will thus find needed packages, libraries that is needed. To ease development, setting LD_LIBRARY_PATH to include ~/projects/shyft_dependencies/lib helps the loader locating boost, dlib, and armadillo libraries.

On windows, we use system SHYFT_DEPENDENCIES environment variable to point to the shyft_dependencies, where all needed extra packages includes and libraries are located. To ease development and enable direct debugging in Visual studio, ensure to add %SHYFT_DEPENDENCIES%lib to the PATH settings. Also ensure to set BOOST_PYTHONHOME env. variable to point to your miniconda/anaconda root directory.

git clone https://github.com/statkraft/shyft

bash shyft/build_support/build_dependencies.sh

on windows:

bash shyft/build_support/win_build_dependencies.h

External Libraries

The Shyft C++ core utilizes several modern frameworks including: dlib, boost, and armadillo.

Instructions on building these are provided in Build Instructions.

Binary installation with the sigbjorn conda channel

If are not yet familiar with Python, we recommend seeing the Python Installation Guide documentation. The simplest way to get started, if you are familiar with conda is to use Sigbjorn’s channel:

conda  create -c sigbjorn -c conda-forge -n shyft python=3.6 pyyaml numpy netcdf4 gdal matplotlib requests nose coverage pip shapely pyproj jupyter pandas shyft scipy bokeh

Accessing pre-built binary releases

Ref to the anaconda.org/sigbjorn channel for prebuilt windows and linxu packages.

Source Installation

If you are interested in modifying the source code and contributing to the project, you will probably want to check out the repositories and install Shyft from source, in which case you’ll need to clone or fork the repository. If you are serious about participating in development, then it is best to fork the repo on github and start editing your own copy.

You are not able to push to the main Shyft repository directly. We follow a standard Git workflow. For more information about working with the Shyft source code, please see the following documents:

Once you are comfortable with your git configuration and python installation, the next steps
are to follow the Build Instructions.

See also

For Contributers it is important to note in the Build Instructions that we have a trick we use for those who don’t want to build the dependencies and C++ core.

Build Instructions

Warning

This portion of the documentation is rapidly changing. We are migrating documentation from github to these pages.

Developers will also do a Source Installation. However, once you have cloned the repositories, you’ll have extra steps to be sure you have all the Build Requirements in place and are able to build the build-dependencies before building the C++ core itself. Documentation for this is currently available at the Shyft repository. Follow the Developer Documentation.

A hack for contributers

We are currently working on a more elegant solution, but for the moment, for those who wish to work with the Python source code, but do not wish to build Shyft, we recommend simply installing the binaries, and then copying the appropriate binary files (e.g. .so or .pyd files) into the appropriate shyft path within the repository. Make sure you PYTHONPATH is pointing to the repository.

Installation Recipes

Warning

The following may not be up to date

A few installation recipes for specific platforms exist: