METSIM: Meteorology Simulator¶

MetSim is a meteorological simulator and forcing disaggregator for hydrologic modeling and climate applications. Metsim is based on MtClim and the preprocessor from version 4 of the VIC hydrologic model.

MetSim consists of 3 main modules that govern the operation of 3 major aspects of its operation:

1. Management of dataset preprocessing and IO

The MetSim object provides high level support for setting up jobs and infrastructure for running simulation/disaggregation steps. It is the main interface through which the other modules are accessed.

2. Simulation of daily meteorological forcings

The base implementation of the meteorological simulator is based off of the algorithms described in [1]. This component has been designed to be flexible in allowing for alternative implementations which may be specified during the setup of the MetSim object. The default implementation allows for the daily simulation of:

Mean daily temperature

Incoming shortwave radiation

Cloud cover fraction

Potential evapotranspiration

Vapor pressure

3. Disaggregation of daily simulation values to sub-daily timesteps

Daily data from given input or simulated via the forcings generation component of MetSim can be disaggregated down to sub-daily values at intervals specified in minutes (provided they divide evenly into 24 hours). The operation of these algorithms is also described in [1]. The variables estimated are:

Temperature

Vapor pressure

Relative and specific humidity

Air pressure

Cloud cover fraction

Longwave radiation

Shortwave radiation

Precipitation

Wind speed

For the “triangle” and “mix” methods of precipitation disaggregation, doumentation can be found here. This will eventually be superceded by a journal article that is currently in review [7].

If you don’t find what you’re looking for here, check out MetSim’s Github page.

Getting Started¶

A tutorial for running MetSim and working with input/output data can be run via binder here: https://github.com/UW-Hydro/MetSim-tutorial

Installation¶

MetSim itself is a pure Python package, but its dependencies are not. You should ensure that you have all of the required dependencies:

Python 3.5 or greater
xarray (0.10.9 or later)
pandas (0.19.0 or later)
numba (0.31.0 or later)
netCDF4
scipy

Then, install MetSim with pip or conda:

$ pip install metsim

or:

$ conda install -c conda-forge metsim

Alternatively, you can install MetSim directly from the source if you desire to:

$ git clone https://github.com/UW-Hydro/MetSim.git
$ cd MetSim
$ python setup.py install
$ py.test --verbose

Basic Usage¶

MetSim provides a simple command line interface which is primarily operated via configuration files. For more information about the options available to be set in the configuration files see the configuration page.

Once installed, MetSim can be used from the command line via:

usage: ms [-h] [-n NUM_WORKERS] [-s SCHEDULER] [-v] [--version] config

positional arguments:
config                Input configuration file

optional arguments:
-h, --help            show this help message and exit
-n NUM_WORKERS, --num_workers NUM_WORKERS
                        Parallel mode: number of processes to use
-s SCHEDULER, --scheduler SCHEDULER
                        Dask scheduler to use
-v, --verbose         Increase the verbosity of MetSim
--version             Name and version number

Bracketed flags are optional; -v activates verbose mode to print messages about the status of a run, and -n activates parallelism. The number given after the -n flag is the number of processes to run. A good rule of thumb is to use one less process than the number of processsors (or threads) that the machine you are running on has.

Warning

Users in environments where OpenMP is available may experience over-utilization of CPU resources, leading to lower performance. If you experience this issue try setting the OMP_NUM_THREADS environment variable to 1 before running MetSim.. This can be done in bash and similar shells by running export OMP_NUM_THREADS=1.

References¶

[1]	(1, 2) Bohn, T. J., B. Livneh, J. W. Oyler, S. W. Running, B. Nijssen, and D. P. Lettenmaier, 2013. Global evaluation of MTCLIM and related algorithms for forcing of ecological and hydrological models, Agricultural and Forest Meteorology, 176:38-49, doi:10.1016/j.agrformet.2013.03.003.

[2]	Bristow, K.L., and G.S. Campbell, 1984. On the relationship between incoming solar radiation and daily maximum and minimum temperature. Agricultural and Forest Meteorology, 31:159-166.

[3]	Running, S.W., R.R. Nemani, and R.D. Hungerford, 1987. Extrapolation of synoptic meteorological data in mountainous terrain and its use for simulating forest evaporation and photosynthesis. Canadian Journal of Forest Research, 17:472-483.

[4]	Glassy, J.M., and S.W. Running, 1994. Validating diurnal climatology of the MT-CLIM model across a climatic gradient in Oregon. Ecological Applications, 4(2):248-257.

[5]	Kimball, J.S., S.W. Running, and R. Nemani, 1997. An improved method for estimating surface humidity from daily minimum temperature. Agricultural and Forest Meteorology, 85:87-98.

[6]	Thornton, P.E., and S.W. Running, 1999. An improved algorithm for estimating incident daily solar radiation from measurements of temperature, humidity, and precipitation. Agricultural and Forest Meteorology, 93:211-228.

[7]	Bohn, T. J., K. M. Whitney, G. Mascaro, and E. R. Vivoni, 2019. A deterministic approach for approximating the diurnal cycle of precipitation for large-scale hydrological simulations. Journal of Hydrometeorology, 20(2):297-317. doi: 10.1175/JHM-D-18-0203.1.

Sitemap¶

Examples¶

Basics¶

Provided in the source are several examples that can help you to get started using MetSim. They are located in the examples directory. For demonstration here is an example YAML configuration file:

# This is an example of an input file for MetSim
# Overall configuration, specification of parameters and input/output
# paths goes in the "MetSim" section
MetSim:
    # Time step in minutes
    time_step: 30
    # Forcings begin here (year-month-day)
    start: 1950-1-1
    # Forcings end at this date (year-month-day)
    stop: 1950-1-31
    # Input and output directories
    forcing: './metsim/data/test.nc'
    domain: './metsim/data/tiny_domain.nc'
    state: './metsim/data/state_nc.nc'
    forcing_fmt: 'netcdf'
    in_format: 'netcdf'
    out_dir: './results'
    out_prefix: 'yaml_output'
    prec_type: 'triangle'
    utc_offset: True

out_vars:
    temp:
        out_name: 'airtemp'
        units: 'K'

    prec:
        out_name: 'pptrate'
        units: 'mm/s'

    shortwave:
        out_name: 'SWradAtm'

    spec_humid:
        out_name: 'spechum'

    air_pressure:
        out_name: 'airpres'
        units: 'kPa'

    wind:
        out_name: 'windspd'

chunks:
    lat: 3
    lon: 3

forcing_vars:
    # Format is metsim_name: input_name
    prec  : 'Prec'
    t_max : 'Tmax'
    t_min : 'Tmin'

state_vars:
    # Format is metsim_name: input_name
    prec  : 'Prec'
    t_max : 'Tmax'
    t_min : 'Tmin'

domain_vars:
    # Format is metsim_name: input_name
    lat  : 'lat'
    lon  : 'lon'
    mask : 'mask'
    elev : 'elev'
    t_pk : 't_pk'
    dur  : 'dur'

constant_vars:
    wind : 2.0

This is a minimal configuration file for MetSim. For a complete description of the input format see the configuration page.

To run this example from the command line, once you have installed MetSim, use the following command:

ms path/to/example.yaml --verbose

This will run MetSim and disaggregate to half-hourly data, and write out the results in a NetCDF file located in the directory specified under out_dir in the configuration file (here ./results). The addition of the --verbose flag provides some information back to you as MetSim runs. In the absence of this flag MetSim will quietly run in the background until finished, or some error has occurred.

Generating daily values¶

Daily values can be output by specifying a time_step of 1440 in the configuration file, such as the one shown in the previous section. This will prevent MetSim’s disaggregation routines from being run, and the results written out will be daily values.

Input Specifications¶

There are 3 required input data sources that must be specified in the configuration file or dictionary. Note that it is possible for a single file to be specified for all three sources, provided that it has all of the required data. For examples of the data see the tests/data directory within the MetSim code.

Input forcings¶

Specified as forcing in the configuration file. This can either be the path to a NetCDF file, or the path to a directory containing ASCII or binary data (in the VIC4 format). The input forcing data is used to provide the base forcing variables. The required variable data is minimum daily temperature, maximum daily temperature, and daily precipitation.

The variable names can be mapped via the configuration file in the forcing_vars section. For more information about how to set up your configuration file see the configuration page.

Domain file¶

Specified as domain in the configuration file. The domain file provides information about the domain MetSim is to be run over. It is required to be a NetCDF file. The domain requires the following variables to be valid:

1. mask: This provides information about which grid cells are valid to run MetSim on. Values that specify grid cells which should be processed are specified via a positive, finite number (one or greater). Cells which MetSim should ignore can be given as 0 or NaN.

It is important to ensure that all valid locations in mask have data in elev and any other variables. Failure to ensure this will result in errors during runtime.

2. elev: This provides elevation data (in m) used for calculation of solar geometry. It only needs to be given at sites which are marked to be processed via the mask variable.

The next two variables are only needed if prec_type = triangle or mix in the input file:

3. dur: This provides the climatological monthly storm event duration (in minutes) used for disaggregating daily precipitation according to the “triangle” method. Requires one value for each month (12).

4. t_pk: This provides the climatological monthly time to storm peak (in minutes starting from midnight) used for disaggregating daily precipitation to sub-daily time scales using the “triangle” method. Requires one value for each month (12).

For more information about the “triangle” method see this description. If you use this feature, please cite Bohn et al. (2019) as listed in the references.

A domain file for the CONUS+Mexico domain, at 0.0625 degree resolution, and containing dur and t_pk values, is available here.

State file¶

The state file provides information about the history of each of the grid cells to be processed. There are four required variables.

The first two are daily minimum and daily maximum temperatures for the 90 days preceeding the start date specified in the configuration file. They should be specified as t_min and t_max respectively. Similarly precipitation should be given as prec. These variables are used to generate seasonal averages which are used in the calculation of shortwave and longwave radiation.

Output Specifications¶

Attention

The time coordinate in MetSim’s output is local to the location of each cell unless the utc_offset is set to True! This means that for a single time slice in the NetCDF file all locations along a parallel (same latitude) will have the same solar geometry at that time.

The output variables that are available are dependent on the time step being used. There are two cases:

Daily Output¶

When time_step is set to 1440 in the configuration file, daily values are generated. The following variables are available for output at a daily time step:

t_min : Minimum temperature (also a required input value) (C)
t_max : Maximum temperature (also a required input value) (C)
prec : Precipitation (also a required input value) (mm/day)
vapor_pressure : Vapor pressure (Pa)
shortwave : Shortwave radiation (W/m^2)
tskc : Cloud cover fraction
pet : Potential evapotranpiration (mm/day)
wind : Wind speed (only if given as an input) (m/s)

Sub-daily Output¶

If disaggregation to shorter time steps is configured, the data is passed from the meteorology simulation module to the disaggregation module. MetSim implements several variable-specific disaggregation routines. Here we briefly mention the disaggregation procedures for completeness, but no substantial changes were made to the earlier algorithms.

Shortwave is disaggregated by multiplying the total daily shortwave by the fraction of radiation received in a given timestep (provided by the solar geometry module). This calculation is corrected for cloud cover by assuming constant transmissivity throughout the day (which is calculated in the meteorological simulation module). Temperature is disaggregated by estimating the time at which the daily maximum and daily minimum temperatures occur. These are chosen so that the daily minimum temperature occurs at sunrise and the daily maximum temperature occurs at a fixed time during the day (which is configurable by the user as a parameter in the configuration file if desired). Then a Hermite polynomial interpolation is used to obtain the full temperature timeseries at sub-daily time steps. Vapor pressure is disaggregated by linearly interpolating between the saturation vapor pressure values calculated based on the daily minimum temperature and that are assumed to occur at the time of the daily minimum temperature. An additional correction is made to ensure that the vapor pressure at any given time step does not exceed the saturation vapor pressure, which is calculated directly from the disaggregated temperature timeseries. Air pressure is disaggregated by using the disaggregated temperature as well as the elevation data provided by the domain file. Both specific and relative humidity are then disaggregated using the disaggregated temperature and air pressure time series. If provided, wind speed is disaggregated, but is assumed to be constant throughout the day.

As part of the model configuration, the user can select from a number of different algorithms to estimate longwave radiation. Sub-daily values are calculated with the selected method using the disaggregated values for vapor pressure and temperature.

Precipitation can be disaggregated in one of two ways. The first and simplest way is to evenly spread the daily precipitation across the sub-daily time steps. The second method requires two additional parameters to be specified in the domain file to represent the average precipitation duration and the time of peak precipitation for each cell. The method then disaggregates precipitation by constructing a triangular kernel with total area equal to the daily precipitation centered at the time which is specified as the time of peak precipitation.

The following variables (and name conventions) are allowed to be specified for subdaily output:

temp : Temperature (C)
prec : Precipitation (mm/timestep)
shortwave : Shortwave radiation (W/m^2)
vapor_pressure : Vapor pressure (Pa)
air_pressure : Air pressure (kPa)
rel_humid : Relative humidity
spec_humid : Specific humidity
longwave : Longwave radiation (W/m^2)
tsck : Cloud cover fraction
wind : Wind speed (only if given as an input) (m/s)

Configuration Specifications¶

This page documents the various options and parameters that can be set in the configuration file. An example configuration file can be found on the examples page.

MetSim Section¶

Required Variables

time_step :: int: The timestep to disaggregate in minutes. If given as 1440 (number of minutes in a day), no disaggregation will occur. This value must divide 1440 evenly.

start :: str: The time to start simulation given in the format yyyy/mm/dd

stop :: str: The time to end simulation given in the format yyyy/mm/dd.

forcing :: path: The path to the input forcing file(s). See the section on __forcing_vars__ for more details.

domain :: path: The path to the input domain file. See the section on __domain_vars__ for more details.

state :: path: The path to the input state file.

out_dir :: path: The location to write output to. If this path doesn’t exist, it will be created.

forcing_fmt :: str: A string representing the type of input files specified in the forcing entry. Can be one of the following: ascii, binary, netcdf, or data.

Optional Variables

method ::str: The method to use for estimation of meteorological quantities. This can be either mtclim to estimate missing variables or passthrough if some of the meteorological variables have already been estimated (for example, by DayMet, PRISM, or GridMET). Defaults to mtclim.

out_prefix :: str: The output file base name. Defaults to forcing.

out_precision :: str: Precision to use when writing output. Defaults to f8. Can be either f4 or f8.

time_grouper :: str: Whether to chunk up the timeseries into pieces for processing. This option is useful to set for when you are limited on memory. Each chunk of output is written as {out_prefix}_{date_range} when active. Any valid pandas.TimeGrouper string may be used (e.g. use ‘10AS’ for 10 year chunks).

verbose :: bool: Whether to print output to stdout. Should be set using the -v flag for command line usage. This can be set for scripting purposes, if desired. Set to 1 to print output; defaults to 0.

sw_prec_thresh :: float: Minimum precipitation threshold to take into account when simulating incoming shortwave radiation. Defaults to 0.

rain_scalar :: float: Scale factor for calculation of cloudy sky transmittance. Defaults to 0.75, range should be between 0 and 1.

utc_offset :: bool: Whether to use UTC timecode offsets for shifting timeseries. Without this option all times should be considered local to the gridcell being processed. Large domain runs probably want to set this option to True.

lw_cloud :: str: Type of cloud correction to longwave radiation to apply. Can be either DEFAULT or CLOUD_DEARDORFF. Defaults to CLOUD_DEARDORFF. Capitalization does not matter.

lw_type :: str: Type of longwave radiation parameterization to apply. Can be one of the following: DEFAULT, TVA, ANDERSON, BRUTSAERT, SATTERLUND, IDSO, or PRATA. Defaults to PRATA. Capitalization does not matter.

tdew_tol :: float: Convergence criteria for the iterative calculation of dewpoint temperature in MtClim. Defaults to 1e-6.

tmax_daylength_fraction :: float : Weight for calculation of time of maximum daily temperature. Must be between 0 and 1. Defaults to 0.67.

tday_coef :: float: Scale factor for calculation of daily mean temperature. Defaults to 0.45, range should be between 0 and 1.

lapse_rate :: float: Used to calculate atmospheric pressure. Defaults to 0.0065 K/m.

out_vars :: list : List of variables to write to output. Should be a list containing valid variables. The list of valid variables is dependent on which simulation method is used, as well as whether disaggregation is used. Defaults to ['temp', 'prec', 'shortwave', 'longwave', 'vapor_pressure', 'red_humid'].

prec_type :: str: Type of precipitation disaggregation method to use. Can be one of the following: uniform, triangle, or mix. Defaults to uniform. Capitalization does not matter. Under uniform method, precipitation is disaggregated by dividing uniformly over all sub-daily timesteps. Under triangle the “triangle” method is employed whereby daily precipitation is distributed assuming an isosceles triangle shape with peak and width determined from two domain variables, t_pk and dur. Under mix, the “uniform” method is used on days when t_min < 0 C, and “triangle” is used on all other days; this hybrid method retains the improved accuracy of “triangle” in terms of warm season runoff but avoids the biases in snow accumulation that the “triangle” method sometimes yields due to fixed event timing within the diurnal cycle of temperature. A domain file for the CONUS+Mexico domain, containing the dur and t_pk parameters is available at: <https://zenodo.org/record/1402223#.XEI-mM2IZPY>. For more information about the “triangle” method see PtriangleMethod.pdf.

For more information about input and output variables see the Input Specifications page.

# Comments begin with hashtags
# The first non-comment line must begin with the following:
Metsim:
    time_step: int
    start: YYYY-MM-DD
    stop: YYYY-MM-DD

    # Paths to input files
    forcing: str
    domain: str
    state: str

    # Output file specification
    out_dir: str
    out_prefix: str

    # Algorithmic controls
    utc_offset: bool
    prec_type: str
    lw_type: str
    lw_cloud: str

chunks section¶

The chunks section describes how parallel computation should be grouped in space. For example, to parallelize over 10 by 10 chunks of latitude and longitude (with netcdf dimensions named lat and lon, respectively) you would use:

chunks:
    lat: 10
    lon: 10

Alternatively, for an HRU based run chunked into 50 element jobs you would use:

chunks:
    hru: 50

As a general rule of thumb, try to evenly chunk the domain in such a way that the number of jobs to run is some multiple of the number of processors you wish to run on.

forcing_vars and state_vars section¶

The forcing_vars and state_vars sections are where you can specify which variables are in your input data, and the corresponding symbols which MetSim will recognize. The in_vars section for acts as a mapping between the variable names in the input dataset to the variable names expected by MetSim. The format is given as metsim_varname: netcdf_varname. The minimum required variables given have metsim_varnames corresponding to t_min, t_max, and prec; these variable names correspond to minimum daily temperature (Celcius), maximum daily temperature (Celcius), and precipitation (mm/day).

domain_vars section¶

The domain_vars section is where information about the domain file is given. Since the domain file is given as a NetCDF file this section has a similar format to that of the NetCDF input file format described above. That is, entries should be of the form metsim_varname = netcdfvarname. The minimum required variables have metsim_varnames corresponding to lat, lon, mask, and elev; these variable names correspond to latitude, longitude, a mask of valid cells in the domain, and the elevation given in meters. If prec_type = triangle or mix, two additonal variables are required including dur and t_pk for disaggregating daily precipitation according to the “triangle” method.

out_vars section¶

The out_vars section is where you can specify the output variables that you want to include. There are two formats for this section. The first is the old format, which we provide backwards compatibility for. You simply specify in the top level [MetSim] section a list of output variables with the names used by MetSim. They will be written out with the same names used internally. Available options are dependent on whether daily or subdaily output is being generated. Options for daily output are:

pet
shortwave
t_max
t_min
tskc

Options for subdaily output are:

prec

shortwave

longwave

temp

vapor_pressure

air_pressure

tskc

rel_humid

spec_humid

wind

The syntax for output specification is as follows:

out_vars:
    metsim_varname:
        out_name: str
        units: str

unit conversions¶

The out_vars section allows for specification of some simple unit conversions for MetSim output. The allowed options are as follows (invalid options will revert to the default after issuing a warning):

prec - mm timestep-1 (default) - mm s-1 - mm h-1

pet (daily output only) - mm timestep-1 (default) - mm s-1 - mm h-1

t_max (daily output only) - C (default) - K

t_min (daily output only) - C (default) - K

temp - C (default) - K

vapor_pressure - Pa (default) - hPa - KPa

air_pressure - kPa (default) - hPa - Pa

tskc (cloud fraction) - fraction (default) - %

rel_humid - % (default) - fraction

constant_vars section¶

The constant_vars section is optional and allows you to set some of the forcing inputs to a constant value. The specification simply consists of entries of the form metsim_varname: value, where value is a number that can be converted to a double. There can only be one entry per line. If the metsim_varname corresponds to an entry that is already in the forcing_vars section, then the constant value will take precedence. In the current implementation there must be at least one non-constant entry in forcings_vars (i.e. at least one entry that is not also in constant_vars).

For example:

constant_vars:
    wind: 2.0

will result in a constant wind field in the output file. In this case wind does not need to be specified in the forcing_vars section. If it was, it will still be set to a constant value of 2 m/s.

Similarly:

constant_vars:
    t_max = 30.0
    t_min = 10.0

will result in output with a diurnal cycle in which the temperature varies at all locations between 10C and 30C. However, all estimation and disaggregation routines are still evaluated, with constant t_max and t_min as input.

INI Configuration Specifications¶

Warning

This page provides documentation only for backwards compatibility. New users should use the YAML configuration file format, as it is more flexible and is where new features will be added. The INI format will be depreciated in version 3 of MetSim.

This page documents the various options and parameters that can be set in the configuration file.