GridStat: Apply separate climatologies for forecast and observations

model_applications/s2s/GridStat_fcstSFSBaseline_obsGHCNCAMS_climoSFSandGHCNCAMS_separateClimo.conf

Scientific Objective

During forecast verification, it is important to keep perspective of the model’s performance relative to a reference forecast. In most use cases, this is accomplished with a climatology comparison; however, the climatologies applied are often from the observational dataset and do not provide as much insight as a climatology based on the forecast itself. This use case shows a basic setup using ensemble-specific climatology and applying that climatology to the forecast, as well as a separate climatology for the observational dataset. This use case will be an important template for those hoping to utilize forecast climatologies within METplus.

Version Added

METplus version 6.1

Datasets

Forecast: Seasonal Forecast System (SFS), 2m temperature

Observation: Global Historical Climatology Network version 2 and the Climate Anomaly Monitoring System (GHCN_CAMS), 2m temperature

Climatology: Forecast is SFS-specific, Observation is GHCN_CAMS specific. Both made outside of METplus and provided via CPC

Location: All of the input data required for this use case can be found in a sample data tarball. Each use case category will have one or more sample data tarballs. It is only necessary to download the tarball with the use case’s dataset and not the entire collection of sample data. Click here to access the METplus releases page and download sample data for the appropriate release: https://github.com/dtcenter/METplus/releases This tarball should be unpacked into the directory that you will set the value of INPUT_BASE. See Running METplus section for more information.

METplus Components

This use case calls GenEnsProd, followed by GridStat. Within GenEnsProd, an ensemble mean is calculated and output. This output is then read in by GridStat for statistical analysis.

METplus Workflow

Beginning time (INIT_BEG): 1994-11-01

End time (INIT_END): 2023-12-01

Increment between beginning and end times (INIT_INCREMENT): 1 year

Sequence of forecast leads to process (LEAD_SEQ): 1 month

With an increment of 1 year and a lead of 1 month, all December 1st’s from 1994 to 2023 are processed for a total of 30 years. For each loop of GenEnsProd, a yearly ensemble mean output file is created. There are 11 ensemble members in total. Afterwards, GridStat will repeat the time loop, using the ensemble mean from GenEnsProd as the forecast input for each year. This use case utilizes separate climatologies for the forecast and observation datasets during verification and outputs the netCDF output for the raw and climatology fields, as well as the CNT line type.

METplus Configuration

METplus first loads all of the configuration files found in parm/metplus_config, then it loads any configuration files passed to METplus via the command line, i.e. parm/use_cases/model_applications/s2s/GridStat_fcstSFSBaseline_obsGHCNCAMS_climoSFSandGHCNCAMS_separateClimo.conf

[config]

# Documentation for this use case can be found at
# https://metplus.readthedocs.io/en/latest/generated/model_applications/s2s/GridStat_fcstSFSBaseline_obsGHCNCAMS_climoSFSandGHCNCAMS_separateClimo.html

# For additional information, please see the METplus Users Guide.
# https://metplus.readthedocs.io/en/latest/Users_Guide

###
# Processes to run
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#process-list
###

PROCESS_LIST = GenEnsProd, GridStat

###
# Time Info
# LOOP_BY options are INIT, VALID, RETRO, and REALTIME
# If set to INIT or RETRO:
#   INIT_TIME_FMT, INIT_BEG, INIT_END, and INIT_INCREMENT must also be set
# If set to VALID or REALTIME:
#   VALID_TIME_FMT, VALID_BEG, VALID_END, and VALID_INCREMENT must also be set
# LEAD_SEQ is the list of forecast leads to process
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#timing-control
###

LOOP_BY = INIT
INIT_TIME_FMT = %Y%m%d
INIT_BEG=19941101
INIT_END=20231201
INIT_INCREMENT = 1Y

LEAD_SEQ = 1m


###
# File I/O
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#directory-and-filename-template-info
###

FCST_GRID_STAT_INPUT_DIR = {GEN_ENS_PROD_OUTPUT_DIR}
FCST_GRID_STAT_INPUT_TEMPLATE = {GEN_ENS_PROD_OUTPUT_TEMPLATE}

OBS_GRID_STAT_INPUT_DIR = {INPUT_BASE}/model_applications/s2s/GridStat_fcstSFSBaseline_obsGHCNCAMS_climoSFSandGHCNCAMS_separateClimo 
OBS_GRID_STAT_INPUT_TEMPLATE = ghcn_cams.1x1.1994-2023.mon.nc

GRID_STAT_FCST_CLIMO_MEAN_FILE_NAME = {INPUT_BASE}/model_applications/s2s/GridStat_fcstSFSBaseline_obsGHCNCAMS_climoSFSandGHCNCAMS_separateClimo/Monthly_Climatology_1994_2023_SFS_Baseline_tmp2m_110100.nc
GRID_STAT_OBS_CLIMO_MEAN_FILE_NAME = {INPUT_BASE}/model_applications/s2s/GridStat_fcstSFSBaseline_obsGHCNCAMS_climoSFSandGHCNCAMS_separateClimo/Monthly_Climatology_1994_2023_Obs_tmp2m_110100.nc

GRID_STAT_FCST_CLIMO_STDEV_FILE_NAME = {INPUT_BASE}/model_applications/s2s/GridStat_fcstSFSBaseline_obsGHCNCAMS_climoSFSandGHCNCAMS_separateClimo/Monthly_StandardDeviation_1994_2023_SFS_Baseline_tmp2m_110100.nc
GRID_STAT_OBS_CLIMO_STDEV_FILE_NAME = {INPUT_BASE}/model_applications/s2s/GridStat_fcstSFSBaseline_obsGHCNCAMS_climoSFSandGHCNCAMS_separateClimo/Monthly_StandardDeviation_1994_2023_Obs_tmp2m_110100.nc

GRID_STAT_OUTPUT_DIR = {OUTPUT_BASE}/model_applications/s2s/GridStat_fcstSFSBaseline_obsGHCNCAMS_climoSFSandGHCNCAMS_separateClimo/GridStat
GRID_STAT_OUTPUT_TEMPLATE = {valid?fmt=%Y%m%d}


GEN_ENS_PROD_INPUT_DIR = {INPUT_BASE}/model_applications/s2s/GridStat_fcstSFSBaseline_obsGHCNCAMS_climoSFSandGHCNCAMS_separateClimo
GEN_ENS_PROD_INPUT_TEMPLATE = SFS_Baseline.tmp2m.{init?fmt=%Y%m}.fcst.nc

GEN_ENS_PROD_N_MEMBERS = 11

GEN_ENS_PROD_OUTPUT_DIR = {OUTPUT_BASE}/model_applications/s2s/GridStat_fcstSFSBaseline_obsGHCNCAMS_climoSFSandGHCNCAMS_separateClimo/GEP
GEN_ENS_PROD_OUTPUT_TEMPLATE = gen_ens_prod_{init?fmt=%Y%m}_ens.nc


###
# Field Info
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#field-info
###

MODEL = SFS
OBTYPE = GHCN_CAMS

GRID_STAT_ONCE_PER_FIELD = False

FCST_IS_PROB = False

FCST_VAR1_NAME = fcst_0_{valid?fmt=%Y%m%d}_000000_all_all_ENS_MEAN
FCST_VAR1_LEVELS = "(*,*)"
FCST_VAR1_THRESH = 

OBS_VAR1_NAME = tmp2m
OBS_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},*,*)"
OBS_VAR1_THRESH = 


ENS_VAR1_NAME = fcst
ENS_VAR1_LEVELS = "(MET_ENS_MEMBER_ID,{valid?fmt=%Y%m%d_%H%M%S},*,*)"
ENS_VAR1_OPTIONS = convert(x)=K_to_C(x);
ENS_FILE_TYPE = NETCDF_NCCF

###
# GridStat Settings
# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#gridstat
###


GRID_STAT_CONFIG_FILE = {PARM_BASE}/met_config/GridStatConfig_wrapped

OBS_GRID_STAT_FILE_TYPE = NETCDF_NCCF

GRID_STAT_REGRID_TO_GRID = OBS

GRID_STAT_DESC = NA

GRID_STAT_OUTPUT_PREFIX = {MODEL}_verfyingWith_{OBTYPE}

GRID_STAT_FCST_CLIMO_MEAN_VAR1_NAME = clim
GRID_STAT_FCST_CLIMO_MEAN_FILE_TYPE = NETCDF_NCCF
GRID_STAT_FCST_CLIMO_MEAN_VAR1_LEVELS = "(@20001201,*,*)"
GRID_STAT_FCST_CLIMO_MEAN_REGRID_METHOD = NEAREST 
GRID_STAT_FCST_CLIMO_MEAN_REGRID_WIDTH = 1
GRID_STAT_FCST_CLIMO_MEAN_REGRID_VLD_THRESH = 0.5
GRID_STAT_FCST_CLIMO_MEAN_TIME_INTERP_METHOD = DW_MEAN
GRID_STAT_FCST_CLIMO_MEAN_DAY_INTERVAL = 31
GRID_STAT_FCST_CLIMO_MEAN_HOUR_INTERVAL = NA

GRID_STAT_FCST_CLIMO_STDEV_VAR1_NAME = stddev
GRID_STAT_FCST_CLIMO_STDEV_FILE_TYPE = NETCDF_NCCF
GRID_STAT_FCST_CLIMO_STDEV_VAR1_LEVELS = "(@20001201,*,*)"
GRID_STAT_FCST_CLIMO_STDEV_REGRID_METHOD = NEAREST
GRID_STAT_FCST_CLIMO_STDEV_REGRID_WIDTH = 1
GRID_STAT_FCST_CLIMO_STDEV_REGRID_VLD_THRESH = 0.5
GRID_STAT_FCST_CLIMO_STDEV_TIME_INTERP_METHOD = DW_MEAN
GRID_STAT_FCST_CLIMO_STDEV_DAY_INTERVAL = 31
GRID_STAT_FCST_CLIMO_STDEV_HOUR_INTERVAL = NA

GRID_STAT_OBS_CLIMO_MEAN_VAR1_NAME = clim
GRID_STAT_OBS_CLIMO_MEAN_FILE_TYPE = NETCDF_NCCF
GRID_STAT_OBS_CLIMO_MEAN_VAR1_LEVELS = "(@20001201,*,*)"
GRID_STAT_OBS_CLIMO_MEAN_REGRID_METHOD = NEAREST
GRID_STAT_OBS_CLIMO_MEAN_REGRID_WIDTH = 1
GRID_STAT_OBS_CLIMO_MEAN_REGRID_VLD_THRESH = 0.5
GRID_STAT_OBS_CLIMO_MEAN_TIME_INTERP_METHOD = DW_MEAN
GRID_STAT_OBS_CLIMO_MEAN_DAY_INTERVAL = 31
GRID_STAT_OBS_CLIMO_MEAN_HOUR_INTERVAL = NA

GRID_STAT_OBS_CLIMO_STDEV_VAR1_NAME = clim
GRID_STAT_OBS_CLIMO_STDEV_FILE_TYPE = NETCDF_NCCF
GRID_STAT_OBS_CLIMO_STDEV_VAR1_LEVELS = "(@20001201,*,*)"
GRID_STAT_OBS_CLIMO_STDEV_REGRID_METHOD = NEAREST
GRID_STAT_OBS_CLIMO_STDEV_REGRID_WIDTH = 1
GRID_STAT_OBS_CLIMO_STDEV_REGRID_VLD_THRESH = 0.5
GRID_STAT_OBS_CLIMO_STDEV_TIME_INTERP_METHOD = DW_MEAN
GRID_STAT_OBS_CLIMO_STDEV_DAY_INTERVAL = 31
GRID_STAT_OBS_CLIMO_STDEV_HOUR_INTERVAL = NA

GRID_STAT_OUTPUT_FLAG_CNT = STAT

GRID_STAT_NC_PAIRS_FLAG_LATLON = TRUE
GRID_STAT_NC_PAIRS_FLAG_RAW = TRUE
GRID_STAT_NC_PAIRS_FLAG_CLIMO = TRUE

###
# GenEnsProd Settings
# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#genensprod
###


GEN_ENS_PROD_ENS_THRESH = 0.3
GEN_ENS_PROD_VLD_THRESH = 0.3

GEN_ENS_PROD_ENSEMBLE_FLAG_LATLON = TRUE
GEN_ENS_PROD_ENSEMBLE_FLAG_MEAN = TRUE

GEN_ENS_PROD_ENS_MEMBER_IDS = 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

MET Configuration

METplus sets environment variables based on user settings in the METplus configuration file. See How METplus controls MET config file settings for more details.

YOU SHOULD NOT SET ANY OF THESE ENVIRONMENT VARIABLES YOURSELF! THEY WILL BE OVERWRITTEN BY METPLUS WHEN IT CALLS THE MET TOOLS!

If there is a setting in the MET configuration file that is currently not supported by METplus you’d like to control, please refer to: Overriding Unsupported MET config file settings

GenEnsProdConfig_wrapped

////////////////////////////////////////////////////////////////////////////////
//
// Gen-Ens-Prod configuration file.
//
// For additional information, please see the MET Users Guide.
//
////////////////////////////////////////////////////////////////////////////////

//
// Output model name to be written
//
//model =
${METPLUS_MODEL}

//
// Output description to be written
// May be set separately in each "obs.field" entry
//
//desc =
${METPLUS_DESC}

////////////////////////////////////////////////////////////////////////////////

//
// Verification grid
// May be set separately in each "field" entry
//
//regrid = {
${METPLUS_REGRID_DICT}

////////////////////////////////////////////////////////////////////////////////

//
// May be set separately in each "field" entry
//
//censor_thresh =
${METPLUS_CENSOR_THRESH}

//censor_val    =
${METPLUS_CENSOR_VAL}

//normalize =
${METPLUS_NORMALIZE}

//cat_thresh    =
${METPLUS_CAT_THRESH}

//nc_var_str    =
${METPLUS_NC_VAR_STR}

//
// Ensemble fields to be processed
//
ens = {
   //file_type =
   ${METPLUS_ENS_FILE_TYPE}

   //ens_thresh =
   ${METPLUS_ENS_THRESH}

   //vld_thresh =
   ${METPLUS_VLD_THRESH}

   //field =
   ${METPLUS_ENS_FIELD}

   ${METPLUS_ENS_CLIMO_MEAN_DICT}
   ${METPLUS_ENS_CLIMO_STDEV_DICT}

}

//ens_member_ids =
${METPLUS_ENS_MEMBER_IDS}

//control_id =
${METPLUS_CONTROL_ID}


////////////////////////////////////////////////////////////////////////////////

//
// Neighborhood ensemble probabilities
//
//nbrhd_prob = {
${METPLUS_NBRHD_PROB_DICT}

//
// NMEP smoothing methods
//
//nmep_smooth = {
${METPLUS_NMEP_SMOOTH_DICT}

////////////////////////////////////////////////////////////////////////////////

//
// Climatology data
//
//climo_mean = {
${METPLUS_CLIMO_MEAN_DICT}

//climo_stdev = {
${METPLUS_CLIMO_STDEV_DICT}

////////////////////////////////////////////////////////////////////////////////

//
// Ensemble product output types
// May be set separately in each "ens.field" entry
//
//ensemble_flag = {
${METPLUS_ENSEMBLE_FLAG_DICT}

////////////////////////////////////////////////////////////////////////////////

//version = "V10.1.0";

////////////////////////////////////////////////////////////////////////////////

tmp_dir = "${MET_TMP_DIR}";

${METPLUS_MET_CONFIG_OVERRIDES}

GridStatConfig_wrapped

////////////////////////////////////////////////////////////////////////////////
//
// Grid-Stat configuration file.
//
// For additional information, see the MET_BASE/config/README file.
//
////////////////////////////////////////////////////////////////////////////////

//
// Output model name to be written
//
// model =
${METPLUS_MODEL}

//
// Output description to be written
// May be set separately in each "obs.field" entry
//
// desc =
${METPLUS_DESC}

//
// Output observation type to be written
//
// obtype =
${METPLUS_OBTYPE}

////////////////////////////////////////////////////////////////////////////////

//
// Verification grid
//
// regrid = {
${METPLUS_REGRID_DICT}

////////////////////////////////////////////////////////////////////////////////

//censor_thresh =
${METPLUS_CENSOR_THRESH}
//censor_val =
${METPLUS_CENSOR_VAL}
//cat_thresh =
${METPLUS_CAT_THRESH}
cnt_thresh  	 = [ NA ];
cnt_logic   	 = UNION;
wind_thresh 	 = [ NA ];
wind_logic  	 = UNION;
eclv_points      = 0.05;
//nc_pairs_var_name =
${METPLUS_NC_PAIRS_VAR_NAME}
nc_pairs_var_suffix = "";
//hss_ec_value =
${METPLUS_HSS_EC_VALUE}

rank_corr_flag   = FALSE;

//
// Forecast and observation fields to be verified
//
fcst = {
  ${METPLUS_FCST_FILE_TYPE}
  ${METPLUS_FCST_FIELD}
  ${METPLUS_FCST_CLIMO_MEAN_DICT}
  ${METPLUS_FCST_CLIMO_STDEV_DICT}
}
obs = {
  ${METPLUS_OBS_FILE_TYPE}
  ${METPLUS_OBS_FIELD}
  ${METPLUS_OBS_CLIMO_MEAN_DICT}
  ${METPLUS_OBS_CLIMO_STDEV_DICT}
}

////////////////////////////////////////////////////////////////////////////////

//
// Climatology mean data
//
//climo_mean = {
${METPLUS_CLIMO_MEAN_DICT}


//climo_stdev = {
${METPLUS_CLIMO_STDEV_DICT}

//
// May be set separately in each "obs.field" entry
//
//climo_cdf = {
${METPLUS_CLIMO_CDF_DICT}

////////////////////////////////////////////////////////////////////////////////

//
// Verification masking regions
//
// mask = {
${METPLUS_MASK_DICT}

////////////////////////////////////////////////////////////////////////////////

//
// Confidence interval settings
//
ci_alpha  = [ 0.05 ];

boot = {
   interval = PCTILE;
   rep_prop = 1.0;
   n_rep    = 0;
   rng      = "mt19937";
   seed     = "";
}

////////////////////////////////////////////////////////////////////////////////

//
// Data smoothing methods
//
//interp = {
${METPLUS_INTERP_DICT}

////////////////////////////////////////////////////////////////////////////////

//
// Neighborhood methods
//
nbrhd = {
   field      = BOTH;
   // shape =
   ${METPLUS_NBRHD_SHAPE}
   // width =
   ${METPLUS_NBRHD_WIDTH}
   // cov_thresh =
   ${METPLUS_NBRHD_COV_THRESH}
   vld_thresh = 1.0;
}

////////////////////////////////////////////////////////////////////////////////

//
// Fourier decomposition
// May be set separately in each "obs.field" entry
//
//fourier = {
${METPLUS_FOURIER_DICT}

////////////////////////////////////////////////////////////////////////////////

//
// Gradient statistics
// May be set separately in each "obs.field" entry
//
//gradient = {
${METPLUS_GRADIENT_DICT}

////////////////////////////////////////////////////////////////////////////////

//
// Distance Map statistics
// May be set separately in each "obs.field" entry
//
//distance_map = {
${METPLUS_DISTANCE_MAP_DICT}


////////////////////////////////////////////////////////////////////////////////
// Threshold for SEEPS p1 (Probability of being dry)

//seeps_p1_thresh =
${METPLUS_SEEPS_P1_THRESH}

////////////////////////////////////////////////////////////////////////////////

//
// Statistical output types
//
//output_flag = {
${METPLUS_OUTPUT_FLAG_DICT}

//
// NetCDF matched pairs output file
// May be set separately in each "obs.field" entry
//
// nc_pairs_flag = {
${METPLUS_NC_PAIRS_FLAG_DICT}

////////////////////////////////////////////////////////////////////////////////

//ugrid_dataset =
${METPLUS_UGRID_DATASET}

//ugrid_max_distance_km =
${METPLUS_UGRID_MAX_DISTANCE_KM}

//ugrid_coordinates_file =
${METPLUS_UGRID_COORDINATES_FILE}

////////////////////////////////////////////////////////////////////////////////

//grid_weight_flag =
${METPLUS_GRID_WEIGHT_FLAG}

tmp_dir = "${MET_TMP_DIR}";

// output_prefix =
${METPLUS_OUTPUT_PREFIX}

////////////////////////////////////////////////////////////////////////////////

${METPLUS_TIME_OFFSET_WARNING}
${METPLUS_MET_CONFIG_OVERRIDES}

Python Embedding

There is no Python Embedding used in this use case.

User Scripting

There is no User Scripting used in this use case.

Running METplus

Pass the use case configuration file to the run_metplus.py script along with any user-specific system configuration files if desired:

run_metplus.py /path/to/METplus/parm/use_cases/model_applications/s2s/GridStat_fcstSFSBaseline_obsGHCNCAMS_climoSFSandGHCNCAMS_separateClimo.conf /path/to/user_system.conf

See Running METplus for more information.

Expected Output

A successful run will output the following both to the screen and to the logfile:

INFO: METplus has successfully finished running.

Refer to the value set for OUTPUT_BASE to find where the output data was generated. Output for this use case will be found in {OUTPUT_BASE}/model_applications/s2s/GridStat_fcstSFSBaseline_obsGHCNCAMS_climoSFSandGHCNCAMS_separateClimo and will contain two separate directories. The first directory is GEP and will contain 30 ensemble mean files from GenEnsProd output. They should have the following naming convention:

* gen_ens_prod_YYYY11_ens.nc

where each date corresponds to the initialization year and month of the model. These files will contain the field info fcst_0_YYYY1201_000000_all_all_ENS_MEAN, where YYYY corresponds to the valid year of the forecast. The second directory named GridStat will have 30 subdirectories corresponding to the valid time for each year in the form of:

* YYYY1201

Each directory will contain two files following the convention below:

* grid_stat_SFS_verfyingWith_GHCN_CAMS_000000L_YYYY1201_000000V.stat
* grid_stat_SFS_verfyingWith_GHCN_CAMS_000000L_YYYY1201_000000V_pairs.nc

The stat file will only contain the CNT line type, while the netCDF file will have the following entries:

* FCST_fcst_0_YYYY1201_000000_all_all_FULL
* OBS_tmp2m_YYYY1201_000000_all_all_FULL
* DIFF_fcst_0_YYYY1201_000000_all_all_tmp2m_YYYY1201_000000_all_all_FULL
* FCST_CLIMO_MEAN_tmp2m_YYYY1201_000000_all_all_FULL
* FCST_CLIMO_STDEV_tmp2m_YYYY1201_000000_all_all_FULL
* OBS_CLIMO_MEAN_tmp2m_YYYY1201_000000_all_all_FULL
* OBS_CLIMO_STDEV_tmp2m_YYYY1201_000000_all_all_FULL
* OBS_CLIMO_CDF_tmp2m_YYYY1201_000000_all_all_FULL

Keywords

Note

• GenEnsProdToolUseCase

• GridStatToolUseCase

• ClimatologyUseCase

• S2SAppUseCase

• NetCDFFileUseCase

Navigate to the METplus Quick Search for Use Cases page to discover other similar use cases.