Point2Grid: Calculate Practically Perfect Probabilities

model_applications/short_range/Point2Grid_obsLSR_ObsOnly_PracticallyPerfect.conf

Scientific Objective

To use storm reports as observations to calculate Practically Perfect probabilities.

Version Added

METplus version 3.1

Datasets

Forecast: None

Observation: NOAA Storm Prediction Center (SPC) Local Storm Reports

Climatology: None

Location: All of the input data required for this use case can be found in a sample data tarball. Each use case category will have one or more sample data tarballs. It is only necessary to download the tarball with the use case’s dataset and not the entire collection of sample data. Click here to access the METplus releases page and download sample data for the appropriate release: https://github.com/dtcenter/METplus/releases This tarball should be unpacked into the directory that you will set the value of INPUT_BASE. See Running METplus section for more information.

METplus Components

This use case runs ASCII2NC to get the storm reports in netcdf format, runs Point2Grid to get those netcdf observations onto a grid, runs RegridDataPlane to use that gridded data as a mask to calculate probabilities

METplus Workflow

Beginning time (INIT_BEG): 2020020500

End time (INIT_END): 2020020500

Increment between beginning and end times (INIT_INCREMENT): 24H

Sequence of forecast leads to process (LEAD_SEQ): 24H

The following tools are used for each run time:

ASCII2NC > Point2Grid > RegridDataPlane

This example runs on a single time/file at a time. Each storm report is assumed to have no more than 24 hours of data inside.

Run times:

2020-02-05

METplus Configuration

METplus first loads all of the configuration files found in parm/metplus_config, then it loads any configuration files passed to METplus via the command line, i.e. parm/use_cases/model_applications/short_range/Point2Grid_obsLSR_ObsOnly_PracticallyPerfect.conf

[config]

# Documentation for this use case can be found at
# https://metplus.readthedocs.io/en/latest/generated/model_applications/short_range/Point2Grid_obsLSR_ObsOnly_PracticallyPerfect.html

# For additional information, please see the METplus Users Guide.
# https://metplus.readthedocs.io/en/latest/Users_Guide

###
# Processes to run
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#process-list
###

PROCESS_LIST = ASCII2NC, Point2Grid, RegridDataPlane 


###
# Time Info
# LOOP_BY options are INIT, VALID, RETRO, and REALTIME
# If set to INIT or RETRO:
#   INIT_TIME_FMT, INIT_BEG, INIT_END, and INIT_INCREMENT must also be set
# If set to VALID or REALTIME:
#   VALID_TIME_FMT, VALID_BEG, VALID_END, and VALID_INCREMENT must also be set
# LEAD_SEQ is the list of forecast leads to process
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#timing-control
###

LOOP_BY = INIT
INIT_TIME_FMT = %Y%m%d%H
INIT_BEG = 2020020500
INIT_END = 2020020500
INIT_INCREMENT = 24H

LEAD_SEQ = 12H


###
# File I/O
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#directory-and-filename-template-info
###

CONFIG_DIR = {PARM_BASE}/use_cases/model_applications/short_range/Point2Grid_obsLSR_ObsOnly_PracticallyPerfect


# ASCII2NC

ASCII2NC_INPUT_DIR = {INPUT_BASE}/model_applications/short_range/practically_perfect
ASCII2NC_INPUT_TEMPLATE = "{CONFIG_DIR}/read_ascii_storm.py {ASCII2NC_INPUT_DIR}/200205_rpts_filtered.csv"

ASCII2NC_OUTPUT_TEMPLATE = {OUTPUT_BASE}/model_applications/short_range/practically_perfect/StormReps.{init?fmt=%Y%m%d%H}.nc


# Point2Grid

POINT2GRID_INPUT_DIR = {OUTPUT_BASE}/model_applications/short_range/practically_perfect
POINT2GRID_INPUT_TEMPLATE = {OUTPUT_BASE}/model_applications/short_range/practically_perfect/StormReps.{init?fmt=%Y%m%d%H}.nc

POINT2GRID_OUTPUT_DIR = {OUTPUT_BASE}/model_applications/short_range/practically_perfect


# RegridDataPlane

OBS_REGRID_DATA_PLANE_INPUT_DIR = {POINT2GRID_OUTPUT_DIR}
OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE = StormReps_211.{init?fmt=%Y%m%d%H}.nc

OBS_REGRID_DATA_PLANE_OUTPUT_DIR = {POINT2GRID_OUTPUT_DIR}
OBS_REGRID_DATA_PLANE_OUTPUT_TEMPLATE = StormReps_211_Probs.{init?fmt=%Y%m%d}.nc

POINT2GRID_OUTPUT_TEMPLATE = {OUTPUT_BASE}/model_applications/short_range/practically_perfect/StormReps_211.{init?fmt=%Y%m%d%H}.nc


###
# ASCII2NC Settings
# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#ascii2nc
###

ASCII2NC_WINDOW_BEGIN = 0
ASCII2NC_WINDOW_END = 0

ASCII2NC_INPUT_FORMAT = python
ASCII2NC_TIME_SUMMARY_FLAG = False
ASCII2NC_TIME_SUMMARY_RAW_DATA = False
ASCII2NC_TIME_SUMMARY_BEG = 000000
ASCII2NC_TIME_SUMMARY_END = 235959
ASCII2NC_TIME_SUMMARY_STEP = 300
ASCII2NC_TIME_SUMMARY_WIDTH = 600
ASCII2NC_TIME_SUMMARY_GRIB_CODES = 11, 204, 211
ASCII2NC_TIME_SUMMARY_VAR_NAMES =
ASCII2NC_TIME_SUMMARY_TYPES = min, max, range, mean, stdev, median, p80
ASCII2NC_TIME_SUMMARY_VALID_FREQ = 0
ASCII2NC_TIME_SUMMARY_VALID_THRESH = 0.0


###
# Point2Grid Settings
# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#point2grid
###

POINT2GRID_REGRID_TO_GRID = G211

POINT2GRID_INPUT_FIELD =Fscale
POINT2GRID_INPUT_LEVEL =

POINT2GRID_ADP =

POINT2GRID_REGRID_METHOD = MAX

POINT2GRID_GAUSSIAN_DX = 81.271
POINT2GRID_GAUSSIAN_RADIUS = 120

POINT2GRID_PROB_CAT_THRESH =

POINT2GRID_VLD_THRESH =


###
# RegridDataPlane Settings
# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#regriddataplane
###

OBS_REGRID_DATA_PLANE_RUN = True

REGRID_DATA_PLANE_ONCE_PER_FIELD = False

OBS_REGRID_DATA_PLANE_VAR1_INPUT_FIELD_NAME = Fscale_mask
OBS_REGRID_DATA_PLANE_VAR1_INPUT_LEVEL = "(*,*)"
OBS_REGRID_DATA_PLANE_VAR1_OUTPUT_FIELD_NAME = PP_probs

REGRID_DATA_PLANE_VERIF_GRID = G211

REGRID_DATA_PLANE_METHOD = MAXGAUSS

REGRID_DATA_PLANE_WIDTH = 1

REGRID_DATA_PLANE_GAUSSIAN_DX = 81.271
REGRID_DATA_PLANE_GAUSSIAN_RADIUS = 120

MET Configuration

METplus sets environment variables based on user settings in the METplus configuration file. See How METplus controls MET config file settings for more details.

YOU SHOULD NOT SET ANY OF THESE ENVIRONMENT VARIABLES YOURSELF! THEY WILL BE OVERWRITTEN BY METPLUS WHEN IT CALLS THE MET TOOLS!

If there is a setting in the MET configuration file that is currently not supported by METplus you’d like to control, please refer to: Overriding Unsupported MET config file settings

Ascii2NcConfig_wrapped

////////////////////////////////////////////////////////////////////////////////
//
// Default ascii2nc configuration file
//
////////////////////////////////////////////////////////////////////////////////

//
// The parameters listed below are used to summarize the ASCII data read in
//

//
// Time periods for the summarization
// obs_var (string array) is added and works like grib_code (int array)
// when the obs name is given instead of grib_code
//
${METPLUS_TIME_SUMMARY_DICT}

//
// Mapping of input little_r report types to output message types
//
message_type_map = [
   { key = "FM-12 SYNOP";  val = "ADPSFC"; },
   { key = "FM-13 SHIP";   val = "SFCSHP"; },
   { key = "FM-15 METAR";  val = "ADPSFC"; },
   { key = "FM-18 BUOY";   val = "SFCSHP"; },
   { key = "FM-281 QSCAT"; val = "ASCATW"; },
   { key = "FM-32 PILOT";  val = "ADPUPA"; },
   { key = "FM-35 TEMP";   val = "ADPUPA"; },
   { key = "FM-88 SATOB";  val = "SATWND"; },
   { key = "FM-97 ACARS";  val = "AIRCFT"; }
];

//
// Indicate a version number for the contents of this configuration file.
// The value should generally not be modified.
//
//version = "V10.0";

tmp_dir = "${MET_TMP_DIR}";

${METPLUS_MET_CONFIG_OVERRIDES}

Python Embedding

This use case uses a Python embedding script to read input data.

parm/use_cases/model_applications/short_range/Point2Grid_obsLSR_ObsOnly_PracticallyPerfect/read_ascii_storm.py

import pandas as pd
import os
import sys

print(f'Python Script: {sys.argv[0]}')

# input file specified on the command line
# load the data into the numpy array

if len(sys.argv) < 2:
    script_name = os.path.basename(sys.argv[0])
    print(f"ERROR: {script_name} -> Must specify exactly one input file.")
    sys.exit(1)

# Read the input file as the first argument
input_file = os.path.expandvars(sys.argv[1])
print(f'Input File: {input_file}')

if not os.path.exists(input_file):
    print("ERROR: Could not find input file")
    sys.exit(2)

# Read and format the input 11-column observations
COLUMN_NAMES = (
    "Message_Type",     # (1)  string
    "Station_ID",       # (2)  string
    "Valid_Time",       # (3)  string (YYYYMMDD_HHMMSS)
    "Lat",              # (4)  numeric (Deg North)
    "Lon",              # (5)  numeric (Deg East)
    "Elevation",        # (6)  numeric (msl)
    "Var_Name",         # (7)  string (or GRIB_Code)
    "Level",            # (8)  numeric
    "Height",           # (9)  numeric (msl or agl)
    "QC_String",        # (10) string
    "Observation_Value" # (11) numeric
)

# Create a blank dataframe based on the 11 column standard
point_frame = pd.DataFrame(columns=COLUMN_NAMES,dtype='str')

#Read in the Storm report, 8 columns not matching the 11 column standard
temp_data = pd.read_csv(input_file,names=['Time', 'Fscale', 'Location', 'County','Stat','Lat', 'Lon', 'Comment'], dtype=str ,skiprows=1)

#Strip out any rows in the middle that are actually header rows
#Allows for concatenating storm reports together
temp_data = temp_data[temp_data["Time"] != "Time"]

# strip out any rows that have any null/NaN values
temp_data = temp_data[~temp_data.isnull().any(axis=1)]

#Change some columns to floats and ints
temp_data[["Lat","Lon"]] = temp_data[["Lat","Lon"]].apply(pd.to_numeric)

#Assign approprite columns to point_frame leaving missing as empty strings
point_frame["Lat"] = temp_data["Lat"]
point_frame["Lon"] = temp_data["Lon"]
#point_frame["Station_ID"] = temp_data["County"]
point_frame["Station_ID"] = "NA"
point_frame["Var_Name"] = "Fscale"
point_frame["Message_Type"] = "StormReport"

#Assign 0.0 values to numeric point_frame columns that we don't have in the csv file
point_frame["Elevation"] = 0.0
point_frame["Level"] = 0.0
point_frame["Height"] = 0.0

#Change Comments into a "QC" string Tornado=1, Hail=2, Wind=3, Other=4
point_frame["QC_String"] = "4"
mask = temp_data["Comment"].str.contains('TORNADO')
point_frame.loc[mask,"QC_String"] = "1"
mask = temp_data["Comment"].str.contains('HAIL')
point_frame.loc[mask,"QC_String"] = "2"
mask = temp_data["Comment"].str.contains('WIND')
point_frame.loc[mask,"QC_String"] = "3"

#Time is HHMM in the csv file so we need to use a piece of the filename and
#this value to create a valid date string
file_without_path = os.path.basename(input_file)
year_month_day = "20"+file_without_path[0:6]
point_frame["Valid_Time"] = year_month_day+"_"+temp_data["Time"]+"00"

#Currently we are only interested in the fact that we have a report at that locaton
#and not its actual value so all values are 1.0
point_frame["Observation_Value"] = 1.0

#Ascii2nc wants the final values in a list
point_data = point_frame.values.tolist()

print("Data Length:\t" + repr(len(point_data)))
print("Data Type:\t" + repr(type(point_data)))

########################################################################

For more information on the basic requirements to utilize Python Embedding in METplus, please refer to the MET User’s Guide section on Python embedding.

User Scripting

User Scripting is not used in this use case.

Running METplus

Pass the use case configuration file to the run_metplus.py script along with any user-specific system configuration files if desired:

run_metplus.py /path/to/METplus/parm/use_cases/model_applications/short_range/Point2Grid_obsLSR_ObsOnly_PracticallyPerfect.conf /path/to/user_system.conf

See Running METplus for more information.

Expected Output

A successful run will output the following both to the screen and to the logfile:

INFO: METplus has successfully finished running.

Refer to the value set for OUTPUT_BASE to find where the output data was generated. Output for this use case will be found in model_applications/short_range/practically_perfect/ (relative to OUTPUT_BASE) and will contain the following files:

• StormReps_211_Probs.20200205.nc

Keywords

Note

• ASCII2NCToolUseCase

• Point2GridToolUseCase

• RegridDataPlaneToolUseCase

• RegriddingInToolUseCase

• NetCDFFileUseCase

• PythonEmbeddingFileUseCase

• ShortRangeAppUseCase

• NCAROrgUseCase

• ProbabilityGenerationUseCase

• MaskingFeatureUseCase

• HMTOrgUseCase

• HWTOrgUseCase

Navigate to the METplus Quick Search for Use Cases page to discover other similar use cases.