导航|地址|刷新
s2s.cma.cn
��ࡱ�>�� bjbj�>�>5��T�T �!n��( @@@��TTT8��h!4T�s��'�'(�'�'�'�(�(�(uswswswswswsws$�w�vzz�s@�(�(�(�(�(�s@@�'�'��sY9Y9Y9�(&@�'@�'usY9�(usY9Y92k��o�'��p,��T�0��l.as�s0�s�l��z�2��z\�o�oJ�z@�o��(�(Y9�(�(�(�(�(�s�s�7��(�(�(�s�(�(�(�(��z�(�(�(�(�(�(�(�(�(� �:WORLD METEOROLOGICAL ORGANIZATION COMMISSION FOR BASIC SYSTEMS OPAG on DPFS Meeting of the Expert Team on Operational Prediction from Sub-seasonal to Longer-time Scales Beijing, China, 11-15 April 2016 CBS-DPFS/ET-OPSLS /Doc. 6.3(4) (06.IV.2016) _______ Agenda item : 6.3 ENGLISH ONLY TITLE OF THE DOCUMENT (Submitted by Caio Coelho and Yuhei Takaya) Summary and purpose of document This document summarizes the responses of GPCs to the questionnaire on subseasonal verification practices in operational centres. Action Proposed The meeting is invited to discuss the questionnaire results. 1) Introduction This document summarizes the responses of GPCs to the questionnaire on subseasonal verification practices in operational centres sent to all GPCs in advance of the ET-OPSLS meeting, Beijing 11-15 April 2016. The questionnaire, which is enclosed in the annex at the end of this document, was designed to support verification research activities of the WWRP/WCRP Subseasonal to Seasonal (S2S) prediction project, which has linkages with Joint CBS-CCl Expert Team on Operational Predictions from Sub-seasonal to Longer-time Scale (ET-OPSLS) activities on subseasonal forecasts through Task Team 3 (TT3): Development of sub-seasonal forecasts. The purpose of the questionnaire was to share current practices used by operational centres to verify subseasonal forecasts (both for operations and research) and also help identify gaps and guide novel developments. The questionnaire was designed in six sections. The following section presents a summary of GPCs responses for each of the six sections, prepared by the S2S sub-project on Verification and Products, to help guide S2S verification research activities and for discussion with ET-OPSLS. 2) Summary of GPCs responses for each of the six sections of the questionnaire 2.1) Section 1: Identification In the first section of the questionnaire respondents were asked to provide identification and information about their organization. All 12 GPCs responded the questionnaire, providing a complete and updated set of information about the current practices used by operational centres to verify subseasonal forecasts. 2.2) Section 2: Verification documentation of S2S systems In the second section of the questionnaire respondents were asked to provide documentation where verification information in terms of forecast quality assessment of their S2S forecast systems is available. Respondents provided various documentation information including peer reviewed journals, conference/workshop publications/presentations, technical reports and websites where verification information about their S2S forecast systems is disseminated. The responses indicated that subseasonal forecast verification is well established in some GPCs but is still an under development activity for a large number of GPCs. 2.3) Section 3: Reference verification datasets In the third section of the questionnaire respondents were asked to provide information about reference verification datasets used for assessing forecast quality of their subseasonal retrospective forecasts (reforecasts; hindcasts). GPCs indicated the use of a variety of datasets for the verification of their subseasonal forecasts including reanalysis products, satellite estimates and station data. Below is a summary of responses provided by GPCs indicating the used datasets for verification of subseasonal hindcasts in terms of different parameters: Atmospheric parameters (e.g. geopotential height, temperature, SLP, etc) NCEP/NCAR Reanalysis 1 (http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis.html) NCEP-DOE Reanalysis 2 (http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis2.html) ECMWF era-interim and the operational analysis for the most recent months for which era-interim is not available (http://apps.ecmwf.int/datasets/data/interim-full-daily/) JRA-55 reanalysis: http://jra.kishou.go.jp/ Oceanic parameters (e.g SST, subsurface temperature, etc.) NOAA Optimum Interpolation (OI) Sea Surface Temperature (SST) V2 (http://www.esrl.noaa.gov/psd/data/gridded/data.noaa.oisst.v2.html) NOAA Optimum Interpolation (OI) Sea Surface Temperature (SST) V2 High Resolution Dataset (http://www.esrl.noaa.gov/psd/data/gridded/data.noaa.oisst.v2.highres.html) NCDC daily OI SST analysis https://www.ncdc.noaa.gov/oisst Era-interim SST (http://apps.ecmwf.int/datasets/data/interim-full-daily/) Extended Reconstructed Sea Surface Temperature (ERSST): https://www.ncdc.noaa.gov/data-access/marineocean-data/extended-reconstructed-sea-surface-temperature-ersst-v3b Sub-surface ocean parameters: Global Ocean Data Assimilation System (GODAS) products http://www.cpc.ncep.noaa.gov/products/GODAS/ Precipitation and other parameters such as near surface temperature, wind, etc CPC Merged Analysis of Precipitation (CMAP) Pentad Dataset (http://www.esrl.noaa.gov/psd/data/gridded/data.cmap.html) ECMWF short term forecasts (precipitation), era-interim and the operational analysis for the most recent months for which era-interim is not available (http://apps.ecmwf.int/datasets/data/interim-full-daily/): Precipitation, surface temperature and surface wind FEWS-NET ARC2 blended gauge satellite data and TRMM to validate predictions of onset timing NCEP/NCAR reanalysis precipitation dataset: http://www.esrl.noaa.gov/psd/cgi-bin/db_search/DBSearch.pl?Dataset=NCEP+Reanalysis+Daily+Averages+Surface+Flux&Variable=Precipitation+rate Global Precipitation Climatology Project (GPCP) version 2 dataset: http://precip.gsfc.nasa.gov/ Climate Anomaly Monitoring System (CAMS) and OLR Precipitation Index (OPI), CAMS-OPI monthly mean precipitation: http://www.cpc.ncep.noaa.gov/products/global_precip/html/wpage.cams_opi.html Global Historical Climate Network (GHCN) Climate Anomaly Monitoring System (CAMS) monthly mean surface temperature: http://www.esrl.noaa.gov/psd/data/gridded/data.ghcncams.html UK station data aggregated to UK climate regions (precipitation, maximum and minimum temperature) Australian region local high-quality rainfall and temperature datasets described at Jones DA, Wang W, Fawcett R. (2009) High�quality spatial climate data�sets for Australia. Australian Meteorological and Oceanographic Journal. 58:233�248. NOAA outgoing long-wave radiation (OLR) and 850 and 200 hPa zonal winds from NCEP/NCAR reanalysis used for MJO indices RMM1 and RMM2. 2.4) Section 4: Reforecast setting In the forth section of the questionnaire respondents were asked to provide information about reforecast setting of their subseasonal forecast systems. Table 1 shows the diverse configuration of GPCs subseasonal forecast systems reforecasts in terms of reforecast period, reforecast initial dates and number of reforecast ensemble members, illustrating the challenge for performing forecast verification intercomparison assessments. Reforecast periodReforecast initial datesNumber of reforecast ensemble membersGPC Beijing1994-2014every day4 for each initial dateGPC CPTECTo be definedTo be definedTo be definedGPC ECMWFThe most recent 20 yearsTwice per week11GPC Exeter1996-2009 (soon to be 1993-2015)1st; 9th; 17th and 25th of each month3 for each initial dateGPC Melbourne1981-20101, 6, 11, 16, 21, 2633GPC Montreal1995-2014Date of Thursday of the current forecast4GOC Moscou1981-2010Every week on Wednesday10GPC Pretoria2000-201304th , 11th , 18th and 25th (flexible for change)24 (4x daily)GPC Seoul1996-2009 (1991-2010 from May2016)1st; 9th; 17th and 25th of each month3 for each initial dateGPC Tokyo1981-201010th, 20th and end of month of each month5 for each initial dateGPC Toulouse1993-20141st and 15th of each month15 for each initial dateGPC Washington1999-2010Daily4/dayTable 1: Configuration of GPCs subseasonal forecast systems reforecasts in terms of reforecast period, reforecast initial dates and number of reforecast ensemble members. In terms of time averaging the questionnaire revealed some GPCs verify reforecast weekly, monthly and week 3-4 (2nd forthnight) averages, but the definition of averaging period is slightly different among GPC (e.g. week 1 is either defined as day 2 to 8 or day 1 to 7 by different GPCs). Table 2 summarizes the number of GPCs indicating the averaging period for which their reforecasts are verified. The relatively small number of GPCs (6 or less) indicating the practice of verifying weekly, monthly and week 3-4 (2nd forthnight) averages suggests that verification procedures for these averaging periods still need to be adopted and consolidated by several GPCs. Averaging periodNumber of GPCsWeek 15Week 26Week 34Week 44Monthly5Week 3-4 (2nd forthnight)3Table 2: Number of GPCs indicating the averaging period for which their reforecasts are verified. Some GPCs reported that the procedure for generating daily information prior to producing weekly and monthly averages for verifying subseasonal reforecasts depends on the variable of interest. Table 3 summarizes the number of GPCs indicating averaging procedures prior to producing weekly and monthly averages. Averaging procedureNumber of GPCsDaily information is generated by averaging hourly instantaneous fields for 24 hours1Daily information is generated by averaging 6-hourly instantaneous fields at 00, 06, 12 and 18 UTC4Daily information is output directly by the forecast model1Averaging all instantaneous fields at every time-step1Depends on the field/variable. 2Table 3: Number of GPCs indicating the averaging procedure prior to producing weekly and monthly averages. The questionnaire also requested GPCs to indicate which model parameters of their subseasonal reforecasts were verified. Table 4 summarizes GPCs responses to this question. The most common variables verified by most GPCs are 500 hPa Geopotential Height, 2 metre temperature and precipitation. VariableNumber of GPCs500 hPa Geopotential Height9Sea level pressure6850 hPa temperature42 metre temperature9Precipitation9200 hPa Velocity potential2200 hPa Stream function2Surface temperature1Table 4: Number of GPCs indicating which model parameters are verified in their subseasonal reforecasts. In terms of spatial resolution Table 5 shows that most GPCs verify their subseasonal reforecasts at either 2.5 by 2.5 or 1.5 by 1.5 degrees in latitude and longitude. Spatial resolutionNumber of GPCs2.5 by 2.5 degrees in latitude and longitude51.5 by 1.5 degrees in latitude and longitude41.0 by 1.0 degrees in latitude and longitude1At model grid1Table 5: Number of GPCs indicating at which spatial resolution their subseasonal reforecasts are verified. Table 6 summarizes the spatial domain for which GPC subseasonal reforecasts verification products are displayed. A total of 8 GPCs display their products globally, and a reduced number of GPCs (3 or less) display regionalized products over selected continental regions where they are located. Spatial domainNumber of GPCsGlobal8AfricaAsia3Europe 3South AmericaNorth America2Oceania1Tropics3Southern Hemisphere3Northern Hemisphere4Artic region1Table 6: Number of GPCs indicating which spatial domain their subseasonal reforecasts verification products are displayed. 2.5) Section 5: Verification metrics In the fifth section of the questionnaire respondents were asked to provide information about verification metrics used to assess forecast quality of their subseasonal forecast systems. Table 7 shows which deterministic metrics are used by GPCs to assess forecast quality and biases of their subseasonal reforecasts. The most commonly used deterministic metrics are mean bias, correlation between forecast and observed anomalies and RMSE (or MSE). Deterministic metricsNumber of GPCsMean bias (map displaying model climate minus observed climate)8Variability bias (map displaying model standard deviation (or variance) divided by observed standard deviation (or variance))2Correlation between forecast and observed anomalies displayed as a map8Root mean squared error (RMSE) or mean squared error (MSE) displayed as a map7Mean squared error skill score (MSSS) displayed as a map5Table 7: Number of GPCs indicating which deterministic metrics are used for assessing forecast quality and biases of their subseasonal reforecasts. Table 8 summarizes which events are assessed by GPCs when investigating probabilistic forecast quality of their subseasonal reforecasts. The most commonly used events are 3 categories (tercile probabilities) as traditionally used in seasonal forecasting. EventsNumber of GPCs2 categories (above/below median or mean) 23 categories (above normal, near normal, below normal)10Quintile categories2Probability of exceedance1Table 8: Number of GPCs indicating which events are assessed when investigating probabilistic forecast quality of their subseasonal reforecasts. Table 9 shows which probabilistic metrics are used by GPCs when investigating probabilistic forecast quality of their subseasonal reforecasts. The most commonly used probabilistic metrics are reliability diagrams, ROC curves, area under ROC curve and the Brier score, all commonly used when assessing seasonal forecasts. Probabilistic metricsPlease indicate Y/NReliability diagrams8Brier score 7Reliability, resolution and resolution components of the Brier score5Continuous Ranked Probability Score (CRPS)3ROC curves16Area under ROC curve displayed as a map8Ignorance score1Hanssen-Kuipers score1Heidke Skill Score1Table 9: Number of GPCs indicating which probabilistic metrics are used to assess forecast quality of their subseasonal reforecasts. Table 10 summarizes which regional and large scale indices are used by GPCs when investigating forecast quality of their subseasonal reforecasts. Large scale teleconnection indices are assessed by 6 GPCs and regional average indices are assessed by 4 GPCs, suggesting that such assessment still needs to be considered for adoption and consolidation by a number of GPCs. IndicesNumber of GPCsTeleconnection indices 6Regional average indices 4Table 10: Number of GPCs indicating which regional and large scale indices are used to assess forecast quality of their subseasonal reforecasts. Table 11 shows which intraseasonal oscillation indices are used by GPCs when investigating forecast quality of their subseasonal reforecasts. The real-time multivariate MJO index is used by 8 GPCs and the boreal summer intraseasonal oscillation index is used by 4 GPCs. One GPC indicated that the real-time multivariate MJO index is used for monitoring purposes but is not operationally verified. Intraseasonal oscillation indicesNumber of GPCsReal-time Multivariate Madden and Jullian Oscillation (MJO) index8Boreal Summer Intraseasonal Oscillation (BSISO) index 4Table 11: Number of GPCs indicating which intraseasonal oscillation indices are used to assess forecast quality of their subseasonal reforecasts. Table 12 summarizes which verification indices and/or diagram/curve are used by GPCs when investigating forecast quality of extreme events in their subseasonal reforecasts. Reliability diagrams are used by only 3 GPCs, ROC curves are used by only 2 GPCs and EFI is currently being investigated by a single GPC, illustrating that verification of extreme events is an area that deserves more attention for an adequate forecast quality assessment of these events. Indices and/or diagram/curve for assessing forecast quality of extreme eventsNumber of GPCsReliability diagrams for events in the 90th and 10th percentiles (or similar thresholds)3ROC curves for events in the 90th and 10th percentiles (or similar thresholds)2Extreme forecast index (EFI) and/or its weighted version EFIR1Extremal dependence indices (e.g. EDI and SEDI)0Table 12: Number of GPCs indicating which indices and/or diagram/curve are used to assess forecast quality of extremes in their subseasonal reforecasts. 2.6) Section 6: Tailored products verification In the sixth section of the questionnaire respondents were asked to provide information about verification of tailored products when assessing forecast quality of their subseasonal forecast systems. Table 13 shows which sector specific verification quantities are used by GPCs when investigating forecast quality in their subseasonal reforecasts. Frequency of heat-wave days and frequency of cold-wave days is assessed by a single GPC. Frequency of heavy rain days is under development by a single GPC, and frequency of heat-wave is also under development by a single GPC. Another GPC reported that is currently investigating predictability of heat and cold waves. These results illustrate that sector specific verification is an area that deserves more attention for a more comprehensive forecast quality assessment of sector specific quantities. Sector specific verification quantitiesNumber of GPCsFrequency of heavy rain days1 (under development)Frequency of no-rain days0Frequency of heat-wave days1 (under development by another GPC)Frequency of cold-wave days1Cluster of heavy rain days (e.g. probability of n consecutive days of heavy rain)0Cluster of no-rain days (e.g. probability of n consecutive days of no-rain)0Cluster of heat-wave days (e.g. probability of n consecutive days with temperature above a high threshold)0Cluster of cold-wave days (e.g. probability of n consecutive days with temperature below a low threshold)0Predictability of heat and cold waves 1Table 13: Number of GPCs indicating which sector specific verification quantities are used to assess forecast quality of their subseasonal reforecasts. Forecast quality assessment of active and break rainfall phases and wet/dry spells was reported by a single GPC based on research experience with seasonal (not sub-seasonal) forecasts by the use of correlation of the ensemble mean number of rain days (no-rain days). Forecast quality assessment of rainy season onset and demise was reported by the same single GPC as being performed based on experience gained with seasonal forecasts by using a tercile description (early/average/late) and generating ROC scores for probabilistic forecasts � calculating the onset tercile category for each member. 3) Summary and final considerations The questionnaire responses are summarized as follows�: All 12 WMO GPCs responded the questionnaire providing a timely update on current practices used by operational centres for verifying subseasonal forecasts. The responses indicated that subseasonal forecast verification is well established in some GPCs but is still an under development activity for a large number of GPCs, contrasting with verification of seasonal forecasts that is a well established activity (guided by SVSLRF) in all 12 GPCs. GPCs indicated the use of a variety of datasets for the verification of their subseasonal forecasts including reanalysis products, satellite estimates and station data. The diverse configuration of GPCs subseasonal forecast systems, in terms of reforecast period, reforecast initial dates and number of reforecast ensemble members, illustrated the challenge for performing forecast verification intercomparison assessments. The relatively small number of GPCs (6 or less) indicating the practice of verifying weekly, monthly and week 3-4 (2nd forthnight) averages suggested that verification procedures for these averaging periods still need to be adopted and consolidated by several GPCs. The most common subseasonal forecast variables verified by most GPCs were found to be 500 hPa Geopotential Height, 2 metre temperature and precipitation, the latter two generally considered of great relevance for a number of societal applications. Most GPCs indicated to verify their subseasonal reforecasts at either 2.5 by 2.5 or 1.5 by 1.5 degrees in latitude and longitude, although the original model configuration may be able to output forecasts at a more refined spatial resolution. A total of 8 GPCs indicated to display their subseasonal forecast products globally, and a reduced number of GPCs (3 or less) indicated to display regionalized products over selected continental regions where they are located and therefore have a particular regional interest. The most commonly used deterministic metrics by GPCs when assessing subseasonal forecasts were found to be mean bias, correlation between forecast and observed anomalies and RMSE (or MSE). The most commonly used events by GPCs when assessing subseasonal forecasts were found to be 3 categories (tercile probabilities) as traditionally used in seasonal forecasting. The most commonly used probabilistic metrics by GPCs when assessing subseasonal forecasts were found to be reliability diagrams, ROC curves, area under ROC curve and the Brier score, all commonly used when assessing seasonal forecasts. Large scale teleconnection indices were indicated to be assessed by 6 GPCs and regional average indices by 4 GPCs, suggesting that such assessment still needs to be considered for adoption and consolidation by a number of GPCs. The real-time multivariate MJO index was indicated to be used by 8 GPCs and the boreal summer intraseasonal oscillation index by 4 GPCs. One GPC indicated that the real-time multivariate MJO index is used for monitoring purposes but is not operationally verified, suggesting room for improved verification practices of the MJO, including the development of appropriate verification approaches for this purpose. Forecast quality assessment of extreme events in subseasonal reforecasts were found to be performed using reliability diagrams by only 3 GPCs, ROC curves by only 2 GPCs and EFI by a single GPC, illustrating that verification of extreme events is an area that deserves more attention for an adequate forecast quality assessment of these events. Sector specific verification (tailored products) was found to be address by a very limited number of GPCs, illustrating that this is an area that deserves more attention for a more comprehensive forecast quality assessment of sector specific quantities. Forecast quality assessment of active and break rainfall phases, wet/dry spells, rainy season onset and demise was reported by a single GPC as a research initiative, suggesting also that this is an area that deserves more attention. The purpose of the questionnaire was to share current practices used by operational centres to verify subseasonal forecasts (both for operations and research) and also help identify gaps and guide novel developments. Although some GPC responses mentioned both operational and research verification practices, the overall summary of responses is likely to be more heavily weighted towards operational activities, with some subseasonal forecast verification research practices performed by some GPCs not necessarily fully incorporated. An interesting aspect that deserves consideration is the distinct possible approaches for subseasonal verification, namely verification of real-time forecasts, verification of reforecasts and verification of outlooks, the latter being an official forecast produced by combining model forecast and expert judgment information. One GPC indicated the common practice of assessing forecast quality of subseasonal outlooks because this is considered the official subseasonal forecast information disseminated to the public. Another GPC indicated that forecast quality assessment of subseasonal operational forecasts is based on verification of the real-time forecasts in line with the practice currently used in numerical weather prediction. For this GPC the assessment of reforecasts is mainly used for diagnostic and predictability studies due to the reduced number of ensemble members in reforecasts when compared to real-time forecasts. Although the questionnaire was designed to address verification of subseasonal reforecasts, the title of the questionnaire �Questionnaire on subseasonal verification practices in operational centres� might have caused some confusion when contrasted with the focus of the questions on reforecast (model) data verification. Annex: Questionnaire on subseasonal verification practices in operational centers You have received this questionnaire as a World Meteorological Organization (WMO) Global Producing Centre of Long-Range Forecasts (GPC). The questionnaire is designed to support verification research activities of the WWRP/WCRP Subseasonal to Seasonal (S2S) prediction project. The purpose of the questionnaire is to share current practices used by operational centres to verify subseasonal forecasts (both for operations and research) and also help identify gaps and guide novel developments. Responses will be summarised by the S2S sub-project
…