导航|地址|刷新
S2S







Introduction


Objectives

Background

Research

Contact

About CMDC


Data Access


API

Database

Products

Data interactive


Description


Models

Parameters


Sub-project


Phase II

Phase I

Agreement


Documents


Reports

News letter


Old Version






Login

Register












S2S sub-project on verification and products


Last modified by Unknown User on 2018/07/28 04:50


------


Highlight: The WWRP/WGNE Joint Working Group on Forecast Verification Research (JWGFVR) is pleased to announce the 7th International Verification Methods Workshop (7IVMW), which will be held in Berlin, Germany, 8-11 May 2017, preceded by a tutorial on forecast verification methods, 3-6 May 2017.

The workshop will have a dedicated session on S2S verification. The S2S community is cordially invited to submit abstracts on S2S verification methodologies to the workshop. Abstract submission is open at the workshop website http://www.7thverificationworkshop.de/

Please note the following deadlines:
- January 31 for tutorial application (open)
- February 27 for abstract submission (open)
- March 31 for registration (to be open soon)

------

The S2S community is cordially invited to enter the Challenge to Develop and Demonstrate the Best New User-Oriented Forecast Verification Metric launched by the WWRP/WGNE Joint Working Group on Forecast Verification Research (JWGFVR)



The aim of this challenge is to promote user-oriented verification, that is, quantitative assessment of forecast quality in terms that are meaningful to particular forecast users. The scope includes all applications of meteorological and hydrological forecasts. The user-oriented verification metrics contest will help support the WWRP/WCRP projects on High Impact Weather (HIWeather), Subseasonal to Seasonal Prediction (S2S), and Polar Prediction (PPP).

The deadline for entries is 31 October 2016. Click here to find out more, or contact verifchallenge@ucar.edu.

------

This S2S verification and products wiki page has the following content:

1) Objectives

2) Membership

3) Linkages with coordinated WMO operational activities

    3.1) Collaboration between S2S and WMO

    3.2) The pilot real-time sub-seasonal MME prediction in WMO LC-LRFMME

4) List of published literature on verification methods of relevance to S2S verification

    4.1) Books and technical reports

    4.2) Scientific papers

5) List of published literature on S2S verification

    5.1) Assessment of S2S systems forecast skill

    5.2) Assessment of MJO/ISO forecast skill

    5.3) Assessment of monsoon systems forecast skill and associated characteristics

    5.4) Applications

    5.5) Seamless verification

6) Available reference verification datasets for assessing S2S forecast quality

    6.1) Atmospheric parameters (e.g. geopotential height, temperature, SLP, wind, etc)

    6.2) Oceanic parameters

    6.3) Surface parameters

    6.4) Datasets accessible via the KNMI Climate Explorer

7) S2S project models

7.1) Accessing S2S models data

7.2) Visualizing S2S forecast products



------

1) Objectives

Recommend verification metrics and datasets for assessing forecast quality of S2S forecasts

Provide guidance for a potential centralized verification effort for comparing forecast quality of different S2S forecast systems, including the comparison of multi-model and individual ensemble systems and consider linkages with users and applications

The S2S verification science plan provides more detailed information about this sub-project.

2) Membership

Caio Coelho (CPTEC/INPE, Brazil)

Andrew Robertson (IRI, USA)

Richard Graham (UKMO, UK)

Yuhei Takaya (JMA, Japan)

Debra Hudson (BoM, Australia)

Joanne Robbins (UKMO, UK)

Angel Muñoz (GFDL, USA)



3) Linkages with coordinated WMO operational activities



3.1) Collaboration between S2S and WMO

The research performed in S2S has strong linkages with WMO operational activities, particularly with the CBS/CCl Expert Team on Operational Prediction from Sub-seasonal to Longer time-scale (ET-OPSLS) and the WMO Lead Centre for Long Range Forecast Multi-Model Ensemble (LC-LRFMME). The S2S project in collaboration with WMO is therefore bridging research and operation activities to drive science and technology forward for producing better weather/climate information for a number of application sectors.

As part of this collaboration between S2S and WMO the S2S sub-project on verification and products has been conducting the following activities:

- Preparation of questionnaire on subseasonal verification practices in operational centres (both in operations and research) to help identify gaps and guide novel developments. This questionnaire was sent to the 12 designated WMO Global Producing Centres of Long-Range Forecasts (GPCs), the results were discussed with the ET-OPSLS and are summarized in this document.

- Preparation of document on S2S application-oriented activities and operational needs as input for the ET-OPSLS







3.2) The pilot real-time sub-seasonal MME prediction in WMO LC-LRFMME



The WMO Lead Centre for Long Range Forecast Multi-Model Ensemble (LC-LRFMME) has recently developed a pilot system for real-time multi-model subseasonal forecasts using real-time forecasts (and hindcasts) from a subset of models contributing to the WWRP/WCRP S2S research project accessible via ECMWF data archive. Following this link the S2S research community has the opportunity to see the initial developments conducted by the LC-LRFMME and provide feedback for future developments and improvements in this pilot under development system. Subseasonal models from four GPCs are currently used: ECMWF, Exeter, Tokyo and Washington. A range of forecast products has been developed including probabilities for tercile categories of weekly averages of 2m temperature and rainfall as well as the MJO and BSISO indices. Verification has also been generated using ROC curves and scores as well as anomaly pattern correlation for a few case studies.





4) List of published literature on verification methods of relevance to S2S verification

Below in sections 4.1 and 4.2 is a selected list of published literature (including books, technical reports and scientific papers) on verification methodologies of relevance for S2S forecast verification. A more comprehensive list and additional information on forecast verification is available at http://www.cawcr.gov.au/projects/verification/ a website of the WWRP/WGNE Joint Working Group on Forecast Verification Research.

Please note that further down on this wiki page Section 5 provides a list of published literature on S2S verification including in section 5.1 papers on the assessment of S2S systems forecast skill, in section 5.2 papers on the assessment of MJO/ISO forecast skill, in section 5.3 papers on the assessment of monsoon systems forecast skill and associated characteristics, in section 5.4 papers on applications and in section 5.5 on seamless verification.

4.1) Books and technical reports

Jolliffe IT, Stephenson DB (2012) Forecast Verification: A Practitioner's Guide in Atmospheric Science. 2nd Edition.  Wiley and Sons Ltd, 274 pp.

Stanski HR, Wilson LJ, Burrows WR (1989) Survey of common verification methods in meteorology. World Weather Watch Tech. Rept. No.8, WMO/TD No.358, WMO, Geneva, 114 pp. Available here.

Wilks DS (2011) Statistical Methods in the Atmospheric Sciences. 3rd Edition.  Elsevier, 676 pp.



4.2) Scientific papers

Bradley AA, Hashino T, Schwartz SS (2003) Distributions-oriented verification of probability forecasts for small data samples. Wea. Forecasting, 18, 903-917. http://dx.doi.org/10.1175/1520-0434(2003)018%3C0903:DVOPFF%3E2.0.CO;2

Bradley AA, Schwartz SS, Hashino T (2008) Sampling uncertainty and confidence intervals for the Brier score and Brier skill score. Wea. Forecasting, 23, 992-1006. doi: http://dx.doi.org/10.1175/2007WAF2007049.1

Brier GW (1950) Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78, 1-3. doi: http://dx.doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2

Epstein ES (1969) A scoring system for probability forecasts of tanked categories. J. App. Met. Vol 8. No 6, 985-987. doi: http://dx.doi.org/10.1175/1520-0450(1969)008<0985:ASSFPF>2.0.CO;2

Ferro CAT (2007) Comparing probabilistic forecasting systems with the Brier score. Weather and Forecasting 22, 1076-1088. doi: http://dx.doi.org/10.1175/WAF1034.1

Ferro CAT, Richardson DS, Weigel AP (2008) On the effect of ensemble size on the discrete and continuous ranked probability scores. Meteorol. Appl., 15, 19-24. http://onlinelibrary.wiley.com/doi/10.1002/met.45/epdf

Ferro CAT, Stephenson DB (2011) Extremal Dependence Indices: improved verifiation measures for deterministic forecasts of rare binary events. Wea. Forecasting, 26, 699-713. doi: http://dx.doi.org/10.1175/WAF-D-10-05030.1

Ferro CAT, Fricker TE (2012) A bias-corrected decomposition of the Brier score. Quarterly Journal of the Royal Meteorological Society, 138, 1954-1960, doi:10.1002/qj.1924.

Ferro CAT (2014) Fair scores for ensemble forecasts. Quarterly Journal of the Royal Meteorological Society, 140, 1917-1923, http://dx.doi.org/10.1002/qj.2270

Gneiting T,  Raftery AE (2007) Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102, Issue 477, 359-378. doi: 10.1198/016214506000001437 http://www.eecs.harvard.edu/cs286r/courses/fall12/papers/Gneiting07.pdf

Hamill TM (2001) Interpretation of rank histograms for verifying ensemble forecasts. Mon. Wea. Rev., 129, 550-560. doi: http://dx.doi.org/10.1175/1520-0493(2001)129<0550:IORHFV>2.0.CO;2

Hersbach H (2000) Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather and Forecasting, 15, 559-570. doi: http://dx.doi.org/10.1175/1520-0434(2000)015<0559:DOTCRP>2.0.CO;2

Hsu W-R., Murphy AH (1986) The attributes diagram: A geometrical framework for assessing the quality of probability forecasts. Int. J. Forecasting, 2, 285-293. http://dx.doi.org/10.1016/0169-2070(86)90048-8

Jolliffe IT (2007) Uncertainty and inference for verification measures. Wea. Forecasting, 22, 637-650. doi: http://dx.doi.org/10.1175/WAF989.1

Jupp TE, Lowe R, Coelho CAS, Stephenson DB (2012): On the visualization, verification and recalibration of ternary probabilistic forecasts. Phil. Trans. R. Soc. A, 370, 1100-1120. doi:10.1098/rsta.2011.0350

Mason I (1982) A model for assessment of weather forecasts. Aust. Met. Mag., 30, 291-303. http://www.nssl.noaa.gov/users/brooks/public_html/feda/papers/mason82.pdf

Mason SJ, Weigel AP (2009) A generic forecast verification framework for administrative purposes. Mon. Wea. Rev., 137, 331-349. doi: http://dx.doi.org/10.1175/2008MWR2553.1

Mason SJ, Graham NE (2002) Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. Quarterly Journal of the Royal Meteorological Society Volume 128, Issue 584, pages 2145–2166, July 2002 Part B. DOI: 10.1256/003590002320603584 http://onlinelibrary.wiley.com/doi/10.1256/003590002320603584/pdf

Mason SJ (2008) Understanding forecast verification statistics. Meteorol. Appl., 15., 31-34. DOI: 10.1002/met.51 http://onlinelibrary.wiley.com/doi/10.1002/met.51/abstract

Murphy AH (1970) The ranked probability score and the probability score: A comparison. UDC 551.509.314. Vol. 98, No. 12. 917-924 http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.395.1780&rep=rep1&type=pdf

Murphy AH (1973) A new vector partition of the probability score. J. Appl. Meteor., 12, 595-600. doi: http://dx.doi.org/10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2

Murphy AH (1988) Skill scores based on the mean square error and their relationships to the correlation coefficient. Mon. Wea. Rev., 116, 2417-2424. doi: http://dx.doi.org/10.1175/1520-0493(1988)116<2417:SSBOTM>2.0.CO;2

Murphy AH (1993) What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8, 281-293. doi: http://dx.doi.org/10.1175/1520-0434(1993)008<0281:WIAGFA>2.0.CO;2

Murphy AH (1995) The coefficients of correlation and determination as measures of performance in forecast verification. Wea. Forecasting, 10, 681-688. doi: http://dx.doi.org/10.1175/1520-0434(1995)010<0681:TCOCAD>2.0.CO;2

Murphy AH (1996) General decompositions of MSE-based skill scores: Measures of some basic aspects of forecast quality. Mon. Wea. Rev., 124, 2353-2369. doi: http://dx.doi.org/10.1175/1520-0493(1996)124<2353:GDOMBS>2.0.CO;2

Murphy AH, Epstein ES (1989) Skill scores and correlation coefficients in model verification. Mon. Wea. Rev., 117, 572-581. doi: http://dx.doi.org/10.1175/1520-0493(1989)117<0572:SSACCI>2.0.CO;2

Richardson DS (2000) Skill and relative economic value of the ECMWF ensemble prediction system. Quart. J. Royal Met. Soc., 126, 649-667. DOI: 10.1002/qj.49712656313 http://onlinelibrary.wiley.com/doi/10.1002/qj.49712656313/abstract

Roulston MS, Smith LA (2002) Evaluating probabilistic forecasts using information theory. Mon. Wea. Rev., 130, 1653-1660. DOI: http://dx.doi.org/10.1175/1520-0493(2002)130<1653:EPFUIT>2.0.CO;2

Stephenson DB, Casati C, Ferro CAT, Wilson CA (2008) The extreme dependency score: a non-vanishing measure for forecasts of rare events. Meteorol. Appl., 15, 41-50. DOI: 10.1002/met.53 http://onlinelibrary.wiley.com/doi/10.1002/met.53/abstract

Stephenson DB, Coelho CAS, Jolliffe IT (2008) Two extra components in the Brier Score Decomposition, Wea. Forecasting, 23, pp 752-757. doi: http://dx.doi.org/10.1175/2007WAF2006116.1

Taylor KE (2001) Summarizing multiple aspects of model performance in a single diagram. J. Geophys. Res., 106 (D7), 7183-7192. DOI: 10.1029/2000JD900719 http://onlinelibrary.wiley.com/doi/10.1029/2000JD900719/abstract

Weigel AP, Liniger MA, Appenzeller C (2007) The discrete Brier and ranked probability skill scores. Mon. Wea. Rev., 135, 118–124. doi: http://dx.doi.org/10.1175/MWR3280.1

Weigel AP, Liniger MA, Appenzeller C (2007) Generalization of the Discrete Brier and Ranked Probability Skill Scores for Weighted Multimodel Ensemble Forecasts. Mon. Wea. Rev., 135, 2778-2785. doi: http://dx.doi.org/10.1175/MWR3428.1

Weigel AP, Mason SJ (2011) The generalized discrimination Score for ensemble forecasts. Mon. Wea. Rev.,, 139, 3069-3074. doi: http://dx.doi.org/10.1175/MWR-D-10-05069.1

Wilson LJ, Burrows WR, Lanzinger A (1999) A strategy for verification of weather element forecasts from an ensemble prediction system. Mon. Wea. Rev., 127, 956-970. doi: http://dx.doi.org/10.1175/1520-0493(1999)127<0956:ASFVOW>2.0.CO;2



5) List of published literature on S2S verification



5.1) Assessment of S2S systems forecast skill

Hudson D, Alves O, Hendon HH, Marshall AG (2011) Bridging the gap between weather and seasonal forecasting: Intraseasonal forecasting for Australia. Quart. J. Roy. Meteor. Soc., 137,673–689, doi:10.1002/qj.769 http://onlinelibrary.wiley.com/doi/10.1002/qj.769/abstract

Hudson D, Marshall AG, Yin Y, Alves O, Hendon HH (2013) Improving intraseasonal prediction with a new ensemble generation strategy. Mon Wea Rev, 141, 4429-4449. doi: http://dx.doi.org/10.1175/MWR-D-13-00059.1

Hudson D, Marshall AG, Alves O, Shi L, Young G (2015) Forecasting upcoming extreme heat on multi-week to seasonal timescales: POAMA experimental forecast products. Bureau Research Report, No. 1. Bureau of Meteorology, Australia (http://www.bom.gov.au/research/research-reports.shtml).

Hudson D, Marshall AG, Alves O, Young G, Jones D, Watkins A (2015) Forewarned is forearmed: Extended range forecast guidance of recent extreme heat events in Australia. Weather and Forecasting. doi: http://dx.doi.org/10.1175/WAF-D-15-0079.1

Jung T, Miller MJ, Palmer TN (2010) Diagnosing the Origin of Extended-Range Forecast Errors. Mon. Wea. Rev., 138, 2434–2446. doi: http://dx.doi.org/10.1175/2010MWR3255.1

Koster RD, Mahanama SPP, Yamada TJ, Balsamo G, Berg AA, Boisserie M, Dirmeyer PA, Doblas-Reyes FJ, Drewitt G, Gordon CT, Guo Z, Jeong J.-H, Lawrence DM, Lee W.-S, Li Z, Luo L, Malyshev S, Merryfield WJ, Seneviratne SI, Stanelle T, van den Hurk BJJM, Vitart F, Wood EF (2010) Contribution of land surface initialization to subseasonal forecast skill: First results from a multi-model experiment. Geophysical Research Letters, Vol. 37, L02402, doi:10.1029/2009GL041677, 2010 http://onlinelibrary.wiley.com/doi/10.1029/2009GL041677/full

Kumar A, Chen M, Wang W (2011) An analysis of prediction skill of monthly mean climate variability. Clim. Dyn., 37, 1119-1131. http://link.springer.com/article/10.1007%2Fs00382-010-0901-4

Li S, Robertson AW (2015) Evaluation of Submonthly Precipitation Forecast Skill from Global Ensemble Prediction Systems. Monthly Weather Review143:7, 2871-2889. doi:http://dx.doi.org/10.1175/MWR-D-14-00277.1

Marshall AG, Hudson D, Wheeler MC, Hendon HH, Alves O. (2012) Simulation and prediction of the Southern Annular Mode and its influence on Australian intra-seasonal climate in POAMA. Climate Dynamics. 38:2483-2502, doi:10.1007/s00382-011-1140-z. http://link.springer.com/article/10.1007%2Fs00382-011-1140-z

Marshall AG, Hudson D, Wheeler MC, Alves O, Hendon HH, Pook MJ, Risbey JS (2013) Intra-seasonal drivers of extreme heat over Australia in observations and POAMA-2. Climate Dynamics, doi: 10.1007/s00382-013-2016-1 http://link.springer.com/article/10.1007%2Fs00382-013-2016-1

Marshall AG, Hudson D, Hendon HH, Pook MJ, Alves O, Wheeler MC (2014) Simulation and prediction of blocking in the Australian region and its influence on intra-seasonal rainfall in POAMA-2. Clim. Dyn., 42, 3271-3288, DOI: 10.1007/s00382-013-1974-7 http://link.springer.com/article/10.1007%2Fs00382-013-1974-7

Mastrangelo D, Malguzzi P, Rendina C, Drofa O, Buzzi A (2012) First Outcomes from the CNR-ISAC monthly forecasting system Adv. Sci. Res., 8, 77–82, 2012
…(内容过长已截断)

------
导航页 打开网址