Atmospheric and Climate Sciences
Vol.3 No.3(2013), Article ID:34079,9 pages DOI:10.4236/acs.2013.33037

Comparative Skill of Numerical Weather Forecasts in Eastern Amazonia

Bergson Cavalcanti de Moraes1*, Douglas Batista da Silva Ferreira1, Luiz Gylvan Meira Filho1, Juarez Ventura de Oliveira1, Everaldo Barreiros de Souza1, Pedro Pereira Ferreira Júnior1, Renata Kelen Cardoso Câmara1, Edson José P. da Rocha2, João Batista M. Ribeiro2

1Vale Institute of Technology, Belém, Brazil

2Federal University of Para (UFPa), Belém, Brazil

Email: *

Copyright © 2013 Bergson Cavalcanti de Moraes et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Received March 27, 2013; revised May 1, 2013; accepted May 10, 2013

Keywords: Meteorological Models; Mean Error; Root Mean Square Error; Precipitation; Eastern Amazonia


The present study evaluates the performance of three numerical weather forecasting models: Global Forecast System (GFS), Brazilian Regional Atmospheric Modelling System (BRAMS) and ETA Regional Model (ETA), by means of the Mean Error (ME) and the Root Mean Square Error (RMSE), during the most rainy four months period (January to April 2012) on Eastern Amazonia. The models displayed errors of superestimation and underestimation with respect to the observed precipitation, mainly over center-north of Pará and all of Amapá, where the precipitation is higher. Among the analyzed models, GFS shows the best performance, except during January and March, when the model to underestimated precipitation, possibly due to the anomalously high values recorded.

1. Introduction

Meteorological forecasting is a complex task, but such complexity has decreased, over the years, making the numerical forecast faster and more practical [1,2], with higher success rate for several variables. In particular, precipitation is one variable that attracts more interest due to its relevance, not only for climate, but also for several parts of society, such as mining, economics, industry, agriculture, and others [3-7].

Many social and economic sectors in Brazil presently use numerical weather forecasts for strategic planning of their activities [8-10]. The main meteorological centers in Brazil use operational models for numerical weather forecasting capable of accurate weather predictions, but in Amazonia, the largest tropical forest in the world, those models do not have a good parameterization of some essential physical processes to represent the atmospheric mechanisms that cause precipitation over that region [11].

Against that background, the aim of this study is to evaluate the performance of the foremost meteorological models that forecast precipitation, for a horizon of 48 hours, during the rainier months (January to April) of Eastern Amazonia.

2. Data and Methodology

2.1. Study Area

The region of interest is the Eastern Amazonia, includes the states of Pará, Amapá, Tocantins and Maranhão. It is located between the longitudes 10S and 2N, and latitudes 60W and 42W. This area is covered mostly with tropical rain forest, with a diversified geography of mountains, rich hydrography and intense physical process of microscale and biosphere—atmosphere interactions, that makes it a difficult region for parameterization [12,13].

2.2. Observational Data

The monthly-accumulated precipitation for January, February, March and April 2012 was obtained from the Surface Synoptic Observing Stations (SYNOP), Data Collection Platforms (PCD), conventional gauges from Weather, Climate and Hydric Resources Monitoring Program (PMTCRH), and from the Tropical Rainfall Measuring Mission (TRMM) satellites. This information was grouped with a statistical technique called Merge [14].

2.3. Meteorological Models

Three numerical weather prediction models were used: the Brazilian Regional Atmospheric Modelling System (BRAMS), the ETA Regional Model (ETA) and the Global Forecast System (GFS).

Originated from the Regional Atmospheric Modelling System (RAMS), the BRAMS model had its parameterization adapted to suit Brazilian needs, being operative on the Center for Weather Forecast and Climatic Studies (CPTEC) and on Amazonia Protection System (SIPAM) in Brazil. RAMS is a numerical model designed to simulate atmospheric circulations at many scales and is equipped with a multiple grid nesting scheme which allows the model equations to be solved simultaneously on any number of two-way interacting computational meshes of increasing spatial resolution [15]. This model has horizontal resolution of 20 km and 38 vertical levels. The output comes on every 12 h, with data assimilation every 06 h.

The ETA Model is a state-of-the-art atmospheric model used for research and operational purposes. The model is a descendent of the earlier HIBU (Hydrometeorological Institute and Belgrade University) model, developed in the seventies in the former Yugoslavia. The name of the model derives from the Greek letter (ETA), which denotes the vertical coordinate [16], one of the model features. The model ETA [17,18], also running operatively on CPTEC, is a mesoscale, non-hydrostatic model with horizontal resolution of 15 km and 42 vertical layers. Assimilates data every, with output every 12 h.

One of the National Oceanic and Atmospheric Administration (NOAA) operational models, the GFS has horizontal resolution of 0.5˚ (≈55 km) and 64 vertical layers (sigma-pressure hybrid). The main time integration is leapfrog for nonlinear advection terms, and semiimplicit for gravity waves and for zonal advection of vorticity and moisture. The forecast comes at every 12 h, with data assimilation every 03 h. It is a global spectral numerical model based on the primitive dynamical equations that includes a suite of parameterizations for atmospheric physics [19-21].

2.4. Statistical Methodology

The model evaluation was performed by calculating the Mean Error (ME) and the Root Mean Square Error (RMSE) between the forecast and the observed precipitation. This methodology has been already applied to many others studies, with consistent results regarding meteorological model evaluation [22].

The ME (1) is given by the mean difference between the forecast and the observed values, indicating the systematic error. Positive error values express overestimation of the observed precipitation and negative values underestimation. When the forecast is perfect, the ME is equal to zero. The ME formula is:


where Pn is the forecast value, On the observed one, and N the number of observations. The ME result has the same unity as the studied variable, in this case, millimeters of precipitation.

The other statistical parameter addressed on this work, is the RMSE (2), that is a frequently used measure of the difference between values predicted by a model and the values actually observed from the environment that is being modeled. These individual differences are also called residuals, and the RMSE serves to aggregate them into a single measure of predictive power. Unlike ME, this parameter gives information regarding the total amplitude of the error, disregarding the signal of positive or negative. The formula that defines the RMSE is:


3. Results and Discussion

The error analyses are displayed in this section, preceded by a discussion about the main meteorological systems responsible for the spatial and temporal distribution of the accumulated precipitation in each month (Figure 1), where dark green color represents the largest amounts of precipitation. On the ME figures, positive values are represented by colors of red tones, and negative values by colors of blue tones. The dark red colors on the RMSE indicate larger errors.

During January 2012 it was observed the presence of three atmospheric mechanisms inductors of precipitation: the Intertropical Convergence Zone (ITCZ) [23-25] over the equatorial Atlantic; the South Atlantic Convergence Zone (SACZ) [26,27], extending from Southern Amazonia to Southeast Brazil, and Upper Tropospheric Cyclonic Vortices (UTCV) [28] over Northeast Brazil. These systems were responsible for a maximum of precipitation over Amapá and Pará (over 400 mm; Figure 1(a)), while the precipitation over Maranhão was reduced (less than 200 mm, except for the South of São Luís). Such pattern was due to the displacement of the UTCV to the Eastern Amazonia, which generates subsiding air and inhibition of precipitation over Maranhão.

The ME and RMSE of the models produced distinct results (Figure 2). The GFS scored the smallest relative values of ME during the forecast horizons, tending more to an underestimation of the precipitation. However, the

Figure 1. Acumulated precipitation (mm) over Eastern Amazon for: (a) January; (b) February; (c) March; (d) April, of 2012.

magnitude of the RMSE was larger, mostly over Pará. The BRAMS and ETA models, which overestimated the precipitation, show similar results regarding the RMSE, with higher values located on the border between Pará and Maranhão. Comparing these models, BRAMS and ETA scored the smallest RMSE. As for the forecast horizons, one can notice that the errors are systematic, so the same pattern seen for 24 h can be observed for 48 h.

During February 2012, the UTCV stopped influencing the weather and Frontal Systems (FS) advances to Southern Amazon was reduced. The ITCZ was therefore the main atmospheric system acting on the region, especially over northeastern Pará and Amapá. The highest precipitations were restricted to Marajó Island and northeastern Pará, where more than 400mm was recorded (Figure 1(b)). The development of Squall Lines (SL) along the Maranhão shore also contributed to the highest accumulated precipitation.

The spatial pattern of ME (Figure 3) shows that the GFS underestimates the precipitation over the north of Pará and Amapá, where the largest amounts of rain were recorded. On the other regions, the model shows ME near zero, which represents an accurate forecast.

In general, BRAMS and ETA presented more areas with under and overestimation errors. At Maranhão, where the total of accumulated precipitation ranged spatially between 150 mm and 350 mm, the RMSE was smaller than the other regions with high precipitation. The error magnitude was higher over Pará, for both forecasts horizons.

March is characterized by the peak of the rainy season (Figure 1(c)), so that many counties experiences extreme events. In addition, Northeastern Pará, Marajó Island and part of the Lower Amazon River received precipitation above the climatology, e.g. the total of precipitation at Belém was above 700 mm, in March 2012. The elevated sea surface temperature of the Atlantic contributed to sustain an intense ITCZ and FS started to impact Southern Amazonia.

With the rising of precipitation, RMSE and ME also increased (Figure 4). GFS continued to display underestimated values over Pará. Still, ETA and BRAMS displayed overestimation for the 24 h forecast, but it reduced for the 48 h horizon, mostly at Pará, Amapá and Northern Tocantins.

The RMSE analysis ratifies that GFS was the one with the largest errors on almost all region during March.

Figure 2. ME and RMSE of GFS (left column), BRAMS (middle column) and ETA (right cloumn) for the forecast of 24 h and 48 h. Mean of January of 2012.

BRAMS and ETA had a better performance; only Northern Pará and Maranhão shore displayed high values (above 20 mm). On the other regions, the less amount of precipitation was determinant for the small value of RMSE, mainly for the 24 h forecast.

It is noteworthy that March was a particularly rainy month, and GFS performance may have suffered from this anomaly. In addition, BRAMS and ETA predicted even higher values for areas with extreme events.

During April 2012 (Figure 1(d)), the ITCZ was virtually the only large-scale atmospheric system influencing Amazonia. Therefore, the most affected areas were in Northern Pará and Amapá, with accumulated values above 400 mm. With these systems displaced further North, other regions, such as the center of Maranhão, received small amounts of precipitation, with values below 150 mm.

The spatial distribution of ME continues the trend of

Figure 3. ME and RMSE of GFS (left column), BRAMS (middle column) and ETA (right column) for the forecast of 24 h and 48 h. Mean of February of 2012.

the other months of the rainy season (Figure 5). The GFS continues to underestimate the precipitation for the 24 h and 48 h forecasts, indicating that the model follows a climatological pattern. The worst performance of BRAMS occurred in April, when it overestimated precipitation in all Eastern Amazonia. One possible answer to BRAMS errors may be the fact that its output represents the mean value of the grid cell, while precipitation shows a high spatial variability [29]. ETA is also shown overestimation, especially in Maranhão, where the model overestimated the observed precipitation for both 24 h and 48 h forecasts.

The RMSE analysis shows that GFS scored the smallest values, mainly over Southeastern Pará, centersouth Maranhão and Northern Tocantins (Figure 5). Similar behavior could be noticed for BRAMS and ETA, where the highest values were in Pará and Amapá. The smallest values of error over Maranhão indicates that GFS repre-

Figure 4. ME and RMSE of GFS (left column), BRAMS (middle column) and ETA (right cloumn) for the forecast of 24 h and 48 h. Mean of March of 2012.

sented well the precipitation at this state, as well as in Northern Tocantins.

4. Conclusions

The ME and RMSE analyses, showed that none of the models could perfectly forecast precipitation in Eastern Amazonia. The BRAMS and ETA models displayed a tendency of overestimation of the observed precipitation, while GFS tended to underestimate it, mostly when the total amount of rain was above the climatology.

The regions with the highest errors for both forecast horizons were Lower Amazonas River, Marajó Island and Northeast Pará. Such results may be associated with the high precipitation during these four months; also in adtition, the density of rain gauges is low, making the interpolation of observed data more uncertain.

In general, the GFS showed good forecasts for Maranhão, Northern Tocantins and Southeast Pará. BRAMS and ETA also had good performances for the same regions, in spite of the tendency for overestimation.

Figure 5. ME and RMSE of GFS (left column), BRAMS (middle column) and ETA (right cloumn) for the forecast of 24 h and 48 h. Mean of April of 2012.

5. Acknowledgements

The authors would like to thank NOAA and CPTEC for the data, Vale Institute of Technology for the grants, equipment and physical structure for the development of this work, which was implemented under the “Integrated monitoring and forecasting of meteorological and hydrological events for Vale, in the Eastern Amazonia” Project.


  1. J. Michalakes, J. Dudhia, D. Gill, T. Henderson, J. Klemp, W. Skamarock and W. Wang, “The Weather and Research and Forecast Model: Software Architecture and Performance,” Proceedings of the 11th Ecmwf Workshop on High Performance Computing in Meteorology, Reading, 25-29 October 2004, pp. 156-168.
  2. B. Rockel, A. Will and A. Hense, “The Regional Climate Model Cosmo-Clm (Cclm),” MeteorologischZeitung, Vol. 17, 2008, pp. 347-348. doi:10.1127/0941-2948/2008/0309
  3. L. B. Lave, “The Value of Better Weather Information for the Raisin Industry,” Econometrica, Vol. 31, No. 1-2, 1963, pp. 151-164. doi:10.2307/1910954
  4. D. A. Olson, N. Junker and B. Korty, “Evaluation of 33 Years of Quantitative Precipitation Forecasting at the Nmc,” Weather and Forecasting, Vol. 10, No. 3, 1995, pp. 498-511. doi:10.1175/1520-0434(1995)010<0498:EOYOQP>2.0.CO;2
  5. J. W. Mjelde, H. S. J. Hill and J. F. Griffiths, “A Review of Current Evidence on Climate Forecasts and Their Economic Effect in Agriculture,” American Journal of Agricultural Economy, Vol. 80, No. 5, 1998, pp. 1089-1095.
  6. Y. Zhu, Z. Toth, R. Wobus, D. Richardson and K. Mylne, “On the Economic Value of Ensemble Based Weather Forecasts,” Bulletin of American Meteorological Society, Vol. 83, No. 1, 2002, pp. 73-83. doi:10.1175/1520-0477(2002)083<0073:TEVOEB>2.3.CO;2
  7. M. J. Best, “Progress towards Better Weather Forecast for City Dwellers: From Short Range to Climate Change,” Theoretical and Applied Climatology, Vol. 84, No. 1-3, 2006, pp. 47-55. doi:10.1007/s00704-005-0143-2
  8. A. Arakawa and V. R. Lamb, “Computational Design of the Basic Dynamical Processes of the UCLA General Circulation Model,” Methods in Computational Physics, Vol. 17, 1977, pp. 174-267. doi:10.1016/B978-0-12-460817-7.50009-4
  9. S. Hastenrath and L. Greischar, “Further Work on the Prediction of Northeast Brazil Rainfall Anomalies,” Journal of Climate, Vol. 6, No. 4, 1993, pp. 743-758. doi:10.1175/1520-0442(1993)006<0743:FWOTPO>2.0.CO;2
  10. M. A. F. Silva Dias, S. Rutledge, P. Kabat, P. L. Silva Dias, C. Nobre, G. Fisch, A. J. Dolman, E. Zipser, M. Garstang, A. O. Manzi, J. D. Fuentes, H. R. Rocha, J. Marengo, A. Plana-Fattori, L. D. A. Sa´, R .C. S. Alvala´, M. O. Andreae, P. Artaxo, R. Gielow and L. Gatti, “Cloud and Rain Processes in a Biosphere-Atmosphere Interactions Context in the Amazon Region,” Journal of Geophysical Research, Vol. 107, No. D20, 2002, pp. LBA 39-1-LBA 39-18.
  11. P. Bechtold, J. P. Chaboureau, A. Baljaars, A. K. Betts, M. Köhler, M. Miller and J. L. Redelsperger, “The Simulation of the Diurnal Cycle of Convective Precipitation over Land in a Global Model,” Quarterly Journal of the Royal Meteorological Society, Vol. 131, 2004, pp. 3119- 3137. doi:10.1256/qj.03.103
  12. W. Grabowski, “Coupling Cloud Processes with the LargeScale Dynamics Using the Cloud-Resolving Convection Parameterization (Crcp),” Journal of The Atmospheric Sciences, Vol. 58, No. 9, 2001, pp. 978-997. doi:10.1175/1520-0469(2001)058<0978:CCPWTL>2.0.CO;2
  13. D. K. Adams, E. P. Souza and A. A. Costa, “Convecção Úmida Na Amazônia: Implicações Para Modelagem Numérica,” Revista Brasileira de Meteorologia, Vol. 24, No. 2, 2009, pp. 168-178. doi:10.1590/S0102-77862009000200006
  14. J. R. Rozante, D. S. Moreira, L. G. G. Gonçalves and D. A. Vila, “Combining Trmm and Surface Observation Precipitation: Technique and Validation over South America,” Weather and Forecasting, 2010. doi:10.1175/2010WAF2222325.1
  15. S. R. Freitas, K. M. Longo, M. A. F. Silva Dias, R. Chatfield, P. L. Silva Dias, P. Artaxo, M. O. Andreae, G. Grell, L. Rodrigues, A. Fazenda and J. Paneta, “The Coupled Aerosol and Tracer Transport Model to the Brazilian Developments on the Regional Atmospheric Modeling System (Catt-Brams)—Part 1: Model Description and Evaluation,” Atmospheric Chemistry and Physics Discussions, Vol. 7, 2009, pp. 8525-8569.
  16. F. Mesinger, “A Blocking for Representation of Mountains in Atmospheric Models,” Rivistadi Meteorologia Aeronautica, Vol. 44, No. 1-4, 1984, pp. 195-202.
  17. S. C. Chou, J. F. Bustamante and J. L. Gomes, “Evaluation of Seasonal Precipitation Forecasts over South America Using Eta Model,” Nonlinear Process in Geophysics, Vol. 12, No. 4, 2005, pp. 537-555. doi:10.5194/npg-12-537-2005
  18. F. Mesinger, S. C. Chou, J. L. Gomes, D. Jovic, P. Bastos, J. F. Bustamante, L. Lazic, A. A. Lyra, S. Morelli, I. Ristic and K. Veljovic, “An Upgraded Version of the Eta Model,” Meteorology and Atmospheric Physics, Vol. 116, No. 3-4, 2012, pp. 63-79. doi:10.1007/s00703-012-0182-z
  19. F. Yang, H.-L. Pan, S. Krueger, S. Moorthi and S. Lord, “Evaluation of NCEP Global Forecast System at the ARM SGP Site,” Weather Review, Vol. 134, No. 12, 2006, pp. 3668-3690. doi:10.1175/MWR3264.1
  20. J. Hamil and J. Juras, “Measuring Forecast Skill: Is It Real Skill or Is It the Varying Climatology?” Quarterly Journal of the Royal Meteorological Society, Vol. 132, No. 621C, 2007, pp. 2905-2923. doi:10.1256/qj.06.25
  21. J. Han and H. L Pan, “Revision of Convection and Vertical Diffusion Schemes in the NCEP Global Forecast System,” Weather and Forecasting, Vol. 26, No. 4, 2011, pp. 520-533. doi:10.1175/WAF-D-10-05038.1
  22. B. G. White, J. Peagle, W. J. Steenburgh, J. D. Horel, R. T. Swanson, L. K. Cook, D. J. Onton and J. G. Miles, “Short-Term Forecast Validation of Six Models,” Weather and Forecasting, Vol. 14, 1990, pp. 84-108. doi:10.1175/1520-0434(1999)014<0084:STFVOS>2.0.CO;2
  23. S. Hastenrath and P. Lamb, “Climatic Atlas of the Tropical Atlantic and Eastern Pacific Oceans,” University of Wisconsin Press, Wisconsin, 1977.
  24. P. Nobre and J. Shukla “Variations of Sst, Wind Stress and Rainfall over the Tropical Atlantic and South America,” Journal of Climate, Vol. 9, No. 10, 1996, pp. 2464- 2479. doi:10.1175/1520-0442(1996)009<2464:VOSSTW>2.0.CO;2
  25. E. B. Souza, M. T. Kayano and T. Ambrizzi, “The Regional Precipitation over the Eastern Amazon/Northeast Brazil Modulated by Tropical Pacific and Atlantic SST Anomalies on Weekly Timescale,” Revista Brasileira de Meteorologia, Vol. 19, No. 2, 2004, pp. 113-122.
  26. S. N. Figueroa and C. A. Nobre, “Precipitation Distribution over Central and Western Tropical South America,” Climanálise, Vol. 5, No. 6, 1990, pp. 36-45.
  27. L. M. V. Carvalho, C. Jones and B. Liebmann, “The South Atlantic Convergence Zone: Persistence, Form, Extreme Precipitation and Relationships with Intraseasonal Activity,” Journal of Climate, Vol. 17, No. 1, 2004, pp. 88-108. doi:10.1175/1520-0442(2004)017<0088:TSACZI>2.0.CO;2
  28. V. E. Kousky and M. A. Gan, “Upper Tropospheric Cyclonic Vortices in the Tropical South Atlantic,” Tellus, Vol. 33, No. 6, 1981, pp. 538-551. doi:10.1111/j.2153-3490.1981.tb01780.x
  29. J. F. Leivas, G. G. Ribeiro, I. Saraiva, J. S. E. Santo, M. B. Souza and J. Rocha Filho, “Avaliacão Dos Prognósticos De Precipitacão Simulada Pelo Modelo Brams Na Amazonia Ocidental Na Estacão Chuvosa,” Acta Amazonica, Vol. 41, 2011, pp. 347-354.


*Corresponding author.