1. Model Evaluation Definition

Model Evaluation implies a broad assessment of model results, considering possible positive and negative outcomes, to understand the value of the model.

(Background: Atmospheric chemistry models try to provide a physically based approximation to real-world behaviour that serves to understand the real world and from there to predict future changes.)

Before comparing with observations, it’s important to choose the suitable observation data (time, location) though the observations are impossible to be at the exact same location and time as the modelled results. Two types of model results: 1. to model at gridded cells, this might cause representation error. 2. To evaluate the model, it’s better to use output at the same location as the receptors.

2. Methods

2.1 Visual Inspections (Figures): to check any prominent features

• temporal: intra-day (diurnal), day-to-day (synoptic), seasonal, and interannual (or long-term trends) – see Chun’s graph (also R script)
• spatial plot
• scatter plot (different site types using different colors)
• rural sites comparison (regional concentration)
• urban background sites (regional concentration, emissions)
• road side sites
• kerbside sited

(I normally use interactive timeplot to sum all scales)

2.2 Metrics:

• BIAS: the mean bias:
• RMSE: root mean squared error
• MAE: the mean absolute error
• MAD: the mean absolute deviation
• MNB: mean normalized bias
• MNAE:mean normalised absolute error
• MFB: mean fractional bias
• MFE: mean fractional error
• NMB: normalised mean bias

3. Possible Errors

3.1 Observations

3.1.1 Choice of observations

• Type
• Location

3.1.2 Errors in observations

(as a modeller, it’s not my job to spend much time to check errors in observations.)

3.2 Models

3.2.1 the model equations and underlying physical and chemical processes

3.2.2 the model parameters input to the equations

• meteorological
• emissions

3.2.3 the numerical approximations in solving the equations

3.2.4 coding errors

3.2.5 Representation error

the model may simulate a spatial average over a grid cell while the observations are from a particular location that may not reflect the grid cell average. (It may be necessary to exclude some sites from the comparison as non-representative. Representation error applies to temporal variability as well.)

4. Reference

Modelling of Atmospheric Chemistry Chapter 10 Atmospheric Observations and Model Evaluation Guy P.Brasseur and Daniel J. Jacob