Daily verification · methodology v0.3.1

How far off was each weather model?

Over the last 90 days, across 23 U.S. stations, the most accurate raw instantaneous temperature forecast 1 day ahead is GraphCast (IFS init): mean absolute error 2.63 °F , bias −1.58 °F, on n = 71 scored days.

Scope: raw model output on the native 0.25° grid. The headline variable is the instantaneous 2 m temperature at 00/06/12/18 UTC, scored against the observed 2 m temperature at the same instant (ASOS routine METAR) — the same four instants on both sides, so nothing in the number depends on a model's own diurnal amplitude. Errors are computed in °C and shown in °F (methodology v0.3.1).

Window
the last 90 days
Variable
instantaneous temperature
Init
00Z
Interpolation
bilinear
Data through
2026-08-30
Updated
2026-08-31T06:29:35+00:00
Next update
2026-08-31T11:00:00+00:00

Leader, lead day 1

2.63°F

MAE

GraphCast (IFS init)

Persistence baseline

3.21°F

yesterday's observation — the bar every model has to clear

Scored days

71

23 stations · 12 systems

in the last 90 days, lead day 1

Systems ranked

1/12

next update 2026-08-31T11:00:00+00:00

below n = 30: published, greyed, unranked

Mean absolute error with 95 % confidence intervals

Lead day 1 · the last 90 days · 23 stations pooled · 00Z · bilinear · instantaneous temperature. Whiskers are moving-block bootstrap intervals; models whose whiskers overlap are not distinguishable at this sample size. Baselines are drawn in grey and never ranked.

MAE by model with 95 % confidence intervals, lead day 1, 90d, 00Z, bilinear012345mean absolute error, °FGraphCast (IFS init)GraphCast (IFS init): 2.63 °F2.63FourCastNet v2 (GFS init)FourCastNet v2 (GFS init): 2.41 °F2.41ECMWF AIFS SingleECMWF AIFS Single: 2.41 °F2.41ECMWF IFS HRESECMWF IFS HRES: 2.6 °F2.6FourCastNet v2 (IFS init)FourCastNet v2 (IFS init): 2.67 °F2.67Aurora (IFS init)Aurora (IFS init): 2.87 °F2.87Pangu-Weather (GFS init)Pangu-Weather (GFS init): 2.89 °F2.89Pangu-Weather (IFS init)Pangu-Weather (IFS init): 2.96 °F2.96NCEP GFSNCEP GFS: 3.11 °F3.11Persistence (baseline)Persistence (baseline): 3.21 °F3.21GraphCast (GFS init)GraphCast (GFS init): 3.25 °F3.25Aurora (GFS init)Aurora (GFS init): 3.43 °F3.43

Source: CastCheck 0.1.0, methodology v0.3.1 · ECMWF open data, NOAA/NCEP GFS, NOAA/CIRA AIWP, NWS ASOS · data through 2026-08-30 · CC BY 4.0. Hover a bar for its value.

Lead day 1target = init + 1 d

permalink

The last 90 days, all 23 stations pooled, 00Z initialization, bilinear interpolation, instantaneous temperature against the observed 2 m temperature at the same instant (ASOS routine METAR). Lower MAE is better; bias is positive when the model is too warm. Every number links to its permanent page.

Skill is 1 − MAE ÷ MAE(persistence) computed on the days both have a value — the small print in that column is that denominator and the size of the intersection, so the number can be checked against the persistence row rather than contradicting it. The persistence row's own n is its whole record, marked all days. Skill, debiased (out-of-sample) repeats the score after removing a bias estimated on the 30 scored days before each day and applied forward, never on the day itself. An interval reads when the bootstrap could not run: fewer than 28 scored days or fewer than 4 blocks. Hit-rate intervals are Wilson score intervals, not bootstraps.

Lead day 1, the last 90 days, 00Z, bilinear, instantaneous temperature.
RankModel MAE °FBias °F ±3 °F Skill Skill, debiased (out-of-sample) nvs leader
1 GraphCast (IFS init)graphcast_ifs 2.63 −1.58[−1.70, −1.46] 63% +0.17[+0.12, +0.22]vs 3.18 (n=71) +0.45n=71 7122.7 stns · 7 QC lowest MAE in this view
FourCastNet v2 (GFS init)fourcastnet_gfs 2.41 −0.58 77% +0.21vs 3.04 (n=1) n=0 115 stns
ECMWF AIFS Singleaifs_single 2.41 −0.86[−0.95, −0.75] 70% +0.21[+0.16, +0.26]vs 3.06 (n=29) +0.38n=14 2922.7 stns · 2 QC
ECMWF IFS HRESifs_hres 2.60 −0.23[−0.39, −0.07] 68% +0.15[+0.10, +0.20]vs 3.06 (n=29) +0.28n=14 2922.7 stns · 2 QC
FourCastNet v2 (IFS init)fourcastnet_ifs 2.67 −0.52 66% +0.24vs 3.51 (n=2) n=0 219 stns
Aurora (IFS init)aurora_ifs 2.87 −1.80 56% +0.18vs 3.51 (n=2) n=0 219 stns
Pangu-Weather (GFS init)pangu_gfs 2.89 −0.16 60% +0.05vs 3.04 (n=1) n=0 115 stns
Pangu-Weather (IFS init)pangu_ifs 2.96 −1.26 58% +0.16vs 3.51 (n=2) n=0 219 stns
NCEP GFSgfs 3.11 +0.71[0.58, 0.85] 58% −0.02[−0.09, +0.04]vs 3.06 (n=29) +0.28n=14 2922.7 stns · 2 QC
Persistence (baseline)persistence · baseline 3.21 −0.05[−0.21, 0.09] 62% 74all days · 7 QC
GraphCast (GFS init)graphcast_gfs 3.25 −1.92 52% −0.07vs 3.04 (n=1) +0.47n=1 115 stns
Aurora (GFS init)aurora_gfs 3.43 −2.22 45% −0.13vs 3.04 (n=1) n=0 115 stns

+ model too warm − model too cold bias interval includes zero ★ lowest MAE = not distinguishable from the leader ▼ worse than the leader ▲ better than the leader — all three Holm-corrected within this table, over the family of comparisons against the leader; the uncorrected verdict for every pair is published in the pairwise table of each permanent link and in pairwise_latest.csv.gzn < 30 greyed and unranked CI is a 95 % moving-block bootstrap interval on the group's own days, and reads — when it could not be computed

Lead day 3target = init + 3 d

permalink

Same sampling as lead day 1. Skill is 1 − MAE ÷ MAE(persistence) on the days both have a value.

Lead day 3, the last 90 days, 00Z, bilinear, instantaneous temperature.
RankModel MAE °FBias °F Skill nvs leader
1 GraphCast (IFS init)graphcast_ifs 2.84 −1.55[−1.69, −1.41] +0.35[+0.28, +0.42]vs 4.38 (n=69) 6922.9 stns · 6 QC lowest MAE in this view
ECMWF AIFS Singleaifs_single 2.64 −0.75 +0.31vs 3.85 (n=27) 2722.6 stns · 2 QC
ECMWF IFS HRESifs_hres 2.89 +0.06 +0.25vs 3.85 (n=27) 2722.6 stns · 2 QC
NCEP GFSgfs 3.46 +0.99 +0.10vs 3.85 (n=27) 2722.6 stns · 2 QC
Persistence (baseline)persistence · baseline 4.40 −0.21[−0.65, 0.16] 74all days · 7 QC

Lead day 5target = init + 5 d

permalink

Same sampling as lead day 1. Skill is 1 − MAE ÷ MAE(persistence) on the days both have a value.

Lead day 5, the last 90 days, 00Z, bilinear, instantaneous temperature.
RankModel MAE °FBias °F Skill nvs leader
1 GraphCast (IFS init)graphcast_ifs 3.18 −1.67[−1.87, −1.46] +0.31[+0.21, +0.40]vs 4.61 (n=69) 6922.8 stns · 6 QC lowest MAE in this view
ECMWF AIFS Singleaifs_single 2.91 −0.79 +0.27vs 4.00 (n=25) 2522.6 stns · 2 QC
ECMWF IFS HRESifs_hres 3.13 +0.08 +0.22vs 4.00 (n=25) 2522.6 stns · 2 QC
NCEP GFSgfs 3.68 +0.60 +0.08vs 4.00 (n=25) 2522.6 stns · 2 QC
Persistence (baseline)persistence · baseline 4.59 −0.22[−0.82, 0.28] 74all days · 7 QC

Lead day 7target = init + 7 d

permalink

Same sampling as lead day 1. Skill is 1 − MAE ÷ MAE(persistence) on the days both have a value.

Lead day 7, the last 90 days, 00Z, bilinear, instantaneous temperature.
RankModel MAE °FBias °F Skill nvs leader
1 GraphCast (IFS init)graphcast_ifs 3.71 −1.83[−2.12, −1.54] +0.24[+0.16, +0.33]vs 4.91 (n=67) 6722.8 stns · 6 QC lowest MAE in this view
ECMWF AIFS Singleaifs_single 3.20 −1.18 +0.24vs 4.23 (n=23) 2322.6 stns · 2 QC
ECMWF IFS HRESifs_hres 3.43 +0.24 +0.19vs 4.23 (n=23) 2322.6 stns · 2 QC
NCEP GFSgfs 4.02 +0.32 +0.05vs 4.23 (n=23) 2322.6 stns · 2 QC
Persistence (baseline)persistence · baseline 4.90 −0.20[−0.99, 0.44] 74all days · 7 QC

The same four samples as a daily maximum and minimum

Lead day 1, the last 90 days, 00Z, bilinear. Here the forecast's max/min of the four samples is scored against the observation's max/min of the same four instants. Like for like: whatever the four-sample definition misses, it misses on both sides. The comparison against the true NWS daily extremes is a different question and lives on the station and permanent-link pages.

Sampled daily maximumtmax_s

RankModel MAE °FBias °F Skilln vs leader
1 GraphCast (IFS init)graphcast_ifs 2.63 −1.99[−2.16, −1.83] +0.17vs 3.16 (n=71) 71
ECMWF AIFS Singleaifs_single 2.02 −0.33[−0.44, −0.21] +0.28vs 2.80 (n=29) 29
ECMWF IFS HRESifs_hres 2.21 −0.10[−0.38, 0.19] +0.21vs 2.80 (n=29) 29
FourCastNet v2 (GFS init)fourcastnet_gfs 2.57 −1.89 +0.20vs 3.22 (n=1) 1
FourCastNet v2 (IFS init)fourcastnet_ifs 2.62 −1.21 +0.25vs 3.48 (n=2) 2
Pangu-Weather (GFS init)pangu_gfs 2.84 −0.33 +0.12vs 3.22 (n=1) 1
Pangu-Weather (IFS init)pangu_ifs 2.94 −0.80 +0.15vs 3.48 (n=2) 2
Persistence (baseline)persistence 3.18 −0.06[−0.25, 0.13] 74all days
NCEP GFSgfs 3.21 +1.74[1.41, 2.13] −0.15vs 2.80 (n=29) 29
Aurora (IFS init)aurora_ifs 3.27 −2.94 +0.06vs 3.48 (n=2) 2
GraphCast (GFS init)graphcast_gfs 4.31 −4.22 −0.34vs 3.22 (n=1) 1
Aurora (GFS init)aurora_gfs 4.58 −4.51 −0.42vs 3.22 (n=1) 1

Sampled daily minimumtmin_s

RankModel MAE °FBias °F Skilln vs leader
1 GraphCast (IFS init)graphcast_ifs 2.27 −1.00[−1.17, −0.82] +0.06vs 2.42 (n=71) 71
Pangu-Weather (IFS init)pangu_ifs 2.03 −0.74 +0.20vs 2.54 (n=2) 2
Aurora (IFS init)aurora_ifs 2.06 −0.42 +0.19vs 2.54 (n=2) 2
GraphCast (GFS init)graphcast_gfs 2.12 +0.34 +0.02vs 2.17 (n=1) 1
FourCastNet v2 (GFS init)fourcastnet_gfs 2.18 +0.60 −0.00vs 2.17 (n=1) 1
FourCastNet v2 (IFS init)fourcastnet_ifs 2.19 +0.29 +0.14vs 2.54 (n=2) 2
Aurora (GFS init)aurora_gfs 2.26 +0.03 −0.04vs 2.17 (n=1) 1
ECMWF IFS HRESifs_hres 2.27 −0.28[−0.46, −0.08] +0.03vs 2.34 (n=29) 29
ECMWF AIFS Singleaifs_single 2.32 −1.25[−1.47, −1.05] +0.01vs 2.34 (n=29) 29
Pangu-Weather (GFS init)pangu_gfs 2.32 +0.13 −0.07vs 2.17 (n=1) 1
Persistence (baseline)persistence 2.43 −0.05[−0.19, 0.09] 74all days
NCEP GFSgfs 2.57 +0.28[−0.04, 0.65] −0.10vs 2.34 (n=29) 29

Every model × every lead day

MAE in °F with the bias and n underneath, the last 90 days, 00Z, bilinear, instantaneous temperature. The sparkline is the same model's MAE across lead days 1–9 on a shared vertical scale.

Model d0d1d2d3d4d5d6d7d8d9 lead 1–9
ECMWF AIFS Singleaifs_single 2.31−0.92 · n=30 2.41−0.86 · n=29 2.55−0.82 · n=28 2.64−0.75 · n=27 2.74−0.78 · n=26 2.91−0.79 · n=25 3.07−1.01 · n=24 3.20−1.18 · n=23 3.50−1.47 · n=22 3.71−1.17 · n=21 ECMWF AIFS Single MAE by lead day 1 to 9
Aurora (GFS init)aurora_gfs 2.87−1.69 · n=2 3.43−2.22 · n=1 Aurora (GFS init) MAE by lead day 1 to 9
Aurora (IFS init)aurora_ifs 2.62−1.67 · n=3 2.87−1.80 · n=2 3.09−1.75 · n=1 Aurora (IFS init) MAE by lead day 1 to 9
FourCastNet v2 (GFS init)fourcastnet_gfs 2.65−1.02 · n=2 2.41−0.58 · n=1 FourCastNet v2 (GFS init) MAE by lead day 1 to 9
FourCastNet v2 (IFS init)fourcastnet_ifs 2.75−1.48 · n=3 2.67−0.52 · n=2 2.35−0.12 · n=1 FourCastNet v2 (IFS init) MAE by lead day 1 to 9
NCEP GFSgfs 2.82+0.59 · n=30 3.11+0.71 · n=29 3.23+0.79 · n=28 3.46+0.99 · n=27 3.46+0.79 · n=26 3.68+0.60 · n=25 3.82+0.67 · n=24 4.02+0.32 · n=23 4.24+0.21 · n=22 4.70+0.28 · n=21 NCEP GFS MAE by lead day 1 to 9
GraphCast (GFS init)graphcast_gfs 2.88−1.45 · n=2 3.25−1.92 · n=1 GraphCast (GFS init) MAE by lead day 1 to 9
GraphCast (IFS init)graphcast_ifs 2.37−1.31 · n=72 2.63−1.58 · n=71 2.74−1.64 · n=70 2.84−1.55 · n=69 3.01−1.57 · n=69 3.18−1.67 · n=69 3.44−1.76 · n=68 3.71−1.83 · n=67 3.99−1.89 · n=66 4.36−1.70 · n=65 GraphCast (IFS init) MAE by lead day 1 to 9
ECMWF IFS HRESifs_hres 2.38−0.33 · n=30 2.60−0.23 · n=29 2.74−0.07 · n=28 2.89+0.06 · n=27 2.97+0.10 · n=26 3.13+0.08 · n=25 3.40+0.25 · n=24 3.43+0.24 · n=23 3.68−0.05 · n=22 4.13+0.51 · n=21 ECMWF IFS HRES MAE by lead day 1 to 9
Pangu-Weather (GFS init)pangu_gfs 2.65−0.31 · n=2 2.89−0.16 · n=1 Pangu-Weather (GFS init) MAE by lead day 1 to 9
Pangu-Weather (IFS init)pangu_ifs 2.64−0.94 · n=3 2.96−1.26 · n=2 3.19−1.44 · n=1 Pangu-Weather (IFS init) MAE by lead day 1 to 9
Persistence (baseline)persistence 3.21−0.05 · n=74 4.04−0.12 · n=74 4.40−0.21 · n=74 4.44−0.20 · n=74 4.59−0.22 · n=74 4.81−0.20 · n=74 4.90−0.20 · n=74 4.94−0.23 · n=74 5.05−0.29 · n=74 Persistence (baseline) MAE by lead day 1 to 9

Where the errors are

Both maps show a fixed quantity per station. Neither shows which model wins where: on samples this short the per-station winner is mostly noise, and a map of winners would invite a comparison the data cannot carry.

ECMWF IFS HRES bias

Mean bias of the fixed reference model ECMWF IFS HRES (lead day 1, instantaneous temperature, 90d window, 00Z, bilinear). One model everywhere, so the colours compare stations, not models. Dot area grows with the number of scored days; fill is the mean bias (warm = model too warm, cool = too cold, grey = interval includes zero). The frame is a latitude/longitude graticule on an Albers projection.

Source: CastCheck · station coordinates NWS/NOAA · Albers conic projection · stations.csv. Hover a dot for its numbers.

model too cold interval includes zero model too warm dot area ∝ scored days

ECMWF IFS HRES bias as a table
StationModel MAE °FBias °F n
KATL Atlanta Hartsfield ECMWF IFS HRES3.08 −1.42 29
KAUS Austin Bergstrom ECMWF IFS HRES2.24 +1.12 29
KBOS Boston Logan ECMWF IFS HRES2.55 −1.05 26
KDCA Washington Reagan ECMWF IFS HRES3.02 −1.71 29
KDEN Denver Intl ECMWF IFS HRES3.78 +2.10 28
KDFW Dallas-Fort Worth ECMWF IFS HRES2.34 +2.25 29
KEWR Newark Liberty ECMWF IFS HRES3.03 −2.10 29
KIAH Houston Bush ECMWF IFS HRES2.02 +0.18 29
KLAS Las Vegas Harry Reid ECMWF IFS HRES3.52 −2.18 28
KLAX Los Angeles Intl ECMWF IFS HRES1.31 +0.55 28
KMIA Miami Intl ECMWF IFS HRES3.05 −1.99 29
KMSP Minneapolis-St Paul ECMWF IFS HRES2.10 −0.34 29
KMSY New Orleans Intl ECMWF IFS HRES2.43 +0.78 29
KNYC New York Central Park ECMWF IFS HRES2.30 −0.12 29
KOKC Oklahoma City ECMWF IFS HRES4.03 +3.22 29
KORD Chicago O'Hare ECMWF IFS HRES2.17 −1.09 29
KPHL Philadelphia Intl ECMWF IFS HRES3.39 −2.80 29
KPHX Phoenix Sky Harbor ECMWF IFS HRES2.84 +0.28 28
KSAN San Diego Lindbergh ECMWF IFS HRES2.13 −2.02 28
KSAT San Antonio Intl ECMWF IFS HRES1.70 +0.90 29
KSEA Seattle-Tacoma ECMWF IFS HRES2.52 +1.23 28
KSFO San Francisco Intl ECMWF IFS HRES1.85 −1.32 28
KTTN Trenton Mercer ECMWF IFS HRES2.23 +0.15 29

All-model mean bias

Mean bias averaged over every scored model at each station (lead day 1, instantaneous temperature, 90d window, 00Z, bilinear). A station that is cold for all of them is a station property — elevation, coastline, a grid cell that is partly sea — not a model ranking. Dot area grows with the number of scored days; fill is the mean bias (warm = model too warm, cool = too cold). The frame is a latitude/longitude graticule on an Albers projection.

Source: CastCheck · station coordinates NWS/NOAA · Albers conic projection · stations.csv. Hover a dot for its numbers.

model too cold interval includes zero model too warm dot area ∝ scored days

All-model mean bias as a table
StationModels MAE °FBias °F n
KATL Atlanta Hartsfield 11 models3.25 −2.51 71
KAUS Austin Bergstrom 11 models3.13 +0.14 70
KBOS Boston Logan 7 models2.50 −0.91 64
KDCA Washington Reagan 11 models2.97 −1.81 71
KDEN Denver Intl 7 models3.13 +0.96 69
KDFW Dallas-Fort Worth 11 models2.22 −0.79 70
KEWR Newark Liberty 11 models2.63 −1.01 71
KIAH Houston Bush 11 models2.19 −0.30 71
KLAS Las Vegas Harry Reid 7 models5.20 −4.75 69
KLAX Los Angeles Intl 7 models1.90 +0.49 70
KMIA Miami Intl 11 models2.55 −1.20 71
KMSP Minneapolis-St Paul 11 models3.06 −2.00 71
KMSY New Orleans Intl 11 models2.60 −1.85 71
KNYC New York Central Park 11 models2.37 +0.23 71
KOKC Oklahoma City 11 models3.59 +1.00 71
KORD Chicago O'Hare 11 models2.25 −1.17 71
KPHL Philadelphia Intl 11 models2.79 −1.85 71
KPHX Phoenix Sky Harbor 7 models4.22 −3.07 70
KSAN San Diego Lindbergh 7 models2.49 −1.73 70
KSAT San Antonio Intl 11 models2.62 −1.12 71
KSEA Seattle-Tacoma 7 models2.53 −0.11 70
KSFO San Francisco Intl 7 models2.18 −1.62 70
KTTN Trenton Mercer 11 models3.45 +2.00 71

Data availability

Each model is scored only over its own available period (2026-01-01 → 2026-08-30); the windows above are therefore not identical across models. Pairwise comparisons on the permanent-link pages use common days only.

ModelPeriod Scored daysCoverage
ECMWF AIFS Single 2026-08-02 → 2026-08-3029
2026-01-012026-08-30
Aurora (GFS init) 2026-08-30 → 2026-08-301
2026-01-012026-08-30
Aurora (IFS init) 2026-08-29 → 2026-08-302
2026-01-012026-08-30
FourCastNet v2 (GFS init) 2026-08-30 → 2026-08-301
2026-01-012026-08-30
FourCastNet v2 (IFS init) 2026-08-29 → 2026-08-302
2026-01-012026-08-30
NCEP GFS 2026-08-02 → 2026-08-3029
2026-01-012026-08-30
GraphCast (GFS init) 2026-01-02 → 2026-08-3034
2026-01-012026-08-30
GraphCast (IFS init) 2026-01-02 → 2026-08-30198
2026-01-012026-08-30
ECMWF IFS HRES 2026-08-02 → 2026-08-3029
2026-01-012026-08-30
Pangu-Weather (GFS init) 2026-08-30 → 2026-08-301
2026-01-012026-08-30
Pangu-Weather (IFS init) 2026-08-29 → 2026-08-302
2026-01-012026-08-30
Persistence (baseline) 2026-01-01 → 2026-08-30215
2026-01-012026-08-30

Stations23

full station table
  • KNYC — New York Central Park
  • KEWR — Newark Liberty
  • KPHL — Philadelphia Intl
  • KTTN — Trenton Mercer
  • KBOS — Boston Logan
  • KDCA — Washington Reagan
  • KATL — Atlanta Hartsfield
  • KMIA — Miami Intl
  • KORD — Chicago O'Hare
  • KMSP — Minneapolis-St Paul
  • KDFW — Dallas-Fort Worth
  • KIAH — Houston Bush
  • KAUS — Austin Bergstrom
  • KSAT — San Antonio Intl
  • KMSY — New Orleans Intl
  • KOKC — Oklahoma City
  • KDEN — Denver Intl
  • KPHX — Phoenix Sky Harbor
  • KLAS — Las Vegas Harry Reid
  • KLAX — Los Angeles Intl
  • KSAN — San Diego Lindbergh
  • KSFO — San Francisco Intl
  • KSEA — Seattle-Tacoma