Model Assessment Summary

Summary plots of model comparisons with the main types of data are shown in this section. The plots show the skill score (see Methodology) from each model-data comparison shown in the subpages. The skill score is not a perfect metric as no metric is, but it is something that can be calculated for each model-data comparison.

Overall, the results point to CIOFS not having enough freshwater input into the system to have realistic salinity variability, let alone accurate salinity. NWGOA may have enough freshwater input, leading to realistic variability, but not enough horizontal resolution to accurate calculate its transport.

The major seasonal sea water temperature signal is well-captured in both models, but temperature anomaly is not well-captured. CIOFS is better at capturing cross-channel variation than NWGOA, and NWGOA better captures vertical variation in temperature.

Both models capture the tidal sea surface height well, and mostly do well at subtidal sea surface height as well.

Both model capture along-channel surface currents in two channels, but mostly do poorly with across-channel currents, and subtidal in either direction. CIOFS performs better than NWGOA for horizontal speed so is capturing the kinetic energy in the model but missing the directionality which may be related to local bathymetric gradients that are not in the bathymetry data.

Salinity

A major question of this report is: given that the CIOFS model is forced with only gauged river data, which is known to be missing freshwater compared to a watershed model like in NWGOA [DHH+20b], how well does the CIOFS model capture the salinity? Horizontal processes should be captured better in CIOFS than NWGOA since CIOFS has higher horizontal resolution, but the model also needs to have adequate freshwater inflow to get it right.

The salinity is poorly captured by both CIOFS and NWGOA, as shown in the summary salinity plots for the moorings (Fig. 33, Fig. 34, Fig. 35) with skill scores that are mostly negative. Skill scores less than 1 indicate that the model is actively disagreeing with the data, and less than -1 is equivalently terrible performance. However, we do see that the NWGOA model skill scores are generally less negative that the CIOFS model, and in digging in more one can see that while the NWGOA skill scores are bad, the actual comparisons look more realistic than with the CIOFS model.

For example, in a typical poor-performing model-data comparison as shown in Fig. 10 (skill score of <-1), the CIOFS salinity is static relative to the data. On the other hand, NWGOA salinity similarly has a negative skill score in the comparison (-0.6), but the behavior demonstrated from the model is much more realistic in Fig. 11. Analogous plots show for the same time period that a similar trend holds at depth (Fig. 12 and Fig. 13)

Fig. 10 Example of CIOFS surface water time series comparison with salinity data (Station homer-dolphin-surface-water-q).

Fig. 11 Example of NWGOA surface water time series comparison with salinity data (Station homer-dolphin-surface-water-q).

Fig. 12 Example of CIOFS deep water time series comparison with salinity data (Station nerrs_kachdwq).

Fig. 13 Example of NWGOA deep water time series comparison with salinity data (Station nerrs_kachdwq).

Consistent with CIOFS results compared with salinity stations at two depths as above, CIOFS often has a homogeneous salinity profile when it should have a surface fresh layer. Fig. 95 shows the skill scores across all CTD profile comparisons. Again, generally both models perform poorly for salinity. However, the Kachemak Bay region does show better performance for NWGOA as compared with CIOFS. As a representative example, the figure below shows a uniform profile of salinity from CIOFS (left) whereas while NWGOA (right) does not capture the data very well, it does become fresher at the surface and is closer to the average salinity in the data.

Similarly, we see in the skill scores from comparisons with CTD transects (Fig. 38) that neither model does great, but NWGOA performs well more often than CIOFS. As before, the CIOFS salinity field (Fig. 14) in both space and time tends to be too uniform compared to the data whereas NWGOA (Fig. 15) shows more variability and is able to capture this transect pretty well (skill score of 0.8).

Fig. 14 Example of CIOFS CTD transect comparison with salinity data (from the OTF KBNERR project of repeated transects from Anchor Point).

Fig. 15 Example of NWGOA CTD transect comparison with salinity data (from the OTF KBNERR project of repeated transects from Anchor Point).

Temperature

The large seasonal temperature range often makes the full temperature relatively easy to capture by a numerical model. Accordingly, both the CIOFS and NWGOA models often capture the full tidal or subtidal temperature time series shown in the moorings data (Fig. 30). However, they do have differences. While the CIOFS model has the full seasonal temperature range at the head of the Inlet (NOAA 9455920, Fig. 16) and the south end of Kodiak Island (NOAA 9457804), it is a bit too cool relative to the data in the summer west of Kodiak Island (WMO 46077) and in the spring and summer east of Kodiak Island (NOAA 9457292). In Kachemak Bay, the CIOFS model is more likely to be somewhat cold relative to the data throughout the year at the surface (Fig. 17) or in the winter at depth (Fig. 18).

The NWGOA model is too cold by almost 10 degrees at the head of the Inlet (NOAA 9455920, Fig. 16), but that grid location is suspect in several of the measures so it may not be representative. NWGOA accurately captures the seasonal temperature range throughout the rest of the Inlet, including in Kachemak Bay at the surface (Fig. 20) and at depth (Fig. 21).

Fig. 16 Example of CIOFS surface water subtidal time series comparison at the head of the Inlet with temperature data (Station noaa_nos_co_ops_9455920).

Fig. 17 Example of CIOFS surface water subtidal time series comparison in Kachemak Bay with temperature data (Station nerrs_kacsswq).

Fig. 18 Example of CIOFS deep water subtidal time series comparison in Kachemak Bay with temperature data (Station nerrs_kachdwq).

Fig. 19 Example of NWGOA surface water subtidal time series comparison at the head of the Inlet with temperature data (Station noaa_nos_co_ops_9455920).

Fig. 20 Example of NWGOA surface water subtidal time series comparison in Kachemak Bay with temperature data (Station nerrs_kacsswq).

Fig. 21 Example of NWGOA deep water subtidal time series comparison in Kachemak Bay with temperature data (Station nerrs_kachdwq).

It requires more model skill to accurate capture the anomaly in a temperature time series, and neither model is able to do this (Fig. 32). CIOFS has some skill capturing the temperature anomaly at the south end of Kodiak Island (NOAA 9457804, Fig. 22) but typically both models perform poorly for this measure.

Fig. 22 Example of CIOFS surface water subtidal temperature anomaly time series comparison south of Kodiak Island (Station noaa_nos_co_ops_9457804).

As with the salinity, instantaneous measures of CTD transects and profiles of temperature are difficult for a model to capture. Neither model does well on this measure and neither does clearly better than the other. However, CIOFS does better at capturing the horizontal variation across a transect whereas NWGOA better captures the vertical variation, which is consistent with CIOFS having higher horizontal resolution and NWGOA having higher vertical resolution. An example of this can be seen in Fig. 23 (for CIOFS) and Fig. 24 (for NWGOA).

Fig. 23 Example of CIOFS CTD transect comparison with temperature data (from the CMI KBNERR, line 1).

Fig. 24 Example of NWGOA CTD transect comparison with temperature data (from the CMI KBNERR, line 1).

Sea Surface Height

Both models can accurately capture the tidal sea surface height. CIOFS captures the subtidal sea surface height a bit better than NWGOA. In the middle of the domain near the entrance of Kachemak Bay, both capture some of the subtidal variability in the sea surface: CIOFS with a skill score of 0.5 (Fig. 25) and NWGOA with a skill score of 0.3 (Fig. 26).

Fig. 25 Example of CIOFS subtidal time series comparison with sea surface height data (from NOAA station noaa_nos_co_ops_9455500).

Fig. 26 Example of NWGOA subtidal time series comparison with sea surface height data (from NOAA station noaa_nos_co_ops_9455500).

Note that NWGOA performs poorly for the station at the head of the Inlet (NOAA 9455920), but given the unusual shape of the tidal and subtidal time series, it seems that a bathymetry/grid issue (maybe the grid cell being compared is being impacted by a wet/dry cell) may be more responsible than model physics.

Currents

Surface currents are measured for three time periods by HF Radar deployments (skill score summary Overview HF Radar Data). The skill scores calculated in time for each measured location in the grid show that CIOFS captures the northward velocity with decent skill. This is because the along-channel flow is dominantly northward. The HF Radar datasets from farther up the Inlet also have some eastward (approximately cross-channel) skill. Subtidal skill scores are almost universally low.

Similar results are shown in the form of tidal ellipses (HF Radar (UAF)). The CIOFS model M2 and K1 show similar results between the data and model.

The NWGOA model shows good skill in the northward/along-channel velocity component, and poor skill in the eastward/across-channel component. Similar to CIOFS, the subtidal velocity performance is poor. NWGOA shows decent comparison between data and model tidal ellipses, though less strong than CIOFS.

Model-data comparison results from the ADCP comparisons (Overview ADCP Data) are consistent with the HF Radar results. Along-channel velocities are accurately modeled by CIOFS. Across-channel velocities and along- and across-channel subtidal velocities are not accurately modeled. Comparisons of the horizontal speed are also shown in order to reduce the impact of small variations in the velocity direction. Both the tidal and subtidal horizontal speed are accurately simulated in CIOFS, implying that the magnitude and timing of the velocities are captured in the model, but the directionality is less accurate.

NWGOA shows the same trend: accurate for along-channel currents and inaccurate for across-channel currents and along- and across-channel subtidal currents. The tidal horizontal speed is not captured well by the NWGOA model, the subtidal horizontal speed is skillfully modeled in parts of the Inlet.