Gulf of Maine & Scotian Shelf (GoMSS) NEMO Model Project
About the Project
The Gulf of Maine (GoM) and the Scotian Shelf (SS) constitute one of the most productive areas on the eastern coast of North America where high biological activity related to upwelling of nutrient-rich deep waters is crucial for major fisheries of various species. With highly variable bathymetry and related strong tidal currents, the region is challenging to model accurately and understand dynamically. The Marine Environmental Observation Prediction and Response (MEOPAR, http://meopar.ca/) network is funding the system presented here under a project entitled “Downscaling Atmosphere-Ocean Forecasts from Global to Harbour Scales with Application to the Maritimes”.
The GoMSS project team has developed a new “pre-operational” system that downscales forecasts from global scales to the GoMSS region. The ocean model uses the NEMO (Nucleus for European Modelling of the Ocean, https://www.nemo-ocean.eu/) modelling architecture configured with a 1/36 degree (~2.8 km) horizontal resolution, 50 levels and five tidal constituents. The hourly surface forcing fields are provided by the Environment and Climate Change Canada High-Resolution Deterministic Prediction System. This website provides public access to the forecast products for project partners with interests in marine forecasting, risk perception, and emergency management.
Model Domain
As indicated by the red rectangle in the figure below, the model domain includes the GoM and SS with portions of George’s Bank on the west and the Grand Banks on the east.
Notes About System Operations and Model Configuration
| Date | Event |
|---|---|
| 2018-09-06 | Started automated daily production of day average surface salinity and temperature figures for web site. Ran figure generation code to backfill figures from 2018-08-24 forward. |
| 2018-08-30 | Daily automated near-real-time runs resumed for the first time since 2018-07-20. |
| 2018-08-23 | Discovered that the period of no model runs due to 2018-07-20 through 2018-08-08 PSY4 downloads outage hid an issue whereby HRDPS weather files contained NaNs for u and v wind fields in hours 1-23 on 2018-07-20 and hour 0 on 2018-07-21. Recovery of 2018-07-20 and 2018-07-21 HRDPS files was not possible, so the missing values were patched by creating symbolic links to persist 2018-07-19 atmospheric forcing as 2018-07-20, and to use 2018-07-22 atmospheric forcing as 2018-07-21. |
| 2018-08-08 |
Updated CMEMS Motu client
(used to download PSY4 files for lateral boundary conditions)
to v1.6.00.
Modified download_psy4 worker re: changes in date range interface in Motu client.
Backfilled PSY4 boundary conditions downloads from 2018-07-20 through 2018-08-08.
Daily automated PSY4 boundary conditions downloads resumed.
|
| 2018-07-20 |
A series of PSY4 boundary conditions download failures started due to
http://marine.copernicus.eu/newsflash/cmems8219-issue-linked-implementation-motu-software-new-version/?idpage=166
.
These failure resulted in daily near-real-time runs not being executed until 2018-08-30. The runs for the period from 2018-07-20 through 2018-08-29 were executed as backfill runs on various dates in that period. |
| 2018-05-30 |
Updated CMEMS Motu client
(used to download PSY4 files for lateral boundary conditions)
to v1.5.00.
Package installed in a new Python 2.7 motu-client-env environment.
|
| 2018-02-26 | Daily automated near-real-time runs resumed for the first time since 2017-08-02. |
| 2018-02-22 |
Clean build of NEMO with pgf90 -Kieee flag in both compile and link to replace
in-line floating point divides with a subroutine call a more accurate math library.
That resolved a NaN sea surface height value issue in backfilling the 2017-09-22 run
|
| 2017-11-30 |
Added code to watch_nemo worker to confirm successful completion of NEMO runs by confirming:
|
| 2017-11-12 |
Added watch_nemo worker to automation to monitor progress of NEMO runs.
|
| 2017-10-17 | HRDPS files downloads failed. The failures were not discovered until 2018-02-23, so recovery of 2017-10-17 HRDPS files was not possible. The missing fields were patched by creating symbolic links to persist 2017-10-16 atmospheric forcing as 2017-10-17. |
| 2017-09-22 |
NEMO executable changed to use pgf90 -Kieee flag in both compile and link to replace
in-line floating point divides with a subroutine call a more accurate math library.
|
| 2017-08-19 | HRDPS files downloads failed. The failures were not discovered until 2017-09-27, so recovery of 2017-08-19 HRDPS files was not possible. The missing fields were patched by creating symbolic links to persist 2017-08-17 atmospheric forcing as 2017-08-19. |
| 2017-08-18 | HRDPS files downloads failed. The failures were not discovered until 2017-09-27, so recovery of 2017-08-18 HRDPS files was not possible. The missing fields were patched by creating symbolic links to persist 2017-08-17 atmospheric forcing as 2017-08-18. |
| 2017-08-09 | HRDPS files downloads failed. The failures were not discovered until 2017-09-27, so recovery of 2017-08-09 HRDPS files was not possible. The missing fields were patched by creating symbolic links to persist 2017-08-08 atmospheric forcing as 2017-08-09. |
| 2017-08-03 |
Missing HRDPS files caused a silent failure of atmospheric forcing files production, PSY4 file downloads, lateral boundary conditions forcing file production, and NEMO runs. The failures were not discovered until 2017-09-06, so recovery of 2017-08-03 HRDPS files was not possible. The missing fields were patched by creating symbolic links to persist 2017-08-02 atmospheric forcing as 2017-08-03. This failure resulted in daily near-real-time runs not being executed until 2018-02-26. The runs for the period from 2017-08-03 through 2018-02-25 were executed as backfill runs on various dates in that period. |
| 2017-05-10 | Changed automation sequencing to run HRDPS files downloads and atmospheric forcing files production, followed by PSY4 file download and lateral boundary conditions forcing files production. Previous concurrency of those operations had developed an unstable race condition due to changes in upstream services. |
| 2017-04-19 |
Updated automation environment to Python 3.6.1. Updated CMEMS Motu client (used to download PSY4 boundary condition files) to v1.4.00. |
| 2017-03-31 |
Changed surface forcing weights file used for on-the-fly interpolation of atmospheric forcing to accommodate change by ECCC that extended HRDPS sub-region used for atmospheric forcing to west and north. Changed Matlab script used to calculate lateral boundary conditions to correct a bug whereby the eastern and northern boundary fields were flipped. |
| 2016-12-07 | Daily model runs commenced. |
Automation Monitoring
This section presents monitoring information from the GoMSS NEMO model daily nowcast runs. It is primarily intended for members of the GoMSS modeling team. Public results of the preceding 21 days runs are linked from the Results section.
Below are:
- A link to the nowcast manager process's checklist that shows the status of the system
- Links to the present and preceding 7 days' log files
Checklist
The nowcast checklist is a YAML representation of the checklist dictionary that is maintained by the nowcast manager process. It reflects the present state of the nowcast system, showing the information returned to the manager from each of the workers. It is both (somewhat) human-readable, and machine-readable. The checklist is reset each morning after the day's nowcast run results have been processed (see the discussion of log file rollovers below). That means that the checklist records only the results of the log file rotations for a period of time each day. The previous day's checklist is also available. The checklist is stored in /data1/dlatornell/nowcast-sys/logs/nowcast/nowcast_checklist.yaml on the nemo cluster.
Log Files
The log files contain messages from the nowcast automation workers and manager processes. Each day's nowcast processing is in a separate log file. When the day's processing is completed the log file is rolled over, as are the previous log files. The log for the current day's run (nowcast.log) is rolled over to nowcast.log.1, and the previous nowcast.log.1 is rolled over to nowcast.log.2, etc. A total of 7 day's log files are retained.
There are 3 flavours of log files:
- nowcast.log which contains messages logged at the info level and above (warning, error, etc.). This is the most concise log file.
- nowcast.debug.log which contains messages logged at the debug level and above; i.e. all of the nitty-gritty details.
- checklist.log which contains a snapshot of the checklist file each morning before it is reset.
The log file rollovers happen each morning after the day's nowcast run results have been processed to produce both a results page.
| nowcast.log nowcast.debug.log | Current log files. Nearly empty after the day's nowcast run results have been processed and the current day's run log files are rolled over to nowcast.log.1 and nowcast.debug.log.1. |
| nowcast.log.1 nowcast.debug.log.1 checklist.log.1 | Previous log files. Roll over to nowcast.log.2, nowcast.debug.log.2, and checklist.log.2 when nowcast.log rolls over to nowcast.log.1. |
| Older log files: | |
| nowcast.log.2 nowcast.debug.log.2 checklist.log.2 | |
| nowcast.log.3 nowcast.debug.log.3 checklist.log.3 | |
| nowcast.log.4 nowcast.debug.log.4 checklist.log.4 | |
| nowcast.log.5 nowcast.debug.log.5 checklist.log.5 | |
| nowcast.log.6 nowcast.debug.log.6 checklist.log.6 | |
| nowcast.log.7 nowcast.debug.log.7 checklist.log.7 | |
Resources
Contributors
The gomss.ocean.dal.ca site is the product of research conducted by scientists at the Dalhousie University Department of Oceanography, the Fisheries and Oceans Canada Bedford Insttitute of Oceanography, and Environment and Climate Change Canada under the MEOPAR “Downscaling Atmosphere-Ocean Forecasts from Global to Harbour Scales with Application to the Maritimes” project.
People who have contributed to the research and to this site:
- Hal Ritchie <hal.ritchie@canada.ca>
- Youyu Lu <youyu.lu@dfo-mpo.gc.ca>
- Keith Thompson <keith.thompson@dal.ca>
- Doug Latornell <doug.latornell@43ravens.ca>
- Daniel Morrison <daniel.morrison@phys.ocean.dal.ca>
- Xianmin Hu <xianmin.hu@dfo-mpo.gc.ca>
- Michael Casey <michael.casey@dfo-mpo.gc.ca>
- Seung-Jong Baek <seung-jong.baek@canada.ca>
- Shiliang Shan <sshan@dal.ca>
- Dominik Jacques <dominik.jacques@canada.ca>
- Christoph Renkl <christoph.renkl@dal.ca>
The web site was designed and implemented by https://43ravens.ca.
License
The gomss.ocean.dal.ca site content, code, and documentation are copyright 2018-2024 by 43ravens and the GoMSS Nowcast project contributors .
They are licensed under the Apache License, Version 2.0. https://www.apache.org/licenses/LICENSE-2.0