Make CMOR tables configurable through new configuration system and deprecate config-developer.yml #2946

bouweandela · 2026-01-16T17:19:18Z

Description

Added features

A new way of configuring the CMOR tables, with separate extensions for CMIP6, CMIP5, and CMIP3 tables to avoid confusion because of differing coordinate definitions.
The possibility to use data that has not been CMORized, as long as it is compliant enough with the CMOR standards for the preprocessor functions and/or diagnostics to be able to function. For preprocessor functions compliance with the CF conventions will typically be enough, while diagnostics will likely expect full CMOR compliance.
Updated the function esmvalcore.preprocessor.align_metadata so it works with branded variables

Closes #2918
Closes #2746
Closes #2364

Links to documentation:

Configure CMOR tables: https://esmvaltool--2946.org.readthedocs.build/projects/ESMValCore/en/2946/quickstart/configure.html#cmor-table-configuration
Updated instructions for contributing new native datasets: https://esmvaltool--2946.org.readthedocs.build/projects/ESMValCore/en/2946/develop/fixing_data.html#add-support-for-new-native-datasets
Upgrade instructions for users of a custom config-developer.yml file: https://esmvaltool--2946.org.readthedocs.build/projects/ESMValCore/en/2946/quickstart/configure.html#developer-configuration-file
Added argument target_branding_suffix to esmvalcore.preprocessor.align_metadata: https://esmvaltool--2946.org.readthedocs.build/projects/ESMValCore/en/2946/recipe/preprocessor.html#align-metadata

Deprecated features

config-developer.yml and the configuration setting config_developer_file, upgrade instructions are available here: https://esmvaltool--2946.org.readthedocs.build/projects/ESMValCore/en/2946/quickstart/configure.html#developer-configuration-file
In the module esmvalcore.cmor.table:
- the function read_cmor_tables which reads the tables based on the deprecated config-developer file
- the cmor_tables_path, default, and default_table_prefix arguments to various CMOR table reader classes and the class CustomInfo for reading custom CMOR tables have been deprecated because they are no longer needed with the new configuration format
- the global variable CMOR_TABLES holding the CMOR tables because of issue Reduce global state and use esmvalcore.config.Session objects instead #2954

Before you get started

☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

🧪 The new functionality is relevant and scientifically sound
🛠 This pull request has a descriptive title and labels
🛠 Code is written according to the code quality guidelines
🧪 and 🛠 Documentation is available
🛠 Unit tests have been added
🛠 Changes are backward compatible
🛠 The list of authors is up to date
🛠 All checks below this pull request were successful

To help with the number pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

codecov · 2026-01-19T16:06:49Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.68%. Comparing base (dee1bcb) to head (b43a466).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2946      +/-   ##
==========================================
+ Coverage   95.61%   95.68%   +0.06%     
==========================================
  Files         266      266              
  Lines       15584    15718     +134     
==========================================
+ Hits        14901    15040     +139     
+ Misses        683      678       -5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

schlunma · 2026-01-20T09:56:19Z

Hello, this pull request has been marked with the v2.14.0 milestone. The release of version 2.14.0 is currently scheduled for February 2026. To get this into the new release, it would be great to get this merged by the end of January.

If you won't be able to finish this in time, don't worry - just unassign the milestone v2.14.0. If you need any support, ping myself (@schlunma; the release manager for v2.14.0) or the @ESMValGroup/technical-lead-development-team. Please note that I won't be available until the beginning of February, though.

…ate default config

…fig-cmor

valeriupredoi · 2026-01-28T12:23:42Z

@bouweandela shout if you need a helping hand here, with the missing tests - I can help. Apart from that, it looks great - and a lot of bookkeeping work too, which is not pleasant 🍻

bouweandela · 2026-01-28T13:32:32Z

Thanks for offering V! The tests shouldn't be too much effort, it's mostly the docs that still need to be updated and I want to think about if I can avoid making #2954 worse. I was actually having a go at fixing that, but realized that it's probably best to leave that for another PR. If you could help out with reviewing and testing that would be great. Hopefully I'll have this ready for review by the end of the week.

valeriupredoi · 2026-01-28T13:34:34Z

sure thing, can do! 🍺

…oading them from config

valeriupredoi · 2026-02-06T16:36:24Z

OK @bouweandela - tests as promised:

basic help functionality OK
basic run with an existing cmorizer (one!) OK
create new cmorizer with no data OK
add data, and run with a standard OBS table OK
run with OBS6 table OK
run with new custom variable (without edit of custom table) OK
add custom variable to custom table and run OK

I am rather happy with the funtionality here, and well done! 🍻 Off to pub now 🍻

bouweandela · 2026-02-06T16:53:06Z

Thanks a lot V, enjoy 🍻 🍻

valeriupredoi

last approval after testing 😀 @schlunma can merge this for 2.14 - you can hold me accountable for any serious issues from it 😆

schlunma · 2026-02-09T09:03:28Z

@bouweandela Could you please fix the merge conflict? I'll try to play around with this today, and hopefully merge it very soon. This is the last feature PR that we will include for the release candidate.

…fig-cmor

bouweandela · 2026-02-09T09:40:50Z

Thanks @schlunma, I've addressed the merge conflict.

schlunma

A couple of questions/comments on the docs, will continue with testing later.

doc/quickstart/configure.rst

doc/develop/fixing_data.rst

doc/quickstart/configure.rst

schlunma · 2026-02-10T15:39:34Z

Removing an entry from the custom CMIP6 tables, e.g. the entry for clhmtisccp and running esmvaltool run ~/ESMValTool/esmvaltool/recipes/examples/recipe_preprocessor_derive_test.yml

with the following user config file

projects:
  CMIP5:
    cmor_table:
      paths:
        - cmip5/Tables
        - cmip5-custom
        - ~/tmp/cmor  # <-- this is the problematic line

fails with

Traceback (most recent call last):
  File "/home/b/b309141/micromamba/envs/esm/bin/esmvaltool", line 3, in <module>
    from esmvalcore._main import run
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_main.py", line 43, in <module>
    from esmvalcore.config._config import warn_if_old_extra_facets_exist
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/config/__init__.py", line 20, in <module>
    from esmvalcore.config._config_object import CFG, Config, Session
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/config/_config_object.py", line 406, in <module>
    CFG = _get_global_config()
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/config/_config_object.py", line 401, in _get_global_config
    config_obj.reload()
    ~~~~~~~~~~~~~~~~~^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/config/_config_object.py", line 157, in reload
    self.load_from_dirs([USER_CONFIG_DIR])
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/config/_config_object.py", line 130, in load_from_dirs
    self.update(new_config_dict)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/config/_validated_config.py", line 105, in update
    self[key] = other[key]
    ~~~~^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/config/_validated_config.py", line 74, in __setitem__
    cval = self._validate[key](val)
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/config/_config_validators.py", line 424, in validate_projects
    validate_cmor_tables(mapping)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/config/_config_validators.py", line 391, in validate_cmor_tables
    project: esmvalcore.cmor.table.get_tables(
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        session={"projects": value},  # type: ignore[arg-type]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        project=project,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/cmor/table.py", line 389, in get_tables
    tables = cls(**kwargs)
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/cmor/table.py", line 1281, in __init__
    self._load_table(table_file)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/cmor/table.py", line 1299, in _load_table
    table = self._read_table_file(table_file)
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/cmor/table.py", line 1308, in _read_table_file
    self._read_line()
    ~~~~~~~~~~~~~~~^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/cmor/table.py", line 1346, in _read_line
    index = line.index(":")
ValueError: substring not found

Somehow the CMIP5 and CMIP6 projects get mixed up.

bouweandela · 2026-02-11T09:55:00Z

Trying to reproduce your issue, I followed these steps:

create a file ~/.config/esmvaltool/cmor_tables.yml with the content

projects:
  CMIP5:
    cmor_table:
      paths:
        - cmip5/Tables
        - cmip5-custom
        - ~/tmp/cmor  # <-- this is the problematic line

create an empty directory ~/tmp/cmor
remove the variable clhmtisccp from the file esmvalcore/cmor/tables/cmip6-custom/CMIP6_custom.json
run the command esmvaltool run examples/recipe_preprocessor_derive_test.yml --diagnostics '*/clhmtisccp'

Does that describe what you did? Or is there some content in your ~/tmp/cmor directory? The stack trace above indicates that there is an issue with reading one of the files of your CMIP5 CMOR configuration. I would expect a message saying in which file the error occurs in, could you share that?

If I follow the 4 steps above, the run ends with the message:

2026-02-11 09:46:40,123 UTC [647747] ERROR   Unable to load CMOR table (project) 'CMIP6' for variable 'clhmtisccp' with mip 'CFmon'

as expected, because I removed the variable from the table file in step 3 above.

valeriupredoi · 2026-02-11T10:03:03Z

Trying to reproduce your issue, I followed these steps:

create a file ~/.config/esmvaltool/cmor_tables.yml with the content
projects:
  CMIP5:
    cmor_table:
      paths:
        - cmip5/Tables
        - cmip5-custom
        - ~/tmp/cmor  # <-- this is the problematic line
create an empty directory ~/tmp/cmor

remove the variable clhmtisccp from the file esmvalcore/cmor/tables/cmip6-custom/CMIP6_custom.json

run the command esmvaltool run examples/recipe_preprocessor_derive_test.yml --diagnostics '*/clhmtisccp'

Does that describe what you did? Or is there some content in your ~/tmp/cmor directory? The stack trace above indicates that there is an issue with reading one of the files of your CMIP5 CMOR configuration. I would expect a message saying in which file the error occurs in, could you share that?

If I follow the 4 steps above, the run ends with the message:
2026-02-11 09:46:40,123 UTC [647747] ERROR   Unable to load CMOR table (project) 'CMIP6' for variable 'clhmtisccp' with mip 'CFmon'
as expected, because I removed the variable from the table file in step 3 above.

I ran this exact test when I ran tests last week - I am starting to get mildly miffed with the approach towards this PR's review - if we're trying to make any fairly/medium large development completely bulletproof, review will take ad infinitum! 🍺

schlunma · 2026-02-11T10:07:59Z

@bouweandela you're right, the content of ~/tmp/cmor matters. I have a single file CMIP6_custom.json with the contents

{
    "Header": {
        "generic_levels": "olevel",
        "table_id": "Table custom"
    },
    "variable_entry": {
        "clhmtisccp": {
            "cell_measures": "area: areacella",
            "cell_methods": "time: mean",
            "comment": "at the top of the atmosphere (to be compared with satellite measurements)",
            "dimensions": "longitude latitude time",
            "long_name": "ISCCP High Level Medium-Thickness Cloud Area Fraction",
            "modeling_realm": "atmos",
            "out_name": "clhmtisccp",
            "standard_name": "",
            "type": "real",
            "units": "%"
        }
    }
}

I thought the CMIP5 table reader only looks for *.dat files. Does every reader type try to read every file in the custom directory? If yes, this should probably be documented (apologies if I missed that).

schlunma · 2026-02-11T10:12:54Z

@valeriupredoi As release manager of the upcoming release,

I want to make sure that there are no obvious problems with the current development version before running 150+ recipes. This will save me a lot of time.
I also want to make sure that I understand the changes implemented in this PR before merging it.

Running some tests helps me with both. Apologies if this repeats some of your tests, but for large changes like this it's probably a good idea if more than one person actually runs some tests. I don't plan to perform any bigger (recipe) tests with this, so if @bouweandela clears up my reported issue, than I am happy to merge this asap.

bouweandela · 2026-02-11T10:44:40Z

Does every reader type try to read every file in the custom directory?

The CMIP6 reader reads files with the .json extension only because these tables are json files, but for CMIP5 style tables there is no extension defined because they do not follow a file format that exists outside of CMOR: https://github.com/ESMValGroup/ESMValCore/tree/main/esmvalcore/cmor/tables/cmip5/Tables

schlunma · 2026-02-11T11:21:11Z

Does every reader type try to read every file in the custom directory?

The CMIP6 reader reads files with the .json extension only because these tables are json files, but for CMIP5 style tables there is no extension defined because they do not follow a file format that exists outside of CMOR: https://github.com/ESMValGroup/ESMValCore/tree/main/esmvalcore/cmor/tables/cmip5/Tables

Would be great to add a note about that to the respective Info classes and/or here.

bouweandela · 2026-02-11T11:49:39Z

Done in b43a466

schlunma · 2026-02-11T13:39:11Z

Thanks!! My tests were successful, merging this now 🚀

valeriupredoi · 2026-02-11T13:46:07Z

Thanks!! My tests were successful, merging this now 🚀

brill, very many thanks, Manu 🍺

bouweandela force-pushed the new-config-cmor branch 2 times, most recently from 2b7ae33 to f7e0982 Compare January 19, 2026 12:41

bouweandela added the enhancement New feature or request label Jan 19, 2026

bouweandela added this to the v2.14.0 milestone Jan 19, 2026

bouweandela added cmor Related to the CMOR standard deprecated feature backwards incompatible change and removed backwards incompatible change labels Jan 19, 2026

bouweandela force-pushed the new-config-cmor branch 4 times, most recently from 2403e51 to c2bd443 Compare January 19, 2026 15:44

Make CMOR tables configurable through new configuration system

5ae3e93

bouweandela force-pushed the new-config-cmor branch from c2bd443 to 5ae3e93 Compare January 19, 2026 16:02

Fix another test

aaaeb9e

bouweandela and others added 7 commits January 23, 2026 17:49

Add docstrings and type hints

ea4aa6a

Gracefully handle missing out_name in CMIP5-style CMOR tables and upd…

64f365b

…ate default config

Merge branch 'main' into new-config-cmor

1b58631

Merge branch 'main' of github.com:ESMValGroup/ESMValCore into new-con…

7e74268

…fig-cmor

Remove config_developer_file from default configuration

d676fbd

Use explicit configuration instead of YAML anchors

d0090d4

Merge branch 'main' into new-config-cmor

d11aa25

bouweandela added 3 commits January 28, 2026 17:08

Small improvements

33ce271

Do not load CMOR tables from built-in config-developer.yml prior to l…

ed35a66

…oading them from config

Add note to docs about what to do if CMOR_TABLES is empty

4e11d10

bouweandela force-pushed the new-config-cmor branch from 221c733 to 610c23d Compare February 6, 2026 14:18

Improve test coverage and fix bug

b5064fd

valeriupredoi approved these changes Feb 6, 2026

View reviewed changes

bouweandela added 2 commits February 9, 2026 10:31

Merge branch 'main' of github.com:ESMValGroup/ESMValCore into new-con…

eb3306c

…fig-cmor

Update siextent units in CMIP6 custom table

81317b1

schlunma reviewed Feb 9, 2026

View reviewed changes

bouweandela added 2 commits February 9, 2026 12:47

Improve docs with suggestions by @schlunma

a982de2

Remove impractical suggestion on how to organize data

5a2147d

schlunma reviewed Feb 9, 2026

View reviewed changes

doc/quickstart/configure.rst Outdated Show resolved Hide resolved

Make defaults/cmor_tables.yml a link

2ee9aad

bouweandela mentioned this pull request Feb 9, 2026

Make the mip facet optional #2970

Open

schlunma mentioned this pull request Feb 10, 2026

Retrieving data with intake esgf fails #2972

Open

Add note about which files are read

b43a466

schlunma merged commit 2784aba into main Feb 11, 2026
4 checks passed

schlunma deleted the new-config-cmor branch February 11, 2026 13:39

schlunma mentioned this pull request Feb 11, 2026

Doc build fails ESMValGroup/ESMValTool#4333

Closed

Make CMOR tables configurable through new configuration system and deprecate config-developer.yml #2946

Make CMOR tables configurable through new configuration system and deprecate config-developer.yml #2946

Conversation

bouweandela commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Deprecated features

Before you get started

Checklist

Uh oh!

codecov bot commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

schlunma commented Jan 20, 2026

Uh oh!

valeriupredoi commented Jan 28, 2026

Uh oh!

bouweandela commented Jan 28, 2026

Uh oh!

valeriupredoi commented Jan 28, 2026

Uh oh!

valeriupredoi commented Feb 6, 2026

Uh oh!

bouweandela commented Feb 6, 2026

Uh oh!

valeriupredoi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

schlunma commented Feb 9, 2026

Uh oh!

bouweandela commented Feb 9, 2026

Uh oh!

schlunma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

schlunma commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bouweandela commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

valeriupredoi commented Feb 11, 2026

Uh oh!

schlunma commented Feb 11, 2026

Uh oh!

schlunma commented Feb 11, 2026

Uh oh!

bouweandela commented Feb 11, 2026

Uh oh!

schlunma commented Feb 11, 2026

Uh oh!

bouweandela commented Feb 11, 2026

Uh oh!

schlunma commented Feb 11, 2026

Uh oh!

Uh oh!

valeriupredoi commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bouweandela commented Jan 16, 2026 •

edited

Loading

codecov bot commented Jan 19, 2026 •

edited

Loading

valeriupredoi left a comment •

edited

Loading

schlunma commented Feb 10, 2026 •

edited

Loading

bouweandela commented Feb 11, 2026 •

edited

Loading