Composing recipes#
This notebooks shows how to fill the datasets section in a recipe.
from pathlib import Path
import yaml
from esmvalcore.config import CFG
from esmvalcore.dataset import Dataset, datasets_to_recipe
from esmvalcore.experimental.recipe import Recipe
Configure ESMValCore so it always searches the ESGF for data
CFG["search_data"] = "complete"
Here is a small example recipe, that uses the datasets_to_recipe function to convert a list of datasets to a recipe:
tas = Dataset(
short_name="tas",
mip="Amon",
project="CMIP6",
dataset="CanESM5-1",
ensemble="r1i1p1f1",
exp="historical",
grid="gn",
timerange="2000/2002",
)
tas["diagnostic"] = "diagnostic_name"
pr = tas.copy(short_name="pr")
print(yaml.safe_dump(datasets_to_recipe([tas, pr])))
datasets:
- dataset: CanESM5-1
diagnostics:
diagnostic_name:
variables:
pr:
ensemble: r1i1p1f1
exp: historical
grid: gn
mip: Amon
project: CMIP6
timerange: 2000/2002
tas:
ensemble: r1i1p1f1
exp: historical
grid: gn
mip: Amon
project: CMIP6
timerange: 2000/2002
A more ambitious recipe might want to use all data that is available on ESGF. We can define a dataset template with a facet value of * where any value can be used. This can then be expanded to a list of datasets using the from_files() method.
dataset_template = Dataset(
short_name="tas",
mip="Amon",
project="CMIP6",
exp="historical",
dataset="*",
institute="*",
ensemble="*",
grid="*",
)
datasets = list(dataset_template.from_files())
len(datasets)
1011
This results in the following recipe:
for dataset in datasets:
dataset.facets["diagnostic"] = "diagnostic_name"
recipe = Path("recipe.yml")
with recipe.open("w") as file:
yaml.safe_dump(datasets_to_recipe(datasets), file, encoding="utf-8")
print(recipe.read_text(encoding="utf-8")[:300])
print("...")
datasets:
- dataset: ACCESS-CM2
ensemble: r(1:10)i1p1f1
grid: gn
institute: CSIRO-ARCCSS
- dataset: ACCESS-ESM1-5
ensemble: r(1:40)i1p1f1
grid: gn
institute: CSIRO
- dataset: AWI-CM-1-1-MR
ensemble: r(1:5)i1p1f1
grid: gn
institute: AWI
- dataset: AWI-ESM-1-1-LR
ensemble: r1i1p1f1
...
The above is a rather long list of datasets, therefore we provide a function to format recipes in a more compact way:
print(Recipe(recipe).to_yaml())
datasets:
- {dataset: ACCESS-CM2, ensemble: 'r(1:10)i1p1f1', grid: gn, institute: CSIRO-ARCCSS}
- {dataset: ACCESS-ESM1-5, ensemble: 'r(1:40)i1p1f1', grid: gn, institute: CSIRO}
- {dataset: AWI-CM-1-1-MR, ensemble: 'r(1:5)i1p1f1', grid: gn, institute: AWI}
- {dataset: AWI-ESM-1-1-LR, ensemble: r1i1p1f1, grid: gn, institute: AWI}
- {dataset: AWI-ESM-1-REcoM, ensemble: r1i1p1f1, grid: gn, institute: AWI}
- {dataset: BCC-CSM2-MR, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: BCC}
- {dataset: BCC-ESM1, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: BCC}
- {dataset: CAMS-CSM1-0, ensemble: 'r(1:2)i1p1f1', grid: gn, institute: CAMS}
- {dataset: CAMS-CSM1-0, ensemble: r1i1p1f2, grid: gn, institute: CAMS}
- {dataset: CAS-ESM2-0, ensemble: 'r(1:4)i1p1f1', grid: gn, institute: CAS}
- {dataset: CESM2, ensemble: 'r(1:11)i1p1f1', grid: gn, institute: NCAR}
- {dataset: CESM2-FV2, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: NCAR}
- {dataset: CESM2-FV2, ensemble: r1i2p2f1, grid: gn, institute: NCAR}
- {dataset: CESM2-WACCM, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: NCAR}
- {dataset: CESM2-WACCM-FV2, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: NCAR}
- {dataset: CIESM, ensemble: 'r(1:3)i1p1f1', grid: gr, institute: THU}
- {dataset: CMCC-CM2-HR4, ensemble: r1i1p1f1, grid: gn, institute: CMCC}
- {dataset: CMCC-CM2-SR5, ensemble: 'r(2:11)i1p2f1', grid: gn, institute: CMCC}
- {dataset: CMCC-CM2-SR5, ensemble: r1i1p1f1, grid: gn, institute: CMCC}
- {dataset: CMCC-ESM2, ensemble: r1i1p1f1, grid: gn, institute: CMCC}
- {dataset: CNRM-CM6-1, ensemble: 'r(1:30)i1p1f2', grid: gr, institute: CNRM-CERFACS}
- {dataset: CNRM-CM6-1-HR, ensemble: r1i1p1f2, grid: gr, institute: CNRM-CERFACS}
- {dataset: CNRM-ESM2-1, ensemble: 'r(1:15)i1p1f2', grid: gr, institute: CNRM-CERFACS}
- {dataset: CanESM5, ensemble: 'r(1:25)i1p1f1', grid: gn, institute: CCCma}
- {dataset: CanESM5, ensemble: 'r(1:40)i1p2f1', grid: gn, institute: CCCma}
- {dataset: CanESM5-1, ensemble: 'r(1:20)i1p1f1', grid: gn, institute: CCCma}
- {dataset: CanESM5-1, ensemble: 'r(1:25)i1p2f1', grid: gn, institute: CCCma}
- {dataset: CanESM5-1, ensemble: 'r(24:39)i1p1f1', grid: gn, institute: CCCma}
- {dataset: CanESM5-1, ensemble: 'r(41:50)i1p1f1', grid: gn, institute: CCCma}
- {dataset: CanESM5-1, ensemble: r22i1p1f1, grid: gn, institute: CCCma}
- {dataset: CanESM5-CanOE, ensemble: 'r(1:3)i1p2f1', grid: gn, institute: CCCma}
- {dataset: E3SM-1-0, ensemble: 'r(1:5)i1p1f1', grid: gr, institute: E3SM-Project}
- {dataset: E3SM-1-0, ensemble: 'r(1:20)i2p2f1', grid: gr, institute: UCSB}
- {dataset: E3SM-1-1, ensemble: r1i1p1f1, grid: gr, institute: E3SM-Project}
- {dataset: E3SM-1-1-ECA, ensemble: r1i1p1f1, grid: gr, institute: E3SM-Project}
- {dataset: E3SM-2-0, ensemble: 'r(1:21)i1p1f1', grid: gr, institute: E3SM-Project}
- {dataset: E3SM-2-0-NARRM, ensemble: 'r(1:5)i1p1f1', grid: gr, institute: E3SM-Project}
- {dataset: E3SM-2-1, ensemble: 'r(1:5)i1p1f1', grid: gr, institute: E3SM-Project}
- {dataset: EC-Earth3, ensemble: 'r(1:7)i1p1f1', grid: gr, institute: EC-Earth-Consortium}
- {dataset: EC-Earth3, ensemble: 'r(9:25)i1p1f1', grid: gr, institute: EC-Earth-Consortium}
- {dataset: EC-Earth3, ensemble: 'r(101:150)i1p1f1', grid: gr, institute: EC-Earth-Consortium}
- {dataset: EC-Earth3-AerChem, ensemble: 'r(3:4)i1p1f1', grid: gr, institute: EC-Earth-Consortium}
- {dataset: EC-Earth3-AerChem, ensemble: r1i1p1f1, grid: gr, institute: EC-Earth-Consortium}
- {dataset: EC-Earth3-AerChem, ensemble: r1i1p4f1, grid: gr, institute: EC-Earth-Consortium}
- {dataset: EC-Earth3-CC, ensemble: 'r(6:13)i1p1f1', grid: gr, institute: EC-Earth-Consortium}
- {dataset: EC-Earth3-CC, ensemble: r1i1p1f1, grid: gr, institute: EC-Earth-Consortium}
- {dataset: EC-Earth3-CC, ensemble: r4i1p1f1, grid: gr, institute: EC-Earth-Consortium}
- {dataset: EC-Earth3-ESM-1, ensemble: r5i1p1f1, grid: gr, institute: EC-Earth-Consortium}
- {dataset: EC-Earth3-HR, ensemble: r1i1p1f1, grid: gr, institute: EC-Earth-Consortium}
- {dataset: EC-Earth3-Veg, ensemble: 'r(1:6)i1p1f1', grid: gr, institute: EC-Earth-Consortium}
- {dataset: EC-Earth3-Veg, ensemble: 'r(10:14)i1p1f1', grid: gr, institute: EC-Earth-Consortium}
- {dataset: EC-Earth3-Veg-LR, ensemble: 'r(1:3)i1p1f1', grid: gr, institute: EC-Earth-Consortium}
- {dataset: FGOALS-f3-L, ensemble: 'r(1:3)i1p1f1', grid: gr, institute: CAS}
- {dataset: FGOALS-g3, ensemble: 'r(1:6)i1p1f1', grid: gn, institute: CAS}
- {dataset: FIO-ESM-2-0, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: FIO-QLNM}
- {dataset: GFDL-CM4, ensemble: r1i1p1f1, grid: gr1, institute: NOAA-GFDL}
- {dataset: GFDL-ESM4, ensemble: 'r(1:3)i1p1f1', grid: gr1, institute: NOAA-GFDL}
- {dataset: GISS-E2-1-G, ensemble: 'r(1:4)i1p5f1', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-G, ensemble: 'r(1:5)i1p1f3', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-G, ensemble: 'r(1:10)i1p1f1', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-G, ensemble: 'r(1:20)i1p1f2', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-G, ensemble: 'r(1:20)i1p3f1', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-G, ensemble: 'r(6:10)i1p5f1', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-G, ensemble: 'r(101:102)i1p1f1', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-G, ensemble: 'r(201:202)i1p1f(5:6)', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-G, ensemble: 'r(201:210)i1p1f2', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-G, ensemble: 'r(201:210)i1p1f4', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-G, ensemble: 'r(301:310)i1p1f2', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-G, ensemble: 'r(301:310)i1p1f4', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-G, ensemble: r1i1p1f5, grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-G-CC, ensemble: r1i1p1f1, grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-H, ensemble: 'r(1:5)i1p1f2', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-H, ensemble: 'r(1:5)i1p3f1', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-H, ensemble: 'r(1:5)i1p5f1', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-1-H, ensemble: 'r(1:10)i1p1f1', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-2-G, ensemble: 'r(1:5)i1p3f1', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-2-G, ensemble: 'r(1:6)i1p1f1', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E2-2-H, ensemble: 'r(1:5)i1p1f1', grid: gn, institute: NASA-GISS}
- {dataset: GISS-E3-G, ensemble: r1i1p101f1, grid: gr, institute: NASA-GISS}
- {dataset: GISS-E3-G, ensemble: r1i1p103f1, grid: gr, institute: NASA-GISS}
- {dataset: HadGEM3-GC31-LL, ensemble: 'r(1:5)i1p1f3', grid: gn, institute: MOHC}
- {dataset: HadGEM3-GC31-LL, ensemble: 'r(11:60)i1p1f3', grid: gn, institute: MOHC}
- {dataset: HadGEM3-GC31-MM, ensemble: 'r(1:4)i1p1f3', grid: gn, institute: MOHC}
- {dataset: ICON-ESM-LR, ensemble: 'r(1:5)i1p1f1', grid: gn, institute: MPI-M}
- {dataset: IITM-ESM, ensemble: r1i1p1f1, grid: gn, institute: CCCR-IITM}
- {dataset: INM-CM4-8, ensemble: r1i1p1f1, grid: gr1, institute: INM}
- {dataset: INM-CM5-0, ensemble: 'r(1:10)i1p1f1', grid: gr1, institute: INM}
- {dataset: IPSL-CM5A2-INCA, ensemble: r1i1p1f1, grid: gr, institute: IPSL}
- {dataset: IPSL-CM6A-LR, ensemble: 'r(1:33)i1p1f1', grid: gr, institute: IPSL}
- {dataset: IPSL-CM6A-LR-INCA, ensemble: r1i1p1f1, grid: gr, institute: IPSL}
- {dataset: KACE-1-0-G, ensemble: 'r(1:3)i1p1f1', grid: gr, institute: NIMS-KMA}
- {dataset: KIOST-ESM, ensemble: r1i1p1f1, grid: gr1, institute: KIOST}
- {dataset: MCM-UA-1-0, ensemble: 'r1i1p1f(1:2)', grid: gn, institute: UA}
- {dataset: MIROC-ES2H, ensemble: 'r(1:3)i1p4f2', grid: gn, institute: MIROC}
- {dataset: MIROC-ES2H, ensemble: 'r1i1p(1:3)f2', grid: gn, institute: MIROC}
- {dataset: MIROC-ES2L, ensemble: 'r(1:30)i1p1f2', grid: gn, institute: MIROC}
- {dataset: MIROC-ES2L, ensemble: r1i1000p1f2, grid: gn, institute: MIROC}
- {dataset: MIROC6, ensemble: 'r(1:50)i1p1f1', grid: gn, institute: MIROC}
- {dataset: MPI-ESM-1-2-HAM, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: HAMMOZ-Consortium}
- {dataset: MPI-ESM1-2-HR, ensemble: 'r(1:10)i1p1f1', grid: gn, institute: MPI-M}
- {dataset: MPI-ESM1-2-LR, ensemble: 'r(1:50)i1p1f1', grid: gn, institute: MPI-M}
- {dataset: MPI-ESM1-2-LR, ensemble: r1i2000p1f1, grid: gn, institute: MPI-M}
- {dataset: MRI-ESM2-0, ensemble: 'r(1:10)i1p1f1', grid: gn, institute: MRI}
- {dataset: MRI-ESM2-0, ensemble: r1i2p1f1, grid: gn, institute: MRI}
- {dataset: MRI-ESM2-0, ensemble: r1i1000p1f1, grid: gn, institute: MRI}
- {dataset: NESM3, ensemble: 'r(1:5)i1p1f1', grid: gn, institute: NUIST}
- {dataset: NorCPM1, ensemble: 'r(1:30)i1p1f1', grid: gn, institute: NCC}
- {dataset: NorESM2-LM, ensemble: 'r(1:43)i1p1f1', grid: gn, institute: NCC}
- {dataset: NorESM2-LM, ensemble: r1i1p4f1, grid: gn, institute: NCC}
- {dataset: NorESM2-MM, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: NCC}
- {dataset: SAM0-UNICON, ensemble: r1i1p1f1, grid: gn, institute: SNU}
- {dataset: TaiESM1, ensemble: 'r(1:2)i1p1f1', grid: gn, institute: AS-RCEC}
- {dataset: UKESM1-0-LL, ensemble: 'r(1:4)i1p1f2', grid: gn, institute: MOHC}
- {dataset: UKESM1-0-LL, ensemble: 'r(5:7)i1p1f3', grid: gn, institute: MOHC}
- {dataset: UKESM1-0-LL, ensemble: 'r(8:12)i1p1f2', grid: gn, institute: MOHC}
- {dataset: UKESM1-0-LL, ensemble: 'r(13:15)i1p1f2', grid: gn, institute: NIMS-KMA}
- {dataset: UKESM1-0-LL, ensemble: 'r(16:19)i1p1f2', grid: gn, institute: MOHC}
- {dataset: UKESM1-1-LL, ensemble: r1i1p1f2, grid: gn, institute: MOHC}
diagnostics:
diagnostic_name:
variables:
tas:
exp: historical
mip: Amon
project: CMIP6