{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "63635ce7-0c74-422a-963d-a92e06ffa3bf",
   "metadata": {},
   "source": [
    "# Composing recipes\n",
    "\n",
    "This notebooks shows how to fill the datasets section in a [recipe](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2c4dfa8a-b1d2-4d3f-9dc1-4aa930501ed5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "import yaml\n",
    "\n",
    "from esmvalcore.config import CFG\n",
    "from esmvalcore.dataset import Dataset, datasets_to_recipe\n",
    "from esmvalcore.experimental.recipe import Recipe"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ad877bb4-1ca7-4819-852d-46d462890b32",
   "metadata": {},
   "source": [
    "Configure ESMValCore so it always searches the ESGF for data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0d0a8ce7-6f42-4956-8fee-82238f5ace85",
   "metadata": {},
   "outputs": [],
   "source": [
    "CFG[\"search_data\"] = \"complete\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a210f70-89d4-4cee-86db-3e0e353a36ab",
   "metadata": {},
   "source": [
    "Here is a small example recipe, that uses the `datasets_to_recipe` function to convert a list of datasets to a recipe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "90d738a2-9934-4fd9-aeeb-edf6ba68a50c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "datasets:\n",
      "- dataset: CanESM5-1\n",
      "diagnostics:\n",
      "  diagnostic_name:\n",
      "    variables:\n",
      "      pr:\n",
      "        ensemble: r1i1p1f1\n",
      "        exp: historical\n",
      "        grid: gn\n",
      "        mip: Amon\n",
      "        project: CMIP6\n",
      "        timerange: 2000/2002\n",
      "      tas:\n",
      "        ensemble: r1i1p1f1\n",
      "        exp: historical\n",
      "        grid: gn\n",
      "        mip: Amon\n",
      "        project: CMIP6\n",
      "        timerange: 2000/2002\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tas = Dataset(\n",
    "    short_name=\"tas\",\n",
    "    mip=\"Amon\",\n",
    "    project=\"CMIP6\",\n",
    "    dataset=\"CanESM5-1\",\n",
    "    ensemble=\"r1i1p1f1\",\n",
    "    exp=\"historical\",\n",
    "    grid=\"gn\",\n",
    "    timerange=\"2000/2002\",\n",
    ")\n",
    "tas[\"diagnostic\"] = \"diagnostic_name\"\n",
    "\n",
    "pr = tas.copy(short_name=\"pr\")\n",
    "\n",
    "print(yaml.safe_dump(datasets_to_recipe([tas, pr])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "392a9235-606d-4270-b0d6-f6895dce4cde",
   "metadata": {},
   "source": [
    "A more ambitious recipe might want to use all data that is available on ESGF. We can define a dataset template with a facet value of `*` where any value can be used. This can then be expanded to a list of datasets using the `from_files()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7764bbee-83e2-4061-a3a4-103b266d55e9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1011"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset_template = Dataset(\n",
    "    short_name=\"tas\",\n",
    "    mip=\"Amon\",\n",
    "    project=\"CMIP6\",\n",
    "    exp=\"historical\",\n",
    "    dataset=\"*\",\n",
    "    institute=\"*\",\n",
    "    ensemble=\"*\",\n",
    "    grid=\"*\",\n",
    ")\n",
    "datasets = list(dataset_template.from_files())\n",
    "len(datasets)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba556a90-ba20-49f3-a103-5bbfd9c332a8",
   "metadata": {},
   "source": [
    "This results in the following recipe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "025952e4-fc31-4193-ad3a-9cef20f93449",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "datasets:\n",
      "- dataset: ACCESS-CM2\n",
      "  ensemble: r(1:10)i1p1f1\n",
      "  grid: gn\n",
      "  institute: CSIRO-ARCCSS\n",
      "- dataset: ACCESS-ESM1-5\n",
      "  ensemble: r(1:40)i1p1f1\n",
      "  grid: gn\n",
      "  institute: CSIRO\n",
      "- dataset: AWI-CM-1-1-MR\n",
      "  ensemble: r(1:5)i1p1f1\n",
      "  grid: gn\n",
      "  institute: AWI\n",
      "- dataset: AWI-ESM-1-1-LR\n",
      "  ensemble: r1i1p1f1\n",
      "...\n"
     ]
    }
   ],
   "source": [
    "for dataset in datasets:\n",
    "    dataset.facets[\"diagnostic\"] = \"diagnostic_name\"\n",
    "recipe = Path(\"recipe.yml\")\n",
    "with recipe.open(\"w\") as file:\n",
    "    yaml.safe_dump(datasets_to_recipe(datasets), file, encoding=\"utf-8\")\n",
    "\n",
    "print(recipe.read_text(encoding=\"utf-8\")[:300])\n",
    "print(\"...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56728d24",
   "metadata": {},
   "source": [
    "The above is a rather long list of datasets, therefore we provide a function\n",
    "to format recipes in a more compact way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f034972c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "datasets:\n",
      "  - {dataset: ACCESS-CM2, ensemble: 'r(1:10)i1p1f1', grid: gn, institute: CSIRO-ARCCSS}\n",
      "  - {dataset: ACCESS-ESM1-5, ensemble: 'r(1:40)i1p1f1', grid: gn, institute: CSIRO}\n",
      "  - {dataset: AWI-CM-1-1-MR, ensemble: 'r(1:5)i1p1f1', grid: gn, institute: AWI}\n",
      "  - {dataset: AWI-ESM-1-1-LR, ensemble: r1i1p1f1, grid: gn, institute: AWI}\n",
      "  - {dataset: AWI-ESM-1-REcoM, ensemble: r1i1p1f1, grid: gn, institute: AWI}\n",
      "  - {dataset: BCC-CSM2-MR, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: BCC}\n",
      "  - {dataset: BCC-ESM1, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: BCC}\n",
      "  - {dataset: CAMS-CSM1-0, ensemble: 'r(1:2)i1p1f1', grid: gn, institute: CAMS}\n",
      "  - {dataset: CAMS-CSM1-0, ensemble: r1i1p1f2, grid: gn, institute: CAMS}\n",
      "  - {dataset: CAS-ESM2-0, ensemble: 'r(1:4)i1p1f1', grid: gn, institute: CAS}\n",
      "  - {dataset: CESM2, ensemble: 'r(1:11)i1p1f1', grid: gn, institute: NCAR}\n",
      "  - {dataset: CESM2-FV2, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: NCAR}\n",
      "  - {dataset: CESM2-FV2, ensemble: r1i2p2f1, grid: gn, institute: NCAR}\n",
      "  - {dataset: CESM2-WACCM, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: NCAR}\n",
      "  - {dataset: CESM2-WACCM-FV2, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: NCAR}\n",
      "  - {dataset: CIESM, ensemble: 'r(1:3)i1p1f1', grid: gr, institute: THU}\n",
      "  - {dataset: CMCC-CM2-HR4, ensemble: r1i1p1f1, grid: gn, institute: CMCC}\n",
      "  - {dataset: CMCC-CM2-SR5, ensemble: 'r(2:11)i1p2f1', grid: gn, institute: CMCC}\n",
      "  - {dataset: CMCC-CM2-SR5, ensemble: r1i1p1f1, grid: gn, institute: CMCC}\n",
      "  - {dataset: CMCC-ESM2, ensemble: r1i1p1f1, grid: gn, institute: CMCC}\n",
      "  - {dataset: CNRM-CM6-1, ensemble: 'r(1:30)i1p1f2', grid: gr, institute: CNRM-CERFACS}\n",
      "  - {dataset: CNRM-CM6-1-HR, ensemble: r1i1p1f2, grid: gr, institute: CNRM-CERFACS}\n",
      "  - {dataset: CNRM-ESM2-1, ensemble: 'r(1:15)i1p1f2', grid: gr, institute: CNRM-CERFACS}\n",
      "  - {dataset: CanESM5, ensemble: 'r(1:25)i1p1f1', grid: gn, institute: CCCma}\n",
      "  - {dataset: CanESM5, ensemble: 'r(1:40)i1p2f1', grid: gn, institute: CCCma}\n",
      "  - {dataset: CanESM5-1, ensemble: 'r(1:20)i1p1f1', grid: gn, institute: CCCma}\n",
      "  - {dataset: CanESM5-1, ensemble: 'r(1:25)i1p2f1', grid: gn, institute: CCCma}\n",
      "  - {dataset: CanESM5-1, ensemble: 'r(24:39)i1p1f1', grid: gn, institute: CCCma}\n",
      "  - {dataset: CanESM5-1, ensemble: 'r(41:50)i1p1f1', grid: gn, institute: CCCma}\n",
      "  - {dataset: CanESM5-1, ensemble: r22i1p1f1, grid: gn, institute: CCCma}\n",
      "  - {dataset: CanESM5-CanOE, ensemble: 'r(1:3)i1p2f1', grid: gn, institute: CCCma}\n",
      "  - {dataset: E3SM-1-0, ensemble: 'r(1:5)i1p1f1', grid: gr, institute: E3SM-Project}\n",
      "  - {dataset: E3SM-1-0, ensemble: 'r(1:20)i2p2f1', grid: gr, institute: UCSB}\n",
      "  - {dataset: E3SM-1-1, ensemble: r1i1p1f1, grid: gr, institute: E3SM-Project}\n",
      "  - {dataset: E3SM-1-1-ECA, ensemble: r1i1p1f1, grid: gr, institute: E3SM-Project}\n",
      "  - {dataset: E3SM-2-0, ensemble: 'r(1:21)i1p1f1', grid: gr, institute: E3SM-Project}\n",
      "  - {dataset: E3SM-2-0-NARRM, ensemble: 'r(1:5)i1p1f1', grid: gr, institute: E3SM-Project}\n",
      "  - {dataset: E3SM-2-1, ensemble: 'r(1:5)i1p1f1', grid: gr, institute: E3SM-Project}\n",
      "  - {dataset: EC-Earth3, ensemble: 'r(1:7)i1p1f1', grid: gr, institute: EC-Earth-Consortium}\n",
      "  - {dataset: EC-Earth3, ensemble: 'r(9:25)i1p1f1', grid: gr, institute: EC-Earth-Consortium}\n",
      "  - {dataset: EC-Earth3, ensemble: 'r(101:150)i1p1f1', grid: gr, institute: EC-Earth-Consortium}\n",
      "  - {dataset: EC-Earth3-AerChem, ensemble: 'r(3:4)i1p1f1', grid: gr, institute: EC-Earth-Consortium}\n",
      "  - {dataset: EC-Earth3-AerChem, ensemble: r1i1p1f1, grid: gr, institute: EC-Earth-Consortium}\n",
      "  - {dataset: EC-Earth3-AerChem, ensemble: r1i1p4f1, grid: gr, institute: EC-Earth-Consortium}\n",
      "  - {dataset: EC-Earth3-CC, ensemble: 'r(6:13)i1p1f1', grid: gr, institute: EC-Earth-Consortium}\n",
      "  - {dataset: EC-Earth3-CC, ensemble: r1i1p1f1, grid: gr, institute: EC-Earth-Consortium}\n",
      "  - {dataset: EC-Earth3-CC, ensemble: r4i1p1f1, grid: gr, institute: EC-Earth-Consortium}\n",
      "  - {dataset: EC-Earth3-ESM-1, ensemble: r5i1p1f1, grid: gr, institute: EC-Earth-Consortium}\n",
      "  - {dataset: EC-Earth3-HR, ensemble: r1i1p1f1, grid: gr, institute: EC-Earth-Consortium}\n",
      "  - {dataset: EC-Earth3-Veg, ensemble: 'r(1:6)i1p1f1', grid: gr, institute: EC-Earth-Consortium}\n",
      "  - {dataset: EC-Earth3-Veg, ensemble: 'r(10:14)i1p1f1', grid: gr, institute: EC-Earth-Consortium}\n",
      "  - {dataset: EC-Earth3-Veg-LR, ensemble: 'r(1:3)i1p1f1', grid: gr, institute: EC-Earth-Consortium}\n",
      "  - {dataset: FGOALS-f3-L, ensemble: 'r(1:3)i1p1f1', grid: gr, institute: CAS}\n",
      "  - {dataset: FGOALS-g3, ensemble: 'r(1:6)i1p1f1', grid: gn, institute: CAS}\n",
      "  - {dataset: FIO-ESM-2-0, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: FIO-QLNM}\n",
      "  - {dataset: GFDL-CM4, ensemble: r1i1p1f1, grid: gr1, institute: NOAA-GFDL}\n",
      "  - {dataset: GFDL-ESM4, ensemble: 'r(1:3)i1p1f1', grid: gr1, institute: NOAA-GFDL}\n",
      "  - {dataset: GISS-E2-1-G, ensemble: 'r(1:4)i1p5f1', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-G, ensemble: 'r(1:5)i1p1f3', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-G, ensemble: 'r(1:10)i1p1f1', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-G, ensemble: 'r(1:20)i1p1f2', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-G, ensemble: 'r(1:20)i1p3f1', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-G, ensemble: 'r(6:10)i1p5f1', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-G, ensemble: 'r(101:102)i1p1f1', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-G, ensemble: 'r(201:202)i1p1f(5:6)', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-G, ensemble: 'r(201:210)i1p1f2', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-G, ensemble: 'r(201:210)i1p1f4', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-G, ensemble: 'r(301:310)i1p1f2', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-G, ensemble: 'r(301:310)i1p1f4', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-G, ensemble: r1i1p1f5, grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-G-CC, ensemble: r1i1p1f1, grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-H, ensemble: 'r(1:5)i1p1f2', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-H, ensemble: 'r(1:5)i1p3f1', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-H, ensemble: 'r(1:5)i1p5f1', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-1-H, ensemble: 'r(1:10)i1p1f1', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-2-G, ensemble: 'r(1:5)i1p3f1', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-2-G, ensemble: 'r(1:6)i1p1f1', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E2-2-H, ensemble: 'r(1:5)i1p1f1', grid: gn, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E3-G, ensemble: r1i1p101f1, grid: gr, institute: NASA-GISS}\n",
      "  - {dataset: GISS-E3-G, ensemble: r1i1p103f1, grid: gr, institute: NASA-GISS}\n",
      "  - {dataset: HadGEM3-GC31-LL, ensemble: 'r(1:5)i1p1f3', grid: gn, institute: MOHC}\n",
      "  - {dataset: HadGEM3-GC31-LL, ensemble: 'r(11:60)i1p1f3', grid: gn, institute: MOHC}\n",
      "  - {dataset: HadGEM3-GC31-MM, ensemble: 'r(1:4)i1p1f3', grid: gn, institute: MOHC}\n",
      "  - {dataset: ICON-ESM-LR, ensemble: 'r(1:5)i1p1f1', grid: gn, institute: MPI-M}\n",
      "  - {dataset: IITM-ESM, ensemble: r1i1p1f1, grid: gn, institute: CCCR-IITM}\n",
      "  - {dataset: INM-CM4-8, ensemble: r1i1p1f1, grid: gr1, institute: INM}\n",
      "  - {dataset: INM-CM5-0, ensemble: 'r(1:10)i1p1f1', grid: gr1, institute: INM}\n",
      "  - {dataset: IPSL-CM5A2-INCA, ensemble: r1i1p1f1, grid: gr, institute: IPSL}\n",
      "  - {dataset: IPSL-CM6A-LR, ensemble: 'r(1:33)i1p1f1', grid: gr, institute: IPSL}\n",
      "  - {dataset: IPSL-CM6A-LR-INCA, ensemble: r1i1p1f1, grid: gr, institute: IPSL}\n",
      "  - {dataset: KACE-1-0-G, ensemble: 'r(1:3)i1p1f1', grid: gr, institute: NIMS-KMA}\n",
      "  - {dataset: KIOST-ESM, ensemble: r1i1p1f1, grid: gr1, institute: KIOST}\n",
      "  - {dataset: MCM-UA-1-0, ensemble: 'r1i1p1f(1:2)', grid: gn, institute: UA}\n",
      "  - {dataset: MIROC-ES2H, ensemble: 'r(1:3)i1p4f2', grid: gn, institute: MIROC}\n",
      "  - {dataset: MIROC-ES2H, ensemble: 'r1i1p(1:3)f2', grid: gn, institute: MIROC}\n",
      "  - {dataset: MIROC-ES2L, ensemble: 'r(1:30)i1p1f2', grid: gn, institute: MIROC}\n",
      "  - {dataset: MIROC-ES2L, ensemble: r1i1000p1f2, grid: gn, institute: MIROC}\n",
      "  - {dataset: MIROC6, ensemble: 'r(1:50)i1p1f1', grid: gn, institute: MIROC}\n",
      "  - {dataset: MPI-ESM-1-2-HAM, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: HAMMOZ-Consortium}\n",
      "  - {dataset: MPI-ESM1-2-HR, ensemble: 'r(1:10)i1p1f1', grid: gn, institute: MPI-M}\n",
      "  - {dataset: MPI-ESM1-2-LR, ensemble: 'r(1:50)i1p1f1', grid: gn, institute: MPI-M}\n",
      "  - {dataset: MPI-ESM1-2-LR, ensemble: r1i2000p1f1, grid: gn, institute: MPI-M}\n",
      "  - {dataset: MRI-ESM2-0, ensemble: 'r(1:10)i1p1f1', grid: gn, institute: MRI}\n",
      "  - {dataset: MRI-ESM2-0, ensemble: r1i2p1f1, grid: gn, institute: MRI}\n",
      "  - {dataset: MRI-ESM2-0, ensemble: r1i1000p1f1, grid: gn, institute: MRI}\n",
      "  - {dataset: NESM3, ensemble: 'r(1:5)i1p1f1', grid: gn, institute: NUIST}\n",
      "  - {dataset: NorCPM1, ensemble: 'r(1:30)i1p1f1', grid: gn, institute: NCC}\n",
      "  - {dataset: NorESM2-LM, ensemble: 'r(1:43)i1p1f1', grid: gn, institute: NCC}\n",
      "  - {dataset: NorESM2-LM, ensemble: r1i1p4f1, grid: gn, institute: NCC}\n",
      "  - {dataset: NorESM2-MM, ensemble: 'r(1:3)i1p1f1', grid: gn, institute: NCC}\n",
      "  - {dataset: SAM0-UNICON, ensemble: r1i1p1f1, grid: gn, institute: SNU}\n",
      "  - {dataset: TaiESM1, ensemble: 'r(1:2)i1p1f1', grid: gn, institute: AS-RCEC}\n",
      "  - {dataset: UKESM1-0-LL, ensemble: 'r(1:4)i1p1f2', grid: gn, institute: MOHC}\n",
      "  - {dataset: UKESM1-0-LL, ensemble: 'r(5:7)i1p1f3', grid: gn, institute: MOHC}\n",
      "  - {dataset: UKESM1-0-LL, ensemble: 'r(8:12)i1p1f2', grid: gn, institute: MOHC}\n",
      "  - {dataset: UKESM1-0-LL, ensemble: 'r(13:15)i1p1f2', grid: gn, institute: NIMS-KMA}\n",
      "  - {dataset: UKESM1-0-LL, ensemble: 'r(16:19)i1p1f2', grid: gn, institute: MOHC}\n",
      "  - {dataset: UKESM1-1-LL, ensemble: r1i1p1f2, grid: gn, institute: MOHC}\n",
      "\n",
      "diagnostics:\n",
      "  diagnostic_name:\n",
      "    variables:\n",
      "      tas:\n",
      "        exp: historical\n",
      "        mip: Amon\n",
      "        project: CMIP6\n"
     ]
    }
   ],
   "source": [
    "print(Recipe(recipe).to_yaml())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.14.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}