CESM Tutorial: Basic Script Modifications

1. CESM Tutorial:Basic Script Modifications Cécile Hannay and Dani Coleman, National Center for Atmospheric Research (NCAR) Reference: This tutorial borrows material presented in:“CESM1 Tutorial: Basic Modifications” by Christine Shields (2011)http://www.cesm.ucar.edu/models/cesm1.0/cesm/cesm1_tutorial.pdf

2. Some basic info • We will use the CESM code located locally on alps, no need to checkout or download any input data. (see: location of these directories next page) 2. We will run with resolution T31_gx3v7. • Scripts will automatically be configured for you using the code/script base prepared uniquely for this tutorial.

3. Review: Summary of CESM directories on alps • Code/data shared by everybody - CESM code directory/home/s07hsu00/cesm_collection/cesm1_0_3/ - Inputdata directory /work/s07hsu00/cesm_inputdata • Your own directories (everyone has his/her own) - Cases directory: ~/cesm_case/ - Run directories /work/$LOGNAME/cesm_run - Short-term archive directory/work/$LOGNAME/cesm_archive where $LOGNAME is your login.

4. Customizing your ~/.profile Let’s customize your ~/.profile and create shortcuts that bring you to the key CESM directories on alps. • If you don’t have a file called ~/.profile, copy the file /home/s07hsu00/.profile to your home directory: cp /home/s07hsu00/.profile ~/.profile • If you have a file called ~/.profile, add at the end of the file: ~/.profile and ### CESM path shortcuts alias cesm_collection='cd /home/s07hsu00/cesm_collection/cesm1_0_3/' alias cesm_data='cd /work/s07hsu00/cesm_inputdata' alias cesm_case='cd ~/cesm_case/' alias cesm_run='cd /work/$LOGNAME/cesm_run/' alias cesm_archive='cd /work/$LOGNAME/cesm_archive’

5. Customizing your ~/.profile For the changes to be effective in the current window, you need to type: source ~/.profile To automatically add the shortcuts in every window you open later, add to your .bashrcthe line: source ~/.profile When you type cesm_collection, it brings you the the cesm collection directory. When you typecesm_data, it brings you to the inputdata directory When you type cesm_case, it brings you to the case directory When you type cesm_run, it brings you to the run directory When you type cesm_archiveit brings you to the archive directory

6. Review: Creating, configure build and run a new case Reminder: you can run CESM with a set of 4 commands (lines in red). # go to CESM root directory cd/home/s07hsu00/cesm_collection/cesm1_0_3 # go into scripts subdir cd scripts # (1) create a new case in your home dir create_newcase –case ~/cesm_case/case01 -res T31_g37 -compset B_1850 -mach alps # go into the case you just created in the last step cd ~/cesm_case/case01/ # (2) configure the case configure -case # (3) build the executable ./case01.alps.build # (4) submit an initial run to the batch queue bsub < case01.alps.run

7. Review of case01 • Yesterday we ran case01. We will review the results now and explore the contents of the various directories of case01. • To navigate between directories, use the shortcuts we created in ~/.profile When you type cesm_collection, it brings you the the cesm collection directory. When you typecesm_data, it brings you to the inputdata directory When you type cesm_case, it brings you to the case directory When you type cesm_run, it brings you to the run directory When you type cesm_archiveit brings you to the archive directory • If your run didn’t complete: use instructions on the next page.

8. Review of case01: If you run didn’t complete • Try to understand why your run didn’t complete. Look at the log files. Try to restart the run case01. Ask for help if necessary. • Once your run is restarted, it will take a while before it completes. Don’t wait until it is completed. Please look at the results from the user “s07hsu00”. Case directory: /home/s07hsu00/cesm_case/case01 Run directory: /work/s07hsu00/cesm_run/case01 Archive directory: /work/s07hsu00/cesm_archive

9. Review of case01: case directory • Case directory (shortcut: cesm_case) Go to the case directory and explore. After the job completes, CESM creates a “timing” directory. Explore the “timing” directory. How long did your job take to complete? What was the “throughput”, i.e. estimated model years per actual day? After the job completes, CESM creates a “log” directory that contains the log files. Go to your “logs” directory and explore the “logs” files.

10. Review of case01: case directory (README file) In your case directory, in addition to your scripts, you will find a README directory and a README.case file. • README directory: automatically created documentation files, please review. • README.case file: one line of text which echos your create_newcase command. In README.case, we highly recommend YOU document any changes you make to the default scripts. It is YOUR “paper trail” and opportunity to list modifications.

11. Review of case01: run directory and archive directory • Run directory (shortcut: cesm_run) Go to the run directory. Can you find your data ? Why ? Where are they ? • Archive directory (shortcut: cesm_archive) When the run ends, output data is moved into a short term archiving directory. Go to the archive directory and explore the structure.

12. How to make simple modifications to your run Now we will learn more about the files env_*.xml located in your case directory. The env_*.xml files contain variables used to setup, configure, build, and run CESM. Yesterday we modified variables in env_run.xml to extent case01 from 5 days to 2 years. Today, we will study some common variables in env_run.xml and env_conf.xml. Then we will create two cases to apply these new learnings.

13. Review: the tool “xmlchange” to modify “xml” files When modifying “xml” files, the user may use the tool, xmlchange (this is the preferred method) NB: the user is also free to use her/his editor of choice, i.e. vi or emacs However, use extra caution if you directly edit the files. • For help, type xmlchange -help 2. Example: To continue an initial run, you set the variable CONTINUE_RUN in env_run.xmlto TRUE To edit env_run.xml via the xmlchange tool, type xmlchange -file env_run.xml -id CONTINUE_RUN -val TRUE

14. variables in env_run.xml The variables in env_run.xml MAY BE CHANGED ANYTIME during a run. Common variables to change include 1. RESUBMIT  sets the number of times to resubmit the run 2. STOP_OPTION  sets the run length time interval type, i.e. nmonths, ndays, nyears or a specific date 3. STOP_N  sets the number of intervals (set by STOP_OPTION) to run the model during the specified wallclock time. Wallclock time is set in your *.run file and is a measure of the actual time.

15. variables in env_run.xml 4. CONTINUE_RUN sets a continuation run that extends an existing CCSM run. A setting of TRUE implies a continuation run. Note: if RESUBMIT is > 0 and it is an initial run (i.e. CONTINUE_RUN=FALSE), CONTINUE_RUN will automatically update to TRUE upon completion of initial run. 5. INFO_DBUG  sets level of stdout (standard out) print statements. If debugging, a higher value may be set. 6. DOUT_S  turns on short-term archiving (short term archiving) . DOUT_S is TRUE by default. Take some time to review all other env_run.xml settings…. You can find a list of all the xml variables set in the env_run.xmlfileathttp://www.cesm.ucar.edu/models/cesm1.0/cesm/cesm_doc_1_0_4/a5013.html#AEN5017

16. how to set your run length env_run.xml STOP_OPTION and STOP_N control the length of the run per computer job submission. A typical simulation is comprised of many job submissions. (You can only stay in the computer queue for a specified time. This queue time limit is often shorter than the desired simulation length.) Question: The special tutorial version of T31_gx3v7 CESM on alps simulates ~7 model years per wallclock day on 48 processors. If you want to run 10 years and you request a maximum wallclock (computing time) of 24 hour per submission, what values should be set for STOP_OPTION, STOP_N, and RESUBMIT? Try to answer the question on your own. The solution is next page.

17. env_run.xml Question: If you want to run 10 years and you request a maximum wallclock of 24 hour per submission, what values should be set for STOP_OPTION, STOP_N, and RESUBMIT? Answer: You are requesting 24-hour jobs. Model runs 7 yrs/day. So you get about 7 years per 24-hour submission. We need a total of 2 submissions to run 10 years. STOP_OPTION = nyears, STOP_N = 5, RESUBMIT = 1

18. Variables in env_conf.xml Sample variables specified in this file include: 1. RUN_TYPE  startup, hybrid, branch 2. RUN_REFCASE  if branch/hybrid, case name you are starting from 3. RUN_REFDATE  if “ “ , date stamp of reference case you are starting from 4. CCSM_CO2_PPM  CO2 value to be propagated to CAM and CLM 5. CAM_NAMELIST_OPTS  CAMnamelist options that differ from default values 6. CLM_NAMELIST_OPTS  CLM namelist options that differ from default values 7. CICE_NAMELIST_OPTS  CICE namelist options for that differ from default values We will look at these options tomorrow

19. run_types CESM has four “types” of runs: STARTUP: All model components are initialized from basic default initial conditions. HYBRID: - The atmosphere and land are initialized from initial condition files generated by a user-specified CESM simulation. - The ocean and ice are initialized from restart files generated by a user-specified CESM simulation. - Initial conditions and restart files use the same reference case and reference date. BRANCH: All model components are initialized from restart files generated by a user-specified CESM simulation. CONTINUE: Continuation runs for all run types. Default T31_g37 cases using compset B_1850_CN are HYBRID runs that are initialized from the CESM1 1850 Control at T31_gx3v7 .

20. More about branch run More about branch runs…. A branch run is useful if you have an experiment which only slightly differs from your control, but you want to initialize with the spun-up basic state of your control. We can do this by initializing the model with restart files from each component model taken from the end of the control. Example: You are running a present day control and have completed 200 years of steady-state (i.e. unchanging) forcing. You want to run a 2xCO2 experiment off the end of your control. You accomplish this by creating a new case, configuring your model to run as a BRANCH case, and specifying the reference case name (your control) and reference case date (the end date of your control) in your env_conf.xml file before you issue the configure command. You would also change your CCSM_CO2_PPMV value in env_conf.xml to the desired value before configuring.

21. case02: hybrid run case02: Create a new case called “case02” on alps using compset B_1850_CN and resolution T31_gx3v7. Configure and build the model. Run the model for 1 month. Try to do this on your own. We give you hints for each step. We provide solutions if you are stuck.

22. case02: hints for each step (1) Create a new_case called case02. (2) Look at the file env_conf.xml located in your case directory. Compare it with the env_conf.xml from case01. Hint: use the command diff to compare the files. (3) Modify the env_*.xml files to set the run length to 1 month. Hint: Use xmlchange to modify the env_*xml files (4) Configure, build and run. (5) Explore the run directory. What are the files b40.t31x3.037* ?

23. case03: create a branch run case03: Create a BRANCH run from the end of case01. Name this new case case03. Double the CO2. Run 1 month. Once again, try to do this on your own. We give you hints for each step. We provide solutions if you are stuck.

24. case03: create a branch run • Create a new_case called case03. Hint: for a branch run, you need to use the same compset and resolution than for case01 (2) Modify the env_*.xml files to set a branch run (modify the run type and info about the reference case), to double the CO2 value, and the length of the run (3) Look at the file env_conf.xml located in your case directory. Compare it with the env_conf.xml from case01 and case02. Hint: use the command diff to compare the files.

25. case03: create a branch run (4) Configure and build the model. • You will need to manually copy your restart and rpointer files from the case01 short-term archive directory into your case03 run directory. • Submit your job

26. Post configure changes Examples of what can be changed after the configure command has been issued: • Swap out a default inputdata set for a home-grown dataset • Namelist changes for POP or CPL • Namelist changes that you forgot to include in env_conf.xml before configuring. • Namelist changes from buildnml scripts created when cloning another case. Examples of what CANNOT be changed after the configure command has been issued: • RUN_TYPE specifications

27. Post configure changes WARNING!!!! . It is very easy to create an incompatibility in the scripts. (You should not assume the model will crash. It may run, just give you wrong answers).

28. Time Step Changes Where and When to Change Time Steps Examples of Pre and Post configure changes When the model crashes due to large, temporary instabilities, one method to overcome the problem is to change the time step. This is typically done in either the atmosphere or ocean components. CAM: dtime, in CAM namelist. Change in env_conf.xml pre or post configure. POP: dt_count in POP namelist. Edit pop.buildnml.csh post configure. GOTCHAS?

29. Time Step Changes GOTCHAS Examples of why we need to know what we are doing! 1. CLM time step = CAM time step, dtime must be changed in CLM namelist as well 2. RTM time step (in lnd.buildnml.csh) must be compatible with the CLM time step. The river runoff model (embedded within CLM) requires the number of rtm time steps per 3 hours be specified. Therefore, if dtime changes, so must rtm_nsteps. 3. ATM_NCPL in env_conf.xml specifies the number of coupling intervals per day between the atmosphere and the coupled system. The atmosphere and land are coupled every time step. If the CAM time step is changed, ATM_NCPL must be updated to reflect the new number of coupling intervals per day.

30. Documentation What files are for documentation purposes? Buildconf/*.input_data_list, Buildconf/pop2doc CaseDocs, CaseStatus, LockedFiles, README.case DOCUMENT everything you do! • A paper trail of your procedures and thoughts is good scientific practice. The README.case file is the perfect place to write notes. You will thank yourself months (years) later, when you are trying to figure out what you did oh-so-long ago!

31. Log Files Model runtime standard output $RUNDIR/* $LOGDIR/* Files are gzipped after model completion. Restore by typing gunzip <logfile>. yyddmm = year, month, day nnnnnn = timeid stamp Log Files: During model execution: After model completion: atm.log.yyddmm-nnnnnn.gz lnd.log.yyddmm-nnnnnn.gz ocn.log.yyddmm-nnnnnn.gz ice.log.yyddmm-nnnnnn.gz cpl.log.yyddmm-nnnnnn.gz ccsm.log.yyddmm-nnnnnn.gz

32. Post Run Tips Check logs • Check your log files to make sure there are no hidden problems and to verify the model is running smoothly and as you expect. The log files may also help you verify your modifications were included in your run. Check output • Check your history files. It is a good idea to run a small test sample of your experiment before launching your full production run. For example, if you want to run a 500 year control with various modifications, first run 10 years. Check the history output files and verify the model is running as you designed before continuing with the full 500 years. It is always best to find errors early, rather than later, in the run. Check timings • Check your timings. After model completion, a timing subdirectory will be placed in your scripts directory. Check the timings after several job completions to verify that the model is running efficiently and as expected. Double check your timings with the CESM default timings for your specific model resolution and machine. Default timings can be found at: http:/www.cesm.ucar.edu/models/cesm1.0/timing/