Cycle Control

Revision History: File created on 13 May 2009.

The CycleControl class is found in the module initexit. It is derived from the standard python dictionary object. It is the only core object of CTDAS that is automatically created in the pipeline, the user (normally) does not need to modify or extend it. The class is created based on options and arguments passes on the command line when submitting your main CTDAS job.

Valid options are defined in

da.tools.initexit.ParseOptions()

Function parses options from the command line and returns the arguments as a dictionary. Accepted command line arguments are:

Argument	Meaning
-v	verbose output in log files
-h	display help
-r	start a simulation by recovering from a previous crash

With the name of a valid rc-file, the CycleControl object is instantiated and validated. An example rc-file looks like this::

! Info on the data assimilation cycle

time.restart        : False                     ! Restart from an existing run T/F
time.start          : 2000-01-01 00:00:00       ! Start time of first cycle
time.finish         : 2000-01-08 00:00:00       ! End time of last cycle
time.cycle          : 7                         ! length of each cycle, 7 means one week
time.nlag           : 5                         ! number of cycles in one smoother window
dir.da_run          : ${HOME}/tmp/test_da       ! the run directory for you project

! Info on the DA system used

da.system           : CarbonTracker             ! an identifier for your inversion system
da.system.rc        : da/rc/carbontracker.rc    ! the settings needed in your inversion system

! Info on the forward model to be used

da.obsoperator         : TM5                                ! an identifier for your observation operator
da.obsoperator.rc      : ${HOME}/Modeling/TM5/tm5-ctdas.rc  ! the rc-file needed to run youobservation operator
da.optimizer.nmembers  : 30                                 ! the number of ensemble members desired in the optimization

The most important method of the CycleControl object are listed below:

class da.tools.initexit.CycleControl(opts=[], args={})

This object controls the CTDAS system flow and functionality.

Initialize()

This method determines how to proceed with the cycle. Three options are implemented:

1. Fresh start : set up the required file structure for this simulation and start
2. Restart : use latest da_runtime variables from the exec dir and restart
3. Recover : restart after crash by getting data from restart/one-ago folder

The choice that gets executed depends on the presence of

# the -r option on the command line, this triggers a recover # the time.restart : True option in the da.rc file

The latter is automatically set if the filter submits the next cycle at the end of the current one, through method SubmitNextCycle().

The specific call tree under each scenario is:

1. Fresh Start

   • dummy = SetupFileStructure() <- Create directory tree

2. Restart

   • dummy = SetupFileStructure()
   • dummy = RandomSeed() <- Read the random seed from file

3. Recover

   • dummy = SetupFileStructure()
   • dummy = RecoverRun() <- Recover files from restart/one-ago dir, reset time.start
   • dummy = RandomSeed()

And is always followed by a call to

• ParseTimes()
• WriteRc(‘jobfilename’)

Finalize()

Finalize the da cycle, this means writing the save data and rc-files for the next run. The following sequence of actions occur:

• Write the randomseed to file for reuse in next cycle
• Write a new rc-file with time.restart : True, and new time.start and time.end
• Collect all needed data needed for check-pointing (restart from current system state)
• Move the previous check pointing data out of the way, and replace with current
• Submit the next cycle

CollectRestartData()

Collect files needed for the restart of this cycle in case of a crash, or for the continuation of the next cycle. All files needed are written to the restart/current directory. The list of files included is read from the attribute “RestartFileList” which is a simple list of files that can be appended by other objects/methods that require restart data to be saved.

Note

Before collecting the files in the RestartFileList, the restart/current directory will be emptied and recreated. This prevents files from accumulating in the restart/current and restart/one-ago folders. It also means that if a file is missing from the RestartFileList, it will not be available for check-pointing if your run crashes or dies!

Currently, the following files are included:

• The da_runtime.rc file
• The randomseed.pickle file
• The savestate.nc file
• The files in the ObservationOperator.RestartFileList, i.e., restart data for the transport model

Note

We assume that the restart files for the ObservationOperator reside in a separate folder, i.e, the ObservationOperator does not write directly to the CTDAS restart dir!

MoveRestartData(io_option='restore')

Store or restore model state to/from a restart directory.

Two IO options are available:

1. io_option = restore : Get data from restart.oneago directory
2. io_option = store : Save data to restart.oneago directory

In case of a ‘store’ command the restart.oneago folder is re-created so that the contents are empty to begin with.

SubmitNextCycle()

Submit the next job of a DA cycle, this consists of

• Changing to the working directory from which the job was started initially
• create a line to start the master script again with a newly created rc-file
• Submitting the jobfile

If the end of the cycle series is reached, no new job is submitted.

CleanUpCycle()

Nothing to do for now anymore

SetupFileStructure()

Create file structure needed for data assimilation system. In principle this looks like:

• ${da_rundir}
• ${da_rundir}/input
• ${da_rundir}/output
• ${da_rundir}/exec
• ${da_rundir}/diagnostics
• ${da_rundir}/analysis
• ${da_rundir}/jobs
• ${da_rundir}/restart/current
• ${da_rundir}/restart/one-ago

Note

The exec dir will actually be a simlink to the directory where the observation operator executable lives. This directory is passed through the da.rc file.

Note

The observation input files will be placed in the exec dir, and the resulting simulated values will be retrieved from there as well.

RecoverRun()

Prepare a recovery from a crashed run. This consists of:

• copying all data from the restart/one-ago folder (MoveRestartData()),
• replacing all rc-file items with those from the da_runtime.rc in the restart/current dir
• resetting the seed of the random number generator to the value it had before the crash (RandomSeed())
• replacing the output dir name, since it has the sample time in it...

RandomSeed(action='read')

Get the randomseed and save it, or read the random seed and set it. The seed is currently stored in a python pickle file, residing in the exec directory

Two important attributes of the CycleControl object are:

1. DaSystem, an instance of a DaSystem
2. DaPlatForm, an instance of a PlatForm

Other functions in the module initexit that are related to the control of a DA cycle are:

da.tools.initexit.StartLogger()

start the logging of messages to screen

da.tools.initexit.ValidateOptsArgs(opts, args)

Validate the options and arguments passed from the command line before starting the cycle. The validation consists of checking for the presence of an argument “rc”, and the existence of the specified rc-file.