tmplate: translate generic tags in templates to content

1. Templates with tags (and R code)

First you need to create a generic template for a target class of source files. Let us assume we will write a Bash script for submitting a parallel job using at least OpenMPI (+ SLURM + environment-modules if they are available). As an example, such a template could be like this one:

# template for parallel computation
T <- readLines(system.file("examples/template.txt", package = "tmplate"))

To display the content of this template run cat(T, sep="\n") and we obtain:

<:SHELL_CALL:>
<:SLURM_PARTITION:>
<:SLURM_NODES:>
<:SLURM_TASKS:>
<:SLURM_MEMORY:>
<:SLURM_TIME:>
@r if(!any(grepl("^<:NULL:>$","<:SLURM_ARRAY:>"))) paste("<:SLURM_ARRAY:>")

@r # R chunk (only assignation)
@r if(.Platform$OS.type=="unix"){
@r     is_mod <- system("mod=$(module --dumpversion 2>&1) || mod=`echo -1`; echo $mod",intern = TRUE)
@r } else {
@r     is_mod <- "-1"
@r }

@r # R chunk (printing)
@r ifelse(is_mod=="-1", "# module environment not found", paste("<:MODULES_LOAD:>"))

<:WORKDIR:>

@r if(!any(grepl("^<:NULL:>$","<:SLURM_ARRAY:>"))) c("<:TASK:>","<:PASS_TASK:>") else paste("<:NULL:>")

<:MPI_N:>

<:MPIRUN:>

<:MESSAGE_CLOSE:>

2. Tags for variables

In the template above, tag variables are marked with <:name:> where the name in between <: and :> is a variable name that will be defined by the input arguments of the function translate. This function will translate those tag variables to their respective input values and will replace its content in the position or positions where the respective tag appears in the template. The variable names that make the template are totally arbitrary. The <:NULL:> symbol is interpreted as a NULL definition and lines containing it can be drop by setting drop = TRUE later in the translate command.

3. R code (chunks, inline or within variables)

Lines starting with @r or @R followed by one space or tabular, define chunks of R code that is also interpreted and translated by translate. The chunks of R code can be assignation or output chunks following the rules from the R package tRnslate (see tRnslate vignette). Assignation chunks are those including <- for assigning an object, while output chunks print R output to the template. Thus several assignation chunks can be placed in adjacent lines, however assignation and output chunks must be separated by one empty line (the same for consecutive output chunks). Alternatively, inline R code can be entered using <r@ code @> or <R@ code @>. Inline R code with assignation does not produce output so is replaced by blank, while inline R code producing output while modify the resulting template. Additionaly, inline R code can be used within tag variable definitions to allow different content.

4. Template tag variables and R code

The R code can use tag variables that point to the value of argument variables which are being used to modify the template content, for example in the R chunk

@r # R chunk (printing)
@r ifelse(is_mod=="-1", "# module environment not found", paste("<:MODULES_LOAD:>"))

it uses the tag <:MODULES_LOAD:> to point to the value of the argument variable MODULES_LOAD. Similarly, the content of tag variables can be modified using inline R code in the definition of the argument when calling translate. For example the argument MPI_ASK_N = '<r@ <:SLURM_ASK_NODES:> * <:SLURM_ASK_TASKS:> @>' will compute (using inline R code) the number of parallel jobs from the input arguments for the number of nodes and tasks. Note that the NULL definition in the template above is used in a R logical expression to decide whether to print or not <:SLURM_ARRAY:> into the source code. Alternatively this decision may have been done purely based on R code.

5. Environment where to evaluate

The evaluation of the inline and chunks of R code to update the input arguments and replace the tags in the template is performed in an environment that can be set by the user. As said before, this environment can contain its own objects which can also be referenced to update the input arguments and modify the content of the template.

6. The translate command

Given the template above, we can define the input arguments directly when calling translate as done below:

## remember to load: library(tmplate) or call tmplate::translate
TT <- translate(
    SHELL_CALL='#!/bin/bash',
    SLURM_SBATCH=ifelse(.Platform$OS.type=="unix", ifelse(system("clu=$(sinfo --version 2>&1) || clu=`echo -1`; echo $clu",intern = TRUE)=="-1", '<:NULL:>', '#SBATCH '), '<:NULL:>'),
    SLURM_PARTITION='<:SLURM_SBATCH:>--partition=defq',
    SLURM_ASK_NODES=2,
    SLURM_NODES='<:SLURM_SBATCH:>--nodes=<:SLURM_ASK_NODES:>',
    SLURM_ASK_TASKS=4,
    SLURM_TASKS='<:SLURM_SBATCH:>--ntasks-per-node=<:SLURM_ASK_TASKS:>',
    SLURM_MEMORY='<:SLURM_SBATCH:>--memory=2gb',
    SLURM_TIME='<:SLURM_SBATCH:>--time=1:00:00',
    SLURM_ARRAY="<:NULL:>",
    MODULES_LOAD='module load module/for/openmpi module/for/R',
    WORKDIR=ifelse('<:SLURM_SBATCH:>'!='#SBATCH','# no slurm machine','cd ${SLURM_SUBMIT_DIR}'),
    TASK="<:NULL:>",
    PASS_TASK="<:NULL:>",
    PASS_TASK_VAR="<:NULL:>",
    MPI_N="<:NULL:>",
    MPI_ASK_N='<r@ <:SLURM_ASK_NODES:> * <:SLURM_ASK_TASKS:> @>',
    R_HOME=R.home("bin"),
    R_OPTIONS='--no-save --no-restore',
    R_FILE_INPUT='script.R',
    R_ARGS='',
    R_FILE_OUTPUT='output.Rout',
    MPIRUN='mpirun --mca mpi_warn_on_fork 0 -n <:MPI_ASK_N:> <:R_HOME:>/Rscript <:R_OPTIONS:> "<:R_FILE_INPUT:>" <r@ ifelse(!any(grepl("^<:NULL:>$","<:SLURM_ARRAY:>")),"<:PASS_TASK_VAR:>","") @> <:R_ARGS:> > <:R_FILE_OUTPUT:>',
    MESSAGE_CLOSE='echo "Job submitted on $(date +%F) at $(date +%T)."',
    drop = TRUE,
    template = T
)

Here we have used a new default environment to evaluate the arguments. the argument drop = TRUE will delete any line containing <:NULL:> in it.

The output from translate is a character vector where each element is a line in the resulting file. We can print it to disk easily using cat (remember to set sep = "\n").

RECOMMENDATIONS

1. Never replace the content of a template writing the output to the same file.

2. Always check the content of the “translated” output before using it for other tasks.

3. Be cautious.