Creating Custom Workflows
A Drona environment consists of a number of files:
- A form file (schema.json), consisting of form elements (field).
- A template file (template.txt) and a driver (driver.sh) that can contain placeholders.
- Optionally, a file (additional_files.json) listing additional files used to generate the workflow.
- A mapping file (named map.json), describing how to map provided input values to placeholders.
Drona engine is the framework handling the process from showing the form to generating and actually submitting workflows. Drona will do that in a number of steps:
- Process schema.json and render it as an interactive form where researcher can enter the information.
- Iterate over map.json to generate a dictionary of placeholder-to-string values, based on input values.
- Replace all placeholders with string values in files template.txt, driver.sh, and all additional files.
- Copy all generated files to the submission directory
- execute the driver.sh file to run the workflow. Typically, the driver contains the command (e.g. sbatch) to submit the template file or one of the additional files to the batch scheduler, but it can do anything (e.g running directly on a compute node)
Drona will search for available environments in the Drona system directory and the researcher's $SCRATCH/drona_composer/environments directory. Every subdirectory under this directory represents a separate environment. When creating new environments, they can be placed in this directory
The following sections will discuss the needed files to generate a workflows and how to create them.
The Form File (file schema.json)
Every Drona environment form consists of a number of form elements (fields), declared in a file named schema.json inside the Drona environment directory. Drona provides a collection of pre-defined form elements as well as powerful conditional logic to control when specific elements will be shown.
Form Elements
File schema.json contains all the individual elements that will be shown in the form. Every element has the following structure.
"<ElementName>": {
"type": "<type>",
"label": "<labelName>",
"name": "<varName>",
"help": "Extra information to assist the researcher. Will be shown as a tooltip"
}
- ElementName It's an element identifier. It's recommended to provide a descriptive name.
- type field; designates the type of element. Examples are text, number, select, etc (for a full list of elements, check the form elements )
- label field; label text shown together with the element.
- name field; represents the variable name (will store the value entered in the form by the researcher).
- help optional field; text shown as additional information when hovering over the label.
Most form element types will have additional fields specific for the type of the element.
Conditional Expressions
By default, Drona will show all of the elements in the form but sometimes certain elements should only be shown depending on the value from another element For example, a form might contain a checkbox if email notifications should be sent. If the researcher checks the box, only then, another field should be shown where the researcher can enter an email address where the mail should be sent to. To add conditions to a Drona element, use the condition field. For example:
"condition": "<ElementName>.value1
In the above example, the element containing the condition will only be shown if element
- select
- dynamicSelect
- radioGroup
- checkbox
Conditions can consist of complex expressions, including && (and) operator, || (or) operator, and ! (negate) operator. Here is an example of more complex example of a condition
"condition": "(<Elem1>.value1 || <Elem1>.value2) && <Elem2.value3"
Template, Driver and Additional Files
Files template.txt, driver.sh, and any additional files are templates for the final workflow files. These files can contain placeholders that will be replaced with actual values by Drona. A placeholder has the form [NAME]. For example:
#SBATCH --account=[ACCOUNT]
In the above example, Drona will replace [ACCOUNT] with the actual string value. Placeholders are defined and evaluated in the mapping file rediscussed in the next section.
File driver.sh is required. This is a Linux bash script that will start the workflow. This file will be executed by Drona engine. In most cases, driver.sh will just contain the code to submit the job. For example
#!/bin/bash
echo "Submitting the job"
sbatch [mybatchfile]
File template.txt can contain any text but typically this file contains the batch script. NOTE for
historical reasons file template.txt will be renamed to
All thes files will be shown in the preview pane.
template.txt and driver.sh are actual files for a Generic environment
Additional files (file additional_files.json)
Additional files to include in the workflow can be declared in a file named additional_files.json. using the structure shown here
[
{
"file_name": "<filename>",
"preview_name": "<preview name>",
"position" " index
},
{
"file_name": "<otherfilename>",
"preview_name": "<other preview name>",
"position" " other_index
}
]
where file_name> represents the actual location of the file to include. preview_name and position are optional and define the tab name and position in the preview pane. A value pf -1 for position indicates to not inlcude this file in the preview pane.
Creating mapping from input values to placeholders (file map.json)
The mapping file (map.json) is a dictionary, mapping input values (the researcher provided in the form) to placeholders. The syntax for every element in the dictionary is as follows;
where STRING_ELEM can be any of the following
- a variable, designated by $variable
- a function call, designated by !some_function($var1, ... , $varN)
- a string literal
variable is the name field of an element defined in schema.json. The function call is a call to a Python function that is defined in optional file utils.py
For example, suppose schema.json defined elements with names cores, time, mem, and version. The snippet below shows a maps.json file with two placeholders.
{
“MODULE”: “module load Matlab/$version”
"batchopts": “!retrieve_batch_opts($cores, $time, $memory)”,
}
When Drona engine evaluates the above mapping it will replace $version provided in the form and it will call Python function !retrieve_batch_opts with the evaluated input parameters. This function will return a string.
For an actual production example of a mapping file, check out the maps.json and utils.py files from the Generic environment that is part of ACES Drona composer
Python Support Library
Drona provides a support library with functions to dynamically add additional files, mappings, and even warnings. Python functions defined inside file utils.py can call these support functions.
Adding mappings
To dynamically add additional mappings, use the following library function:
drona_add_mapping(String,String)
where the first String represents the left-hand side of the mapping (the placeholder) and the second string representing the righthand side of the mapping.
NOTE: Drona engine will evaluate dynamic mappings before evaluating static mappings (from file map.json). This means that the righthand side of a dynamic mappings can contain placeholders defined in map.json.
Adding warnings
To add warnings, use the following library function:
drona_add_warning(String)
Warnings can be useful to provide feedback to the researcher based on values entered in the form or even inform of potential issues. The example below shows an actual code snippet from utils.py in the Generic environment that adds a warning when a researchers requests a PVC accelerator and a wall time exceeding the time limits for PVCs
if gpu == "PVC" and total_hours > (24*2):
drona_add_warning("Requested walltime "+walltime+" (hh:mm). Reducing to max of 48 hours in GPU/PVC queues")
Adding additional files
To dynamically add additional files use the following library function:
drona_add_additional_file(<filename>,< preview_tab_name>, <position_in_tab>)
Only the first parameters is required.
if gpu == "PVC":
drona_add_additional_file("pre_process_pvc.py", "PVC preprocess", 1)
else:
drona_add_additional_file("pre_process.py", "Regular preprocess", 1)