Skip to content

Under Construction

Using Drona Composer

Drona Composer provides a 100% graphical interface to create and run any type of workflow.

Custom Workflows

Elements of a Workflow

Any custom environment will contain the following specification files:

  • schemas.json
  • maps.json
  • template.txt
  • driver.sh (potentially optional in future)
  • utils.py (optional)
  • additional_files.json (optionsl, for advanced environments)

Creating the Form

The description for the from will be declared in File schema.json, in the environment's directory. This file contains all the individual elements that will be shown in the form. Every element will have the following minimum structure 

"<ElementName>": {
      "type": "<type>",
      "label": "<labelName>",
      "name": "<varName>",
      "help": "Here you can provide some extra information to assist the reasercher"
 }

ElementName identifies the element. It's recommended to provide a descriptive name. The type field designates the type of element. The next subsections will discuss all available types. Every element has an assigned label that will be shown in the form together with the element. The name field designates the variable name that will store the value the researcher enters in the form. For example, in a number element (discussed next) the researcher migt enter a value of 100. That value will be stored in variable varName. The help field provides an option to show additional information when hovering over the label. The help field is optional (in theory, the name and label fields are also optional)

The following sections will discuss all the available elements. For an actual production example, check out schema.json from the Generic environment that is part of ACES Drona composer

Number element

The number element is a very generic element, that shows a single input field where the user can only enters numerical values. The element includes field to set the minimum, maximumm, and step size. Below, see the specific fields for number elements.

"<ElmentName>": {
   "type": "number",
   "min": 0,
   "max": 100,
   "step": 2,
   "placeholder": "0-100, step of 2"

}

Text element

The text element is a very generic element, that shows a single input field where the user can specify any text input. The element includes one single text specific field to set the default value.

"<ElementName>": {
   "type": "text",
   "value": "<Default text>"

}

Time element

The time element shows a combined input with three number, representing days, hours, and minutes. Keep in mind, the element will return the values in HH:MM format.

"<ElementName>": {
   "type": "time"
}

Unit element

The unit elelment shows a combined input with a number field and a dropdown with options for the unit. This element is typically used to specify memory, where the dropdown has options for MB and GB.

"<ElementName>": {
   "type": "unit",
   "units": [
      { "label": "MB", "value": "M" },
      { "label": "GB", "value": "G" }
   ]

}

Module element

The module element shows a combined search bar and toolchain dropdown. When the researcher starts typing in the search bar, the element will show all available modules for the selected toolchain matching the search string. Selected modules will be automatically assigned to a predefined variable named module_list. Note this element uses an external SQL database to retrieve modules from. 

"<ElementName>": {
   "type": "module",
   "toolchains": [
     {
        "label": "intel/2023a",
        "value": "modules-intel2023a"
     },
     {
        "label": "Foss/2023a",
        "value": "modules-foss2023a"
     }
  ]

}

Checkbox element

The checkbox element shows a single checkbox. When the checkbox is not checked the assigned variable will (through the name field) will be set to the empty string. If the the checkbox is checked, the assigned variable will be set to the value of the value field

"<ElementName>": {
   "type": "checkbox",
   "value": "Yes"
},

Radiogroup element

The radiogoup element shows a group of radio buttons where only one can be selected. The element provides an option list with labels and values for every choice. the variable name assoicated with this this element (designated by the name field), will be set to the selected value field in the options list.

"<ElementName>": {
   "type": "radioGroup",
   "options": [
      {
         "label": "Radio 1",
         "value": "value1"
      },
      {
         "label": "Radio 2",
         "value": "value2"
      }
   ]
}

select element

The select element shows a dropdown of options where only one can be selected. The element provides an option list with labels and values for every choice. the variable name assoicated with this this element (designated by the name field), will be set to the selected value field in the options list.

"<ElementName>": {
   "type": "select",
   "options": [
      {
         "label": "Option 1",
         "value": "option1"
      },
      {
         "label": "Option 2",
         "value": "option2"
      }
    ]
}

dynamic_select element

The dynamic_select element shows a dropdown of options where only one can be selected. This is similar to the select element mentioned above. The only different is that the options list is generated by an external script. The retriever field designates the script to be called. The variable name assoicated with this element (designated by the name field), will be set to the selected value field in the options list.

"<ElementName>": {
   "type": "dynamic_select",
   "retriever" : "<retrieval_script>"
}

Picker element

The picker element shows a combined element where users can select a file or directory (either from researcher's local machine or from the a location on the machine where Drona composer os running). If the researcher selects a file from their local machine, Drona composer will upload the file as well. The variable name assoicated with this element (designated by the name field), will be set to the selected file or directory.

"<ElementName>": {
   "type": "picker",
   "remoteLabel": "Upload",
   "localLabel": "Cluster",
   "showFiles": true
}

Upload element

The upload element shows an upload button, where the researcher will be able to upload fikes and directories. Drona composer will automatically copy all the uploaded files to the job directory. 

"<ElementName>": {
   "type": "uploader"
}

Conditional expressions

The above section show all the currently available elements. By default, Drona composr will show all of these elements. However, many environments might only want to show certain elements depending on the selected values in other elements. For example, Drona composer provides the condition field for this. This field can be added to any element. 

"condition": "<ElementName>.value1

In the above example the element containing the condition will only be shown if another element ElementName has been selected with a value op value1. Currently elements can only depend on values selected in radiogroups, select, and dynamic_select. Future versions will be able to handle more complex conditions.

Mapping input values to placeholders

For an actual production example of a mapping file, check out maps.json from the Generic environment that is part of ACES Drona composer

Substitutuing placeholders

Adding warnings

Drona composer provides an option to add warnings to the preview window. This can be used to provide feedback to the researcher based on values entered in the form, for examle, on the ACES clusters the maximum walltime for PVC jobs (i.e. jobs where the researcher requests a PVC GPU) is 2 days. The screenshot below shows the warning when the researcher tries to request one or multiple PVC GPUs and a walltime of five days.

Drona Warning

To add the warning, you can call the following convenience function inside any Python function definition in utils.py

drona_add_warning(String)

For example, above we saw a screenshot of a warning when a researcher requests a PVC and a walltime of more than 2 days. The snipet below is actual code from utils.py in the Generic environment that adds the warning: 

if gpu == "PVC" and total_hours > (24*2):
    drona_add_warning("Requested  walltime "+walltime+" (hh:mm). Reducing to max of 48 hours in GPU/PVC queues")

The Drona engine will iterate over all the warnings and show them in the preview window. In future versions of the Composer we might add options to add severity levels for warnings.

Adding additional files

By default Drona composer will process (substitute placeholders, preview, and copy to the job directory) the following two files:

  • template.txt (which typically is the job script that is being created)
  • driver.sh

This is sufficient for most environments, but for more advanced environments it's possible additional files need to be proccesed. For example, some enviroments might require different templates, depending on the input the researcher provided in the form. Drona composer provides two ways to accomplish this.

Statically adding files

You can specify a list of additonal files that Drona composer will process in a JSON file named additional_files.json. Suppose in our environemnt we have two Python scripts that will be executed, named pre_process.py and post_process.py. The format of this JSON file will be as follows:

{
    "files" : ["pre_process.py","post_process.py"]
}

Drona composer will look for file additional_files.json in the environment directory and include all the entries as additional files to be processed. In the example, pre_process.py and post_process.py will be handled exactly the same as driver.sh and template.txt. Any placeholders will be substituted and they will be shown in the fully editable preview window. Once the researcher clicks the submit button, those files will be copied to the job directory.

Dynamically adding files

Another way to add additional files is to call the following convenience function inside any Python function definition in utils.py

drona_add_additional_file(String)

As You can see, this is very similar to the convenience function to add warnings, discussed above. As an example, let's supose, we have an environment that needs to run pre-processing step, that depends on values the researcher provided in the form.

if gpu == "PVC":
    drona_add_additional_file("pre_process_pvc.py")
else:
    drona_add_additional_file("pre_process.py")

In the above example, if the researcher selected a PVC GPU, it will include a specialized pre processing script (pre_process_pvc.py). The files that are added dynamically will be handled exactly the same as driver.sh and template.txt. Any placeholders will be substituted and they will be shown in the fully editable preview window. Once the researcher clicks the submit button, those files will be copied to the job directory.

Adding additional mappings

A previous section discussed the mapping from input values the researcher provided in the form to plaeholders that will be substituted in the template files (e.g. template.txt and driver.sh). The mapping is defined in file maps.json. Drona also provides a function to add mappings dynamically. This is very useful for cases where multiple placeholder depend on a similar set of input variables and conditions. To add mappings dynamically, use the following function

drona_add_mapping(String,String)

where the first String represents the lefhand side in the mapping amd second String represents the righhand side of the mapping.

Back to top