This site is old; please update your bookmark to https://cbiit.github.com/sdsi/candle!
This page will show you how to make your Python* script CANDLE-compliant.
We assume that your code already runs successfully as a standalone script on Biowulf. For help with this, reach out to us at the contact information at the bottom of this page.
Step (1): Place your entire Python script in a run
function
Before:
# First line of your code
# Second line of your code
# ...
# Last line of your code
After:
def run(gParameters):
# First line of your code
# Second line of your code
# ...
# Last line of your code
return(history) # ensure you define history in your code; it is likely the return variable from model.fit()
Don’t forget to even wrap your import statements in the run
function and to indent your code.
Step (2): Prepend the initialize_parameters
function
Before:
def run(gParameters):
# First line of your code
# Second line of your code
# ...
After:
def initialize_parameters():
# Add the candle_keras library to the Python path
import sys, os
sys.path.append(os.getenv("CANDLE")+'/Candle/common')
# Instantiate the Benchmark class
# The values of the prog and desc parameters don't really matter
import candle_keras as candle
mymodel_common = candle.Benchmark(os.path.dirname(os.path.realpath(__file__)), os.getenv("DEFAULT_PARAMS_FILE"), 'keras', prog='myprogram', desc='My CANDLE example')
# Read the parameters (in a dictionary format) pointed to by the environment variable DEFAULT_PARAMS_FILE
gParameters = candle.initialize_parameters(mymodel_common)
# Return this dictionary of parameters
return(gParameters)
def run(gParameters):
# First line of your code
# Second line of your code
# ...
Step (3): Append a main
function and allow the script to be called from the command line
Before:
# ...
# Last line of your code
return(history) # ensure you define history in your code; it is likely the return variable from model.fit()
After:
# ...
# Last line of your code
return(history) # ensure you define history in your code; it is likely the return variable from model.fit()
def main():
gParameters = initialize_parameters()
run(gParameters)
if __name__ == '__main__':
main()
Step (4): Declare the variables you would like to vary in a hyperparameter optimization
For example, if you define the variable epochs
in the second line of your code, do this:
Before:
def run(gParameters):
# First line of your code
epochs = 10
# ...
After:
def run(gParameters):
# First line of your code
epochs = gParameters['epochs']
# ...
Note that this only needs to be done once per variable, when it is first defined.
While the dictionary “keys” (e.g., the argument to gParameters
above) need not match the names of the variables being assigned, they must match the names of the keywords defined in the default parameters file, described next.
Step (5): Create a default parameters file
In order to initialize the parameters for any given run, you need a configuration file that sets their default values. Here is an example such file:
[Global Params]
epochs=20
batch_size=128
activation='relu'
optimizer='rmsprop'
Note that this file is to be pointed to by the DEFAULT_PARAMS_FILE
variable in the submission script submit_candle_job.sh
described here.
Testing your script
You should be able to run your CANDLE-compliant script outside of the CANDLE “supervisor” by running your script manually on Biowulf. This is an excellent test to show whether you’ve made your code properly CANDLE-compliant.
For example, we would test that we’ve made our template MNIST example properly CANDLE-compliant by placing the following in a file called run_without_candle.sh
and running sbatch run_without_candle.sh
on Biowulf:
#!/bin/bash
#SBATCH --partition=gpu
#SBATCH --mem=20G
#SBATCH --gres=gpu:k80:1
#SBATCH --time=00:05:00
#SBATCH --job-name=mnist_test_no_candle
# Always load the candle module for, e.g., finding the Benchmark class... DO NOT MODIFY
module load candle
# Load desired Python version or Conda environment
# Load other custom environment settings here
module load python/3.6
# Set the file that the Python script below will read in order to determine the model parameters
export DEFAULT_PARAMS_FILE="$CANDLE/Supervisor/templates/model_params/mnist1.txt"
# Run the model
python $CANDLE/Supervisor/templates/models/mnist/mnist_mlp.py
In fact, this exact file is one of our CANDLE templates. Feel free to run that file as-is to help you see how things work (sbatch $CANDLE/Supervisor/templates/run_without_candle.sh
) or copy it to your local directory and edit it to test how well you made your code CANDLE-compliant.
Notes
- *CANDLE supports scripts written in Python by default; documentation for our language-agnostic scripts is forthcoming.
- Sample CANDLE-compliant scripts can be found here.
- More details on writing CANDLE-compliant code can be found here.
- You can check GPU utilization on a compute node using
watch nvidia-smi
(use Ctrl+C to exit out of thewatch
command). - The
Supervisor
top directory referred to by links above that point to GitHub is present on Biowulf at$CANDLE/Supervisor
(if you have thecandle
module loaded) or/data/BIDS-HPC/public/candle/Supervisor
. In the links, just remove “blob/fnlcr/” or “tree/fnlcr/”. E.g., the file described by the link https://github.com/ECP-CANDLE/Supervisor/blob/fnlcr/templates/model_params/mnist1.txt is located on Biowulf at/data/BIDS-HPC/public/candle/Supervisor/templates/model_params/mnist1.txt
. - Feel free to email Andrew Weisman with any questions.