Graphcore IPUs

From the ACES login node, ssh into poplar1 system.

[username@aces1 ~]$ ssh poplar1

Set up the Poplar SDK environment

In this step, set up several environment variables to use the Graphcore tools and Poplar graph programming framework.

[username@poplar1 ~]$ source /opt/gc/poplar/poplar_sdk-ubuntu_20_04-[ver]/poplar-ubuntu_20_04-[ver]/enable.sh 
[username@poplar1 ~]$ source /opt/gc/poplar/poplar_sdk-ubuntu_20_04-[ver]/popart-ubuntu_20_04-[ver]/enable.sh

[ver] indicates the version number of the package.

Example commands with an existing version on ACES:

source /opt/gc/poplar/poplar_sdk-ubuntu_20_04-3.1.0+1205-58b501c780/poplar-ubuntu_20_04-3.1.0+6824-9c103dc348/enable.sh
source /opt/gc/poplar/poplar_sdk-ubuntu_20_04-3.1.0+1205-58b501c780/popart-ubuntu_20_04-3.1.0+6824-9c103dc348/enable.sh

mkdir -p /localdata/$USER/tmp
export TF_POPLAR_FLAGS=--executable_cache_path=/localdata/$USER/tmp`  
export POPTORCH_CACHE_DIR=/localdata/$USER/tmp
# export POPLAR_LOG_LEVEL=INFO
# export POPLIBS_LOG_LEVEL=INFO

Set up environments of frameworks for IPU

PyTorch (Poptorch)

Set up PyTorch (Poptorch) for IPU

The local home dir is small (300G total). You can store large files in /localdata/username (or use localdata symlink from your home dir). /localdata has 3.5TB available.

[username@poplar1 ~]$ cd /localdata/$USER
[username@poplar1 localdata]$ python -m venv -p python3 poptorch_test
[username@poplar1 localdata]$ source poptorch_test/bin/activate
[username@poplar1 localdata]$ python -m pip install -U pip
[username@poplar1 localdata]$ python -m pip install <sdk_path>/poptorch_x.x.x.whl

For /poptorch_x.x.x.whl, you can use /opt/gc/poplar/poplar_sdk-ubuntu_20_04-3.1.0+1205-58b501c780/poptorch-3.1.0+98660_0a383de63f_ubuntu_20_04-cp38-cp38-linux_x86_64.whl, which exists on ACES

Clone a copy of the Graphcore tutorials repository and change the directory to mnist

[username@poplar1 localdata]$ git clone https://github.com/graphcore/tutorials.git
[username@poplar1 localdata]$ cd tutorials/simple_applications/pytorch/mnist/

Install the dependencies and run the model

[username@poplar1 mnist]$ pip install -r requirements.txt
[username@poplar1 mnist]$ python mnist_poptorch.py

TensorFlow 2

Set up TensorFlow 2 for IPU

The local home dir is small (300G total). You can store large files in /localdata/NetID (or use localdata symlink from your home dir). /localdata has 3.5TB available.

[username@poplar1 ~]$ cd /localdata/$USER
[username@poplar1 localdata]$ python -m venv venv_tf2 -p python3.6
[username@poplar1 localdata]$ source venv_tf2/bin/activate
[username@poplar1 localdata]$ python -m pip install <sdk_path>/tensorflow_x.x.x.whl

For /tensorflow_x.x.x.whl, you can use /opt/gc/poplar/poplar_sdk-ubuntu_20_04-3.1.0+1205-58b501c780/tensorflow-2.6.3+gc3.1.0+246224+2b7af067dae+amd_znver1-cp38-cp38-linux_x86_64.whl, which exists on ACES

Clone a copy of the Graphcore tutorials repository and change the directory to tensorflow2/keras/completed_demos

[username@poplar1 localdata]$ https://github.com/graphcore/tutorials.git
[username@poplar1 localdata]$ cd tutorials/tutorials/tensorflow2/keras/completed_demos/

Run the model

[username@poplar1 completed_demos]$ python completed_demo_ipu.py

gc-monitor

gc-monitor is a command-line utility that provides a comprehensive overview of IPU device information, including details about any processes that are utilizing the IPUs in the form of a table.

Graphcore Documentation can be found at https://docs.graphcore.ai/en/latest/