Vector Engine
Introduction
The NEC Vector Engine (VE) compute node in ACES has 8 VE Type 20B-P cards. Each VE card has 8 VE cores and 48GB HBM2 memory.
Accessing Vector Engines
This VE compute node is available via the nec partition.
Interactive
To access the VE compute node interactively to develop your programs against the VE cards:
srun --partition nec --pty --time=4:00:00 --nodes=1 --gres=gpu:ve:8 /bin/bash
The requested number of VEs can be 1 to 8.
To setup your interactive job environment for the NEC compiler:
export PATH=/opt/nec/ve/bin/:$PATH
source /opt/nec/ve/mpi/3.4.0/bin64/necmpivars.sh
Turn on printing details about VE card usage:
export VE_PROGINF=DETAIL
Job Submission
To submit batch jobs against the VE node, include the following Slurm directives in your job scripts:
#SBATCH --partition=nec
#SBATCH --gres=gpu:ve:N
The requested number of VEs can be 1 to 8.
Information
MPI Usage
Example MPI runs:
mpirun –ve 0 –np 8 executable # run on 1 VE card (ve card 0) with 8 VE cores
mpirun –ve 0-1 –np 16 executable # run on 2 VE cards (VE card 0 and 1) with 8 VE cores per card
mpirun –ve 0-7 –np 64 executable # run on 8 VE cards (VE cards 0-7) with 8 VE cores per card
References
Additional details and tutorials:
NEC Vector Engine: Vectorization for HPC applications VASP on NEC VE
NEC Compiler user manuals: