Hardware

ACES: A Dell x86 HPC Cluster

System Name:	ACES
Operating System:	Red Hat Enterprise 8
Total Compute Cores/Nodes:	11,888 cores 130 nodes
Compute Nodes:	110 Intel Sapphire Rapids Nodes 17 Intel Ice Lake Nodes 1 AMD Rome Graphcore Node with 16 Mk2 Colossus GC200 IPUs 1 Intel Ice Lake Graphcore Node with 16 Bow-2000 IPUs 1 Intel Cascade Lake Node with 8 NEC Vector Engine Type 20B-P cards
Composable Components	30 NVIDIA H100 GPUs (In 15 SPR nodes. Composability Coming Soon) 4 NVIDIA A30 GPUs 2 Bittware Agilex FPGAs 2 Intel D5005 FPGAs 2 NextSilicon coprocessors 48 Intel Optane Memory 120 Intel GPU Max 1100 (Formerly Ponte Vecchio) (Coming Soon)
Interconnect:	NVIDIA Mellanox NDR200 InfiniBand (MPI and storage) Liqid PCIe Gen4 Fabrics (composability), Gen 5 Fabrics (Coming Soon)
Peak Performance:	1.6 PFLOPS
Global Disk:	2.3PB (usable) via a DDN Lustre appliance
File System:	Lustre
Batch Facility:	Slurm by SchedMD
Location:	West Campus Data Center
Production Date:	July 2023

ACES is a Dell cluster with a rich accelerator testbed consisting of Intel Max GPUs (Graphics Processing Units), Intel FPGAs (Field Programmable Gate Arrays), NVIDIA H100 GPUs, NEC Vector Engines, NextSilicon co-processors, Graphcore IPUs (Intelligence Processing Units). The ACES cluster consists of compute nodes using a mix of the following processors:

• Intel Xeon 8468 Sapphire Rapids processors
• Intel Xeon Ice Lake 8352Y processors
• Intel Xeon Cascade Lake 8268 processors
• AMD Epyc Rome 7742 processors

The compute nodes interconnected with NVIDIA NDR200 connections for MPI and access to the Lustre storage. The Intel Optane SSDs and all accelerators (except the Graphcore IPUs and NEC Vector Engines) are accessed using Liqid's composable framework via PCIe (Peripheral Component Interconnect express) Gen4 and Gen5 fabrics.

Compute Nodes

Table 2: Details of Compute Nodes

	Sapphire Rapids Nodes	Ice Lake Nodes	Cascade Lake Node + NEC VEs	AMD Rome 7742 Node
Processor Type	Intel Xeon 8468	Intel Xeon 8352Y	Intel Xeon 8268	AMD EPYC Rome 7742
Sockets per node	2	2	2	2
Cores per socket	48	32	24	64
Cores per Node	96	64	48	128
Clock rate (Base/Turbo)	2.10GHz / 3.8GHz	2.20GHz / 3.40 GHz	2.90GHz / 3.90GHz	2.25GHz / 3.4GHz
Memory	512 GB DDR5-4800	256 GB DDR4-3200	768 GB DDR4-3200	768 GG DDR4-3200
Cache	105 MB	48 MB	35.75 MB	256 MB
Local Disk Space	1.6 TB NVMe (/tmp)	3.84 TB NVMe (/tmp)	480 GB SATA	3.5 TB NVMe /localdata

System Interconnect

The ACES compute nodes are interconnected with NDR200 links. The leaf and core switches are interconnected with NDR (400Gb) links in a fat tree topology with a 2:1 oversubscription factor.

Data Transfer Nodes

ACES has two 100Gb data transfer nodes that can be used to transfer data to ACES via the Globus Connect web interface or Globus command line. Globus Connect Server v5.4 is installed on the data transfer nodes. One data transfer node is dedicated to XSEDE users and its collection is listed as “ACCESS TAMU ACES DTN”.