Ada:Batch:JobTracking

Ada/LSF Job Tracking Techniques

Introduction

The information on this page will help you understand the LSF scheduling behavior on Ada, but will not tell you when your job will run. Scheduling is a complex task. There are many factors that contribute to whether a job will exit the queue next. These next few sections will cover common bottlenecks users encounter, but should not be considered a comprehensive guide.

After reviewing the following sections, you should be able to estimate whether your job will start running quickly or if you should expect to wait.

Hardware Limitations

Ada is composed mostly of 20 core + 64GB nodes. There is a small set of 20 core + 256GB nodes. Mixed between these two sets are some GPU and PHI nodes.

The compute node hardware details can be seen at: Ada Hardware Summary.

The compute node batch job memory limitations can be seen at: Ada Memory Specification Clarification.

Advice

It is much more common for all the 256GB, GPU, or 1/2TB hardware to be occupied than the 64GB hardware. If your program works on a 64GB general compute node (<54GB of RAM), then ensure your job file fits on 64GB nodes.

If you need GPU nodes, then you want to request as few nodes as possible. Requesting many GPU nodes almost guarantees that you will be waiting in queue for a while. The same applies to PHI and TB nodes.

Overall Impact: Major

Typical Job Requests

It is most common for users to request either 2^n cores or 20*n cores. This means that there are many single-node jobs that request 1, 2, 4, 8, 16, or 20 cores.

Common memory per core requests are typically 2700MB on 64GB nodes and 12700MB on 256GB nodes. This is in part due to memory limitations and the SU surcharge for memory equivalent cores.

Advice

If possible, it is best to fit your job into one of the common job configurations. This is because an 4 core + (2700*4)MB job fits nicely with 16 core + (2700*16)MB jobs.

On the other hand, a 15 core + (2700*15)MB job won't fit with the common 8 core job.

Likewise, a 2 core + (20000*2)MB job will need a node with about 40GB of RAM unreserved. It is advised that users take advantage of the full 2700MB per core they can request without extra charge, so this 40GB job will likely need a node with at most 5 cores already reserved. This is can cause major stalls if you need multiple 2-core-40GB nodes for a single job.

Overall Impact: Minor

Batch Queue Structure

The queue structure determines several limitations. While the queues enforce per-queue and per-user limits, queues themselves are determined by the job's properties. Thus, the queue structure also enforces limits on walltime, hardware requests, and job configurations.

With a few exceptions, the Ada batch queue implementation is fire-and-forget. Most users do not need to be concerned about which queue they get placed into as the limits are relatively high.

Special cases (many jobs, long jobs) will want to observe queue limits and structure their workflow around the established structure.

Advice

Overall Impact: Minor in the typical case

Overall Impact: Major in the special case

Job Priority

View priority