Using the CCEB High Performance Computing Cluster: Terminology and Specifications
Introduction
This is the fourth blog post in a series of articles about using the CCEB cluster. An overview of the series is available here. This post focuses on the general terminology surrounding high performance computing clusters and an overview of the CCEB cluster specifications.
Most of the information presented in this article was taken from IBM resources here and Microsoft here. You should explore these sites on your own for specific details.
Cluster Versus Grid
Cluster computing uses several nodes that are made to run as a single entity. These nodes are connected by a local area network (LAN). A cluster is homogeneous in that they have all the same hardware and operating systems.
Grid computing is the segregation of resources across multiple sites. Grids are are heterogeneous. The computers that are part of a grid can run different operating systems, have different hardware, and be in different geographical locations.
Cloud computing is a new computing paradigm which provides a large pool of dynamic, scalable, and virtual resources to run processes. Think Amazon Web Services.
Grid computing is sometimes distinguished from conventional high-performance computing systems such as cluster computing in that grid computers have each node set to perform a different task/application or may have clusters in different geographical locations. A grid is a cluster but a cluster is not necessarily a grid.
Our CCEB resources were previously housed in different buildings meeting the spatial definition of a grid, but they were recently all moved into the same location. The operating system is the same across the entire machine but different machines are able to run different software programs (i.e. SAS, MATLAB, R). Technically, I believe the CCEB high performance computing resources are categorized as grid computing. Personally, I use cluster to reference the machine though. Others use grid. To be honest, for our purposes, the language of how we speak of the CCEB High Performance Computing (HPC) resources doesn’t matter. Both won’t affect how effectively we Google questions or discuss the resources among ourselves.
If you want to read more, this content was mostly taken from a really nice article in the International Journal of Advanced Research in Computer and Communication Engineering and Wikipedia. The links are provided below:
Hosts
Hosts on the cluster perform different functions.
Master host: A server host acts as the overall coordinator for the cluster, doing all job scheduling and dispatch.
Server host: A host that submits and runs jobs.
Client host: A host that only submits jobs and tasks.
Execution host: A host that runs jobs and tasks.
Submission host: A host from which jobs and tasks are submitted.
Typically, we only interact with server, client, execution, and submission hosts. The master host is working in the background after we submit jobs. In our department, we most commonly refer to all of these generically as host and don’t clarify which type of host. It can get confusing so for this document I’ll try to always clarify.
If you simply ssh onto the CCEB cluster you are put onto a submission host. No jobs can be run on this host directly but are submit and sent to run on other execution hosts. If you are part of a different queue, for example Taki or PennSIVE’s group, then they use a server host, takim, so when you ssh onto the cluster your jobs are submit from there but can also be run on content on takim. Once you submit a job from these hosts, you tasks are run on execution hosts.
More information is provided about hosts on IBM’s website here. A nice graphic about hosts is available here.
Nodes, Sockets, and Cores
A node (a.k.a. host, machine, computer) refers to an entire compute node. Each node contains 1 or more sockets. Notice, this is synonymous with host/machine.
A socket (a.k.a. numa node) refers to collection of cores with a direct pipe to memory. Each socket contains 1 or more cores. Note that this does not necessarily refer to a physical socket, but rather to the memory architecture of the machine, which will depend on your chip vendor. I pretty much never think about this level or use this language.
A core (a.k.a. processor, cpu, cpu core, logical processor) refers to a single processing unit capable of performing computations. A core is the smallest unit of allocation available in high performance computing.
When you bsub an interactive session with 1 core requested by definition you are using 1 core on 1 node or host. When you submit a normal job you can request multiple cores and unless specified they may be across multiple hosts/machines.
Job
A job is the unit of work (i.e. your code) that is running on the cluster. A job is submit using a command. The master host schedules, controls, and tracks the job according to configured policies. Jobs can be complex problems, simulation scenarios, extensive calculations, or anything that needs compute power.
This language often gets confusing because we refer to the entire thing running as a batch or array job (i.e. all 1000 of your simulations running) and we refer to each 1000 iterations as single jobs. If you ever get confused just ask for clarification.
When you submit a job it goes into a job slot or a bucket from which a single unit of work is assigned on the grid system. Hosts can be configured with multiple job slots and you can dispatch jobs from queues until all the job slots are filled. You can correlate job slots with the total number of CPUs in the cluster.
Each job is assigned a number. You can use this number to check on the memory of your job, kill or stop your job, or include when emailing issues to PMACs.
Queue
A queue is a cluster-wide container for jobs. All jobs wait in queues until they are scheduled and dispatched to hosts. Queues do not correspond to individual hosts; each queue can use all server hosts on the cluster, or a configured subset of the server hosts. When you submit a job to a queue, you do not need to specify an execution host. The master host dispatches the job to the best available execution host on the cluster to run that job. You can specify an execution host if you want but this is only useful for quality control.
Queues implement different job scheduling and control policies.
“Layout”
Figure 1: Pictoral representation of the hardware and layout of the grid. This graphic is not the accurate physical layout but simply provided as an example.
- The submission host also has nodes but cannot compute on this host. A server host can also be used to submit jobs but it is not a good idea to use this for running code. Imagine if a server host crashed then all dependent execute hosts would crash and everyone running jobs in your queue would be very angry.
The machine that you are put on after ssh-ing with ssh pennkey@scisub is simply a submission host. You are not permitted to run any processes on this host but rather have to bsub onto another machine. On the other hand, Taki’s queue puts you onto a server host. This host can both submit jobs and also run jobs. We can compute on takim but it is dangerous.
We will talk about the bsub command in the next few blog posts but this will be how you get onto an execute host to actually run code and jobs.
PMACS High Performance Computing (HPC) Grids
These specifications were true as of April 22, 2019. The internal wiki for up to date information can be found here.
PMACS Limited Performance Computing (LPC) cluster is a new grid (IBM Spectrum) and the cluster is comprised a variety of hosts running CentOS Linux. Currently, two Linux hosting servers are available to submit jobs to Grid and/or run applications in an Interactive session. The CCEB HPC Statistical Grid has been run and supported over the past 15+ years by PMACS. The CCEB HPC is an LSF (“LSF”, short for load sharing facility) software. LSF is an IBM platform.
This is industry-leading enterprise-class software that distributes work across existing heterogeneous IT resources to create a shared, scalable, and fault-tolerant infrastructure, that delivers faster, more reliable workload performance and reduces cost.
LSF balances load, allocates resources, and provides access to those resources.
This Statistical HPC is composed of 144 Matlab cores, 80 SAS cores, 144 Stata cores, and 524 unrestricted R). More information about machine specification and memory is available through the Wiki.
The current setup defaults to assigning one core for each job submitted to the HPC Grid, although proper set-up and usage of Matlab pools and R parallel packages will allow up to 12 cores to be used concurrently for a job.
Any one user may only use up to 50 cores (e.g., 50 1-core jobs) running concurrently on the grid.