HPC

This page page needs expert review!

Introduction

HPC stands for High Performance Computing, which is shorthand for the next generation of large-scale computing that is being developed. (This is being written in 2017.) We expect mu2e to be come more involved in these sites and advanced styles of programming, and taking full advantage of these resource by the time we take data.

As physical limitations to Moore's and other scaling laws approach, the economics of computing has evolved. Instead of many machines with large memory and few cores, the trend is towards fewer machines with limited memory and many cores. This forces users into reducing memory where possible and sharing as much of the rest as possible. It also motivates making programs with parts that can run in parallel on the cores.

On top of this trend is another strong trend for the user to bring their operating system environment with them. Instead of the user just submitting a job to a machine with a known environment (such as SL6) coordinated with the site, the user submits their jobs as a virtual machine plus their workflow. The worker node runs the virtual machine which then starts the workflow. In this way, the site can be much more generic and host almost any type of user. The virtual machine may or may not be visible to the user. The user may simply check a box for what VM to use, or may build a custom VM for the experiment, or even specific to the workflow. An light-weight way to provide a virtual machine is by a Docker image. This object provides everything but the kernel - OS utilities and libraries, products and collaboration code.

Methods

There are different levels of, loosely speaking, memory sharing and threading...

One method that is being used at several sites is the "container", such as Docker or Singularity. (See a FIFE summary.) This is a type of a pseudo virtual machine that can maximize flexibility and memory sharing between your jobs. The worker node is usually running a lunix kernel, but the exact OS is not important. An image is built up from the all libraries and executables that the workflow will need. Contents such has the OS libraries, lab products, and art libraries might be put in an image by an expert. The experiment or user then would add the particular version of mu2e code libraries and utilities needed. This image is moved to the worker node, and then one or many copies of the job workflow (in this language, each is a container) can be started from the image. The memory sharing comes from the many jobs running on one set of shared libraries in memory instead of each job being run in its own virtual machine, with its own copy of the libraries. The convenience and interoperability comes from the fact that the user is bringing their whole environment to the worker node and therefore is very likely to run successfully and correctly.

There are different levels of parallelization with one run of an exe:

• bit-level - larger buses and FPU's, fixed by the hardware
• multithreading - the CPU decides at the level of a few instructions what can be computed in parallel on the fly
• vectorization of code - each line of code is examined by a programmer and threads inserted where possible. For example, if an initialization of several objects is independent, then each can be done in a separate parallel thread.
• function level - some tasks are naturally independent and and be done in a parallel, such as tracing a particle or fitting a track. This approach is being pursued in geant, where each particle is naturally a thread.
• module level - art's rules enforce that analysis and output modules can be run independently and in parallel, for example
• event level - a central input module distributes whole events out to a set of threads ("multi-scheduling"). This is the focus of the art team in 2017. Will use Intel Thread Building Blocks (TBB)

Machines

See also Doc 9040 mu2e might be able to run on Edison and CORI I with little effort. MIRA with dedicated effort.

• NERSC (National Energy Research Scientific Computing Center)
  • Edison - 2.57 Petaflops, Cray XC30, Intel "Ivy Bridge" processor,2.5 GB of memory per core, uses Docker, no special compiler. Max wal time 36h. To be retired 3/31/19.
  • CORI Phase I - Cray XC40, Intel Xeon Haswell processors, 2 GB of memory per thread, uses Docker. Max wal time 48h.
  • CORI Phase II - second generation Intel Xeon Phi, Knights Landing (KNL), 4 hardware threads per core, 350 MB/thread requires function-level parallel programming, prefers special compilation. The standard executable runs on this machine, but with considerable inefficiency. Max wall time 24h. KNL training slides

Simulation job relative timing test. KNL is slower because the architecture is much more specialized and we do not take advantage of it. It may still be the most valuable because it has the most cores.

FermiGrid	1.0
OSG	1.2
Edison	0.56
Cori I	0.66
Cori II KNL	2.6

• ALCF (Argonne Leadership Computing Facility)
  • BeBop - KNL. Here time is charged 0.58 times slower than wall on KNL, to account for the slower clock
  • MIRA 10 petaflops IBM Blue Gene/Q, 200 MB/core, requires special compilation
  • Theta 9.65 petaflops, 4GB/core
  • Aurora - future

• SDSC (San Diego Supercomputer Center)
  • COMET, similar to CORI I.
• XSEDE - a collaboration helping users with supercomputers

Activities and References

Geant4 with threading is a project called Geant4-MT. This version allows many Geant4 events to be generated in parallel with one call to the Geant event generator. Each set of Geant4 events would appear in one art event, so this will be a different type of event which will have to be accomodated. Some recent talks: talk1 talk2.

The art team is developing the event-level and module-level threading capability for art executables. As of fall 2017, the infrastructure is in place and the process of rolling it out and matching user needs is underway. There is an art multi-threading forum. There is an associated mailing list "art-mt-forum@listserv.fnal.gov" (see listserv instructions).

You can see mu2e talks and docs on the topic of threading at this doc-db search link.

Some tools being investigated by SCD (2017)

• HDF5 - a unique suite of technologies and supporting services that make possible the management of extremely large and complex data collections. (Replacement for root ntuples, interface to industry machine learning.)
• Apache Spark a fast and general engine for large-scale data processing. (In-memory computing..)
• Dask - flexible parallel computing library for analytic computing in Python
• Apache Flink An open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications