Elastic Analysis Facility (EAF)

Introduction

This page is intended as a guide to the Fermilab Elastic Analysis Facility (EAF) for Mu2e collaborators. The official EAF documentation from SCD can be found at test-1-eaf.fnal.gov.

EAF is a web-based platform intended for Python analysis and ML tasks. The key to EAF is the container-based infrastructure, which distinguishes it from traditional virtual machines. The benefit of this approach is that underlying resources (the hardware) can be swapped without breaking the container, adding elasticity.

EAF is a a powerful and flexible platform, ideal for running Mu2e Python analyses and ML tasks.

Accessing EAF

EAF is entirely web-based at https://analytics-hub.fnal.gov, that is, there is no interactive ssh access at present. This means that in order to access EAF from outside the FNAL network you will either need to use the Fermilab VPN, or set up a proxy. You will also need an active services account.

There are number of ways to set up a proxy. One method, which is endorsed by SCD and should be applicable to all operating systems, is use Firefox with some modified network settings. Instructions on how to do this are given below.

1. Ensure you have a valid kerberos ticket: check with klist, run kinit <username>@FNAL.GOV to make a new one. Make sure to replace <username> with your FNAL username.
2. Open a terminal and start an ssh tunnel to an FNAL machine on which you have an account (such as mu2egpvm01).

   ssh -f -N -D 9999 <username>@mu2egpvm01.fnal.gov

3. Open Firefox and type about:config into the address bar, click OK to ignore the warning.
4. Use the search bar to change the parameters by to the values shown in the table below.

Parameter	Value
network.proxy.socks	127.0.0.1
network.proxy.socks_port	9999
network.proxy.socks_remote_dns	true
network.proxy.type	1

To stop the proxy, change network.proxy.type back to its default value by pressing the reset button to the right of the edit button.

See https://library.fnal.gov/off-site-electronic-access-instructions for more information.

Starting an EAF server

Once you have started your proxy server and ssh tunnel (if you are off-site), go the EAF home page at

https://analytics-hub.fnal.gov

on which you should see a welcome page which will invite you to sign in with your Fermilab Services (SSO) account.

Make sure you're using the same browser as the one you have configured to use a proxy! You should see a page with a Start My Server button. If you click on this button it will take you to a Server Options page.

Follow the instructions below to start an AL9 server.

1. Go to the FIFE server box:

1. Click CPU Interactives.
2. Select AL9.
3. Scroll down to bottom of the page and click Start.

The server may take a few minutes to start up. You will see a page like this

There is also an option to start a server from a Mu2e AL9 image, which is now deprecated. Instead, it is recommended to use the mu2e_env environment described in The Mu2e environment.

JupterHub and the EAF area

On loading an EAF server you be land on a JupyterHub launcher page. From here you should see various options to applications such as a terminal, a python notebook, a python file editor, or an interactive python console. It also provides options run the interactive applications with different Python kernels, where kernels represent both the execution engine and the packages, libraries, and dependencies available to the Python interpreter. You can also access these options by clicking the blue "+" button on the top right.

If you open a terminal and run pwd, you will see that a user area has been automatically created for you in /home. From here, you have direct access to the /exp/mu2e/app and /exp/mu2e/data areas. Access to /pnfs requires xroot, which is included in the mu2e_env environment described in The Mu2e environment.

Each user has eight guaranteed cores, a 64 GB memory limit, and 23 GB of storage on their EAF user area.

Conda/Mamba

Conda and Mamba are open-source packages and environment management systems. Mamba is a C++ reimplementation of Conda: it has the same command syntax but is supposed to be more efficient. Mamba is the tool used in this example to set up our Mu2e environment, and can be used to create and manage custom user environments.

Upon initialising, Mamba will write some lines to your $HOME/.bashrc files, so first make sure that your $HOME/.bash_profile is set up to read from the .bashrc by adding the following lines to your .bash_profile using any command line text editor (such as emacs or vim).

# Get aliases and functions
if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi

To start using Mamba, again using a terminal from the JupyterHub launcher, and initialise Mamba as shown below.

mamba init

This will prompt you to open a new shell, so close the current session and start a new one.

You only need to do this once.

The Mu2e environment

Custom environments allow for flexibility beyond the base image, which is managed by SCD. To provide users with the tools needed to conduct analysis, we have an installed a Python environment on /exp/data that can used on by EAF and the virtual machines. This will eventually be moved to the /cvmfs area.

To use this environment, create a symlink in your EAF ~/.conda directory that points to the current environment, as shown below.

ln -s /cvmfs/mu2e.opensciencegrid.org/env/ana/current ~/.conda/envs/mu2e_env

You can then activate the environment using mamba.

mamba activate mu2e_env

Once inside the environment, you have access to the libraries listed below.

matplotlib
pandas
uproot
scipy
scikit-learn
pytorch
tensorflow
jupyterlab
notebook
statsmodels
awkward
urllib3==1.26.16
ipykernel
conda-pack
fsspec-xrootd
htop
vector
plotly 
dash
anapytools==2.0.0

As of version 1.2.0, an interactive kernel called mu2e_env is automatically available upon activating the environment. If you are using a version prior to this, and would like to use the environment interactively, you must install an ipython kernel in your user area with following command.

python -m ipykernel install --user --name mu2e_env --display-name "mu2e_env"

After refreshing the page, you should then see mu2e_env appear as an option when launching a notebook or interactive console.

You can also install the mu2e_env environment in your user area from a YAML file, for example:

mamba env create -f /exp/mu2e/data/users/sgrant/EAF/env/yml/mu2e_env.vX.X.X.yml
mamba activate mu2e_env.vX.X.X

In any case, you should now see the environment display name as a prefix on your command line.

(current) [<username>@jupyter-<username> ~]$ 

To use this environment on the Mu2e virtual machines, you can use the activate/deactivate scripts in the environment directory directly. So, to activate:

source /cvmfs/mu2e.opensciencegrid.org/env/ana/current/bin/activate

and to deactivate:

source /cvmfs/mu2e.opensciencegrid.org/env/ana/current/bin/deactivate

It also might be convenient to alias this command in your .my_bashrc file, to something like

alias pystart="source /cvmfs/mu2e.opensciencegrid.org/env/ana/current/bin/activate"
alias pystop="source /cvmfs/mu2e.opensciencegrid.org/env/ana/current/bin/deactivate"

Once you have activated your environment, you can run a brief test to check that your packages are working:

$ python
>>> import numpy

This first time you import a package may take a minute -- subsequent imports should be faster (thanks to file caching).

When using the the Mu2e environment on EAF along with muse, it is important to activate/reactivate the environment after running any muse setup commands, for example:

$ mu2einit
$ muse setup ops
$ pystart # (source /cvmfs/mu2e.opensciencegrid.org/env/ana/current/bin/activate)

This will ensure that your Python environment variables are set correctly.

Change log

1.0.0

• This was the first official version of the environment. It contains a set of packages which mirror the contents of its predecessor pyana, as well as including anapytools.

matplotlib
pandas
uproot
scipy
scikit-learn
pytorch
tensorflow
jupyterlab
notebook
statsmodels
awkward
urllib3==1.26.16
ipykernel
conda-pack
htop
anapytools==2.0.0

1.1.0

This includes:

• The addition of the vector package, which is needed by mu2epyutils;
• A patch for "Qt platform plugin" and "Fontconfig" errors seen when running the matplotlib GUI. The patch involved introducing the script 1.1.0/etc/conda/activate.d/env_vars.sh which sets the following environment variables upon activation:

export QT_QPA_PLATFORM_PLUGIN_PATH="$CONDA_PREFIX/lib/qt6/plugins/platforms" # Matplotlib GUI plugin
export FONTCONFIG_FILE="$CONDA_PREFIX/etc/fonts/fonts.conf" # Matplotlib fonts

1.1.1

This includes a patch for a conflict with the default Python interpreter which occurs when using the environment alongside muse. This patch involved explicitly pointing to the environments Python in PATH and pointing to the environments packages in the PYTHONPATH. This variables were added env_vars.sh so as to be available upon activation:

export PATH="$CONDA_PREFIX/bin:$PATH" # Interpreter
export PYTHONPATH="$CONDA_PREFIX/lib/python3.12/site-packages:$PYTHONPATH" # Packages

1.2.0

This includes:

• Additional packages, plotly and dash, requested for DQM development;
• An in-built kernel called mu2e_env which becomes available automatically after activating the environment. This works on both EAF and the VMs, and means that users do not have to install their own local kernel to use the environment interactively. The kernel is installed within the environment, here 1.2.0/share/jupyter/kernels/mu2e_env.v1.2.0, and is described by a kernel.json which simply points to the Python currently in-use (the one in the environment). As an additional complication, when you install a kernel into a prefix that does not contain a default python kernel (as I have done here), ipykernel automatically creates one. So, the environment also contains 1.2.0/share/jupyter/kernels/python3. This creates all kinds of problems, but conda-pack complains if it doesn't exist. As a workaround, I replaced this kernel with a symlink back to the environment kernel so that python3/kernel.json -> mu2e_env.v1.2.0/kernel.json. Finally, in order to ensure that jupyter can see the kernel, I also included the following to env_vars.sh:

export JUPYTER_PATH="$CONDA_PREFIX/share/jupyter/kernels:$JUPYTER_PATH" # Path to kernel

Custom environments

To start entirely from scratch, you can create a clean custom environment with the commands below.

mamba create -q -y -n my_env 
mamba activate my_env

You can then install whatever packages you need using mamba install <package_name>.

anapytools

Along with some standard Python libraries, mu2e_env comes with a some additional utilities from https://github.com/Mu2e/anapytools.git.

At present, anapytools allows users to interface with SAM and /pnfs from EAF, and provides a multithreading tool. These can be imported as packages from anapytools called read_data and parallelise.

Before reading files, you will need a valid access token/certificate. Run the following:

source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh
kinit ${USER}@FNAL.GOV
/cvmfs/mu2e.opensciencegrid.org/bin/vomsCert

To create a file list from a SAM dataset:

$ python
>>> from anapytools.read_data import DataReader
>>> reader = DataReader()
>>> file_list = reader.get_file_list(defname='nts.mu2e.CeEndpointMix1BBSignal.Tutorial_2024_03.tka')

To read a file from /pnfs using xroot:

$ python
>>> from anapytools.read_data import DataReader
>>> reader = DataReader()
>>> file = reader.read_file(filename='nts.sgrant.CosmicCRYExtractedCatTriggered.MDC2020ae_best_v1_3.001205_00000000.root') 

An example of read_data in use is shown below. The read_data functions also include a quiet flag to suppress their printouts, if needed.

To run parallel reads on a list of files using multithreading, run:

$ python
>>> from anapytools.read_data import DataReader
>>> from anapytools.parallelise import ParallelProcessor
>>> reader = DataReader()
>>> processor = ParallelProcessor()
>>> file_list = reader.get_file_list(defname='nts.sgrant.CosmicCRYExtractedCatTriggered.MDC2020ae_best_v1_3.root', quiet=False)
>>> def process_function(filename):
        file = reader.read_file(filename, quiet=True)
        return 
>>> processor.multithread(process_function, file_list)

An example of parallelise in use is shown below.

More tools will be added in future.