View source for Elastic Analysis Facility (EAF)

[[Category:EAF]] [[ category:Introduction]]
= Introduction = 

This page is intended as a guide to the Fermilab Elastic Analysis Facility (EAF) for Mu2e collaborators. The official EAF documentation from SCD can be found at https://eafjupyter.readthedocs.io/en/latest/.

EAF is a web-based platform intended for Python analysis and ML tasks. The key to EAF is the container-based infrastructure, which distinguishes it from traditional virtual machines. The benefit of this approach is that underlying resources (the hardware) can be swapped without breaking the container, adding elasticity. 

EAF is a a powerful and flexible platform, ideal for running Mu2e Python analyses and ML tasks. 

[[File:EAF_scheme.png|768px|center|EAF.]]

[[Category:EAF]] [[ category:Accessing EAF]]

= Accessing EAF = 

EAF is entirely web-based, that is, there is no interactive <code>ssh</code> access at present. This means that in order to access EAF from outside the FNAL network you will either need to use the Fermilab VPN, or set up a proxy. You will also need an active services account. 

There are number of ways to set up a proxy. One method, which is endorsed by SCD and should be applicable to all operating systems, is use Firefox with some modified network settings. Instructions on how to do this are given below. 

# Ensure you have a valid kerberos ticket: check with <code>klist</code>, run <code>kinit <username>@FNAL.GOV</code> to make a new one. Make sure to replace <code><username></code> with your FNAL username.
# Open a terminal and start an <code>ssh</code> tunnel to an FNAL machine on which you have an account (such as <code>mu2egpvm01</code>). <pre>ssh -f -N -D 9999 <username>@mu2egpvm01.fnal.gov</pre>
# Open Firefox and type <code>about:config</code> into the address bar, click OK to ignore the warning. 
# Use the search bar to change the parameters by to the values shown in the table below.
<!-- 
<div align="center">
<table class="center" style="margin: auto;"> -->
<div style="text-align: center;">
{| class="wikitable" style="display: inline-block;"
|-
! Parameter
! Value
|-
| network.proxy.socks 
| 127.0.0.1
|-
| network.proxy.socks_port 
| 9999
|-
| network.proxy.socks_remote_dns 
| true
|-
| network.proxy.type 
| 1
|}
</div>

To stop the proxy, change <code>network.proxy.type</code> back to its default value by pressing the reset button to the right of the edit button.

See https://library.fnal.gov/off-site-electronic-access-instructions for more information.

[[Category:EAF]] [[ category:Starting an EAF server]]

= Starting an EAF server =

Once you have started your proxy server and <code>ssh</code> tunnel (if you are off-site), go the EAF home page at 

https://analytics-hub.fnal.gov 

on which you should see a welcome page which will invite you to sign in with your Fermilab Services (SSO) account. 

Make sure you're using the same browser as the one you have configured to use a proxy! You should see a page with a <code>Start My Server</code> button. If you click on this button it will take you to a <code>Server Options</code> page. 

[[File:StartEAF.png|768px|center|start server for EAF]]

Follow the instructions below to start an AL9 server. 

# Go to the <code>FIFE</code> server box:

[[File:EAFFIFE.png|768px|center|EAF FIFE.]]

# Click <code>CPU Interactives</code>.
# Select <code>AL9</code>.
# Scroll down to bottom of the page and click <code>Start</code>.

The server may take a few minutes to start up. You will see a page like this

[[File:ServerStarting.png|768px|center|EAF FIFE.]]

There is also an option to start a server from a Mu2e AL9 image, which is now deprecated. Instead, it is recommended to use the <code>mu2e_env</code> environment described in [[#The Mu2e environment|The Mu2e environment]]. 

[[Category:EAF]] [[ category:JupterHub and the EAF area]]

= JupterHub and the EAF area = 

On loading an EAF server you be land on a <code>JupyterHub</code> launcher page. From here you should see various options to applications such as a terminal, a python notebook, a python file editor, or an interactive python console. It also provides options run the interactive applications with different Python kernels, where kernels represent both the execution engine and the packages, libraries, and dependencies available to the Python interpreter. You can also access these options by clicking the blue "+" button on the top right. 

[[File:JupyterHub.png|768px|center|JupyterHub]]

If you open a terminal and run <code>pwd</code>, you will see that a user area has been automatically created for you in <code>/home</code>. From here, you have direct access to the <code>/exp/mu2e/app</code> and <code>/exp/mu2e/data</code> areas. Access to <code>/pnfs</code> requires <code>xroot</code>, which is included in the <code>mu2e_env</code> environment described in [[#The Mu2e environment|The Mu2e environment]].

Each user has eight guaranteed cores, a 64 GB memory limit, and 23 GB of storage on their EAF user area.

[[Category:EAF]] [[ category:Conda/Mamba]]

= Conda/Mamba =

Conda and Mamba are open-source packages and environment management systems. Mamba is a <code>C++</code> reimplementation of Conda: it has the same command syntax but is supposed to be more efficient. Mamba is the tool used in this example to set up our Mu2e environment, and can be used to create and manage custom user environments. 

Upon initialising, Mamba will write some lines to your <code>$HOME/.bashrc</code> files, so first make sure that your <code>$HOME/.bash_profile</code> is set up to read from the <code>.bashrc</code> by adding the following lines to your <code>.bash_profile</code> using any command line text editor (such as <code>emacs</code> or <code>vim</code>). 

<pre>
# Get aliases and functions
if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi
</pre>

To start using Mamba, again using a terminal from the JupyterHub launcher, and initialise Mamba as shown below. <pre>mamba init</pre>This will prompt you to open a new shell, so close the current session and start a new one. 

You only need to do this once. 

[[Category:EAF]] [[ category:The Mu2e environment]]

= The Mu2e environment = 

Custom environments allow for flexibility beyond the base image, which is managed by SCD. To provide users with the tools needed to conduct analysis, we have an installed a Python environment on <code>/exp/data</code> that can used on by EAF and the virtual machines. This will eventually be moved to the <code>/cvmfs</code> area.

To use this environment, create a symlink in your EAF <code>.conda</code> directory that points to the environment, as shown below.

<pre>
ln -s /exp/mu2e/data/users/sgrant/EAF/env/mu2e_env ~/.conda/envs/mu2e_env
</pre>

You can then activate the environment using mamba.

<pre>
mamba activate mu2e_env
</pre>

Once inside the environment, you have access to the libraries listed below.

<pre>
matplotlib
pandas
uproot
scipy
scikit-learn
pytorch
tensorflow
jupyterlab
notebook
statsmodels
awkward
urllib3==1.26.16
ipykernel
conda-pack
fsspec-xrootd
mu2etools==0.0.1
</pre>

To use these packages interactively, you can install an <code>ipython</code> kernel in your user area with following command.

<pre>
python -m ipykernel install --user --name mu2e_env --display-name "mu2e_env"
</pre>

After refreshing the page, you should then see <code>mu2e_env</code> appear as an option when launching a notebook or interactive console. 

If you prefer, you can to install a copy of the environment locally from the environment <code>YMAL</code> file. This method should allow you to customise the environment if needed.

<pre>
mamba env create -f /exp/mu2e/data/users/sgrant/EAF/env/mu2e_env/environment.yml
mamba activate mu2e_env
</pre> 

In any case, you should now see <code>(mu2e_env)</code> as a prefix on your command line. 

<pre>
(mu2e_env) [<username>@jupyter-<username> ~]$ 
</pre>

To you this environment on the Mu2e virtual machines, you can use the <code>activate</code>/<code>deactivate</code> scripts in the environment directory directly. So, to activate:

<pre>
source /exp/mu2e/data/users/sgrant/EAF/env/mu2e_env/bin/activate
</pre>

and to deactivate: 

<pre>
source /exp/mu2e/data/users/sgrant/EAF/env/mu2e_env/bin/deactivate
</pre>

It also might be convenient to alias this command in your <code>.my_bashrc</code> file, to something like

<pre>
alias activate_mu2e_env="source /exp/mu2e/data/users/sgrant/EAF/env/mu2e_env/bin/activate"
alias deactivate_mu2e_env="source /exp/mu2e/data/users/sgrant/EAF/env/mu2e_env/bin/deactivate"
</pre>
 

[[Category:EAF]] [[ category:Custom environments]]

= Custom environments =

To start entirely from scratch, you can create a clean custom environment with the commands below. 

<pre>
mamba create -q -y -n my_env
mamba activate my_env
</pre>

You can then install whatever packages you need using <code>mamba install <package_name></code>. 
[[Category:Mu2e EAF tools]] [[ category:Mu2e EAF tools]]

= mu2etools =

Along with some standard Python libraries, <code>mu2e_env</code> contains a set of custom tools defined in https://github.com/sam-grant/mu2etools.git. This contains some generalised methods from Yuri Oksuzian's https://github.com/oksuzian/mu2etools. 

At present, <code>mu2etools</code> allows users to interface with <code>SAM</code> and <code>/pnfs</code> from EAF, and provides a multithreading tool. These are tools can be imported as packages from <code>mu2etools</code> called <code>read_data</code> and <code>parallelise</code>.

Before reading files, you will need a valid access token/certificate. Run the following:

<pre>
source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh
kinit ${USER}@FNAL.GOV
/cvmfs/mu2e.opensciencegrid.org/bin/vomsCert
</pre>

To create a file list from a SAM dataset:

<pre>
from mu2etools import read_data as rd
file_list = rd.get_file_list(defname='nts.mu2e.CeEndpointMix1BBSignal.Tutorial_2024_03.tka')
</pre>

To read a file from <code>/pnfs</code> using <code>xroot</code>:

<pre>
from mu2etools import read_data as rd
file = rd.read_file(filename='nts.sgrant.CosmicCRYExtractedCatTriggered.MDC2020ae_best_v1_3.001205_00000000.root') 
</pre>

An example of <code>read_data</code> in use is shown below. The <code>read_data</code> functions also include a <code>quiet</code> flag to suppress their printouts, if needed. 

[[File:read_data.png|768px|center|read_data()]]

To run parallel reads on a list of files using multithreading, run:

<pre>
from mu2etools import parallelise as pa
def process_function(filename):
    file = rd.read_file(filename, quiet=True)
    return 
pa.multithread(process_function, file_list) 
</pre>

An example of <code>parallelise</code> in use is shown below. 

[[File:parallelise.png|768px|center|parallelise.png()]]

More tools will be added as needed.