Pyana

Introduction

Pyana is a UPS product provided by the Mu2e offline group. It adds several common python analysis package to you path

setup mu2e
muse setup ...
setup pyana

It has several advantages over a personal python environment

• uses cvfms disk space which is mounted everywhere
• it allows coordination with the version of python in Offline so you can use features such as PyWrap
• is enables easy sharing of scripts with other users

Content

as of v1_04_00

matplotlib
pandas
uproot
scikit-learn
torch
tensorflow
jupyterlab
notebook
statsmodels
awkward
urllib3==1.26.16

other modules, such as keras are installed as dependencies.

Notes

jupyter-lab

The following error for jupyter-lab should be fixed in v1_05_00 onwards

JupyterLab Error
JupyterLab application assets not found in "/cvmfs/mu2e.opensciencegrid.org/artexternals/python/v3_9_13/Linux64bit+3.10-2.17/share/jupyter/lab"
Please run `jupyter lab build` or use a different app directory

can be fixed by adding the switch:

--app-dir=$PYANA_FQ_DIR/share/jupyter/lab

The following errors (and similar I/O errors) writing default files to your home directory, also for jupyter-lab:

[W 2023-03-23 17:09:31.510 ServerApp] 500 PUT /api/contents/tutorial2_dqm.ipynb?1679609372139 (127.0.0.1): Unexpected error while saving file: tutorial2_dqm.ipynb disk I/O error
[W 2023-03-23 17:09:31.510 ServerApp] wrote error: 'Unexpected error while saving file: tutorial2_dqm.ipynb disk I/O error'

can be addressed by the following. The issue stems from the issue that sqlite has with the NFS file system. The workaround requires not only having /.ipython/profile_default/history.sqlite , but also ~/.local/share/jupyter/nbsignatures.db in the /tmp/ area:

1. run ipython profile create if ~/.ipython/profile_default/ipython_config.py does not exist.
2. Add c.HistoryManager.hist_file='/tmp/ipython_hist.sqlite' to the config file.
3. run jupyter notebook --generate-config or jupyter-lab --generate-config if ~/.jupyter/jupyter_notebook_config.py or ~/.jupyter/jupyter_lab_config.py does not exist.
4. Add c.NotebookNotary.data_dir = "/tmp" to each config file.

create kernel locally

jupyter kernelspec install $PYANA_FQ_DIR/share/jupyter/kernels/python3/ --user

notebook missing schemas

The command

jupyter-notebook --no-browser --port=3001

once opened in a browser gives the error

tornado.web.HTTPError: HTTP 404: Not Found (Schema not found: /cvmfs/mu2e.opensciencegrid.org/artexternals/python/v3_9_15/Linux64bit+3.10-2.17/share/jupyter/lab/schemas/@jupyterlab/docmanager-extension/plugin.json)

note that it is looking for schema in the python directory.

One way to fix this is to generate a config file:

 jupyter-notebook --generate-config

it will tell you the file it generated in ~/.jupyter, which you can edit:

c.LabServerApp.schemas_dir = '/cvmfs/mu2e.opensciencegrid.org/artexternals/pyana/v1_04_00/Linux64bit+3.10-2.17//share/jupyter/lab/schemas'
c.LabServerApp.themes_dir = '/cvmfs/mu2e.opensciencegrid.org/artexternals/pyana/v1_04_00/Linux64bit+3.10-2.17//share/jupyter/lab/themes'

unfortunately it doesn't seem to accept environmentals. There may be other ways to fix this, with JUPYTER paths variables, for example.

Anaconda

The network block described in the following section was removed on September 13, 2022. Anaconda has reconsidered it's previous decision and again considers Fermilab to be a not-for-profit organization. So once again it is legal to download and use Anaconda packages for free for use for Fermilab related activities.

Short Lived Blocking of Anaconda on the Fermilab site

Between February and September 2022, Fermilab blocked network access to the Anaconda repository at repo.anaconda.com. This was done because, for that period of time, Anaconda had notified Fermilab that the terms of their licensing agreement do not allow for free usage at the lab of the default Anaconda and Miniconda repositories. The community-supported distribution from Conda Forge, Miniforge, remains free.

During the blockage Fermilab recommended that you use Miniforge to replace the functionality provided by Anaconda or Miniconda. You continue to use Miniforge.

It is your responsibility as a Fermilab employee, user or affiliate to respect licensing agreements on Fermilab-owned hardware and networks. Some notes on using Miniforge in place of Anaconda can be found in this article.

This decision was reversed on Sept 13, 2022.