Pyana
Introduction
Pyana is a UPS product provided by the Mu2e offline group. It adds several common python analysis package to you path
mu2einit muse setup ... setup pyana
It has several advantages over a personal python environment
- uses cvfms disk space which is mounted everywhere
- it allows coordination with the version of python in Offline so you can use features such as PyWrap
- is enables easy sharing of scripts with other users
Content
as of v1_04_00
matplotlib pandas uproot scikit-learn torch tensorflow jupyterlab notebook statsmodels awkward urllib3==1.26.16
other modules, such as keras are installed as dependencies.
Notes
jupyter-lab
The following error for jupyter-lab should be fixed in v1_05_00 onwards
JupyterLab Error JupyterLab application assets not found in "/cvmfs/mu2e.opensciencegrid.org/artexternals/python/v3_9_13/Linux64bit+3.10-2.17/share/jupyter/lab" Please run `jupyter lab build` or use a different app directory
can be fixed by adding the switch:
--app-dir=$PYANA_FQ_DIR/share/jupyter/lab
The following errors (and similar I/O errors) writing default files to your home directory, also for jupyter-lab:
[W 2023-03-23 17:09:31.510 ServerApp] 500 PUT /api/contents/tutorial2_dqm.ipynb?1679609372139 (127.0.0.1): Unexpected error while saving file: tutorial2_dqm.ipynb disk I/O error [W 2023-03-23 17:09:31.510 ServerApp] wrote error: 'Unexpected error while saving file: tutorial2_dqm.ipynb disk I/O error'
can be addressed by the following. The issue stems from the issue that sqlite has with the NFS file system. The workaround requires not only having /.ipython/profile_default/history.sqlite , but also ~/.local/share/jupyter/nbsignatures.db in the /tmp/ area:
- run ipython profile create if ~/.ipython/profile_default/ipython_config.py does not exist.
- Add c.HistoryManager.hist_file='/tmp/ipython_hist.sqlite' to the config file.
- run jupyter notebook --generate-config or jupyter-lab --generate-config if ~/.jupyter/jupyter_notebook_config.py or ~/.jupyter/jupyter_lab_config.py does not exist.
- Add c.NotebookNotary.data_dir = "/tmp" to each config file.
create kernel locally
jupyter kernelspec install $PYANA_FQ_DIR/share/jupyter/kernels/python3/ --user
notebook missing schemas
The command
jupyter-notebook --no-browser --port=3001
once opened in a browser gives the error
tornado.web.HTTPError: HTTP 404: Not Found (Schema not found: /cvmfs/mu2e.opensciencegrid.org/artexternals/python/v3_9_15/Linux64bit+3.10-2.17/share/jupyter/lab/schemas/@jupyterlab/docmanager-extension/plugin.json)
note that it is looking for schema in the python directory. This error should be fixed in v1_05_00 onwards
One way to fix this is to generate a config file:
jupyter-notebook --generate-config
it will tell you the file it generated in ~/.jupyter, which you can edit:
c.LabServerApp.schemas_dir = '/cvmfs/mu2e.opensciencegrid.org/artexternals/pyana/v1_04_00/Linux64bit+3.10-2.17//share/jupyter/lab/schemas' c.LabServerApp.themes_dir = '/cvmfs/mu2e.opensciencegrid.org/artexternals/pyana/v1_04_00/Linux64bit+3.10-2.17//share/jupyter/lab/themes'
unfortunately it doesn't seem to accept environmentals. There may be other ways to fix this, with JUPYTER paths variables, for example.
Anaconda
On October 16, 2024 access to Anaconda and MiniConda from Fermilab will be blocked. See the annoucement betow for details.
From: Associate Lab Director for Computational Science and AI Jim Amundson
Re: Anaconda and Miniconda access
Dear colleagues
This message is to let you know that Fermilab is blocking network access to the default channel of Anaconda and Miniconda. You will need to use Miniforge to replace the functionality provided by Anaconda or Miniconda.
When: Starting on October 16
Why: Anaconda, Inc. has decided that the terms of their licensing agreement do not allow for free use of the commercial Anaconda offering ("the default channel"). The community-supported distribution from Conda Forge, Miniforge, remains free to use.
Additional Information:
• Some notes on using Miniforge in place of Anaconda can be found in this article: https://fermi.servicenowservices.com/kb_view.do?sysparm_article=KB0014262
• While neither Anaconda nor Conda Forge are supported products at Fermilab, we recommend Miniforge for users who are looking for the functionality provided by Anaconda or Miniconda and are willing to provide their own support.
Please contact the Service Desk at ext. 2345 or servicedesk@fnal.gov if you have questions.
Regards,
Jim
Short Lived Blocking of Anaconda on the Fermilab site During 2022
Between February and September 2022, Fermilab blocked network access to the Anaconda repository at repo.anaconda.com. This was done because, for that period of time, Anaconda had notified Fermilab that the terms of their licensing agreement do not allow for free usage at the lab of the default Anaconda and Miniconda repositories. The community-supported distribution from Conda Forge, Miniforge, remains free.
During the blockage Fermilab recommended that you use Miniforge to replace the functionality provided by Anaconda or Miniconda. You continue to use Miniforge.
It is your responsibility as a Fermilab employee, user or affiliate to respect licensing agreements on Fermilab-owned hardware and networks. Some notes on using Miniforge in place of Anaconda can be found in this article.
This decision was reversed on Sept 13, 2022.