Mu2epro

Introduction

These are notes on using the group account mu2epro, which is used for collaboration-managed production activities. Access is controlled by the k5login. As quick start quide, you can get the correct environment and authentication with (the -k is important!):

ssh -k mu2epro@nodename

Here is SCD note 5644 on group accounts.

Kerberos

installing keytabs

In 8/2016 we asked the service desk to reset the keytab passwords and then ran the procedure /usr/bin/make-cron-keytab (servicedesk article servicedesk form). This moves the keytab to a secure system location under /var. This solves the security problem of using the keytab from a network-mounted disk, exposing the kerberos data to the network.

In 2021, we added principles to mu2esamgpvm01,2,3.

At times, such as when the machines we re-installed, we loose the keytabs.

The procedure is ask for the existing principles to be reset with new passwords ("Kerberos (FNAL.GOV) Standard" "Access to Kerberized Machines"), then run make-cron-keytab on each machine again. We have keytabs for each machine mu2egpvm01-07, mu2ebuild02, and mu2egpvmsam01-03.

In 2024, we added mu2eprodgpvm01 and 2, and get the keytab this time, we had to install fermilab-util_makehostkeys-1.3-1.el9.noarch for the /usr/sbin/makehostkeys exe, then ask the service desk for the principles and a password for each. Then

/usr/libexec/kcron/init-kcron-keytab
makehostkeys -s mu2epro/cron -n mu2eprodgpvm02.fnal.gov -k $(/usr/libexec/kcron/client-keytab-name)

using keytabs

To get a new ticket, you can run

 kinit -k -t /var/adm/krb5/mu2epro.cron.$(hostname).keytab mu2epro/cron/$(hostname)@FNAL.GOV
   for older nodes, or for mu2eprodgpmv*
 kinit -k -t /var/kerberos/krb5/user/44592/client.keytab mu2epro/cron/$(hostname)@FNAL.GOV

at any time. However, a cron job on all of the interactive machines runs this command every 4 hours and writes the ticket to /tmp/krb5cc_44592_auto. As long as you point to this ticket location, you will always have a valid ticket. Your process can use the ticket by setting

export KRB5CCNAME=FILE:/tmp/krb5cc_44592_auto

Ideally users should

ssh -k mu2epro@nodename

The -k switch stops your personal ticket from being forwarded to your interactive sessions as mu2epro. During the login, .bashrc will set KRB5CCNAME to FILE:/tmp/krb5cc_44592_auto for you.

If you forward your ticket (not recommended):

ssh mu2epro@nodename

you will end up in the mu2epro account, and

KRB5CCNAME=FILE:/tmp/krb5cc_44592_XXXXXXXXXX

which is a ticket with your personal principle. But in .bash_profile, KRB5CCNAME will be reset to FILE:/tmp/krb5cc_44592_auto since this is the only reasonable and convenient default for the mu2epro account. If you do forward your personal ticket, it is possible for other people using the mu2epro account to gain your kerberos identity. Also, nfs has a behavior which caches which kerberos identity is associated with a UID. In this case, other uses (such as cron jobs) will not be able to access any disks that you as your personal UID cannot access. Also if you are using your personal ticket and run bare jobsub_submit, the job will be run in your personal account, not mu2epro.

Anyone can get their personal principal back with

kinit your_user_name@FNAL.GOV

But note that if KRB5CCNAME is not set, the ticket will default to /tmp/krb5cc_44592, which may be found by utilities or other users. If it is set to FILE:/tmp/krb5cc_44592_auto, it will overwrite the mu2epro ticket, so please don't do that!

Bottom line, just ssh -k and only use the default mu2epro automatic tickets! Do not use your kerberos identity inside the mu2epro account.

Andrei has an ssh config setup that will effectively set "-k" for you automatically. Add this text block to $HOME/.ssh/config

#----------------
Host mu2epro
User mu2epro
HostName mu2egpvm01.fnal.gov
ForwardAgent no
GSSAPIDelegateCredentials no
#----------------

and then the command

ssh mu2epro

translates to

ssh -k mu2epro@mu2egpvm01.fnal.gov

You can see the kerberos ticket with

klist

Certificate

This account has a OSG Service Certificate consisting of certificate and key files. The service cert is for a "service" instead of a person. Certificate authorities always require a responsible person behind the cert and for this one that is Ray Culbertson.

The cert has been copied to a Computing Division server. They will renew the cert yearly, as needed.

The SCD generates a VOMS proxy every 4 hours and that proxy is copied in a secure way to

/opt/mu2epro/mu2epro.Production.proxy

on all the interactive machines. The SCD has notes on the system (bottom of the page). We believe the proxy is updated by an atomic command, so there is no chance of a user finding a partial proxy file. You can see the proxy with

voms-proxy-info -all

When you log into mu2epro, an environmental is set to point to this proxy.

Production Role

When the mu2epro account submits production jobs, it should use the --role=Production switch. This switch is then also necessary for jobsub_rm and jobsub_fetchlog. This puts the jobs in a different category than the user default role Analysis. For example, I believe the username on the grid node is different. We presumably can give more advanced privileges to this role, but we have not used this so far. Other users can also have the Production role and if the authenticate with their personal certificate and submit with this role, the job will run as if it were submitted by mu2epro and the Production role. In particular, the output will be owned by mu2epro and you can remove or examine any production jobs. We do not use this feature. It has the one advantage that the SCD can see what user submitted this job, if they need to contact you. It might useful if we have lots of operators submitting jobs, since even today we occasionally are left wondering who is submitting mu2epro jobs.

Setup scripts

The login file .bash_profile defines KRB5CCNAME to point to the mu2epro automatic kerberos ticket. It also makes sure /usr/krb5/bin and ~/bin are in the path, defines some environmentals which allow you to use the mu2epro authentication for commands, and aliases for setting further things up.

.bashrc defines aliases for personalization for specific operators and projects.

After logging in, the user can chose their environment. You can set your preferences through a personal setup script alias such as "setup_rlc". Please see

~mu2epro/bin/setup_alias.sh

for examples. These scripts should only set minor interactive preferences such as the prompt or "rm -i". It should set the environmental

export OPERATOR=rlc

to your username. It should not do anything required for real production projects. The original (TDR era) setup is saved as "setup_old".

In addition, you can run the setup for a project, such as "setup_upload". These should set the environmental MUE2PRO_PROJECT to a unique value. This script should then setup all environment needed for the project. This should be complete, but not include any enviroment that is not necessary to run the project. Necessary setups might include mu2einit, offline version setup, jobsub, ifdhc and useful environmentals, such as pointers to scratch disk directories.

If the environment for a project must be strictly controlled, then you would only setup the project, and this setup should overwrite the value of OPERATOR to stop personalized .bashrc behavior.

You should write project scripts which only depend on setup_project and with alias-defeating /bin/mv etc to avoid differences in preferences. A final useful convention would be to keep project scripts in ~/projectname.

There are also hooks in .bashrc to add behaviour as needed.

source ~/bin/setup_alias.sh
[ "$OPERATOR" == "SOMENAME"      ] && source ~/bin/setup_SOMENAME.sh rc

cron jobs should do all their setup explicitly, which should include sourcing these setup scripts.

GitHub and mu2epro

Some GitHub repos, for example OfflineOps, are normally cloned into the mu2epro disk space and used from that disk space; they need to be used from the mu2epro account in order to have the required authorizations. In that sense they are "live".

Issues arise because there is no mu2epro GitHub account and different people will use the mu2epro account to edit the clone. One issue is how do we authenticate to GitHub in order to push and another is how do we identify who made which commit. This section outlines a procedure to resolve these issues. The procedure is applicable to other live repos, such as some of those in the TDAQ environment.

The short version is that everyone should use their own ssh-agent credentials: see Authentication#SSH_Keys. And everyone should use one of the forms of the --author option to the git commit command:

--author=<name> 
--author="Full Name <name@subdomain.domain>"

Roughly speaking, the first form will work if you have already made at least one commit to the repo; the second form is required if you have not. Consult the git documentation to understand exactly how the first form works; in short, it pattern matches <name> to the author field of previous commits. The two Dave Browns will need to figure our how to use the short version correctly.

An alternative to providing --author on the command line is to define the environment variables GIT_AUTHOR_NAME, for the human-readable name, and GIT_AUTHOR_EMAIL for the author email address. Since mu2epro is a shared account we must not set these in the login scripts.

We are investigating using a pre-commit hook to enforce the requirement of supplying --author or providing the environment variables.

There are two ways that we might choose to organize this work. For some live repos we might chose to follow the feature branch and pull request model that we use for Offline. For other live repos we may, instead, choose to work on the one "official" branch and to push directly to the Mu2e repo from the clone; perhaps the official branch will be the main branch but it could be a different one. In the both options, users will need to communicate with each other to avoid two people making changes to the live system at the same time - but that is an inherent feature of a live repo, not something peculiar to this discussion.

Option 1: Using Pull Request

1. Before you log into mu2epro, add the ssh-key for your GitHub identity to your ssh-agent. See Authentication#SSH_Keys.
2. When you log into mu2epro you must forward your ssh-agent. You can do this with the -A option on the ssh command line or by having "ForwardAgent yes" in your ~/.ssh/config file for the mu2e interactive machines.
3. Add your own GitHub fork as a remote to the local clone; you only need to do this once
4. Modify files as needed; commit them to the local clone using --author and push the working branch to your own GitHub fork.
5. Make a pull request from your GitHub fork.

If many people have their forks added as remotes, the ssh-agent mechanism ensures that each can only push to their own remote.

Option 2: Pushing Directly to Mu2e

1. Before you log into mu2epro, add the ssh-key for your GitHub identity to your ssh-agent. See Authentication#SSH_Keys.
2. When you log into mu2epro you must forward your ssh-agent. You can do this with the -A option on the ssh command line or by having "ForwardAgent yes" in your ~/.ssh/config file for the mu2e interactive machines.
3. Check with the Mu2e GitHub admins to ensure that your GitHub identity has write (or better) privileges on the repo in the Mu2e organization.
4. Work on the "standard" working branch; commit to the local clone using --author and push the working branch directly to the repo in the Mu2e organization.

Cron jobs

cron job information is kept in ~mu2epro/cron. We will try to run all cron jobs on mu2egpvm01 unless they have to run elsewhere.

Backup

The home area is on /mu2e/app and is not backed up automatically like the user home areas. It is backed up by a cron job to /pnfs/mu2e/scratch/users/mu2epro/backup. Older backups are purged.

Mailing Lists

There exist (11/2022) two mailing lists:

mu2e-production@listserv.fnal.gov

for registering Docker, Github, and communication with the computing division

mu2epro-sim@listserv.fnal.gov

for receiving mailed grid job summaries from production jobs