Mu2epro

Introduction

These are notes on using the group account mu2epro, which is used for collaboration-managed production activities. Access is controlled by the k5login. As quick start quide, you can get the correct environment and authentication with (the -k is important!):

ssh -k mu2epro@nodename

Here is SCD note 5644 on group accounts.

Kerberos

In 4/2015 we (rlc) installed kerberos keytabs in ~mu2epro/keytab. The principals are created by the servicedesk, and a temp password returned. The actual keytab file is made according to this article.

In 8/2016 we asked the service desk to reset the keytab passwords and then ran the procedure /usr/krb5/config/make-cron-keytab This moves the keytab to a secure system location under /var. This solves the security problem of using the keytab from a network-mounted disk, exposing the kerberos data to the network.

To get a new ticket, you run

/usr/krb5/bin/kcron

at any time, but please see the notes below on where the ticket goes.

A cron job on all of the interactive machines runs every 4 hours, runs a kcron command and writes the ticket to /tmp/krb5cc_44592_auto. As long as you use this ticket location, you will always have a valid ticket.

It is important to setup how the ticket is located. If you do not have the KRB5CCNAME environmental set when you run kcron, the ticket will go to /tmp/krb5cc_44592cronXXXXX and no other utility will find it, not even klist, so you will always want to have KRB5CCNAME set. What follows is how it gets set.

Ideally users should

ssh -k mu2epro@nodename formatted with code and indentx2

The -k switch stops your personal ticket from being forwarded to your interactive sessions as mu2epro. During the login, .bash_profile will set KRB5CCNAME to FILE:/tmp/krb5cc_44592_auto for you. If you avoid using a "login shell" you will not run .bash_profile and KRB5CCNAME will not be set (in some philosophical sense this is what you asked for) We recommend to just use login shells.

If you forward your ticket (not recommended):

ssh mu2epro@nodename

you will end up in the mu2epro account, and

KRB5CCNAME=FILE:/tmp/krb5cc_44592_XXXXXXXXXX

which is a ticket with your personal principle. But in .bash_profile, KRB5CCNAME will be reset to FILE:/tmp/krb5cc_44592_auto since this is the only reasonable and convenient default for the mu2epro account. If you do forward your personal ticket, it is possible for other people using the mu2epro account to gain your kerberos identity. Also if you are using your personal ticket and run bare jobsub_submit, the job will be run in your personal account, not mu2epro.

Anyone can get their personal principal back with

kinit your_user_name@FNAL.GOV

But note that if KRB5CCNAME is not set, the ticket will default to /tmp/krb5cc_44592, which may be found by utilities or other users. If it is set to FILE:/tmp/krb5cc_44592_auto, it will overwrite the mu2epro ticket, so please don't do that!

Bottom line, just ssh -k and only use the default mu2epro automatic tickets! Do not use your kerberos identity inside the mu2epro account.

Andrei has an ssh config setup that will effectively set "-k" for you automatically. Add this text block to $HOME/.ssh/config

#----------------
Host mu2epro
User mu2epro
HostName mu2egpvm01.fnal.gov
ForwardAgent no
GSSAPIDelegateCredentials no
#----------------

and then the command

ssh mu2epro

translates to

ssh -k mu2epro@mu2egpvm01.fnal.gov

You can see the kerberos ticket with

klist

Certificate

This account has a OSG Service Certificate consisting of certificate and key files. The service cert is for a "service" instead of a person. Certificate authorities always require a responsible person behind the cert and for this one that is Ray Culbertson.

The cert has been copied to a Computing Division server. They will renew the cert yearly, as needed. The current cert will expire about 3/1/2017.

The SCD generates a VOMS proxy every 8 hours and that proxy is copied in a secure way to

/opt/mu2epro/mu2epro.Production.proxy

on all the interactive machines. The SCD has notes on the system (bottom of the page). I don't know if the proxy is updated by an atomic command, (so there is no chance of a user finding a partial proxy file). You can see the proxy with

voms-proxy-info -all

Andrei has installed the actual certificate (cert and key files) in the /opt/mu2epro area too. We believe the /opt area is a local disk so this will not expose the cert over the network. This will allow scripts to authenticate https communication with the SAM server. There may be other methods to authenticate, but the concern is that they will time-out.

Older certs are kept in ~mu2epro/osg* directories. This directory should have only owner access permissions. Old notes on obtaining the cert are also in that directory.

Production Role

When the mu2epro account submits production jobs, it should use the --role=Production switch. This switch is then also necessary for jobsub_rm and jobsub_fetchlog. This puts the jobs in a different category than the user default role Analysis. For example, I believe the username on the grid node is different. We presumably can give more advanced privileges to this role, but we have not used this so far. Other users can also have the Production role and if the authenticate with their personal certificate and submit with this role, the job will run as if it were submitted by mu2epro and the Production role. In particular, the output will be owned by mu2epro and you can remove or examine any production jobs. We do not use this feature. It has the one advantage that the SCD can see what user submitted this job, if they need to contact you. It might useful if we have lots of operators submitting jobs, since even today we occasionally are left wondering who is submitting mu2epro jobs.

Setup scripts

The login file .bash_profile defines KRB5CCNAME to point to the mu2epro automatic kerberos ticket. It also makes sure /usr/krb5/bin and ~/bin are in the path, defines some environmentals which allow you to use the mu2epro authentication for commands, and aliases for setting further things up.

.bashrc defines aliases for personalization for operators and for projects (see next). It also runs

/cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups

to point PRODUCTS to the common products, allowing "setup mu2e" as well as fermilab common products, such as jobsub_client.

After logging in, the user can chose their environment. You can set your preferences through a personal setup script alias such as "setup_rlc". Please see

~mu2epro/bin/setup_alias.sh

for examples. These scripts should only set minor interactive preferences such as the prompt or "rm -i". It should set the environmental

export OPERATOR=rlc

to your username. It should not do anything required for real production projects. The original (TDR era) setup is saved as "setup_old".

In addition, you can run the setup for a project, such as "setup_upload". These should set the environmental MUE2PRO_PROJECT to a unique value. This script should then setup all environment needed for the project. This should be complete, but not include any enviroment that is not necessary to run the project. Necessary setups might include "setup mu2e", offline version setup, jobsub, ifdhc and useful environmentals, such as pointers to scratch disk directories.

If the environment for a project must be strictly controlled, then you would only setup the project, and this setup should overwrite the value of OPERATOR to stop personalized .bashrc behaviour.

You should write project scripts which only depend on setup_project and with alias-defeating /bin/mv etc to avoid differences in preferences. A final useful convention would be to keep project scripts in ~/projectname.

There are also hooks in .bashrc to add behaviour as needed.

source ~/bin/setup_alias.sh
[ "$OPERATOR" == "rlc"      ] && source ~rlc/bin/setup_rlc.sh rc
...
[ "$MUE2PRO_PROJECT" == "upload" ] && source ~/bin/setup_upload.sh rc

cron jobs should do all their setup explicitly, which should include sourcing these setup scripts.

Jobsub and ifdh

ifdhc with version less than 1_7 may not work to transfer files, it may generate an incorrect gridftp url resulting in globus errors. If you write to bluearc, you will need to force the transfer through gridftp to get the permissions to work.

ifdh cp --force=gridftp file /mu2e/data/..

You will also need this switch for ifdh non-transfer commands

ifdh rm /mu2e/data/.. --force=gridftp

Writes to dCache (/pnfs/...) using x509 authentication (like gridftp or expftp) currently end up owned by mu2eana or mu2epro. gridftp-ls does not always list permissions correctly, uberftp fndca1 "ldir" does.

jobsub needs X509_USER_PROXY to point to the OSG cert to work. This is done in the authentication script run during login.

As of jobsub_client v1_1_0_2, the --role switch is needed for submit, rm, and fetchlog. It is not needed for q and history, which list the jobs as user mu2epro.

Note that when a user becomes mu2epro, they may still have their personal kerberos ticket active, and that may be used accidentally to authenticate the submission, so the user should kdestroy their ticket. Please see the discussion of kerberos above. The job submission should be authenticated by the proxy.

"ifdh cp --force=expftp" will not work with the service cert, i.e. interactively. It works on the grid because a different grid-based cert is used there.

Cron jobs

cron job information is kept in ~mu2epro/cron. We will try to run all cron jobs on mu2egpvm01 unless they have to run elsewhere.

Backup

The home area is on /mu2e/app and is not backed up automatically like the user home areas. It is backed up by a cron job to /pnfs/mu2e/scratch/users/mu2epro/backup. Older backups are purged.