User:Rlc

Introduction

These are notes on using the group account mu2epro, which is used for collaboration-managed production activities. Access is controlled by the k5login. As quick start quide, you can get the correct environment and authentication with:

ssh -k mu2epro@nodename (-k is important!)     formatted with pre

Kerberos

In 4/2015 we (rlc) installed kerberos keytabs in ~mu2epro/keytab (code inline). The principals are created by the servicedesk, and a temp password returned. The actual keytab (tt inline) file is made according to this article.

In 8/2016 we asked the service desk to reset the keytab passwords and then ran the procedure /usr/krb5/config/make-cron-keytab This moves the keytab to a secure system location under /var. This solves the security problem of using the keytab from a network-mounted disk, exposing the kerberos data to the network.

To get a new ticket, you run

/usr/krb5/bin/kcron      formatted with pre and indentx2

at any time, but please see the notes below on where the ticket goes.

A cron job on all of the interactive machines runs every 4 hours, runs a kcron command and writes the ticket to /tmp/krb5cc_44592_auto. As long as you use this ticket location, you will always have a valid ticket.

Using the keytab with

kinit -k -t $HOME/keytab/mu2egpvm01.keytab mu2epro/cron/mu2egpvm01.fnal.gov@FNAL.GOV (formatted with code)

is no longer available.

It is important to setup how the ticket is located. If you do not have the KRB5CCNAME environmental set when you run kcron, the ticket will go to /tmp/krb5cc_44592cronXXXXX and no other utility will find it, not even klist, so you will always want to have KRB5CCNAME set. What follows is how it gets set.

Ideally users should

ssh -k mu2epro@nodename formatted with code and indentx2

The -k switch stops your personal ticket from being forwarded to your interactive sessions as mu2epro. During the login, .bash_profile will set KRB5CCNAME to FILE:/tmp/krb5cc_44592_auto for you. If you avoid using a "login shell" you will not run .bash_profile and KRB5CCNAME will not be set. In some philosophical sense this is what you asked for, so just use login shells.

If you forward your ticket (not recommended):

ssh mu2epro@nodename (formatted with code and newlines)

you will end up in the mu2epro account, and

KRB5CCNAME=FILE:/tmp/krb5cc_44592_XXXXXXXXXX

which is a ticket with your personal principle. But in .bash_profile, KRB5CCNAME will be reset to FILE:/tmp/krb5cc_44592_auto since this is the only reasonable and convenient default for the mu2epro account. If you do forward your personal ticket, it is possible for other people in the mu2epro account to gain your credentials. Also if you are using your personal ticket and run bare jobsub_submit, the job will be run in your personal account, not mu2epro.

Anyone can get their personal principal back with

kinit your_user_name@FNAL.GOV

But note that if KRB5CCNAME is not set, the ticket will default to /tmp/krb5cc_44592, which may be found by utilities or other users. If it is set to FILE:/tmp/krb5cc_44592_auto, it will overwrite the mu2epro ticket, so please don't do that!

Bottom line, just ssh -k and only use the default mu2epro automatic tickets!

Andrei has an ssh config setup that will effectively set "-k" for you automatically. Add this text block to $HOME/.ssh/config

#----------------
Host mu2epro
User mu2epro
HostName mu2egpvm01.fnal.gov
ForwardAgent no
GSSAPIDelegateCredentials no
#----------------

and then the command

ssh mu2epro

translates to "ssh -k mu2epro@mu2egpvm01.fnal.gov".

Certificate

This account has a OSG Service Certificate consisting of certificate and key files kept in ~mu2epro/osg_service_cert_2015. This directory should have only owner access permissions. Notes on obtaining the cert are also in that directory. The cert is valid until 2/24/17.

The cert has been copied to a Computing Division server. There, it is converted to a VOMS proxy every 8 hours and that proxy is copied in a secure way to

/opt/mu2epro/mu2epro.Production.proxy

on all the interactive machines. I don't know if the proxy is updated by an atomic command, (so there is no chance of a user finding a partial proxy file).

Andrei has installed the actual certificate in the /opt area too. We believe the /opt area is a local disk so this will not expose the cert over the network. This will allow scripts to authenticate https communication with the SAM server. There may be other methods to authenticate, but the concern is that they will time-out.

Setup scripts

The login file .bash_profile defines KRB5CCNAME to point to the mu2epro automatic kerberos ticket. It also makes sure /usr/krb5/bin and ~/bin are in the path, defines some environmentals which allow you to use the mu2epro authentication for commands, and aliases for setting further things up.

.bashrc defines aliases for personalization for operators and for projects (see next). It also runs

/cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups

to point PRODUCTS to the common products, allowing "setup mu2e" as well as fermilab common products, such as jobsub_client.

After logging in, the user can chose their environment. You can set your preferences through a personal setup script alias such as "setup_rlc". Please see

~mu2epro/bin/setup_alias.sh

for examples. These scripts should only set minor interactive preferences such as the prompt or "rm -i". It should set the environmental

export OPERATOR=rlc

to your username. It should not do anything required for real production projects. The original (TDR era) setup is saved as "setup_old".

In addition, you can run the setup for a project, such as "setup_upload". These should set the environmental MUE2PRO_PROJECT to a unique value. This script should then setup all environment needed for the project. This should be complete, but not include any enviroment that is not necessary to run the project. Necessary setups might include "setup mu2e", offline version setup, jobsub, ifdhc and useful environmentals, such as pointers to scratch disk directories.

If the environment for a project must be strictly controlled, then you would only setup the project, and this setup should overwrite the value of OPERATOR to stop personalized .bashrc behaviour.

You should write project scripts which only depend on setup_project and with alias-defeating /bin/mv etc to avoid differences in preferences. A final useful convention would be to keep project scripts in ~/projectname.

There are also hooks in .bashrc to add behaviour as needed. source ~/bin/setup_alias.sh [ "$OPERATOR" == "rlc" ] && source ~rlc/bin/setup_rlc.sh rc ... [ "$MUE2PRO_PROJECT" == "upload" ] && source ~/bin/setup_upload.sh rc cron jobs should do all their setup explicitly, which should include sourcing these setup scripts.

Jobsub and ifdh

ifdhc with version less than 1_7 may not work to transfer files, it may generate an incorrect gridftp url resulting in globus errors. If you write to bluearc, you will need to force the transfer through gridftp to get the permissions to work.

ifdh cp --force=gridftp file /mu2e/data/..

You will also need this switch for ifdh non-transfer commands ifdh rm /mu2e/data/.. --force=gridftp Writes to dCache (/pnfs/...) using x509 authentication (like gridftp or expftp) currently end up owned by mu2eana or mu2epro. gridftp-ls does not always list permissions correctly, uberftp fndca1 "ldir" does.

jobsub needs to be pointed to the OSG cert to work. export X509_USER_PROXY=/tmp/x509up_u44592 this should be done though the command

setup_mu2epro

As of jobsub_client v1_1_0_2, the --role switch is needed for submit, rm, and fetchlog. It is not needed for q and history, which list the jobs as user mu2epro.

Note that when a user becomes mu2epro, they may still have their personal kerberos ticket active, and that may be used accidentally to authenticate the submission, so the user should kdestroy their ticket. Please see the discussion of kerberos and the setup commands below. If there is no user kerberos ticket, the job will be authenticated by the proxy.

jobsub_submit -G mu2e --role=Production -N 1 \
  --generate-email-summary --email-to=rlc@fnal.gov  --OS=SL6 \
  --resource-provides=usage_model=DEDICATED  \
  file:///mu2e/app/users/rlc/jobs/remote.bash

"ifdh cp --force=expftp" will not work with the service cert, i.e. interactively. It works on the grid because a different grid-based cert is used there. Cron jobs cron job information is kept in ~mu2epro/cron. We will try to run all cron jobs on mu2egpvm01 unless they have to run elsewhere.

Backup

The home area is on /mu2e/app and is not backed up automatically like the user home areas. It is backed up by a cron job to /pnfs/mu2e/scratch/users/mu2epro/backup.