Authentication
Introduction
The lab services and utilities used by mu2e require gaining and using several kinds of authentication. You will have one username for all computing purposes lab.
You login to the virtual machines with kerberos authentication. You will need a permanent ID called a kerberos "principal" which is looks like "xyz@FNAL.GOV", where xyz is your username. You will have a password associated with your principal. You will use this principal and password to log into the various personal linux desktops located at Fermilab or to ssh into the collaboration interactive machines from your home institution.
The second identity you will need is the services principal, which looks like xyz@services.fnal.gov, or often just xyz, and also has a password (different from your kerberos password). You will need this identity to log into Fermilab email, the servicedesk web site and some other services based at the lab. You would typically only use this authentication at the point you log into the service.
The third identity you will need is a CILogon certificate. This "cert" is the basis of authentication to the mu2e documents database, the computing farms, and a few other services. You will use this cert in two ways. The first way is to load it into your browser, which then gives you access to web pages and web services. The second is by using your kerberos authentication to access a copy of your certificate maintained in a remote database. You get this certificate once and then renew it only once a year.
hypernews is an archived blog and email list - for access here, you will need a hypernews password and your services password!
Finally, the mu2e internal web pages require a collaboration username and password, please ask your mu2e mentor.
There is a standard lab seminar on authentication for users.
If you need to do any computing in mu2e, please go ahead and start the procedure on the ComputingAccounts to create your accounts and authentication.
Kerberos
You login to the virtual machines with kerberos authentication. You will need a permanent ID called a kerberos "principal" which is looks like "xyz@FNAL.GOV", where xyz is your username. You will have a password associated with your principal. You will use this principal and password to log into the various personal linux desktops located at Fermilab or to ssh into the collaboration interactive machines from your home institution.
Your kerberos authentication is stored in a file
/tmp/krb5cc_`id -u`_*
where "*" will be a random string. Each time you ssh into a machine, it will produce a new file in /tmp for that process. The environmental KRB5CCNAME will point to the ticket file.
The ticket may be viewed with
klist
If you log into a lab desktop, you would typically user you username (just "xyz" without the "@FNAL.GOV") a new ticket is created. You can renew the ticket, or create it if you logged on my some other means, using
kinit
kinit takes an argument which is the user name (or full principle) and asked for you password. Tickets are "forwardable" by default. This means if you are logged into machine A with a ticket, and ssh to machine B, the ticket will also be moved to machine B, so you can ssh (or scp, etc) once you are on B.
Tickets are only valid for 26h and you typically refresh your kerberos authentication every day. This would normally happen as you log in to a desktop. If you have an ssh session from machine A to machine B, and leave the session up, you may renew the ticket on A, but the ticket on B will not automatically be updated. in this case, you can use
k5push <node>
to push your fresh ticket out though all ssh sessions and update the remote tickets.
There are two utilities in kerberos which we will only note here. One is kcron and kcroninit. These allows you to gain a kerberos ticket in a cron job. The other variant is the keytab file. This file can hold a ticket that is good for a year and can be accessed by anyone with access to the keytab file. This feature is not as secure, so it is usually only issued by the lab when needed, for example, in a group account.
Read more at the FNAL kerberos link (Miscellaneous Kerberos Topics for the User -> Automated Processes). Reset password here.
Services
The second identity you will need is the services principal, which looks like xyz@services.fnal.gov, or often just xyz, and also has a password (different from your kerberos password). You will need this identity to log into Fermilab email, the servicedesk web site, sharepoint and some other services based at the lab. You would typically only use this authentication at the point you log into the service and their is no local credential cache.
Certificate
For some interactive purposes on linux, via the command line or browsers, you will need a certificate to prove your identity. Our certificates are based on the CILogon Certification Authority (CA) which is geared to big science. Some of the systems which require cert authentication are jobsub job submission, ifdh data transfer and writing to the SAM database as part of file upload to tape.
You should have received a certificate as part of registering for computing accounts. (See also Fermilab docs on certs for creating or renewing a cert.) The cert can be downloaded as a password-protected pem or p12 file and imported in your browser. Your cert is good for a year and only needs to be updated in your browser once a year.
When your cert is first created, it is also communicated to the lab which can then manage and provide the cert for you. When you access your cert at the linux command line, you usually access it from this cache.
kinit kx509
kinit makes sure your kerberos identity is valid, and kx509 uses that authentication to make a local, temporary copy of your cert, called a proxy, and writes it to a file named with your UID:
kx509 ls -l /tmp/x509up_u`id -u` -rw------- 1 rlc mu2e 8171 Aug 15 10:07 /tmp/x509up_u1311
It is this proxy, in this standard location, that commands can use to authenticate you. Note that jobsub and ifdh can automatically run kx509 for you if it is needed (so you only need to remember to kinit), however, samweb does not run kx509 automatically, and you will need to run it yourself if you get authentication errors.
kx509 is equivalent to
setup cigetcert cigetcert --institution="Fermi National Accelerator Laboratory"
which you might see some places. If you do not have a kerberos ticket when you run cigetcert, it will prompt you for your services password, and you can use this authentication to access your cert and make a proxy.
You can print your proxy certificate with
voms-proxy-info -all
Your cert works by providing encrypted identity information. This packet will be "signed" by a Certificate Authority (CA). The party, such as a lab service, that wants to check your identity can ask the CA if your packet is valid. CA's may be signed by other CA's in turn up to a nationally-recognized organization, in a "trust chain".
Certificate Error
If you believe you have a valid certificate and you still see errors, for example,
Error creating dataset definition for ... 500 SSL negotiation failed: .
Then try removing the cert and recreating it.
kinit rm /tmp/x509up_u`id -u` kx509
The problem is that the proxy created by jobsub or ifdh may require extra steps in authentication because it is "issued" by you. The cert created by kx509 is issued by the CILogon CA, and requires fewer steps to authenticate. While this delete-and-kx509 procedure works fine, the best solution is to provide the intermediate CA in the trust chain to the command (more below).
Types of certs and proxies
A proxy is a copy of your certificate that expires quickly, usually in a few days or hours (see the printouts for "timeleft"). If a bad actor were to gain access to the proxy, they could only use it for the valid time, and they could not replicate it. The proxy is considered safe enough to pass around the grid and over networks. At the command line, we only use proxies.
kx509 creates "end entity certificates", and are temporary copies of your cert. (I've had experts tell me these are technically proxies, but this seems to not be helpful.) Here is what one print looks like:
subject : /DC=org/DC=cilogon/C=US/O=Fermi National Accelerator Laboratory/OU=People/CN=Raymond Culbertson/CN=UID:rlc issuer : /DC=org/DC=cilogon/C=US/O=CILogon/CN=CILogon Basic CA 1 identity : /DC=org/DC=cilogon/C=US/O=CILogon/CN=CILogon Basic CA 1 type : unknown strength : 2048 bits path : /tmp/x509up_u1311 timeleft : 167:59:40 key usage : Digital Signature, Key Encipherment, Data Encipherment
The term "end-entity credential" just means it is below a CA in the certificate trust chain.
If you print your cert and see "subject' with an appendage like "/CN=2707985426" then this is a proxy. It may also say "proxy" in the "type" field. The kx509 certs are not fully RFC-compliant (the current industry standard) for backwards compatibility, so have type "unknown".
You can generate a proxy from your certificate, the easiest way I know is:
kx509 --proxyhours=1
which gives
subject : /DC=org/DC=cilogon/C=US/O=Fermi National Accelerator Laboratory/OU=People/CN=Raymond Culbertson/CN=UID:rlc/CN=528207979 issuer : /DC=org/DC=cilogon/C=US/O=Fermi National Accelerator Laboratory/OU=People/CN=Raymond Culbertson/CN=UID:rlc identity : /DC=org/DC=cilogon/C=US/O=Fermi National Accelerator Laboratory/OU=People/CN=Raymond Culbertson/CN=UID:rlc type : RFC compliant proxy strength : 2048 bits path : /tmp/x509up_u1311 timeleft : 0:59:55
This is a proxy derived from your end entity cert, so you are the issuer. It has type proxy and has the "/CN=..." and limited time features of a proxy.
Under some circumstances, you might end up with a voms proxy. voms is a system to track identities for uses on the grid. When you run a job on the grid, this will be the form of the cert you have on the grid node as you run. These proxies have been "extended" with additional fields that can be considered part of your identity, such as your VO (Virtual Organization, i.e. your experiment) and your role ("Analysis" or "Production").
In our current procedures, you should never need to do this directly, but if you want, you can create a voms proxy:
voms-proxy-init -noregen -rfc -voms fermilab:/fermilab/mu2e/Role=Analysis
This will create a proxy with an extension:
subject : /DC=org/DC=cilogon/C=US/O=Fermi National Accelerator Laboratory/OU=People/CN=Raymond Culbertson/CN=UID:rlc/CN=333302448 issuer : /DC=org/DC=cilogon/C=US/O=Fermi National Accelerator Laboratory/OU=People/CN=Raymond Culbertson/CN=UID:rlc identity : /DC=org/DC=cilogon/C=US/O=Fermi National Accelerator Laboratory/OU=People/CN=Raymond Culbertson/CN=UID:rlc type : RFC compliant proxy strength : 1024 bits path : /tmp/x509up_u1311 timeleft : 11:59:57 key usage : Digital Signature, Key Encipherment, Data Encipherment === VO fermilab extension information === VO : fermilab subject : /DC=org/DC=cilogon/C=US/O=Fermi National Accelerator Laboratory/OU=People/CN=Raymond Culbertson/CN=UID:rlc issuer : /DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=voms2.fnal.gov attribute : /fermilab/mu2e/Role=Analysis/Capability=NULL attribute : /fermilab/mu2e/Role=NULL/Capability=NULL attribute : /fermilab/Role=NULL/Capability=NULL timeleft : 11:59:57 uri : voms2.fnal.gov:15001
The "fermilab" VO is used for all smaller experiments at the lab. In the fermilab VO, there is a "mu2e group". Again, this has all the features of a proxy.
A different location of the proxy can be indicated to most services with some combination of
X509_USER_PROXY=/tmp/x509up_u`id -u` X509_USER_CERT=/tmp/x509up_u`id -u` X509_USER_KEY=/tmp/x509up_u`id -u` X509_CERT_DIR=/etc/grid-security/certificates
With a real cert, you might need to point CERT and KEY to different files, but with a proxy you can point them both to the proxy. The CERT_DIR variable points the command to the intermediate CA certs.
There is an equivalent set for the http protocol:
HTTPS_CA_FILE HTTPS_CERT_FILE HTTPS_KEY_FILE HTTPS_CA_DIR
The RFC for proxy certificates is here: link and for X.509 certificates here: link.
Browsers
When you first get your certificate, you want to load it in your browsers. This is usually straightforward following the browser instructions. When you visit secure web sites, such as Jenkins, you may have to select the certificate to present to the web site.
When you connect to a secure web site that expects a certificate from you, that site will also present your browser with a certificate of its own. Your browser will then attempt to authenticate the certificate. If it cannot, it will open a dialog box telling you that it does not recognize the site's certificate and asking you if you would like to "add an exception". If you add the exception, then your browser will accept this site even though the browser cannot itself authenticate the certificate. This is usually OK, but not ideal.
The way that your browser authenticates a certificate is that it contacts a recognized, trusted, Certificate Authority (CA). It then forwards the certificate in question to the CA and asks "Can I trust this?". If all is well, the CA replies that you can trust it. If your browser does not know the relevant CA to use, or if it does not trust the CA that the certificate says to use, then your browser will start the "add exception" dialog.
Out of the box, your browser usually has a set of CA's from commercial services, but not the CILogon CA, which we are using currently. You can see what CA's are needed by entering the url in at digicert or running
openssl s_client -connect URL:PORT
If it is uses the CILogon certs, you can usually find them here:
/etc/grid-security/certificates/cilogon-basic.pem /etc/grid-security/certificates/cilogon-osg.pem
Or you can download the same files from CILogon CA certs.
After following the browser instructions for importing the CA cert to your browser, you usually need to check a box to "trust" these certs.
KCA
KCA refers to an old lab-based Certificate Authority system which is now disabled (2016).
Grid Workflows
The following operations (from Workflows) require certs:
- samweb write operations (such as mu2eFileDeclare) require a kx509 cert
- ifdh copies to dCache (including mu2eFileUpload --ifdh) require a voms proxy
- jobsub grid submission requires a voms proxy
- reading files via xrootd (filespec like xroot://..) requires a voms proxy
When you land on a grid node, you will have a voms proxy, so all procedures should "just work". But when you perform procedure interactively, you need to manage your certificates.
Two procedures, jobsub and ifdh, will try to make a cert for you. We believe the pattern is:
- check for a voms proxy pointed to by X509 environmentals
- check for a voms proxy in /tmp/x509up_u$UID
- check for a voms proxy in /tmp/x509up_voms_mu2e_Analysis_$UID
- if a voms proxy is found, stop and use it (issue a warning if proxy has less than an hour left)
- if no voms proxy found look for x509 cert: in X509 environmentals, or in /tmp/x509up_u$UID
- if a x509 is found, create a 12 h voms proxy in /tmp/x509up_voms_mu2e_Analysis_$UID, use that
- if no x509 proxy is found, look for one-session (daily, default) kerberos ticket in /tmp/krb5cc_${UID}_*, as pointed to by KRB5CCNAME environmental.
- if a kerberos ticket is found, create x509 cert in /tmp/x509up_u$UID, and voms proxy in /tmp/x509up_voms_mu2e_Analysis_$UID, use voms proxy
- if the voms proxy expires repeat the above
While this code is trying to be helpful, it can fall short. For example, if it finds a x509 cert with 2h left on it, the voms proxy will also be limited to 2h. Kerberos session tickets will be deleted when the interactive process that created it exits, which will prevent ifdh and jobsub from using it to renew the voms proxy. (Certs are not deleted when you log off.) Cert-based authentication is unique to a node and user, but not a session which means you might start a script while you have good authentication, then overwrite those certs by an action in a different session on the same machine. These effects can lead to the procedures failing at unexpected times. The only complete solution is to understand, plan, and monitor your authentication.
To setup voms authentication for a long time, so you can run a long mu2eFileUpload, for example, you probably want to refresh your certs first:
TMPP=$(mktemp) TMPV=$(mktemp) FINV=/tmp/x509up_u$UID /usr/krb5/bin/klist # this is the same as kx509, but with more options /usr/bin/cigetcert -i "Fermi National Accelerator Laboratory" -o $TMPP voms-proxy-info -file $TMPP voms-proxy-init -hours 120 -noregen -rfc \ -voms fermilab:/fermilab/mu2e/Role=Analysis \ -cert $TMPP -out $TMPV mv $TMPV $FINV rm $TMPP voms-proxy-info -all
A command has been provided on cvmfs:
/cvmfs/mu2e.opensciencegrid.org/bin/vomsCert
At this point, you should see an x509 cert and voms proxy with 120 h lifetime in /tmp/x509up_u$UID and all procedures should work. You must simply be aware and careful to not overwrite these certs on this machine. The "mv" in the above script makes swapping the new cert in an atomic action, so the cert is never invalid and other scripts can use the cert uninterrupted.
Theoretically, this procedure could be put in your login scripts, but we wouldn't recommend that since you won't be so aware of what's happening.
We have also established that you can create a kcron kerberos ticket (see above) and run the authentication procedure off of the kcron ticket in a cron job, thereby establishing perpetual authentication. To do this, you need to create the kcron keytab file (see also above). To do that, just run:
kcroninit
you'll be prompted for you kerberos password and the keytab will be put on /var/adm. At this point you can see your kcron principle:
kdestroy kcron klist
You can run the authentication with a crontab entry:
0 */8 * * * kcron /cvmfs/mu2e.opensciencegrid.org/bin/vomsCert >& ~/vomsCert.log
but first you must get your kcron kerberos principle associated to your certificate id as explained on the fife pages, which points to this servicedesk item.
You will need to do this procedure once on each machine you want to work on. (kcroninit is permanent until you kcrondestroy..)
Tokens
Mu2epro
The SCD FIFE group provides a service to push a valid token to the mu2epro account every 10 min (probably will be set to 1-2 hours in the future). The write access is granted by adding
mu2epro/managedtokens/fifeutilgpvm01.fnal.gov@FNAL.GOV
to the k5login. and setting
export HTGETTOKENOPTS="--credkey=mu2epro/managedtokens/fifeutilgpvm01.fnal.gov"
in the bashrc causes token commands to find it.
SSH Keys
What are SSH keys
SSH has many options for performing authentication, one of which is RSA public key cryptography. If you want you, can read more about that at, for example, https://en.wikipedia.org/wiki/RSA_(cryptosystem) . SSH keys are a pair of files one of which contains the private key and the other of which contains the public key. You share the public key with people/applications with whom you wish to communicate securely. It is important to keep your private key secure.
One of the ways to authenticate to github is with SSH keys and github has some good information about SSH keys:
- Generating and testing keys: https://help.github.com/en/articles/connecting-to-github-with-ssh
- Trouble shooting problems: https://help.github.com/en/categories/authenticating-to-github
The following contains a shortened version of that information plus some additional details.
Generating a public and private key pair
Do this step on your desktop or laptop, not on one of the Fermilab interactive machines.
The instructions below are good for most unix based systems, including Mac OS. If someone knows the instructions for other operating systems, please add them here.
Instructions for Unix and Mac
The first step is to choose a password that you will provide when prompted. Please use something unique and do not recycle your kerberos or services password. You will rarely need this password so make sure you have a means to remember it.
- cd ~/.ssh
- ssh-keygen
- it will also prompt you for a file name; I chose robk_rsa .
- it will prompt you for password; use the new one you chose above.
This will create two files in the current directory named robk_rsa and robk_rsa.pub; the former holds your private key and the latter your public key.
Adding a key to your ssh-agent
Do this step on your desktop or laptop, not on one of the Fermilab interactive machines.
The next step is to add your private key to the ssh-agent running on your machine. You can check to see if there are already any registered private keys with the command:
- ssh-add -l
The option is a lower case letter L, not the numeral one.
To add your new key: (n.b. rhb was having trouble with this step after a recent Mac upgrade, and tried with the unqualified pathname, and then it (ahem) just worked. The old key which had been working did not, the new key did not, and only the unqualified pathname worked)
- cd ~/.ssh
- ssh-add ~/.ssh/robk_rsa
- You will be prompted a password; use the password you gave to ssh-keygen.
RHB has also determined that at least on OSX 10.14.6 every reboot requires this ssh-add step.
If you get the error message "Could not open a connection to your authentication agent", you don't have the agent running, you can start it with
- eval `ssh-agent`
You can verify that the ssh-agent has your key:
- ssh-add -l
If you do this in one terminal window during an interactive session it will immediately be present in all other terminal windows and all other applications.
When you reboot your computer the key will no longer be in your ssh-agent. To add it you need again to run
- ssh-add ~/.ssh/robk_rsa
so you had better remember that password! You could put ssh-add -l in your .profile and then before you get too far you'll see if you need to add the agent again. Once this step is complete you will be able to communicate securely from your laptop/desktop to people/applications with whom you have shared your public key.
To delete a key from ssh-agent:
- ssh-add -d robk_rsa
Forwarding your keys from your laptop/desktop to the Mu2e interactive machines
Do this step on your desktop or laptop, not on one of the Fermilab interactive machines.
The final step is to forward the private key information from the ssh-agent on your laptop/desktop to a Mu2e interactive machine so that you can also have secure communications when logged into one of those machines. Forwarding your private keys via ssh is more secure than placing a copy of your private key on the Mu2e interactive machines.
There are two ways to do this. The first way is to add the -A option to your ssh command line:
ssh -A mu2egpvm01
You will need to add the -A option everytime that you ssh into a machine. You can verify that your credentials have been forwarded by logging into, for example, mu2egpvm01 and issuing the following command:
ssh-add -l
In my case I see the forwarded key information plus a warning message that indicates that the older protocol 1 is still active but not in use.
error fetching identities for protocol 1: agent refused operation
This is only a warning message.
The second way is that you can modify your ~/.ssh/config file so that the -A option is present by default on every ssh command. Make a backup copy of ~/.ssh/config. The structure of this file is a series of blocks beginning with "Host pattern", followed by lines with option-value pairs. When you ssh into a machine, ssh will look in the config file and pattern match the name of your target machine to the patterns given in the lines beginning with "Host". If the name will match multiple patterns, the first match wins. The final Host block should begin with "Host *" to match all machine names that have not already matched. The options given in the matched Host block will be applied to your ssh session. The option that controls forwarding of ssh-agent credentials is: "ForwardAgent". The spirit of the following instructions is that you should only forward these credentials to machines that need them and that you trust; forwarding them to all machines risks theft of the credentials.
In the block beginning:
Host *.fnal.gov
add the line:
ForwardAgent yes
The ssh config protocol requires an exact match for machine names. So if you normally type ssh mu2egpvm01, not mu2egpvm01.fnal.gov, it will not match the *.fnal.gov pattern. To rectify this you can copy the "Host *.fnal.gov" block to a new block named "Host mu2e*" or simply use the "Host mu2e*" block only, if that matches your needs. The final block of the file should begin:
Host *
which specifies options for all machines not matched by other patterns. It should contain:
ForwardAgent no
Authenticating to github
GitHub has recently announced that password based authentication is deprecated and will soon be disabled. Several types of token based authentication, including ssh keys, will continue to be supported. This section describes how to use ssh keys for authentication.
GitHub has good instructions for registering ssh keys with GitHub. The instructions below add a few details:
- Follow the instructions Authentication#SSH Keys to enable ssh keys on your laptop/desktop and forward them to the Mu2e interactive machines.
- Log in to your github account via the web interface: https://github.com
- In the upper right corner, click on your picture or avatar icon to get a pop-up menu
- From the menu select "Settings"
- In the left hand task bar select "SSH and GPG Keys"
- In the upper right of the main pane there is a green button "New SSH Key". Click it.
- Mouse the content of your public key file into the place provided. Fill in the title field with a unique name.
- Click on the green "Add SSH Key" below the field for the content of your key file.
- The system will prompt your for the password to your *github* account, not the password you used to encrypt the key.
On your laptop or desktop you can test that everything is working by cloning a repository:
git clone git@github.com:username/Offline
where "username" is the GitHub name of the fork that you are cloning; the official Mu2e repository has the username "mu2e".
Also test that your ssh keys have been forwarded to the mu2e interactive machines as follows:
ssh mu2egpvm01 setup mu2e git clone git@github.com:username/Offline
Jupyter
Jupyter is a web-based interactive development environment which allows to have notebooks containing code, text, and plots. The most convenient way to use Jupyter and access Mu2e files is to run a session on a Mu2e gpvm and create a SSH tunnel to your local machine. The recommended way to install Jupyter is through conda, a package management system which manages self-contained and independent Python environments.
Install Miniconda
Anaconda has notified Fermilab that the terms of their licensing agreement do not allow for free usage at the lab of the commercial Anaconda and Miniconda distributions.
The first thing we do is to download and run Miniconda, a minimal installer of conda. So we log into a Mu2e gpvm (e.g. mu2egpvm01) and run:
cd setup mu2e setup python v3_8_3b wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh chmod +x Miniconda3-latest-Linux-x86_64.sh ./Miniconda3-latest-Linux-x86_64.sh echo ". ~/miniconda3/etc/profile.d/conda.sh" >> ~/.bash_profile
Install Jupyter
Now, we logout and login again to create a conda environment containing Jupyter:
conda create -n jupyter_env conda activate jupyter_env conda install -c conda-forge jupyterlab
Create a SSH tunnel
Now it is possible to run a Jupyter session (in this case JupyterLab):
jupyter-lab --no-browser
A token, needed to access the session from another browser, will appear. Take note of it and open a SSH tunnel on your local machine:
ssh -L localhost:8888:localhost:8888 user@mu2egpvm01.fnal.gov
Now, open your browser at http://localhost:8888 on your local machine and enter the token. The Jupyter session will appear and you should be able to access and the files on the gpvm.
If you terminate the session on the gpvm you will now only need to load the conda environment and create a new Jupyter session:
conda activate jupyter_env jupyter-lab --no-browser