ComputingLogin

Introduction

Fermilab maintains several machines for interactive login by Mu2e members. Typically all interactive work would be done on the mu2evm (GPCF) nodes. This web page describes how to log in to these machines and and a few others. We use the bash shell exclusively.

Machines

The interactive login nodes come in these groups:

• mu2evm.fnal.gov This is a pool of virtual machines within the General Purpose Computing Farm (GPCF), the main interactive computing facility for use by the Intensity Frontier experiments. They are identical virtual quad-core machines running SL7 linux. A virtual system looks like a single machine but is actually a copy of the operating system that may share the hardware with other copies of the OS. If you login to mu2evm you will be diverted to one of the machines in the pool, currently mu2egpvm01 through mu2egpvm06. You can also ssh directly to one of these nodes. These machines are intended for developing, compiling and linking jobs and other general purpose uses. You can also run short jobs here - longer jobs should be submitted to the grid. These nodes mount all the disks.
  • mu2egpvm02 has updates live - they follow the repos from the night before
  • all others are delayed - they follow the clone from 30 days ago

• mu2ebuild01.fnal.gov This is a 16-core bare metal machine setup very similarly to the virtual machines. This machine is only for building code (running scons) and running a few simple tests on the build results. It explicitly is not for running production jobs, analysis, or anything at all that takes more than a few minutes. As soon as anyone uses the machine for running jobs, its purpose as a build machine would be defeated.

• mu2edaq01.fnal.gov This is a 24-core virtual machine, running SL7, dedicated to developing the DAQ and triggers. Special login permission required.

• mu2eimagegpmv01.fnal.gov A dedicated machine, running Docker server, for building Docker images. Ask offline management for login permissions.

• fnalu.fnal.gov This is a legacy machine which we have access to, but it is is only used for a few very specific purposes.

You can only log in to the above nodes if you have a Fermilab computer account. The only permitted access method is ssh authenticated with a kerberos ticket, as described more below.

Monitoring of infrastructure and production runs on mu2egpvm01 (in mu2epro account). mu2egpvm02 gets updated to the next minor release of scientific linux as soon as it is available, but the rest are updated at the next monthly opportunity. The idea is that if something bad happens during an upgrade, we catch it on 02 first.

Logging in from Linux or Mac's

From a Unix or Mac system you need to issue the following commands to log into one of the Fermilab interactive machines:

> kinit -l 6d -f
> ssh -AKX -l your_kerberos_principal mu2egpvm01.fnal.gov

The -f argument to kinit requests a forwardable kerberos ticket; this means that when you are logged in to your target machine you can request services that ask for kerberos authentication. Otherwise you just have permission to get into the machine, not to get further authorization once there. The -AK argument to ssh requests ssh to forward your forwardable ticket. The X argument might or might not be necessary: see the next section.

If your username on your desktop/laptop is he same as your kerberos principal, you may omit the "-l your_kerberos_principal".

Mac OpenGL 3D Error

When using OPEN GL based 3D event displays on MAC OS some people will get an error message containing the string "cannot load swrast driver". When this occurs no graphics is produced. The specific circumstances in which this error has been observed are:

• You are working on a Mac laptop or desktop.
• You are logged into one of the Mu2e linux machines ( for example one of the mu2egpvm machines ).
• You are running a graphics program on the linux machine that opens a graphics window on your display.
• Your Mac is using XQuartz installed as its X11 server and the version is at least 2.7.10.
• Two cases in which this error has been seen are:
  • using the Geant4 based event display ( see the g4test_01.fcl example on the G4 instructions page.
  • using the ROOT based geometry browser.

The most general solution is:

1. Quit XQuartz.
2. Open a fresh Terminal window and issue the command
defaults write org.macosforge.xquartz.X11 enable_iglx -bool true
3. Restart XQuartz and open a new Terminal window (either from the Applications menu or using the cmd-N keyboard shortcut). ( It is different from the default Mac Terminal - it is really an xTerm. )
4. Use this xTerm window to login to a mu2egpvmxx machine:
ssh -KAX userID@mu2egpvmxx.fnal.gov where userID is of course your own user ID.
5. Your 3D graphics should now work, although you may still see an initial complaint.

In the error message, "swrast" refers to software rasterization. Depending on the details of your computer, you may have a high end graphics card capable of hardware rasterization of 3D graphics. If you do not, then X11 installation needs to do software rasterization of 3D graphics (which will work but will be slower).

If you have both a low and and a high end graphics card, your Mac normally automatically switches between the two (it uses the low end card when it can in order to save energy ). However there is a bug in the automatic switching that leads to the same error as seen above. In this case a possible solution is to disable automatic switching and to always use the high end card. The instructions for this are:

1. To see if you have video cards: open up the Apple Menu and chose “About This Mac”. Click on “System Report”. In the left hand sidebar click on Graphics/Displays. On my machine this shows:
AMD Radeon R9 M370X Intel Iris Pro The first is the more powerful video card; the second is the default on-chip video “card”.
2. If you see only one video card then these instructions are not useful; use the instructions above.
3. Go to the System Preferences (Gear icon) and choose energy saver. At the top is a check box for automatic graphics switching. Uncheck the box.
4. Log out of the remote linux machine; restart XQuartz; log in again into the remote linux machine and retry.

Note on Mac's

• You may need to download and install the current krb5.conf and consult the authentication page, this Mac kerberos article, the Mac at Fermilab page, and the web page with suggestions on how to resolve ssh problems.

• Any time that you upgrade your OS you must check whether or not the upgrade overwrite /etc/krb5.conf; if necessary restore it from the link above.

• Starting Thursday, Oct. 31, 2019, any Mac computer running macOS Sierra (v10.12) will be blocked from accessing the Fermilab network.

• 9/2015 - Users switching from OSX 10.9.5 to OSX 10.10.5 have noticed that their ticket lifetime defaults to 10h instead of the 26h that is standard at the lab. The solution is to update krb5.conf. At this time, this official source was not up-to-date, so the working conf file had to be downloaded directly from the authentication experts.

• 1/2017: Problems logging into lab Linux machines from Sierra: the lab's official answer

• 10/2017 If running Yosemite (v10.10), you must upgrade it to Sierra (v10.12) or El Captain (v10.11) by Nov. 10, 2017.

• 10/2018 macOS El Capitan (v10.11) end-of-life Thursday, Nov. 1, 2018

• upgrade notes from Eric Prebys.

• Allowing remote nodes to Open a Window on your Linux or Mac Laptop or Desktop

• Mac OS X users can acquire the X.Org X Window System disk image at xquartz.org.

• If you use Scientific Linux Fermi (SLF) it is normally configured so that when you use ssh on your laptop to log in to a remote node ( like one of the GPCF nodes), software running on that remote node is permitted to open a new window on your laptop display. If this does not work, add the -X or -Y options to your ssh command:

> ssh -X -AK -l your_kerberos_principal mu2egpvm01.fnal.gov

• On some very old versions of MAC OS you must run ssh from within an XTerm. If you run it from within a Terminal it will not be possible to have windows from the remote machine appear on your display.

• kerberos ticket lifetimes on a Mac. As a linux system, a Mac may have keberos installed. You will probably need to download a custom krb5.conf.

• As of October 1st of 2019, Fermi will no longer be support the 10.12 Sierra OS and systems will be blocked from the network until upgraded.

ssh on Very Old MAC's

There were some issues related to older MACs running MAC OS X with a version earlier than Leopard. Since Snow Leopard all of these issues are resolved. Follow this link to find an explanation of how to access Fermilab machines using Macs running OS X Panther or earlier.

If you have and Apple Mac that is still running a version of OS X that is earlier than Leopard ( ie Tiger or Panther ), then there are some tricks to login in to the Fermilab machines.

The first is that some of these MACs require you to use an XTerm instead of a Terminal; if you log in using a Terminal then the Fermilab machine will not be able to open windows that appear on your laptop.

The second issue is that the version of kerberos that comes with the older Mac OS X version is not compatible with the version of kerberos installed on the FNALU nodes. The solution is to install a second version of ssh on your Mac. This installs as scp3a and ssh3a, so it does not overwrite the native ssh.

Installation packages for both Panther and Tiger are available:

http://home.fnal.gov/~mzs/fnalssh103.dmg
http://home.fnal.gov/~mzs/fnalssh104.dmg

fnalssh103.dmg is for OS X 10.3 and fnalssh104.dmg is for OS X 10.4. I believe, but am not 100% sure, that neither Leopard nor Snow Leopard require this fix.

Just use ssh3a, scp3a, and sftp3a instead of ssh, scp, and sftp after installation to connect to hosts running older versions of sshd.

It's all explained in the README file that comes with the downloads. This is probably enough for most people.

Additional information is available at the Mac section of the FNAL strong authentication site.

Logging in From PC's

Fermilab does not officially support logging in to one of the Mu2e interactive nodes from a PC. We are trying to change this. Your options are:

• The current recommended option is to use puTTy ssh client and xming xwindows server. Unoffical but helpful links: CMS doc link link link.
• Another option is to install cyqwin , which provides a Linux-like environment for Windows. Useful instructions can be found here cygwin-instructions. You can install kerberized ssh in this environment. Consult with the Service Desk to learn how to configure ssh to work correctly.
• The service desk may recommend Reflection software. You can run terminals on a remote linux host and display the terminal and other xwindows on your PC, and use ftp to move files.
• You can install SLF as a guest virtual OS hosted by your Windows machine.
• Purchase and install WRQ, which is X-Windows software that runs on PCs; configure it to do kerberized ssh. The Service Desk may be able to help with this configuration because they support WRQ but only for Fermilab employees. You need to pay for WRQ and non-employees are not covered by the Fermilab license.
• You can configure your machine for dual boot and install both Linux and Windows. Boot to Linux when you wish to log in to Fermilab. This is very inconvenient if you need to switch from one OS to the other frequently.

VPN

Some web pages at the lab are restricted to viewing only on the lab network. If you are offsite and want to access one of these pages, you can use a VPN, which runs on a gateway node, authenticates you, then redirects your web traffic onto the lab network. Here is the lab VPN (use your services [email] password) and some VPN help. As of 8/2019, the VPN requires multi-factor authentication. There are two forms, a software or RSA form which is good enough for most users, and a yubikey (usb dongle) stringer form for people who need to access personnel and business information.

Remote Desktops

VNC

General information

VNC stands for Virtual Network Connection. It allows you to run your session on one of Mu2e interactive machines and display the running session it on your local desktop of laptop. VNC runs in a client-server mode, with client performing only display functions. Therefore, if you disconnect and reconnect later, you will continue to work in the same session. VNC handles efficiently the X11-compression and works over the network much faster than a direct SSH connection. It works well on network connections with ping times up to 200 ms, getting in trouble only if TCP packets start getting lost.

VNC server software is installed on MU2EGPVM* machines. VNC server runs in a user mode - each user starts his own server(s). The local computer only needs a VNC client and a kerberized SSH installations.

Below are a few links to general VNC tutorials:

• VNC
• wikipedia
• tutorial

Setting up a VNC server

To set up a VNC server follow these instructions:

1. ssh into one of the mu2egpvm machines (like mu2egpvm04)
2. once logged in, do:

kinit                  # get Kerberos ticket
vncpasswd              # create your VNC server password

3. check already running VNC servers:

   murat@mu2egpvm02:~>uname -a; ps -efl | grep -i Xvnc | cut -c-120
   Linux mu2egpvm02.fnal.gov 3.10.0-1127.13.1.el7.x86_64 #1 SMP Tue Jun 23 10:32:27 CDT 2020 x86_64 x86_64 x86_64 GNU/Linux
   0 S bvitali   3704     1  0  80   0 - 286048 ep_pol Jul18 ?       02:37:19 /usr/bin/Xvnc :6 -auth /nashome/b/bvitali/.Xa
   0 S murat     3797  3326  0  80   0 - 28204 pipe_w 17:11 pts/57   00:00:00 grep --color=auto -i Xvnc
   0 S mmackenz  3888     1  0  80   0 - 125007 ep_pol Jul31 ?       03:32:25 /usr/bin/Xvnc :23 -auth /nashome/m/mmackenz/.
   0 S defelice  4232     1  0  80   0 - 205764 ep_pol Jul17 ?       03:49:00 /usr/bin/Xvnc :08 -auth /nashome/d/defelice/.
   0 S zanetti  11491     1  0  80   0 - 65755 ep_pol Sep08 ?        00:00:13 /usr/bin/Xvnc :2 -auth /nashome/z/zanetti/.Xa
   0 S mmackenz 14149     1  0  80   0 - 63366 ep_pol Jul17 ?        00:03:11 /usr/bin/Xvnc :1 -auth /nashome/m/mmackenz/.X
   0 S bbarton  18286     1  0  80   0 - 77543 ep_pol Jul17 ?        02:33:33 /usr/bin/Xvnc :5 -auth /nashome/b/bbarton/.Xa
   0 S chenj    21733     1  0  80   0 - 129571 ep_pol Jul28 ?       06:07:57 /usr/bin/Xvnc :9 -auth /nashome/c/chenj/.Xaut
   0 S gianipez 30193     1  0  80   0 - 83661 ep_pol Aug19 ?        00:09:30 /usr/bin/Xvnc :4 -auth /nashome/g/gianipez/.X
   0 S huangs   32386     1  0  80   0 - 92075 ep_pol Aug04 ?        03:27:20 /usr/bin/Xvnc :29 -auth /nashome/h/huangs/.Xa
   

4. start your VNC server with unused Xserver ID, for example, use ID=7:

vncserver :7 -name MY_VNC -depth 24 -geometry 1920x1080 -localhost

Using '-localhost' option is a must, the server should be listening to a local port

Setting up a VNC client on Linux

1. return to your local machine
2. download a vncviewer. There is a multitude of those, for example, TigerVNC, RealVNC,etc. Mac OS has a native vncviewer, but it is slower than the others. TigerVNC is the most performant one, we recommend trying it first.
3. once you have a VNC viewer installed, on the local machine, start a SSH tunnel connecting the local machine to the remote server:
ssh -v -f -X -N -L PORT:localhost:PORT <kerberos_username>@mu2egpvm04.fnal.gov notes:
• replace PORT with 5900+ID , replace mu2egpvm04 with the name of the machine where you started the vncserver.
4. open the vncviewer and type: localhost:ID, where ID is the same as above, then click "connect"

5. you will be asked to enter the password you set on step (3)

This is it! you now have acccess to your server that will look like a linux machine.

Setting up a VNC client on Windows

• install Cygwin in a minimal configuration, you only need the core and krb5 client
• copy /etc/krb5.conf from one of the Mu2e interactive nodes to /etc directory of Cygwin installation
• create the following .ssh/config file in the home directory

Host *
    GSSAPIAuthentication      yes
    GSSAPIDelegateCredentials no
    ForwardX11Trusted         yes

• start Cygwin
• the rest is the same, as for Linux

Setting up a VNC client on MacOS

• mostly, the same as on Linux
• to setup an ssh tunnel, run

 ssh -v -f -KXY -N -L PORT:localhost:PORT <kerberos_username>@mu2egpvm04.fnal.gov 

• when connecting a client, use 5900+ID instead of ID, i.e. localhost:5901 instead of localhost:1

Few useful scripts on mu2egpvm* nodes

• check VNC servers running on a given machine, in this example - mu2egpvm01. Look at several (mu2egpvm01-mu2egpvm06), choose the one which is less loaded

kinit
ssh YOUR_FNAL_USERNAME@mu2egpvm01.fnal.gov /mu2e/app/users/murat/bin/check_running_vnc_servers
/usr/bin/Xvnc  :2 huangs               
/usr/bin/Xvnc :22 mchattor             
/usr/bin/Xvnc  :3 kharrig              
/usr/bin/Xvnc :99 murat                
/usr/bin/Xvnc  :1 merrill              

• choose the display number which is not used, for this example display=23 would do. Start VNC server and create password:

ssh YOUR_FNAL_USERNAME@mu2egpvm01.fnal.gov 
/mu2e/app/users/murat/bin/start_vnc_server :23 1920x1080  #  1920x1080 - is the screen size displayed by the server 
vncpasswd

• kill server running on display :23 :

ssh YOUR_FNAL_USERNAME@mu2egpvm01.fnal.gov /mu2e/app/users/murat/bin/kill_vnc_server :23

Comments

• you must limit the terminal history scrolling to not more than 10000 lines. Otherwise, the /tmp area on the central machine

will get filled up and the machine will need to be rebooted.

• Once in a while, you need to reconnect. When restarting the SSH tunnel on the local machine, don't forget to kill the stale SSH tunnel processes.

Issue the following command to find those:

  pf -efl | grep ssh

• you can add the following function to your .bashrc and use it to restart the SSH tunnel - it automatically finds and kills stale tunnel processes.

it is assumed that the used port numbers on local and remote machines are the same and that the server is running within the fnal.gov domain

#---------------------------------------------------------------------------------------
# to restart SSH port tunneling for VNC server running on mu2egpvm04 and using port 5902: 
# 
# vnc 2 mu2egpvm04
#----------------------------------------------------------------------------------------
function vnc  () {
    display=$1
    host=$2                              # assume "$host.fnal.gov" exists
    port=`printf "59%02i" $display`      # don't forget to check if the port is available
    forwarding=$port:localhost:$port
    kill -9 `ps -efl | grep ssh | grep $forwarding | awk '{print $4}'` ;
    ssh -v -f -X -N -L $forwarding $host.fnal.gov ;
}

Known problems

• VNC server v1.8.0 crashes on SL7 : disable back store - use -bs option when starting the VNC server

see details at https://mu2e-hnews.fnal.gov/HyperNews/Mu2e/get/Sim/744/6/2/2/1/1.html

• 8/8/19 - Using "-bs" seems to break the root geometry viewer ogl (OpenGL) option - driver upgrade in the end of Sep'2019 fixed that

• copy and paste doesn't work with TigerVNC 1.8.0 / SL7 : see workaround at https://mu2e-hnews.fnal.gov/HyperNews/Mu2e/get/Sim/744/6/2/2/1/2.html

Accessing windows on linux

rdesktop on SL6, xfreerdp on SL7 (xfreerdp comes from the RPM called 'freerdp'). Command:

xfreerdp fermi-ts -u <username> -d fermi

More info