User:Rlc: Difference between revisions

From Mu2eWiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(20 intermediate revisions by the same user not shown)
Line 1: Line 1:


[[Category:Test]]
{{Special:PrefixIndex/User:Rlc/}}
[[Category:Computing]]


==Introduction==
α β γ δ ε ζ η θ ι κ λ μ ν ξ ο π ρ σ ς τ υ φ χ ψ ω Γ Δ Θ Λ Ξ Π Σ Φ Ψ Ω
These are notes on using the group account mu2epro, which is used for collaboration-managed production activities. Access is controlled by the k5login. As quick start quide, you can get the correct environment and authentication with:


<pre>ssh -k mu2epro@nodename (-k is important!)    formatted with pre</pre>
flavor flavour
favor
uniqword


==Kerberos==
[https://www.nevis.columbia.edu/~seligman/root-class nevis root]<br>
In 4/2015 we (rlc) installed kerberos keytabs in <code>~mu2epro/keytab (code inline)</code>. The principals are created by the servicedesk, and a temp password returned. The actual <tt>keytab (tt inline)</tt> file is made according to this article.
[http://www.slac.stanford.edu/BFROOT/www/doc/workbook/root1/root1.html Babr root]<br>


In 8/2016 we asked the service desk to reset the keytab passwords and then ran the procedure '''/usr/krb5/config/make-cron-keytab''' This moves the keytab to a secure system location under /var. This solves the security problem of using the keytab from a network-mounted disk, exposing the kerberos data to the network.
- how to log out (AG)<br>
- figure the visibility and log in rules<br>
- populate some of internal<br>


To get a new ticket, you run
put this in
Template:H:title
<span title="{{{1}}}" {{#ifeq:{{{dotted|yes}}}|no||style="border-bottom:1px dotted"}} class="help">{{{2}}}</span><noinclude>{{Documentation}}</noinclude>


:: <pre>/usr/krb5/bin/kcron      formatted with pre and indentx2</pre>


at any time, but please see the notes below on where the ticket goes.
{{H:title|Yes you will!|
Can you see the answer by hovering over this text?}}


A cron job on all of the interactive machines runs every 4 hours, runs a kcron command and writes the ticket to /tmp/krb5cc_44592_auto. As long as you use this ticket location, you will always have a valid ticket.
<div class="usermessage mw-customtoggle-myTable">Answer.</div>


Using the keytab with
{| class="wikitable mw-collapsible" id="mw-customcollapsible-myTable"
! Hello
! World
|-
| Content
| Goes
|-
| In
| Here
|}


<code>kinit -k -t $HOME/keytab/mu2egpvm01.keytab mu2epro/cron/mu2egpvm01.fnal.gov@FNAL.GOV (formatted with code)</code>
<div style="width:600">
{| class="mw-collapsible mw-collapsed wikitable" style="text-align: left; width=300"
! style="background-color: white; width=300" | What is Mu2e trying to measure?||
|-
| colspan="2"| Mu2e is an experiment designed to measure the conversion of a muon into an electron (in the field of the nucleus) without the emission of the neutrinos. In the standard model, lepton conservation tells us that when a muon decays to an electron there should be an accompanying muon neutrino to conserve muon number in the decay and an anti-electron neutrino to conserve electron number.  
|}
</div>


is no longer available.


It is important to setup how the ticket is located. If you do not have the KRB5CCNAME environmental set when you run kcron, the ticket will go to /tmp/krb5cc_44592cronXXXXX and no other utility will find it, not even klist, so you will always want to have KRB5CCNAME set. What follows is how it gets set.
<div class="toccolours mw-collapsible" style="width:40em;">
Q: question here <br>
A: answer here
</div>


Ideally users should
{{FAQ}}
{{Expert}}
{{Draft}}
{{Obsolete}}


:: <code>ssh -k mu2epro@nodename formatted with code and indentx2</code>
Pages that need help:
*Authentication
* review examples in workflow
* wiki tips


The -k switch stops your personal ticket from being forwarded to your interactive sessions as mu2epro. During the login, .bash_profile will set KRB5CCNAME to FILE:/tmp/krb5cc_44592_auto for you. If you avoid using a "login shell" you will not run .bash_profile and KRB5CCNAME will not be set. In some philosophical sense this is what you asked for, so just use login shells.
Meeting:
* Lockdown
* syops
* getting users in


If you forward your ticket (not recommended):
mixing is now done with particle/POT (used to be something else)


<code>
ssh mu2epro@nodename (formatted with code and newlines)
</code>


you will end up in the mu2epro account, and
your login can expire while you are editing - copy web page to a file,
then log in.  Partial edit will not be saved.  Not straightforward, but possible
to go back into the editor, not sure of the algorithm..


KRB5CCNAME=FILE:/tmp/krb5cc_44592_XXXXXXXXXX
https://www.mediawiki.org/wiki/Extension:LinkTarget
or set global parameters


which is a ticket with your personal principle. But in .bash_profile, KRB5CCNAME will be reset to FILE:/tmp/krb5cc_44592_auto since this is the only reasonable and convenient default for the mu2epro account. If you do forward your personal ticket, it is possible for other people in the mu2epro account to gain your credentials. Also if you are using your personal ticket and run bare jobsub_submit, the job will be run in your personal account, not mu2epro.
https://www.mediawiki.org/wiki/Extension:CirrusSearch


Anyone can get their personal principal back with
https://www.mediawiki.org/wiki/Extension:Lockdown
LocalSettings.php


kinit your_user_name@FNAL.GOV


But note that if KRB5CCNAME is not set, the ticket will default to /tmp/krb5cc_44592, which may be found by utilities or other users. If it is set to FILE:/tmp/krb5cc_44592_auto, it will overwrite the mu2epro ticket, so please don't do that!
==  Introduction ==
Keeping all Intensity Frontier data on disks is not practical,
so large datasets must be written to tape.  At the same time, the data
must always be available and delivered efficiently.
The solution is coordinating several subsystems:


Bottom line, just ssh -k and only use the default mu2epro automatic tickets!
<ul>
<li><b>dCache</b>: a set of disk servers, a database of files
on the servers, and services to deliver those files with high throughput
<ul>
<li>[scratchDcache.shtml <b>scratch dCache</b>] : a dCache where least used files are purged as space is needed.
<li><b>tape-backed dCache</b>: a dCache where all files are on tape and are
cycled in and out of the dCache as needed
</ul>
<li><b>pnfs</b>: an nfs server behind the /pnfs/mu2e parition
which looks like a file system to users, but
is actually a interface to the dCache file database.
<li><b>Enstore</b>: the Fermilab system of tape and tape drive management
<li><b>SAM</b>: Serial Access to Metadata, a database of file metadata
and a system for managing large-scale file delivery
<li><b>FTS</b>:File Transfer Service, a process which manages the intake
of files into the tape-backed dCache and SAM.
<li><b>jsonMaker</b>: a piece of mu2e code which helps create
and check metadata when creating a SAM record of a file
<li><b>SFA</b>: Small File Aggregation, enstore can tar up small
files into a single large file before it goes to tape,
to increase tape efficiency.
</ul>


Andrei has an ssh config setup that will effectively set "-k" for you automatically. Add this text block to $HOME/.ssh/config


<blockquote>
The basic procedure is for the user to run the jsonMaker
<pre>
on a data file to make the
#----------------
[http://json.org json file] ,
Host mu2epro
then copy both the data file and the json
User mu2epro
into an FTS area in </a "scratchDcache.shtml">scratch dCache</a>  
HostName mu2egpvm01.fnal.gov
called a dropbox.
ForwardAgent no
The json file is
GSSAPIDelegateCredentials no
essentially a set of metadata fields with the
#----------------
corresponding values.  The
</pre>
[http://mu2esamgpvm02.fnal.gov:8787/fts/status FTS] 
</blockquote>
will see the file with its json
and then the command
file, and copy the file to a permanent location in
the tape-backed dCache and use the json to create a metadata record in SAM.
The tape-backed dCache will migrate the file to tape quickly
and the SAM record will be updated with the tape location.
Users will use [sam.shtml SAM]  to read the files
in tape-backed dCache.


ssh mu2epro


translates to "ssh -k mu2epro@mu2egpvm01.fnal.gov".
Since there is some overhead in uploading, storing and retrieving
each file, the ideal file size is as large as reasonable.
This size limit should be determined by how long an executable
will typically take to read the file.  This will vary according to  
exe settings and other factors, so a conservative estimate
should be used.    A file should be sized so that the longest
jobs reading it should take about 4 to 8 hours to run, which
generally provides efficient large-scale job processing.
A grid job that reads a few files in 4 hours is nearly as efficient,
so you can err on the small size.
You definately want to avoid a single
job section requiring only part of a large file.
Generally,
file sizes should not go over 20 GB in any case because they get
less convenient in several ways.  
Files can be concatenated
to make them larger, or split to make them smaller.
Note - we have agreed that a subrun will only appear in one file.
Until we get more experience with data handling, and
see how important these effects are, we will
often upload files in the size we make them or find them.


==Certificate==
Once files have been moved into the FTS directories,
This account has a OSG Service Certificate consisting of certificate and key files kept in ~mu2epro/osg_service_cert_2015. This directory should have only owner access permissions. Notes on obtaining the cert are also in that directory. The cert is valid until 2/24/17.
please do not try to move or delete them since this will
confuse the FTS and require a hand cleanup.  Once files
are on tape, there is an expert procedure to delete them,
and files of the same name can then be uploaded to replace
the bad files.


The cert has been copied to a Computing Division server. There, it is converted to a VOMS proxy every 8 hours and that proxy is copied in a secure way to
<!********************************************************>
==  Recipe==


<code>
If you are about to run some new Monte Carlo in the official framework,
/opt/mu2epro/mu2epro.Production.proxy
then the upload will be built into the scripts and documented
</code>
with the mu2egrid
[ Monte Carlo submission]  process.
<font color=red>this is under development,
please ask Andrei for the status</font>


on all the interactive machines. I don't know if the proxy is updated by an atomic command, (so there is no chance of a user finding a partial proxy file).


Andrei has installed the actual certificate in the /opt area too. We believe the /opt area is a local disk so this will not expose the cert over the network. This will allow scripts to authenticate https communication with the SAM server. There may be other methods to authenticate, but the concern is that they will time-out.
Existing files on local disks can be uploaded using the following steps.
The best approach would be to read quickly through the rest of this
page for concepts then focus on the  
[uploadExample.shtml upload examples]  page.


==Setup scripts==
<ul>
<li>choose values for the [#metadata SAM Metadata] , including
      the appropriate [#ff file family]
<li>record the above items in a json file fragment
that will apply to all the files in your dataset
<li>[#name rename]  your files by the upload convention
(This can also be done by jsonMaker in the next step.)
<li>setup an offline release and run the [#jsonMaker jsonMaker] 
to write the json file, which will include the fragment from the previous step
<li>use "ifdh cp" to copy the data file and the full json file to the FTS area
/pnfs/mu2e/scratch/fts (This step can also can be done by jsonMaker.)
<li>use [sam.shtml SAM]  to access the file or its metadata
</ul>


The login file .bash_profile defines KRB5CCNAME to point to the mu2epro automatic kerberos ticket. It also makes sure /usr/krb5/bin and ~/bin are in the path, defines some environmentals which allow you to use the mu2epro authentication for commands, and aliases for setting further things up.
</ol>
The following is some detail you should be aware of in general, but
a detailed knowledge is not required.


.bashrc defines aliases for personalization for operators and for projects (see next). It also runs


/cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups
{{Special:PrefixIndex/Help:Subpages/}}


to point PRODUCTS to the common products, allowing "setup mu2e" as well as fermilab common products, such as jobsub_client.
<pre>
 
1.35.11  skin is vector
After logging in, the user can chose their environment. You can set your preferences through a personal setup script alias such as "setup_rlc". Please see
$wgAllowUserJs  = true;
 
$wgAllowUserCss  = true;
~mu2epro/bin/setup_alias.sh
https://en.wikipedia.org/wiki/Special:MyPage/skin.css
 
https://mu2ewiki.fnal.gov/wiki/User:Rlc/skin.css  common.css vector.css vector-legacy.css
for examples. These scripts should only set minor interactive preferences such as the prompt or "rm -i". It should set the environmental
mw.config.get( 'skin' )
 
https://en.wikipedia.org/wiki/Help:Link_color
export OPERATOR=rlc
#3366bb
 
/* standard link colors */
to your username. It should not do anything required for real production projects. The original (TDR era) setup is saved as "setup_old".
.mw-body-content a:link { color: #0000FF; } /* normal unvisited links */
 
.mw-body-content a:link:visited { color: #0B0080; } /* visited links */
In addition, you can run the setup for a project, such as "setup_upload". These should set the environmental MUE2PRO_PROJECT to a unique value. This script should then setup all environment needed for the project. This should be complete, but not include any enviroment that is not necessary to run the project. Necessary setups might include "setup mu2e", offline version setup, jobsub, ifdhc and useful environmentals, such as pointers to scratch disk directories.
.mw-body-content a:link:active { color: #FF0000; } /* active links */
 
.mw-body-content a:link.new { color: #FF0000; } /* new links */
If the environment for a project must be strictly controlled, then you would only setup the project, and this setup should overwrite the value of OPERATOR to stop personalized .bashrc behaviour.
.mw-body-content a:link.extiw { color: #3366BB; } /* interwiki links */
 
.mw-body-content a:link.external { color: #3366BB; } /* external links */
You should write project scripts which only depend on setup_project and with alias-defeating /bin/mv etc to avoid differences in preferences. A final useful convention would be to keep project scripts in ~/projectname.
.mw-body-content a:link.stub { color: #772233; } /* hovered links */
 
There are also hooks in .bashrc to add behaviour as needed.
<code>
source ~/bin/setup_alias.sh
[ "$OPERATOR" == "rlc"      ] && source ~rlc/bin/setup_rlc.sh rc
...
[ "$MUE2PRO_PROJECT" == "upload" ] && source ~/bin/setup_upload.sh rc
</code>
cron jobs should do all their setup explicitly, which should include sourcing these setup scripts.
 
==Jobsub and ifdh==
 
ifdhc with version less than 1_7 may not work to transfer files, it may generate an incorrect gridftp url resulting in globus errors. If you write to bluearc, you will need to force the transfer through gridftp to get the permissions to work.
 
<code>
ifdh cp --force=gridftp file /mu2e/data/..
</code>
 
You will also need this switch for ifdh non-transfer commands
<code>
ifdh rm /mu2e/data/.. --force=gridftp
</code>
Writes to dCache (/pnfs/...) using x509 authentication (like gridftp or expftp) currently end up owned by mu2eana or mu2epro. gridftp-ls does not always list permissions correctly, uberftp fndca1 "ldir" does.


jobsub needs to be pointed to the OSG cert to work.
.mw-body-content a:link {color: #FF0000}
<code>
.mw-body-content a:visited {color: #00FF00}
export X509_USER_PROXY=/tmp/x509up_u44592
.mw-body-content a:hover {color: #FF00FF}
</code>
.mw-body-content a:active {color: #0000FF}
this should be done though the command


setup_mu2epro
.mw-body-content a.external {color: #008000}
.mw-body-content a.external:visited {color: #008000}


As of jobsub_client v1_1_0_2, the --role switch is needed for submit, rm, and fetchlog. It is not needed for q and history, which list the jobs as user mu2epro.
.mw-body-content a:link {color: #000000; text-decoration: underline; }
 
Note that when a user becomes mu2epro, they may still have their personal kerberos ticket active, and that may be used accidentally to authenticate the submission, so the user should kdestroy their ticket. Please see the discussion of kerberos and the setup commands below. If there is no user kerberos ticket, the job will be authenticated by the proxy.
<blockquote>
<pre>
jobsub_submit -G mu2e --role=Production -N 1 \
  --generate-email-summary --email-to=rlc@fnal.gov  --OS=SL6 \
  --resource-provides=usage_model=DEDICATED  \
  file:///mu2e/app/users/rlc/jobs/remote.bash
</pre>
</pre>
</blockquote>
"ifdh cp --force=expftp" will not work with the service cert, i.e. interactively. It works on the grid because a different grid-based cert is used there.
Cron jobs
cron job information is kept in ~mu2epro/cron. We will try to run all cron jobs on mu2egpvm01 unless they have to run elsewhere.
==Backup==
The home area is on /mu2e/app and is not backed up automatically like the user home areas. It is backed up by a cron job to /pnfs/mu2e/scratch/users/mu2epro/backup.

Latest revision as of 03:04, 30 September 2023

α β γ δ ε ζ η θ ι κ λ μ ν ξ ο π ρ σ ς τ υ φ χ ψ ω Γ Δ Θ Λ Ξ Π Σ Φ Ψ Ω

flavor flavour favor uniqword

nevis root
Babr root

- how to log out (AG)
- figure the visibility and log in rules
- populate some of internal

put this in Template:H:title {{{2}}}Template:Documentation


Template:H:title

Answer.
Hello World
Content Goes
In Here
What is Mu2e trying to measure?
Mu2e is an experiment designed to measure the conversion of a muon into an electron (in the field of the nucleus) without the emission of the neutrinos. In the standard model, lepton conservation tells us that when a muon decays to an electron there should be an accompanying muon neutrino to conserve muon number in the decay and an anti-electron neutrino to conserve electron number.


Q: question here
A: answer here

Template:FAQ

Expert.jpeg This page page needs expert review!

Construction.jpeg This page is a draft, please help complete it!

Outofdate.jpeg This page is out of date, please help update it!

Pages that need help:

  • Authentication
  • review examples in workflow
  • wiki tips

Meeting:

  • Lockdown
  • syops
  • getting users in

mixing is now done with particle/POT (used to be something else)


your login can expire while you are editing - copy web page to a file, then log in. Partial edit will not be saved. Not straightforward, but possible to go back into the editor, not sure of the algorithm..

https://www.mediawiki.org/wiki/Extension:LinkTarget or set global parameters

https://www.mediawiki.org/wiki/Extension:CirrusSearch

https://www.mediawiki.org/wiki/Extension:Lockdown LocalSettings.php


Introduction

Keeping all Intensity Frontier data on disks is not practical, so large datasets must be written to tape. At the same time, the data must always be available and delivered efficiently. The solution is coordinating several subsystems:

  • dCache: a set of disk servers, a database of files on the servers, and services to deliver those files with high throughput
    • [scratchDcache.shtml scratch dCache] : a dCache where least used files are purged as space is needed.
    • tape-backed dCache: a dCache where all files are on tape and are cycled in and out of the dCache as needed
  • pnfs: an nfs server behind the /pnfs/mu2e parition which looks like a file system to users, but is actually a interface to the dCache file database.
  • Enstore: the Fermilab system of tape and tape drive management
  • SAM: Serial Access to Metadata, a database of file metadata and a system for managing large-scale file delivery
  • FTS:File Transfer Service, a process which manages the intake of files into the tape-backed dCache and SAM.
  • jsonMaker: a piece of mu2e code which helps create and check metadata when creating a SAM record of a file
  • SFA: Small File Aggregation, enstore can tar up small files into a single large file before it goes to tape, to increase tape efficiency.


The basic procedure is for the user to run the jsonMaker on a data file to make the json file , then copy both the data file and the json into an FTS area in </a "scratchDcache.shtml">scratch dCache</a> called a dropbox. The json file is essentially a set of metadata fields with the corresponding values. The FTS will see the file with its json file, and copy the file to a permanent location in the tape-backed dCache and use the json to create a metadata record in SAM. The tape-backed dCache will migrate the file to tape quickly and the SAM record will be updated with the tape location. Users will use [sam.shtml SAM] to read the files in tape-backed dCache.


Since there is some overhead in uploading, storing and retrieving each file, the ideal file size is as large as reasonable. This size limit should be determined by how long an executable will typically take to read the file. This will vary according to exe settings and other factors, so a conservative estimate should be used. A file should be sized so that the longest jobs reading it should take about 4 to 8 hours to run, which generally provides efficient large-scale job processing. A grid job that reads a few files in 4 hours is nearly as efficient, so you can err on the small size. You definately want to avoid a single job section requiring only part of a large file. Generally, file sizes should not go over 20 GB in any case because they get less convenient in several ways. Files can be concatenated to make them larger, or split to make them smaller. Note - we have agreed that a subrun will only appear in one file. Until we get more experience with data handling, and see how important these effects are, we will often upload files in the size we make them or find them.

Once files have been moved into the FTS directories, please do not try to move or delete them since this will confuse the FTS and require a hand cleanup. Once files are on tape, there is an expert procedure to delete them, and files of the same name can then be uploaded to replace the bad files.

<!********************************************************>

Recipe

If you are about to run some new Monte Carlo in the official framework, then the upload will be built into the scripts and documented with the mu2egrid [ Monte Carlo submission] process. this is under development, please ask Andrei for the status


Existing files on local disks can be uploaded using the following steps. The best approach would be to read quickly through the rest of this page for concepts then focus on the [uploadExample.shtml upload examples] page.

  • choose values for the [#metadata SAM Metadata] , including the appropriate [#ff file family]
  • record the above items in a json file fragment that will apply to all the files in your dataset
  • [#name rename] your files by the upload convention (This can also be done by jsonMaker in the next step.)
  • setup an offline release and run the [#jsonMaker jsonMaker] to write the json file, which will include the fragment from the previous step
  • use "ifdh cp" to copy the data file and the full json file to the FTS area /pnfs/mu2e/scratch/fts (This step can also can be done by jsonMaker.)
  • use [sam.shtml SAM] to access the file or its metadata

The following is some detail you should be aware of in general, but a detailed knowledge is not required.



1.35.11  skin is vector
$wgAllowUserJs  = true;
$wgAllowUserCss  = true;
https://en.wikipedia.org/wiki/Special:MyPage/skin.css
https://mu2ewiki.fnal.gov/wiki/User:Rlc/skin.css  common.css vector.css vector-legacy.css
mw.config.get( 'skin' )
https://en.wikipedia.org/wiki/Help:Link_color
#3366bb
/* standard link colors */
.mw-body-content a:link { color: #0000FF; } /* normal unvisited links */
.mw-body-content a:link:visited { color: #0B0080; } /* visited links */
.mw-body-content a:link:active { color: #FF0000; } /* active links */
.mw-body-content a:link.new { color: #FF0000; } /* new links */
.mw-body-content a:link.extiw { color: #3366BB; } /* interwiki links */
.mw-body-content a:link.external { color: #3366BB; } /* external links */
.mw-body-content a:link.stub { color: #772233; } /* hovered links */

.mw-body-content a:link {color: #FF0000}
.mw-body-content a:visited {color: #00FF00}
.mw-body-content a:hover {color: #FF00FF}
.mw-body-content a:active {color: #0000FF}

.mw-body-content a.external {color: #008000}
.mw-body-content a.external:visited {color: #008000}

.mw-body-content a:link {color: #000000; text-decoration: underline; }