Rollout the minimal LCG system

James Werner

These are the installation procedures for a complete installation site LCG independent of any external resources (Deliverable A3.3 - 2005Q4).

Thanks to Sabah Salin and Alessandra Forti for support and discussions.

Installation tasks.

Hardware architecture
Requesting certificates
Network configuration
Adapt Scientific Linux to LCG
Java installation from RPM
Node synchronisation
Configuration tool:YAIM
Site configuration file
Installing Apt-get
Installing information system level 0 (BDII), monitor (MON), and long term proxy (PX)
Installing Resource Broker (RB)
Installing Computer Element (CE)
Installing Torque Worker Nodes (WN)
Installing Storage Element (SE)
Installing User Interface (UI)
AFS configuration
Configuration BDII PX MON
Configuration RB
Configuration CE
Configuration WNs
Configuration UI
Configuration SE
Testing LCG middleware
Operational procedures
The firewall issue

1. Hardware/Software architecture.
Savannah portal: http://savannah.cern.ch/
http://www.gridpp.ac.uk/deployment/links.html
LCG rollout archives: http://www.listserv.rl.ac.uk/archives/lcg-rollout.html
Download releases: http://lcg.web.cern.ch/LCG/Sites/releases.html
Wiki: http://goc.grid.sinica.edu.tw/gocwiki/FrontPage

LCG software is a set of specific functional computers running modules that interact
with each other during the job submission, running and deliver processes.
The processes are:

User interface (UI): It is the module that allows users interact with grid. It contains
programs to submit jobs and commands to the grid, check job status, recover outputs, etc.
To install, perform tasks 1 to 9,14, 16, 21, and 23.
Resource broker (RB): It is responsable to match the jobs with resources, implement policies,
and send jobs for processing in a remote resource (Computer element).The definition of resources
are written in the JDF file using ClassAds (see condor manual and LCG user manual for more details).
To install, perform tasks 1 to 9,11, 16, 18, and 23.
Computer element (CE): it is the batch manager (PBS) that allocates resources in the farm to run users'
software. There are interactive jobs, parallel jobs, and simple jobs.
To install, perform tasks 1 to 9,12, 16, 19, and 23.
Worker Nodes (WN): it is the computer that will run your software. It contains the packages
installed, libraries, and necessary local data for processing.
To install, perform tasks 1 to 9,13, 16, 20, and 23.
Storage Element (SE): it is the mass storage element in grid.
To install, perform tasks 1 to 9,15, 16, 22, and 23.
Berkeley Database (BDII): it is the catalog of resources published in the grid. It is distributed in
two levels. The first catalog all CE BDII entries. The second, in the CE, contains the information about the
farm managed by the CE.
Proxy manager (PX): stores proxies for long time jobs in the grid.

CE and RB never could be installed in the same computer. There are installations where RB and BDII are together.
These are the conputers with standard Scientific Linux 3.0.4 installation from
www.scientificlinux.org. The log is available here.

IP Hostname nick Function Cert
194.36.3.43 bf30.hep.man.ac.uk bf30 BDII PX MON *
194.36.3.44 bf31.hep.man.ac.uk bf31 RB *
194.36.3.45 bf32.hep.man.ac.uk bf32 CE *
194.36.3.46 bf33.hep.man.ac.uk bf33 WN
194.36.3.47 bf34.hep.man.ac.uk bf34 WN
194.36.3.48 bf35.hep.man.ac.uk bf35 WN
194.36.3.49 bf36.hep.man.ac.uk bf36 WN
194.36.3.50 bf37.hep.man.ac.uk bf37 WN
194.36.3.51 bf38.hep.man.ac.uk bf38 WN
194.36.3.52 bf39.hep.man.ac.uk bf39 UI
194.36.3.53 bf40.hep.man.ac.uk bf40 SE *

* e-Science certificate are required in these machines
The data server using NFS is cap.hep.man.ac.uk.
Most of the current servers get delivered with two ethernet: eth0 should be connected to the public network
(194.36.XX.XX) and eth1 to the private network (192.0.168.XX). The advantage is for example, when accessing
data server (cap) will not share network connection with AFS.
Another important point is: when writing data from your code, do not write records remotely (AFS, NFS, etc).
Write outputs in a local disk, and at the end of your job copy it to a remote resource.

2. Requesting certificates

3. Network configuration

Add to /etc/hosts after the local definition the following hosts: [root@bf30 etc]# vi hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost 194.36.3.43 bf30.hep.man.ac.uk 194.36.3.222 cap.hep.man.ac.uk cap 194.36.3.43 bf30.hep.man.ac.uk bf30 194.36.3.44 bf31.hep.man.ac.uk bf31 194.36.3.45 bf32.hep.man.ac.uk bf32 194.36.3.46 bf33.hep.man.ac.uk bf33 194.36.3.47 bf34.hep.man.ac.uk bf34 194.36.3.48 bf35.hep.man.ac.uk bf35 194.36.3.49 bf36.hep.man.ac.uk bf36 194.36.3.50 bf37.hep.man.ac.uk bf37 194.36.3.51 bf38.hep.man.ac.uk bf38 194.36.3.52 bf39.hep.man.ac.uk bf39 194.36.3.53 bf40.hep.man.ac.uk bf40

CAP NFS auto mounting:

At cap.hep.man.ac.uk, as root user: [root@cap root]# cat /etc/exports # /work bf*.hep.man.ac.uk(rw,no_root_squash) /work bf*.tier2.hep.man.ac.uk(rw,no_root_squash) /work bohr*.tier2.hep.man.ac.uk(rw,no_root_squash) /work *.hep.man.ac.uk(rw,no_root_squash) [root@cap root]# exportfs -a [root@cap root]# exportfs /work bf*.hep.man.ac.uk /work bf*.tier2.hep.man.ac.uk /work bohr*.tier2.hep.man.ac.uk /work *.hep.man.ac.uk -at each network computer: [root@onegrid ~]# ssh -l root bfXX.hep.man.ac.uk [root@bf30 root]# cd /etc [root@bf30 etc]# vi auto.master # # $Id: auto.master,v 1.3 2003/09/29 08:22:35 raven Exp $ # # Sample auto.master file # This is an automounter map and it has the following format # key [ -mount-options-separated-by-comma ] location # For details of the format look at autofs(5). #/misc /etc/auto.misc --timeout=60 #/misc /etc/auto.misc #/net /etc/auto.net /exp_software /etc/auto.experiment -rw,intr,noquota,nosuid,noatime [root@bf30 etc]# vi auto.experiment work cap.hep.man.ac.uk:/work babardisk1 -rw,nosuid,noexec,rsize=65536,wsize=65536 celeste.hep.man.ac.uk:/babardisk1 babardisk2 -rw,nosuid,noexec,rsize=65536,wsize=65536 cornelius.hep.man.ac.uk:/babardisk2 babardisk3 -rw,nosuid,noexec,rsize=65536,wsize=65536 arthur.hep.man.ac.uk:/mnt/babardisk3 [root@bf30 etc]# mkdir /exp_software [root@bf30 etc]# ln -s /exp_software /nfs [root@bf30 etc]# /etc/rc.d/init.d/autofs restart [root@bf30 etc]# /sbin/service nfs restart Shutting down NFS mountd: [ OK ] Shutting down NFS daemon: [ OK ] Shutting down NFS quotas: [ OK ] Shutting down NFS services: [ OK ] Starting NFS services: [ OK ] Starting NFS quotas: [ OK ] Starting NFS daemon: [ OK ] Starting NFS mountd: [ OK ] [root@bf30 root]# ls /nfs/work 00_index.txt babar lost+found newstore SE0 store users This mount point will be used by kanaccess.conf to assembly data physical file name for babar experiment. The direct mounting of CAP can be done as: [root@bf30 root]# mkdir /nfs [root@bf30 root]# mkdir /nfs/work [root@bf30 root]# mount cap:/work /nfs/work [root@bf30 root]# ls /nfs/work 00_index.txt babar lost+found newstore SE0 store users

4. Adapt Scientific Linux to LCG

This is necessary to avoid conflict with message passing packages in LCG: [root@bf30 root]# rpm -e lam-6.5.9-1

5. Java installation from RPM

Verify if java is installed: [root@bf30 usr]# ls /usr/java j2sdk1.4.2_10 [root@bf30 root]# ls /usr/java/j2sdk1.4.2_10 bin demo jre LICENSE README src.zip COPYRIGHT include lib man README.html THIRDPARTYLICENSEREADME.txt and get the address: /usr/java/j2sdk1.4.2_10 If java is not installed: http://java.sun.com/j2se/1.4.2/download.html Download J2SE SDK 1.4.2_10 accept RPM in self extracting file: j2sdk-1_4_2_10-linux-i586-rpm.bin The file was saved in: cap.hep.man.ac.uk:/nfs/work/users/jamwer/GRID [root@onegrid backup]# scp j2sdk-1_4_2_10-linux-i586-rpm.bin root@cap.hep.man.ac.uk:/nfs/work/users/jamwer/GRID root@cap.hep.man.ac.uk's password: j2sdk-1_4_2_10-linux-i586-rpm.bin 100% 34MB 4.2MB/s 00:08 [root@bf30 root]# cd /nfs/teste/users/jamwer/GRID [root@bf30 GRID]# chmod +x j2sdk-1_4_2_10-linux-i586-rpm.bin [root@bf30 GRID]# ls -l total 34508 -rwxr-xr-x 1 root root 35292766 Nov 1 14:48 j2sdk-1_4_2_10-linux-i586-rpm.bin root@bf30 GRID]# cd /usr/local [root@bf30 local]# ls bin etc games include lib libexec sbin share src [root@bf30 local]# mkdir java [root@bf30 local]# cd java [root@bf30 java]# /nfs/teste/users/jamwer/GRID/j2sdk-1_4_2_10-linux-i586-rpm.bin Sun Microsystems, Inc. Binary Code License Agreement for the JAVATM 2 SOFTWARE DEVELOPMENT KIT (J2SDK), STANDARD EDITION, VERSION 1.4.2_X ... (LFI#129530/Form ID#011801) Do you agree to the above license terms? [yes or no] yes Unpacking... Checksumming... 0 0 Extracting... UnZipSFX 5.40 of 28 November 1998, by Info-ZIP (Zip-Bugs@lists.wku.edu). inflating: j2sdk-1_4_2_10-linux-i586.rpm Done. [root@bf30 java]# ls j2sdk-1_4_2_10-linux-i586.rpm [root@bf30 root]# cd /usr/local/java [root@bf30 java]# ls j2sdk-1_4_2_10-linux-i586.rpm [root@bf30 java]# mv j2sdk-1_4_2_10-linux-i586.rpm /nfs/work/users/jamwer/GRID/ [root@bf30 java]# cd .. [root@bf30 local]# rmdir java [root@bf30 java]# rpm -ivh /nfs/work/users/jamwer/GRID/j2sdk-1_4_2_10-linux-i586.rpm Preparing... ########################################### [100%] 1:j2sdk ########################################### [100%] [root@bf30 root]# ls /usr/java/j2sdk1.4.2_10 bin demo jre LICENSE README src.zip COPYRIGHT include lib man README.html THIRDPARTYLICENSEREADME.txt Set JVM address (verify the correct version): vi /etc/profile export JAVA_LOCATION=/usr/java/j2sdk1.4.2_10 export JAVA_LIBDIR=$JAVA_LOCATION/lib export JNI_LIBDIR=$JAVA_LIBDIR export JVM_LIBDIR=$JAVA_LIBDIR PATH=$JAVA_LOCATION/bin:$PATH [root@bf30 root]# . /etc/profile [root@bf30 root]# java Usage: java [-options] class [args...] (to execute a class) ...

6. Node synchronisation

[root@bf30 yaim]# ping ntp2a.mcc.ac.uk PING maverick.mcc.ac.uk (130.88.202.49) 56(84) bytes of data. 64 bytes from maverick.mcc.ac.uk (130.88.202.49): icmp_seq=0 ttl=63 time=0.419 ms [root@bf30 etc]# mv ntp.conf ntp.conf.orig [root@bf30 etc]# vi ntp.conf [root@bf30 etc]# cat ntp.conf server ntp2a.mcc.ac.uk server ntp2b.mcc.ac.uk server ntp2c.mcc.ac.uk server ntp2d.mcc.ac.uk [root@bf30 etc]# cat /etc/ntp/step-tickers ntp2a.mcc.ac.uk ntp2b.mcc.ac.uk ntp2c.mcc.ac.uk ntp2d.mcc.ac.uk Start ntp: [root@bf30 etc]# ntpdate ntp2a.mcc.ac.uk 1 Nov 16:01:10 ntpdate[2426]: adjust time server 130.88.202.49 offset -0.370997 sec [root@bf30 etc]# service ntpd start ntpd: Synchronizing with time server: [ OK ] Starting ntpd: [ OK ] [root@bf30 etc]# chkconfig ntpd on

7. Configuration tool:YAIM

[root@bf30 teste]# cd /nfs/work/users/jamwer/GRID [root@bf30 GRID]# rpm -ivh lcg-yaim-2.6.0-9.noarch.rpm Preparing... ########################################### [100%] 1:lcg-yaim ########################################### [100%] [root@bf30 GRID]# cd /opt/lcg [root@bf30 lcg]# ls bin yaim [root@bf30 lcg]# ls bin configure_node install_node [root@bf30 lcg]# cd /opt/lcg/yaim [root@bf30 yaim]# ls examples functions scripts

8. Site configuration file

[root@bf30 yaim]# vi wn-list.conf bf33.hep.man.ac.uk bf34.hep.man.ac.uk bf35.hep.man.ac.uk bf36.hep.man.ac.uk bf37.hep.man.ac.uk bf38.hep.man.ac.uk [root@bf30 yaim]# vi users.conf 24014:babar001:2739:babar:babar:: 24022:babar002:2739:babar:babar:: 24023:babar003:2739:babar:babar:: 24024:babar004:2739:babar:babar:: 24028:babar005:2739:babar:babar:: 24029:babar006:2739:babar:babar:: 24030:babar007:2739:babar:babar:: 24031:babar008:2739:babar:babar:: 24032:babar009:2739:babar:babar:: 24033:babar010:2739:babar:babar:: 24034:babar011:2739:babar:babar:: 24038:babar012:2739:babar:babar:: 24044:babar013:2739:babar:babar:: 24045:babar014:2739:babar:babar:: 24046:babar015:2739:babar:babar:: 24047:babar016:2739:babar:babar:: 24048:babar017:2739:babar:babar:: 24049:babar018:2739:babar:babar:: 24050:babar019:2739:babar:babar:: 24051:babar020:2739:babar:babar:: [root@bf30 yaim]# vi /nfs/work/users/jamwer/GRID/JSTestBed.def Verify the file /nfs/work/users/jamwer/GRID/JSTestbed.def for complete implementation. The information necessary to configure the complete site is: Latitude and longitude: www.infoplease.com/ipa/A0001769.html MY_DOMAIN=hep.man.ac.uk CE_HOST=bf32.$MY_DOMAIN SE_HOST=bf40.$MY_DOMAIN RB_HOST=bf31.$MY_DOMAIN PX_HOST=bf30.$MY_DOMAIN #PX_HOST=lcgrbp01.gridpp.rl.ac.uk BDII_HOST=bf30.$MY_DOMAIN #BDII_HOST=lcgbdii02.gridpp.rl.ac.uk MON_HOST=bf30.$MY_DOMAIN #MON_HOST=bohr0002.tier2.hep.man.ac.uk REG_HOST=lcgic01.gridpp.rl.ac.uk # there is only 1 central registry for now WN_LIST=/opt/lcg/yaim/wn-list.conf USERS_CONF=/opt/lcg/yaim/users.conf FUNCTIONS_DIR=/opt/lcg/yaim/functions JAVA_LOCATION="/usr/java/j2sdk1.4.2_10" MYSQL_PASSWORD=carnaval SITE_NAME=JSTestBed SITE_LOC="Manchester, UK" SITE_LAT=53.30 SITE_LONG=2.15 SITE_WEB="http://www.hep.man.ac.uk" SITE_TIER="TIER 3" SITE_SUPPORT_SITE="tier2.hep.man.ac.uk" CE_CLOSE_SE="SE1" CE_CLOSE_SE1_HOST=$SE_HOST CE_CLOSE_SE1_ACCESS_POINT=/storage CE_CLOSE_SE2_HOST=another-se.$MY_DOMAIN CE_CLOSE_SE2_ACCESS_POINT=/somewhere BDII_HTTP_URL="http://bf30.hep.man.ac.uk/lcg-testbed.conf" #BDII_HTTP_URL="http://grid-deployment.web.cern.ch/grid-deployment/gis/lcg2-bdii/dteam/lcg2-all-sites.conf" VO_SW_DIR=/nfs/work/lcg VO_BABAR_SW_DIR=$VO_SW_DIR/babar VO_BABAR_DEFAULT_SE=$SE_HOST VO_BABAR_STORAGE_DIR=$CE_CLOSE_SE1_ACCESS_POINT/babar VO_BABAR_QUEUES="babar" VO_BABAR_USERS=ldap://babar-vo.gridpp.ac.uk/ou=babar,dc=gridpp,dc=ac,dc=uk The YAIM entries to configure the VOMS server are: site-info.def: VO_BABAR_VOMS_SERVERS="vomss://voms.gridpp.ac.uk:8443/voms/babar?/babar/" VO_BABAR_VOMSES="'babar voms.gridpp.ac.uk 15002 /C=UK/O=eScience/OU=Manchester/L=HEP/CN=voms.gridpp.ac.uk/Email=hostmaster@hep.man.ac.uk babar'" groups.conf: "/VO=babar/GROUP=/babar":::: "/VO=babar/GROUP=/babar/ROLE=lcgadmin":::sgm: