Notes about Installation and Configuration of EMI-3 (SL6) LFC server

These notes are provided by site admins on a best effort base as a contribution to the IGI communities and MUST not be considered as a subsitute of the Official IGI documentation.
This document is addressed to site administrators responsible for middleware installation and configuration.
The goal of this page is to provide some hints and examples on how to install and configure an IGI LFC server based on EMI/UMD middleware.

NB: LFC service is a CORE service, it should not be installed at Resource Center level. If you want to provide an official instances which will be added to the IGI endpoint, please contact igi-noc@lists.italiangrid.it

References

Recommendations and Sceanrios

A full virtualized machine based on KVM has been used in the following notes. There are four installation scenarios that can be follow.

Scenario 1 (all in one)

LFC frontend and backend are installed on the same server. The service needs at least two cores, 4 GB RAM, 10 GB disk space (disk space highly depends on the number of VOs that will be use the instance).

Scenario 2 ( n frontends and one backend)

In this scenario the frontend and the backend are installed on separate servers. The number of frontends could be higher than one for load balancing and high availability purpose.

Scenario 3 ( n forntends and high availability backend)

In this scenario the frontends and backends are installaed on separate server and the backend, based on mysql, is configured in high availability mode (master-master replica).

Scenario 4 ( n forntends and high availability mysql using pacemaker and corosync)

In this scenario the frontends and backends are installaed on separate server and the backend, based on mysql, is configured in high availability mode (master-slave) through pacemaker and corosync.

Service Installation

In this note, scenario 4 has been followed.

HOSTNAME
lfcserver01
lfcserver02

O.S. and Repos

Starts from a fresh installation of Scientific Linux 5.x (x86_64).

# cat /etc/redhat-release 
Scientific Linux release 6.4 (Carbon)

Install the additional repositories: EPEL, Certification Authority, EMI-3

# cd /etc/yum.repos.d/
# rpm -ivh http://mirror.switch.ch/ftp/mirror/epel/6/x86_64/epel-release-6-8.noarch.rpm
# wget http://repo-pd.italiangrid.it/mrepo/repos/egi-trustanchors.repo
# rpm -ivh  http://emisoft.web.cern.ch/emisoft/dist/EMI/3/sl6/x86_64/base/emi-release-3.0.0-2.el6.noarch.rpm
# yum install yum-priorities yum-protectbase

Be sure that SELINUX is disabled (or permissive). Details on how to disable SELINUX are here:

# getenforce 
Disabled

GlusterFS, pacemaker and corosync

The idea is:

to use glusterfs to setup a volume to be mounted under /var/lib/mysql;
to use pacemaker and corosync to managed primary/slave configuration for mysqld.

GlusterFS volume setup

GlusterFS configuration is not covered here.

gluster volume info
 
Volume Name: volume-mysql
Type: Replicate
Volume ID: 5e8e3ed1-ab16-45cd-ac07-7fa61b7056f2
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: lfcserver01.cnaf.infn.it:/glusterfs/br-mysql
Brick2: lfcserver02.cnaf.infn.it:/glusterfs/br-mysql
Options Reconfigured:
auth.allow: 131.154.101.61,131.154.101.156

The volume is mounted under /var/lib/mysql on both lfc nodes.

Update host and perform the installation of middleware package(s)

# yum clean all
Loaded plugins: downloadonly, kernel-module, priorities, protect-packages, protectbase, security, verify, versionlock
Cleaning up Everything

# yum install -y ca-policy-egi-core emi-lfc_mysql lfc-dli
[...]

lfc-dll rpm must be installed due to a bug (GGUS 95081

MySQL server installation and configuration

NB: If you want to install/run a MySQL server in the same host as the LFC, add mysql-server to the yum install command above.

# yum install -y mysql-server
Loaded plugins: downloadonly, kernel-module, priorities, protect-packages, protectbase, security, verify, versionlock
[...]                                                                                                                                                                   
Complete!

Configure MySQL to start at boot time

# chkconfig mysqld on

Start MySQL server

# /etc/init.d/mysqld start
Initializing MySQL database:  Installing MySQL system tables...
OK
Filling help tables...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:
/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h lfcserver04.cnaf.infn.it password 'new-password'

Alternatively you can run:
/usr/bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:
cd /usr ; /usr/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl
cd mysql-test ; perl mysql-test-run.pl

Please report any problems with the /usr/bin/mysqlbug script!

The latest information about MySQL is available on the web at
http://www.mysql.com
Support MySQL by buying support/licenses at http://shop.mysql.com
                                                           [  OK  ]
Starting MySQL:                                            [  OK  ]

Configure MySQL service using /usr/bin/mysql_secure_installation

# /usr/bin/mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MySQL to secure it, we'll need the current
password for the root user.  If you've just installed MySQL, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none): 
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MySQL
root user without the proper authorisation.

Set root password? [Y/n] Y
New password: 
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!

By default, a MySQL installation has an anonymous user, allowing anyone
to log into MySQL without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] Y
 ... Success!

By default, MySQL comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MySQL
installation should now be secure.

Thanks for using MySQL!

LFC Service configuration

LFC YAIM variables are reported here

Start from example:

# cp -vr /opt/glite/yaim/examples/siteinfo .
`/opt/glite/yaim/examples/siteinfo' -> `./siteinfo'
`/opt/glite/yaim/examples/siteinfo/services' -> `./siteinfo/services'
`/opt/glite/yaim/examples/siteinfo/services/lfc_oracle' -> `./siteinfo/services/lfc_oracle'
`/opt/glite/yaim/examples/siteinfo/services/glite-bdii_site' -> `./siteinfo/services/glite-bdii_site'
`/opt/glite/yaim/examples/siteinfo/services/lfc_mysql' -> `./siteinfo/services/lfc_mysql'
`/opt/glite/yaim/examples/siteinfo/site-info.def' -> `./siteinfo/site-info.def'

# rm /root/siteinfo/services/lfc_oracle /root/siteinfo/services/glite-bdii_site

site-info.def

The configuration file for this service is really basic:

# cat /root/siteinfo/site-info.def 
################################
# Site configuration variables #
################################
SITE_NAME=IGI-BOLOGNA
SITE_EMAIL="grid-operations@lists.cnaf.infn.it"
SITE_LAT=44.4948
SITE_LONG=11.3417
# Download the last files (groups.conf and users.conf) at https://wiki.italiangrid.it/twiki/bin/view/IGIRelease/IgiEmi#IGI_YAIM_configuration_files
GROUPS_CONF=/root/igi-siteinfo/groups.conf
USERS_CONF=/root/igi-siteinfo/users.conf

MYSQL_PASSWORD=XXXXXXXXX
VOS="argo ams02.cern.ch babar bio cdf compchem comput-er.it superbvo.org enea glast.org gridit inaf infngrid libi pacs.infn.it pamela planck theophys tps.in
fn.it virgo euchina enmr.eu euindia cyclops compassit ops dteam infngrid"

emi_lfc_mysql

The configuration file for this service is really basic:

# cat /root/siteinfo/services/emi_lfc_mysql
# LFC hostname
LFC_HOST=lfcserver01.cnaf.infn.it

LFC_HOST_ALIAS="lfcserver.cnaf.infn.it"

LFC_DB_HOST=gridstore1.cnaf.infn.it
LFC_DB=cns_db
LFC_DB_PASSWORD="XXXXXXXXX"

LFC_LOCAL="ops infngrid dteam"
LFC_CENTRAL="argo ams02.cern.ch babar bio cdf compchem comput-er.it superbvo.org enea glast.org gridit inaf infngrid libi pacs.infn.it pamela planck theophys tps.in
fn.it virgo euchina enmr.eu euindia cyclops compassit"

vo.d directory

Create the directory siteinfo/vo.d and fill it with a file for each supported VO. You can download them from HERE

and here an example for some VOs. Information about the several VOs are available at the CENTRAL OPERATIONS PORTAL

# ls vo.d/
ams02.cern.ch  bio                       compassit     cyclops                                enea             euchina    gridit    libi            pacs.infn.it  superbvo.org  virgo
argo           cdf                       compchem      d4science.research-infrastructures.eu  enmr.eu          euindia    inaf      lights.infn.it  pamela        theophys
babar          climate-g.vo.eu-egee.org  comput-er.it  dteam                                  eticsproject.eu  glast.org  infngrid  ops             planck        tps.infn.it

yaim check

#  chmod -R 600 /root/siteinfo

# # /opt/glite/yaim/bin/yaim  -v -s /root/igi-siteinfo/site-info.def -n emi_lfc_mysql
   INFO: Configuring HOST: lfcserver01.cnaf.infn.it
   INFO: Using site configuration file: /root/igi-siteinfo/site-info.def
   INFO: Sourcing service specific configuration file: /root/igi-siteinfo/services/emi_lfc_mysql
   INFO: 
         ###################################################################
         
         .             /'.-. ')
         .     yA,-"-,( ,m,:/ )   .oo.     oo    o      ooo  o.     .oo
         .    /      .-Y a  a Y-.     8. .8'    8'8.     8    8b   d'8
         .   /           ~ ~ /         8'    .8oo88.     8    8  8'  8
         . (_/         '===='          8    .8'     8.   8    8  Y   8
         .   Y,-''-,Yy,-.,/           o8o  o8o    o88o  o8o  o8o    o8o
         .    I_))_) I_))_)
         
         
         current working directory: /root/igi-siteinfo
         site-info.def date: Aug 9 11:08 /root/igi-siteinfo/site-info.def
         yaim command: -v -s /root/igi-siteinfo/site-info.def -n emi_lfc_mysql
         log file: /opt/glite/yaim/bin/../log/yaimlog
         Thu Aug  9 11:09:05 CEST 2012 : /opt/glite/yaim/bin/yaim
         
         Installed YAIM versions:
         glite-yaim-bdii 4.3.9-1
         glite-yaim-core 5.1.0-1
         glite-yaim-lfc 4.2.4-1
[...]
   INFO: YAIM terminated succesfully.

yaim config

Please use the debug flag ( "-d 6") to configure the services in order to have detailed information. For your convenience yo can save all the configuration information in a log file you can look at any time, separated from the yaimlog defulat one.

#  /opt/glite/yaim/bin/yaim  -d 6 -c -s /root/igi-siteinfo/site-info.def -n emi_lfc_mysql
   DEBUG: Checking siteinfo dir is not world readable
   DEBUG: Checking site-info.def is syntactically correct
   INFO: Configuring HOST: lfcserver01.cnaf.infn.it
   DEBUG: Sourcing /opt/glite/yaim/bin/../defaults/site-info.pre
   DEBUG: Sourcing /opt/glite/yaim/bin/../defaults/emi_lfc_mysql.pre
   INFO: Using site configuration file: /root/igi-siteinfo/site-info.def
   DEBUG: Sourcing site-info.def file: /root/igi-siteinfo/site-info.def
   INFO: Sourcing service specific configuration file: /root/igi-siteinfo/services/emi_lfc_mysql
   DEBUG: Sourcing /opt/glite/yaim/bin/../defaults/site-info.post
   DEBUG: Sourcing /opt/glite/yaim/bin/../defaults/mapping
   INFO: 
         ###################################################################
         
         .             /'.-. ')
         .     yA,-"-,( ,m,:/ )   .oo.     oo    o      ooo  o.     .oo
         .    /      .-Y a  a Y-.     8. .8'    8'8.     8    8b   d'8
         .   /           ~ ~ /         8'    .8oo88.     8    8  8'  8
         . (_/         '===='          8    .8'     8.   8    8  Y   8
         .   Y,-''-,Yy,-.,/           o8o  o8o    o88o  o8o  o8o    o8o
         .    I_))_) I_))_)
         
         
         current working directory: /root/igi-siteinfo
[...]
   INFO: Configuration Complete.                                               [  OK  ]
   INFO: YAIM terminated succesfully.

Service checks

From a User Interfaces, try the following commands (replace lfcserver01.cnaf.infn.it with your LFC frontend hostname):

Create a valid proxy for a VO supported by the LFC server, e.g. voms-proxy-init --voms dteam
Export the following variables:

export LCG_CATALOG_TYPE=lfc
export LFC_HOST=lfcserver01.cnaf.infn.it
export LFC_HOME=/grid/dteam

Ping the service (take a look at /var/log/lfc/log on the LFC server)

$  lfc-ping -h lfcserver01.cnaf.infn.it
1.8.3.1-0

Use lcg-cr command to copy & register (and the delete) a file on LFC, take a look at /var/log/lfc/log on the frontend

# lcg-cr -v --vo dteam -d se-srm-00.to.infn.it -P host-wms-list-no-checksum.txt -l lfn:host-wms-list-no-checksum.txt file:///home/TIER1/veronesi/host-wms-list.txt
# lcg-del -a --vo dteam  lfn:/grid/dteam/host-wms-list-no-checksum.txt