References:General structure. Database structure Using a Local Database Data Import and Bookkeeping at Remote Sites Setting up BaBar Book Keeping database for local use Relational Database Query Classes
The mapping of files between the file server Sun1 and the nfs systemn is: Sun1 bfb.tier2 u01 /nfs/babar01 u02 /nfs/babar02 u03 /nfs/babar03 u04 /nfs/babar04 u05 /nfs/babar05 u06 /nfs/babar06
this procedure DOES NOT work due a problems at SLAC, RAL, ... when running BbkMirror Some of the errors are:
bfactory@pc91>./bin/Linux24RH72_i386_gcc2953/BbkDbMirror -ss ral -su anyuser -ts local -tu bfactory
bbkr14.bbk_streams number of entries to INSERT: 128 and maxId= bbkr14.bbk_runs number of entries to INSERT: 20000 and maxId= bbkr14.bbk_runs number of entries to INSERT: 20000 and maxId=20037 bbkr14.bbk_runs number of entries to INSERT: 20000 and maxId=40037 bbkr14.bbk_runs number of entries to INSERT: 20000 and maxId=60037 ... bbkr14.bbk_dsetorun number of entries to INSERT: 20000 and maxId=4129707 bbkr14.bbk_dsetorun number of entries to INSERT: 20000 and maxId=4149707 bbkr14.bbk_dsetorun number of entries to INSERT: 20000 and maxId=4169707 bbkr14.bbk_dsetorun number of entries to INSERT: 0 and maxId=4189707 bbkr14.bbk_dataset number of entries to INSERT: 0 and maxId= bbkr14.bbk_sitecontent number of entries to INSERT: 0 and maxId= bbkr14.bbk_sites number of entries to INSERT: 0 and maxId= bbkr14.bbk_aliases number of entries to INSERT: 0 and maxId= bbkr14.bbk_dstodse number of entries to INSERT: 0 and maxId= bbkr14.bbk_releases number of entries to INSERT: 0 and maxId= bbkr14.bbk_files number of entries to INSERT: 0 and maxId= 1088607643-Report-writing time info in /afs/hep.man.ac.uk/g/bfactory/.bbk/mirror/mirrorTimes
The import process ends with all database marked as external. The number of records is much less then the real one. At the end of the mirror the number of entries in bbk_dsetorun was only ~4M which is not the ~32M at slac.
./bin/Linux24RH72_i386_gcc2953/BbkDbMirror -ss slac -su anyuser -ts local -tu bfactory
install_driver(Proxy) failed: Can't locate RPC/PlClient.pm in @INC (@INC contains: /afs/hep.man.ac.uk/g/bfactory/bbk1442a/bin/Linux24RH72_i386_gcc2953 /afs/hep.man.ac.uk/g/bfactory/bbk1442a/include -I/afs/hep.man.ac.uk/g/bfactory/dist/releases/14.4.2a/include /usr/lib/perl5/5.6.1/i386-linux /usr/lib/perl5/5.6.1 /usr/lib/perl5/site_perl/5.6.1/i386-linux /usr/lib/perl5/site_perl/5.6.1 /usr/lib/perl5/site_perl/5.6.0 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.6.1/i386-linux /usr/lib/perl5/vendor_perl/5.6.1 /usr/lib/perl5/vendor_perl . /afs/hep.man.ac.uk/g/bfactory/bbk1442a/include /afs/hep.man.ac.uk/g/bfactory/dist/releases/14.4.2a/include /afs/hep.man.ac.uk/g/bfactory/lib.shared/perl/i386-linux /afs/hep.man.ac.uk/g/bfactory/lib.shared/perl) at /afs/hep.man.ac.uk/g/bfactory/lib.shared/perl/i386-linux/DBD/Proxy.pm line 28. BEGIN failed--compilation aborted at /afs/hep.man.ac.uk/g/bfactory/lib.shared/perl/i386-linux/DBD/Proxy.pm line 28. Compilation failed in require at (eval 3) line 3. Perhaps a module that DBD::Proxy requires hasn't been fully installed at BbkTools/BbkConnect.pm line 93
This is an example - always from slac, but I reduced to only one table - of what is happening:
aforti@tersk05>BbkDbMirror -ss slac -su anyuser -ts local -tu bfactory --trgtable files
bbkr14.bbk_files number of entries to INSERT: 20000 and maxId=40004 bbkr14.bbk_files number of entries to INSERT: 0 and maxId=60053 1089029503-Report-writing time info in /u/ec/aforti/.bbk/mirror/mirrorTimes 13.16u 2.66s 48:51.56 rss+sh+dat+stack +io pf+w
It copies certain amount of entries (but not all of them) and then it cleanly exits as if it had finished.
Reference about the mirror process at RAL: http://babar-hn.slac.stanford.edu:5090/HyperNews/get/Bookkeeping/670.html
Reference about similar error: http://www.slac.stanford.edu/BFROOT/www/Computing/Distributed/Bookkeeping/Do cumentation/updates.html
mysqldump --opt -u root bbkr14 >/u06/bbkr14_load.sql &
Bookkeeping Configuration file was stored at: ~/.bbk/sites/man/connect
vi ~/.bbk/sites/man/connect
type=MySQL connection=Network server=sun1.hep.man.ac.uk database=bbkr14 driver=dbi:mysql:bbkr14:sun1.hep.man.ac.uk
Loading the database with RAL dump at Manchester:
ssh -l root sun1 cd /usr/local/mysql
There are two options: load the database with initial values when the database was empty or after download of tau experiment data and monte carlo.
./bin/mysql -u root < /u06/bbkr14.sql <-- Loading db with database empty ./bin/mysql -u root < /u06/bbkr14_load.sql <-- Loading db with TAU data in the database
The load can be monitored using the following command:
./bin/mysqladmin -i 2 -u root status processlist <- executando a cada 2 segundos mysqladmin -u root processlist status <- Executando so uma vez
+------+----------+-------------------------------+--------+---------+------+--------------+------------------------------------------------------------------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | +------+----------+-------------------------------+--------+---------+------+--------------+------------------------------------------------------------------------------------------------------+ | 1554 | bfactory | bfb.tier2.hep.man.ac.uk:34276 | bbkr14 | Query | 314 | Sending data | SELECT bbkr14.bbk_dsentities.id AS dse_id,bbkr14.bbk_dstodse.link_status,bbkr14.bbk_runs.run,bbkr14. | | 1555 | root | localhost | | Query | 0 | | show processlist | +------+----------+-------------------------------+--------+---------+------+--------------+------------------------------------------------------------------------------------------------------+ Uptime: 1646329 Threads: 2 Questions: 15068 Slow queries: 40 Opens: 115 Flush tables: 1 Open tables: 46 Queries per second avg: 0.009
To kill a process in the database:
mysqladmin -u root kill 1554 <- killing process 1554 above
Reestarting Bbk database, if some mistake was done, needs to reset columns that especify local data availability:
These commands should be used only when the database is loaded with empty schema.
ssh -l root sun1
bash
/usr/local/mysql/bin/mysql -u root
mysql> use bbkr14;
mysql> update data_files set status=2;
Query OK, 5735 rows affected (0.71 sec)
Rows matched: 211324 Changed: 5735 Warnings: 0
mysql> update bbk_dsentities set is_local=0;
Query OK, 129509 rows affected (17.25 sec)
Rows matched: 129509 Changed: 129509 Warnings: 0
mysql> exit
exit
exit
exit
Connection to sun1 closed.
Download process requires twice the password for each file.
To overcame this security check, it is necessary to create a public/private key
without inform the passfrase (just press enter when required) and download the
public key at bbr-xfer01. See
http://noether.vassar.edu/~myers/help/PublicKey.html
ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/jamwer/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/jamwer/.ssh/id_dsa.
Your public key has been saved in /home/jamwer/.ssh/id_dsa.pub.
The key fingerprint is:
1a:8e:89:c7:ec:c3:63:4b:de:5a:20:bd:e6:f3:0c:6f jamwer@bfb.tier2.hep.man.ac.uk
cat id_dsa.pub
ssh-dss AAAAB3NzaC1kc3MAAACBALb0t9jzmVNizjH4K0Ul0d6F1+VRqLeVpeJ/6jDe0T1MGz0AeejyySEKUWGJ6uqS61tIfDK9H7FNDbdHzkSqgXhQYRdsu8R/vziuYyEGMibhwYWF+cc6Ud7bAzPrVVkAm8ys7KumCRCCS0Mh7RU0fz9feDHZ74jzqoV/i6dlNuRDAAAAFQDw4R8CVDtKgMo9xZmVJNQUeTH0nQAAAIEAjhceteteob/YsC6+m6AwIKX4wtQs6RjhnpgxaZC0W+JcCKzP4p7w0K35KVV1m7xKLJoVjejjfMcwq0REE13/n2Iql1g0hy6l8QblUXgV8n/ms5nEcEOujfOpQunAG+5SA1lTbzvChBRGleXtZhDyEMdOixxG2lEMDL4yDCTaBYoAAACAAryPHs8f35qSHqH43omPrOb6iC2ssAn/ViEut4MpItYcqyMkBHUYZPeqT0lRFmWLSMnSKqoXTFUG4AH4BPwNxGYTVsapSPOOVYIz/ri8jYbI16frNulgd8K51/2u6vWN9yxae8ySMEq45S0bFygp8yf9Op5v7dF9QhPyPa+1OP8= jamwer@bfb.tier2.hep.man.ac.uk
cd .ssh
ls
id_dsa id_dsa.pub known_hosts
W a r n i n g : Always delete the directory .ssh before any update in the destination machine, or the new key will not be active.
If the passphrase is empty, the computer do not ask for it and the transfer will be without interruption.
The key should be copied in the file /afs/slac.stanford.edu/u/br/jamwer/.ssh/authorized_keys at bbr-xfer01:
mkdir .ssh
cd .ssh
scp jamwer@bfb.tier2.hep.man.ac.uk:/home/jamwer/.ssh/id_dsa.pub authorized_keys
Warning: Permanently added 'bfb.tier2.hep.man.ac.uk,194.36.3.61' (RSA1) to the list of known hosts.
jamwer@bfb.tier2.hep.man.ac.uk's password:
id_dsa.pub 100% 620 4.1KB/s 00:00
ls -l
total 14
lrwxr-xr-x 1 jamwer br 23 Jul 15 02:58 authorized_keys -> .public/authorized_keys
lrwxr-xr-x 1 jamwer br 24 Jul 15 02:58 authorized_keys2 -> .public/authorized_keys2
lrwxr-xr-x 1 jamwer br 18 Jul 15 02:58 id_dsa.pub -> .public/id_dsa.pub
lrwxr-xr-x 1 jamwer br 18 Jul 15 02:58 id_rsa.pub -> .public/id_rsa.pub
lrwxr-xr-x 1 jamwer br 20 Jul 15 02:58 identity.pub -> .public/identity.pub
lrwxr-xr-x 1 jamwer br 19 Jul 15 02:58 known_hosts -> .public/known_hosts
lrwxr-xr-x 1 jamwer br 20 Jul 15 02:58 known_hosts2 -> .public/known_hosts2
ssh -v -l jamwer bbr-xfer01.slac.stanford.edu
ssh bbrdist@bbr-xfer01.slac.stanford.edu
There will be no request of password or passphrase.
|
|
|
Feedback to: jamwer@hep.man.ac.uk |
