Installing Ambra and needed services on multiple boxes using RPMs

This page outlines the process to install the publishing system across multiple machines using the RPMs packages.

See also:

The latest packages can be downloaded from:

Configuration Summary

Ports required:

init start-pristop-priport ssl shutdown-port
ecqs 91 15 ---- 7443 7005
mulgara 92 14 8111 ---- 6789
fedora 93 13 9090 9443 9005
ambra 96 10 8080 (http)
8112,8113 (ehcache)
8009 if webhead 		10443 8005
mysql -- -- 3306 ---- ----
postgresql-- -- 5432 ---- ----

ehcache is set to do peer discovery via multicast at 230.0.0.1 port 4446. If you are running multiple installations on the same network, you'll want to firewall multicast traffic appropriately to avoid cache pollution, or edit ehcache.xml and specify different multicast IP addresses for different installations.

Assumptions:

  • Remove all other Java packages (all gcc-java and gcj related packages)
  • Edit /etc/sysconfig/topaz-common and set up JAVA_HOME. Fox example: 
    export JAVA_HOME=/home/tools/java
  • PATH starts with $JAVA_HOME/bin (no other JDE or JDK is in the path)

Important Directories:

  • /etc/topaz - Configuration files
  • /usr/local/topaz/ - Binaries
  • /usr/local/topaz/bin
    • genkey stores
    • multicast
    • tomcat
  • /var/log/topaz - Log files
  • /var/spool/ambra - Ingest files
  • /var/cache/ambra - Cache files

Common Tools

Ambra tools RPM needs to be installed on all machines running ambra and ambra_cas

  • Install rpms 
    rpm -ivh ambra-tools-${version}.noarch.rpm

non standard utilities

  • (A) Install the appropriate rpm for ImageMagick? version 6.2.6
  • (B) Determine the full path to ImageMagick?'s convert utility.
  • (C) Choose a temporary directory where ImageMagick? may create files.
  • (D) Open file /etc/topaz/ambra.xml and you should see the following elements: 
    <utilities>
  <image-magick>
    <executable-path>/usr/bin/convert</executable-path>
    <temp-directory>/tmp</temp-directory>
  </image-magick>
</utilities>
  • (E) Ensure that the answer from step (B) agrees with the content of element executable-path.
  • (F) Ensure that the answer from step (C) agrees with the content of element temp-directory.

Multicast Setup

The ehcache peer discovery (Ambra and Mulgara servers) is configured to use Multicast.

In a permissive LAN, this should work with no extra set up. However, if you have a restrictive set of firewall rules, you'll need to specifically allow Mulitcast traffic between these servers.

For IPTables, add an instruction such as: 

-A RH-Firewall-1-INPUT -d 224.0.0.0/240.0.0.0 -j ACCEPT

If you are running multiple topaz installations on the same LAN, you'll need to reject Multicast traffic between the two installs.

For IPTables, add rules allowing Multicast traffic from member Mulgara and Ambra servers, and reject all other Multicast traffic on each Mulgara and Ambra server: 

-A RH-Firewall-1-INPUT -s GOOD-IP-ADDRESS -d 224.0.0.0/240.0.0.0 -j ACCEPT
...
-A RH-Firewall-1-INPUT -d 224.0.0.0/240.0.0.0 -j DROP

Also for rpm versions up to and including 0.7, TCP any-to-any should be allowed so that ehcache peers can collaborate.

For rpm versions above 0.7 that include [3082], the TCP port 8111 should be allowed thru (see the commit message for [3082])

Generate Keys

Generate the keys for Ambra, CAS & Fedora. Once those components are installed, you will be copying the generated keys to the machine the component is installed on.

  • Edit dname in /usr/local/topaz/bin/gen_keystores 
  keytool -genkey -alias "$key_alias" -keyalg "$key_alg" -keysize "$key_size" \
          -sigalg "$sig_alg" -dname "CN=$h, O=Public Library of Science, L=San Francisco, S=California, C=US" -validity "$validity" \
          -keystore "$ks" -keypass "$passwd" -storepass "$passwd"
  • Generate certificates for all BUT Mulgara 
/usr/local/topaz/bin/gen_keystores
 Hostname> [FQHN ambra]
 Hostname> [FQHN cas]
 Hostname> [FQHN fedora]
  • examine some keys 
    keytool -list -keystore /keystore.[FQHN ambra].jks -storepass changeit | grep topazproject | more
keytool -list -keystore /keystore.[FQHN cas].jks -storepass changeit | grep topazproject | more
keytool -list -keystore /keystore.[FQHN fedora].jks -storepass changeit | grep topazproject | more

Mulgara Server

  • Install rpms 
    rpm -ivh topaz-mulgara-${version}.noarch.rpm
  • Start Mulgara 
    service topaz-mulgara start

Fedora Server

Install MySQL

  • Install MySQL server from Yum
  • Set MySQL to use innodb, not isam tables. Editing /etc/my.cnf 
    [mysqld]
...
    default-storage-engine=INNODB

Install Fedora

  • Install rpms 
    rpm -ivh topaz-fedora-${version}.noarch.rpm
  • Copy keystore 
    scp <user>@<key generation server>:/tmp/keystore.[fedora FQHN].jks /etc/topaz/keystore.jks
  • Initialize MySQL 
    /usr/local/topaz/topaz-fedora/server/bin/mysql-config.sh [mysql_home] [mysql_dba_user] [mysql_dba_pass] [fedoradba_user] [fedora_dba_pass] [mysql_db_name] [mysql41_flag]
# arg 1 - the location where MySQL is installed (e.g., /usr/local/mysql, or /usr for a centos RPM install of mysql-server)
# arg 2 - the name of the MySQL user with dba privileges (e.g., root)
# arg 3 - the password for the MySQL user with dba privileges (use "" to indicate no password)
# arg 4 - fedoraAdmin
# arg 5 - fedoraAdmin
# arg 6 - fedora21
# arg 7 - y
  • Initialize Fedora (start and stop) 
    service topaz-fedora start
--- wait ---
service topaz-fedora stop
  • Edit /etc/topaz/fedora.fcfg 
    Change
<param name="fedoraServerHost" value="localhost">
to 
<param name="fedoraServerHost" value="[FQHN of fedora server]">
  • Edit /usr/local/topaz/topaz-fedora/data/fedora-xacml-policies/repository-policies/default/deny-apim-if-not-localhost.xml 
    Copy this line:
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">127.0.0.1</AttributeValue>
And create additional lines below it (one per ambra server):
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">[IP ADDRESS OF PLOSONE SERVER]</AttributeValue>
  • Start Fedora 
    service topaz-fedora start

CAS Server

Install PostgreSQL

  • Install pgsql server with Yum (yum install postgresql-server)

  • Start up postgresql to initialize database and config files
  • Edit /var/lib/pgsql/data/postgresql.conf: 
    tcpip_socket = true
  • Set /var/lib/pgsql/data/pg_hba.conf to: (order is important) 
    # TYPE  DATABASE    USER        IP-ADDRESS        IP-MASK           METHOD
host    all         all         127.0.0.1         255.255.255.255   md5
local   all         topaz                                           md5
local   all         all                                             ident   sameuser
  • Create a user 
    su postgres -c 'createuser --createdb --pwprompt --encrypted plosone'
password = 'loot2_debonAir'
  • Create the database 
    su postgres -c 'createdb --owner=plosone plosdb'
  • Update runlevel 
    chkconfig --level 345 postgresql on

Install CAS

  • Install rpms 
    rpm -ivh ambra-cas-${version}.noarch.rpm
  • Copy keystore 
    scp <user>@<key generation server>:/tmp/keystore.[cas FQHN].jks /etc/topaz/keystore.jks
  • generate a csr from keystore and then get a real ssl cert 
    keytool -certreq -keystore /etc/topaz/keystore.jks -file /tmp/csr.req -alias tomcat -keypass changeit
  • import ssl certs 
    keytool -import -alias intermed -keystore keystore.jks -trustcacerts -keypass changeit -file sf_issuing.crt
keytool -import -alias tomcat -keystore /etc/topaz/keystore.jks -keypass changeit -file <the-certificate-file>
  • Edit /etc/topaz/cas.xml and set the following values:
    • config.network.hosts.cas - Change from localhost to your FQHN
    • config.network.hosts.default - Change from localhost to webhead
    • config.network.ports.cas-ssl
    • config.network.ports.default - Webhead port (or blank)
    • config.cas.db.user - USERNAME
    • config.cas.db.password - PASSWORD
    • config.cas.db.url - jdbc:postgresql://localhost/ plosdb
  • Start CAS 
    service ambra-cas start

Ambra

Ambra Install

  • Install rpms 
    rpm -ivh ambra-${version}.noarch.rpm
  • Copy keystore 
    scp <user>@<key generation server>:/tmp/keystore.[ambra FQHN].jks /etc/topaz/keystore.jks
  • Configure the doi-journals entries in /etc/topaz/ambra.xml with appropriate journals and regular expressions for journals you wish to support in the doi resolver
  • Point doi-resolver configuration at mulgara
  • Initialize Ambra (IMPORTANT: To create Mulgara models) 
    service ambra start
--- wait ---
service ambra stop
  • Start Ambra 
    service ambra start

* Check that Ambra is running - make sure port 8009 is up 

/etc/init.d/ambra status

Webhead Installation

Install mod_jk

mod_jk for i386: 

yum --enablerepo=jpackage* install mod_jk-ap20

mod_jk for x64_86: (Download latest source from http://tomcat.apache.org/connectors-doc/) 

tar -xzvf tomcat-connectors-#.#.##-src.tgz
cd tomcat-connectors-#.#.##-src/native
./configure --with-apxs=/usr/sbin/apxs (or where ever the apxs/apxs2 is)
make
su -c 'make install'

A very simple /etc/httpd/conf.d/mod_jk.conf: 

LoadModule      jk_module modules/mod_jk.so
JkWorkersFile   /etc/httpd/conf/workers.properties
JkLogFile       logs/mod_jk.log
JkLogLevel      info

A very simple /etc/httpd/conf/workers.properties: 

worker.list=ajp13
worker.ajp13.port=8009
worker.ajp13.host={ambra FQDN}
worker.ajp13.type=ajp13

Add to webhead vhost: 

JkMount /* ajp13
AddEncodedSlashes On
ErrorDocument 503 503.html

Exclude 503 error doc files and feed from mod_jk 

SetEnvIf Request_URI "^/503.html" no-jk
SetEnvIf Request_URI "^/images/503_hdr_logo.jpg" no-jk
SetEnvIf Request_URI "^/feed/*" no-jk

Remove port 8080 and update FQHN from config files

Static files and RSS

  • Copy 503 error doc files to webhead docroot
  • Install topaz-tools on webhead and build feeds with cron

Forward 7443 to 443

  • Add the following lines to /etc/sysconfig/iptables on CAS server 
          *nat
      :PREROUTING ACCEPT [167:22182]
      :POSTROUTING ACCEPT [1:60]
      :OUTPUT ACCEPT [1:60]
      -A PREROUTING -d <IP ADDR> -i eth1 -p tcp -m tcp --dport 443 -j DNAT --to-destination <IP ADDR>:7443
      -A OUTPUT -d <IP ADDR> -p tcp -m tcp --dport 443 -j DNAT --to-destination <IP ADDR>:7443
      COMMIT

Add additional pairs if cas server communicates on multiple IP addresses (eg, different address for external and internal communication)

  • Restart iptables
  • Remove port 7443 from the following files:
    • ambra: /etc/topaz/ambra.xml
    • cas: /etc/topaz/nonJmxApplicationContext.xml
    • ambra: /etc/topaz/ambra.properties

Post Installation

Create Admin User

This needs to be done only once on a freshly installed system

  1. Register as user "Admin" with valid email address
  2. Go through the email confirmation process
  3. Login as the user and fill out the profile
  4. Go to http://[FQHN:port]/admin/findUser.action
  5. Look up your userid using 'Find User by Email' (should be "info:doi/10.1371/account/<GUID>")
  6. Go to the following URL and enter the user id: http://[FQHN:port]/admin/assignAdminRole.action

Ingest Articles

  1. On Ambra server, copy articles into /var/spool/ambra/ingestion-queue
    • Run /usr/local/topaz/bin/prepare_sip on articles if necessary
  2. chown topaz.topaz /var/spool/ambra/ingestion-queue/*.zip
  3. Go to http://[FQHN:port]/admin/adminTop.action
  4. Select an article from the "Ingestable Articles" section
  5. Click on "Ingest Selected Archives"

Publish Articles

  1. Go to http://[FQHN:port]/admin/adminTop.action
  2. Ingested articles will display in the "Publishable Documents" section. You can view an article before it is published by clicking on the article URL. This article is not publicly viewable.
  3. Check "Publish" for an article
  4. Click on "Publish/Delete Articles"

Upgrade Instructions

Stop Apache web server

Stop apache on webhead 

service httpd stop

Stop Applications

Stop all applications in the following order:

  • Ambra server 
    service ambra stop
  • CAS server 
    service ambra-cas stop
  • Fedora server 
    service topaz-fedora stop
  • Mulgara server 
    service topaz-mulgara stop

Tools upgrade

  • Upgrade tools rpms on all servers that are running Ambra or any of its components. 
    rpm -Uvh ambra-tools-${version}.noarch.rpm

Mulgara upgrade

  • Upgrade Mulgara rpms and start Mulgara 
    rpm -Uvh topaz-mulgara-${version}.noarch.rpm
service topaz-mulgara start

Fedora upgrade

  • Upgrade Fedora rpms 
    rpm -Uvh topaz-fedora-${version}.noarch.rpm
  • Initialize Fedora - necessary to create xacml policies, etc. 
    service topaz-fedora start
--- wait ---
service topaz-fedora stop
  • Edit /usr/local/topaz/topaz-fedora/data/fedora-xacml-policies/repository-policies/default/deny-apim-if-not-localhost.xml 
    Copy this line:
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">127.0.0.1</AttributeValue>
And create additional lines below it (one per ambra server):
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">[IP ADDRESS OF PLOSONE SERVER]</AttributeValue>
  • Fedora server 
    service topaz-fedora start

CAS upgrade

  • Upgrade CAS rpms 
    rpm -Uvh ambra-cas-${version}.noarch.rpm
  • If not using default database password, update /usr/local/topaz/ambra-cas/webapps/cas/web.xml
  • Start CAS 
    service ambra-cas start

Ambra upgrade

  • Upgrade Ambra rpms 
    rpm -Uvh ambra-${version}.noarch.rpm
  • Copy settings from ambra.xml.rpmsave to ambra.xml

  • If on stage servers, rsync webapps
    • [ambra]$> rsync -rt /usr/local/topaz/ambra/webapps/ROOT/* webhead.topazproject.org:/var/www/webhead
  • If on production servers
    • chmod -R g+w /usr/local/topaz/ambra/webapps/ROOT
    • chown -R web.web /usr/local/topaz/ambra/webapps/ROOT
    • chmod -R g+w /var/spool/ambra
    • sync plosfail01 ROOT folder to production
  • Start Ambra 
    service ambra start

Webhead Upgrade

* Make sure to upgrade topaz-tools on the webhead. Currently feeds are run from there.

Start Apache web server

Start apache on webhead 

service httpd start