This article contains instructions on how to migrate a Operations Automation management node host running RHEL/CentOS 5.x to a host running RHEL/CentOS 6.x.
Important: If the Operations Automation system database is located on a remote host, use this instruction for migration.
Important: The migration procedure does not include transferring old log files, in order to minimize the downtime. It is advised to save the old log files /var/log/poa*
to allow historical root cause investigation of arising issues.
A migration process includes a full backup and restore of the system database. You can estimate total amount of time required for the migration as time of full database backup and restore plus two hours to migrate other components. This time depends on the database size and a number of available CPU cores. For example:
- number of subscriptions: 150K
- number of accounts: 100K
- number of users: 250K
- number of processing cores on MN: 8
- backup time: 1h
- restore time 1h
- total estimated migration time: 3h
Rolling back of migration is fast and it does not require any considerable time.
Changes
Version 1.0.1
- initial release
Version 1.0.2
- migration steps for Windows Asure Pack
Preparation
Make sure that a destination host meets all the requirements listed at Linux Platform Deployment Guide > Installing Linux-based Operations Automation Management Node > Installation Preconditions.
Assign any temporary IP addresses on the destinaton node.
Additional packages will be installed automatically by a migration scripts.
Check that Service Automation Central YUM Repository http://download.automation.odin.com/pa/6.0/RPMS/
is accessible from the destination node.
Names of network interfaces should be the same on the original and on the destination hosts. That is, if the BackNet IP address of a source node is bound to eth0 interface, then on the destination node the eth0 interface should be also configured for the BackNet IP.
Before backing up the Operations Automation management node, make sure that the source node is ready for migration:
- Log in to the source management node as
root
. - Download the following scripts.
Extract content of the archive into
/usr/local/pem/bin
using the following command:tar -xvzf mn_el5-1.0.2.tar.gz -C /usr/local/pem/bin/
Run the pre-check script:
/usr/local/pem/bin/backup.sh -t mn-migration-precheck
If the pre-check failed for some reason, follow recommendations in the script output.
Backup
Important: Before backing up the Operations Automation management node, make sure that there are no custom services (for example, apache, proftpd, bind, and so on) installed on the host. In case there are custom services which you also need to migrate, refer to Migrating Services to Nodes Running Up-to-Date OSes.
To back up all necessary data from the Operations Automation management node do the following:
- Log in to the management node as
root
. - Make sure that there is enough free space to store the backed up data.
Make sure that Operations Automation is not running. You can use the following commands to shut Operations Automation down:
service pemui stop service pem stop service pau stop
Back up necessary data using the following command:
/usr/local/pem/bin/backup.sh -t mn-migration -o /Operations_Automation_backup
After the operation is performed, the backup data will be stored in the
/Operations_Automation_backup
directory.The data contains all the information needed for migration:
- Entire
pem
folder (including tarballs, APS, binaries, etc.) - Full DB dump
- ssh configs
- redis configs
- Configs of OACI, pvps APS
- Entire
- Copy the backup to any safe external storage.
- Write down the host name, the external and internal IP addresses of the host.
- If Windows Azure Pack component is installed on source management node, use Backup section of this instruction to backup additional data.
- Shutdown the original host.
Restore
To finish migration of the Operations Automation management node, you need to restore the backed up data to a new host.
- Log in to the destination host as
root
. - Assign the original host's IP addresses to the new host. Make sure that the IP-addresses are assigned to the same network interfaces, as it was configured on the source host.
- Configure the host name so that it matches the name of the original host. Then make sure that hostname resolution is the same as on the source node.
- If a virtualization technology is used, make sure that it have correct settings (IP addresses, hostname, name servers) for the destination node.
- Copy the backed up data to the host.
Perform the following command from the directory with the backed up data:
cd Operations_Automation_backup ./install.py --migrate
- If Windows Azure Pack component is installed on source management node, use Restore section of this instruction to restore additional data.
Once the data is restored all necessary services will be launched automatically, and you will be able to work with Operations Automation.
NOTE: the restoration process may stop, indicating some failure, and ask for confirmation to proceed, for example:
2016-08-11 09:50:14.876 WARN Package 'parallels-plesk-cloud' failed to re-install due to error: ['/usr/local/pem/bin/ppm_ctl', '-f', '/usr/local/pem/etc/pleskd.props', '-b', '-q', 'reinstall', '1', '1980'] exited with non-zero status 1, stderr: None, stdout: None
Ignore this package and continue migration?
((Y)es/(N)o): Y
Pressing 'N' will stop the whole process, pressing 'Y' will skip the failure and continue. There is no retry attempt, so the failed operation should be re-run manually after the restore ends.
Rollback
To rollback the migration:
- Revert the hostname and IP addresses on the target host.
- Shutdown the target host.
- Start the original host.
Known Issues
Restoration may fail with the following error:
2016-08-10 14:56:34.883 WARN Package 'PrivilegesCommon' failed to re-install due to error: ['/usr/local/pem/bin/ppm_ctl', '-f', '/usr/local/pem/etc/pleskd.props', '-b', '-q', 'reinstall', '1', '38'] exited with non-zero status 1, stderr: None, stdout: None Ignore this package and continue migration? ((Y)es/(N)o): Y
In
/var/log/poa-migrate.log
:package 1734 other-any-any-any-PrivilegesCommon-1.14 ERROR [ xinstallPackage] ExSystem: module_id:'db_service', ex_type_id:'1',Message:'DBService invocation error: ERROR: permission denied for relation dual Query: INSERT INTO privileges(name, area, allow_locked) SELECT ?, ?, ? FROM dual WHERE NOT EXISTS (SELECT 1 FROM privileges WHERE name = ? ) Parameters: [User Included Operations, C, 1, User Included Operations]', deprecated_codes = (1, 1), properties = { empty }
Solution:
switch to another screen
log in to psql with posgresq user:
# psql -U postgres -h `hostname` plesk
assign all permissions for the user plesk to dual table:
plesk=# GRANT ALL PRIVILEGES ON TABLE dual to plesk; GRANT
re-run the package installation:
# /usr/local/pem/bin/ppm_ctl -f /usr/local/pem/etc/pleskd.props -b reinstall 1 38
- return to migration, press 'Y' and proceed with the restore
Restoration may fail during some package resinstallation with the following error in
/var/log/poa-migrate.log
:package 6526 sc-i686-RHEL-6-VPSManager-6.0.7.52 ERROR [ xinstallPackage] ExSystem: module_id:'NativeDepends', ex_type_id:'1',Message:'Destination host 'osscore.home-whs.pl' (#1), IP '10.128.32.10' : Native dependency processing failed; list of dependences: libuuid.i686 . Please either install the packages above, or application templates with them. Error: /usr/local/pem/sbin/pem-pkg-tool:21: DeprecationWarning: The popen2 module is deprecated. Use the subprocess module. import popen2 Missing or unsatisfied requirements: { libuuid.i686 (=any) }.', deprecated_codes = (0, 0), properties = { packages: 'libuuid.i686 ',reason: '/usr/local/pem/sbin/pem-pkg-tool:21: DeprecationWarning: The popen2 module is deprecated. Use the subprocess module. import popen2 Missing or unsatisfied requirements: { libuuid.i686 (=any) }', }
The error indicates some unsatisifed dependencies of the package.
Solution:
switch to another screen
install the highlighted dependency:
# yum install libuuid.i686
re-run the package installation:
# /usr/local/pem/bin/ppm_ctl -f /usr/local/pem/etc/pleskd.props -b reinstall 1 84
- return to migration, press 'Y' and proceed with the restore
The list of known required packages for migration:
libuuid.i686 mysql-libs.i686 boost-serialization.i686 libxslt.i686 apr-util.i686 apr.i686 expat.i686 nmap python-paramiko mod_ssl php php-xml webalizer
Migration fails with the following error:
2017-02-02 02:07:23.259 INFO Restoring PAU Starting pau oom_adj modified Waiting pau to start: ESC[60G[ESC[0;31mFAILEDESC[0;39m] 2017-02-02 02:09:24.733 ERROR ExecFailed service pau start exited with non-zero status 1, stderr: None, stdout: None 2017-02-02 02:09:24.733 ERROR <class 'poaupdater.uUtil.ExecFailed'> service pau start exited with non-zero status 1, stderr: None, stdout: None 2017-02-02 02:09:24.735 ERROR at execCommand (/usr/lib/python2.6/site-packages/poaupdater/uUtil.py :83) in raise ExecFailed(command, status) at restorePAU (/root/pem-backup-all/install_routines.py :582) in uUtil.execCommand('service pau start') at rv (install.py :186) in return fun(*args, **kwds) at doInstall (install.py :316) in phase_fun(install_routines.restorePAU, progress, "Restoring PAU")(installation)
/tmp
partition is mounted withnoexec
option:# mount | grep /dev/mapper/VolGroup-lv_tmp on /tmp type ext4 (rw,noexec,nosuid,nodev)
Solution:
Remount /tmp with exec permission and correct
/etc/fstab
file to correct the same:# mount -o remount,exec /tmp # grep tmp /etc/fstab /dev/mapper/VolGroup-lv_tmp /tmp ext4 rw,nosuid,nodev 1 2
In case the restore operation is re-run, it may fail on the stage of starting PAU service with:
2016-08-09 04:05:30.224 INFO *** Restoring PAU *** 2016-08-09 04:05:30.224 INFO Restoring PAU Starting pau oom_adj modified pau is already runningESC[60G[ESC[0;31mFAILEDESC[0;39m] 2016-08-09 04:05:30.278 ERROR ExecFailed service pau start exited with non-zero status 1, stderr: None, stdout: None 2016-08-09 04:05:30.278 ERROR <class 'poaupdater.uUtil.ExecFailed'> service pau start exited with non-zero status 1, stderr: None, stdout: None 2016-08-09 04:05:30.279 ERROR at execCommand (/usr/lib/python2.6/site-packages/poaupdater/uUtil.py :83) in raise ExecFailed(command, status) at restorePAU (/root/pem-all/install_routines.py :582) in uUtil.execCommand('service pau start') at rv (install.py :186) in return fun(*args, **kwds) at doInstall (install.py :316) in phase_fun(install_routines.restorePAU, progress, "Restoring PAU")(installation) at <module> (install.py :701) in doInstall(config)
Solution:
Stop
pau
service before another run:# service pau stop