This article contains instructions on how to migrate a Operations Automation management node host running RHEL/CentOS 5.x to a host running RHEL/CentOS 6.x.

Important: If the Operations Automation system database is located on a remote host, use this instruction for migration.

Important: The migration procedure does not include transferring old log files, in order to minimize the downtime. It is advised to save the old log files /var/log/poa* to allow historical root cause investigation of arising issues.

A migration process includes a full backup and restore of the system database. You can estimate total amount of time required for the migration as time of full database backup and restore plus two hours to migrate other components. This time depends on the database size and a number of available CPU cores. For example:

  • number of subscriptions: 150K
  • number of accounts: 100K
  • number of users: 250K
  • number of processing cores on MN: 8
  • backup time: 1h
  • restore time 1h
  • total estimated migration time: 3h

Rolling back of migration is fast and it does not require any considerable time.

Changes

Version 1.0.1

  • initial release

Version 1.0.2

  • migration steps for Windows Asure Pack

Preparation

Make sure that a destination host meets all the requirements listed at Linux Platform Deployment Guide > Installing Linux-based Operations Automation Management Node > Installation Preconditions.

Assign any temporary IP addresses on the destinaton node.

Additional packages will be installed automatically by a migration scripts.

Check that Service Automation Central YUM Repository http://download.automation.odin.com/pa/6.0/RPMS/ is accessible from the destination node.

Names of network interfaces should be the same on the original and on the destination hosts. That is, if the BackNet IP address of a source node is bound to eth0 interface, then on the destination node the eth0 interface should be also configured for the BackNet IP.

Before backing up the Operations Automation management node, make sure that the source node is ready for migration:

  1. Log in to the source management node as root.
  2. Download the following scripts.
  3. Extract content of the archive into /usr/local/pem/bin using the following command:

    tar -xvzf mn_el5-1.0.2.tar.gz -C /usr/local/pem/bin/
    
  4. Run the pre-check script:

    /usr/local/pem/bin/backup.sh -t mn-migration-precheck
    

If the pre-check failed for some reason, follow recommendations in the script output.

Backup

Important: Before backing up the Operations Automation management node, make sure that there are no custom services (for example, apache, proftpd, bind, and so on) installed on the host. In case there are custom services which you also need to migrate, refer to Migrating Services to Nodes Running Up-to-Date OSes.

To back up all necessary data from the Operations Automation management node do the following:

  1. Log in to the management node as root.
  2. Make sure that there is enough free space to store the backed up data.
  3. Make sure that Operations Automation is not running. You can use the following commands to shut Operations Automation down:

        service pemui stop
        service pem stop
        service pau stop
    
  4. Back up necessary data using the following command:

        /usr/local/pem/bin/backup.sh -t mn-migration -o /Operations_Automation_backup
    

    After the operation is performed, the backup data will be stored in the /Operations_Automation_backup directory.

    The data contains all the information needed for migration:

    • Entire pem folder (including tarballs, APS, binaries, etc.)
    • Full DB dump
    • ssh configs
    • redis configs
    • Configs of OACI, pvps APS
  5. Copy the backup to any safe external storage.
  6. Write down the host name, the external and internal IP addresses of the host.
  7. If Windows Azure Pack component is installed on source management node, use Backup section of this instruction to backup additional data.
  8. Shutdown the original host.

Restore

To finish migration of the Operations Automation management node, you need to restore the backed up data to a new host.

  1. Log in to the destination host as root.
  2. Assign the original host's IP addresses to the new host. Make sure that the IP-addresses are assigned to the same network interfaces, as it was configured on the source host.
  3. Configure the host name so that it matches the name of the original host. Then make sure that hostname resolution is the same as on the source node.
  4. If a virtualization technology is used, make sure that it have correct settings (IP addresses, hostname, name servers) for the destination node.
  5. Copy the backed up data to the host.
  6. Perform the following command from the directory with the backed up data:

        cd Operations_Automation_backup
        ./install.py --migrate
    
  7. If Windows Azure Pack component is installed on source management node, use Restore section of this instruction to restore additional data.

Once the data is restored all necessary services will be launched automatically, and you will be able to work with Operations Automation.

NOTE: the restoration process may stop, indicating some failure, and ask for confirmation to proceed, for example:

2016-08-11 09:50:14.876  WARN Package 'parallels-plesk-cloud' failed to re-install due to error: ['/usr/local/pem/bin/ppm_ctl', '-f', '/usr/local/pem/etc/pleskd.props', '-b', '-q', 'reinstall', '1', '1980'] exited with non-zero status 1, stderr: None, stdout: None
Ignore this package and continue migration?
((Y)es/(N)o): Y

Pressing 'N' will stop the whole process, pressing 'Y' will skip the failure and continue. There is no retry attempt, so the failed operation should be re-run manually after the restore ends.

Rollback

To rollback the migration:

  1. Revert the hostname and IP addresses on the target host.
  2. Shutdown the target host.
  3. Start the original host.

Known Issues

  1. Restoration may fail with the following error:

    2016-08-10 14:56:34.883  WARN Package 'PrivilegesCommon' failed to re-install due to error: ['/usr/local/pem/bin/ppm_ctl', '-f', '/usr/local/pem/etc/pleskd.props', '-b', '-q', 'reinstall', '1', '38'] exited with non-zero status 1, stderr: None, stdout: None
    Ignore this package and continue migration?
    ((Y)es/(N)o): Y
    

    In /var/log/poa-migrate.log:

     package 1734 other-any-any-any-PrivilegesCommon-1.14
    ERROR [ xinstallPackage] ExSystem: module_id:'db_service', ex_type_id:'1',Message:'DBService invocation error: ERROR: permission denied for relation dual Query:  INSERT INTO privileges(name, area, allow_locked) SELECT ?, ?, ? FROM dual WHERE NOT EXISTS (SELECT 1 FROM privileges WHERE name = ? ) Parameters: [User Included Operations, C, 1, User Included Operations]', deprecated_codes = (1, 1), properties = { empty }
    

    Solution:

    • switch to another screen

    • log in to psql with posgresq user:

      # psql -U postgres -h `hostname` plesk
      
    • assign all permissions for the user plesk to dual table:

      plesk=# GRANT ALL PRIVILEGES ON TABLE dual to plesk;
      GRANT
      
    • re-run the package installation:

      # /usr/local/pem/bin/ppm_ctl -f /usr/local/pem/etc/pleskd.props -b reinstall 1 38
      
    • return to migration, press 'Y' and proceed with the restore
  2. Restoration may fail during some package resinstallation with the following error in /var/log/poa-migrate.log:

     package 6526 sc-i686-RHEL-6-VPSManager-6.0.7.52
    ERROR [ xinstallPackage] ExSystem: module_id:'NativeDepends', ex_type_id:'1',Message:'Destination host 'osscore.home-whs.pl' (#1), IP '10.128.32.10' : Native dependency processing failed;                     list of dependences: libuuid.i686  .                    Please either install the packages above,                       or application templates with them.                     Error: /usr/local/pem/sbin/pem-pkg-tool:21: DeprecationWarning: The popen2 module is deprecated.  Use the subprocess module.
      import popen2
    Missing or unsatisfied requirements: { libuuid.i686 (=any) }.', deprecated_codes = (0, 0), properties = { packages: 'libuuid.i686  ',reason: '/usr/local/pem/sbin/pem-pkg-tool:21: DeprecationWarning: The popen2 module is deprecated.  Use the subprocess module.
      import popen2
    Missing or unsatisfied requirements: { libuuid.i686 (=any) }', }
    

    The error indicates some unsatisifed dependencies of the package.

    Solution:

    • switch to another screen

    • install the highlighted dependency:

      # yum install libuuid.i686
      
    • re-run the package installation:

      # /usr/local/pem/bin/ppm_ctl -f /usr/local/pem/etc/pleskd.props -b reinstall 1 84
      
    • return to migration, press 'Y' and proceed with the restore

    The list of known required packages for migration:

    libuuid.i686
    mysql-libs.i686
    boost-serialization.i686
    libxslt.i686
    apr-util.i686
    apr.i686
    expat.i686
    nmap
    python-paramiko
    mod_ssl
    php
    php-xml
    webalizer
    
  3. Migration fails with the following error:

    2017-02-02 02:07:23.259  INFO Restoring PAU
    Starting pau
    oom_adj modified
    Waiting pau to start: ESC[60G[ESC[0;31mFAILEDESC[0;39m]
    2017-02-02 02:09:24.733 ERROR ExecFailed service pau start exited with non-zero status 1, stderr: None, stdout: None
    2017-02-02 02:09:24.733 ERROR <class 'poaupdater.uUtil.ExecFailed'>
    service pau start exited with non-zero status 1, stderr: None, stdout: None
    2017-02-02 02:09:24.735 ERROR
      at execCommand (/usr/lib/python2.6/site-packages/poaupdater/uUtil.py :83) in raise ExecFailed(command, status)
      at restorePAU (/root/pem-backup-all/install_routines.py :582) in uUtil.execCommand('service pau start')
      at rv (install.py :186) in return fun(*args, **kwds)
      at doInstall (install.py :316) in phase_fun(install_routines.restorePAU, progress, "Restoring PAU")(installation)
    

    /tmp partition is mounted with noexec option:

    # mount | grep
    /dev/mapper/VolGroup-lv_tmp on /tmp type ext4 (rw,noexec,nosuid,nodev)
    

    Solution:

    Remount /tmp with exec permission and correct /etc/fstab file to correct the same:

    # mount -o remount,exec /tmp
    # grep tmp /etc/fstab
    /dev/mapper/VolGroup-lv_tmp /tmp                    ext4    rw,nosuid,nodev        1 2
    
  4. In case the restore operation is re-run, it may fail on the stage of starting PAU service with:

    2016-08-09 04:05:30.224  INFO *** Restoring PAU ***
    2016-08-09 04:05:30.224  INFO Restoring PAU
    Starting pau
    oom_adj modified
    pau is already runningESC[60G[ESC[0;31mFAILEDESC[0;39m]
    2016-08-09 04:05:30.278 ERROR ExecFailed service pau start exited with non-zero status 1, stderr: None, stdout: None
    2016-08-09 04:05:30.278 ERROR <class 'poaupdater.uUtil.ExecFailed'>
    service pau start exited with non-zero status 1, stderr: None, stdout: None
    2016-08-09 04:05:30.279 ERROR
      at execCommand (/usr/lib/python2.6/site-packages/poaupdater/uUtil.py :83) in raise ExecFailed(command, status)
      at restorePAU (/root/pem-all/install_routines.py :582) in uUtil.execCommand('service pau start')
      at rv (install.py :186) in return fun(*args, **kwds)
      at doInstall (install.py :316) in phase_fun(install_routines.restorePAU, progress, "Restoring PAU")(installation)
      at <module> (install.py :701) in doInstall(config)
    

    Solution:

    Stop pau service before another run:

        # service pau stop
    

Internal content

Link on internal Article