VE migration from VPS Hosting (non-OACI-based) to VPS Hosting (OACI-based) : CloudBlue Technical Support

Pre-Requisites for Migration

Before the migration, make sure the following requirements are met:

Automation Platform 7.4, CloudBlue Commerce 20.4, or later
Cloud Infrastructure Automation module is deployed
Virtuozzo 6.0 or higher is installed

Migration Preparations

The following preparations needed to be performed:

Cloud Infrastructure Automation upgraded to 17.13 or higher
All Cloud Infrastructure Automation hotfixes are installed (for 17.13 – Hotfix 10 or later, for 18.0 – Hotfix 3 or later)
HW nodes registered in Cloud Infrastructure Automation
IP pools of the source VPSs should be registered as OACI IP pools (with the purpose Cloud Infrastructure)
Target Service Plans are configured. VPS parameters in the target service plans should be equal or more than the source VPS (to be migrated) has. During the migration the VPS parameters will be adjusted to the bigger values. Be aware that timeout is expected for VE resizing.
Each VPS has a link to the OS template from which it was created. The OS templates, which are used in VPSs to be migrated, should be registered in OACI. If the VPSs are linked to the OS templates that are not supported and can't be registered on Virtuozzo where you want to migrate them, then during the migration in the parameters you can define which OS template will be used in that case.
Add OS templates, on which your legacy VPSs are based, in the VPS virtual server template to provide a customer with capability to recreate a migrated server.

Important: Please back up the whole Virtuozzo node hosting VPSs to be migrated or the particular VPSs before the migration to VPS Hosting (OACI-based).

Before the migration you need to make the following changes in the IM DB.

Why do you need to make changes in the IM DB?

The table public_nets in the IM DB lists the OACI IP pool and a pool used for legacy VPS Hosting. Both pools are marked with the "Cloud Infrastructure" purpose. It means that the IM can use IP addresses from these pools when assigning to new OACI servers. During the migration the legacy VPS IP pool should be blocked so that the IM could not assign IP addresses for new OACI servers from there. The following values are possible in this column:

f – false. An IP address pool is not disabled and can be used for OACI purpose.
t - true. An IP address pool is disabled and cannot be used for OACI purpose.

Before the migration, set the disabled column value to true for the legacy VPS Hosting pool. As a result, the IM will not use this IP pool for newly created OACI servers.

Perform the following actions to make changes in the IM DB:

View the content of the public_nets table by running the command:

su postgres
psql im
im=# select * from public_nets;
 id | netgroup_id | label |  gateway  |    ip_min    |    ip_max    | mask | disabled
----+-------------+-------+-----------+--------------+--------------+------+----------
  1 |           1 | 2     | 10.31.0.1 | 10.31.18.128 | 10.31.18.255 |   16 | f
  2 |           1 | 5     | 10.31.0.1 | 10.31.66.1   | 10.31.66.66  |   16 | f
(2 rows)

Set the true value for the legacy VPS Hosting IP pool:

im=# update public_nets set disabled = 't' where id = <id_of_legacy_vps_ip_pool>;
UPDATE 1
im=# select * from public_nets;
 id | netgroup_id | label |  gateway  |    ip_min    |    ip_max    | mask | disabled
----+-------------+-------+-----------+--------------+--------------+------+----------
  1 |           1 | 2     | 10.31.0.1 | 10.31.18.128 | 10.31.18.255 |   16 | f
  2 |           1 | 5     | 10.31.0.1 | 10.31.66.1   | 10.31.66.66  |   16 | t
(2 rows)

Exit from the psql utility and superuser shell.
```
\q
exit
```

As a result, the IM will not use the legacy VPS IP pool for OACI purpose.

Overview of Migration Procedure

Migration of VE to VPS Hosting (OACI-based) consists of the following steps:

Registering the HW node with legacy VEs in OACI.
Preparing the VPS Hosting service template.
Preparing the VPS Hosting service plan.
Running the migration script per list of subscriptions.

Important: Be aware that VEs will be rebooted during migration.

Step 1. Registering HW Node in OACI

In Operations PCP, go to Cloud Infrastructure > Infrastructure > Virtualization Nodes.
Register the HW node that hosts legacy VEs.

Step 2 and 3. Preparing VPS Hosting Service Template and Service Plan

Follow the common guidelines to prepare the VPS Hosting service template and plan.

Important: Make sure that the limits of VE hardware (CPU, RAM, HDD) are the same as the size of VE that you are going to migrate or higher. Be aware that timeout is expected during VE resizing.

Step 4. Running Migration Script

Upload the migration script archives legacy-vps-migration-tool-1.0.23-bundle.tar.xz to the Management Node. MD5SUM: 455621B7E7843956BA1475D66ADCA7E0
Unpack the uploaded archives.
Enter the folder with the installation files.
Prepare the migration_params.json file where you should define the parameters for migration.
Migration Parameters Defined in JSON File
- pba_ip – BackNet IP address of the Billing Application node.
- pba_user – admin user on the Billing Application node.
- pba_password – password for connection to the Billing Application node. Billing Application node parameters can be found in /usr/local/bm/conf/_amt_service_.res on the Billing Application node.
- mn_certificate – the MN certificate is located in /usr/local/pem/APS/certificates/poa.pem on the MN.
- plesk_configuration – ID of Plesk configuration which will be installed in a migrated virtual server. If there is only one Plesk configuration in a service plan, then it is not required to define this parameter. If there are several Plesk configurations in a service plan, this parameter is mandatory otherwise an error will be shown during the migration. You can find ID of Plesk configuration in PCP > Operations > Products > Resources, searching by "Plesk" resource type.
- plan_id – target Service Plan ID for a new subscription (for installations with Billing).
- plan_period_id – Subscription Period ID for a new subscription (for installations with Billing).
- service_template_id – target Service Template ID for a new subscription (for installations without Billing).
- expand_cpu_cunt_to_subscription_limits – defines what to do if the actual VE parameters are smaller than the limits of VPS Hosting service plan. If "true“ value is set, VE will be resized up to the limit in the plan.
- expand_ram_to_subscription_limits – the same as above.
- expand_hdd_to_subscription_limits – the same as above.
- default_os_template – first, the migration script checks if the OS template associated with the VE is registered in OACI. If it is not registered, the script will use this parameter.
- pba_db_user – Billing DB user ("pba").
- pba_db_name – Billing DB name ("pba").
- pba_db_ip – BackNet IP address of the Billing DB node.
- pba_db_pass – password for accessing the Billing DB located in /usr/local/bm/etc/ssm.conf.d/global.conf on the Billing Application node.
- im_ip – BackNet IP address of the IM node.
- im_db_user – IM DB user (commonly used "im")
- im_db_name – IM DB name (commonly used "im")
- im_db_ip – BackNet IP address of the node where the IM DB is hosted.
- im_db_pass – IM DB password. You can get it in /usr/local/share/PACI-im/IM-config.xml on the IM node.
- im_host_id – this is an optional parameter which defines id of the IM node in the list of service nodes in PCP (in PCP go to Infrastructure > Service Nodes and find the id of the IM node).
- im_if_id – this is an optional parameter which defines id of the IM network interface. You can specify a particular interface otherwise it will be defined randomly. To get the list of network interfaces for the IM node, you can perform a request to the OSS DB:
- corba_port - port for CORBA connections used by migration script, by default, 8354 and usually you do not need to change it
- corba_user - name of CORBA user. By default "admin" and usually you do not need to change it
- corba_password - password that is needed for CORBA protocol. By default the same as provider admin password
```
select h.primary_name, n.if_id, n.host_id from net_interfaces n inner join hosts h on n.host_id = h.host_id;
```
  Where if_id – id of the network interface of a host.
  host_id – node id in the list of service nodes in PCP.
- subscription_cancelation_type
- subscription_cancelation_reason_id

For information about cancellation types and reason ids, refer to PlaceSubscriptionCancellationOrder_API

The example of migration_params.json file is shown below:

{
   "pba_ip" : "192.168.66.122",
   "pba_user" : "system",
   "pba_password" : "HUmYIs7PdrKZgl68t",
   "mn_certificate": "/usr/local/pem/APS/certificates/poa.pem",
   "plesk_configuration": 1000068,
   "plan_id" : 3,
   "plan_period_id" : 5,
   "service_template_id" : 3,
   "expand_cpu_count_to_subscription_limits" : "true",
   "expand_ram_to_subscription_limits" : "true",
   "expand_hdd_to_subscription_limits": "true",
   "default_os_template" : "centos-6-x86_64",
   "pba_db_user": "pba",
   "pba_db_name": "pba",
   "pba_db_ip": "10.26.167.95",
   "pba_db_pass": " ",
   "im_ip": "192.168.67.36",
   "im_db_user": "im",
   "im_db_name": "im",
   "im_db_ip": "192.168.116.48",
   "im_db_pass": " ",
   "im_host_id" : 7,
   "im_if_id": 14,
   "subscription_cancelation_type" : 30,
   "subscription_cancelation_reason_id" : 1,
    "corba_port": 8354,
    "corba_user": "admin",
    "corba_password": "1q2w3e$R%T^Y"

}

Prepare the list of subscriptions to be migrated by doing the following:
1. In PCP, go to Operations > Subscriptions.
2. Filter the subscriptions by plan.
3. Click Export to Excel.
4. Copy the subscription ids in a file. For example, the subscriptions_list file can be as in the example below:
```
# cat ./subscriptions_list 
1000044
1000045
1000030
...
```
5. Use this file for migration. Important: Make sure that there are no empty strings in the subscriptions_list file because the migration script considers them as subscription ids and it can cause migration failure.

Run the migration script:

sh vps-migrate.sh [-h]
                  [--mode {dry-run,normal-run,retry-failed-stage,skip-failed-stage}]
                  [--configuration-file CONFIGURATION_FILE]
                  (--subscription-id SUBSCRIPTION_ID | --subscription-list SUBSCRIPTION_LIST)

Where

mode:
- dry-run – check that a target plan has sufficient resource limits for migration of a VPS. Actual migration is not performed.
- normal-run – run migration. This is a default mode.
- retry-failed-stage, skip-failed-stage – these modes are used to repeat migration that has failed previously. If retry-failed-stage is chosen, the script will run again the stage that has failed during the previous run. If skip-failed-stage is chosen, the failed stage will be skipped and migration will start from the next stage. Note: these modes cannot be used for batch migration, only one subscription at a time can be migrated in these modes.
configuration-file: a file with migration parameters in the JSON format. Note: the default file is migration_params.json in the current directory.
subscription-id or subscription-list (only one option can be specified): subscription ID to be migrated or a file with the list of subscription IDs to be used in batch migration.
h: displays general information about the migration utility, including the list of allowed attributes and parameters, and its current version.

Examples of running the migration script:

Migration of a single subscription, the dry-run mode:

sh vps-migrate.sh --subscription-id 1000011 --mode dry-run

Migration of a list of subscriptions:

sh vps-migrate.sh --subscription-list list.file

Resuming migration after failure (skipping the stage that failed):

sh vps-migrate.sh --subscription-id 1000011 --mode skip-failed-state

Resuming migration after failure (retrying the stage that failed):

sh vps-migrate.sh --subscription-id 1000011 --mode retry-failed-state

After the successful migration of legacy VPSs, allow the IM use the legacy VPS Hosting IP pool by changing the value in the column "Disabled" for the legacy VPS IP pool in the IM DB:

Run the following commands:

su postgres
psql im
im=# update public_nets set disabled = 'f' where id = <id_of_legacy_vps_ip_pool>;
UPDATE 1
\q
exit

What Actions Does Migration Script Perform

When you run the migration script, it automatically performs the following:

Collects limits from the VPS Hosting service plan and template and puts them in the new-ve-limits-for-subscr-35.json;
Executes the script normalize-vps.pyon HW node, which does the following:
- Reconfigures network parameters of VE – creates a 'Bridged' adapter instead of a ‘Host-routed’ one and assigns the existing public IP address to it; this is necessary because OACI uses the ‘Host-routed’ adapter for a private IP address.
- Ensures that the CPU, RAM and HDD limits do not exceed the limits in the new VPS Hosting Service Plan. If they exceed, the script fails. If they are less than those in the plan and the corresponding parameter "expand_xxx_to_subscription_limits" is "true" in config file, then the VE is resized.
- Creates two json files on HW node like /usr/local/share/oaci/original--1010.json, /usr/local/share/oaci/reconfigured--1010.json which have the original VE parameters and reconfigured VE parameters.
Creates a new empty VPS Hosting subscription based on the created service template and plan (no order is created).
Starts the task that imports the existing VE into the new subscription.
Places the Cancellation Order for the legacy VPS subscription (VE is not deleted on subscription cancellation). A provider should process this Cancellation Order in PCP.

Subscription Lifecycle

A new subscription will be created while cancellation order will be placed for an original one
Original subscription will be switched into disabled state by cancellation order placement
No refunds or additional charges will be applied while cancelling original VPS subscription
Original subscription is being paid by the end of billing period, where migration occurs

Billing After Migration

Let’s consider two cases with different billing models – Before Billing Period, After Billing Period.

alttext

Troubleshooting

Let’s consider the two stages when some issues may occur.

During VE migration script performance

If you have any issues related to the script execution, investigate logs:

On the Management Node, the migration script log is located in /var/log/pa/vps-migration/subscription-{subscription_ID}.log.

On the HW node, the script working directory is /usr/local/share/oaci. You can find files with VE configuration before and after VE normalization, for example: original–1010.json, reconfigured–1010.json

During VE import

On the event of VE import, the corresponding task is created in the Task Manager in PCP. You should ensure that VE import task (like shown below) is completed successfully.

Execute operation 'import' with ID 2ee2fdba-72de-4101-b96d-b88527991d87 on resource ed02a8d6-f312-478c-b3c2-09c5e94b0fc2

After the VE import, VE name on the node will be changed according to the naming pattern used by OACI: {customerID}.server-{subscriptionID}-{Ordinal}, for example:

Before migration

# prlctl list
UUID STATUS IP_ADDR T NAME
{ca08ce57-2f9a-4c36-82b2-2d6493622280} running 10.26.18.179 CT 1012

After migration

# prlctl list
UUID STATUS IP_ADDR T NAME
{ca08ce57-2f9a-4c36-82b2-2d6493622280} running 172.21.3.57 CT 1000001.server-1000067-1

General Recommendations

Let's consider the case when you first run the migration script in the "dry-run" mode and get the "Ok" status. Then you run it in the "normal-run" mode and get the "Failed" status for subscription migration such as the following:

# sh vps-migrate.sh --subscription-list subscriptions_list --mode normal-run
===========================================================================================================================
| Subscription Id | Target Plan Id | Target Plan Period Id | Target Service Template Id | Target Subscription Id | Status |
===========================================================================================================================
| 30              |                |                       | 3                          | 68                     | Failed |
 ---------------------------------------------------------------------------------------------------------------------------
[INFO] Result log is saved in /var/log/pa/vps-migration/summary.log
[INFO] Done. Log is saved in /var/log/pa/vps-migration/migration.log

You see that the subscription with the ID=30 failed. Now you must investigate the log files containing this id number.

To investigate the issue, do the following:

Find out on which stage the migration failed by investigating the migration-state-30.json file on the MN:

# less /var/log/pa/vps-migration/migration-state-30.json
    {
        "stages": [
            {
                "status": "COMPLETED",
                "name": "normalize_ve"
            },
            {
                "status": "COMPLETED",
                "name": "create_new_subscription"
            },
            {
                "status": "FAILED",
                "name": "import_ve"
            },
            {
                "status": "NOT_STARTED",
                "name": "plesk_post_import"
            },
            {
                "status": "NOT_STARTED",
                "name": "prepare_disabling_old_subscription"
            },
            {
                "status": "NOT_STARTED",
                "name": "disable_old_subscription"
            }
        ],
        "properties": {
            "new_subscription_id": 1000131,
            "new_environment_uuid": "45be3508-7016-4e17-a3e7-9f3f80a7854c"
        }
    }

You can see that the "FAILED" status means that the migration failed on the "Import_VE" task (importing a normalized VE to a new subscription).

Investigate the migration log for the subscription with the id=30 by searching for the POST API request for import:

# less /var/log/pa/vps-migration/migration-30.log
[INFO] POST https://localhost:6308/aps/2/resources/ac9cb848-7cdf-4936-9268-2af9d02673a7/import/9447db41-ab5e-4ef8-8cf5-a12668e14172/at/LegacyVPSNode-1fd4177c792a.aqa.int.zone
[INFO] Status: 202
[INFO] Response headers:
        Expires: Mon, 01 Jan 2001 00:00:01 GMT
        Cache-Control: no-cache, no-store
        Set-Cookie: JSESSIONID=F00787F63BF8819B67153ECC7AACE888; Path=/; Secure; HttpOnly
        X-XSS-Protection: 1; mode=block
        Pragma: no-cache
        X-Frame-Options: DENY
        Date: Mon, 28 Jan 2019 16:12:29 GMT
        Connection: close
        X-APSBooster-Response: true
        X-Content-Type-Options: nosniff
        Content-Type: text/plain
        Content-Length: 2
        APS-Retry-Timeout: 5
        APS-Info: Importing VE ...

Where ac9cb848-7cdf-4936-9268-2af9d02673a7 – APS User ID,

9447db41-ab5e-4ef8-8cf5-a12668e14172 – APS VE ID

Open /var/log/pa/vps.log on the IM node and find the POST request by the APS VE ID:

/POST.*ve\/9447db41-ab5e-4ef8-8cf5-a12668e14172
2019-01-28 19:16:57,510 (web_fdfe508c-d39c-433d-9ac8-ff1400e333ae_2206) INFO  CustomLoggingFilter [Grizzly-worker(3)] - 442 * Server in-bound request
442 > POST http://192.168.69.143:4465/paci/v1.0/of/4/ve/9447db41-ab5e-4ef8-8cf5-a12668e14172/import
442 > authorization: <hidden>
442 > x-paci-request-id: web_fdfe508c-d39c-433d-9ac8-ff1400e333ae_2206
442 > group-id: 87b118c8-a407-487b-805c-0f601e5efd9a
442 > x-callback-url: https://10.31.108.47:4477/f9d813d5-ecca-49b4-abf6-8b1f0218be09
442 > accept: application/xml
442 > content-type: application/xml
442 > user-agent: Java/1.8.0_191
442 > host: 192.168.69.143:4465
442 > connection: keep-alive
442 > content-length: 311
442 >
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><imported-ve><name>server-68-1</name><subscription-id>68</subscription-id><hn-name>LegacyVPSNode-1fd4177c792a
.aqa.int.zone</hn-name><template-name>centos-6-x86_64-pcs6</template-name><user-uuid>ac9cb848-7cdf-4936-9268-2af9d02673a7</user-uuid></imported-ve>

You can find the reason of VE import failure, for example, invalid configuration of the imported VE. For example, VE resources do not fit resource limits in the ST/SP or node.

Import VE b70a4aa7-e1ed-44ae-9cd7-72eda320b0c0 - invalid data in ve_info: VE cpu_number 4 exceeds the one of the node: 2

Open /var/log/pa/vps.aps.log on the CI Endpoint node and find the POST request by the APS VE ID:

/POST.*import/9447db41-ab5e-4ef8-8cf5-a12668e14172/at/
* [APSC->PACI]
> POST https://10.31.108.47:9000/api/environments/ac9cb848-7cdf-4936-9268-2af9d02673a7/import/9447db41-ab5e-4ef8-8cf5-a12668e14172/at/LegacyVPSNode-1fd4177c792a.aqa.int.
zone
> accept-encoding: gzip
> aps-actor-id: 0e7845b6-4ddd-4df0-869a-0390dd7b74d0
> aps-controller-uri: https://192.168.68.138:6308/
> aps-instance-id: c3f9089c-598b-4968-bc02-b4f91ac653d4
> aps-request-id: cf7bd163-db38-4296-b94a-c9034949c251
> aps-request-phase: async
> aps-version: 2.2
> connection: close
> content-type: application/json
> host: 10.31.108.47:9000
> user-agent: Python-urllib/2.7
> x-correlation-id: task:2039:1307
> content-length: 0
> cookie: JSESSIONID=***hidden***
2019-01-28 19:12:35,444 web_a22f9d3f-31f0-4d6a-a3c8-b6aed8deef5d_2201 ERROR IaaSExceptionMapper [https-jsse-nio-9000-exec-1] - failure
com.parallels.paci.vps.exceptions.ImException: [IM] P10001: Operation failed
2019-01-28 19:12:35,444 web_a22f9d3f-31f0-4d6a-a3c8-b6aed8deef5d_2201 INFO  HttpLoggingFeature [https-jsse-nio-9000-exec-1] - Server responded with a response on thread https-jsse-nio-9000-exec-1
* [APSC<-PACI]

Where aps-request-id: cf7bd163-db38-4296-b94a-c9034949c251 – ID of the failed task in PCP > Tasks.

Now fix the cause of the issue and then continue VE migration by doing the following:
1. In Operations PCP, go to Operations > Tasks.
2. Find the failed task by the ID which you have found at the previous step. You will find a task like the following: Execute operation 'import' with ID fc3f30bc-5160-4bd3-9e2b-3ac9af4c9025 on resource 515bee48-b25b-4de4-bc24-068802052819,
  where fc3f30bc-5160-4bd3-9e2b-3ac9af4c9025 – unique APS ID of the task.
  515bee48-b25b-4de4-bc24-068802052819 – APS ID of the new VE. APS ID can be found in the migration-state-30.json file in the parameter "new_environment_uuid": "515bee48-b25b-4de4-bc24-068802052819".
3. Select this task and click Run Tasks.
4. Wait until the task is successfully completed.
Run the migration script in one of the following modes:
- "skip-failed-stage" mode: if the migration failed on the "import_ve" stage.
- "retry-failed-stage" mode: if the migration failed on the "normalize_ve" or "create_new_subscription" stage.

Migration Restrictions

After the migration Plesk will be operational as before but will not be shown in the control panel. If you need Plesk to be available in the control panel, prepare the respective Plesk configuration (PCP > Cloud Infrastructure > Plesk in VPS > Plesk Configurations > ), disable the Install updates automatically option and the Update pre-installed Plesk to the latest minor version option, and manually provision it.
Bundled service subscriptions are not supported
Low-level Virtuozzo config parameters for CPU and RAM (QoS) are not supported by OACI
Legacy VPS Backups will not be available, new backups should be created
Existing applications would not be available for management in CCP
OACI VPS requires private IP for every VE. Corresponding network configuration and IP pools are required.

Internal content

Link on internal Article