Introduction

This instruction describes how to migrate subscriptions from the CI Hosting model (service controller based) to the CS Hosting model (APS-based).

Pre-requisites for CI Hosting migration

Make sure that the following requirements are met before the migration:

  1. Cloud Infrastructure Automation module, Cloud Infrastructure APS package and endpoint are installed. The minimum required Cloud Infrastructure package version is oaci-17.13-396.
  2. CS Hosting resource types are present in PCP. Create two resource types – CS Instance- AUTOPROVISION and Virtual Server- NO AUTOPROVISION – as described below. They will be used in CI service template for migration.
  3. The following services are up and running:
    • pa-service
    • pau
    • pba
    • PACI-im
  4. The Management Node can communicate with the Cloud Infrastructure Instance Manager and Billing Application Server.
  5. Service plans created based on a migrated CI service template are prohibited.

Creating 'CS Instance- AUTOPROVISION' resource for migration

  1. Go to Application > Cloud Infrastructure > Resource Types > Create.
  2. Choose the Application Service type.
  3. Specify the name CS Instance- AUTOPROVISION (use this naming).
  4. Select service of application with the name Cloud Servers.
  5. Make sure that the auto-provisioning option is on.

Creating 'Virtual Server- NO AUTOPROVISION' resource for migration

  1. Go to Application > Cloud Infrastructure > Resource Types > Create.
  2. Choose the Application Service type.
  3. Specify the name Virtual Server- NO AUTOPROVISION (use this naming).
  4. Select service of application with the name Virtual Server.
  5. Make sure that the auto-provisioning option is off.

Overview of migration process

The migration process consists of the following steps:

  1. Uploading the package ci2cs-1.18.zip (md5sum: ffe0ab3fa30eaa3a9cf9109fc72e1f65 ci2cs-1.18.zip ) with migration tools on the Management Node and edit the config.py file.
  2. Preparing the subs.example.json file with the list of subscriptions to be migrated.
  3. Preparing the resources.py file.
  4. Evaluation of CI subscriptions.
  5. Insertion of CS Hosting resources in the CI Hosting service template and subscriptions.
  6. Provisioning of CS Hosting resources.
  7. Insertion of CS Hosting resource rates in CI Hosting service plan.
  8. Final stage (synchronizing CI subscriptions with changes in the CI Hosting service plan).

Let's consider each step in details.

Step 1. Uploading the migration tools.

  1. Unpack the archive ci2cs-x.xx.zip on the Management Node. This archive has scripts and files that are needed for migration:

    • config.py - contains configuration settings for connection to Billing Application server, Billing DB, and performance settings.
    • main.py - main script that performs evaluation, insertion and provisioning of CS resources, finalizes the migration process.
    • resources.py - this file is used by the main script on the stage when insertion of CS resources in subscriptions is performed. The list of resources in the file should be adjusted according to the list of CS resources in PCP. You should list only those resources that you want to be added to one CI Hosting service template. If you have several CI Hosting service templates, do not use the resources.py file for inserting CS resources in both of them simultaneously.
    • subs.example.json - this file contains the list of IDs of CI subscriptions that should be migrated to the CS Hosting model. Note that all the subscriptions should be based on the same CI service template.
  2. Open the config.py file and replace all the placeholders (inside the angle brackets) with your system data.

    #!/usr/bin/python
    # -*- coding: utf-8 -*-
    
    # Billing DB IP and Pass could be taken from /usr/local/bm/etc/ssm.conf.d/global.conf
    billing_db_user = 'pba'
    billing_db_name = 'pba'
    billing_db_ip = '<Billing DB IP Address>'
    billing_db_pass = '<Billing DB password>'
    
    # Performance configuration
    openapi_threads = 5
    aps_api_threads = 5
    pba_api_threads = 5
    
    # Migration Database setting. pypaci.db will be saved in log folder /var/log/pa/ci2cs
    database = "./pypaci.db"
    
    # Days to consider subscription inactive
    days_of_inactivity = 90
    
    # Default batch size of subscriptions to be processed
    default_batch_size = 10
    
    # Allow local and network storage types in subscription
    allow_all_storage_types_in_subscription = False
    

Note: The allow_all_storage_types_in_subscription parameter was added in version 1.17 and is optional. If you wish to use it, don't forget to update the configuration file.

Note: If you plan to migrate a short list of subscriptions (< 100), it is recommended to set the performance parameters to 1 thread.

Step 2. Preparing the `subs.example.json` file.

The subs.example.json file has the list of subscription ids to be migrated. Preparing this file is not required, you can list all the subscriptions in the migration script explicitly. You can use this file if the list of subscriptions is long.

Important: Make sure that all the subscriptions listed in the file subs.example.json are based on the same CI service template. Do not mix subscriptions based on different CI service templates.

    {
        "subscriptions": [1000100, 1000101, 1000102, 1000103, 1000104, 1000105, 1000106, 1000107, 1000108]
    }

The subs.example.json file can be created automatically. In the folder with the migration tools, you can find the propose.py file that can generate the suggested subs.json files based on your CI subscriptions and CI Service Templates. The script searches for subscriptions in the paci_usage_timestamps table, which includes all subscriptions based on the service template with CI resources, and group them by the service template ID.

To generate the subs.example.json file, run the following command:

# python propose.py --prepare

As a result, you will get the files such as st_5.subs.json, st_6.subs.json. For example, the st_6.subs.json will look like the following:

# cat st_6.subs.json 
{
  "subscriptions": [
    4,
    5,
    6
  ]

Where the st_6 means that the file contains all CI subscriptions based on the Service Template #6.

Step 3. Preparing the `resources.py` file.

The resources.py file has a set of resources, their names and limits. Note that the list of CS resources in the file is dedicated for insertion in no more than one CI Hosting service template.

You can use the following example for reference:

    cs_resources = [
        {'name': u'DNS Hosting', 'resource_class_name': 'DNS Hosting', 'resource_id': 0, 'limit': -1},
        {'name': u'DNS Management', 'resource_class_name': 'DNS Management', 'resource_id': 0, 'limit': -1},
        {'name': u'OACI Location DEFAULT', 'resource_class_name': 'rc.saas.service.link', 'resource_id': 0, 'limit': 1},
        {'name': u'OACI Backup Schedule - Daily', 'resource_class_name': 'rc.saas.service.link', 'resource_id': 0, 'limit': -1},
        {'name': u'OACI Backup Schedule - Weekly', 'resource_class_name': 'rc.saas.service.link', 'resource_id': 0, 'limit': -1},
        {'name': u'OACI Backup Schedule - Monthly', 'resource_class_name': 'rc.saas.service.link', 'resource_id': 0, 'limit': -1},
        {'name': u'OACI Backup Schedule - Custom', 'resource_class_name': 'rc.saas.service', 'resource_id': 0, 'limit': -1},
        {'name': u'CS Application', 'resource_class_name': 'rc.saas.service.link', 'resource_id': 0, 'limit': 1},
        {'name': u'CS Virtual Server Template', 'resource_class_name': 'rc.saas.service.link', 'resource_id': 0, 'limit': -1},
        {'name': u'CS UI Policy', 'resource_class_name': 'rc.saas.service.link', 'resource_id': 0, 'limit': 1},
        {'name': u'CS Load Balancer', 'resource_class_name': 'rc.saas.service', 'resource_id': 0, 'limit': 10},
        {'name': u'CS Virtual Environment Image Limit', 'resource_class_name': 'rc.saas.service', 'resource_id': 0, 'limit': -1},
        {'name': u'CS Virtual Environment Backup Limit', 'resource_class_name': 'rc.saas.service', 'resource_id': 0, 'limit': -1},
        {'name': u'CS Public IPv4 Address', 'resource_class_name': 'rc.saas.service', 'resource_id': 0, 'limit': 10},
        {'name': u'CS Public IPv6 Address', 'resource_class_name': 'rc.saas.service', 'resource_id': 0, 'limit': 100},
        {'name': u'CS CPU Power Usage Running Container', 'resource_class_name': 'rc.saas.resource.mhzh', 'resource_id': 0, 'limit': -1},
        {'name': u'CS CPU Power Usage Stopped Container', 'resource_class_name': 'rc.saas.resource.mhzh', 'resource_id': 0, 'limit': -1},
        {'name': u'CS CPU Power Usage Running Virtual Machine', 'resource_class_name': 'rc.saas.resource.mhzh', 'resource_id': 0, 'limit': -1},
        {'name': u'CS CPU Power Usage Stopped Virtual Machine', 'resource_class_name': 'rc.saas.resource.mhzh', 'resource_id': 0, 'limit': -1},
        {'name': u'CS Inbound traffic', 'resource_class_name': 'rc.saas.resource.kba', 'resource_id': 0, 'limit': -1},
        {'name': u'CS Outbound traffic', 'resource_class_name': 'rc.saas.resource.kba', 'resource_id': 0, 'limit': -1},
        {'name': u'CS RAM Usage Running', 'resource_class_name': 'rc.saas.resource.mbh', 'resource_id': 0, 'limit': -1},
        {'name': u'CS RAM Usage Stopped', 'resource_class_name': 'rc.saas.resource.mbh', 'resource_id': 0, 'limit': -1},
        {'name': u'CS Local Diskspace Usage Running', 'resource_class_name': 'rc.saas.resource.mbh', 'resource_id': 0, 'limit': -1},
        {'name': u'CS Local Diskspace Usage Stopped', 'resource_class_name': 'rc.saas.resource.mbh', 'resource_id': 0, 'limit': -1},
        {'name': u'CS PCS Diskspace Running', 'resource_class_name': 'rc.saas.resource.mbh', 'resource_id': 0, 'limit': -1},
        {'name': u'CS PCS Diskspace Stopped', 'resource_class_name': 'rc.saas.resource.mbh', 'resource_id': 0, 'limit': -1},
        {'name': u'CS Bandwidth Running', 'resource_class_name': 'rc.saas.resource.kbpsh', 'resource_id': 0, 'limit': -1},
        {'name': u'CS Bandwidth Stopped', 'resource_class_name': 'rc.saas.resource.kbpsh', 'resource_id': 0, 'limit': -1},
        {'name': u'CS Image Space Usage', 'resource_class_name': 'rc.saas.resource.mbh', 'resource_id': 0, 'limit': -1},
        {'name': u'CS Backup Space Usage', 'resource_class_name': 'rc.saas.resource.mbh', 'resource_id': 0, 'limit': -1},
        {'name': u'Virtual Server- NO AUTOPROVISION', 'resource_class_name': 'rc.saas.service', 'resource_id': 0, 'limit': -1},
        {'name': u'CS Instance- AUTOPROVISION', 'resource_class_name': 'rc.saas.service', 'resource_id': 0, 'limit': 1},
    ]

Your list of resources can be different (you may have different 'Virtual Server Template' or you may not have 'DNS Management'), therefore adjust the list of resources in the file according to the set of CS resources in your PCP.

Important: Make sure that the resource 'CS Virtual Environment Limit' which will be used for migration (in our example it is called 'Virtual Server- NO AUTOPROVISION') has the activation parameter 'Automatically provision service' turned off.

Important: Make sure there is only one 'Virtual Server Template' resource included in the file, as CS Hosting model does not support several VSTs per a single service template.

resources.py contains the list of application service resources, whose usage will be reduced to zero after the migration, so that customers will be billed for these resources only from start billing period till the migration date (provisioning stage)

    DROP_USAGE = [
        {'name': u'IP addresses', 'resource_class_name': 'ips', 'resource_id': 0},
        {'name': u'IPv6 addresses', 'resource_class_name': 'ipv6_addresses', 'resource_id': 0},
        {'name': u'CI Virtual Environments Limit', 'resource_class_name': 'c2u.vecount', 'resource_id': 0}
    ]

Important: Do not include the CI Cloud Hosting resource in this section, because in this case the customer will not be billed for CI-related counters.

Step 4. Evaluation of CI subscriptions.

During this step, you should run the script which performs the following:

  • Collects information about all CI subscriptions.
  • Prepares a virtual server template with defined limits and step. This template has the same parameters and presets as in the original CI Service Templates.
  • Prepares a set of CS resources to be inserted into live subscriptions (w\o provisioning).

Run the main.py script for evaluation. In the command parameters you can specify either the file with the list of subscriptions to be migrated or define explicitly the subscription ids.

    # python main.py --subscriptions_file subs.example.json --stage evaluation

OR

    # python main.py --subscriptions 1000005 1000006 1000007 --stage evaluation

    2018-07-11 10:27:28.129 [DEBUG] collecting information about CI subscriptions [1000005, 1000006, 1000007] and it's Service Template
    2018-07-11 10:27:30.010 [DEBUG] [evaluation] parsed service templates and virtual server templates
    2018-07-11 10:27:30.338 [DEBUG] [evaluation] parsed subscriptions

Then, check the status of subscriptions:

    # python main.py --status
    2018-07-11 10:28:17.589 [DEBUG] getting status of subscriptions and migration itself
    |subscription_id|       |migration_status|      |oaci_status|   |migration_type|
    1000005                 injected                active          regular
    1000006                 evaluated               was_active      regular
    1000007                 evaluated               inactive        regular

After evaluation is done, you can see the status of each subscription:

  • active status – currently have VEs in OACI
  • was_active status – currently no VEs in OACI but the subscription has usage for the past 3 months. The period during which the script checks usage is configurable in the config.py file by means of the days_of_inactivity option.
  • inactive status – no VEs in OACI at all.

You can also find the 'migration_type' option:

  • regular – means that you run one stage at a time (it is a default option if no other options are specified in the script arguments)
  • fast – a suitable option to migrate one subscription fast, see the description below in the 'Migration in One Run' section
  • only-ui – a suitable option for migration of inactive subscriptions. A provider will turn off the old CI panel for them, see the 'Partial Switch to CS Hosting' section.

Note: During the evaluation stage, you can see warnings such as

    "[evaluation] going to parse_ve_size_restrictions of service template: 78"
    "[evaluation] trying to load Legacy OACI CI resources activation parameters json '' in order to create possible CS Sample Resource"
    "[evaluation] a non-critical error occurred on attempt to parse Legacy OACI CI resources activation parameters '' falling to defaults"

This means that in the Service Template with the id 78 the CI Resource has wrong activation parameters, for example, additional dots, spaces, or it is empty.

It is not critical since the CS Sample Resource is composed by parsing the legacy CI resources.

This CS Sample Resource is not mandatory, you can use it, edit it or create your own one manually. When this warning occurred in the log, the migration will continue, the script will set the default values if the generated ones are not appropriate.

Now you can choose which type of subscriptions to process by means of the --oaci option of the main.py script. By default, active subscriptions will be selected for processing.

See the example, how to use the script with the --oaci option:

    usage: main.py
                   [--oaci {inactive,was_active,active}]


    # python main.py  --subscriptions 1000005 1000006 1000007 --stage injection --oaci active
    2018-09-11 09:04:10.242 [DEBUG] Call openapi.OpenAPI method to get PACI settings
    2018-09-11 09:04:10.850 [DEBUG] injecting CS resources into subscriptions [1000005 1000006 1000007]
    active
    2018-09-11 09:04:10.850 [DEBUG] [injection] these of specified subscriptions are in 'evaluated' state and could be injected [1000005]

In the example, only the 100005 subscription has been chosen as appropriate for insertion of CS resources because only it has active OACI VEs.

After performing the evaluation stage, a new virtual server template is created. This template has the same parameters and presets as in the original CI Service Templates. It will have a name similar to 'Virtual Server Template For 5 Service Template'. In this name, 5 means the ID of CI Hosting service template for which a corresponding virtual server template was created. You can review it, adjust as needed and add to the resource.py file instead of the virtual server template defined previously.

Step 5. Insertion of CS Hosting resources in CI service template and subscriptions.

  1. In Operations PCP, deactivate the CI Hosting service template.
  2. Add CS Hosting resources to it:

    • DNS Hosting; limit = -1
    • DNS Management; limit = -1
    • OACI Location DEFAULT; limit = 1
    • OACI Backup Schedule - Daily; limit = -1
    • OACI Backup Schedule - Weekly; limit = -1
    • OACI Backup Schedule - Monthly; limit = -1
    • OACI Backup Schedule - Custom; limit = -1
    • CS Load Balancer; limit = 10
    • CS Virtual Environment Image Limit; limit = -1
    • CS Virtual Environment Backup Limit; limit = -1
    • CS Public IPv4 Address; limit = 10
    • CS Public IPv6 Address; limit = 100
    • CS CPU Power Usage Running Container; limit = -1
    • CS CPU Power Usage Stopped Container; limit = -1
    • CS Inbound traffic; limit = -1
    • CS Application; limit = 1
    • CS Virtual Server Template; limit = -1
    • CS UI Policy; limit = 1
    • CS CPU Power Usage Running Virtual Machine; limit = -1
    • CS CPU Power Usage Stopped Virtual Machine; limit = -1
    • CS Outbound traffic; limit = -1
    • CS RAM Usage Running; limit = -1
    • CS RAM Usage Stopped limit; limit = -1
    • CS Local Diskspace Usage Running; limit = -1
    • CS Local Diskspace Usage Stopped; limit = -1
    • CS PCS Diskspace Running; limit = -1
    • CS PCS Diskspace Stopped; limit = -1
    • CS Bandwidth Running; limit = -1
    • CS Bandwidth Stopped; limit = -1
    • CS Image Space Usage; limit = -1
    • CS Backup Space Usage; limit = -1
    • CS Instance- AUTOPROVISION; limit = 1
    • Virtual Server- NO AUTOPROVISION; limit = -1
  3. Activate the CI Hosting service template.
  4. Insert CS resources in the specified CI subscriptions by running the main.py with the option --stage injection.

    # python main.py --subscriptions 1000053 1000054 --stage injection
    2018-07-11 10:50:22.676 [DEBUG] injecting CS resources into subscriptions [1000053, 1000054]
    2018-07-11 10:50:39.913 [DEBUG] [injection] injected resources into specified subscriptions
    
    # python main.py --status
    2018-07-11 10:50:42.027 [DEBUG] getting status of subscriptions and migration itself
    |subscription_id|       |migration_status|      |oaci_status|   |migration_type|
    1000053                 injected                active          regular
    1000054                 injected                was_active      regular
    

As a result, you have CS resources in the specified CI subscriptions now. They are not active but can be provisioned now.

For more information about configuration of the CS Hosting resource model, refer to the Cloud Infrastructure Provider's Guide.

Step 6. Provisioning of CS Hosting resources.

  1. Disabling usage collection.

    During provisioning and finalization steps, usage collection must be disabled. To do this, stop the following task:

    Synchronize resource usage for APS application (Cloud Infrastructure) for CS resources

    To stop a task, in the Provider Control Panel go to Operations > Periodic, open this task and click Cancel Task.

  2. Run the main.py with the option --stage provisioning.

    # python main.py --subscriptions 1000053 1000054 --stage provisioning
    2018-07-11 10:46:31.837 [DEBUG] provisioning CS APS resources for subscriptions [1000053, 1000054]
    2018-07-11 10:50:50.009 [DEBUG] [linking] legacy IM subscription 1000053 has been attached to CS model
    2018-07-11 10:50:51.969 [DEBUG] [linking] legacy IM subscription 1000054 has been attached to CS model
    
    # python main.py --status
    2018-07-11 10:59:31.119 [DEBUG] getting status of subscriptions and migration itself
    |subscription_id|       |migration_status|      |oaci_status|   |migration_type|
    1000053                 provisioned                active          regular
    1000054                 provisioned                was_active      regular
    
  3. Enabling usage collection.

    To enable usage collection, run the following periodic tasks:

Synchronize resource usage for APS application instance #18 (Cloud Infrastructure) for CS resources

To run a task, in the Provider Control Panel go to Operations > Periodic, open this task and click Run Task.

As a result:

  • the old usage reporting will be turned off,
  • you have provisioned CS resources for CI Hosting subscriptions.

Your customer can manage the server in both the UX1 panel and CCP v1 (migration is not completed yet).

You can also revert the provisioning stage in case you need it.

    # python main.py --subscriptions 1000010 --stage rollback
    2018-08-22 13:30:16.590 [DEBUG] Call openapi.OpenAPI method to get PACI settings
    2018-08-22 13:30:16.827 [DEBUG] rolling back migrated subscriptions [1000010]
    2018-08-22 13:30:16.828 [DEBUG] [rollback] these of specified subscriptions are in 'provisioned' state and could be reverted back [1000010]
    2018-08-22 13:30:17.942 [DEBUG] [rollback] subscription 1000010 has been rolled back to CI model

After reverting, your subscriptions will return to the injected stage with no provisioned CS resources, no UX1 panel. The server will be active and manageable from old UI.

Step 7. Insertion of CS Hosting resource rates in service plan.

  1. In Billing PCP, find the CI Hosting service plan.
  2. Add CS Hosting resource rates to it and set prices. As a result, you get a service plan with both CI and CS Hosting resource rates while the respective existing subscriptions do not have CS resources.

IMPORTANT: Please make sure that your CS hosting resources are configured in BA as measurable, in accordance with the following instructions.

Step 8. Final Stage.

Run the main.py with the option --stage finalize. This action will turn off the old UI, and the CI Hosting subscriptions with changes in the CI Hosting service plan.

    # python main.py --subscriptions 1000053 1000054 --stage finalize

    2018-07-11 11:01:43.956 [DEBUG] getting status of subscriptions and migration itself
    |subscription_id|       |migration_status|      |oaci_status|   |migration_type|
    1000053                 migrated                active          regular
    1000054                 migrated               was_active      regular

Now your CI Hosting subscriptions are switched to the CS Hosting model (APS-based). Customers can now manage these subscriptions in UX1 panel.

Migration in One Run

Please use this migration approach only for staging testing.

If you want to migrate one or few subscriptions at a time, and you don't want to run every stage, there is a stage called 'fast'.

  1. Perform the manual steps (add CS resources in the service template, add and set up resource rates in the service plan).
  2. Run the main.py with the option --stage fast:

    # python main.py  --subscriptions 1002000 --stage fast
    2018-08-24 15:43:31.078 [DEBUG] [finalize] all old UI has been turned off for subscription 1002000
    2018-08-24 15:43:31.079 [DEBUG] [finalize] finished migration for subscriptions
    

Partial Switch to CS Hosting (no provisioning of CS resources)

You might have non-active CI subscriptions that do not need the CS resources and UX1 panel now. They also don't need CI Hosting UI (CCP v1) to be available. These subscriptions will need a possibility to be switched to the CS Hosting model in future.

In that case, there is a method of CI > CS Hosting migration during which the CS resources will be inserted into CI subscriptions, but they will not be provisioned, and CI usage reporting and CI UI will be turned off.

A provider will have a possibility to provide the CS Hosting to the customer at any time.

  1. Perform the manual steps (add CS resources in the service template, add and set up resource rates in the service plan).
  2. Run the main.py with the option --stage evaluation.

    # python main.py  --subscriptions 1002001 --stage evaluation
    2018-08-24 15:51:30.671 [DEBUG] [evaluation] parsed subscriptions [1002001]
    
  3. Run the main.py with the option --stage only-ui.

    # python main.py  --subscriptions 1002001 --stage only-ui
    2018-08-24 15:51:40.727 [DEBUG] [fast_turnoff_ui] finished migration for subscriptions
    

Migration in Batches

By default, migration will be performed with the batch size 100 because the parameter default_batch_size = 100 in the config.py file. But you can adjust the size of a batch by specifying the --batch option.

In the following example, four subscriptions are passed for processing to turn off the UI only, but the batch size is defined as 1. As a result, the list was reduced to only 1 subscription.

# python main.py  --subscriptions 1002001 1002002 1002003 1002004 --stage only-ui --oaci inactive --batch 1
2018-09-26 11:09:21.266 [DEBUG] Call openapi.OpenAPI method to get PACI settings
2018-09-26 11:09:21.432 [DEBUG] batch size of current migration is: 1, reducing list of subscriptions to that size
2018-09-26 11:09:22.648 [DEBUG] [evaluation] parsed service templates and virtual server templates
2018-09-26 11:09:22.839 [DEBUG] [evaluation] parsed subscriptions [1002001]

|subscription_id|       |migration_status|      |oaci_status|   |migration_type|
1002001                 evaluated                inactive          only-ui

Troubleshooting

  • Logs are stored in /var/log/pa/ci2cs
  • Logs are created for every script launch: stage or status
  • Status db pypaci.db is stored in /var/log/pa/ci2cs for backup purpose

Internal content

Link on internal Article