Symptoms

A Billing Upgrade Order fails with the message after 10 minutes:

"Change Service Plan failed: Operations Automation is not available. Connection to 172.16.3.12:8440 has failed: Connection timeout. Please check network settings.".

Cause

This case is related to hard-coded timeout of 10 minutes on the Billing side. Corresponding pem.upgradeSubscription call to OA is still runing, despite of the fact, that order has been already failed. This case has been submitted to the Maintenance Team as internal issue with id APS-37640: "Impossible to migrate subscription from LSH to NG in case it contains many domains".

Resolution

Option 1:

  1. Resubmit order for provisioning;

  2. Find the beginning of upgradeSubscription transaction in poa.log:

    INF [openapi:XX.XX.XX>XX:30140 1:21869:ae7f4b70 OpenAPI ]: ==>>> Executing OpenAPI method 'pem.upgradeSubscription' with parameters: [{ 'custom_resource_limits' => [ { 'limit' => '0', 'resource_type_id' => 1000010, }, ... 'subscription_id' => XXXXXXX,'txn_id' => '123456', } ]

    Check that you have found correct request by parameter subscription_id. Note txn_id field value;

  3. Wait for result of pem.upgradeSubscription request in poa.log (request can be identified by openapi prefix (For example openapi:172.16.3.10:42153)

    Like this:

    INF [openapi:172.16.3.10:42153 1:22531:ae7f6b70 OpenAPI ]: <<<== Method 'pem.upgradeSubscription' is executed, with status:
ERR [openapi-firewall:ad8e7b70 1:22531:ad8e7b70 OpenAPI ]: [ iqxmlrpc-server] iqxmlrpc::Http_server_connection: Socket::send: Broken pipe

    Message "iqxmlrpc::Http_server_connection: Socket::send: Broken pipe" after pem.upgradeSubscription, means that this method was executed successfully, but you need to commit transaction.

  4. Perform manual commit for transaction:

    # curl -k -d@/tmp/txn.Commit.xml http://XX.XX.XX.XX:8440

<?xml version="1.0" encoding="UTF-8"?>
<methodResponse><...><name>status</name><value><i4>0</i4><...></methodResponse>

    Content of "txn.Commit.xml" (use the number from step 2):

    <?xml version="1.0"?>
<methodCall>
<methodName>txn.Commit</methodName>
<params>
    <param>
        <value>
            <struct>
                <member>
                    <name>txn_id</name>
                    <value><string>123456</string></value>
                </member>
            </struct>
        </value>
    </param>
</params>
</methodCall>

    You can find additional information about this API request in OSA API documentation.

  5. Resubmit billing order again. If the order fails again, but OA completed necessary actions (customer's subscription's ST has been changed), then complete order manually.

Option 2: (in rare cases option 1 may not work)

  1. Get the whole set of parameters of pem.upgradeSubscription call from poa.log and run it manually, example:

    [root@core ~]# cat upgradeSubscription.py
#!/usr/bin/env python
from poaupdater.openapi import OpenAPI

api = OpenAPI()

api.pem.upgradeSubscription(custom_resource_limits = [ { 'limit' : '1', 'resource_type_id' : 1000010, }, { 'limit' : '10485760', 'resource_type_id' : 1000150, }, { 'limit' : '209715200', 'resource_type_id' : 1000152, }, { 'limit' : '30', 'resource_type_id' : 1000168, }, { 'limit' : '30', 'resource_type_id' : 1000169, }, { 'limit' : '5', 'resource_type_id' : 1000176, }, { 'limit' : '3145728', 'resource_type_id' : 1000177, }, { 'limit' : '30', 'resource_type_id' : 1000186, }, { 'limit' : '30', 'resource_type_id' : 1000374, }, { 'limit' : '30', 'resource_type_id' : 1000376, }, { 'limit' : '50', 'resource_type_id' : 1000417, }, { 'limit' : '5', 'resource_type_id' : 1000419, }, { 'limit' : '200', 'resource_type_id' : 1000420, }, { 'limit' : '30', 'resource_type_id' : 1000437, }, { 'limit' : '30', 'resource_type_id' : 1000438, }, { 'limit' : '30', 'resource_type_id' : 1000439, }, { 'limit' : '30', 'resource_type_id' : 1000440, }, { 'limit' : '30', 'resource_type_id' : 1000591, }, { 'limit' : '30', 'resource_type_id' : 1001803, }, { 'limit' : '30', 'resource_type_id' : 1002626, }, { 'limit' : '0', 'resource_type_id' : 1002647, }, { 'limit' : '30', 'resource_type_id' : 1002670, }, { 'limit' : '50', 'resource_type_id' : 1002700, }, { 'limit' : '25000', 'resource_type_id' : 1002706, }, { 'limit' : '20', 'resource_type_id' : 1002718, }, { 'limit' : '100', 'resource_type_id' : 1002724, }, { 'limit' : '50', 'resource_type_id' : 1002730, }, { 'limit' : '1556480', 'resource_type_id' : 1002736, }, { 'limit' : '1556480', 'resource_type_id' : 1002742, }, { 'limit' : '10485760', 'resource_type_id' : 1002748, }, { 'limit' : '10485760', 'resource_type_id' : 1002754, }, { 'limit' : '1048576', 'resource_type_id' : 1002760, }, { 'limit' : '100', 'resource_type_id' : 1002776, }, { 'limit' : '0', 'resource_type_id' : 1003167, }, { 'limit' : '30', 'resource_type_id' : 1003200, }, { 'limit' : '1', 'resource_type_id' : 1003560, }, { 'limit' : '-1', 'resource_type_id' : 1003561, }, { 'limit' : '50', 'resource_type_id' : 1003562, }, { 'limit' : '1048576', 'resource_type_id' : 1003580, }, { 'limit' : '0', 'resource_type_id' : 1003587, }, { 'limit' : '0', 'resource_type_id' : 1003592, }, { 'limit' : '0', 'resource_type_id' : 1003595, }, { 'limit' : '0','resource_type_id' : 1003598, }, { 'limit' : '0', 'resource_type_id' : 1003601, }, { 'limit' : '0', 'resource_type_id' : 1003604, }, { 'limit' : '0', 'resource_type_id' : 1003637, }, { 'limit' : '0', 'resource_type_id' : 1003640, }, { 'limit' : '50', 'resource_type_id' : 1003750, }, { 'limit' : '0', 'resource_type_id' : 1003753, }, { 'limit' : '0', 'resource_type_id' : 1003756, }, { 'limit' : '0', 'resource_type_id' : 1004002, } ], service_template_id = 121, subscription_id = 1003020)

[root@core ~]# python upgradeSubscription.py
  2. Once the API call is completed successfully, re-submit the order.

Internal content

Link on internal Article