[Office 365] Cannot Create Subscription: "Customer subscriptions are not ready yet" : CloudBlue Technical Support

Symptoms

Operations Automation (OA) background task to provision a new Office 365 subscription fails with this timeout: Method edit_customercalling failed. Reason:Timeout exception. Customer subscriptions are not ready yet, waiting for request completion.

The task also contains an XML request with a subscription ID that is not ready, as in the example below:

Task name               APS application 'Microsoft Office 365', id 101, instance 102 -> service 'customer', instance 103: executing configuration script, site 'http://portal.customer.com/'
Output  Application configuration script reported errors: '
Method edit_customer calling failed. Reason:Timeout exception. Customer subscriptions are not ready yet, waiting for request completion. _**XML Payload**_: _Request_: <s:Envelope xmlns:a="http://www.w3.org/2005/08/addressing" xmlns:s="http://www.w3.org/2003/05/soap-envelope"> <s:Header> <a:Action s:mustUnderstand="1">http://www.microsoftonline.com/MOSI/1/0/action/PlaceOrder</a:Action> <a:MessageID>urn:uuid:12345678-42b8-abcd-9d3f-b7a0fce2bd78</a:MessageID> <a:ReplyTo> <a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address> </a:ReplyTo> </s:Header> <s:Body> <PlaceOrder xmlns="http://www.microsoftonline.com/MOSI/1/0/service/"> <request xmlns:d4p1="http://www.microsoftonline.com/MOSI/1/0/messages/" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"> _**<d4p1:CustomerId>abcdef09-86ee-1234-8ba7-98765432b6b7</d4p1:CustomerId>**_ <d4p1:OfferId>1017D7F3-6D7F-4bfa-BDD8-79BC8F104E0C</d4p1:OfferId> <d4p1:PartnerId>0</d4p1:PartnerId> <d4p1:Quantity>50</d4p1:Quantity> <d4p1:RelatedSubscription i:nil="true" /> <d4p1:RequestId>98765432-48c9-dcba-866c-123456fdfe00</d4p1:RequestId> _**<d4p1:SubscriptionId>dcba0987-1ba6-1234-a681-567890b36004</d4p1:SubscriptionId>**_ </request> </PlaceOrder> </s:Body> </s:Envelope> _Response_: <s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://www.w3.org/2005/08/addressing" xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"> <s:Header> <a:Action s:mustUnderstand="1">http://www.microsoftonline.com/MOSI/1/0/action/PlaceOrderResponse</a:Action> <a:RelatesTo>urn:uuid:12345678-42b8-abcd-9d3f-b7a0fce2bd78</a:RelatesTo> <ActivityId CorrelationId="12345678-90e9-4b88-abcd-e28e277c32e1" xmlns="http://schemas.microsoft.com/2004/09/ServiceModel/Diagnostics">00000000-0000-0000-0000-000000000000</ActivityId> <o:Security s:mustUnderstand="1" xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"> <u:Timestamp u:Id="_0"> <u:Created>2012-11-16T17:25:52.693Z</u:Created> <u:Expires>2012-11-16T17:30:52.693Z</u:Expires> </u:Timestamp> </o:Security> </s:Header> <s:Body> <PlaceOrderResponse xmlns="http://www.microsoftonline.com/MOSI/1/0/service/" /> </s:Body> </s:Envelope>'

Cause

Odin Service Automation is waiting for the provisioning operations for the subscription in question to be completed or for the callback from Microsoft Online Portal.

There is a record in the waiting_subscriptions table in the [MOSI Gateway Database][1] for the problem subscription, indicating that a callback about the previous operation's completion was not received by the MOSI Gateway.

This can indicate a number of things:

Provisioning has not been completed on the Office 365 side and Microsoft has not sent a callback to Parallels about completing the provisioning. Provisioning can take quite a while in certain cases.
Provisioning has been completed on the Office 365 side, but OA has not received a callback because Microsoft did not send it or it was lost.

When a callback about subscription provisioning is received by the MOSI Gateway, this will remove the record about the problem subscription from the waiting_subscriptions table and the failed OA task will complete at the next execution.

Resolution

Verify whether provisioning on the Microsoft side has been completed:

Check the record in the waiting_subscriptions table in the MOSI Gateway Database to understand what the desired outcome is.

In the example below, OA wants to provision a new subscription for 50 seats of the offer 1017D7F3-6D7F-4bfa-BDD8-79BC8F104E0C, also known as ENTERPRISEPACK, and Plan E3 (see the Knowledgebase article #114227 [Office 365] Configuring Offer Mappings on Office 365 Gateway Host for the list of Microsoft Office 365 offers):

sqlite> SELECT * FROM waiting_subscriptions WHERE id = "dcba0987-1ba6-1234-a681-567890b36004"
dcba0987-1ba6-1234-a681-567890b36004
{
    "_waiting_for_": "**_Create_**:
Request:
<s:Envelope xmlns:a=\"http://www.w3.org/2005/08/addressing\" xmlns:s=\"http://www.w3.org/2003/05/soap-envelope\">
  <s:Header>
    <a:Action s:mustUnderstand=\"1\">http://www.microsoftonline.com/MOSI/1/0/action/PlaceOrder</a:Action>
    <a:MessageID>urn:uuid:12345678-42b8-abcd-9d3f-b7a0fce2bd78</a:MessageID>
    <a:ReplyTo>
      <a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address>
    </a:ReplyTo>
  </s:Header>
  <s:Body>
    <PlaceOrder xmlns=\"http://www.microsoftonline.com/MOSI/1/0/service/\">
      <request xmlns:d4p1=\"http://www.microsoftonline.com/MOSI/1/0/messages/\" xmlns:i=\"http://www.w3.org/2001/XMLSchema-instance\">
        _**<d4p1:CustomerId>abcdef09-86ee-1234-8ba7-98765432b6b7</d4p1:CustomerId>**_
        <d4p1:OfferId>1017D7F3-6D7F-4bfa-BDD8-79BC8F104E0C</d4p1:OfferId>
        <d4p1:PartnerId>0</d4p1:PartnerId>
        <d4p1:Quantity>50</d4p1:Quantity>
        <d4p1:RelatedSubscription i:nil=\"true\" />
        <d4p1:RequestId>98765432-48c9-dcba-866c-123456fdfe00</d4p1:RequestId>
        _**<d4p1:SubscriptionId>98765432-48c9-dcba-866c-123456fdfe00</d4p1:SubscriptionId>**_
      </request>
    </PlaceOrder>
  </s:Body>
</s:Envelope>
Response:
<s:Envelope xmlns:s=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:a=\"http://www.w3.org/2005/08/addressing\" xmlns:u=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd\">
  <s:Header>
    <a:Action s:mustUnderstand=\"1\">http://www.microsoftonline.com/MOSI/1/0/action/PlaceOrderResponse</a:Action>
    <a:RelatesTo>urn:uuid:12345678-42b8-abcd-9d3f-b7a0fce2bd78</a:RelatesTo>
    <ActivityId CorrelationId=\"12345678-90e9-4b88-abcd-e28e277c32e1\" xmlns=\"http://schemas.microsoft.com/2004/09/ServiceModel/Diagnostics\">00000000-0000-0000-0000-000000000000</ActivityId>
    <o:Security s:mustUnderstand=\"1\" xmlns:o=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd\">
      <u:Timestamp u:Id=\"_0\">
        <u:Created>2012-11-16T17:25:52.693Z</u:Created>
        <u:Expires>2012-11-16T17:30:52.693Z</u:Expires>
      </u:Timestamp>
    </o:Security>
  </s:Header>
  <s:Body>
    <PlaceOrderResponse xmlns=\"http://www.microsoftonline.com/MOSI/1/0/service/\" />
  </s:Body>
</s:Envelope>"

The Office 365 subscription ID dcba0987-1ba6-1234-a681-567890b36004 used in the SQL query above must be replaced with the actual ID, which can be taken from the XML Payload in the failed OA task provided in the Symptoms section:

<d4p1:SubscriptionId>dcba0987-1ba6-1234-a681-567890b36004</d4p1:SubscriptionId>

As you can see from the above output, the MOSI Gateway is waiting for completion of the PlaceOrder request - it is waiting for the subscription to be created on the Microsoft side.

Check in the Microsoft Online Portal web interface or in PowerShell whether the problem subscription was created in Office 365 and its limits. The example below shows that the subscription already exists in Microsoft with the correct number of seats:
```
PS> Get-MsolSubscription -TenantID (Get-MsolPartnerContract -Domain _**customersubdomain.onmicrosoft.com**_).TenantID | select skupartnumber, status, totallicenses

SkuPartNumber                                  Status             TotalLicenses
-------------                                  ------             -------------
ENTERPRISEPACK                                Enabled                        50
```
The domain customersubdomain.onmicrosoft.com in the PowerShell cmdlet above must be replaced with the actual customer domain in Office 365. The domain can be found in the OA Customer Control Panel (CCP) in the login of the administrative user, which is created along with the customer account in Office 365. The login of the admin user is displayed in the CCP at Microsoft Office 365 > Configuration > Settings > Generated Values. See this screenshot:

In fact, any domain the customer delegated to Microsoft can be used in the PowerShell cmdlet above.

Further actions depend on the result of checking if the subscription was properly created in the Office 365 cloud:

If provisioning has not been completed, wait for it to complete on the Microsoft side. As soon as provisioning is completed in Office 365, Microsoft will send a callback to the MOSI Gateway and the failed OA task will complete at the next run.
If provisioning has been completed in Office 365 and OA did not receive a callback:

2.1. Contact Microsoft support to clarify if the callback was sent for this subscription. In some cases Microsoft would require the list of incoming/outgoing callbacks for the customer - they may be located inside MOSI GW folder in Callbacks folder.

2.2. If Microsoft claims they sent the callback, contact Odin technical support and provide the above investigation results.

Workaround

If the issue is urgent or you do not wish to contact Microsoft support, you can remove the record for the problem subscription from the waiting_subscriptions table in the MOSI Gateway Database manually, then resubmit the failed task in OA:

    sqlite> delete from waiting_subscriptions where id = "dcba0987-1ba6-1234-a681-567890b36004";

After that, run the failed task in OA.

Internal content

Link on internal Article