Symptoms
Customer ordered new VPS subscription, but for some reasons (which are not the subject of this article) it creation has failed. Customer found that Sales Order was completed regardless of that. Along with order, provisioning task "Provisioning "PACIVirtualEnvironment" for APS application Cloud Infrastructure" was also completed successfully.
Chain of events could be restored from core.log and the most important part is following:
Provisioning operation failed on IM side. Callback with non-zero RC has reached OA:
DEBUG RESTCallbackListener [Grizzly(11)] - Got IM callback, uri = [/fdf3076b-c705-49e1-bd17-7c0209226e6c], rc = [-1]
DEBUG ConcurrentTimedMatcher [Grizzly(11)] - checkOut(/fdf3076b-c705-49e1-bd17-7c0209226e6c, ...)
ERROR EnvsResourceImpl [IMCallbackThread-5] - Failed to create environment
java.util.concurrent.CompletionException: com.parallels.iaas.exceptions.BadRequestException: P100001: Operation failed; details: [Failed to create VE]
This event was processed by OA, failed request has been handled, APS resource of VPS - removed with provisioning workflow item (that will be important in further steps):
INF [rest:34535 1:10766:7ff1f77fe700 SAAS ]: [ HTTP] IN 127.0.0.1:45714 Begin request: "DELETE /aps/2/application/environments/939213c9-48fa-4677-962d-2f9e8d47eef0"
MailMessengerBean sending mail message "server-1000372-1" VPS creation has failed to
...
DBG [rest:34535 1:10766:7ff1f77fe700 SAAS 451619256]: [ RDBMS] Prepare [0x7ff274026600]: DELETE FROM "aps_workflow_item" WHERE "aps_workflow_item"."resource_id" = $1
DBG [rest:34535 1:10766:7ff1f77fe700 SAAS 451619256]: [ RDBMS] Execute [0x7ff274026600]: 4922
After a couple of minutes provisioning task, that handles async provisioning, checked the workflow and came into conclusion that since no items left in workflow, provisioning is finished:
DBG [task:580294:41334 1:10766:7ff252fed700 SAAS ]: [ APSC] [WF] No one unprocessed Workitem found
Then task immediately completes.
Cause
This behavior recognized as software-related issue CCU-17343. Please use the provided ID to clarify the status of the request with your Technical Manager.
Resolution
The workaround to this issue is to change the APS VE resource limit of the affected subscription (in OA PCP) to 0 then to 1 - the VE then will be autoprovisioned.