Symptoms
After update of Windows Azure Pack environment to UR6 all WAP-related tasks fail with errors like:
Task ID 37887991
Queue name aps_application_instance54406 subscription_id1
Task name Provisioning "AddonInstallationTask" for APS application Windows Azure Pack APS Package
Task description Provisioning resource 254ee2b1-553d-47d2-90fd-75e4a41df2ca of type 'addon-task' (http://www.parallels.com/infrastructure/internal/addon-task/1.0) for APS application 'Windows Azure Pack APS Package-1.4-297'
Subscription ID 1
Queue status Failed
Start not earlier than Jun-08-2015 15:35
Method name taskExecuteWorkflow on SCREF:SaaSManagerTasks:0
Last execution output
APS Application Error:
406 Not Acceptable
[ApplicationUnknownError] Provisioning: resource 254ee2b1-553d-47d2-90fd-75e4a41df2ca of type 'addon-task' (http://www.parallels.com/infrastructure/internal/addon-task/1.0) for APS application 'Windows Azure Pack APS Package-1.4-297': P100001: Operation failed; details: [Failed to synchronization subscription , 1082631].
Body: {
"error": "ApplicationUnknownError",
"message": "P100001: Operation failed; details: [Failed to synchronization subscription , 1082631]",
"http_request": "POST /addon-task to '0.0.0.0:4485'"
}
The following error apper in /var/log/WAP/wap.log
:
2015-06-08 16:25:07,810 INFO LoggingFilter [apsc(5)] - 70 * [WAP->AZURE] Client out-bound request
70 > POST https://wapadminapi.ccccloud.com/subscriptions/a7056607-9fee-4bdc-9a0a-6900e37079d2/addons
70 > Authorization: <hidden>
70 > X-ReqId: 548
70 > Accept: text/plain,application/json
70 > Content-Type: application/json
{
"AddOnId" : "GBAzuhskh0ild"
}
2015-06-08 16:25:16,201 INFO LoggingFilter [apsc(5)] - 70 * [WAP<-AZURE] Client in-bound response
70 < 500
70 < X-AspNet-Version: 4.0.30319
70 < Date: Mon, 08 Jun 2015 13:25:16 GMT
70 < Content-Length: 206
70 < Expires: -1
70 < Content-Type: application/json; charset=utf-8
70 < X-Powered-By: ASP.NET
70 < Server: Microsoft-IIS/8.5
70 < Pragma: no-cache
70 < Cache-Control: no-cache
70 <
{"Code":"ErrorFromUnderlyingNotificationSubscribers","Message":"One or more errors occurred while contacting the underlying notification subscribers. The operation may be partially completed.","Details":[]}
The following error apper in Event Viewer on the WAP node in MgmtSvc-Usage\Operational
log:
Publishing notification to an end point failed with the exception: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 172.16.32.41:4484
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.WindowsAzure.Server.Usage.Service.NotificationRequestSender.<PostAsync>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.WindowsAzure.Server.Usage.Service.NotificationRequestHandler.<PublishNotificationsAsync>d__1`1.MoveNext()
Cause
Rollup installation changed settings on WAP node, it could be checked with command below:
PS C:\Windows\system32> get-MgmtSvcNotificationSubscriber
Name : Billing
Enabled : True
SubscriberType : BillingService
Endpoint : https://wap-aps:4484/wap/billing
AuthenticationMode : Basic
AuthenticationUsername : wap
AuthenticationPassword : pass_word
Port was changed to 4484
instead of 4486
and trailing slash was removed
Resolution
Apply proper configuration with:
PS C:\Windows\system32> Set-MgmtSvcNotificationSubscriber -Name Billing -SubscriberType BillingService -Enabled $true -Endpoint 'https://wap-aps:4486/wap/billing/' -AuthenticationMode Basic -AuthenticationPassword 'pass_word' -AuthenticationUsername 'wap' -force
And restart IIS after:
PS C:\Windows\system32> iisreset
Attempting stop...
Internet services successfully stopped
Attempting start...
Internet services successfully restarted