Symptoms

After update of Windows Azure Pack environment to UR6 all WAP-related tasks fail with errors like:

Task ID 37887991
Queue name      aps_application_instance54406 subscription_id1
Task name       Provisioning "AddonInstallationTask" for APS application Windows Azure Pack APS Package
Task description Provisioning resource 254ee2b1-553d-47d2-90fd-75e4a41df2ca of type 'addon-task' (http://www.parallels.com/infrastructure/internal/addon-task/1.0) for APS application 'Windows Azure Pack APS Package-1.4-297'
Subscription ID 1
Queue status    Failed
Start not earlier than  Jun-08-2015 15:35
Method name     taskExecuteWorkflow on SCREF:SaaSManagerTasks:0
Last execution output   

APS Application Error:
406 Not Acceptable
[ApplicationUnknownError] Provisioning: resource 254ee2b1-553d-47d2-90fd-75e4a41df2ca of type 'addon-task' (http://www.parallels.com/infrastructure/internal/addon-task/1.0) for APS application 'Windows Azure Pack APS Package-1.4-297': P100001: Operation failed; details: [Failed to synchronization subscription , 1082631].

Body: {
"error": "ApplicationUnknownError",
"message": "P100001: Operation failed; details: [Failed to synchronization subscription , 1082631]",
"http_request": "POST /addon-task to '0.0.0.0:4485'"
}

The following error apper in /var/log/WAP/wap.log:

2015-06-08 16:25:07,810 INFO  LoggingFilter [apsc(5)] - 70 * [WAP->AZURE] Client out-bound request
70 > POST https://wapadminapi.ccccloud.com/subscriptions/a7056607-9fee-4bdc-9a0a-6900e37079d2/addons
70 > Authorization: <hidden>
70 > X-ReqId: 548
70 > Accept: text/plain,application/json
70 > Content-Type: application/json
{
  "AddOnId" : "GBAzuhskh0ild"
}

2015-06-08 16:25:16,201 INFO  LoggingFilter [apsc(5)] - 70 * [WAP<-AZURE] Client in-bound response
70 < 500
70 < X-AspNet-Version: 4.0.30319
70 < Date: Mon, 08 Jun 2015 13:25:16 GMT
70 < Content-Length: 206
70 < Expires: -1
70 < Content-Type: application/json; charset=utf-8
70 < X-Powered-By: ASP.NET
70 < Server: Microsoft-IIS/8.5
70 < Pragma: no-cache
70 < Cache-Control: no-cache
70 <
{"Code":"ErrorFromUnderlyingNotificationSubscribers","Message":"One or more errors occurred while contacting the underlying notification subscribers. The operation may be partially completed.","Details":[]}

The following error apper in Event Viewer on the WAP node in MgmtSvc-Usage\Operational log:

Publishing notification to an end point failed with the exception: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 172.16.32.41:4484
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
   at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.WindowsAzure.Server.Usage.Service.NotificationRequestSender.<PostAsync>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.WindowsAzure.Server.Usage.Service.NotificationRequestHandler.<PublishNotificationsAsync>d__1`1.MoveNext()

Cause

Rollup installation changed settings on WAP node, it could be checked with command below:

PS C:\Windows\system32> get-MgmtSvcNotificationSubscriber

Name                   : Billing
Enabled                : True
SubscriberType         : BillingService
Endpoint               : https://wap-aps:4484/wap/billing
AuthenticationMode     : Basic
AuthenticationUsername : wap
AuthenticationPassword : pass_word

Port was changed to 4484 instead of 4486 and trailing slash was removed

Resolution

Apply proper configuration with:

PS C:\Windows\system32> Set-MgmtSvcNotificationSubscriber -Name Billing -SubscriberType BillingService -Enabled $true -Endpoint 'https://wap-aps:4486/wap/billing/' -AuthenticationMode Basic -AuthenticationPassword 'pass_word' -AuthenticationUsername 'wap' -force

And restart IIS after:

PS C:\Windows\system32> iisreset

Attempting stop...
Internet services successfully stopped
Attempting start...
Internet services successfully restarted

Internal content