Windows Azure Pack task fails: One or more errors occurred while contacting the underlying notification subscribers. The operation may be partially completed. : CloudBlue Technical Support

Symptoms

Task for provisioning of WAP subscription fails with the error:

An unexpected exception occurred while calling the notification subscriber

The following errors may be found in Event Viewer on the server when WAP Admin API is deployed:

Error: HttpStatusCode: InternalServerError, ErrorCode: ErrorFromUnderlyingNotificationSubscribers, ClientErrorMessage:One or more errors occurred while contacting the underlying notification subscribers. The operation may be partially completed.


Publishing notification to an end point failed with the exception:     System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.


Resource provider unexpected exception for fanout request with verb 'POST', operation name 'Fanout', request URI 'https://aaapi.dorukcloud.com:30022/notifications/subscriptions',  version '', client request Id '', server request Id '', status code 'InternalServerError', reason 'Internal Server Error', body 'An unexpected exception occurred while calling the notification subscriber.'.


Error:System.InvalidOperationException: No MediaTypeFormatter is available to read an object of type 'ResourceProviderError' from content with media type 'text/plain'. 
at System.Net.Http.HttpContentExtensions.ReadAsAsync[T](HttpContent content, Type type, IEnumerable`1 formatters, IFormatterLogger formatterLogger) at System.Net.Http.HttpContentExtensions.ReadAsAsync[T](HttpContent content, IEnumerable`1 formatters) at Microsoft.WindowsAzure.Server.Management.Core.RequestSenderExtensions.<EnsureSuccessResponseAsync>d__1f.MoveNext(), operationName:, version:, accept language:, subscription Id:, client request Id:, principal Id:, page request Id:, server request id:


Publishing notification to an end point failed with the exception: System.Net.Http.HttpRequestException: Response status code does not indicate success: 403 (Forbidden).

Cause

Issues with certificate or incorrect WAP_billing_password

Resolution

Such errors could occur due to multiple reasons. First of all, please login to WAP node via RDP and open the endpoint link in browser on this node (by default it is https://wap-aps:4486/wap/metadata.xml). The following cases can occur:

Certificate is shown with warnings. Then most likely it was incorrectly generated during deployment. Please repeat steps from WAP Deployment Guide (Section 3 Configure Windows Azure Pack, steps 10-26).
There are no certificate warnings, but some errors are shown in browser (403, 404, 500). Then it is necessary to check that POA management node (wap-aps) receives requests from WAP node. On MN node tail /var/log/WAP/wap.log while refreshing https://wap-aps:4486/wap/metadata.xml on WAP node.

If there is an error like "java.security.UnrecoverableKeyException: Cannot recover key" in wap.log check that it is possible to list certificates using the password from /usr/local/share/WAP/wap-config.xml:
```
# /usr/java/default/bin/keytool  -list -keystore /usr/local/share/WAP/WAPKeyStore -storepass <passwrod from wap-config.xml>
```
If there is nothing in wap.log, check that wap-aps resolves to POA MN node from WAP node

If there is "You are not authorized to perform this operation" in wap.log, it could mean that the incorrect password was specified during deployment when executing the command:
```
PS> Set-MgmtSvcNotificationSubscriber -Name Billing -SubscriberType BillingService -Enabled $true -Endpoint https://WAP-APS:4486/wap/billing/ -AuthenticationMode Basic -AuthenticationPassword '<wap_billing_pass>' -AuthenticationUsername 'wap' -force
```
To fix it, execute the command with the correct password (can be found in wap-config.xml) and restart IIS on WAP node.
If above-mentioned steps do not help, compare wap-config.xml with wap-config.xml.template to be sure that it has proper format (i.e. it contains all the sections that .template file has).

Important Do not forget to restart WAP service on POA MN each time you are modifying WAPKeyStore or wap-config.xml.

Internal content

Link on internal Article