Search Engine: Elastic

Article ID: 134014, created on May 13, 2019, last review on May 13, 2019

  • Applies to:
  • Operations Automation

Symptoms

Office 365 became unmanageable:

  1. All operations related to Office 365 application in CCPv1/UX1 and Online Store are hanging or failing with the message unknown error;
  2. All tasks related to Office 365 APS are failing with different errors;
  3. sitelog on Office 365 APS endpoint is filled with the error messages like the following:

    2019-05-06 08:51:43,092 <3836> [  93] ERROR aps_endpoint: AzureAuthorizationTokenHttp  server returned an web exception 'The remote server returned an error: (400) Bad Request.'.
    System.Net.WebException: The remote server returned an error: (400) Bad Request.
       at System.Net.HttpWebRequest.GetResponse()
       at Parallels.Office365.Gateway.BaseHttpWebRequest.ExecuteAndHandleResponse[TResponse,TErrorResponse,TException](String Prefix, HttpWebRequest webRequest, String requestBodyForPrint, IJsonSettings jsonSettings, Func`4 errorAction) in c:\inetpub\wwwroot\O365AppPanama\App_Code\HttpWebRequest\BaseHttpWebRequest.cs:line 226
    2019-05-06 08:51:43,092 <3836> [  93] ERROR aps_endpoint: Server returned body '{"error":"invalid_grant","error_description":"AADSTS700082: The refresh token has expired due to inactivity. The token was issued on 2019-02-03T15:22:19.6643819Z and was inactive for 90.00:00:00.\r\nTrace ID: 75718cf2-3f05-46f2-83b8-d96ca89c2e00\r\nCorrelation ID: 3617a41b-bc3a-47c8-a646-493dd1204692\r\nTimestamp: 2019-05-06 13:51:58Z","error_codes":[700082],"timestamp":"2019-05-06 13:51:58Z","trace_id":"75718cf2-3f05-46f2-83b8-d96ca89c2e00","correlation_id":"3617a41b-bc3a-47c8-a646-493dd1204692"}'.
    
  4. Operations to update refresh token for Office365 APS instance via PCP are hanging;
  5. console.log on the Management Node is filled with the following messages:

    190503 01:26:18,441 ERROR [io.undertow.request] (aps-service task-229) UT005023: Exception handling request to /aps/2/resources/555bd77d-dd7f-4224-ac58-32508c842fa0/getMSSubscriptionsInfoFast: com.parallels.pa.service.aps.api.APSEndpointExecutionException: AADSTS700082: The refresh token has expired due to inactivity. The token was issued on 2019-02-02T05:28:03.6345003Z and was inactive for 90.00:00:00.
    ...
    190503 02:43:39,280 ERROR [io.undertow.request] (aps-service task-66) UT005023: Exception handling request to /aps/2/resources/72dc1c7e-c4ba-4fcd-ba1f-2140b4014630/checkAccount: com.parallels.pa.service.aps.api.APSEndpointExecutionException: Could not invoke endpoint url 'https://o365aeh.domain.tld:443/Office365gwAppcspmxpr/aps/office365app/72dc1c7e-c4ba-4fcd-ba1f-2140b4014630/checkAccount?accountId=1000003&planId=-1' of application instance with UUID '46223094-dea3-45b8-a1ff-d30995600fb0'. RESTEASY004655: Unable to invoke request
    
  6. In rare cases all operations on the OA platform became unmanageable along with presence of the big amount of the aforementioned errors in console.log and messages in core.log for lost transactions:

    May  3 09:49:30.903 : ERR [rest:542821 1:16536:7fadcf776700 SAAS ]: [ resolveCosControl] [APSC] The transaction with id '1540609877' is not found by unknown reason
    

Cause

Expired token for Office 365 APS instances causes APS and Platform performance degradation due to software-related issue APSA-21162.

Resolution

Contact your Technical Account Manager or PTA team in order to trace the status of the issue APSA-21162.

To workaround the issue in case of only Office 365 unavailability:

  1. Cancel periodic tasks "Office 365 Synchronization with Office 365 Portal" for all instances of Office 365 application in PCP > Operations > Tasks > Periodic Tasks tab;
  2. Restart IIS service on the Office 365 APS endpoint server:

    iisreset
    
  3. Update refresh token for APS instances via PCP;
  4. Run previously canceled periodic tasks;

In case of whole platform unavailability with aforementioned symptoms it will be also required to restart OA services on the Management Node.

5356b422f65bdad1c3e9edca5d74a1ae 8fc71f07abe5b233fea1ae0377cd5e3d

Email subscription for changes to this article
Save as PDF