Symptoms

During the Synchronize resource usage for APS application task run for some APS package, REST API requests invoked at the same time during provisioning / modification requests to APS resources become unstable and may take a very long time or fail:

 Oct 25 06:28:50.929 : INF [openapi:131387 1:18696:7f73171ef700 SAAS ]: [ HTTP] OUT  10.253.20.9  [2082] "POST /backupservice/globals/c29e5371-29e5-4438-9753-713a1875cdad/tenant HTTP/1.1"  [0] "204 No Content"  299901ms

or:

Oct 15 16:28:12.512 : ERR [rest:23287 1:3942:7f51e81c1700 SAAS 799579269]: [ APSC] REST Error: PUT /aps/2/application/globals/c29e5371-29e5-4438-9753-713a1875cdad code=500: ERROR:  canceling statement due to statement timeout
CONTEXT:  while locking tuple (72,52) in relation "aps_resource"

The endpoint has default Apache configuration that allows 10 simultaneous connections:

 [root@endpoint ~]# apachectl -M | egrep "prefork|worker"
 ...
  mpm_prefork_module (static)

 # prefork MPM
 # StartServers: number of server processes to start
 # MinSpareServers: minimum number of server processes which are kept spare
 # MaxSpareServers: maximum number of server processes which are kept spare
 # ServerLimit: maximum value for MaxClients for the lifetime of the server
 # MaxClients: maximum number of server processes allowed to start
 # MaxRequestsPerChild: maximum number of requests a server process serves
 <IfModule prefork.c>
 StartServers       1
 MinSpareServers    1
 MaxSpareServers    5
 ServerLimit       10
 MaxClients        10
 MaxRequestsPerChild  4000
 </IfModule>

Cause

The Synchronize resource usage for APS application task runs in 16 threads by default and, in case of a large amount of processed APS resources, makes the Endpoint unavailable for any other requests, as it allows less threads to be executed at the same time.

The behavior is planned to be optimized in scope of APS-43355 request.

Resolution

  1. Tune Apache configuration on the Endpoint: modify /etc/httpd/conf.d/httpd.conf file and increase ServerLimit and MaxClients parameters to 30.
  2. Schedule the Synchronize resource usage for APS application task to run during non-business time, when there is a minimum amount of provisioning requests.

Internal content