Symptoms
During the Synchronize resource usage for APS application task run for some APS package, REST API requests invoked at the same time during provisioning / modification requests to APS resources become unstable and may take a very long time or fail:
Oct 25 06:28:50.929 : INF [openapi:131387 1:18696:7f73171ef700 SAAS ]: [ HTTP] OUT 10.253.20.9 [2082] "POST /backupservice/globals/c29e5371-29e5-4438-9753-713a1875cdad/tenant HTTP/1.1" [0] "204 No Content" 299901ms
or:
Oct 15 16:28:12.512 : ERR [rest:23287 1:3942:7f51e81c1700 SAAS 799579269]: [ APSC] REST Error: PUT /aps/2/application/globals/c29e5371-29e5-4438-9753-713a1875cdad code=500: ERROR: canceling statement due to statement timeout
CONTEXT: while locking tuple (72,52) in relation "aps_resource"
The endpoint has default Apache configuration that allows 10 simultaneous connections:
[root@endpoint ~]# apachectl -M | egrep "prefork|worker"
...
mpm_prefork_module (static)
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers 1
MinSpareServers 1
MaxSpareServers 5
ServerLimit 10
MaxClients 10
MaxRequestsPerChild 4000
</IfModule>
Cause
The Synchronize resource usage for APS application task runs in 16 threads by default and, in case of a large amount of processed APS resources, makes the Endpoint unavailable for any other requests, as it allows less threads to be executed at the same time.
The behavior is planned to be optimized in scope of APS-43355 request.
Resolution
- Tune Apache configuration on the Endpoint: modify
/etc/httpd/conf.d/httpd.conf
file and increase ServerLimit and MaxClients parameters to 30. - Schedule the Synchronize resource usage for APS application task to run during non-business time, when there is a minimum amount of provisioning requests.