Symptoms
An APS endpoint server faces rapid increase in memory usage on permanent basis.
netstat
shows a huge amount of connections hanging in TIME_WAIT state to Apache service (port :80), originating from OA Management Node:
[root@endpoint ~]$ netstat -aon | fgrep :80
tcp 0 0 192.0.2.2:8008 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 :::80 :::* LISTEN off (0.00/0/0)
tcp 0 0 ::ffff:192.0.2.2:80 ::ffff:192.0.2.10:38748 TIME_WAIT timewait (56.34/0/0)
tcp 0 0 ::ffff:192.0.2.2:80 ::ffff:192.0.2.10:38632 TIME_WAIT timewait (41.07/0/0)
tcp 0 0 ::ffff:192.0.2.2:80 ::ffff:192.0.2.10:38710 TIME_WAIT timewait (53.86/0/0)
...
There is a huge amount of GET requests for the APS applications, related to this Endpoint, recorded in /var/log/apsc.http.log
:
[root@mn ~]# grep -c myapspp /var/log/apsc.http.log
236739
Synchronize resource usage for instances of APS applications periodic task is set to be run with a very short period, e.g. 5 minutes.
Cause
By its design, Synchronize resource usage for instances of APS applications periodic task goes to each APS application endpoint and retrieves the details on resource usage of each APS resource it owns that includes APS counters, see Resource model in APS2.0: counters for more details.
Resolution
Reconfigure Synchronize resource usage for instances of APS applications periodic task to run once a day - the default and recommended period for this task.