Symptoms
OACI IM was found unmanageable and according to /var/log/pa/vps.log, heap space of OACI IM JVM instance has been exceeded:
ERROR ServletHandler [Grizzly-worker(1)] - service exception:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236) ~[na:1.8.0_112]
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.8.0_112]
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.8.0_112]
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) ~[na:1.8.0_112]
The closest request to the crash was related to autoscale and resource usage history retrieval:
INFO CustomLoggingFilter [Grizzly-worker(1)] - 69613 * Server in-bound request
69613 > GET http://192.168.139.2:4465/paci/v1.0/of/1087335/ve/main/autoscale/history/2017-12-23%2014%3A20%20%2B0300/2018-03-24%2014%3A20%20%2B0300
Target VE was checked, but no outstanding records were found.
IM automatically saves heap dumps upon OutOfMemory error. Name and path to heap dump (*.hprof file) could be retrieved from vps.startup.log on IM node.
Cause
Heap usage distribution shows that the most of it were consumed by a line of text (byte array) which was processed inside of ResourceConsumptionAndAutoscaleHistoryExtImpl class. Checking the value which consumed that memory showed repeating <time-out/> in XML construct:
<resource-consumption-and-autoscale-history>
<time-out/>
<time-out/>
<time-out/>
Cause of this is an infinite loop which occurred due to CCU-17675.
Resolution
Since the loop that caused an issue is infinite, increase of heap will not help. There is no workaround for the issue, but the conditions for its occurrence are rare, so it should not appear frequently. CCU-17675 will be fixed in future product versions.
To obtain more information on CCU-17675 you may contact your Technical Manager.