Symptoms
VPS hanged with "configuring firewall" or any other transitional status. This status appeared again in 20 mins after VE was restarted. Recreation of VE also may hang just as any other operations with VE.
As one of other symptoms, backup creation process for VE may report failure while backup has been done correctly on the storage node. That will generate a lot of unregistered backups that consume disk space.
In logs one may found the errors listed below.
/var/log/terracotta/terracotta-server.log:
[TC Memory Monitor] WARN tc.operator.event - NODE : localhost Subsystem: MEMORY_MANAGER Message: Current Memory usage(93%) crossed critical threshold(90%).
/var/log/PACI-vm2vf.log:
INFO NativeVm2VfCode [Thread-85851] - [59780:484244] INF pthread_cond_timedwait [110]: cbp 0x7f1e380b4be0 job 0x4a23a [__PrlVm_Commit] index [9060] (__configure_firewall_cb, firewall.c, 248) @[common/generic_sdk_cb_tools.c][139][cbps_timeout_sleep][26479])
WARN LocalVm2Vf [Thread-85854] - CORBA exception OBJECT_NOT_EXIST:Server-side Exception: caught at 'in_progress' method invocation
WARN LocalVm2Vf [Thread-85862] - CORBA exception OBJECT_NOT_EXIST:Server-side Exception: caught at 'in_progress' method invocation
And also exception:
INFO CallbackProxy [RequestProcessor-4] - vm2vf.callbackproxy InvocationTargetException exception java.lang.NullPointerException: null
Cause
Above shows that request was sent by IM but when callback returned from vm2vf, object that should receive it, was already destroyed and IM was not able to process callback.
This behavior is caused by CCU-10599 and CCU-11519.
Resolution
Fix the IM configuration file and terracotta startup file manually as a workaround. Set Java Heap limit to 2GB as shown below:
/usr/local/share/PACI-im/scripts/im-daemon:
Line 47:
-Xmx2048m -Xms2048m -XX:+HeapDumpOnOutOfMemoryError ${JAVA_CONSOLE_SETTINGS}\
for terracotta startup script:
# grep -n Xms /usr/local/terracotta-3.6.1/bin/start-tc-server.sh
60:${JAVA_COMMAND} -Xms512m -Xmx512m -XX:+HeapDumpOnOutOfMemoryError \
Full restart of IM services and purging terracotta cache are required after that:
service PACI-im stop
service terracotta stop
service activemq-server stop
service terracotta clearcache
service activemq-server start
service terracotta start
service PACI-im start