Symptoms
Provisioning of new PVPS environments fails with the following error message in the output of corresponding task "Provisioning "PACIVirtualEnvironment" for APS application Cloud Infrastructure":
APS Application Error:
406 Not Acceptable
[ApplicationUnknownError] P100001: Operation failed; details: [Failed to perform DELETE operation].
Body: {
"error": "ApplicationUnknownError",
"message": "P100001: Operation failed; details: [Failed to perform DELETE operation]",
"http_request": "POST /vps/api/environments"
}
Vm2vf service on Virtuozzo nodes is running but does not write logs. The latest messages in /var/log/pa/vps.vm2vf.log are:
ERROR CallbackPublisher [ActiveMQ Transport: tcp:///10.128.40.10:61616@48358] - JMS transport interrupted
ERROR CallbackPublisher [ActiveMQ Transport: tcp:///10.128.40.10:61616@48361] - JMS transport interrupted
After stop, vm2vf service on Virtuozzo node is unable to start. No additional error output is given.
Cause
Broken communication between Virtuozzo nodes and OACI IM host on port 61616. Connection issues or firewall rules blocked communication between Virtuozzo hosts and IM via JMS protocol and IM was not able to send any messages to corresponding vm2vf service on nodes. Communication between IM <-> Virtuozzo on port 61616 on a backnet interface is a new requirement of OA 7.0 since vm2vf service was de-centralized and moved to Virtuozzo nodes.
Resolution
If connectivity is restricted by firewall on IM host or Virtuozzo node, add corresponding iptables rules. For example:
# iptables -I INPUT 1 -s $VZ_node_ip/32 -p tcp -m tcp --dport 61616 -j ACCEPT
If port is blocked on Virtuozzo node, swap $VZ_node_ip with OACI IM host IP.