OA Control Panel is down or inaccessible: Known issues and Troubleshooting : CloudBlue Technical Support

Symptoms

A provider or a customer cannot log into the Operations Automation (OA) Control Panel. The Control Panel either is not accessible at all, or shows error messages such as:

Fatal error

An unforeseen error occurred

System error

Service Unavailable The corresponding service is not available at the moment, try it within 2-3 minutes.

This article describes basic troubleshooting steps when the Control Panel is down or giving an internal server error.

Cause

There may be several causes of Paralels Operations Automation Control Panel unavailability:

Stopped or hung servers
Stopped or hung services
Lack of resources (memory, diskspace) on servers
Network connectivity issues (firewall)
Misconfigured services
Expired license

Below, you will find more details on checking for the above issues and troubleshooting them.

Resolution

Important: in order to allow root cause analysis of each OA Control Panel outage, collect debug details to be passed to the Odin Technical Support team. Use the following script:

oa_dump.bash

Launch it on both OA Management Node and UI node, when CP is inaccessible. The script will produce a .tar.bz2 archive inside /var/log/pa directory. Note: both nodes should have gdb package installed preliminarily.

# wget https://kb.cloudblue.com/Attachments/kcs-35818/oa_dump.bash
# bash oa_dump.bash

In case CP is completely down and huge CPU usage of Java process is observed on either of the nodes, run the script in 'dump' mode instead to collect a core dump file additionally. Note: in case of a live Control Panel, this mode will cause CP unavailability during the dump process:

# bash oa_dump.bash --heap

After collecting the debug data, proceed with restarting the services to see if it helps to restore CP functionality.

How to restart OA system services: UI, Management Node, Agents

If the services restart didn't help and you are using IDP, try restarting the IDP pod:

(get the pod name and delete the currently running pod so it could be recreated by K8s):

# kubectl get pod -n NAMESPACE | grep idp
idp-backend-7b45d4b7d8-6j95z   1/1     Running   4 (5d16h ago)   5d19h
# kubectl delete pod idp-backend-7b45d4b7d8-6j95z

Check the status of servers

Verify that all the servers listed below are up and running (the malfunctioning of any server below may affect the Control Panel). Start any servers that are down:

1. OA Management Node

2. OA Database server

3. OA UI (User Interface) server/cluster

Check the status of services

Make sure all required services are running on the corresponding servers.
1. OA Management Node:
```
# systemctl status pau pa-agent
```
  - If any of them is down, check this article for possible solution: https://cloudblue.freshdesk.com/support/solutions/articles/44002155089-service-pa-agent-fails-to-start-when-trying-to-restart-cbc-core-services
2. OA UI node:
```
# systemctl status pau
```
Make sure the OA system database service is running on the corresponding server:
```
# systemctl status postgresql-9.6
```

Check the status of APS booster

To check whether APS booster is enabled, do the following: 1. Connect to a UI node using SSH as root. 2. Run the following commands:

# sh <widlfy-working-directory>/bin/jboss-cli.sh
[standalone@localhost:9990 /] /system-property=com.parallels.pui.apsbooster.enabled:read-resource

If it is enabled, the output will be similar to this:

{
    "outcome" => "success",
    "result" => {"value" => "true"}
}

If the output value is not true, you can enable APS booster using the following commands:

[standalone@localhost:9990 /] /system-property=com.parallels.pui.apsbooster.enabled:remove
[standalone@localhost:9990 /] /system-property=com.parallels.pui.apsbooster.enabled:add(value=true)

Analyze logs

Analyze logs related to the problem with OA Control Panel:

Apache logs on the UI node(s) if the issue is on web server level.
/var/log/pa/pui/pui.log and /var/log/pa/console.log on the UI node(s) if the issue is on UI service level.
/var/log/pa/core.log and /var/log/pa/console.log on the OA Management Node if the issue is on OSS Core level.

Refer to the Knowledgebase article Odin Automation system logs location to find the required OA logs.

Analyze the part of the logs that are being appended to the log file when trying to log into the Control Panel. Use search in the Odin Knowledgebase for the error and warning messages you find in the logs.

Use the 'tail' utility to get the necessary part of the log. The 'tail' utility for Windows can be found in the Support Tools pack.

Check system resources

If OA servers are deployed as virtual environments, check that the proper amount of memory is allocated to the servers.
Refer to the hardware requirements in the corresponding installation guide. For example, OA Management Node on OA 7.3.
If you are using Virtuozzo products, use the native PVC/PSBM/PCS tools to check and tune memory and other resource limits configured for virtual servers: vzlist, vzctl.
Check /proc/user_beancounters for failed counters for specific resources and increase them if necessary.
For example, in case vzctl enters into container with OSA core throws:
```
# vzctl enter 10
Unable to fork: Cannot allocate memory
```
It means kmemsize is hit, as also mentioned in /var/log/messages. To increase it:
```
# vzctl set 10 --save --kmemsize <softlimit>:<hardlimit>
```
Where the limits would be bigger than the current ones.
Check if OOM killed process events registered on in /var/log/messages on Virtuozzo hardware node.
Check and tune (if necessary) the number of Linux threads on the OA UI server. Refer to Odin Knowledgebase article #118465 for more details.
Make sure there is enough free disk space on the OA Management Node, UI and Database servers.
In the case of a Linux-based installation, you also should check the number of disk inodes available on servers using the 'df -i' command.
On PVC, PSBM or PCS servers, the amount of disk inodes allocated for a container can be changed with the vzctl utility:
```
# vzctl set CT_ID --save --diskspace SOFTLIMIT:HARDLIMIT --diskinodes SOFTLIMIT:HARDLIMIT
```
See Virtuozzo Knowledgebase article #114019 for more details.

Check the firewall

Make sure the OA UI and branding servers can access the OA Management Node and each other over the network (Backnet).
Check the HTTPS port is open in the firewall for external incoming connections to the branding servers.
Check that ports 8080, 6308, 8352, 8354 are reachable on MN by UI node.

Refer to the Odin Automation Firewall Configuration Guide for the list of network ports to be open between different servers in a Odin Automation infrastructure.