A provider or a customer cannot log into the Operations Automation (OA) Control Panel. The Control Panel either is not accessible at all, or shows error messages such as:
An unforeseen error occurred
The corresponding service is not available at the moment, try it within 2-3 minutes.
This article describes basic troubleshooting steps when the Control Panel is down or giving an internal server error.
There may be several causes of Paralels Operations Automation Control Panel unavailability:
- Expired license
Below, you will find more details on checking for the above issues and troubleshooting them.
Important: in order to allow root cause analysis of each OA Control Panel outage, collect debug details to be passed to the Odin Technical Support team. Use the following script:
Launch it on both OA Management Node and UI node, when CP is inaccessible. The script will produce a .tar.bz2 archive inside
/var/log/pa directory. Note: both nodes should have
gdb package installed preliminarily.
# wget https://kb.cloudblue.com/Attachments/kcs-35818/oa_dump.bash # bash oa_dump.bash
In case CP is completely down and huge CPU usage of Java process is observed on either of the nodes, run the script in 'dump' mode instead to collect a core dump file additionally. Note: in case of a live Control Panel, this mode will cause CP unavailability during the dump process:
# bash oa_dump.bash --heap
After collecting the debug data, proceed with restarting the services to see if it helps to restore CP functionality.
Check the status of servers
Verify that all the servers listed below are up and running (the malfunctioning of any server below may affect the Control Panel). Start any servers that are down:
1. OA Management Node 2. OA Database server 3. OA UI (User Interface) server/cluster
Make sure all required services are running on the corresponding servers.
OA Management Node:
# systemctl status pau pa-agent
OA UI node:
# systemctl status pau
Make sure the OA system database service is running on the corresponding server:
# systemctl status postgresql-9.6
Analyze logs related to the problem with OA Control Panel:
- Apache logs on the UI node(s) if the issue is on web server level.
/var/log/pa/console.logon the UI node(s) if the issue is on UI service level.
/var/log/pa/console.logon the OA Management Node if the issue is on OSS Core level.
Refer to the Knowledgebase article Odin Automation system logs location to find the required OA logs.
Analyze the part of the logs that are being appended to the log file when trying to log into the Control Panel. Use search in the Odin Knowledgebase for the error and warning messages you find in the logs.
Use the 'tail' utility to get the necessary part of the log. The 'tail' utility for Windows can be found in the Support Tools pack.
If OA servers are deployed as virtual environments, check that the proper amount of memory is allocated to the servers.
Refer to the hardware requirements in the corresponding installation guide. For example, OA Management Node on OA 7.3.
If you are using Virtuozzo products, use the native PVC/PSBM/PCS tools to check and tune memory and other resource limits configured for virtual servers: vzlist, vzctl.
/proc/user_beancountersfor failed counters for specific resources and increase them if necessary.
For example, in case vzctl enters into container with OSA core throws:
# vzctl enter 10 Unable to fork: Cannot allocate memory
It means kmemsize is hit, as also mentioned in
/var/log/messages. To increase it:
# vzctl set 10 --save --kmemsize <softlimit>:<hardlimit>
Where the limits would be bigger than the current ones.
Check if OOM killed process events registered on in
/var/log/messageson Virtuozzo hardware node.
Check and tune (if necessary) the number of Linux threads on the OA UI server. Refer to Odin Knowledgebase article #118465 for more details.
Make sure there is enough free disk space on the OA Management Node, UI and Database servers.
In the case of a Linux-based installation, you also should check the number of disk inodes available on servers using the 'df -i' command.
On PVC, PSBM or PCS servers, the amount of disk inodes allocated for a container can be changed with the vzctl utility:
# vzctl set CT_ID --save --diskspace SOFTLIMIT:HARDLIMIT --diskinodes SOFTLIMIT:HARDLIMIT
See Virtuozzo Knowledgebase article #114019 for more details.
Make sure the OA UI and branding servers can access the OA Management Node and each other over the network (Backnet).
Check the HTTPS port is open in the firewall for external incoming connections to the branding servers.
- Check that ports 8080, 6308, 8352, 8354 are reachable on MN by UI node.
Refer to the Odin Automation Firewall Configuration Guide for the list of network ports to be open between different servers in a Odin Automation infrastructure.