Symptoms

  1. APS Booster was configured and all tasks were completed successfully.

  2. UI hosts were connected to Database Replica Nodes be steps from Documentation.

  3. Following error appears in /var/log/pa/pui/pui.log on UI hosts:

    00:00:00.001 [EJB default - 7] DEBUG c.o.a.s.b.e.ReplicationAndBoosterStatusBean - replica status check
00:00:00.005 [EJB default - 7] DEBUG c.o.a.s.b.ejb.WebServerStatusChecker - replica status: replay delay = 168909 ms, WAL diff = 0 bytes
00:00:00.005 [EJB default - 7] DEBUG c.o.a.s.b.ejb.WebServerStatusChecker - replica is out of sync - 168909 ms since last WAL replay
00:00:00.005 [EJB default - 7] DEBUG c.o.a.s.b.ejb.WebServerStatusChecker - booster is inactive, replica is out of sync

  4. APS REST API requests with OAuth authentication fail with:

    Authorization failed. Invalid authorization signature. Please check application settings.

Cause

  1. APS Booster is disabled
  2. PostgreSQL replication is not working.
  3. Database replication does not work due difference in time between the UI host and the Database Replica node.

Resolution

Check if the APS booster is Enabled or not. If not, enable it.

  • Make sure that APS Booster is enabled on the UI node:

    # grep apsbooster /usr/local/pem/wildfly-21.0.1.Final/standalone/configuration/standalone_xml_history/standalone-full-ha.boot.xml
<property name="com.parallels.pui.apsbooster.enabled" value="true"/>

    and in runtime:

    # sh /usr/local/pem/wildfly-21.0.1.Final/bin/jboss-cli.sh -c --commands="/system-property=com.parallels.pui.apsbooster.enabled:read-resource"
{
    "outcome" => "success",
    "result" => {"value" => "true"}
}

Check if the PostgreSQL replica is working or not.

  • To check if PostgreSQL replication is working, run this oss database query:                                                      

    oss=#  select * from pg_stat_replication;
 pid  | usesysid |      usename       | application_name | client_addr | client_hostname | client_port |         backend_start
    | backend_xmin |   state   |  sent_lsn   |  write_lsn  |  flush_lsn  | replay_lsn  |    write_lag    |    flush_lag    |   repl
ay_lag    | sync_priority | sync_state
------+----------+--------------------+------------------+-------------+-----------------+-------------+---------------------------
----+--------------+-----------+-------------+-------------+-------------+-------------+-----------------+-----------------+-------
----------+---------------+------------
 1295 |    31716 | slave_oa_10_1_10_7 | walreceiver      | 10.1.10.7   |                 |       33128 | 2023-01-27 19:43:16.991738
+09 |     48984320 | streaming | 69/B35994B8 | 69/B35994B8 | 69/B35994B8 | 69/B35994B8 | 00:00:00.001291 | 00:00:00.004878 | 00:00:
00.005063 |             0 | async
(1 row)

    On oss replica, use this query:                                                                                                                                       

    oss=# select * from pg_stat_wal_receiver;
  pid  |  status   | receive_start_lsn | receive_start_tli | received_lsn | received_tli |      last_msg_send_time       |     last
_msg_receipt_time     | latest_end_lsn |        latest_end_time        | slot_name | sender_host | sender_port |
                                                                               conninfo


-------+-----------+-------------------+-------------------+--------------+--------------+-------------------------------+---------
----------------------+----------------+-------------------------------+-----------+-------------+-------------+-------------------
-----------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------
 32592 | streaming | 69/82000000       |                 1 | 69/B35AE770  |            1 | 2023-01-30 10:09:45.012388+09 | 2023-01-
30 10:09:45.012485+09 | 69/B35AE770    | 2023-01-30 10:09:45.012388+09 |           | 10.1.10.6   |        5432 | user=slave_oa_10_1
_10_7 password=******** dbname=replication host=10.1.10.6 port=5432 fallback_application_name=walreceiver sslmode=verify-ca sslcomp
ression=1 krbsrvname=postgres target_session_attrs=any
(1 row)

            If it is not working, make sure that oss and oss replica has the same version of PostgreSQL.


Check if it has more than a one-minute time difference.

  • Synchronize time between UI host and replication node. More than one-minute time difference causes the issue:
maxReplayDelay = Duration.ofMinutes(1);

NOTE: In case UI cluster deployed in Azure, follow this article:

How to change timezone on Azure UI cluster

  • Make sure that /etc/localtime file is linked to a proper timezone. If not, correct this like in the example below:

    localtime -> /usr/share/zoneinfo/Etc/GMT+8

    As an alternative, configure time synchronization services like Chrony or NTPD on the affected hosts.

Internal content

Link on internal Article