Symptoms

All VPS backup tasks regularly start failing with the following error:

422 : Failed to connect to backup server: Can't connect: System errors : unauthorized request

The backup node has 20 established connections on port 4434:

[root@vzbackup01 ~]# netstat -antp | grep 4434
tcp        0      0 0.0.0.0:4434                0.0.0.0:*                   LISTEN      17245/pvaagent [gen
tcp        0      0 192.0.2.2:4434           192.0.2.3:56426         ESTABLISHED 17245/pvaagent [gen
tcp        0      0 192.0.2.2:4434           192.0.2.3:36510         ESTABLISHED 17245/pvaagent [gen
...

[root@vzbackup01 ~]# netstat -antp | grep 4434 | wc -l
20

While the other end does not have any connections open:

[root@vz01 ~]# netstat -antp | grep 4434
tcp        0      0 0.0.0.0:4434                0.0.0.0:*                   LISTEN      39194/pvaagent [gen
tcp        0      0 :::4434                     :::*                        LISTEN      39194/pvaagent [gen
[root@vz01 ~]# netstat -antp | grep 192.0.2.2

The backup node was already tweaked according to this article, but the limit is still getting reached.

Cause

Bad network configuration between the node(s) and the backup node. The intermediate network equipment interrupts the idle connection that transmits the backup data, and TCP connections remain stuck on the backup node.

Resolution

Configure the intermediate network equipment to terminate the connections properly, with sending FIN or RST packets to both the source node and the backup node, when terminating idle connections.

As an alternative, but less preferable solution, it is possible to tune TCP keepalive parameters on the backup node, the instructions can be found in the following article:

Backup fails: Connection timed out

Internal content