Symptoms
All VPS backup tasks regularly start failing with the following error:
422 : Failed to connect to backup server: Can't connect: System errors : unauthorized request
The backup node has 20 established connections on port 4434:
[root@vzbackup01 ~]# netstat -antp | grep 4434
tcp 0 0 0.0.0.0:4434 0.0.0.0:* LISTEN 17245/pvaagent [gen
tcp 0 0 192.0.2.2:4434 192.0.2.3:56426 ESTABLISHED 17245/pvaagent [gen
tcp 0 0 192.0.2.2:4434 192.0.2.3:36510 ESTABLISHED 17245/pvaagent [gen
...
[root@vzbackup01 ~]# netstat -antp | grep 4434 | wc -l
20
While the other end does not have any connections open:
[root@vz01 ~]# netstat -antp | grep 4434
tcp 0 0 0.0.0.0:4434 0.0.0.0:* LISTEN 39194/pvaagent [gen
tcp 0 0 :::4434 :::* LISTEN 39194/pvaagent [gen
[root@vz01 ~]# netstat -antp | grep 192.0.2.2
The backup node was already tweaked according to this article, but the limit is still getting reached.
Cause
Bad network configuration between the node(s) and the backup node. The intermediate network equipment interrupts the idle connection that transmits the backup data, and TCP connections remain stuck on the backup node.
Resolution
Configure the intermediate network equipment to terminate the connections properly, with sending FIN or RST packets to both the source node and the backup node, when terminating idle connections.
As an alternative, but less preferable solution, it is possible to tune TCP keepalive parameters on the backup node, the instructions can be found in the following article:
Backup fails: Connection timed out