Symptoms
When checking traffic in a container, e.g., in Plesk Panel for hosted domains, the amount of traffic is less than the amount of traffic shown in Power Panel, PBA-S, or POA CP.
Is there any way for an owner of a container to verify accounted traffic?
Cause
Details of traffic accounting in Plesk Panel.
Plesk checks log files of the services in the container on a per-domain basis. This is only a part of the handled traffic, e.g., if any CGI application connects with some external host, this traffic is not reflected anywhere in the log files.
Also, if there is no domain defined as the default domain for IP address(es) in Plesk, HTTP(s) traffic handled by the default Plesk site will not be counted in Plesk.
The points above give incomplete traffic details.
How does Virtuozzo kernel count containers' and VMs' traffic?
The traffic, which is the number of packets and the amount of data in those packets, is counted by the kernel for both directions (from and to a container). The traffic is counted on a per-class basis for network classes defined in the system.
The definition of network classes is configured with the file
/etc/sysconfig/vz-scripts/networks_classes. If the peer IP address matches with a network of the specific class, the corresponding packet is counted for that network class only (exclusive accounting).There is no way to define services or ports to count; the traffic is not divided this way, as only per-network segregation is possible.
The accounted number of packets and the amount of data can be retrieved by the
vznetstatorpnetstatcommands, which use the syscall to the kernel.How does PVA get and store this information?
PVA Agent queries the traffic stats from the kernel periodically (every minute), checks those counters, and puts the difference (relative to the information retrieved last time) in its database. Generally it is stored in
/var/opt/pva/agent/etc/VZAgentDB.sqldb.The node's administrator can query traffic stats from the database in the following way.
Find out the EID of the container, e.g., #1929:
~# /opt/pva/agent/bin/findByVeid.sh 1929 c03d9bb7-b4ff-ab4f-bfe5-8ea32b2040b4Query per-month traffic details from the database (creating a copy is needed to avoid DB locking):
~# cp -f /var/opt/pva/agent/etc/VZAgentDB.sqldb /root/VZAgentDB.sqldb ~# EID=$(/opt/pva/agent/bin/findByVeid.sh 1929) ~# echo "select strftime('%Y-%m', datetime(ldat.startTime, 'unixepoch', 'localtime')), ldef.counter, sum(ldat.avg) from log_def ldef join log_data ldat on ldat.defid=ldef.id where ldef.eid='${EID}' and ldef.lclass='counters_vz_net' and ldef.instance='' and ldef.counter like '%_bytes' group by 1, 2;" | sqlite3 -separator " " VZAgentDB.sqldbThe output might be like this:
2013-01 counter_net_incoming_bytes 687322275 2013-01 counter_net_outgoing_bytes 9975158133 2013-02 counter_net_incoming_bytes 367518150 2013-02 counter_net_outgoing_bytes 5290068106 2013-03 counter_net_incoming_bytes 524999712 2013-03 counter_net_outgoing_bytes 7306852350 2013-04 counter_net_incoming_bytes 614414197 2013-04 counter_net_outgoing_bytes 9153944033 2013-05 counter_net_incoming_bytes 622662523 2013-05 counter_net_outgoing_bytes 9077983708 2013-06 counter_net_incoming_bytes 128571718401 2013-06 counter_net_outgoing_bytes 79157203480 2013-07 counter_net_incoming_bytes 121901158967 2013-07 counter_net_outgoing_bytes 143672611791 2013-08 counter_net_incoming_bytes 205573570015 2013-08 counter_net_outgoing_bytes 168925814349 2013-09 counter_net_incoming_bytes 248664202 2013-09 counter_net_outgoing_bytes 4235004876Retrieve per-day information:
~# cp -f /var/opt/pva/agent/etc/VZAgentDB.sqldb /root/VZAgentDB.sqldb ~# EID=$(/opt/pva/agent/bin/findByVeid.sh 1929) ~# echo "select strftime('%Y-%m-%d', datetime(ldat.startTime, 'unixepoch', 'localtime')), ldef.counter, sum(ldat.avg) from log_def ldef join log_data ldat on ldat.defid=ldef.id where ldef.eid='${EID}' and ldef.lclass='counters_vz_net' and ldef.instance='' and ldef.counter like '%_bytes' group by 1, 2;" | sqlite3 -separator " " VZAgentDB.sqldbThe sample output is:
... 2013-09-01 counter_net_incoming_bytes 119501615 2013-09-01 counter_net_outgoing_bytes 2138801965 2013-09-02 counter_net_incoming_bytes 98970769 2013-09-02 counter_net_outgoing_bytes 1666390529 2013-09-03 counter_net_incoming_bytes 30191818 2013-09-03 counter_net_outgoing_bytes 429812382Retrieve per-hour information:
~# cp -f /var/opt/pva/agent/etc/VZAgentDB.sqldb /root/VZAgentDB.sqldb ~# EID=$(/opt/pva/agent/bin/findByVeid.sh 1929) ~# echo "select strftime('%Y-%m-%d %H-%M-%S', datetime(ldat.startTime, 'unixepoch', 'localtime')), ldef.counter, sum(ldat.avg) from log_def ldef join log_data ldat on ldat.defid=ldef.id where ldef.eid='${EID}' and ldef.lclass='counters_vz_net' and ldef.instance='' and ldef.counter like '%_bytes' group by 1, 2;" | sqlite3 -separator " " VZAgentDB.sqldbThe sample output is:
2014-02-11 15-19-19 counter_net_incoming_bytes 780578.0 2014-02-11 15-19-19 counter_net_outgoing_bytes 0.0 2014-02-11 19-19-19 counter_net_incoming_bytes 80635.0 2014-02-11 19-19-19 counter_net_outgoing_bytes 406095.0 2014-02-21 07-18-59 counter_net_incoming_bytes 128986.0 2014-02-21 07-18-59 counter_net_outgoing_bytes 840.0 2014-02-21 08-18-39 counter_net_incoming_bytes 84465.0 2014-02-21 08-18-39 counter_net_outgoing_bytes 0.0
Also, API requests to PVA Agent can be used to get the traffic details, getting information for the counter
counters_vz_netas described in the documentation.The router shows less outgoing traffic. How can it be?
The kernel counts traffic as it receives it from a virtual environment.
- For a container in routed mode, it is done on the interface
venet0. - For a container in bridged mode, it is done on the appropriate interface,
vethCTID.N. - For a virtual machine, it is done on the appropriate interface,
vmeN.M.
The effective bandwidth of these virtual interfaces is higher than the bandwidth of the physical interface of the node, and some packets are to be dropped under a high network load. This is especially visible if the node has traffic shaping enabled for virtual environments.
For example, ICMP and UDP traffic is not guaranteed to be delivered and packets might be freely dropped. If there is some malicious process sending large amounts of such traffic, then the amount of accounted traffic will differ between the kernel and the router.
In this case, carefully review the output of the the following command about ICMP and UDP output packets:
[root@virtual.env ~]# netstat -as
Another point is communication with a neighbor virtual environment which is hosted on the same Hardware Node: Packets do not leave the node. In terms of configuration, this can be avoided by configuring the proper network class.
- For a container in routed mode, it is done on the interface
Resolution
As an owner of a container, the only way to double-check the information is to review the counters of IPtables.
root@ct-1929:/# uptime
19:27:38 up 1 day, 4:15, 0 users, load average: 0.00, 0.00, 0.00
root@ct-1929:/# iptables-save -t filter
# Generated by iptables-save v1.3.6 on Tue Sep 3 19:27:47 2013
*filter
:INPUT ACCEPT [357117:39056474]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [509801:552679387]
COMMIT
# Completed on Tue Sep 3 19:27:47 2013
root@ct-1929:/#
The numbers in brackets for :INPUT and :OUTPUT show the number of packets and amount of data in those packets that was handled in the container's network sockets.
For 1 day of uptime, the received traffic is 39056474 bytes and sent traffic is 552679387. This includes local traffic as well, so these values are a bit bigger than the ones that are counted on the Hardware Node. But it is about the same as in the PVA Agent database.
The most simple way to count traffic on a per-service basis would be to configure per-port/service rules in IPtables and dump stats periodically.
IPtables allows you to configure rules with different matching criteria, e.g., for a given user, or to/from a specified port/service, for a specific binary. Many traffic accounting tools use IPtables for this and configure the rules accordingly, so you can take a look and set up any such service in the container. For example, Shorewall can do such a task.