How to deal with situations when the container hits the inodes limit : CloudBlue Technical Support

Symptoms

The website inside the container does not work properly.
It is not possible to rename/create a file inside the container.

Cause

The container has reached the limit of allocated inodes:

CT-999-bash-4.1# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vzfs  350000 350000 0 100% /
none 131072 151 130921 1% /dev
# vzquota stat 999
   resource          usage       softlimit      hardlimit    grace
  1k-blocks        5119784       945741824     1073741824
     inodes         350000          350000         370000

Resolution

For a container in VZFS format, increase the number of diskinodes allocated to the container:

# vzquota stat 999
   resource          usage       softlimit      hardlimit    grace
  1k-blocks        5119784       945741824     1073741824
     inodes         350000          350000         370000

# vzctl set 999 --save --diskinodes 450000:470000
# vzquota stat 999
   resource          usage       softlimit      hardlimit    grace
  1k-blocks        5119784       945741824     1073741824
     inodes         350054          450000         470000

For a container with Ploop-device, the number of inodes is proportional to the maximum disk space allocated for the container with the fixed initial ratio. Increasing the diskspace for this container triggers increasing number of available inodes, afterwards the diskspace limit can be set back to the previous value.

In case the used inodes count rapidly increases, it is necessary to find out which process generates excessive amount of inodes.

The first place to check is subfolders of /var/ directory, for example under certain conditions new requests to show PHP-powered pages create session files faster then the old ones can be deleted, which results in inode number increase in container.

# find /var -type d -size +256k

Directories with size of 1 megabyte or more cause significant performance problem, and having big amount of files is the source of high number of indes. Once the directory is spotted, the set of actions should be clear, depending on the purpose of the directory.

Let us check the following case when the culprit directory is /var/lib/php/session. To confirm this you can try to list the contents of the this directory (in a separate shell session) - the listing will work very slowly and on the first glance will appear to hang. Since the purpose of the directory is to store temporary files with session data of web-site visitors, this directory can be recreated, and the system configured to avoid such situation in the future.

To delete the directory follow these instructions:

Schedule some maintenance time for the container (not long, enough for stop, mount, unmount and start of the container)
Ensure that the container did not hit the inodes limit once again
Stop the container
```
# vzctl stop 999
```
Mount the container
```
# vzctl mount 999
```

Rename the sessions folder:

# mv /vz/root/999/var/lib/php/session{,_1}

Create a new sessions folder with the same permissions as the old one
Modify the /etc/php.ini configuration file to change the way of storing PHP sessions: change session.save_path = "/var/lib/php/session" to session.save_path= "3;/var/lib/php/session" in /vz/root/999/etc/php.ini configuration file
Start the container
```
# vzctl start 999
```
Remove the old folder with PHP session files:
```
# rm -rf /vz/root/999/var/lib/php/session_1
```
(it is highly advised to launch the removal operation is screen, it will take a very long time).

Also see

120305 How to change the default inodes limit for ploop containers?