How to setup virtual containers with LXC and quota support on Debian 8
Quota support is an often requested feature in lxc. Linux filesystem quota is required when you want to give multiple users access to a container and want to control that one user is not using all the disk space. Quota is also required for web hosting servers, e.g. with ISPConfig 3, for the same reason: one website shall not be able to fill up the whole disk. This howto shows you, how you can use lxc with hard disk quota using qemu nbd with a qcow image file on Debian 8.
Prerequisites
To use lxc, you need the qemu utilities and the lxc package itself. Install them by calling:
apt-get install lxc qemu-utils
The installer will ask you to choose the directory where the lxc virtual machine images get installed later. This directory should be on a partition with a lot of free space. If you have enough space in /var, then accept the default /var/lib/lxc, otherwise, choose a free directory on your largest partition. When you use a non-default path, then ensure to change the path in all commands and config files below.
Preparing
Check if the kernel loop module is loaded with:
lsmod | grep '^loop'
If you get not result, you can enable the module by running:
modprobe loop
Create the virtual machine
Now we can start creating the VM. In this tutorial, I will use Debian Jessie in both the host and the container, but you can use other lxc templates of course, e. g. Debian wheezy or ubuntu.
lxc-create -B loop -t debian -n mydebianvm --fssize=20G -- -r jessie
The -t argument selects the main template, -r decides which release to use. To set the hard disk size for the virtual machine, you can alter the --fssize argument. Let's say you want to create a disk with 50 gigabytes, you'd change the argument to --fssize=50G.
The argument -n sets the name of the vm. I used mydebianvm in this tutorial. Please change the name in all following commands according to what you chose.
As we don't want to use a raw image file, we need to convert the disk image to the qemu qcow2 format. This is done by the following command
qemu-img convert -O qcow2 /var/lib/lxc/mydebianvm/rootdev /var/lib/lxc/mydebianvm/rootdev.qcow2
To make backup handling easier, later on, we create an image file set, i. e. a second file that records all changes to the device.
qemu-img create -f qcow2 -b /var/lib/lxc/mydebianvm/rootdev.qcow2 /var/lib/lxc/mydebianvm/rootdev-live.qcow2
You can now delete the original raw image file with:
rm /var/lib/lxc/mydebianvm/rootdev
Configure the Network Bridge
Install the bridge-utils:
apt-get install bridge-utils
Open the Debian Network configuration file /etc/network/interfaces in an editor
vim /etc/network/interfaces
and add the following lines:
auto br0
iface br0 inet static
address 192.168.1.254
netmask 255.255.255.0
bridge_ports eth0
bridge_stp off
bridge_fd 2
bridge_maxwait 20
Replace the address and netmask with values for your local network.
Then activate the network bridge with the command:
ifup br0
Configure the VM
Backup the old config file:
mv /var/lib/lxc/mydebianvm/config /var/lib/lxc/mydebianvm/config_bak
And create the config for the container:
vim /var/lib/lxc/mydebianvm/config
And add the following content in the file:
lxc.rootfs = /var/lib/lxc/mydebianvm/rootfs
lxc.rootfs.options = usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0
lxc.hook.pre-start = /var/lib/lxc/prestart-nbd.sh
lxc.hook.post-stop = /var/lib/lxc/poststop-nbd.sh
# Common configuration
lxc.include = /usr/share/lxc/config/debian.common.conf
# only if bridge is set up (or use other method)
lxc.network.type = veth
lxc.network.name = veth0
lxc.network.flags = up
lxc.network.link = br0
lxc.network.ipv4 = 192.168.1.101/24
lxc.network.ipv4.gateway = 0.0.0.0
# Container specific configuration
lxc.mount = /var/lib/lxc/mydebianvm/fstab
lxc.utsname = debian8
lxc.arch = amd64
lxc.autodev = 1
lxc.kmsg = 0
Replace the IP-Address 192.168.1.101 with a free IP from your network.
Add the prestart script /var/lib/lxc/prestart-nbd.sh
vim /var/lib/lxc/prestart-nbd.sh
with the following content:
#!/bin/bash
CHK=$(lsmod | grep '^nbd');
if [[ "$CHK" == "" ]] ; then
modprobe nbd nbds_max=64 max_part=8
fi
DEV=""
for D in /dev/nbd* ; do
F=$(basename $D)
if [[ $(lsblk | grep "^${F} ") == "" ]] ; then
DEV="$D"
break;
fi
done
echo "Next free NBD is $DEV";
CHK=$(lsof /var/lib/lxc/${LXC_NAME}/rootdev-live.qcow2 | grep 'qemu-nbd' | awk '{ print $2 }');
if [[ "$CHK" == "" ]] ; then
if [[ "$DEV" == "" ]] ; then
print "No free nbd device found";
exit 1;
fi
echo "Connecting $DEV to /var/lib/lxc/${LXC_NAME}/rootdev-live.qcow2"
qemu-nbd -c ${DEV} -n --aio=native /var/lib/lxc/${LXC_NAME}/rootdev-live.qcow2
else
NBD=$(lsof -p ${CHK} | grep '/dev/nbd' | awk '{ print $9 }');
if [[ "$NBD" != "" ]] ; then
echo "/var/lib/lxc/${LXC_NAME}/rootdev-live.qcow2 is already connected to $NBD"
DEV="$NBD";
else
echo "/var/lib/lxc/${LXC_NAME}/rootdev-live.qcow2 is used by suspicious PID";
exit 1;
fi
fi
CHK=$(mount | grep " /var/lib/lxc/${LXC_NAME}/rootfs ")
if [[ "$CHK" == "" ]] ; then
echo "/var/lib/lxc/${LXC_NAME}/rootfs not mounted";
echo "Mounting ${DEV} to /var/lib/lxc/${LXC_NAME}/rootfs"
mount ${DEV} /var/lib/lxc/${LXC_NAME}/rootfs
fi
echo "${DEV} ${DEV:1} none bind,create=file,optional 0 0" > /var/lib/lxc/${LXC_NAME}/fstab
and make it executable:
chmod +x /var/lib/lxc/prestart-nbd.sh
Add the poststop script /var/lib/lxc/poststop-nbd.sh
vim /var/lib/lxc/poststop-nbd.sh
with the following content:
#!/bin/bash
CHK=$(mount | grep " /var/lib/lxc/${LXC_NAME}/rootfs " | awk '{ print $1 }')
if [[ "$CHK" != "" ]] ; then
echo "Unmounting ${CHK} from /var/lib/lxc/${LXC_NAME}/rootfs"
echo "Disconnecting ${CHK}"
umount /var/lib/lxc/${LXC_NAME}/rootfs && qemu-nbd -d ${CHK}
fi
and make it executable:
chmod +x /var/lib/lxc/poststop-nbd.sh
Start the VM and set-up quota
We can now start the container in background mode by typing:
lxc-start -n mydebianvm -d
Install the neccessary packages for quota. We don't have to enter the container for this. Using lxc-attach we can run commands from outside the container.
lxc-attach -n mydebianvm -- apt-get -y update
lxc-attach -n mydebianvm -- apt-get -y install quota
It is not possible to activate quota through lxc-attach. So we create a bash script, that is executed on next boot of the container
vim /var/lib/lxc/mydebianvm/rootfs/opt/actquota.sh
with the following content:
#!/bin/bash
mount -o remount,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 /
touch /aquota.user /aquota.group
chmod 0600 /aquota.*
quotacheck -cmug /
quotaon -avug
echo '#!/bin/sh -e
exit 0' > /etc/rc.local
rm -f /opt/actquota.sh
The script will delete itself and empty the /etc/rc.local of the container afterwards.
Now make sure the bash script is executeable and called at start-up:
Make it executable:
chmod 700 /var/lib/lxc/mydebianvm/rootfs/opt/actquota.sh
Add the call to the vm's rc.local file:
echo '#!/bin/bash
if [[ -e "/opt/actquota.sh" ]] ; then
/opt/actquota.sh
fi' > /var/lib/lxc/mydebianvm/rootfs/etc/rc.local
With all pre-requisites set-up you can now restart the container. If you have followed the steps correctly, this will activate quota.
lxc-stop -r -n mydebianvm
Verify the results
You should now check if quota is working. Change to the container.
lxc-attach -n mydebianvm
Inside the container type:
repquota -avug
You should see used quota of users and groups now.
Destroying the virtual machine
It is very important to use the commands in the correct order. Before you can disconnect the nbd device you have to stop the container if it is running:
lxc-stop -n mydebianvm
Afterwards, you have to unmount the root fs.
umount /var/lib/lxc/mydebianvm/rootfs
The last step is to disconnect the nbd. Be sure to select the correct device number.
NEVER disconnect the nbd before you unmounted the rootfs. This will lead to a lot of problems and will require a full forced reboot of your host.
qemu-nbd -d /dev/nbd0
Backup the VM
Because we have created two files when creating the image file for the container, we can easily backup without stopping the vm. First we need to commit the changes that occured in the meantime to the base file.
qemu-img commit /var/lib/lxc/mydebianvm/rootdev-live.qcow2
The /var/lib/lxc/mydebianvm/rootdev.qcow2 now contains the current state of the vm's hard disk, so you can backup this file.
