Cloudstack virtual router pains

On my Cloudstack based Xenserver pool suddenly VM creation started resulting in error, when deploying VM into a specific client network.
According to management-server.log:

" com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Unable to apply userdata and password entry on router"

Since I have recently added a new host to the pool I was sure, that I screwed up something around the switch. After an interesting hour spent with ports and VLANs, I concluded, that the network config is perfect. (Okay, I found 1 typo in it, but it had nothing to do with the current situation.)

Time to read those logs in the Xenserver dom0.
I found an interesting entry in SMlog:

Jun 27 15:59:09 xen5 SM: [30660] ['/bin/bash', '/opt/cloud/bin/router_proxy.sh', 'vmdata.py', '169.xxx.yyy.zzz', '-d', ....']
Jun 27 15:59:10 xen5 SM: [30660] FAILED in util.pread: (rc 1) stdout: 'failed to make directories /var/www/html/userdata/192.168.44.110 due to :No space left on device

Errr…wtf? How can a system VM run out of space?
After ssh-ing into the system VM ( ssh -p 3922 -i /root/.ssh/id_rsa.cloud root@IP.AD.D.R) I saw, that the /var file system was full.
Great news… 2 minutes later I found out the reason: conntrackd is writing heavily conntrackd-stats.log. Seriously, who the hell needs this logfile in a system VM?
To disable it I had to edit /etc/conntrackd/conntrackd.conf, and replace “LogFile on”
with “LogFile off” at the end of the file, and run a /etc/init.d/conntrackd restart.

The issue is logged in Cloudstack’s bug tracking system too: https://issues.apache.org/jira/browse/CLOUDSTACK-6751
The ticket says that the affected version is 4.3, but it also affects version 4.2 (I am running 4.2)

1 thought on “Cloudstack virtual router pains

  1. Pingback: Apache CloudStack Virtual Router disk is full | WHO, WHAT, WHERE, WHEN, WHY, HOW ?

Leave a comment