Blocker Description
I am using PHD Virtual Backup Appliance to perform snapshot-based backups every day, starting late at night and continuing into the early hours. The backup solution works as expected with XenServer 7.0.
Since installing XenServer 7.1, the backup appliance is crashing every morning at 04:02:01 (the time on the console) and the CPU #0 usage is shooting up to 100%. The console is also freezing and completely stops responding. I have to do a forceful reboot of the VM, but even then as soon as it starts it crashes and multiple CPU cores shoot up to 100%. Repeating the forceful reboot process results in the VM crashing every time until the XenServer host is rebooted after which the VM comes back to life and does not crash when it's started.
After the VM crashes and before rebooting the host, I have tried reimporting the VM using the XVA template (supplied by Unitrends) but upon starting, it crashes immediately. I have also tried using the latest version of the Unitrends Backup software (v9.2) but that does the exact same thing.
The older backup appliance runs Ubuntu 12.04.1 LTS compared to the newer version which runs CentOS. However all appliances have 1 thing in common - they all run in PV mode. I don't know whether that has any relevance.
According to the cron log file, the cron.daily jobs are running and completing at 04:02:01:
Mar 15 04:02:01 l1ps02xen CROND[29507]: (root) CMD ([ ! -f /etc/cron.hourly/0anacron ] && run-parts /etc/cron.daily)
Mar 15 04:02:01 l1ps02xen run-parts(/etc/cron.daily)[29508]: starting 0yum-daily.cron
Mar 15 04:02:01 l1ps02xen run-parts(/etc/cron.daily)[29515]: finished 0yum-daily.cron
Mar 15 04:02:01 l1ps02xen run-parts(/etc/cron.daily)[29508]: starting license-check
Mar 15 04:02:01 l1ps02xen run-parts(/etc/cron.daily)[29530]: finished license-check
Mar 15 04:02:01 l1ps02xen run-parts(/etc/cron.daily)[29508]: starting logrotate
Mar 15 04:02:02 l1ps02xen run-parts(/etc/cron.daily)[29581]: finished logrotate
Mar 15 04:02:02 l1ps02xen run-parts(/etc/cron.daily)[29508]: starting man-db.cron
Mar 15 04:02:03 l1ps02xen run-parts(/etc/cron.daily)[29592]: finished man-db.cron
Again I don't know whether this is relevant but I have disabled the cron.daily jobs and will see what happens tonight when I reboot the host again.
I ran the "xen-bugtool -y" command this morning before doing a forceful reboot. If you would like the logs, please supply a secure upload folder and I will send them across.
