Enabling AMD IOMM generate a VM starting issue

Good morning Sirs,
I use XenServer in my Lab to test different sw solutions based on Windows or Linux OS.

I have previous installed the XenServer 6.5 Beta 2 on the above hw, without any problems or issues. All the tests made on the Beta2 work fine, and XenServer is really fast and stable virtualization environment !!!

Now I have installed today the latest XenServer Beta available (3) and I found the following issue:

• If I leave enabled the IOMM option in the motherboard BIOS, with a new and clean XenServer 6.4.96-88386c installation, any new or already exixting VMs, Linux or Windows do not starts, if they are configured with more than one vCPU (two or more).
  Starting a VM, the XenCenter display the green led for that machine, the console display only a white empty rectangle, and the vCPU in the performance tab remain to 100% (I waited up to 15 minutes). Nothing more happen, no OS setup start, no error or issue displayed on XenCenter and the vm seems to be frozen, until to a force shutdown.
  Instead, in case of a single vCPU VM, it can start after the initial creation (from a default template, ex: Windows 2012 R2), but after the first phase of Windows installation, at the first reboot asked by the Windows OS, the VM do not restart, displaying an error message like "The OS is unaivable".

• If I disable the IOMM option in the motherboard BIOS, all work fine: All the VMs starts with one or more vCPUs, the VM OS setup work fine and the OS run normally with XenServer Tools installed.

I tried to enable the IOMM and disable others CPU settings in the motherboard BIOS (like C6 switch or the AMD Cool and Quiet), but the really effective solution to solve the above problem is to disable IOMM.

This issue never arise on XenServer Beta2, that I ever used with IOMM switch enabled.
I also tried to generate the Status report and I have submitted in TAAS Portal, but the analysis end with no issues found.

In attach you can find two "Status-report", one with IOMM swith enabled (with issue), and another with the IOMM switch disabled (without issue).

Thank you very much for your support

Best Regards,

Roberto Luongo

Analysis of the problem:
Latest logging from the problem case with a modified Xen shows:

[2014-09-30 13:52:45] (d1) [  151.703497] pci dev 01:2 bar 20 size 000000020: 00000c201
[2014-09-30 13:52:45] (d1) [  151.703907] pci dev 01:1 bar 20 size 000000010: 00000c221
[2014-09-30 13:52:45] (d1) [  151.704210] Trampoline: 29 bytes at 0x103010
[2014-09-30 13:52:45] (d1) [  151.704230]   copying to: 0x1000
[2014-09-30 13:52:45] (d1) [  151.704241] Source hex:
[2014-09-30 13:52:45] (d1) [  151.704283]  8c c8 8e d8 0f 01 16 17 00 31 c0 40 0f 01 f0 66
[2014-09-30 13:52:45] (d1) [  151.704320]  ea 2e 30 10 00 08 00 17 00 48 30 10 00 cc
[2014-09-30 13:52:45] (d1) [  151.704330] Dest hex:
[2014-09-30 13:52:45] (d1) [  151.704371]  8c c8 8e d8 0f 01 16 17 00 31 c0 40 0f 01 f0 66
[2014-09-30 13:52:45] (d1) [  151.704408]  ea 2e 30 10 00 08 00 17 00 48 30 10 00 c2
[2014-09-30 13:52:45] (d1) [  151.704435] Multiprocessor initialisation:
[2014-09-30 13:52:45] (d1) [  151.704447] Booting cpu 0
[2014-09-30 13:52:45] (d1) [  151.704522]  - CPU0 ... 48-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done.
[2014-09-30 13:52:45] (d1) [  151.704535] Booting cpu 1
[2014-09-30 13:52:45] (d1) [  151.704562] Sending INIT-SIPI-SIPI to cpu 1
[2014-09-30 13:52:45] (XEN) [  151.704700] MMIO emulation failed: d1v1 Real @ 0100:0000000c -> 

0100:0000000c is the lmsw instruction from the AP boot trampoline, which does unconditionally need emulating in software for AMD systems. (This does use the MMIO emulation codepath despite not expecting any MMIO as a result. This is changed to be more sane in xen-unstable)

The lack of any instruction bytes to the right-hand-side of -> suggests that the emulation failure is caused by an inability to fetch the 3 bytes of the lmsw instruction. As the vcpu is currently in RM16, there is no possibility of pagetables causing the fetch issue.

From the extra debugging present for -dbg1-d.log, when attempting to emulate lmsw (part of the instruction fetch) hvm_fetch_from_guest_virt_nofault() fails with HVMCOPY_bad_gfn_to_mfn which implies that the NPT pagetables are bad for vcpu1.

There are very few changes to Xen between beta2 and beta3, and certainly no changes which look like they could plausibly cause this issue