Details
-
Bug
-
Resolution: Done
-
Major
-
None
-
6.5 SP1
-
None
Description
Cross-posted at discussions.citrix.com (http://discussions.citrix.com/topic/382504-bnx2x-soft-lockup-bug-on-xenserver-65-sp1/)
Nov 8 15:53:20 vhp4 kernel: [2374459.160507] ------------[ cut here ]------------ Nov 8 15:53:20 vhp4 kernel: [2374459.160518] WARNING: at net/sched/sch_generic.c:255 dev_watchdog+0x19d/0x270() Nov 8 15:53:20 vhp4 kernel: [2374459.160520] NETDEV WATCHDOG: eth1 (bnx2x): transmit queue 0 timed out Nov 8 15:53:20 vhp4 kernel: [2374459.160521] Modules linked in: ipv6 stp llc arptable_filter arp_tables ebtable_filter ebtables tun ipmi_si nls_iso8859_1 nls_cp437 vfat fat mpt3sas(O) mpt2sas(O) scsi_transport_sas raid_class mptctl mptbase nfsv3 nfs_acl nfs fscache dell_rbu ipmi_devintf lockd sunrpc openvswitch(O) gre ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_tcpudp xt_conntrack nf_conntrack iptable_filter ip_tables x_tables dm_multipath nls_utf8 isofs video backlight sbs sbshc acpi_ipmi ipmi_msghandler nvram sr_mod cdrom usb_storage mxm_wmi sg dcdbas wmi hed tpm_tis tpm tpm_bios nvme bnx2x(O) ptp pps_core libcrc32c mdio ahci libahci lpc_ich mfd_core libata microcode crc32_pclmul ehci_pci scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua scsi_dh dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod shpchp megaraid_sas(O) sd_mod scsi_mod uhci_hcd ohci_hcd ehci_hcd [last unloaded: ipmi_si] Nov 8 15:53:20 vhp4 kernel: [2374459.160581] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 3.10.0+2 #1 Nov 8 15:53:20 vhp4 kernel: [2374459.160583] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.1.7 06/16/2016 Nov 8 15:53:20 vhp4 kernel: [2374459.160584] ffffffff81827a86 ffff880180603cd0 ffffffff812ce2d9 ffff880180603d10 Nov 8 15:53:20 vhp4 kernel: [2374459.160587] ffffffff810532b0 ffff880180613fc0 0000000000000000 ffff88016c39f080 Nov 8 15:53:20 vhp4 kernel: [2374459.160590] 000000000000005b ffff88016d114000 0000000000000000 ffff880180603df0 Nov 8 15:53:20 vhp4 kernel: [2374459.160593] Call Trace: Nov 8 15:53:20 vhp4 kernel: [2374459.160594] <IRQ> [<ffffffff812ce2d9>] dump_stack+0x19/0x20 Nov 8 15:53:20 vhp4 kernel: [2374459.160603] [<ffffffff810532b0>] warn_slowpath_common+0x70/0xa0 Nov 8 15:53:20 vhp4 kernel: [2374459.160605] [<ffffffff810533de>] warn_slowpath_fmt+0x6e/0x70 Nov 8 15:53:20 vhp4 kernel: [2374459.160610] [<ffffffff8108746f>] ? wake_up_process+0x3f/0x50 Nov 8 15:53:20 vhp4 kernel: [2374459.160615] [<ffffffff8106e9ac>] ? wake_up_worker+0x1c/0x30 Nov 8 15:53:20 vhp4 kernel: [2374459.160617] [<ffffffff8106ebd0>] ? insert_work+0xa0/0xc0 Nov 8 15:53:20 vhp4 kernel: [2374459.160621] [<ffffffff81506b7e>] ? _raw_spin_lock+0xe/0x20 Nov 8 15:53:20 vhp4 kernel: [2374459.160623] [<ffffffff8146c73d>] dev_watchdog+0x19d/0x270 Nov 8 15:53:20 vhp4 kernel: [2374459.160625] [<ffffffff8146c5a0>] ? dev_deactivate+0x40/0x40 Nov 8 15:53:20 vhp4 kernel: [2374459.160627] [<ffffffff8146c5a0>] ? dev_deactivate+0x40/0x40 Nov 8 15:53:20 vhp4 kernel: [2374459.160629] [<ffffffff8106232e>] call_timer_fn+0x6e/0x110 Nov 8 15:53:20 vhp4 kernel: [2374459.160631] [<ffffffff8146c5a0>] ? dev_deactivate+0x40/0x40 Nov 8 15:53:20 vhp4 kernel: [2374459.160633] [<ffffffff810636a9>] run_timer_softirq+0x1f9/0x250 Nov 8 15:53:20 vhp4 kernel: [2374459.160637] [<ffffffff8105b8fa>] __do_softirq+0x10a/0x220 Nov 8 15:53:20 vhp4 kernel: [2374459.160639] [<ffffffff810d31c8>] ? handle_percpu_irq+0x48/0x70 Nov 8 15:53:20 vhp4 kernel: [2374459.160643] [<ffffffff8151061c>] call_softirq+0x1c/0x30 Nov 8 15:53:20 vhp4 kernel: [2374459.160647] [<ffffffff810140a0>] do_softirq+0x50/0xa0 Nov 8 15:53:20 vhp4 kernel: [2374459.160649] [<ffffffff8105b3bd>] irq_exit+0x4d/0xa0 Nov 8 15:53:20 vhp4 kernel: [2374459.160652] [<ffffffff81363375>] xen_evtchn_do_upcall+0x35/0x50 Nov 8 15:53:20 vhp4 kernel: [2374459.160655] [<ffffffff8151067e>] xen_do_hypervisor_callback+0x1e/0xa0 Nov 8 15:53:20 vhp4 kernel: [2374459.160656] <EOI> [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20 Nov 8 15:53:20 vhp4 kernel: [2374459.160660] [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20 Nov 8 15:53:20 vhp4 kernel: [2374459.160663] [<ffffffff81009b90>] ? xen_safe_halt+0x10/0x20 Nov 8 15:53:20 vhp4 kernel: [2374459.160666] [<ffffffff8101a735>] ? default_idle+0x65/0xc0 Nov 8 15:53:20 vhp4 kernel: [2374459.160668] [<ffffffff8101a418>] ? arch_cpu_idle+0x18/0x30 Nov 8 15:53:20 vhp4 kernel: [2374459.160671] [<ffffffff8109e626>] ? cpu_startup_entry+0x1a6/0x210 Nov 8 15:53:20 vhp4 kernel: [2374459.160674] [<ffffffff814f3362>] ? rest_init+0x72/0x80 Nov 8 15:53:20 vhp4 kernel: [2374459.160677] [<ffffffff81ad018f>] ? start_kernel+0x3ef/0x400 Nov 8 15:53:20 vhp4 kernel: [2374459.160679] [<ffffffff81acf8e0>] ? repair_env_string+0x70/0x70 Nov 8 15:53:20 vhp4 kernel: [2374459.160681] [<ffffffff81acf56e>] ? x86_64_start_reservations+0x2e/0x40 Nov 8 15:53:20 vhp4 kernel: [2374459.160683] [<ffffffff81ad3241>] ? xen_start_kernel+0x7a1/0x7c0 Nov 8 15:53:20 vhp4 kernel: [2374459.160686] ---[ end trace 2b03c9edeaaf7d00 ]--- Nov 8 15:53:20 vhp4 kernel: [2374459.180377] bnx2x: [bnx2x_stats_comp:217(eth1)]timeout waiting for stats finished Nov 8 15:53:20 vhp4 kernel: [2374459.200633] bnx2x: [bnx2x_stats_comp:217(eth1)]timeout waiting for stats finished Nov 8 15:53:22 vhp4 kernel: [2374461.208857] [bnx2x_clean_tx_queue:1488(eth1)]timeout waiting for queue[0]: txdata->tx_pkt_prod(5650) != txdata->tx_pkt_cons(5348) Nov 8 15:53:24 vhp4 kernel: [2374463.217565] [bnx2x_clean_tx_queue:1488(eth1)]timeout waiting for queue[0]: txdata->tx_pkt_prod(5650) != txdata->tx_pkt_cons(5348) Nov 8 15:53:24 vhp4 xenstored: D22.46695 rm attr/eth0 Nov 8 15:53:24 vhp4 xenstored: D22 write attr/eth0/ip 10.1.1.170 Nov 8 15:53:24 vhp4 xenstored: D22 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:1aa Nov 8 15:53:27 vhp4 xenstored: D11.63941 rm attr/eth0 Nov 8 15:53:27 vhp4 xenstored: D11 write attr/eth0/ip 10.1.0.75 Nov 8 15:53:27 vhp4 xenstored: D11 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:4b Nov 8 15:53:31 vhp4 xenstored: D14 write data/meminfo_free 9707000 Nov 8 15:53:31 vhp4 xenstored: A342899 w event /local/domain/14/data/meminfo_free /local/domain/14/data/meminfo_free Nov 8 15:53:31 vhp4 xenstored: D14.46679 rm attr/eth0 Nov 8 15:53:31 vhp4 xenstored: D14 write attr/eth0/ip 10.1.2.48 Nov 8 15:53:31 vhp4 xenstored: D14 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:230 Nov 8 15:53:31 vhp4 xenstored: D14 write data/updated Tue Nov 8 15:53:31 CST 2016 Nov 8 15:53:31 vhp4 xenstored: A6 w event /local/domain/14/data/updated /local/domain/14/data/updated Nov 8 15:53:31 vhp4 xenstored: A10 w event /local/domain/14/data/updated /local/domain/14/data/updated Nov 8 15:53:33 vhp4 xenstored: D13.59183 rm attr/eth0 Nov 8 15:53:33 vhp4 xenstored: D13 write attr/eth0/ip 10.1.0.66 Nov 8 15:53:33 vhp4 xenstored: D13 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:42 Nov 8 15:53:33 vhp4 xenstored: D20.46633 rm attr/eth0 Nov 8 15:53:33 vhp4 xenstored: D20 write attr/eth0/ip 10.1.0.193 Nov 8 15:53:33 vhp4 xenstored: D20 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:c1 Nov 8 15:53:33 vhp4 xenstored: D27 write data/meminfo_free 672220 Nov 8 15:53:33 vhp4 xenstored: A342899 w event /local/domain/27/data/meminfo_free /local/domain/27/data/meminfo_free Nov 8 15:53:33 vhp4 xenstored: D27.61040 rm attr/eth0 Nov 8 15:53:33 vhp4 xenstored: D27 write attr/eth0/ip 10.1.1.24 Nov 8 15:53:33 vhp4 xenstored: D27 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:118 Nov 8 15:53:33 vhp4 xenstored: D27 write attr/eth0:16/ip 10.1.1.239 Nov 8 15:53:33 vhp4 xenstored: D27 error EINVAL Nov 8 15:53:33 vhp4 xenstored: D27 write attr/eth0:30/ip 10.1.1.216 Nov 8 15:53:33 vhp4 xenstored: D27 error EINVAL Nov 8 15:53:33 vhp4 xenstored: D27 write data/updated Tue Nov 8 15:53:33 CST 2016 Nov 8 15:53:33 vhp4 xenstored: A6 w event /local/domain/27/data/updated /local/domain/27/data/updated Nov 8 15:53:33 vhp4 xenstored: A10 w event /local/domain/27/data/updated /local/domain/27/data/updated Nov 8 15:53:34 vhp4 kernel: [2374473.255366] [bnx2x_state_wait:319(eth1)]timeout waiting for state 0 Nov 8 15:53:34 vhp4 kernel: [2374473.255376] bnx2x: [bnx2x_del_all_macs:9117(eth1)]Failed to delete MACs: -16 Nov 8 15:53:34 vhp4 kernel: [2374473.255381] bnx2x: [bnx2x_chip_cleanup:9946(eth1)]Failed to schedule DEL commands for UC MACs list: -16 Nov 8 15:53:40 vhp4 xenstored: D9.47400 rm attr/eth0 Nov 8 15:53:40 vhp4 xenstored: D9 write attr/eth0/ip 10.1.1.156 Nov 8 15:53:40 vhp4 xenstored: D9 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:19c Nov 8 15:53:44 vhp4 xenstored: D24 write data/meminfo_free 580164 Nov 8 15:53:44 vhp4 xenstored: A342899 w event /local/domain/24/data/meminfo_free /local/domain/24/data/meminfo_free Nov 8 15:53:44 vhp4 xenstored: D24.46924 rm attr/eth0 Nov 8 15:53:44 vhp4 xenstored: D24 write attr/eth0/ip 10.1.1.179 Nov 8 15:53:44 vhp4 xenstored: D24 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:1b3 Nov 8 15:53:44 vhp4 xenstored: D24 write data/updated Tue Nov 8 15:53:44 CST 2016 Nov 8 15:53:44 vhp4 xenstored: A6 w event /local/domain/24/data/updated /local/domain/24/data/updated Nov 8 15:53:44 vhp4 xenstored: A10 w event /local/domain/24/data/updated /local/domain/24/data/updated Nov 8 15:53:44 vhp4 kernel: [2374483.308913] [bnx2x_state_wait:319(eth1)]timeout waiting for state 9 Nov 8 15:53:47 vhp4 xenstored: D16 write data/meminfo_free 473436 Nov 8 15:53:47 vhp4 xenstored: A342899 w event /local/domain/16/data/meminfo_free /local/domain/16/data/meminfo_free Nov 8 15:53:47 vhp4 xenstored: D16.47440 rm attr/eth0 Nov 8 15:53:47 vhp4 xenstored: D16 write attr/eth0/ip 10.1.1.119 Nov 8 15:53:47 vhp4 xenstored: D16 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:177 Nov 8 15:53:47 vhp4 xenstored: D16 write data/updated Tue Nov 8 15:53:47 CST 2016 Nov 8 15:53:47 vhp4 xenstored: A6 w event /local/domain/16/data/updated /local/domain/16/data/updated Nov 8 15:53:47 vhp4 xenstored: A10 w event /local/domain/16/data/updated /local/domain/16/data/updated Nov 8 15:53:48 vhp4 xenstored: D2 write data/meminfo_free 8106812 Nov 8 15:53:48 vhp4 xenstored: A342899 w event /local/domain/2/data/meminfo_free /local/domain/2/data/meminfo_free Nov 8 15:53:48 vhp4 xenstored: D2.47391 rm attr/eth0 Nov 8 15:53:48 vhp4 xenstored: D2 write attr/eth0/ip 10.1.0.80 Nov 8 15:53:48 vhp4 xenstored: D2 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:50 Nov 8 15:53:48 vhp4 xenstored: D2 write data/updated Tue Nov 8 15:53:48 CST 2016 Nov 8 15:53:48 vhp4 xenstored: A6 w event /local/domain/2/data/updated /local/domain/2/data/updated Nov 8 15:53:48 vhp4 xenstored: A10 w event /local/domain/2/data/updated /local/domain/2/data/updated Nov 8 15:53:49 vhp4 xenstored: D15 write data/meminfo_free 26227316 Nov 8 15:53:49 vhp4 xenstored: A342899 w event /local/domain/15/data/meminfo_free /local/domain/15/data/meminfo_free Nov 8 15:53:49 vhp4 xenstored: D15.59765 rm attr/eth0 Nov 8 15:53:49 vhp4 xenstored: D15 write attr/eth0/ip 10.1.1.220 Nov 8 15:53:49 vhp4 xenstored: D15 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:1dc Nov 8 15:53:49 vhp4 xenstored: D15 write data/updated Tue Nov 8 15:53:49 CST 2016 Nov 8 15:53:49 vhp4 xenstored: A6 w event /local/domain/15/data/updated /local/domain/15/data/updated Nov 8 15:53:49 vhp4 xenstored: A10 w event /local/domain/15/data/updated /local/domain/15/data/updated Nov 8 15:53:51 vhp4 xenstored: D17 write data/meminfo_free 429252 Nov 8 15:53:51 vhp4 xenstored: A342899 w event /local/domain/17/data/meminfo_free /local/domain/17/data/meminfo_free Nov 8 15:53:51 vhp4 xenstored: D17.47486 rm attr/eth0 Nov 8 15:53:51 vhp4 xenstored: D17 write attr/eth0/ip 10.1.1.110 Nov 8 15:53:51 vhp4 xenstored: D17 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:16e Nov 8 15:53:51 vhp4 xenstored: D17 write data/updated Tue Nov 8 15:53:51 CST 2016 Nov 8 15:53:51 vhp4 xenstored: A6 w event /local/domain/17/data/updated /local/domain/17/data/updated Nov 8 15:53:51 vhp4 xenstored: A10 w event /local/domain/17/data/updated /local/domain/17/data/updated Nov 8 15:53:52 vhp4 xenstored: D8.47461 rm attr/eth0 Nov 8 15:53:52 vhp4 xenstored: D8 write attr/eth0/ip 10.1.1.164 Nov 8 15:53:52 vhp4 xenstored: D8 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:1a4 Nov 8 15:53:54 vhp4 kernel: [2374493.322189] [bnx2x_state_wait:319(eth1)]timeout waiting for state 2 Nov 8 15:53:54 vhp4 kernel: [2374493.322201] bnx2x: [bnx2x_func_stop:9717(eth1)]FUNC_STOP ramrod failed. Running a dry transaction Nov 8 15:53:54 vhp4 kernel: [2374493.643194] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:54 vhp4 kernel: [2374493.643210] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:55 vhp4 kernel: [2374493.842172] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:55 vhp4 kernel: [2374493.842187] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:55 vhp4 kernel: [2374494.041105] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:55 vhp4 kernel: [2374494.041130] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:55 vhp4 kernel: [2374494.240103] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:55 vhp4 kernel: [2374494.240118] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:55 vhp4 kernel: [2374494.438966] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:55 vhp4 kernel: [2374494.439062] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:55 vhp4 xenstored: D12.71744 rm attr/eth0 Nov 8 15:53:55 vhp4 xenstored: D12 write attr/eth0/ip 10.1.1.16 Nov 8 15:53:55 vhp4 xenstored: D12 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:110 Nov 8 15:53:55 vhp4 xenstored: D12 write attr/eth0:30/ip 10.1.1.224 Nov 8 15:53:55 vhp4 xenstored: D12 error EINVAL Nov 8 15:53:55 vhp4 kernel: [2374494.637881] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:55 vhp4 kernel: [2374494.637899] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:56 vhp4 kernel: [2374494.836922] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:56 vhp4 kernel: [2374494.836936] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:56 vhp4 kernel: [2374495.035705] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:56 vhp4 kernel: [2374495.036147] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:56 vhp4 kernel: [2374495.235456] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:56 vhp4 kernel: [2374495.236465] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:56 vhp4 kernel: [2374495.435400] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:56 vhp4 kernel: [2374495.436367] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:56 vhp4 kernel: [2374495.635235] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:56 vhp4 kernel: [2374495.635531] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:57 vhp4 kernel: [2374495.834518] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:57 vhp4 kernel: [2374495.835305] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:57 vhp4 kernel: [2374496.034234] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:57 vhp4 kernel: [2374496.035205] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:57 vhp4 kernel: [2374496.233985] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:57 vhp4 kernel: [2374496.235243] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:57 vhp4 kernel: [2374496.433995] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:57 vhp4 kernel: [2374496.435028] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:57 vhp4 kernel: [2374496.634180] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:57 vhp4 kernel: [2374496.634400] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:58 vhp4 kernel: [2374496.833238] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:58 vhp4 kernel: [2374496.834317] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:58 vhp4 kernel: [2374497.033187] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:58 vhp4 kernel: [2374497.033872] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:58 vhp4 kernel: [2374497.232821] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:58 vhp4 kernel: [2374497.233809] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:58 vhp4 kernel: [2374497.432685] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:58 vhp4 kernel: [2374497.433458] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:58 vhp4 kernel: [2374497.632415] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:58 vhp4 kernel: [2374497.632941] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:58 vhp4 kernel: [2374497.831781] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:59 vhp4 kernel: [2374497.832631] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:59 vhp4 kernel: [2374498.031578] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:59 vhp4 kernel: [2374498.032170] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:59 vhp4 kernel: [2374498.231235] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:59 vhp4 kernel: [2374498.232409] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:59 vhp4 kernel: [2374498.431338] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:59 vhp4 kernel: [2374498.432307] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:59 vhp4 kernel: [2374498.631076] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:59 vhp4 kernel: [2374498.631572] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:53:59 vhp4 kernel: [2374498.830542] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:53:59 vhp4 kernel: [2374498.831501] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:00 vhp4 kernel: [2374499.030401] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:00 vhp4 kernel: [2374499.031010] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:00 vhp4 kernel: [2374499.229915] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:00 vhp4 kernel: [2374499.230848] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:00 vhp4 kernel: [2374499.429769] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:00 vhp4 kernel: [2374499.430954] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:00 vhp4 kernel: [2374499.630139] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:00 vhp4 kernel: [2374499.630767] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:00 vhp4 xenstored: D19 write data/meminfo_free 1796904 Nov 8 15:54:00 vhp4 xenstored: A342899 w event /local/domain/19/data/meminfo_free /local/domain/19/data/meminfo_free Nov 8 15:54:00 vhp4 xenstored: D19.46598 rm attr/eth0 Nov 8 15:54:00 vhp4 xenstored: D19 write attr/eth0/ip 10.1.2.50 Nov 8 15:54:00 vhp4 xenstored: D19 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:232 Nov 8 15:54:00 vhp4 xenstored: D19 write data/updated Tue Nov 8 15:54:00 CST 2016 Nov 8 15:54:00 vhp4 xenstored: A6 w event /local/domain/19/data/updated /local/domain/19/data/updated Nov 8 15:54:00 vhp4 xenstored: A10 w event /local/domain/19/data/updated /local/domain/19/data/updated Nov 8 15:54:00 vhp4 kernel: [2374499.829559] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:00 vhp4 kernel: [2374499.830709] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:01 vhp4 kernel: [2374500.029737] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:01 vhp4 kernel: [2374500.030273] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:01 vhp4 kernel: [2374500.229138] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:01 vhp4 kernel: [2374500.230312] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:01 vhp4 kernel: [2374500.429359] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:01 vhp4 kernel: [2374500.430354] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:01 vhp4 kernel: [2374500.629254] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:01 vhp4 kernel: [2374500.630673] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:01 vhp4 kernel: [2374500.829669] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:01 vhp4 kernel: [2374500.831725] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:02 vhp4 kernel: [2374501.030671] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:02 vhp4 kernel: [2374501.031611] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:02 vhp4 xenstored: D21.46986 rm attr/eth0 Nov 8 15:54:02 vhp4 xenstored: D21 write attr/eth0/ip 10.1.0.107 Nov 8 15:54:02 vhp4 xenstored: D21 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:6b Nov 8 15:54:02 vhp4 kernel: [2374501.230422] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:02 vhp4 kernel: [2374501.231875] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:02 vhp4 kernel: [2374501.430619] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:02 vhp4 kernel: [2374501.431479] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:02 vhp4 kernel: [2374501.630269] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:02 vhp4 kernel: [2374501.631384] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:02 vhp4 kernel: [2374501.830169] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:02 vhp4 kernel: [2374501.831613] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:03 vhp4 kernel: [2374502.030542] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:03 vhp4 kernel: [2374502.031421] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:03 vhp4 kernel: [2374502.230222] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:03 vhp4 kernel: [2374502.231587] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:03 vhp4 kernel: [2374502.430610] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:03 vhp4 kernel: [2374502.431698] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:03 vhp4 kernel: [2374502.630627] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:03 vhp4 kernel: [2374502.631964] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:03 vhp4 kernel: [2374502.830933] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:03 vhp4 kernel: [2374502.832001] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:04 vhp4 kernel: [2374503.030891] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:04 vhp4 kernel: [2374503.031867] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:04 vhp4 kernel: [2374503.230781] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:04 vhp4 kernel: [2374503.231847] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:04 vhp4 kernel: [2374503.430605] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:04 vhp4 kernel: [2374503.431475] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:04 vhp4 kernel: [2374503.630204] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:04 vhp4 kernel: [2374503.631267] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:04 vhp4 kernel: [2374503.830135] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:04 vhp4 kernel: [2374503.831359] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:05 vhp4 kernel: [2374504.030220] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:05 vhp4 kernel: [2374504.030719] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:05 vhp4 kernel: [2374504.229582] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:05 vhp4 kernel: [2374504.230885] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:05 vhp4 kernel: [2374504.429648] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:05 vhp4 kernel: [2374504.430723] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:05 vhp4 kernel: [2374504.629535] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:05 vhp4 kernel: [2374504.630584] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:05 vhp4 kernel: [2374504.829430] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:05 vhp4 kernel: [2374504.830609] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:06 vhp4 kernel: [2374505.029346] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:06 vhp4 kernel: [2374505.029859] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:06 vhp4 kernel: [2374505.228679] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:06 vhp4 kernel: [2374505.228738] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:06 vhp4 kernel: [2374505.427475] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:06 vhp4 kernel: [2374505.427517] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:06 vhp4 kernel: [2374505.626285] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:06 vhp4 kernel: [2374505.626311] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:06 vhp4 kernel: [2374505.825102] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:06 vhp4 kernel: [2374505.825118] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:07 vhp4 kernel: [2374506.024252] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:07 vhp4 kernel: [2374506.024270] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:07 vhp4 kernel: [2374506.223248] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:07 vhp4 kernel: [2374506.223272] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:07 vhp4 kernel: [2374506.422058] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:07 vhp4 kernel: [2374506.422080] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:07 vhp4 xenstored: D1 write data/meminfo_free 368576 Nov 8 15:54:07 vhp4 xenstored: A342899 w event /local/domain/1/data/meminfo_free /local/domain/1/data/meminfo_free Nov 8 15:54:07 vhp4 xenstored: D1.84641 rm attr/eth0 Nov 8 15:54:07 vhp4 xenstored: D1 write attr/eth0/ip 10.1.0.130 Nov 8 15:54:07 vhp4 xenstored: D1 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:82 Nov 8 15:54:07 vhp4 xenstored: D1 write data/updated Tue Nov 8 15:54:07 CST 2016 Nov 8 15:54:07 vhp4 xenstored: A6 w event /local/domain/1/data/updated /local/domain/1/data/updated Nov 8 15:54:07 vhp4 xenstored: A10 w event /local/domain/1/data/updated /local/domain/1/data/updated Nov 8 15:54:07 vhp4 kernel: [2374506.621017] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:07 vhp4 kernel: [2374506.621038] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:07 vhp4 kernel: [2374506.819917] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:07 vhp4 kernel: [2374506.819953] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:08 vhp4 kernel: [2374507.018729] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:08 vhp4 kernel: [2374507.018743] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:08 vhp4 kernel: [2374507.217504] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:08 vhp4 kernel: [2374507.217523] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:08 vhp4 kernel: [2374507.416311] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:08 vhp4 kernel: [2374507.416325] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:08 vhp4 kernel: [2374507.615166] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:08 vhp4 kernel: [2374507.615181] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:08 vhp4 kernel: [2374507.813974] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:08 vhp4 kernel: [2374507.814000] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:09 vhp4 kernel: [2374508.012903] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:09 vhp4 kernel: [2374508.012939] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:09 vhp4 kernel: [2374508.211839] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:09 vhp4 kernel: [2374508.211862] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:09 vhp4 kernel: [2374508.410747] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:09 vhp4 kernel: [2374508.410770] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:09 vhp4 kernel: [2374508.609720] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:09 vhp4 kernel: [2374508.609744] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:09 vhp4 kernel: [2374508.808873] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:09 vhp4 kernel: [2374508.808895] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:10 vhp4 kernel: [2374509.007894] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:10 vhp4 kernel: [2374509.007915] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:10 vhp4 kernel: [2374509.206937] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:10 vhp4 kernel: [2374509.206970] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:10 vhp4 kernel: [2374509.405762] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:10 vhp4 kernel: [2374509.405784] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:10 vhp4 kernel: [2374509.604846] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:10 vhp4 kernel: [2374509.604871] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:10 vhp4 kernel: [2374509.803670] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:10 vhp4 kernel: [2374509.803698] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:11 vhp4 kernel: [2374510.002707] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:11 vhp4 kernel: [2374510.002728] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:11 vhp4 kernel: [2374510.201627] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:11 vhp4 kernel: [2374510.201682] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:11 vhp4 kernel: [2374510.400544] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:11 vhp4 kernel: [2374510.400592] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:11 vhp4 kernel: [2374510.599522] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:11 vhp4 kernel: [2374510.599551] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:11 vhp4 kernel: [2374510.798473] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:11 vhp4 kernel: [2374510.798497] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:12 vhp4 kernel: [2374510.997262] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:12 vhp4 kernel: [2374510.997289] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:12 vhp4 kernel: [2374511.196236] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:12 vhp4 kernel: [2374511.196258] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:12 vhp4 kernel: [2374511.395165] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:12 vhp4 kernel: [2374511.395189] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:12 vhp4 kernel: [2374511.593947] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:12 vhp4 kernel: [2374511.593988] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:12 vhp4 kernel: [2374511.792797] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:12 vhp4 kernel: [2374511.792820] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:13 vhp4 kernel: [2374511.991657] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:13 vhp4 kernel: [2374511.991685] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:13 vhp4 kernel: [2374512.190670] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:13 vhp4 kernel: [2374512.190699] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:13 vhp4 kernel: [2374512.389618] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:13 vhp4 kernel: [2374512.389646] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:13 vhp4 kernel: [2374512.588752] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:13 vhp4 kernel: [2374512.588784] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:13 vhp4 xenstored: D5.69250 rm attr/eth0 Nov 8 15:54:13 vhp4 xenstored: D5 write attr/eth0/ip 10.1.0.238 Nov 8 15:54:13 vhp4 kernel: [2374512.787727] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:13 vhp4 kernel: [2374512.787771] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:13 vhp4 xenstored: D5 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:ee Nov 8 15:54:14 vhp4 xenstored: D18 write data/meminfo_free 157604 Nov 8 15:54:14 vhp4 xenstored: A342899 w event /local/domain/18/data/meminfo_free /local/domain/18/data/meminfo_free Nov 8 15:54:14 vhp4 kernel: [2374512.986628] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:14 vhp4 kernel: [2374512.986657] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:14 vhp4 xenstored: D18.47338 rm attr/eth0 Nov 8 15:54:14 vhp4 xenstored: D18 write attr/eth0/ip 10.1.1.82 Nov 8 15:54:14 vhp4 xenstored: D18 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:152 Nov 8 15:54:14 vhp4 xenstored: D18 write attr/eth0:0/ip 10.1.1.34 Nov 8 15:54:14 vhp4 xenstored: D18 error EINVAL Nov 8 15:54:14 vhp4 xenstored: D18 write attr/eth0:1/ip 10.1.1.35 Nov 8 15:54:14 vhp4 xenstored: D18 error EINVAL Nov 8 15:54:14 vhp4 xenstored: D18 write attr/eth0:2/ip 10.1.1.38 Nov 8 15:54:14 vhp4 xenstored: D18 error EINVAL Nov 8 15:54:14 vhp4 xenstored: D18 write attr/eth0:30/ip 10.1.1.162 Nov 8 15:54:14 vhp4 xenstored: D18 error EINVAL Nov 8 15:54:14 vhp4 xenstored: D18 write attr/eth0:31/ip 10.1.1.168 Nov 8 15:54:14 vhp4 xenstored: D18 error EINVAL Nov 8 15:54:14 vhp4 xenstored: D18 write data/updated Tue Nov 8 15:54:14 CST 2016 Nov 8 15:54:14 vhp4 xenstored: A6 w event /local/domain/18/data/updated /local/domain/18/data/updated Nov 8 15:54:14 vhp4 xenstored: A10 w event /local/domain/18/data/updated /local/domain/18/data/updated Nov 8 15:54:14 vhp4 kernel: [2374513.185942] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:14 vhp4 kernel: [2374513.185973] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:14 vhp4 kernel: [2374513.384826] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:14 vhp4 kernel: [2374513.384857] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:14 vhp4 kernel: [2374513.583911] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:14 vhp4 kernel: [2374513.583934] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:14 vhp4 kernel: [2374513.782826] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:14 vhp4 kernel: [2374513.782860] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:15 vhp4 kernel: [2374513.981685] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:15 vhp4 kernel: [2374513.981718] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:15 vhp4 kernel: [2374514.180710] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:15 vhp4 kernel: [2374514.180734] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:15 vhp4 kernel: [2374514.379762] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:15 vhp4 kernel: [2374514.379789] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:15 vhp4 kernel: [2374514.578771] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:15 vhp4 kernel: [2374514.578795] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:15 vhp4 kernel: [2374514.777771] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:15 vhp4 kernel: [2374514.777794] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:16 vhp4 kernel: [2374514.976677] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:16 vhp4 kernel: [2374514.980412] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:16 vhp4 kernel: [2374515.179382] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:16 vhp4 kernel: [2374515.184127] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:16 vhp4 xenstored: D25 write data/meminfo_free 2197504 Nov 8 15:54:16 vhp4 xenstored: A342899 w event /local/domain/25/data/meminfo_free /local/domain/25/data/meminfo_free Nov 8 15:54:16 vhp4 kernel: [2374515.383057] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:16 vhp4 kernel: [2374515.388338] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:16 vhp4 kernel: [2374515.587494] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:16 vhp4 kernel: [2374515.592089] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:16 vhp4 kernel: [2374515.791077] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:16 vhp4 kernel: [2374515.796387] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:17 vhp4 kernel: [2374515.995302] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:17 vhp4 kernel: [2374516.000679] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:17 vhp4 xenstored: D10 write data/meminfo_free 70136 Nov 8 15:54:17 vhp4 xenstored: A342899 w event /local/domain/10/data/meminfo_free /local/domain/10/data/meminfo_free Nov 8 15:54:17 vhp4 xenstored: D10.47422 rm attr/eth0 Nov 8 15:54:17 vhp4 xenstored: D10 write attr/eth0/ip 10.1.1.135 Nov 8 15:54:17 vhp4 xenstored: D10 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:187 Nov 8 15:54:17 vhp4 xenstored: D10 write data/updated Tue Nov 8 15:54:17 CST 2016 Nov 8 15:54:17 vhp4 xenstored: A6 w event /local/domain/10/data/updated /local/domain/10/data/updated Nov 8 15:54:17 vhp4 xenstored: A10 w event /local/domain/10/data/updated /local/domain/10/data/updated Nov 8 15:54:17 vhp4 kernel: [2374516.199813] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:17 vhp4 kernel: [2374516.204627] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:17 vhp4 xenstored: D4.46708 rm attr/eth0 Nov 8 15:54:17 vhp4 xenstored: D4 write attr/eth0/ip 10.1.1.128 Nov 8 15:54:17 vhp4 xenstored: D4 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:180 Nov 8 15:54:17 vhp4 kernel: [2374516.403529] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:17 vhp4 kernel: [2374516.408138] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:17 vhp4 kernel: [2374516.607031] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:17 vhp4 kernel: [2374516.613493] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:17 vhp4 kernel: [2374516.812295] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:17 vhp4 kernel: [2374516.818309] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:18 vhp4 kernel: [2374517.017088] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:18 vhp4 kernel: [2374517.023075] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:18 vhp4 kernel: [2374517.221985] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:18 vhp4 kernel: [2374517.227619] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:18 vhp4 kernel: [2374517.426555] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:18 vhp4 kernel: [2374517.430854] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:18 vhp4 kernel: [2374517.629717] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:18 vhp4 kernel: [2374517.635068] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:18 vhp4 xenstored: D7 write data/meminfo_free 4879700 Nov 8 15:54:18 vhp4 xenstored: A342899 w event /local/domain/7/data/meminfo_free /local/domain/7/data/meminfo_free Nov 8 15:54:18 vhp4 xenstored: D7.46706 rm attr/eth0 Nov 8 15:54:18 vhp4 xenstored: D7 write attr/eth0/ip 10.1.0.92 Nov 8 15:54:18 vhp4 xenstored: D7 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:5c Nov 8 15:54:18 vhp4 xenstored: D7 write data/updated Tue Nov 8 15:54:18 CST 2016 Nov 8 15:54:18 vhp4 xenstored: A6 w event /local/domain/7/data/updated /local/domain/7/data/updated Nov 8 15:54:18 vhp4 xenstored: A10 w event /local/domain/7/data/updated /local/domain/7/data/updated Nov 8 15:54:19 vhp4 kernel: [2374517.833900] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:19 vhp4 kernel: [2374517.838795] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:19 vhp4 kernel: [2374518.037938] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:19 vhp4 kernel: [2374518.043749] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:19 vhp4 kernel: [2374518.242685] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:19 vhp4 kernel: [2374518.247319] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:19 vhp4 kernel: [2374518.446219] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:19 vhp4 kernel: [2374518.452079] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 8 15:54:19 vhp4 xenstored: D3 write data/meminfo_free 516020 Nov 8 15:54:19 vhp4 xenstored: A342899 w event /local/domain/3/data/meminfo_free /local/domain/3/data/meminfo_free Nov 8 15:54:19 vhp4 xenstored: D3.81063 rm attr/eth0 Nov 8 15:54:19 vhp4 xenstored: D3 write attr/eth0/ip 10.1.0.241 Nov 8 15:54:19 vhp4 xenstored: D3 write attr/eth0/ipv6/0/addr fe80::216:3eff:fe01:f1 Nov 8 15:54:19 vhp4 xenstored: D3 write data/updated Tue Nov 8 15:54:19 CST 2016 Nov 8 15:54:19 vhp4 xenstored: A6 w event /local/domain/3/data/updated /local/domain/3/data/updated Nov 8 15:54:19 vhp4 xenstored: A10 w event /local/domain/3/data/updated /local/domain/3/data/updated Nov 8 15:54:19 vhp4 kernel: [2374518.651141] bnx2x: [bnx2x_issue_dmae_with_comp:744(eth1)]DMAE timeout! Nov 8 15:54:19 vhp4 kernel: [2374518.655442] bnx2x: [bnx2x_write_dmae:793(eth1)]DMAE returned failure -1 Nov 08 15:56:59 vhp4 syslogd 1.4.1: restart.
