Uploaded image for project: 'XenServer Org'
  1. XenServer Org
  2. XSO-552

Openvswitch high CPU usage on dom0 & logging isn't working

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • None
    • 6.5 SP1, 7.0, 7.3, 7.4, 7.5, 7.6
    • Networking
    • None

    Description

      Across all our hosts (20~) there are many Openvswitch processes running on the dom0s that regularly use a lot of CPU resources (between 30-100% of one Xeon E5-2660 v2 core).

       11  3407 root      -10 1270M  156M  8864 S 72.5  2.0 24h58:09 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:info -vfile:emer --mlockall --no-chdir --log-file=/var/log/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
        5  4100 root      -10 1270M  156M  8864 S 14.1  2.0  4h48:21 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:info -vfile:emer --mlockall --no-chdir --log-file=/var/log/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
        3  4097 root      -10 1270M  156M  8864 S 14.1  2.0  4h47:29 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:info -vfile:emer --mlockall --no-chdir --log-file=/var/log/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
        9  4096 root      -10 1270M  156M  8864 S 14.1  2.0  4h50:41 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:info -vfile:emer --mlockall --no-chdir --log-file=/var/log/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
       13  4101 root      -10 1270M  156M  8864 S 14.1  2.0  4h48:11 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:info -vfile:emer --mlockall --no-chdir --log-file=/var/log/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
        6  4099 root      -10 1270M  156M  8864 S 14.1  2.0  4h47:30 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:info -vfile:emer --mlockall --no-chdir --log-file=/var/log/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
      

      We went to check the openvswitch logs but they're empty and contain nothing:

      [root@s1-b8 log]# ps -ef | grep -iE 'ovs|openvswitch'
      root      3369     1  0 May29 ?        00:00:00 ovsdb-server: monitoring pid 3371 (healthy)
      root      3371  3369  0 May29 ?        00:04:27 ovsdb-server /run/openvswitch/conf.db -vconsole:emer -vsyslog:info -vfile:emer --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir --log-file=/var/log/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
      root      3406     1  0 May29 ?        00:00:00 ovs-vswitchd: monitoring pid 3407 (healthy)
      root      3407  3406 36 May29 ?        1-00:59:07 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:info -vfile:emer --mlockall --no-chdir --log-file=/var/log/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
      root      3557     1  0 May29 ?        00:01:26 /usr/bin/python /usr/share/openvswitch/scripts/ovs-xapi-sync unix:/var/run/openvswitch/db.sock --log-file
      [root@s1-b8 log]# file /var/log/ovs-vswitchd.log
      /var/log/ovs-vswitchd.log: empty
      

      and because journalctl has been set not to log in XenServer 7 I can't get any logs or metrics from systemd regarding the service (See XSO-543)

      This occurs both on XenServer 6.5 SP1 and 7.0

      Attachments

        Issue Links

          Activity

            People

              anoobs Anoob Soman
              s_mcleod Sam McLeod
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: