Uploaded image for project: 'XenServer Org'
  1. XenServer Org
  2. XSO-552

Openvswitch high CPU usage on dom0 & logging isn't working

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • None
    • 6.5 SP1, 7.0, 7.3, 7.4, 7.5, 7.6
    • Networking
    • None

    Description

      Across all our hosts (20~) there are many Openvswitch processes running on the dom0s that regularly use a lot of CPU resources (between 30-100% of one Xeon E5-2660 v2 core).

       11  3407 root      -10 1270M  156M  8864 S 72.5  2.0 24h58:09 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:info -vfile:emer --mlockall --no-chdir --log-file=/var/log/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
        5  4100 root      -10 1270M  156M  8864 S 14.1  2.0  4h48:21 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:info -vfile:emer --mlockall --no-chdir --log-file=/var/log/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
        3  4097 root      -10 1270M  156M  8864 S 14.1  2.0  4h47:29 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:info -vfile:emer --mlockall --no-chdir --log-file=/var/log/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
        9  4096 root      -10 1270M  156M  8864 S 14.1  2.0  4h50:41 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:info -vfile:emer --mlockall --no-chdir --log-file=/var/log/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
       13  4101 root      -10 1270M  156M  8864 S 14.1  2.0  4h48:11 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:info -vfile:emer --mlockall --no-chdir --log-file=/var/log/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
        6  4099 root      -10 1270M  156M  8864 S 14.1  2.0  4h47:30 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:info -vfile:emer --mlockall --no-chdir --log-file=/var/log/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
      

      We went to check the openvswitch logs but they're empty and contain nothing:

      [root@s1-b8 log]# ps -ef | grep -iE 'ovs|openvswitch'
      root      3369     1  0 May29 ?        00:00:00 ovsdb-server: monitoring pid 3371 (healthy)
      root      3371  3369  0 May29 ?        00:04:27 ovsdb-server /run/openvswitch/conf.db -vconsole:emer -vsyslog:info -vfile:emer --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir --log-file=/var/log/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
      root      3406     1  0 May29 ?        00:00:00 ovs-vswitchd: monitoring pid 3407 (healthy)
      root      3407  3406 36 May29 ?        1-00:59:07 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:info -vfile:emer --mlockall --no-chdir --log-file=/var/log/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
      root      3557     1  0 May29 ?        00:01:26 /usr/bin/python /usr/share/openvswitch/scripts/ovs-xapi-sync unix:/var/run/openvswitch/db.sock --log-file
      [root@s1-b8 log]# file /var/log/ovs-vswitchd.log
      /var/log/ovs-vswitchd.log: empty
      

      and because journalctl has been set not to log in XenServer 7 I can't get any logs or metrics from systemd regarding the service (See XSO-543)

      This occurs both on XenServer 6.5 SP1 and 7.0

      Attachments

        1. Capture.JPG
          114 kB
          Tobias Kreidl
        2. Capture2.JPG
          94 kB
          Tobias Kreidl
        3. openvswitch sort by active.jpg
          587 kB
          Sam McLeod
        4. openvswitch sort by most time taken.jpg
          667 kB
          Sam McLeod

        Issue Links

          Activity

            People

              anoobs Anoob Soman
              s_mcleod Sam McLeod
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: