This discussion has been locked.

You can no longer post new replies to this discussion. If you have a question you can start a new discussion

100% CPU - high IO wait

Hi,

I'm running an ASG 220 with v7.002. Since some days I have severe trouble because CPU is running on 100%. I found that IO wait is often using between 70% and 80%.
How can I find out which process is causing this?

-tim-

This thread was automatically locked due to age.

Parents

0 drees over 18 years ago

The just released v7.003 will likely fix your issue as it's probably the http content scanner sucking up CPU and memory. Update and see if it helps or not.
Cancel
Vote Up 0 Vote Down

Cancel
0 -tim- over 18 years ago in reply to drees

I'll update to v7.003 today as it hopefully also fixes the IPsec stability issues I have.

I got the info about high IO wait out of top. All processes were showing %CPU below 10 and us and sy values were very low in top.
So I couldn't identify which process is causing this.
Cancel
Vote Up 0 Vote Down

Cancel
0 drees over 18 years ago in reply to -tim-

Besides top, grabbing a dozen lines from the output of `vmstat 1` is also helpful.

If you grab a screenshot of both that would be helpful if you wish to get further input on what is causing your load issues.
Cancel
Vote Up 0 Vote Down

Cancel
0 ScottG over 18 years ago in reply to drees

I get 100% usage when itunes is downloading videos (https://community.sophos.com/products/unified-threat-management/astaroorg/f/54/t/38328).

I'm hoping v7.003 fixes it.

--Scott
Cancel
Vote Up 0 Vote Down

Cancel
0 -tim- over 18 years ago in reply to ScottG

great, cpu dropped from 80-100% to below 20% after upgrading to v7.003.
Cancel
Vote Up 0 Vote Down

Cancel

0 -tim- over 18 years ago in reply to -tim-

well, all problems are back with v7.004.
here is my output from top and vmstat 1:


top - 15:03:25 up 6 days, 23:07,  1 user,  load average: 2.80, 3.67, 2.99

Tasks: 106 total,   3 running, 103 sleeping,   0 stopped,   0 zombie

Cpu(s):  6.0%us,  2.3%sy,  0.0%ni,  0.0%id, 91.1%wa,  0.7%hi,  0.0%si,  0.0%st

Mem:    515304k total,   510292k used,     5012k free,     2376k buffers

Swap:  1052248k total,   531240k used,   521008k free,   117320k cached



  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                            

22710 root      25   0 19676 3928 2996 S  1.3  0.8   0:00.04 httpproxy                                                          

 3120 root      15   0 14784 5136 1452 S  1.0  1.0  33:25.02 selfmonng.plx                                                      

22693 root      16   0  5044 1904 1560 S  0.7  0.4   0:00.02 sshd                                                               

   58 root      15   0     0    0    0 D  0.3  0.0   0:40.07 kswapd0                                                            

22634 root      16   0  2172 1132  828 R  0.3  0.2   0:00.04 top                                                                

22700 sshd      15   0  5020 1408 1080 S  0.3  0.3   0:00.01 sshd                                                               

    1 root      16   0   716   68   40 S  0.0  0.0   0:09.82 init                                                               

    2 root      34  19     0    0    0 R  0.0  0.0   0:08.65 ksoftirqd/0                                                        

    3 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/0                                                           

    4 root      10  -5     0    0    0 S  0.0  0.0   0:00.01 khelper                                                            

    5 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kthread                                                            

    7 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/0                                                          

   10 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khubd                                                              

   59 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0                                                              

  649 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kseriod                                                            

 1451 root      15   0     0    0    0 D  0.0  0.0   0:18.48 kjournald                                                          

 1492 root      12  -4  1824  404  344 S  0.0  0.1   0:12.35 udevd                                                              

 2201 root      19   0     0    0    0 S  0.0  0.0   0:00.00 kjournald                                                          

 2203 root      15   0     0    0    0 S  0.0  0.0   0:15.76 kjournald                                                          

 2205 root      15   0     0    0    0 D  0.0  0.0   0:27.68 kjournald                                                          

 2207 root      15   0     0    0    0 D  0.0  0.0   0:05.56 kjournald                                                          

 2371 root      15   0  1504  184  164 S  0.0  0.0   0:10.85 rngd                                                               

 2456 root      16   0  2904 1292  556 S  0.0  0.3   7:18.67 syslog-ng                                                          

 2491 messageb  16   0  3420  468  464 S  0.0  0.1   0:00.05 dbus-daemon                                                        

 2494 root      16   0 18820 1032  772 D  0.0  0.2   2:06.77 ulogd                                                              

 2541 root      15   0  2284  912  584 S  0.0  0.2   0:27.38 lcd4linux                                                          

 2569 root      18   0  1596  204  200 S  0.0  0.0   0:00.00 resmgrd                                                            

 2599 root      16   0  4964  860  652 S  0.0  0.2   0:04.39 hald                                                               

 2708 root      16   0 12108 4012 1124 S  0.0  0.8  10:08.06 dns-resolver.pl                                                    

 2710 root      16   0  8752 2352 1044 S  0.0  0.5   0:59.51 emailpki-sync.p                                                    

 2736 root      15   0 52832  20m 1304 S  0.0  4.1 176:18.98 confd.plx                                                          

 2738 root      17   0  8188 2568 1012 S  0.0  0.5   0:30.61 sysmond                                                            

 2838 root      16   0 28500  10m 1184 S  0.0  2.1   0:03.91 aua.bin                                                            

 2862 root      16   0 44692 3556  900 S  0.0  0.7   0:05.18 confd.plx                                                          

 2868 root      16   0 55828  23m 1320 S  0.0  4.6  69:30.85 confd.plx                                                          

 2872 root      16   0 56084  18m 1320 S  0.0  3.7   0:14.53 confd.plx                                                          

 2873 root      16   0 55620  23m 1316 S  0.0  4.6 129:04.08 confd.plx                                                          

 2971 root      16   0  1804  648  560 S  0.0  0.1   0:01.39 cron                                                               

 3036 root      16   0  1764  336  332 S  0.0  0.1   0:00.00 asg_ha_zeroconf                                                    

 3099 root      16   0 22816 9004 1448 S  0.0  1.7  65:59.83 mdw_daemon.plx                                                     

 3109 root      16   0  2528  484  480 S  0.0  0.1   0:00.00 S90beeps                                                           

 3123 root      16   0  1504  492  448 S  0.0  0.1   0:00.87 daemon-watcher


# vmstat 1

procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

 2  8 531156   5160   2400 117404   16    9    28    43   11   39 13  2 81  4  0

 0  8 530448   5904   2396 117304 1016 2052  1032  2060  437  653  4  2  0 94  0

 0  8 530024   5712   2388 117296 1208  352  1208   352  366  549  0  0  0 100  0

 0  8 529816   5620   2392 117284  928 1348   932  1380  362  586  2  4  0 94  0

 0  8 529444   5336   2400 117256 1000  172  1012   180  377  553  1  0  0 99  0

 0  7 529152   5164   2412 117256  608  180   608   260  370  600  3  3  0 94  0

 1  7 528904   5904   2416 117236  912  112   912   116  361  501  2  1  0 97  0

 0  6 528460   5784   2416 117236 1084    0  1084    72  366  554  3  2  0 95  0

 0  7 528116   5940   2416 117232  936  120   936   120  383  374  1  0  0 99  0

 0  6 527540   5788   2432 117204  900   88   904   148  351  384  1  1  0 98  0

 0  6 526652   5436   2456 117252  892 2384  1072  2396  365  547  5  1  0 94  0

 0 10 526400   5284   2480 117668  556  848   640  1288  432  695  2  4  0 94  0

 0 10 526188   5604   2480 117568  564  656   572   664  388  587  2  2  0 96  0

 0  7 526124   5720   2484 117644  656   56   760    56  406  590  5  3  0 92  0

 0  3 526124   6332   2548 117448  476    0   488   320  394  569  7  1  0 92  0

 0  1 526124   5848   2584 117344  816    0   924    96  357  753 25  6  0 69  0

 0  3 526300   5696   2624 118320 1132 1012  1952  1300  364  563 11  3  0 86  0

 0  2 526124   5940   2628 118524  648 2340   880  2744  476  732 10  2  0 88  0

 0  2 526124   6428   2632 118520  608    0   608    60  411  608  6  1  0 93  0

 0  1 526124   6248   2696 118328 1496    0  1500   512  432  853 27  5  0 68  0

Reply

0 -tim- over 18 years ago in reply to -tim-

well, all problems are back with v7.004.
here is my output from top and vmstat 1:


top - 15:03:25 up 6 days, 23:07,  1 user,  load average: 2.80, 3.67, 2.99

Tasks: 106 total,   3 running, 103 sleeping,   0 stopped,   0 zombie

Cpu(s):  6.0%us,  2.3%sy,  0.0%ni,  0.0%id, 91.1%wa,  0.7%hi,  0.0%si,  0.0%st

Mem:    515304k total,   510292k used,     5012k free,     2376k buffers

Swap:  1052248k total,   531240k used,   521008k free,   117320k cached



  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                            

22710 root      25   0 19676 3928 2996 S  1.3  0.8   0:00.04 httpproxy                                                          

 3120 root      15   0 14784 5136 1452 S  1.0  1.0  33:25.02 selfmonng.plx                                                      

22693 root      16   0  5044 1904 1560 S  0.7  0.4   0:00.02 sshd                                                               

   58 root      15   0     0    0    0 D  0.3  0.0   0:40.07 kswapd0                                                            

22634 root      16   0  2172 1132  828 R  0.3  0.2   0:00.04 top                                                                

22700 sshd      15   0  5020 1408 1080 S  0.3  0.3   0:00.01 sshd                                                               

    1 root      16   0   716   68   40 S  0.0  0.0   0:09.82 init                                                               

    2 root      34  19     0    0    0 R  0.0  0.0   0:08.65 ksoftirqd/0                                                        

    3 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/0                                                           

    4 root      10  -5     0    0    0 S  0.0  0.0   0:00.01 khelper                                                            

    5 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kthread                                                            

    7 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/0                                                          

   10 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khubd                                                              

   59 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0                                                              

  649 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kseriod                                                            

 1451 root      15   0     0    0    0 D  0.0  0.0   0:18.48 kjournald                                                          

 1492 root      12  -4  1824  404  344 S  0.0  0.1   0:12.35 udevd                                                              

 2201 root      19   0     0    0    0 S  0.0  0.0   0:00.00 kjournald                                                          

 2203 root      15   0     0    0    0 S  0.0  0.0   0:15.76 kjournald                                                          

 2205 root      15   0     0    0    0 D  0.0  0.0   0:27.68 kjournald                                                          

 2207 root      15   0     0    0    0 D  0.0  0.0   0:05.56 kjournald                                                          

 2371 root      15   0  1504  184  164 S  0.0  0.0   0:10.85 rngd                                                               

 2456 root      16   0  2904 1292  556 S  0.0  0.3   7:18.67 syslog-ng                                                          

 2491 messageb  16   0  3420  468  464 S  0.0  0.1   0:00.05 dbus-daemon                                                        

 2494 root      16   0 18820 1032  772 D  0.0  0.2   2:06.77 ulogd                                                              

 2541 root      15   0  2284  912  584 S  0.0  0.2   0:27.38 lcd4linux                                                          

 2569 root      18   0  1596  204  200 S  0.0  0.0   0:00.00 resmgrd                                                            

 2599 root      16   0  4964  860  652 S  0.0  0.2   0:04.39 hald                                                               

 2708 root      16   0 12108 4012 1124 S  0.0  0.8  10:08.06 dns-resolver.pl                                                    

 2710 root      16   0  8752 2352 1044 S  0.0  0.5   0:59.51 emailpki-sync.p                                                    

 2736 root      15   0 52832  20m 1304 S  0.0  4.1 176:18.98 confd.plx                                                          

 2738 root      17   0  8188 2568 1012 S  0.0  0.5   0:30.61 sysmond                                                            

 2838 root      16   0 28500  10m 1184 S  0.0  2.1   0:03.91 aua.bin                                                            

 2862 root      16   0 44692 3556  900 S  0.0  0.7   0:05.18 confd.plx                                                          

 2868 root      16   0 55828  23m 1320 S  0.0  4.6  69:30.85 confd.plx                                                          

 2872 root      16   0 56084  18m 1320 S  0.0  3.7   0:14.53 confd.plx                                                          

 2873 root      16   0 55620  23m 1316 S  0.0  4.6 129:04.08 confd.plx                                                          

 2971 root      16   0  1804  648  560 S  0.0  0.1   0:01.39 cron                                                               

 3036 root      16   0  1764  336  332 S  0.0  0.1   0:00.00 asg_ha_zeroconf                                                    

 3099 root      16   0 22816 9004 1448 S  0.0  1.7  65:59.83 mdw_daemon.plx                                                     

 3109 root      16   0  2528  484  480 S  0.0  0.1   0:00.00 S90beeps                                                           

 3123 root      16   0  1504  492  448 S  0.0  0.1   0:00.87 daemon-watcher


# vmstat 1

procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

 2  8 531156   5160   2400 117404   16    9    28    43   11   39 13  2 81  4  0

 0  8 530448   5904   2396 117304 1016 2052  1032  2060  437  653  4  2  0 94  0

 0  8 530024   5712   2388 117296 1208  352  1208   352  366  549  0  0  0 100  0

 0  8 529816   5620   2392 117284  928 1348   932  1380  362  586  2  4  0 94  0

 0  8 529444   5336   2400 117256 1000  172  1012   180  377  553  1  0  0 99  0

 0  7 529152   5164   2412 117256  608  180   608   260  370  600  3  3  0 94  0

 1  7 528904   5904   2416 117236  912  112   912   116  361  501  2  1  0 97  0

 0  6 528460   5784   2416 117236 1084    0  1084    72  366  554  3  2  0 95  0

 0  7 528116   5940   2416 117232  936  120   936   120  383  374  1  0  0 99  0

 0  6 527540   5788   2432 117204  900   88   904   148  351  384  1  1  0 98  0

 0  6 526652   5436   2456 117252  892 2384  1072  2396  365  547  5  1  0 94  0

 0 10 526400   5284   2480 117668  556  848   640  1288  432  695  2  4  0 94  0

 0 10 526188   5604   2480 117568  564  656   572   664  388  587  2  2  0 96  0

 0  7 526124   5720   2484 117644  656   56   760    56  406  590  5  3  0 92  0

 0  3 526124   6332   2548 117448  476    0   488   320  394  569  7  1  0 92  0

 0  1 526124   5848   2584 117344  816    0   924    96  357  753 25  6  0 69  0

 0  3 526300   5696   2624 118320 1132 1012  1952  1300  364  563 11  3  0 86  0

 0  2 526124   5940   2628 118524  648 2340   880  2744  476  732 10  2  0 88  0

 0  2 526124   6428   2632 118520  608    0   608    60  411  608  6  1  0 93  0

 0  1 526124   6248   2696 118328 1496    0  1500   512  432  853 27  5  0 68  0

Children

0 drees over 18 years ago in reply to -tim-

Tim, your machine is swapping horribly which is why it's so slow.

512MB of swap space used is not good and it is actively swapping memory in/out. Adding another 512MB memory to you ASG to get a total of 1GB would be extremely helpful, too bad it's not upgradeable from what I understand.

You need to figure out how to reduce memory utilization. I would also open a trouble ticket with Astaro for further troubleshooting.

IMO, Astaro needs to do a lot of work in the memory usage area, it simply uses way too much memory. Supposedly some fixes are coming in the next revision.

You can see what your top memory hogs by either sorting while viewing top (press until the output is sorted by VIRT) or using the command: ps aux --sort -vsz | head -25 which will give you the top 24 memory hogs.

On my test v7 the top 10 memory hogs are:

loginuser@fw-test:/home/login > ps aux --sort -vsz | head -11
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
mysql     3594  1.2  3.8 116572 19992 ?        Sl   May23 222:49 /usr/sbin/mysqld --basedir=/usr --datadir=/var/storage/mysql --user=mysql --pid-file=/var/lib/mysql/mysqld.pid --skip-locking --port=3306 --socket=/var/chroot-cffd/var/lib/mysql/mysql.sock
root      5553  0.0  8.9  87300 46012 ?        Ss   May23   1:40 /sbin/snort_inline -D -Q -c /etc/snort/snort.conf
root      3340  0.0  6.4  45672 33464 ?        S    May23   0:06 /var/mdw/mdw_daemon.plx
root      4746  0.0  3.3  39432 17440 ?        S    May23   1:02 prpc [system]
root      3398  0.0  3.2  39400 16760 ?        S    May23   0:03 prpc [system]
root      3083  0.0  3.2  39276 16692 ?        S    May23   0:00 prpc [system]
root      3105  0.0  3.3  39276 17060 ?        S    May23   0:36 prpc [system]
root      3141  0.0  3.2  39272 16940 ?        S    May23   0:00 prpc [system]
root      2972  0.0  3.1  38436 16204 ?        Ss   May23   0:00 confd [master]
root      3057  0.0  2.9  38164 15224 ?        S    May23   0:07 confd [prpc]
loginuser@fw-test:/home/login > free
             total       used       free     shared    buffers     cached
Mem:        515264     480468      34796          0      84232     142740
-/+ buffers/cache:     253496     261768
Swap:      1052248      44440    1007808

MySQL is the biggest memory hog on my system and you while the default settings are fairly memory frugal, you can reduce it's utilization by about 30MB by editing /etc/my.cnf and /etc/my.cnf-default and changing these settings (old settings are commented):

# key_buffer = 16M
key_buffer = 1M
# sort_buffer_size = 512K
sort_buffer_size = 128K
# read_buffer_size = 256K
read_buffer_size = 128K
# myisam_sort_buffer_size = 8M
myisam_sort_buffer_size = 128K
# Set this value, default is 8M
innodb_buffer_pool_size = 1M

Then restart mysql: service mysql restart

On a system that isn't swapping this will hurt the performance of whatever Astaro uses mysql for (appears to use it to store certificates and quarantined emails) but if a machine is swapping it should help.

Have a look at this post/thread for great info on how to significantly reduce squid memory utilization:

https://community.sophos.com/products/unified-threat-management/astaroorg/f/55/p/43343/153434#153434

0 -tim- over 18 years ago in reply to drees

hi,

many thanks for the detailed infos.
i'll open a ticket with astaro with these detailes and i'm very interested in what they say. it's an asg which came with v7 factory-installed, so not enough ram shouldn't be my problem at all....

thanks again...

tim
Cancel
Vote Up 0 Vote Down

Cancel
0 William Warren over 18 years ago in reply to -tim-

http://astaro.org/showthread.php?t=18776
Cancel
Vote Up 0 Vote Down

Cancel
0 BarryG over 18 years ago in reply to -tim-

On my machine running 7.004 with 54 days of uptime, MySQL was using 291M VSZ and 165M RSS!

This was causing some swapping.

I'm not running any of the proxies, btw.

I've upgraded to 7.005 and upgraded my RAM to 512MB, and followed William's suggestion of changing the kernel swappiness value, so we'll see how it goes.
I haven't edited the my.cnf yet.

The strange thing is the /var/storage/mysql/ directory is only 29MB, so I don't see what it could possibly be caching,
UNLESS lots of temporary tables are being created by Astaro and not being freed.

Barry
Cancel
Vote Up 0 Vote Down

Cancel