[8.920][BUG] Huge CPU increase because of httpd process

Hi All

The CPU went up suddenly because of httpd and since then it stayed there (55%). Before the increase the cpu was around 15%

attached is the diagram as well

wwwrun   31676 99.2  0.1  18656  4724 ?        R    Mar30 1452:42  \_ /bin/httpd -f /etc/httpd/httpd.conf

(not sure if related to https://community.sophos.com/products/unified-threat-management/astaroorg/f/75/t/63913 as I don't have SSL scanning enabled-Mantis #20610 )

The issues started 30.3.2012 11.00 AM

Only error I can see is on middleware log (the times are not the same as the high CPU)

2012:03:30-00:10:16 ***** middleware[3523]: >=========================================================================
2012:03:30-00:10:16 ***** middleware[3523]: E Failed: /usr/sbin/tc qdisc del dev eth1.10 parent 1:5 handle 2:
2012:03:30-00:10:16 ***** middleware[3523]: 
2012:03:30-00:10:16 ***** middleware[3523]:  1. utils::IPRoute::tc_cmd:949() /utils/IPRoute.pm
2012:03:30-00:10:16 ***** middleware[3523]:  2. utils::IPRoute::tc_qdisc_cmd:919() /utils/IPRoute.pm
2012:03:30-00:10:16 ***** middleware[3523]:  3. utils::IPRoute::tc_qdisc_del:911() /utils/IPRoute.pm
2012:03:30-00:10:16 ***** middleware[3523]:  4. modules::qos::cleanSystem:257() /modules/qos.pm
2012:03:30-00:10:16 ***** middleware[3523]:  5. modules::qos::setAll:157() /modules/qos.pm
2012:03:30-00:10:16 ***** middleware[3523]:  6. core::Config::load:278() /core/Config.pm
2012:03:30-00:10:16 ***** middleware[3523]:  7. main::top-level:221() mdw.pl
2012:03:30-00:10:16 ***** middleware[3523]: >=========================================================================
2012:03:30-00:10:16 ***** middleware[3523]: E Failed: /usr/sbin/tc qdisc del dev eth1.20 parent 1:3 handle 2:
2012:03:30-00:10:16 ***** middleware[3523]: 
2012:03:30-00:10:16 ***** middleware[3523]:  1. utils::IPRoute::tc_cmd:949() /utils/IPRoute.pm
2012:03:30-00:10:16 ***** middleware[3523]:  2. utils::IPRoute::tc_qdisc_cmd:919() /utils/IPRoute.pm
2012:03:30-00:10:16 ***** middleware[3523]:  3. utils::IPRoute::tc_qdisc_del:911() /utils/IPRoute.pm
2012:03:30-00:10:16 ***** middleware[3523]:  4. modules::qos::cleanSystem:257() /modules/qos.pm
2012:03:30-00:10:16 ***** middleware[3523]:  5. modules::qos::setAll:157() /modules/qos.pm
2012:03:30-00:10:16 ***** middleware[3523]:  6. core::Config::load:278() /core/Config.pm
2012:03:30-00:10:16 ***** middleware[3523]:  7. main::top-level:221() mdw.pl

or config daemon

2012:03:30-14:14:43 ***** confd[31682]: >=========================================================================
2012:03:30-14:14:43 ***** confd[31682]: E Maximal count of pending signals (120) exceeded at /rpc.pm line 426.
2012:03:30-14:14:43 ***** confd[31682]: 
2012:03:30-14:14:43 ***** confd[31682]:  1. sys::_warn:51() /sys.pm
2012:03:30-14:14:43 ***** confd[31682]:  2. sys::_die:57() /sys.pm
2012:03:30-14:14:43 ***** confd[31682]:  3. Astaro::RPC::get_request:426() /rpc.pm
2012:03:30-14:14:43 ***** confd[31682]:  4. Astaro::RPC::server_loop:180() /rpc.pm
2012:03:30-14:14:43 ***** confd[31682]:  5. rpc::launch:50() /rpc.pm
2012:03:30-14:14:43 ***** confd[31682]:  6. main::_rpc:2135() confd.pl
2012:03:30-14:14:43 ***** confd[31682]:  7. main::top-level:394() confd.pl

no /var/storage/cores available.
Thanks

Parents

0 wingman over 13 years ago

more useful info

:/var/storage/cores # ps ax -o pcpu,cmd|sort -n -r|head
99.2 /bin/httpd -f /etc/httpd/httpd.conf
 6.5 [confd.plx] 
 1.4 /var/chroot-http/usr/bin/httpproxy -f -c /var/chroot-http -u httpproxy
 1.2 confd [worker[:P]rpc:webadmin]
 0.9 /sbin/snort_inline -Q -D -c /etc/snort/snort.conf -R 1 -K none -P 65535 -g 800 -u 800 --daq nfqmnlmmap --daq-var queue=0 --daq-var queue_len=1024 --daq-var verdictcache_len=10 --daq-var verdictcache_timeout=1000 --daq-var proto=ip*
 0.6 /var/webadmin/index.plx
 0.5 /usr/local/bin/selfmonng.plx
 0.5 /usr/lib/ctasd/ctasd -p /var/run/ctasd.pid -l /usr/lib/ctasd
 0.5 ./ctipd.bin -l /usr/lib/ctipd
 0.3 /var/mdw/mdw.plx

:/var/storage/cores # ps ax -o rss,cmd |sort -n -r|head
295428 /var/chroot-http/usr/bin/httpproxy -f -c /var/chroot-http -u httpproxy
246280 /usr/bin/cssd -d
155012 /sbin/snort_inline -Q -D -c /etc/snort/snort.conf -R 1 -K none -P 65535 -g 800 -u 800 --daq nfqmnlmmap --daq-var queue=0 --daq-var queue_len=1024 --daq-var verdictcache_len=10 --daq-var verdictcache_timeout=1000 --daq-var proto=ip*
81372 /var/webadmin/index.plx
63312 /var/mdw/mdw.plx
50096 confd [worker[:P]rpc:webadmin]
35460 confd [worker[:P]rpc:system]
35180 /var/jape/index.plx
35112 confd [worker[:P]rpc:system]
34204 confd [master]

:/ # lsof -nP -i | grep 8080
httpproxy 20915 httpproxy    5u  IPv4 1307069      0t0  TCP 192.168.2.100:8080->192.168.2.5:53745 (ESTABLISHED)
httpproxy 20915 httpproxy   73u  IPv4 1300615      0t0  TCP *:8080 (LISTEN)
httpproxy 20915 httpproxy   76u  IPv6 1300617      0t0  TCP *:8080 (LISTEN)
httpproxy 20915 httpproxy   77u  IPv4 1300619      0t0  TCP *:18080 (LISTEN)
httpproxy 20915 httpproxy   78u  IPv6 1300621      0t0  TCP *:18080 (LISTEN)
httpproxy 20915 httpproxy   84u  IPv4 1302571      0t0  TCP 192.168.2.100:8080->192.168.2.2:4731 (ESTABLISHED)
httpproxy 20915 httpproxy   86u  IPv4 1301055      0t0  TCP 192.168.2.100:8080->192.168.2.5:53348 (ESTABLISHED)
httpproxy 20915 httpproxy  104u  IPv4 1302144      0t0  TCP 192.168.2.100:8080->192.168.2.5:53654 (ESTABLISHED)

I've tried to run the command "strace -p 31676 " or even output it to a file but the file is empty

Reply

0 wingman over 13 years ago

more useful info

:/var/storage/cores # ps ax -o pcpu,cmd|sort -n -r|head
99.2 /bin/httpd -f /etc/httpd/httpd.conf
 6.5 [confd.plx] 
 1.4 /var/chroot-http/usr/bin/httpproxy -f -c /var/chroot-http -u httpproxy
 1.2 confd [worker[:P]rpc:webadmin]
 0.9 /sbin/snort_inline -Q -D -c /etc/snort/snort.conf -R 1 -K none -P 65535 -g 800 -u 800 --daq nfqmnlmmap --daq-var queue=0 --daq-var queue_len=1024 --daq-var verdictcache_len=10 --daq-var verdictcache_timeout=1000 --daq-var proto=ip*
 0.6 /var/webadmin/index.plx
 0.5 /usr/local/bin/selfmonng.plx
 0.5 /usr/lib/ctasd/ctasd -p /var/run/ctasd.pid -l /usr/lib/ctasd
 0.5 ./ctipd.bin -l /usr/lib/ctipd
 0.3 /var/mdw/mdw.plx

:/var/storage/cores # ps ax -o rss,cmd |sort -n -r|head
295428 /var/chroot-http/usr/bin/httpproxy -f -c /var/chroot-http -u httpproxy
246280 /usr/bin/cssd -d
155012 /sbin/snort_inline -Q -D -c /etc/snort/snort.conf -R 1 -K none -P 65535 -g 800 -u 800 --daq nfqmnlmmap --daq-var queue=0 --daq-var queue_len=1024 --daq-var verdictcache_len=10 --daq-var verdictcache_timeout=1000 --daq-var proto=ip*
81372 /var/webadmin/index.plx
63312 /var/mdw/mdw.plx
50096 confd [worker[:P]rpc:webadmin]
35460 confd [worker[:P]rpc:system]
35180 /var/jape/index.plx
35112 confd [worker[:P]rpc:system]
34204 confd [master]

:/ # lsof -nP -i | grep 8080
httpproxy 20915 httpproxy    5u  IPv4 1307069      0t0  TCP 192.168.2.100:8080->192.168.2.5:53745 (ESTABLISHED)
httpproxy 20915 httpproxy   73u  IPv4 1300615      0t0  TCP *:8080 (LISTEN)
httpproxy 20915 httpproxy   76u  IPv6 1300617      0t0  TCP *:8080 (LISTEN)
httpproxy 20915 httpproxy   77u  IPv4 1300619      0t0  TCP *:18080 (LISTEN)
httpproxy 20915 httpproxy   78u  IPv6 1300621      0t0  TCP *:18080 (LISTEN)
httpproxy 20915 httpproxy   84u  IPv4 1302571      0t0  TCP 192.168.2.100:8080->192.168.2.2:4731 (ESTABLISHED)
httpproxy 20915 httpproxy   86u  IPv4 1301055      0t0  TCP 192.168.2.100:8080->192.168.2.5:53348 (ESTABLISHED)
httpproxy 20915 httpproxy  104u  IPv4 1302144      0t0  TCP 192.168.2.100:8080->192.168.2.5:53654 (ESTABLISHED)

I've tried to run the command "strace -p 31676 " or even output it to a file but the file is empty

Children

No Data