Performance monitoring with Sysdig

The Sysdig software seems highly geared towards containers like Docker but it works just as well on bare metal. The software is scriptable much like dtrace is.

In order to work with this utility we must first install it.

  • Download the following playbook, set the correct IP address in the ansible hosts file and run it:
git clone https://gitlab.com/youvegotmoxie/sysdig-ansible.git
cd sysdig-ansible/
$ ansible-playbook -u root playbook.yml

Using sysdig:

  • See the top processes in terms of network bandwidth usage:
# sysdig -c topprocs_net
Bytes               Process             PID
--------------------------------------------------------------------------------
71.88KB             php-fpm             2865
71.70KB             memcached           1774
760B                nginx               2148
68B                 sshd                4608
  • List processes that are using files by highest first, use ctrl+c to display statistics after the command is run:
# sysdig -c fdcount_by proc.name "fd.type=file"
lfd - processin       211
leechprotect          186
httpd                 92
service               19
sh                    11
lfd                   10
sed                    6
cat                    5
python2.7              5
grep                   5
nscd                   4
basename               4
tailwatchd             3
env                    3
consoletype            2
sshd                   2
bash                   2
  • Show all incoming nginx connections:
# sysdig -p"%proc.name %fd.name" "evt.type=accept and proc.name=nginx"
nginx 67.23.232.183:28198->192.34.58.244:https
nginx 67.23.232.183:redstorm_join->192.34.58.244:https
nginx 67.23.232.183:20087->192.34.58.244:https
nginx 67.23.232.183:24520->192.34.58.244:https
nginx 67.23.232.183:8980->192.34.58.244:https
nginx 67.23.232.183:31640->192.34.58.244:https
  • See the top files in terms of read+write bytes:
# sysdig -c topfiles_bytes
Bytes               Filename
--------------------------------------------------------------------------------
63.97KB             /var/www/html/blog/wp-includes/script-loader.php
32.00KB             /var/mysqltmp/#sql_792_0.MAD
31.98KB             /var/www/html/blog/wp-includes/class-wp-term-query.php
20.12KB             /var/www/html/blog/wp-content/w3tc-config/master.php
17.50KB             /var/mysqltmp/#sql_792_0.MAI
8.00KB              /var/www/html/blog/wp-content/themes/screenr/style.css
2.00KB              /proc/interrupts
1.18KB              /proc/stat
817B                /var/log/nginx/blog.donthurt.us-access_log
165B                /dev/ptmx
  • Show where a process is spending the most time:
# sysdig -c topfiles_time proc.name=exim
Time                Filename
--------------------------------------------------------------------------------
566us               /var/spool/exim/db/wait-remote_smtp
240us               /etc/pki/tls/cert.pem
113us               /var/spool/exim/input/1cmS7T-0001Jg-NB-J
80us                /var/log/exim/main.log
51us                /var/spool/exim/msglog/1cmS7T-0001Jg-NB
43us                /var/spool/exim/db/retry
41us                /proc/meminfo
26us                /home/mike/Maildir/tmp/1489181440.H114196P5075.centoscloud.donthurt.us
24us                /etc/services
18us                /var/spool/exim/input/1cmS7T-0001Jg-NB-D
  • Display file I/O calls that take longer than 1ms to complete:
sysdig -c fileslower 1
evt.datetime            proc.name    evt.type LATENCY(ms)  fd.name
----------------------- ------------ -------- ------------ -----------------------------------------
2017-03-10 16:32:20.318 rsync        read                6 /usr/src/kernels/3.10.0-514.10.2.el7.x86_64/include/linux/fcdevice.h
2017-03-10 16:32:20.324 rsync        read                5 /usr/src/kernels/3.10.0-514.10.2.el7.x86_64/include/linux/fd.h
2017-03-10 16:32:23.588 rsync        read                1 /usr/src/kernels/3.10.0-514.10.2.el7.x86_64/include/linux/firmware.h
2017-03-10 16:32:27.453 rsync        read                5 /usr/src/kernels/3.10.0-514.10.2.el7.x86_64/include/linux/frame.h
  • All sysdig filters can be saved to a log file for later viewing:
# sysdig -c fileslower 1 -w trace.scap
  • They can be read using the following command, replace the filter as needed depending on the output captured:
# sysdig -c fileslower 1 -r trace.scap
  • Sysdig also comes with a bunch of pre-made filters which they call chisels.

  • You can get a list of all filters using sysdig -cl

  • You can get information on a filter using the following syntax:
# sysdig -i $chisel_name
# sysdig -i bottlenecks

Category: Performance
---------------------
bottlenecks     Slowest system calls

Lists the 10 system calls that took the longest to return during the capture interval.
Args:
(None)
  • To run a filter use the -c flag:
# sysdig -c spy_users
5552 17:59:36 root) /usr/bin/id -un
5552 17:59:36 root) /usr/bin/hostname
5552 17:59:36 root) /bin/sh /usr/libexec/grepconf.sh -c
5552 17:59:36 root) grep -qsi ^COLOR.*none /etc/GREP_COLORS
5552 17:59:36 root) /usr/bin/tty -s
5552 17:59:36 root) /usr/bin/tput colors
5552 17:59:36 root) /usr/bin/dircolors --sh /etc/DIR_COLORS
5552 17:59:36 root) /usr/bin/grep -qi ^COLOR.*none /etc/DIR_COLORS
5552 17:59:36 root) /usr/bin/id -u
5552 17:59:37 root) ls --color=auto

One thought on “Performance monitoring with Sysdig

Leave a Reply

Your email address will not be published. Required fields are marked *