Last week
Draios company made bold move - they made their Linux tracing tool
Sysdig open-source.
What is Sysdig? As it says on own website - "strace + tcpdump + lsof + awesome sauce".
And I think that tool is really quite awesome.
Installation for daredevils is quite simple -
curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash
But for more responsible sysadmins there's
good manual - set up sysdig repo and install it (you'll need linux headers and
DKMS for automatic kernel module build and installation).
You can start learning sysdig using this
simple examples:
- See the top processes in terms of network bandwidth usage:
sysdig -c topprocs_net
See the top local server ports (in terms of total bytes)
sysdig -c fdbytes_by fd.sport
See the top client IPs (in terms of established connections)
sysdig -c fdcount_by fd.cip "evt.type=accept"
Топ процессов по использованию диска
sysdig -c topprocs_file
Print the top files that apache has been reading from or writing to
sysdig -c topfiles_bytes proc.name=httpd
List the processes that are using a high number of files
sysdig -c fdcount_by proc.name "fd.type=file"
See the files where most time has been spent
sysdig -c topfiles_time
See queries made via apache to an external MySQL server happening in real time
sysdig -A -c echo_fds fd.sip=192.168.30.5 and proc.name=apache2 and evt.buffer contains SELECT
You can also record all events server-wide (or process-wide, or using other sysdig filter):
sysdig -w out.scap proc.name=httpd
and analyze that later, using even MAC or Windows workstation.
Also there is a framework for Lua -
Chisel - you can write a simple script and execute them immediately at sysdig run.
However, there's an open question still - how much additional load sysdig brings to server.
Let's make a simple test. I have an small virtual machine on Ubuntu 12.04, 1 GB of RAM, with Percona Mysql 5.6 installed.
- Install sysbench:
sudo apt-get install sysbench
- Create empty database 'sbtest' and fill it with test data:
sysbench --test=oltp --mysql-table-engine=innodb --oltp-table-size=10000 --mysql-user=root --mysql-password=root --db-driver=mysql --test=oltp prepare
- Run sysbench
root@ubuntu:~# sysbench --num-threads=8 --max-requests=5000 --oltp-table-size=10000 --mysql-user=root --mysql-password=root --db-driver=mysql --test=oltp run
- Results
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 8
Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 5000
Threads started!
Done.
OLTP test statistics:
queries performed:
read: 70014
write: 25003
other: 10001
total: 105018
transactions: 5000 (123.58 per sec.)
deadlocks: 1 (0.02 per sec.)
read/write requests: 95017 (2348.49 per sec.)
other operations: 10001 (247.19 per sec.)
Test execution summary:
total time: 40.4587s
total number of events: 5000
total time taken by event execution: 323.5886
per-request statistics:
min: 5.83ms
avg: 64.72ms
max: 8020.75ms
approx. 95 percentile: 168.71ms
Threads fairness:
events (avg/stddev): 625.0000/24.51
execution time (avg/stddev): 40.4486/0.01
Delete sbtest database, reboot virtual machine, repeat p.1 and 2
- Run sysdig in separate terminal:
root@ubuntu:~# sysdig -w /root/mysqld.scap proc.name=mysqld
- Re-run test
root@ubuntu:~# sysbench --num-threads=8 --max-requests=5000 --oltp-table-size=10000 --mysql-user=root --mysql-password=root --db-driver=mysql --test=oltp run
- Results
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 8
Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 5000
Threads started!
Done.
OLTP test statistics:
queries performed:
read: 70014
write: 25002
other: 10001
total: 105017
transactions: 5000 (71.62 per sec.)
deadlocks: 1 (0.01 per sec.)
read/write requests: 95016 (1360.97 per sec.)
other operations: 10001 (143.25 per sec.)
Test execution summary:
total time: 69.8150s
total number of events: 5000
total time taken by event execution: 558.1830
per-request statistics:
min: 9.35ms
avg: 111.64ms
max: 1590.65ms
approx. 95 percentile: 304.89ms
Threads fairness:
events (avg/stddev): 625.0000/39.17
execution time (avg/stddev): 69.7729/0.02
So, average query time has almost doubled - 111 ms instead of 65 ms. Not very impressive. Truthfully speaking, test was quite artificial and not very methodologically correct though....
