Hello !
So, did you already missing for Github digest? OK, OK, it's in progress, gimme more time please.
But now I want to talk about new exciting tool, again.
If you consider yourself as experienced Linux sysadmin and still do not reading Brendan Gregg's blog - you are making big mistake! Please follow it through RSS or follow Gregg's twitter @brendangregg right now - I will wait. :)
Before Gregg was working in Joyent and most of his posts was connected with DTrace, but now he moved to Netflix, and most of his work now connected with Linux.
So, Gregg was ported own iosnoop tool for Linux now. What is iosnoop? It's like a tcpdump, but for disk activity. It's hard to believe, but despite of its practical top of 10000 IOPS - really iosnoop is a bash script (more precisely - combination of bash with optimized awk) and it works out of the box on most modern kernels (even debug symbols not required)!
You can say - 'Ok, it's cool new small nifty tool, which provides some Kbs of text output, meh'. Not really impressive, right? But Gregg also provide cool thing to visualize that (or similar) data - heatmaps - and also provide tooling for that. Check that great post for making latency heatmap from iosnoop output.
So, let's try that stuff on something practical. I'm suspecting that newly installed KVM server is not behaving OK, I/O seems too slow. How we can detect that? Ok, let's reboot some VM, run iosnoop for 180 sec and compare heatmaps of reboot process.
Let's do that first on some normal server first:
I cap output at 100ms, just for simplicity, but it's not critical - real value took about 150 ms, because of queueing. Server was rebooted near 80 sec timepoint, and process was finished near 120 sec.
But compare it with target server:
It looks definitely not OK. Results are much more scattered, and queueing effect looks really terrible - top delay reach almost 6000 ms! Why? It's another question, I'll try to find that tomorrow. :)
One more time - please check Gregg's blog and his github repo - you can find much more cool things there!
So, did you already missing for Github digest? OK, OK, it's in progress, gimme more time please.
But now I want to talk about new exciting tool, again.
If you consider yourself as experienced Linux sysadmin and still do not reading Brendan Gregg's blog - you are making big mistake! Please follow it through RSS or follow Gregg's twitter @brendangregg right now - I will wait. :)
Before Gregg was working in Joyent and most of his posts was connected with DTrace, but now he moved to Netflix, and most of his work now connected with Linux.
So, Gregg was ported own iosnoop tool for Linux now. What is iosnoop? It's like a tcpdump, but for disk activity. It's hard to believe, but despite of its practical top of 10000 IOPS - really iosnoop is a bash script (more precisely - combination of bash with optimized awk) and it works out of the box on most modern kernels (even debug symbols not required)!
You can say - 'Ok, it's cool new small nifty tool, which provides some Kbs of text output, meh'. Not really impressive, right? But Gregg also provide cool thing to visualize that (or similar) data - heatmaps - and also provide tooling for that. Check that great post for making latency heatmap from iosnoop output.
So, let's try that stuff on something practical. I'm suspecting that newly installed KVM server is not behaving OK, I/O seems too slow. How we can detect that? Ok, let's reboot some VM, run iosnoop for 180 sec and compare heatmaps of reboot process.
Let's do that first on some normal server first:
I cap output at 100ms, just for simplicity, but it's not critical - real value took about 150 ms, because of queueing. Server was rebooted near 80 sec timepoint, and process was finished near 120 sec.
But compare it with target server:
One more time - please check Gregg's blog and his github repo - you can find much more cool things there!
