Random Ops Github Digest #9 (August 3rd 2014)

Hello, my fellow readers!
I decided not ot wait until tomorrow and make new digest right now - still not sure if this digest will be weekly or more random and sporadic. :)

So, let's start, again, my favourite Github projects (sometimes Ops-like) in no particular order.

1. cheat
cheat allows you to create and view interactive cheatsheets on the command-line.
https://github.com/chrisallenlane/cheat
Like man command, but you'll get small cheatsheet just with commands instead big man page. Still not sure is it good thing or not....

2. awesome-django
A curated list of awesome Django apps and projects.
https://github.com/rosarior/awesome-django
Another Awesome list, but for Django developers. Can be useful for Django Ops too.

For you, fellow Puppeteers:

3. r10k
Smarter Puppet deployment, powered by killer robots
https://github.com/adrienthebo/r10k
Relatively new tool for Puppet installations, useful if you using librarian and dynamic environments.

4. hiera-consul
Hiera backend plugin for Consul
https://github.com/lynxman/hiera-consul
If you have working Consul cluster you can get/set vualues there directly from Hiera.

Going to DevOps stuff:

5. terraform
Terraform is a tool for building, changing, and combining infrastructure safely and efficiently. http://www.terraform.io
https://github.com/hashicorp/terraform
"Infrastructure-as-code" tool - describe your infrastructure, deploy and control it on different providers. It's not Puppet replacement though - http://www.terraform.io/intro/vs/chef-puppet.html - it's just more high level tool.

6. infratester
Infrastructure Behavior Testing Framework http://infrataster.net
https://github.com/ryotarai/infrataster
Another "Infrastructure-as-code" tool - it tests infrastructure's behaviour from outside of servers:



7. devstep
Development environment builder powered by Docker and buildpacks
https://github.com/fgrehm/devstep
A dead simple, no frills development environment builder that is based around a simple goal:
"I want to git clone and run a single command to hack on any software project."
For more information please check http://fgrehm.viewdocs.io/devstep

8. libcloud-vagrant
Apache Libcloud compute provider for local Vagrant boxes
https://github.com/carletes/libcloud-vagrant
"libcloud-vagrant is a compute provider for Apache Libcloud which uses Vagrant to create VirtualBox nodes. With libcloud-vagrant installed, you could prototype a small cluster on your laptop, for instance, and then deploy it later on to Amazon, Rackspace, or any of the other clouds supported by Libcloud."

Rest of things:

9. seL4
seL4 microkernel
https://github.com/seL4/seL4
Freshly opensourced L4-based microkernel, which has formal verification proof. Not really practical thing, but interesting - check its FAQ if curious.

10. snapzend
zfs send/receive backup system http://www.znapzend.org
https://github.com/oetiker/znapzend/
Cool new backup system specifically for ZFS!

11. pghero
Database insights made easy

https://github.com/ankane/pghero
Personally I didn't work with more-less loaded instance of PostgreSQL for quite long time. But if you do - please check it out - it's nice web dashboard for pg:


Today's fun section is quite big:

12. cool-old-term
A good looking terminal emulator which mimics the old cathode display
https://github.com/Swordifish90/cool-old-term 
Looks cool - but not sure is it OK for work:




13. ld-preload-sounds

Generates raw WAV output by hooking malloc() and read().
https://github.com/gordol/ld_preload-sounds
You can now hear how your compilation sounds like!

14. conway
A real-time, persistent, multiplayer version of Conway's Game of Life
https://github.com/drewblaisdell/conway
You can roll out your own server and play with friends or play with strangers right now on http://lifecompetes.com


Small disclaimer: I'm just big fan of Github and since we have so many sources of information around I pick Github as source of information for my digest of interesting open-source projects. And because I'm Ops/Sysadmin person - I choose projects which somewhat ops-related, but not necessary. I'm not GitHub employee and this digest is not connected with GitHub in any way.

Weekly Ops Github Digest #8 (July 29th 2014)

So, it seems that our digest becoming quite regular, it's issue number 8 and number 7 was out exactly one week ago, same as issue 6 was two weeks ago, nice.

Let's try to keep it that way - new issue every Monday/Tuesday, for fresh start of new week!
... but please note that possible upcoming vacation can interfere with my intentions. :)

OK, let's go.

1. elvish
An experimental Unix shell
https://github.com/xiaq/elvish
Nice new experimental shell (written in Go), more traditional than Xiki but also with intresting new features, like nice autocompletion and navigation mode, built in daemon for syncing command history from different instances and new Clojure/Lisp-like built-in language. Worth a try, but only if you have Linux, it doesn't compile on Mac, so, if you have one and want to be a contributor - welcome.

2. perf-tools
Performance analysis tools based on Linux perf_events (aka perf) and ftrace.
https://github.com/brendangregg/perf-tools
Set of performance analyzing tools for Linux, using (relatively) new Ftrace interface, written by famous Brendan D. Gregg, author of Systems Performance book, DTrace book etc.

Will not talk much about that, in short - amazing stuff, check my previous post about it.

3. Next 3 repos connected with Docker, lets merge them together:

Fig.
Fast, isolated development environments using Docker http://www.fig.sh
https://github.com/orchardup/fig
Describe your environment in single file and run it with single command "fig up".
Looks cool? Then check out

Octohost.
Simple web focused Dockerfile based PaaS server.
http://www.octohost.io/
https://github.com/octohost/octohost
Similar idea, but realization is little bit different - and idea is more general and limited at the same time. Just add Dockerfile to your webapp and after that run "git push" to deploy your project on choosen environment (Vagrant/Amazon/Digital Ocean/Rackspace/own server) - but only limited (but quite poular) languages and platforms are supported.

Next thing -
Registrator.
Service registry bridge for Docker
https://github.com/progrium/registrator
will help you with automatic "register/deregisters services for containers based on published ports and metadata from the container environment". I.e. you can automatically add/remove services to LB, for example. External service registers/pairing/discovery are supported also. Nice thing if your building scalable application on Docker.


Going to more general developer's stuff.

4. Two Git help pages. More general

Git Cheat Sheet
https://github.com/ArslanBilal/Git-Cheat-Sheet

and more specific

git-flight-rules
https://github.com/k88hudson/git-flight-rules

First one is normal Git cheat sheet, but second one is more like FAQ and list of common tasks and workarounds.


5. CockroachDB.
A Scalable, Geo-Replicated, Transactional Datastore
https://github.com/cockroachdb/cockroach

So, it looks like NoSQL is not very popular buzzword now, NewSQL is much better!
But if talking seriously it really looks promising - it took better parts from both worlds:



ALPHA for now, we'll see how it'll going.

6. pycrastinate
TODO less, DO more. Keep your code clean without changing the way you code.
https://github.com/isaacbernat/pycrastinate
Pycrastinate is a language-agnostic tool that helps you keep your codebase (whether it is legacy or new) under control in a transparent way (i.e. without interfering with your coding). I.e. it checks for FIXMEs and TODOs and gently tring to force you to do something with that. :)
Check this talk or slides if interested.

7. Streisand.
https://github.com/jlund/streisand
"Streisand sets up a new server running L2TP/IPsec, OpenSSH, OpenVPN, Shadowsocks, Stunnel, and a Tor bridge. It also generates custom configuration instructions for all of these services. At the end of the run you are given an HTML file with instructions that can be shared with friends, family members, and fellow activists."
Not like very funny thing, especially for persons who struggle with internet censorshop somewhere but worth to checking then.

And time for fun!

8. IPFS
The Permanent Web http://ipfs.io
https://github.com/jbenet/ipfs
According to website:
"IPFS is a global, versioned, peer-to-peer filesystem. It combines good ideas from Git, BitTorrent, Kademlia, SFS, and the Web. It is like a single bittorrent swarm, exchanging git objects. IPFS provides an interface as simple as the HTTP web, but with permanence built in. You can also mount the world at /ipfs."
Very experimental thing for now, but maybe it's the source of new Internet, who knows?

Iosnoop is a great tool, and Gregg has more!

Hello !
So, did you already missing for Github digest? OK, OK, it's in progress, gimme more time please.
But now I want to talk about new exciting tool, again.
If you consider yourself as experienced Linux sysadmin and still do not reading Brendan Gregg's blog - you are making big mistake! Please follow it through RSS or follow Gregg's twitter @brendangregg right now - I will wait. :)
Before Gregg was working in Joyent and most of his posts was connected with DTrace, but now he moved to Netflix, and most of his work now connected with Linux.
So, Gregg was ported own iosnoop tool for Linux now. What is iosnoop? It's like a tcpdump, but for disk activity. It's hard to believe, but despite of its practical top of 10000 IOPS - really iosnoop is a bash script (more precisely - combination of bash with optimized awk) and it works out of the box on most modern kernels (even debug symbols not required)!
You can say - 'Ok, it's cool new small nifty tool, which provides some Kbs of text output, meh'. Not really impressive, right? But Gregg also provide cool thing to visualize that (or similar) data - heatmaps - and also provide tooling for that. Check that great post for making latency heatmap from iosnoop output.
So, let's try that stuff on something practical. I'm suspecting that newly installed KVM server is not behaving OK, I/O seems too slow. How we can detect that? Ok, let's reboot some VM, run iosnoop for 180 sec and compare heatmaps of reboot process.
Let's do that first on some normal server first:

I cap output at 100ms, just for simplicity, but it's not critical - real value took about 150 ms, because of queueing. Server was rebooted near 80 sec timepoint, and process was finished near 120 sec.
But compare it with target server:

It looks definitely not OK. Results are much more scattered, and queueing effect looks really terrible - top delay reach almost 6000 ms! Why? It's another question, I'll try to find that tomorrow. :)
One more time - please check Gregg's blog and his github repo - you can find much more cool things there!

Ops Github Digest #7 (July 21st 2014)

Hello Everybody!

Summer is going on, it's too hot for thinking much about work and stuff, everybody is want to go to vacation, right?
Right, but for everyone else trapped in stifling offices - next issue of Ops Github Digest!

1. Anode - Utility for analyzing graphite metrics. 

Experimental package.

https://github.com/mattrco/anode.exp 

Anode is a tool for experimenting with different analysis algorithms on metrics and other time series.
You have your data from Graphite, but don't know what to do with it? Try to experiment with that tool to find out:



WARNING! DANGER AHEAD! 

Some math required. :)

Please check that link for explanation how it works.


2. More awesomeness!
We have another list of awesome lists - https://github.com/sindresorhus/awesome, awesome algorithms - https://github.com/tayllan/awesome-algorithms, awesome machine learning - https://github.com/josephmisiti/awesome-machine-learning, and awesome bigdata - https://github.com/onurakpolat/awesome-bigdata
But my favorite - list of awesome T-Shirt sites - https://github.com/elder-cb/teesites !

3. Log file navigator http://lnav.org
https://github.com/tstack/lnav
Nice log reading / colorizing tool, especially for own server.


4. Another PuppetDB frontend:
Spotify's Puppetexplorer https://github.com/spotify/puppetexplorer
Has own query language and able to combine data from different PuppetDBs into one.

5. Facette
Time series data visualization and graphing software http://facette.io/
https://github.com/facette/facette
Facette is a web application to display time series data from various sources — such as collectd or Graphite — on graphs, designed to be easy to setup and to use.




6. storm is a command line tool to manage your ssh connections. 

https://github.com/emre/storm/

7. maestro-ng - Orchestration of Docker-based, multi-host environmentshttps://github.com/signalfuse/maestro-ng
Quite self-explanatory, please check if you use Docker.

8. miniLock - miniLock is a small, portable file encryption software.
https://github.com/kaepora/miniLock
New generation of modern file encryption software, which tries to proof that JS-crypto suck less nowadays.

9. And now - for you, my fellow esoteric programming language freaks!




Escher: A language for programming in metaphors 

https://github.com/gocircuit/escher 

Truly mind blowing if you have enough patience to wrap your mind around it. :)

Sporadic Ops Github Digest #6 (July 14th 2014)

Hi All!
I'm (again, still fighting with lazyness - and overcoming it) presenting our next digest:

1. cv - Linux tool to show progress for cp, rm, dd, ...
https://github.com/Xfennec/cv
One image is better than thousand words:

2. powerline-shell - A beautiful and useful prompt for your shell
https://github.com/milkbikis/powerline-shell
Yes, it is nice shell from picture above:


3. awesome-perl - A curated list of awesome Perl frameworks and libraries.
https://github.com/mackee/awesome-perl
At last! Perl!!!!11111

4. awesone-awesome-awesome - A a curated list of curated lists of awesome lists.
https://github.com/t3chnoboy/awesome-awesome-awesome
Ha-ha-ha-ha! Made my day!
And of course, there's awesome-awesome-awesome-awesome list too - for now, containing only list above...

5. More stuff for Python learning:

https://github.com/rasbt/python_reference - Useful functions, tutorials, and other Python-related things

https://github.com/s16h/py-must-watch - Must-watch videos about Python

6. The automated Pokémon speedrunning machine
https://github.com/kylecoburn/PokeBot
An automated computer program that speedruns Pokémon, which you can run on own computer. I'm not big fan of Pokemon and speedruns, but fan of old consoles. :) Also nice piece of good Lua code with AI.

Sporadic Ops Github Digest #5 (July 10th 2014)

"Did you missed me?" :)

Was quite busy. So, let's continue our fascinating journey to amazing world of interesting and/or useful (to the Ops engineers) Github projects!

1. A shell console with GUI features http://xiki.org
https://github.com/trogdoro/xiki
Did you see that awesomeness? If not - go to http://xiki.org and give it a try. It's definitely some fresh air for old and stale world of 70's unix shells.

Did you start own "awesome" list of something? Not yet?
Now we have bunch of them - Scala, Go, but my favorite is :

2. Awesome awesomeness - A curated list of awesome awesomeness.
https://github.com/bayandin/awesome-awesomeness

3. A Python library for automating interaction with websites.
https://github.com/hickford/MechanicalSoup
Same thing as Perl's WWW-Mechanize and Ruby's Mechanize - but compatible with python 2.x and 3.x and using Requests and BeautifulSoup as engine.

4. Snabb Switch: Fast open source packet processing
https://github.com/SnabbCo/snabbswitch
Very interesting project of open-source Lua-powered bare metal ethernet switch/router/firewall. Check that article for start - Snabb Switch - Skip The OS And Get 40 Million Requests Per Second In Lua

5. The next-generation email platform 

https://www.inboxapp.com/
https://github.com/inboxapp/inbox
Also very interesting project - Mail retrieval engine + API - you can collect email from various sources and retrieve it in your app/service - python-based and ready to run on your laptop or server.

6. 500 Lines or Less
https://github.com/aosabook/500lines
Not very interesting for now, but put it to your bookmarks or watch it on Github. New "The Architecture of Open Source Applications"-like book will be there soon.

7. Easily use tmux to collaborate with distant friends!
https://github.com/non/pairing
Small tool for pair programming over sharing your terminal with tmux.

8. Flappybird clone for Vim
https://github.com/mattn/flappyvird-vim
Just for fun! Awesome stuff! :)

Sporadic Gihub Ops Digest #4 (July 5th 2014)

I'm continuing to collect interesting Github repos (for Ops people but other geeks too):

1. Awesome Ruby.
A collection of awesome Ruby libraries, tools, frameworks and software https://github.com/markets/awesome-ruby
We have a Amazing PHP, Python, Ruby and sysadmin repos. But where's Perl ?

2. Vex.
Run a command in the named virtualenv. https://github.com/sashahart/vex

3. Rainbow Stream.
Terminal-based Twitter Client. https://github.com/DTVD/rainbowstream

4. statsrelay
A consistent-hashing relay for statsd metrics https://github.com/uber/statsrelay
5. EulerPy
Python-based Project Euler command line tool.
http://iKevinY.github.io/EulerPy
https://github.com/iKevinY/EulerPy

6. ReplayLastGoal
Automatically create an animated gif with the latest goal at the world cup
https://github.com/xdamman/ReplayLastGoal
Just for fun!