Semi-irregular Sysadmin Ninja's Github Digest (Vol. 21)

Hello, fellow readers!
Issue 21 of "Semi-irregular Sysadmin Ninja's Github Digest" is here. The last issue was very dry, will add more of my thoughts and funny pictures. :)
Let's go!


teeproxy
"A reverse HTTP proxy that duplicates requests."
"You may have production servers running, but you need to upgrade to a new system. You want to run A/B test on both old and new systems to confirm the new system can handle the production load, and want to see whether the new system can run in shadow mode continuously without any issue."
https://github.com/chrislusf/teeproxy

kicksat
"A tiny open source spacecraft project. http://kicksat.io"

WOW. Just W-O-W. Your eyes are not lying, it's open-source spacecraft. "Our goal is to dramatically lower the cost of spaceflight, making it easy enough and affordable enough for anyone to explore space. We can do this by shrinking the size and mass of the spacecraft, allowing many to be launched together."

I hope guys will be succeeded and we'll see KickSat launch soon!
https://github.com/kicksat

web2web
"P2P web powered by torrents and blockchain."
Rejoice, my paranoid brothers and sisters! New Internet is here! Wear our foil hats on!
It's a combination of webtorrent and blockhain to make not-seizable internet!
"When you open index.html in the browser (live demo), here's what happens:
Bitcoin address 1DhDyqB4xgDWjZzfbYGeutqdqBhSF7tGt4 is searched for the latest outgoing transaction containing OP_RETURN script. Inside the script there is a torrent infohash of webpage.html. webpage.html is downloaded from torrent via webtorrent and displayed."
https://github.com/elendirx/web2web

ironssh
"IronSSH - End-to-end secure file transfer"

"While sftp and scp use ssh to keep files secure while they are being transferred over the network, once those files hit the remote server, they are no longer protected. The ironsftp executable provides additional security. When you put a file on the server using ironsftp, the file is encrypted before it is uploaded, and it stays that way on the server. When you get a file from the server, it is downloaded then decrypted. So the file remains secure until it is at the place you want to use it - on your local machine."
https://github.com/ironcorelabs/ironssh

quinedb
"QuineDB is a quine that is also a key/value store.
If your database can't print its own source code, can you really trust it?"Very interesting and funny project! It's simple K/V storage, written in bash4, but it's also a quine!
"When you run it, the (possibly modified) source code of quinedb is printed to STDOUT, and the results of the specific command run are printed to STDERR."
https://github.com/gfredericks/quinedb


flask_jsondash
"Build javascript chart dashboards without any front-end code. Uses any json endpoint. JSON config only. Ready to go."
Long time dream is fulfilled! Yes, dashboards w/o any front-end code. !https://github.com/christabor/flask_jsondash


lograil
"LogTrail is a plugin for Kibana to view, analyze, search and tail log events from multiple hosts in realtime with devops friendly interface inspired by Papertrail."
Like "tail -f", but for ELK!

https://github.com/sivasamyk/logtrail

python-remote-companies

"A list of companies that allow remote work and use Python."
Yep, that simple, but maybe useful.https://github.com/mariusavram91/python-remote-companies

Games on the Github 
"list of open source games and game-related projects that can be found on GitHub"
https://github.com/leereilly/games

cog
"Bringing the power of the command line to chat http://operable.io"
"Cog is an open chatops platform that gives you a secure, collaborative command line right in your chat window. It is designed to be secure, highly available, chat provider agnostic, and to be extensible using your favorite programming language."
https://github.com/operable/cog


borg
"BORG - A terminal based search engine for bash commands"

Searching bash-related questions on Stackoverlow not leaving your terminal!
https://github.com/crufter/borg

git-stacktrace
"Easily figure out which git commit caused a given stacktrace https://pypi.python.org/pypi/git-stacktrace"

A little bit naive tool which helps you to find out wich commit is responsible for specific stacktrace. Python and Java are supported.
https://github.com/pinterest/git-stacktrace

pyinfra
"⚡ Deploy stuff by diff-ing the state you want against the remote server"
Interesting deploy tool. Looks nice, but IMO it better uses real configuration management tool in this case, e.g. Salt or Ansible.
https://github.com/Fizzadar/pyinfra

And something more for ML fans:

deepregex
"Neural Generation of Regular Expressions from Natural Language with Minimal Domain Knowledge"
For now, it's purely academic project - creating regexes using natural language and machine learning.
But beware - regexes it's a start, maybe sometimes computers maybe be able to write own programs.
Why they will need people then? :)
https://github.com/nicholaslocascio/deep-regex

tensorflow_image_classifier
Machine Learning is simple! You can make own TF image classifier in 5 minutes!

https://github.com/llSourcell/tensorflow_image_classifier

Introducing collectd-iostat-python

Collectd-iostat-python is an iostat plugin for collectd that allows you to graph Linux iostat metrics in Graphite or other output formats that are supported by collectd.
This plugin (and mostly this README) is rewrite of kieran's collectd-iostat in Python and collectd-python instead of Ruby and collectd-exec

Why ?

Disk performance is quite crucial for most of modern server applications, especially databases. E.g. MySQL - check out this slides from Percona Live conference.
Although collectd provides disk statistics out of the box, graphing the metrics as shown by iostat was found to be more useful and graphic, because iostat reports usage of block devices, partitions, multipath devices and LVM volumes.
Also this plugin was rewritten in Python, because its a preferable language for siteops' tools on my current job, and choice of using collectd-python instead of collectd-exec was made for performance and stability reasons.

How ?

Collectd-iostat-python functions by calling iostat with some predefined intervals and push that data to collectd using collectd-python plugin.
Collectd can be then configured to write the collected data into many output formats that are supported by it's write plugins, such as graphite, which was the primary use case for this plugin.

Setup

Deploy the collectd python plugin into a suitable plugin directory for your collectd instance.
Configure collectd's python plugin to execute the iostat plugin using a stanza similar to the following:

Once functioning, the iostat data should then be visible via your various output plugins.

In the case of Graphite, collectd should be writing data to graphite in thehostname_domain_tld.collectd_iostat_python.DEVICE.column-name style namespaces. Symbols like '/','-' and '%' in metric names (but not in device names) automatically replacing by underscores (i.e. '_')
Please note that plugin will take only last line of iostat output, so big Count numbers also have no sense, but Count needs to be more than 1 to get actual and not historical data. And please make Interval * Count << Collectd.INTERVAL (20 seconds by default). I found e.g. Count=2 and Interval=2 works quite well for me.

Technical notes

For parsing iostat output I'm using jakamkon's python-iostat python module, but as internal part of script instead of separate module because of couple of fixes - using Kbytes instead of blocks, adding -N to iostat for LVM endpoint resolving, migration to subprocess module as replacement of deprecated popen3, objectification etc.

TODO

Maybe some data aggregation needed, e.g. we can use some max / avg aggregation of data across intervals instead of picking last line of iostat output.

Semi-irregular SysadminNinja's Github Digest (vol. 13)

Hi All,

Third part of digest, most current one!
Two more to come.

1. gogs
Gogs(Go Git Service) is a painless self-hosted Git Service written in Go.
http://gogs.io
https://github.com/gogits/gogs

Another Github clone, but written in Go. Worth checking, if you need one.

2. pyston
An open-source Python implementation using JIT techniques.
https://tech.dropbox.com/2014/04/introducing-pyston-an-upcoming-jit-based-python-implementation/
https://github.com/dropbox/pyston

New Python implementation, blazingly fast, looks promising, recommended by Guido himself :).

3. profiling
An interactive Python profiler.
https://github.com/what-studio/profiling

Nice looking Python profiler, still under development but looks good:

4. blackbox
Safely store secrets in Git/Mercurial http://the-cloud-book.com
https://github.com/StackExchange/blackbox

Another solution for storing secrets in your repos, but now from StackExchange and @yesthattom himself. Fresh and professional looking.

5. sfr1-lite
Search Formula-1 - A distributed high performance massive data engine for enterprise/vertical search
https://github.com/izenecloud/sf1r-lite

Vertical search engines are quite rare in open-source landscape. It's a new one from some Chinese startup, check tech docs in English for details.

6. mannaggia
automatic saint calling for depressed Veteran unix Admins, in italian
https://github.com/LegolasTheElf/mannaggia

Hah, funny thing. "Automatic saint calling for depressed Veteran unix Admins, in italian
Developed in italian, can be easily adapted in other languages."
TIL what "mannaggia" means. :)

7. shellshoc_poc
Shellshock DHCP RCE Proof of Concept
https://github.com/mschwager/shellshock_pocProof of concept of shellshock attack on DHCP server. See https://www.trustedsec.com/september-2014/shellshock-dhcp-rce-proof-concept/ for details:

8. bash_shellshock 

Wrapper for /bin/bash that mitigates 'shellshock' 

https://github.com/dlitz/bash-shellshock 

If you do not trust current Shellshock fixes you can use own wrapper to secure your server(s).

Weekly Ops Github Digest #8 (July 29th 2014)

So, it seems that our digest becoming quite regular, it's issue number 8 and number 7 was out exactly one week ago, same as issue 6 was two weeks ago, nice.

Let's try to keep it that way - new issue every Monday/Tuesday, for fresh start of new week!
... but please note that possible upcoming vacation can interfere with my intentions. :)

OK, let's go.

1. elvish
An experimental Unix shell
https://github.com/xiaq/elvish
Nice new experimental shell (written in Go), more traditional than Xiki but also with intresting new features, like nice autocompletion and navigation mode, built in daemon for syncing command history from different instances and new Clojure/Lisp-like built-in language. Worth a try, but only if you have Linux, it doesn't compile on Mac, so, if you have one and want to be a contributor - welcome.

2. perf-tools
Performance analysis tools based on Linux perf_events (aka perf) and ftrace.
https://github.com/brendangregg/perf-tools
Set of performance analyzing tools for Linux, using (relatively) new Ftrace interface, written by famous Brendan D. Gregg, author of Systems Performance book, DTrace book etc.

Will not talk much about that, in short - amazing stuff, check my previous post about it.

3. Next 3 repos connected with Docker, lets merge them together:

Fig.
Fast, isolated development environments using Docker http://www.fig.sh
https://github.com/orchardup/fig
Describe your environment in single file and run it with single command "fig up".
Looks cool? Then check out

Octohost.
Simple web focused Dockerfile based PaaS server.
http://www.octohost.io/
https://github.com/octohost/octohost
Similar idea, but realization is little bit different - and idea is more general and limited at the same time. Just add Dockerfile to your webapp and after that run "git push" to deploy your project on choosen environment (Vagrant/Amazon/Digital Ocean/Rackspace/own server) - but only limited (but quite poular) languages and platforms are supported.

Next thing -
Registrator.
Service registry bridge for Docker
https://github.com/progrium/registrator
will help you with automatic "register/deregisters services for containers based on published ports and metadata from the container environment". I.e. you can automatically add/remove services to LB, for example. External service registers/pairing/discovery are supported also. Nice thing if your building scalable application on Docker.


Going to more general developer's stuff.

4. Two Git help pages. More general

Git Cheat Sheet
https://github.com/ArslanBilal/Git-Cheat-Sheet

and more specific

git-flight-rules
https://github.com/k88hudson/git-flight-rules

First one is normal Git cheat sheet, but second one is more like FAQ and list of common tasks and workarounds.


5. CockroachDB.
A Scalable, Geo-Replicated, Transactional Datastore
https://github.com/cockroachdb/cockroach

So, it looks like NoSQL is not very popular buzzword now, NewSQL is much better!
But if talking seriously it really looks promising - it took better parts from both worlds:



ALPHA for now, we'll see how it'll going.

6. pycrastinate
TODO less, DO more. Keep your code clean without changing the way you code.
https://github.com/isaacbernat/pycrastinate
Pycrastinate is a language-agnostic tool that helps you keep your codebase (whether it is legacy or new) under control in a transparent way (i.e. without interfering with your coding). I.e. it checks for FIXMEs and TODOs and gently tring to force you to do something with that. :)
Check this talk or slides if interested.

7. Streisand.
https://github.com/jlund/streisand
"Streisand sets up a new server running L2TP/IPsec, OpenSSH, OpenVPN, Shadowsocks, Stunnel, and a Tor bridge. It also generates custom configuration instructions for all of these services. At the end of the run you are given an HTML file with instructions that can be shared with friends, family members, and fellow activists."
Not like very funny thing, especially for persons who struggle with internet censorshop somewhere but worth to checking then.

And time for fun!

8. IPFS
The Permanent Web http://ipfs.io
https://github.com/jbenet/ipfs
According to website:
"IPFS is a global, versioned, peer-to-peer filesystem. It combines good ideas from Git, BitTorrent, Kademlia, SFS, and the Web. It is like a single bittorrent swarm, exchanging git objects. IPFS provides an interface as simple as the HTTP web, but with permanence built in. You can also mount the world at /ipfs."
Very experimental thing for now, but maybe it's the source of new Internet, who knows?

Sporadic Ops Github Digest #6 (July 14th 2014)

Hi All!
I'm (again, still fighting with lazyness - and overcoming it) presenting our next digest:

1. cv - Linux tool to show progress for cp, rm, dd, ...
https://github.com/Xfennec/cv
One image is better than thousand words:

2. powerline-shell - A beautiful and useful prompt for your shell
https://github.com/milkbikis/powerline-shell
Yes, it is nice shell from picture above:


3. awesome-perl - A curated list of awesome Perl frameworks and libraries.
https://github.com/mackee/awesome-perl
At last! Perl!!!!11111

4. awesone-awesome-awesome - A a curated list of curated lists of awesome lists.
https://github.com/t3chnoboy/awesome-awesome-awesome
Ha-ha-ha-ha! Made my day!
And of course, there's awesome-awesome-awesome-awesome list too - for now, containing only list above...

5. More stuff for Python learning:

https://github.com/rasbt/python_reference - Useful functions, tutorials, and other Python-related things

https://github.com/s16h/py-must-watch - Must-watch videos about Python

6. The automated Pokémon speedrunning machine
https://github.com/kylecoburn/PokeBot
An automated computer program that speedruns Pokémon, which you can run on own computer. I'm not big fan of Pokemon and speedruns, but fan of old consoles. :) Also nice piece of good Lua code with AI.

Sporadic Ops Github Digest #5 (July 10th 2014)

"Did you missed me?" :)

Was quite busy. So, let's continue our fascinating journey to amazing world of interesting and/or useful (to the Ops engineers) Github projects!

1. A shell console with GUI features http://xiki.org
https://github.com/trogdoro/xiki
Did you see that awesomeness? If not - go to http://xiki.org and give it a try. It's definitely some fresh air for old and stale world of 70's unix shells.

Did you start own "awesome" list of something? Not yet?
Now we have bunch of them - Scala, Go, but my favorite is :

2. Awesome awesomeness - A curated list of awesome awesomeness.
https://github.com/bayandin/awesome-awesomeness

3. A Python library for automating interaction with websites.
https://github.com/hickford/MechanicalSoup
Same thing as Perl's WWW-Mechanize and Ruby's Mechanize - but compatible with python 2.x and 3.x and using Requests and BeautifulSoup as engine.

4. Snabb Switch: Fast open source packet processing
https://github.com/SnabbCo/snabbswitch
Very interesting project of open-source Lua-powered bare metal ethernet switch/router/firewall. Check that article for start - Snabb Switch - Skip The OS And Get 40 Million Requests Per Second In Lua

5. The next-generation email platform 

https://www.inboxapp.com/
https://github.com/inboxapp/inbox
Also very interesting project - Mail retrieval engine + API - you can collect email from various sources and retrieve it in your app/service - python-based and ready to run on your laptop or server.

6. 500 Lines or Less
https://github.com/aosabook/500lines
Not very interesting for now, but put it to your bookmarks or watch it on Github. New "The Architecture of Open Source Applications"-like book will be there soon.

7. Easily use tmux to collaborate with distant friends!
https://github.com/non/pairing
Small tool for pair programming over sharing your terminal with tmux.

8. Flappybird clone for Vim
https://github.com/mattn/flappyvird-vim
Just for fun! Awesome stuff! :)

Sporadic Gihub Ops Digest #4 (July 5th 2014)

I'm continuing to collect interesting Github repos (for Ops people but other geeks too):

1. Awesome Ruby.
A collection of awesome Ruby libraries, tools, frameworks and software https://github.com/markets/awesome-ruby
We have a Amazing PHP, Python, Ruby and sysadmin repos. But where's Perl ?

2. Vex.
Run a command in the named virtualenv. https://github.com/sashahart/vex

3. Rainbow Stream.
Terminal-based Twitter Client. https://github.com/DTVD/rainbowstream

4. statsrelay
A consistent-hashing relay for statsd metrics https://github.com/uber/statsrelay
5. EulerPy
Python-based Project Euler command line tool.
http://iKevinY.github.io/EulerPy
https://github.com/iKevinY/EulerPy

6. ReplayLastGoal
Automatically create an animated gif with the latest goal at the world cup
https://github.com/xdamman/ReplayLastGoal
Just for fun!

Sporadic Ops GitHub digest #2 (01 Jul 2014)

Second issue of mine Sporadic Ops GitHub digest.

1. A curated list of awesome Python frameworks, libraries and resources. 

Inspired by awesome-php
https://github.com/vinta/awesome-python
Tons of useful stuff for pythonistas.

2. A well-written BSD licensed C language TLS implementation
https://github.com/wbl/SolidTLS
Another fork/rewrite of OpenSSL 'bloatware', good to know.

3. Simplistic interactive filtering tool
https://github.com/peco/peco
Cool little console tool, check screencasts for details.

4. Wrk - Modern HTTP benchmarking tool
https://github.com/wg/wrk
Lua-powered multi- thread / core HTTP benchmarking tool.

Sporadic Ops GitHub digest #1 (30 Jun 2014)

For no specific reason I decided to put some interesting for Ops engineer (on my opinion) Github projects to periodical (I hope) but quite sporadic digest.

So, what we have for today, in no particular order:

1. UNIX command line client for Twitter written in Go 

https://github.com/ajensenwaud/gwitter 

Just toy for now, please contribute.

2. Terminal-based SoundCloud client. 

Without all these stupid CSS files. Runs on OSX and Linux https://github.com/grobie/soundcloud2000

3. An open-source graph database by Google 

https://github.com/google/cayley
Good competitor for Neo4J (maybe, someday, in future).

4. Securely and anonymously share a file of any size 

https://onionshare.org/
https://github.com/micahflee/onionshare 

Tricky mix of Tor and python, I like that!

5. A drop-in, low-overhead monitoring web dashboard for a linux machine https://github.com/afaqurk/linux-dash 

Simple PHP-based dashboard for one server.

6. Ajenti - the web admin panel everyone wants. http://ajenti.org
https://github.com/Eugeny/ajenti 

Neat web admin panel, looks cool and modern!

7. Offline version of Project Euler 

https://github.com/imsky/LocalEuler

8. A Python framework for creating interactive Twitter bots 

https://github.com/thricedotted/twitterbot

Perl vs Python

100% true

New skill test site !

There is a new website with skill tests - http://smarterer.com/
They're proud about their scoring system and looks like it works - I have Expert grade in Perl, and just beginner for Python. :)
You can create special page to measure d**k length proud - http://smarterer.com/deniszh (and you can cowardly hide "not-so-good-test-results" on that page :D)