Semi-irregular Sysadmin Ninja's Github Digest (Vol. 12)

Let's continue with review of most interesting Github projects.
N.B. this is vol. 12 - from quite old draft, I hope it's still relevant :)
Sorry for small comments, mostly projects are speak for themselves.

1. changelog
"What's changed in the last twenty minutes?"
https://github.com/prezi/changelog
The basic idea is that you'll send any event that has even a remote chance of causing problems to simple web service with REST interface. Later, when something goes wrong, you can quickly check what's changed in the last minutes / hours.
It's interface looks like this:



Very nice idea, indeed, especially with some graphing solution w/anomaly detection.
Clients for bash/python/java/etc. already exists.


2. db-readings
Readings in Databases
https://github.com/rxin/db-readings
A list of papers essential to understanding databases and building new data systems. Very nice reading for all sysadmins and developers. Only one problem exists - where I can find enough time for that????? :)

3. bitcoinbook
Mastering Bitcoin - Unlocking digital currencies - Early Release Draft
https://github.com/aantonop/bitcoinbook
Another "must-read" book for everyone who interested in digital currencies - it's about Bitcoin internals and technical realization. Early release of book is also available by O'Reilly.

4. github-cheat-sheet
A collection of cool hidden and not so hidden features of Git and GitHub
https://github.com/tiimgreen/github-cheat-sheet
Good stuff, even for experienced Git-hubbers.

5. Awesome Chef
A curated list of amazingly awesome Chef resources
https://github.com/obazoud/awesome-chef/
Where's our "Awesome Puppet"?

Ah, there it is:
6. Awesome Puppet
A curated list of amazingly awesome puppet resources
https://github.com/olindata/awesome-puppet

7. mcrouter
Mcrouter is a memcached protocol router for scaling memcached deployments.
https://github.com/facebook/mcrouter
Another nice piece of stuff from Facebook Engineering team. Must have if you're (still) using memcached on more than couple of servers.

8. masscan
TCP port scanner, spews SYN packets asynchronously, scanning entire Internet in under 5 minutes
https://github.com/robertdavidgraham/masscan

9. powa
PostgreSQL Workload Analyzer http://dalibo.github.io/powa
https://github.com/dalibo/powa

10. elasticsearch-HQ
Monitoring and Management Web Application for ElasticSearch instances and clusters.
https://github.com/royrusso/elasticsearch-HQ
Also nice tool for all ElasticSearch users.

Semi-irregular SysadminNinja's Github Digest (vol. 13)

Hi All,

Third part of digest, most current one!
Two more to come.

1. gogs
Gogs(Go Git Service) is a painless self-hosted Git Service written in Go.
http://gogs.io
https://github.com/gogits/gogs

Another Github clone, but written in Go. Worth checking, if you need one.

2. pyston
An open-source Python implementation using JIT techniques.
https://tech.dropbox.com/2014/04/introducing-pyston-an-upcoming-jit-based-python-implementation/
https://github.com/dropbox/pyston

New Python implementation, blazingly fast, looks promising, recommended by Guido himself :).

3. profiling
An interactive Python profiler.
https://github.com/what-studio/profiling

Nice looking Python profiler, still under development but looks good:

4. blackbox
Safely store secrets in Git/Mercurial http://the-cloud-book.com
https://github.com/StackExchange/blackbox

Another solution for storing secrets in your repos, but now from StackExchange and @yesthattom himself. Fresh and professional looking.

5. sfr1-lite
Search Formula-1 - A distributed high performance massive data engine for enterprise/vertical search
https://github.com/izenecloud/sf1r-lite

Vertical search engines are quite rare in open-source landscape. It's a new one from some Chinese startup, check tech docs in English for details.

6. mannaggia
automatic saint calling for depressed Veteran unix Admins, in italian
https://github.com/LegolasTheElf/mannaggia

Hah, funny thing. "Automatic saint calling for depressed Veteran unix Admins, in italian
Developed in italian, can be easily adapted in other languages."
TIL what "mannaggia" means. :)

7. shellshoc_poc
Shellshock DHCP RCE Proof of Concept
https://github.com/mschwager/shellshock_pocProof of concept of shellshock attack on DHCP server. See https://www.trustedsec.com/september-2014/shellshock-dhcp-rce-proof-concept/ for details:

8. bash_shellshock 

Wrapper for /bin/bash that mitigates 'shellshock' 

https://github.com/dlitz/bash-shellshock 

If you do not trust current Shellshock fixes you can use own wrapper to secure your server(s).

So, welcome to new SysadminNinja's house!

Hi All!

My old domain was suddenly expired, so I decided to make some changes in my blog. So, now it's named "Sysadmin Ninja's Blog" and it's nocated on neat and fancy URL http://iamsysadmin.ninja ! (yes, I know, but I was little bit slowpok-ey and most of yummy domains in .ninja TLD were already taken).
And I made some blog redesign - I hope current theme is not very dark... Maybe I'll become "light side" ninja soon. :)
So, and it's time to continue our GitHub digest. Last one was more than month ago, but I was quite busy with my moving, sorry. But we have fancy name now - "Semi-irregular Sysadmin Ninja's Github Digest" !
I had a ton of interesting repos for (more than)  last month, so, I decided to split it for at three volumes, will publish them this weekend.

Semi-irregular Intresting Github Repos Ops Digest #10 (August 15)

Hi All,

New name for that digest come into my mind - Semi-irregular! It sounds good for me, let's stick with that.
I want to make some celebration with Issue #10, but first I found out that no really cool projects to show for a week - maybe because of summer/vacation time, I don't know. So, I was waiting for one more week, and now -

Semi-irregular Intresting Github repos (for) Ops Digest, issue 10:

First, fresh docker projects:

1. docker-cheat-sheet
Docker Cheat Sheet
https://github.com/wsargent/docker-cheat-sheet
Good cheat sheet for Docker, must read.

2. flocker
Easily manage Docker containers and their data https://clusterhq.com
https://github.com/ClusterHQ/flocker
Open-source Docker orcestration service. Also, not much to say, please try tutorial.

3. flynn
A next generation open source platform as a service (PaaS) https://flynn.io
https://github.com/flynn/flynn
Ah, that's more complicated thing, but looks very promising - latyered open-sourced PaaS platform, check it out.

4. github-trending
Tracking the most popular Github repos, updated daily
https://github.com/josephyzhou/github-trending
Self explanatory, tracking the most popular Github repos and updating list of top objective-c, javasript and go repos in markdown output daily. You can pick other languages and run own tracker if you want.

5. cloud-ssh
Cloud enhanced SSH client replacement with host auto-completion
http://leonsbox.com/cloud-ssh/
https://github.com/buger/cloud-ssh
Cloud enhanced SSH client replacement with host auto-completion, espicially nice for AWS hosts.

6. cpubars
Lightweight terminal-based multicore CPU usage monitor
https://github.com/aclements/cpubars
cpubars is a simple terminal-based tool for monitoring CPU load in real-time, especially tailored to monitoring large multicores over SSH.


7. Password-Repo
https://github.com/Tek-Security-Group/Password-Repo
Quite good collection of password lists, which help to your services be more secure, I mean if you will deny using that obvious passwords for users.

8. mooltipass
Github repository dedicated to the mooltipass project
https://github.com/limpkin/mooltipass
Very nice project, indeed, like it very much! Aruino-based hardware password storage, powered by smartcard - wow! Fully open-sourced. Prototype is looking like that:


Ceators selling it for 80$, but real paranoid needs to built this by own, of course.
More and more sysadmin tools starts to be written in Go language. This is collection of Go stuff for ops engineers, check it out

9. golang-devops-stuff
https://github.com/mindreframer/golang-devops-stuff

And useful port of psutil library for Go:
10. gopsutil
psutil for golanghttps://github.com/shirou/gopsutil

Also some vim plugins was detected by by Github radar this week:

11. vim-plug
Minimalist Vim Plugin Manager https://github.com/junegunn/vim-plug
New Vim plugin manager, like it:



12. vim-grammarous
A powerful grammar checker for Vim using LanguageTool.
https://github.com/rhysd/vim-grammarous
"vim-grammarous is a powerful grammar checker for Vim. Simply do :GrammarousCheck to see the powerful checking"

13. vim-go
Go development plugin for Vim https://github.com/fatih/vim-go
"Full featured Go (golang) support for Vim. It comes with pre-defined sensible settings (like auto gofmt on save), has autocomplete, snippet support, improved syntax highlighting, go toolchain commands, etc... It's highly customizable and has settings for disabling/enabling features easily."Nuff said.

Fun section:

14. carp
"interesting" VM in C. Let's see how this goes. 

https://github.com/tekknolagi/carp
Sample realization ov virtual machine in C. Check if you want to how to find how VM works.

15. sunfish
Sunfish: a Python Chess Engine in 111 lines of code
https://chessprogramming.wikispaces.com/Sunfish
https://github.com/thomasahle/sunfish

"Sunfish is a simple, but strong chess engine, written in Python, mostly for teaching purposes. Without tables and its simple interface, it takes up just 111 lines of code!
The clarity of the Sunfish code provides a great platform for experimenting, be it with evaluation functions, search extensions or anything. Fork it today and see what you can do!"

16. anagramatron
twitter anagram hunter http://anagramatron.tumblr.com/
https://github.com/cmyr/anagramatron
"Anagramatron hunts for pairs of tweets that use the exact same set of letters.
This script connects to the twitter stream. When it receives a new tweet it runs it through some filters, ignoring tweets that contain things like links or @mentions, or that contain less then a minimum number of characters."
You can check results of it on http://anagramatron.tumblr.com/

Small disclaimer: I'm just big fan of Github and since we have so many sources of information around I pick Github as source of information for my digest of interesting open-source projects. And because I'm Ops/Sysadmin person - I choose projects which somewhat ops-related, but not necessary. I'm not GitHub employee and this digest is not connected with GitHub in any way.

Google hiring task (first 10-digit prime in e) in bash one-liner

Hi All,
I was on a bit of short vacation, and according to my vacation rules I'm trying to not to think about my work-related stuff, or any computer-related stuff overall. I was not successfull completely with that target thanks to my smartphone :) but tried to. But during my train voyage I was quite bored, so, I was thinking about task which my friend and fellow colleague master Victor mentioned - it was famous foto of billboard which was used by Google for hiring purposes back in 2004 -

And master Victor said - "assuming you know about the e constant, solve in a Bash one-liner to get a job @Google :)" - and I was thinking about that during my trip. And I was succeded - with some remarks, of course.
Disclaimer: IMHO solving this thask in Bash is NOT GOOD, because it violates first principle of engeneering - "choosing the right tool for the job" - and bash definitely is not good for that task.
So, first I tried to find is that task could be solved in "pure bash" and I found that it is not. You can check that thorough investigation to find out that calculating e with required precision is not posible in bash because of integer overflow.
So, let's "cheat" - i.e. use some tools with bash. Which tools are suitable and not count as cheating? I think bc is absolutely required for math operations, also other unix tools, like sed and coreutils are also OK.
Let's divide task into pieces:

1. Generate e with enough precision (1000 digits)?
I found that you can easily do it with bc:
root@ubuntu:~# echo "scale=100;e(1)" | bc -l 2.718281828459045235360287471352662497757247093699959574966967627724\ 0766303535475945713821785251664274

2. We need to go through digits of e and take each 10 digits to check, then move to 1 digit forward and repeat, and so forth. We can do it using temporary file and "cut -cX..Y" syntax, but I found that using bash variable and bash string manipulation (${string:X:10}) is much convenient.

3. We need to check is current number prime or not. We can do it on "pure bash" too, but I found that we can use infamous factor tool from coreutils to do that task:

root@ubuntu:~# factor 100 100: 2 2 5 5
root@ubuntu:~# factor 177 177: 3 59
root@ubuntu:~# factor 179 179: 179

As you can see, factor tool make number factorization for any not really big integer number - it's quite enough for our purposes.
So, let's glue everything together:

And output is:
7427466391
7413596629
6059563073
3490763233
2988075319
1573834187
7021540891
5408914993
6480016847
9920695517
1838606261
6062613313
3845830007
1692836819
4425056953
2505695369
5490598793
1782154249
8215424999
9229576351
9519366803
5193668033
1825288693
8294887933
1730123819
4039701983
4804295311
8194558153
9455815301
1332069811
6181881593
1881593041
5930416903
1934580727
3858942287
4841984443
1978623209
3140934317
3640546253
8887070167
7683964243
4563549061

Answer is 7427466391. And we can also fold it to one-liner of course:

Misson accomplished. :)

Random Ops Github Digest #9 (August 3rd 2014)

Hello, my fellow readers!
I decided not ot wait until tomorrow and make new digest right now - still not sure if this digest will be weekly or more random and sporadic. :)

So, let's start, again, my favourite Github projects (sometimes Ops-like) in no particular order.

1. cheat
cheat allows you to create and view interactive cheatsheets on the command-line.
https://github.com/chrisallenlane/cheat
Like man command, but you'll get small cheatsheet just with commands instead big man page. Still not sure is it good thing or not....

2. awesome-django
A curated list of awesome Django apps and projects.
https://github.com/rosarior/awesome-django
Another Awesome list, but for Django developers. Can be useful for Django Ops too.

For you, fellow Puppeteers:

3. r10k
Smarter Puppet deployment, powered by killer robots
https://github.com/adrienthebo/r10k
Relatively new tool for Puppet installations, useful if you using librarian and dynamic environments.

4. hiera-consul
Hiera backend plugin for Consul
https://github.com/lynxman/hiera-consul
If you have working Consul cluster you can get/set vualues there directly from Hiera.

Going to DevOps stuff:

5. terraform
Terraform is a tool for building, changing, and combining infrastructure safely and efficiently. http://www.terraform.io
https://github.com/hashicorp/terraform
"Infrastructure-as-code" tool - describe your infrastructure, deploy and control it on different providers. It's not Puppet replacement though - http://www.terraform.io/intro/vs/chef-puppet.html - it's just more high level tool.

6. infratester
Infrastructure Behavior Testing Framework http://infrataster.net
https://github.com/ryotarai/infrataster
Another "Infrastructure-as-code" tool - it tests infrastructure's behaviour from outside of servers:



7. devstep
Development environment builder powered by Docker and buildpacks
https://github.com/fgrehm/devstep
A dead simple, no frills development environment builder that is based around a simple goal:
"I want to git clone and run a single command to hack on any software project."
For more information please check http://fgrehm.viewdocs.io/devstep

8. libcloud-vagrant
Apache Libcloud compute provider for local Vagrant boxes
https://github.com/carletes/libcloud-vagrant
"libcloud-vagrant is a compute provider for Apache Libcloud which uses Vagrant to create VirtualBox nodes. With libcloud-vagrant installed, you could prototype a small cluster on your laptop, for instance, and then deploy it later on to Amazon, Rackspace, or any of the other clouds supported by Libcloud."

Rest of things:

9. seL4
seL4 microkernel
https://github.com/seL4/seL4
Freshly opensourced L4-based microkernel, which has formal verification proof. Not really practical thing, but interesting - check its FAQ if curious.

10. snapzend
zfs send/receive backup system http://www.znapzend.org
https://github.com/oetiker/znapzend/
Cool new backup system specifically for ZFS!

11. pghero
Database insights made easy

https://github.com/ankane/pghero
Personally I didn't work with more-less loaded instance of PostgreSQL for quite long time. But if you do - please check it out - it's nice web dashboard for pg:


Today's fun section is quite big:

12. cool-old-term
A good looking terminal emulator which mimics the old cathode display
https://github.com/Swordifish90/cool-old-term 
Looks cool - but not sure is it OK for work:




13. ld-preload-sounds

Generates raw WAV output by hooking malloc() and read().
https://github.com/gordol/ld_preload-sounds
You can now hear how your compilation sounds like!

14. conway
A real-time, persistent, multiplayer version of Conway's Game of Life
https://github.com/drewblaisdell/conway
You can roll out your own server and play with friends or play with strangers right now on http://lifecompetes.com


Small disclaimer: I'm just big fan of Github and since we have so many sources of information around I pick Github as source of information for my digest of interesting open-source projects. And because I'm Ops/Sysadmin person - I choose projects which somewhat ops-related, but not necessary. I'm not GitHub employee and this digest is not connected with GitHub in any way.

Weekly Ops Github Digest #8 (July 29th 2014)

So, it seems that our digest becoming quite regular, it's issue number 8 and number 7 was out exactly one week ago, same as issue 6 was two weeks ago, nice.

Let's try to keep it that way - new issue every Monday/Tuesday, for fresh start of new week!
... but please note that possible upcoming vacation can interfere with my intentions. :)

OK, let's go.

1. elvish
An experimental Unix shell
https://github.com/xiaq/elvish
Nice new experimental shell (written in Go), more traditional than Xiki but also with intresting new features, like nice autocompletion and navigation mode, built in daemon for syncing command history from different instances and new Clojure/Lisp-like built-in language. Worth a try, but only if you have Linux, it doesn't compile on Mac, so, if you have one and want to be a contributor - welcome.

2. perf-tools
Performance analysis tools based on Linux perf_events (aka perf) and ftrace.
https://github.com/brendangregg/perf-tools
Set of performance analyzing tools for Linux, using (relatively) new Ftrace interface, written by famous Brendan D. Gregg, author of Systems Performance book, DTrace book etc.

Will not talk much about that, in short - amazing stuff, check my previous post about it.

3. Next 3 repos connected with Docker, lets merge them together:

Fig.
Fast, isolated development environments using Docker http://www.fig.sh
https://github.com/orchardup/fig
Describe your environment in single file and run it with single command "fig up".
Looks cool? Then check out

Octohost.
Simple web focused Dockerfile based PaaS server.
http://www.octohost.io/
https://github.com/octohost/octohost
Similar idea, but realization is little bit different - and idea is more general and limited at the same time. Just add Dockerfile to your webapp and after that run "git push" to deploy your project on choosen environment (Vagrant/Amazon/Digital Ocean/Rackspace/own server) - but only limited (but quite poular) languages and platforms are supported.

Next thing -
Registrator.
Service registry bridge for Docker
https://github.com/progrium/registrator
will help you with automatic "register/deregisters services for containers based on published ports and metadata from the container environment". I.e. you can automatically add/remove services to LB, for example. External service registers/pairing/discovery are supported also. Nice thing if your building scalable application on Docker.


Going to more general developer's stuff.

4. Two Git help pages. More general

Git Cheat Sheet
https://github.com/ArslanBilal/Git-Cheat-Sheet

and more specific

git-flight-rules
https://github.com/k88hudson/git-flight-rules

First one is normal Git cheat sheet, but second one is more like FAQ and list of common tasks and workarounds.


5. CockroachDB.
A Scalable, Geo-Replicated, Transactional Datastore
https://github.com/cockroachdb/cockroach

So, it looks like NoSQL is not very popular buzzword now, NewSQL is much better!
But if talking seriously it really looks promising - it took better parts from both worlds:



ALPHA for now, we'll see how it'll going.

6. pycrastinate
TODO less, DO more. Keep your code clean without changing the way you code.
https://github.com/isaacbernat/pycrastinate
Pycrastinate is a language-agnostic tool that helps you keep your codebase (whether it is legacy or new) under control in a transparent way (i.e. without interfering with your coding). I.e. it checks for FIXMEs and TODOs and gently tring to force you to do something with that. :)
Check this talk or slides if interested.

7. Streisand.
https://github.com/jlund/streisand
"Streisand sets up a new server running L2TP/IPsec, OpenSSH, OpenVPN, Shadowsocks, Stunnel, and a Tor bridge. It also generates custom configuration instructions for all of these services. At the end of the run you are given an HTML file with instructions that can be shared with friends, family members, and fellow activists."
Not like very funny thing, especially for persons who struggle with internet censorshop somewhere but worth to checking then.

And time for fun!

8. IPFS
The Permanent Web http://ipfs.io
https://github.com/jbenet/ipfs
According to website:
"IPFS is a global, versioned, peer-to-peer filesystem. It combines good ideas from Git, BitTorrent, Kademlia, SFS, and the Web. It is like a single bittorrent swarm, exchanging git objects. IPFS provides an interface as simple as the HTTP web, but with permanence built in. You can also mount the world at /ipfs."
Very experimental thing for now, but maybe it's the source of new Internet, who knows?