Semi-irregular Sysadmin Ninja's Github Digest (Vol. 16)

Ok, I'm still trying to finish with my old drafts and return to normal, weekly issues. Let's go!

1. devopsbookmarks.com
Website of devopsbookmarks.com http://www.devopsbookmarks.com
https://github.com/devopsbookmarks/devopsbookmarks.com

Cool new website which tries to collect all modern DevOps tools in one place (open-source and commercial too). And what is most exciting - everyone can participate through Github. :)

2. using-ngxlua-in-upyun
2014 Beijing OSC
https://github.com/timebug/using-ngxlua-in-upyun

It's also not standalone repo, but just code repo for this presentation from some Chinese conference. If you're interesting in Nginx + Lua / Openresty - check it out, quite good intro to subject. Don't afraid, it's in English - 

Using ngx_lua in UPYUN from timebug

3. sshrc
bring your .bashrc, .vimrc, etc. with you when you ssh
https://github.com/Russell91/sshrcIf you're making some remote admin tasks on "not your" servers from time to time you're usually quite frustrated that working environment there is not like perfectly crafted precious configs. You can fix that problem with that script, but beware of big Vim plugins - they're need to be transferred to your home dir on remote host during every login.

4. tmux-resurrect
Persists tmux environment across system restarts.
https://github.com/tmux-plugins/tmux-resurrect

Doing exactly that was promised - "saves all the little details from your tmux environment so it can be completely restored after a system restart (or when you feel like it). No configuration is required. You should feel like you never quit tmux."

5. Openstackgeek
StackGeek OpenStack Deploy
https://github.com/StackGeek/openstackgeek

"StackGeek provides these scripts and this guide to enable you to get a working installation of OpenStack Icehouse going in about 10 minutes."
Nuff said.

6. weave
The Docker Network
https://github.com/zettio/weave

Very interesting project, missing part of Docker, really. Networking is still weakest part of Docker IMO, and this project will help you with creation of virtual networks for your containers:

7. ZeroTierOne
Create flat virtual Ethernet networks of almost unlimited size. https://www.zerotier.com/
https://github.com/zerotier/ZeroTierOne

This project is similar with previous one, but main target of it is "normal" VMs and clouds and not containers. Looks quite mature and feature-full.

8. msr-cloud-tools
MSR Cloud Tools
https://github.com/brendangregg/msr-cloud-tools

Again, another tools from Brendan Gregg. For this time you can check is your cloud "hardware" support TurboBoost or read CPU temperature directly from CPU's MSRs (Model Specific Registers).

9. pcstat
Page Cache stat: get page cache stats for files on Linux
https://github.com/tobert/pcstat

Yes, that tool can show for given file how many memory pages lies in Linux' file cache. Nice to know for tuning DBs, e.g. Cassandra (that's why it was written for). Not like very new tool, you can use fadvise tool from https://code.google.com/p/linux-ftools/ too - but Go code looks prettir IMO.

Also TIL mincore(2) syscall on which both tolls were based on.

10. lsleases
list assigned ip from any device in your network
https://github.com/j-keck/lsleases
Simple DHCP sniffer - will list all IP/MACs from devices in your network. Could be useful.

11. inspeqtor
Monitor your application infrastructure!
https://github.com/mperham/inspeqtor
"Famous" inspector tool - modern rewrite of Monit on Go language with extended syntax and commercially available extension (because of which it was DMCAed by Monit developers first, but they're dismissed their claim after)

12. puppet-catalog-diff
Tool to diff Puppet catalogs
https://github.com/acidprime/puppet-catalog-diff
"A tool to compare two Puppet catalogs. While upgrading versions of Puppet or refactoring Puppet code you want to ensure that no unexpected changes will be made prior to doing the upgrade."

Very useful tool for upgrade Puppet between versions, indeed.

13. logsend
Logsend is high-performance tool for processing logs
https://github.com/ezotrank/logsend
"This like Logstash but more tiny and written by Golang. Supported outputs: influxdb, statsd and
MySQL". If you need some tool for log processing, but logstash looks somewhat bloated - check it out.

Semi-irregular Sysadmin Ninja's Github Digest (Vol. 12)

Let's continue with review of most interesting Github projects.
N.B. this is vol. 12 - from quite old draft, I hope it's still relevant :)
Sorry for small comments, mostly projects are speak for themselves.

1. changelog
"What's changed in the last twenty minutes?"
https://github.com/prezi/changelog
The basic idea is that you'll send any event that has even a remote chance of causing problems to simple web service with REST interface. Later, when something goes wrong, you can quickly check what's changed in the last minutes / hours.
It's interface looks like this:



Very nice idea, indeed, especially with some graphing solution w/anomaly detection.
Clients for bash/python/java/etc. already exists.


2. db-readings
Readings in Databases
https://github.com/rxin/db-readings
A list of papers essential to understanding databases and building new data systems. Very nice reading for all sysadmins and developers. Only one problem exists - where I can find enough time for that????? :)

3. bitcoinbook
Mastering Bitcoin - Unlocking digital currencies - Early Release Draft
https://github.com/aantonop/bitcoinbook
Another "must-read" book for everyone who interested in digital currencies - it's about Bitcoin internals and technical realization. Early release of book is also available by O'Reilly.

4. github-cheat-sheet
A collection of cool hidden and not so hidden features of Git and GitHub
https://github.com/tiimgreen/github-cheat-sheet
Good stuff, even for experienced Git-hubbers.

5. Awesome Chef
A curated list of amazingly awesome Chef resources
https://github.com/obazoud/awesome-chef/
Where's our "Awesome Puppet"?

Ah, there it is:
6. Awesome Puppet
A curated list of amazingly awesome puppet resources
https://github.com/olindata/awesome-puppet

7. mcrouter
Mcrouter is a memcached protocol router for scaling memcached deployments.
https://github.com/facebook/mcrouter
Another nice piece of stuff from Facebook Engineering team. Must have if you're (still) using memcached on more than couple of servers.

8. masscan
TCP port scanner, spews SYN packets asynchronously, scanning entire Internet in under 5 minutes
https://github.com/robertdavidgraham/masscan

9. powa
PostgreSQL Workload Analyzer http://dalibo.github.io/powa
https://github.com/dalibo/powa

10. elasticsearch-HQ
Monitoring and Management Web Application for ElasticSearch instances and clusters.
https://github.com/royrusso/elasticsearch-HQ
Also nice tool for all ElasticSearch users.

Random Ops Github Digest #9 (August 3rd 2014)

Hello, my fellow readers!
I decided not ot wait until tomorrow and make new digest right now - still not sure if this digest will be weekly or more random and sporadic. :)

So, let's start, again, my favourite Github projects (sometimes Ops-like) in no particular order.

1. cheat
cheat allows you to create and view interactive cheatsheets on the command-line.
https://github.com/chrisallenlane/cheat
Like man command, but you'll get small cheatsheet just with commands instead big man page. Still not sure is it good thing or not....

2. awesome-django
A curated list of awesome Django apps and projects.
https://github.com/rosarior/awesome-django
Another Awesome list, but for Django developers. Can be useful for Django Ops too.

For you, fellow Puppeteers:

3. r10k
Smarter Puppet deployment, powered by killer robots
https://github.com/adrienthebo/r10k
Relatively new tool for Puppet installations, useful if you using librarian and dynamic environments.

4. hiera-consul
Hiera backend plugin for Consul
https://github.com/lynxman/hiera-consul
If you have working Consul cluster you can get/set vualues there directly from Hiera.

Going to DevOps stuff:

5. terraform
Terraform is a tool for building, changing, and combining infrastructure safely and efficiently. http://www.terraform.io
https://github.com/hashicorp/terraform
"Infrastructure-as-code" tool - describe your infrastructure, deploy and control it on different providers. It's not Puppet replacement though - http://www.terraform.io/intro/vs/chef-puppet.html - it's just more high level tool.

6. infratester
Infrastructure Behavior Testing Framework http://infrataster.net
https://github.com/ryotarai/infrataster
Another "Infrastructure-as-code" tool - it tests infrastructure's behaviour from outside of servers:



7. devstep
Development environment builder powered by Docker and buildpacks
https://github.com/fgrehm/devstep
A dead simple, no frills development environment builder that is based around a simple goal:
"I want to git clone and run a single command to hack on any software project."
For more information please check http://fgrehm.viewdocs.io/devstep

8. libcloud-vagrant
Apache Libcloud compute provider for local Vagrant boxes
https://github.com/carletes/libcloud-vagrant
"libcloud-vagrant is a compute provider for Apache Libcloud which uses Vagrant to create VirtualBox nodes. With libcloud-vagrant installed, you could prototype a small cluster on your laptop, for instance, and then deploy it later on to Amazon, Rackspace, or any of the other clouds supported by Libcloud."

Rest of things:

9. seL4
seL4 microkernel
https://github.com/seL4/seL4
Freshly opensourced L4-based microkernel, which has formal verification proof. Not really practical thing, but interesting - check its FAQ if curious.

10. snapzend
zfs send/receive backup system http://www.znapzend.org
https://github.com/oetiker/znapzend/
Cool new backup system specifically for ZFS!

11. pghero
Database insights made easy

https://github.com/ankane/pghero
Personally I didn't work with more-less loaded instance of PostgreSQL for quite long time. But if you do - please check it out - it's nice web dashboard for pg:


Today's fun section is quite big:

12. cool-old-term
A good looking terminal emulator which mimics the old cathode display
https://github.com/Swordifish90/cool-old-term 
Looks cool - but not sure is it OK for work:




13. ld-preload-sounds

Generates raw WAV output by hooking malloc() and read().
https://github.com/gordol/ld_preload-sounds
You can now hear how your compilation sounds like!

14. conway
A real-time, persistent, multiplayer version of Conway's Game of Life
https://github.com/drewblaisdell/conway
You can roll out your own server and play with friends or play with strangers right now on http://lifecompetes.com


Small disclaimer: I'm just big fan of Github and since we have so many sources of information around I pick Github as source of information for my digest of interesting open-source projects. And because I'm Ops/Sysadmin person - I choose projects which somewhat ops-related, but not necessary. I'm not GitHub employee and this digest is not connected with GitHub in any way.

Ops Github Digest #7 (July 21st 2014)

Hello Everybody!

Summer is going on, it's too hot for thinking much about work and stuff, everybody is want to go to vacation, right?
Right, but for everyone else trapped in stifling offices - next issue of Ops Github Digest!

1. Anode - Utility for analyzing graphite metrics. 

Experimental package.

https://github.com/mattrco/anode.exp 

Anode is a tool for experimenting with different analysis algorithms on metrics and other time series.
You have your data from Graphite, but don't know what to do with it? Try to experiment with that tool to find out:



WARNING! DANGER AHEAD! 

Some math required. :)

Please check that link for explanation how it works.


2. More awesomeness!
We have another list of awesome lists - https://github.com/sindresorhus/awesome, awesome algorithms - https://github.com/tayllan/awesome-algorithms, awesome machine learning - https://github.com/josephmisiti/awesome-machine-learning, and awesome bigdata - https://github.com/onurakpolat/awesome-bigdata
But my favorite - list of awesome T-Shirt sites - https://github.com/elder-cb/teesites !

3. Log file navigator http://lnav.org
https://github.com/tstack/lnav
Nice log reading / colorizing tool, especially for own server.


4. Another PuppetDB frontend:
Spotify's Puppetexplorer https://github.com/spotify/puppetexplorer
Has own query language and able to combine data from different PuppetDBs into one.

5. Facette
Time series data visualization and graphing software http://facette.io/
https://github.com/facette/facette
Facette is a web application to display time series data from various sources — such as collectd or Graphite — on graphs, designed to be easy to setup and to use.




6. storm is a command line tool to manage your ssh connections. 

https://github.com/emre/storm/

7. maestro-ng - Orchestration of Docker-based, multi-host environmentshttps://github.com/signalfuse/maestro-ng
Quite self-explanatory, please check if you use Docker.

8. miniLock - miniLock is a small, portable file encryption software.
https://github.com/kaepora/miniLock
New generation of modern file encryption software, which tries to proof that JS-crypto suck less nowadays.

9. And now - for you, my fellow esoteric programming language freaks!




Escher: A language for programming in metaphors 

https://github.com/gocircuit/escher 

Truly mind blowing if you have enough patience to wrap your mind around it. :)