MySQL Replication: What’s New in MySQL 5.7 and Beyond

Very interesting presentation about current and upcoming features of replication in MySQL 5.7.x:

MySQL Replication: What’s New in MySQL 5.7 and Beyond from Andrew Morgan

Semi-irregular Sysadmin Ninja's Github Digest (Vol. 15)

Hi All,

Let me introduce you Volume 15 of Sysadmin Ninja's Github Digest! (as usual, in no particular order).
But why volume 15, if previous one was 13 and where's 11 and 14 then?
Well, because 11 and 14 are still in my drafts, and although I'm doubt that I'm publish them sometimes... well, let's be at least consistent - I'll try to continue digest publishing (will try to do it) on weekly basis.

1. KeyBox.
A web-based SSH console that executes commands on multiple shells. KeyBox allows you to manage keys, share terminal commands, and upload files to multiple systems simultaneously. http://sshkeybox.com
https://github.com/skavanagh/KeyBox

"A web-based ssh console to execute commands and manage multiple systems simultaneously. KeyBox allows you to share terminal commands and upload files to all your systems. Once the sessions have been opened you can select a single system or any combination to run your commands. Additional system administrators can be added and their terminal sessions and history can be audited. Also, KeyBox can manage and distribute public keys that have been setup and defined."

Can be useful for small distributed team of sysadmins across the globe.

2. FNordmetric
FnordMetric allows you collect and visualize timeseries data with SQL. http://fnordmetric.io
https://github.com/paulasmuth/fnordmetric

Hot stuff. Client-server application which "aims to be a StatsD+graphite competitor, it implements a wire compatible StatsD API". The main idea behind it that you write your query in SQL-like language, called "ChartSQL" and get your graph in SVG format. For it has quite big amount of graphing modes, but almost lacks of any functions for now. Written in C++, so, quite fast, but good luck with porting Graphite function library. Maybe it's a good idea to port Graphite as backend for it. Looks promising anyway.

And talking about Graphite -
3. graphite-stresser
A stress testing tool for Graphite
https://github.com/feangulo/graphite-stresser

Nothing unusual, but nice tool if you want to stress test your Graphite instance. Check author blog's entry for details.

4. pstop
pstop - a top-like program for MySQL
https://github.com/sjmudd/pstop

"pstop is a program which collects information from MySQL 5.6+'s performance_schema database and uses this information to display server load in real-time." For example, you can get IOPS for innodb file, or locks / operations / latencies per table. At least, you can start using performance schema in your MySQL 5.6 instance for something useful. Another useful P_S tool is "sys schema", you can read recent entry in Percona blog about it.

5. Consul-template
Generic template rendering and notifications with Consul
https://github.com/hashicorp/consul-template

Quite recent addition to service discovery tool Consul.io - now you can use it for any service which didn't understand service discovery through DNS - you can format config file templates for that service and reload it when Consul will see configuration change. Agian, blog post is better than 1000 words.

6. VCLfiddle
VclFiddle is hosted at http://www.vclfiddle.net/
https://github.com/vclfiddle/vclfiddle
"VclFiddle is an online tool for experimenting with the Varnish Cache HTTP reverse-proxy in a sandboxed environment. The name comes from a combination of the Varnish Configuration Language (VCL) and another tool that inspired this project, JSFiddle."

I.e. you can edit your VCL config on-line, using web editor, and check how it caches your website.

7. Racher.io
Rancher is an open source project that provides infrastructure services designed specifically for Docker. http://www.rancher.io
https://github.com/rancherio/rancher

Quite ambitious project for creating AWS-like environment, but for Docker containers. 

8. Atlas
A high-performance and stable proxy for MySQL
https://github.com/Qihoo360/Atlas

Another MySQL proxy. Well... Personally I never saw any production running on some MySQL-proxy solution (even on MySQL Fabric), but some China company named Qihoo360 developed this solution and insists that it's running on their production infrastructure.

9. Bosun
An advanced, open-source monitoring and alerting system by Stack Exchange http://bosun.io
https://github.com/bosun-monitor/bosun

Another Graphite competitior - OpenTSDB-backed service with own system metric's collector scollector and graphing and alerting interface. Written in Go. Looks neat and scalable:

And speaking about Go -

10. Go-opstocat
Collection of Ops related patterns for Go apps at GitHub.
https://github.com/github/go-opstocat

and
11. Delve
Delve is a Go debugger, written in Go.
https://github.com/derekparker/delve

Going further.

12. bup
Very efficient backup system based on the git packfile format, providing fast incremental saves and global deduplication (among and within files, including virtual machine images).
https://bup.github.io/
https://github.com/bup/bup

Interesting new backup tool, quite green and fresh, but looks promising.

13. osquery
SQL powered operating system instrumentation, monitoring, and analytics.
http://osquery.io
https://github.com/facebook/osquery

Facebook quite recently open-sourced that tool. Idea is looks very promising - again present system state as SQL tables on which you can run queries, by interactive console or automatically, as daemon.

And fun section

15. C4
C in four functions
https://github.com/rswier/c4

C-compile in 500 lines of C. Reading of it's sources is quite fun. :)

16. Gravity
An orbital simulation game written in Elm
https://github.com/stephenbalaban/Gravity

You can play it here.

17. Convergence
Python/OpenCl Cellular Automata design & manipulation tool
https://github.com/InfiniteSearchSpace/PyCl-Convergence

Looks like fun, but can't run that on my Mac for some reason, so, no screenshots.

Semi-irregular Sysadmin Ninja's Github Digest (Vol. 12)

Let's continue with review of most interesting Github projects.
N.B. this is vol. 12 - from quite old draft, I hope it's still relevant :)
Sorry for small comments, mostly projects are speak for themselves.

1. changelog
"What's changed in the last twenty minutes?"
https://github.com/prezi/changelog
The basic idea is that you'll send any event that has even a remote chance of causing problems to simple web service with REST interface. Later, when something goes wrong, you can quickly check what's changed in the last minutes / hours.
It's interface looks like this:



Very nice idea, indeed, especially with some graphing solution w/anomaly detection.
Clients for bash/python/java/etc. already exists.


2. db-readings
Readings in Databases
https://github.com/rxin/db-readings
A list of papers essential to understanding databases and building new data systems. Very nice reading for all sysadmins and developers. Only one problem exists - where I can find enough time for that????? :)

3. bitcoinbook
Mastering Bitcoin - Unlocking digital currencies - Early Release Draft
https://github.com/aantonop/bitcoinbook
Another "must-read" book for everyone who interested in digital currencies - it's about Bitcoin internals and technical realization. Early release of book is also available by O'Reilly.

4. github-cheat-sheet
A collection of cool hidden and not so hidden features of Git and GitHub
https://github.com/tiimgreen/github-cheat-sheet
Good stuff, even for experienced Git-hubbers.

5. Awesome Chef
A curated list of amazingly awesome Chef resources
https://github.com/obazoud/awesome-chef/
Where's our "Awesome Puppet"?

Ah, there it is:
6. Awesome Puppet
A curated list of amazingly awesome puppet resources
https://github.com/olindata/awesome-puppet

7. mcrouter
Mcrouter is a memcached protocol router for scaling memcached deployments.
https://github.com/facebook/mcrouter
Another nice piece of stuff from Facebook Engineering team. Must have if you're (still) using memcached on more than couple of servers.

8. masscan
TCP port scanner, spews SYN packets asynchronously, scanning entire Internet in under 5 minutes
https://github.com/robertdavidgraham/masscan

9. powa
PostgreSQL Workload Analyzer http://dalibo.github.io/powa
https://github.com/dalibo/powa

10. elasticsearch-HQ
Monitoring and Management Web Application for ElasticSearch instances and clusters.
https://github.com/royrusso/elasticsearch-HQ
Also nice tool for all ElasticSearch users.

Semi-irregular SysadminNinja's Github Digest (vol. 13)

Hi All,

Third part of digest, most current one!
Two more to come.

1. gogs
Gogs(Go Git Service) is a painless self-hosted Git Service written in Go.
http://gogs.io
https://github.com/gogits/gogs

Another Github clone, but written in Go. Worth checking, if you need one.

2. pyston
An open-source Python implementation using JIT techniques.
https://tech.dropbox.com/2014/04/introducing-pyston-an-upcoming-jit-based-python-implementation/
https://github.com/dropbox/pyston

New Python implementation, blazingly fast, looks promising, recommended by Guido himself :).

3. profiling
An interactive Python profiler.
https://github.com/what-studio/profiling

Nice looking Python profiler, still under development but looks good:

4. blackbox
Safely store secrets in Git/Mercurial http://the-cloud-book.com
https://github.com/StackExchange/blackbox

Another solution for storing secrets in your repos, but now from StackExchange and @yesthattom himself. Fresh and professional looking.

5. sfr1-lite
Search Formula-1 - A distributed high performance massive data engine for enterprise/vertical search
https://github.com/izenecloud/sf1r-lite

Vertical search engines are quite rare in open-source landscape. It's a new one from some Chinese startup, check tech docs in English for details.

6. mannaggia
automatic saint calling for depressed Veteran unix Admins, in italian
https://github.com/LegolasTheElf/mannaggia

Hah, funny thing. "Automatic saint calling for depressed Veteran unix Admins, in italian
Developed in italian, can be easily adapted in other languages."
TIL what "mannaggia" means. :)

7. shellshoc_poc
Shellshock DHCP RCE Proof of Concept
https://github.com/mschwager/shellshock_pocProof of concept of shellshock attack on DHCP server. See https://www.trustedsec.com/september-2014/shellshock-dhcp-rce-proof-concept/ for details:

8. bash_shellshock 

Wrapper for /bin/bash that mitigates 'shellshock' 

https://github.com/dlitz/bash-shellshock 

If you do not trust current Shellshock fixes you can use own wrapper to secure your server(s).

So, welcome to new SysadminNinja's house!

Hi All!

My old domain was suddenly expired, so I decided to make some changes in my blog. So, now it's named "Sysadmin Ninja's Blog" and it's nocated on neat and fancy URL http://iamsysadmin.ninja ! (yes, I know, but I was little bit slowpok-ey and most of yummy domains in .ninja TLD were already taken).
And I made some blog redesign - I hope current theme is not very dark... Maybe I'll become "light side" ninja soon. :)
So, and it's time to continue our GitHub digest. Last one was more than month ago, but I was quite busy with my moving, sorry. But we have fancy name now - "Semi-irregular Sysadmin Ninja's Github Digest" !
I had a ton of interesting repos for (more than)  last month, so, I decided to split it for at three volumes, will publish them this weekend.

Semi-irregular Intresting Github Repos Ops Digest #10 (August 15)

Hi All,

New name for that digest come into my mind - Semi-irregular! It sounds good for me, let's stick with that.
I want to make some celebration with Issue #10, but first I found out that no really cool projects to show for a week - maybe because of summer/vacation time, I don't know. So, I was waiting for one more week, and now -

Semi-irregular Intresting Github repos (for) Ops Digest, issue 10:

First, fresh docker projects:

1. docker-cheat-sheet
Docker Cheat Sheet
https://github.com/wsargent/docker-cheat-sheet
Good cheat sheet for Docker, must read.

2. flocker
Easily manage Docker containers and their data https://clusterhq.com
https://github.com/ClusterHQ/flocker
Open-source Docker orcestration service. Also, not much to say, please try tutorial.

3. flynn
A next generation open source platform as a service (PaaS) https://flynn.io
https://github.com/flynn/flynn
Ah, that's more complicated thing, but looks very promising - latyered open-sourced PaaS platform, check it out.

4. github-trending
Tracking the most popular Github repos, updated daily
https://github.com/josephyzhou/github-trending
Self explanatory, tracking the most popular Github repos and updating list of top objective-c, javasript and go repos in markdown output daily. You can pick other languages and run own tracker if you want.

5. cloud-ssh
Cloud enhanced SSH client replacement with host auto-completion
http://leonsbox.com/cloud-ssh/
https://github.com/buger/cloud-ssh
Cloud enhanced SSH client replacement with host auto-completion, espicially nice for AWS hosts.

6. cpubars
Lightweight terminal-based multicore CPU usage monitor
https://github.com/aclements/cpubars
cpubars is a simple terminal-based tool for monitoring CPU load in real-time, especially tailored to monitoring large multicores over SSH.


7. Password-Repo
https://github.com/Tek-Security-Group/Password-Repo
Quite good collection of password lists, which help to your services be more secure, I mean if you will deny using that obvious passwords for users.

8. mooltipass
Github repository dedicated to the mooltipass project
https://github.com/limpkin/mooltipass
Very nice project, indeed, like it very much! Aruino-based hardware password storage, powered by smartcard - wow! Fully open-sourced. Prototype is looking like that:


Ceators selling it for 80$, but real paranoid needs to built this by own, of course.
More and more sysadmin tools starts to be written in Go language. This is collection of Go stuff for ops engineers, check it out

9. golang-devops-stuff
https://github.com/mindreframer/golang-devops-stuff

And useful port of psutil library for Go:
10. gopsutil
psutil for golanghttps://github.com/shirou/gopsutil

Also some vim plugins was detected by by Github radar this week:

11. vim-plug
Minimalist Vim Plugin Manager https://github.com/junegunn/vim-plug
New Vim plugin manager, like it:



12. vim-grammarous
A powerful grammar checker for Vim using LanguageTool.
https://github.com/rhysd/vim-grammarous
"vim-grammarous is a powerful grammar checker for Vim. Simply do :GrammarousCheck to see the powerful checking"

13. vim-go
Go development plugin for Vim https://github.com/fatih/vim-go
"Full featured Go (golang) support for Vim. It comes with pre-defined sensible settings (like auto gofmt on save), has autocomplete, snippet support, improved syntax highlighting, go toolchain commands, etc... It's highly customizable and has settings for disabling/enabling features easily."Nuff said.

Fun section:

14. carp
"interesting" VM in C. Let's see how this goes. 

https://github.com/tekknolagi/carp
Sample realization ov virtual machine in C. Check if you want to how to find how VM works.

15. sunfish
Sunfish: a Python Chess Engine in 111 lines of code
https://chessprogramming.wikispaces.com/Sunfish
https://github.com/thomasahle/sunfish

"Sunfish is a simple, but strong chess engine, written in Python, mostly for teaching purposes. Without tables and its simple interface, it takes up just 111 lines of code!
The clarity of the Sunfish code provides a great platform for experimenting, be it with evaluation functions, search extensions or anything. Fork it today and see what you can do!"

16. anagramatron
twitter anagram hunter http://anagramatron.tumblr.com/
https://github.com/cmyr/anagramatron
"Anagramatron hunts for pairs of tweets that use the exact same set of letters.
This script connects to the twitter stream. When it receives a new tweet it runs it through some filters, ignoring tweets that contain things like links or @mentions, or that contain less then a minimum number of characters."
You can check results of it on http://anagramatron.tumblr.com/

Small disclaimer: I'm just big fan of Github and since we have so many sources of information around I pick Github as source of information for my digest of interesting open-source projects. And because I'm Ops/Sysadmin person - I choose projects which somewhat ops-related, but not necessary. I'm not GitHub employee and this digest is not connected with GitHub in any way.

Google hiring task (first 10-digit prime in e) in bash one-liner

Hi All,
I was on a bit of short vacation, and according to my vacation rules I'm trying to not to think about my work-related stuff, or any computer-related stuff overall. I was not successfull completely with that target thanks to my smartphone :) but tried to. But during my train voyage I was quite bored, so, I was thinking about task which my friend and fellow colleague master Victor mentioned - it was famous foto of billboard which was used by Google for hiring purposes back in 2004 -

And master Victor said - "assuming you know about the e constant, solve in a Bash one-liner to get a job @Google :)" - and I was thinking about that during my trip. And I was succeded - with some remarks, of course.
Disclaimer: IMHO solving this thask in Bash is NOT GOOD, because it violates first principle of engeneering - "choosing the right tool for the job" - and bash definitely is not good for that task.
So, first I tried to find is that task could be solved in "pure bash" and I found that it is not. You can check that thorough investigation to find out that calculating e with required precision is not posible in bash because of integer overflow.
So, let's "cheat" - i.e. use some tools with bash. Which tools are suitable and not count as cheating? I think bc is absolutely required for math operations, also other unix tools, like sed and coreutils are also OK.
Let's divide task into pieces:

1. Generate e with enough precision (1000 digits)?
I found that you can easily do it with bc:
root@ubuntu:~# echo "scale=100;e(1)" | bc -l 2.718281828459045235360287471352662497757247093699959574966967627724\ 0766303535475945713821785251664274

2. We need to go through digits of e and take each 10 digits to check, then move to 1 digit forward and repeat, and so forth. We can do it using temporary file and "cut -cX..Y" syntax, but I found that using bash variable and bash string manipulation (${string:X:10}) is much convenient.

3. We need to check is current number prime or not. We can do it on "pure bash" too, but I found that we can use infamous factor tool from coreutils to do that task:

root@ubuntu:~# factor 100 100: 2 2 5 5
root@ubuntu:~# factor 177 177: 3 59
root@ubuntu:~# factor 179 179: 179

As you can see, factor tool make number factorization for any not really big integer number - it's quite enough for our purposes.
So, let's glue everything together:

And output is:
7427466391
7413596629
6059563073
3490763233
2988075319
1573834187
7021540891
5408914993
6480016847
9920695517
1838606261
6062613313
3845830007
1692836819
4425056953
2505695369
5490598793
1782154249
8215424999
9229576351
9519366803
5193668033
1825288693
8294887933
1730123819
4039701983
4804295311
8194558153
9455815301
1332069811
6181881593
1881593041
5930416903
1934580727
3858942287
4841984443
1978623209
3140934317
3640546253
8887070167
7683964243
4563549061

Answer is 7427466391. And we can also fold it to one-liner of course:

Misson accomplished. :)