Показаны сообщения с ярлыком postgresql. Показать все сообщения
Показаны сообщения с ярлыком postgresql. Показать все сообщения

воскресенье, 26 марта 2017 г.

TSDB on PostgreSQL? But why?

This week I stumbled across two new time-series databases (TSDB) - which is good, but both were based on PostgreSQL, which is... confusing, to be honest.

First was named Tgres, it was created by Grisha Trubetskoy and already reach v0.10.0, after 18 month in development, which is quite impressive (no joke). It's written in Go, partially Graphite-compatible and even outperforming "ye olde Graphite" on a single ec2 i2.2xlarge instance (not go-carbon, just normal python Graphite, which is quite not fair IMO). It's also based on rollup archives idea as Whisper and also more effective as storage - around 8 bytes per point (whisper has 12 bytes).

I'm totally fine with people who want to develop something, and I'm not gonna say that Grisha does not understand what he's doing - he's experienced developer and Tgres looks very impressive. But to be honest all rational behind Tgres is really puzzling me. You can read it on the link above (you can go to part named "Avoid Solving the Storage Problem", but it's worth to read all article).
Grisha says: "Someone once said that “anything is possible when you don’t know what you’re talking about”, and nowhere is it more evident than in data storage. File systems and relational databases trace their origin back to the late 1960s and over half a century later I doubt that any field experts would say “the storage problem is solved”. And so it seems almost foolish to suppose that by throwing together a key-value store and a consensus algorithm or some such it is possible to come up with something better? Instead of re-inventing storage, why not focus on how to structure the data in a way that is compatible with a storage implementation that we know works and scales reliably?"
With all respect, but I think that's a wrong direction. Yes, filesystems and databases are in development from the 1960s - and what result do we have? The storage problem is not solved, indeed, but saying "OK, screw it, let's create something on top of weak foundation and hope that it'll fine" is also wrong.
I think that storage engine is the best part of any database and it creates and limits any DB - relational, column or time-series - doesn't matter. Whisper is a good example. It has its own weak points (e.g. no subsecond resolution, IO intensity, 12 bytes per point, only local storage) - and its good points (quite good speed, built-in rollups). But most of Graphite users know its limitations very well - and these limitations limiting Graphite usage from one side - but on the other hand, they created all this new generation of TSDBs / monitoring solutions which are flourishing last time.
And in the same way Tgres inherits all scalability flaws as PostgreSQL (as any relational database) has e.g. good vertical scalability, but quite weak horizontal one. Yes, the author mentions clustering for Tgres, but it's the same approach as we saw already in Whisper - it's external clustering, not built-in in storage.

Another PostgreSQL-based database, named TimescaleDB looks bit better - it still based on Postgres although it uses an own storage engine with built-in clustering and sharding. You can check their paper, it's quite interesting. Now it looks like early InfluxDB, but authors are saying that their approach is better because you can use all real SQL power across all your timeseries.
Let's see. TimescaleDB is quite young, less than 6 months in development, maybe we'll get something useful out there. They have a good and stable foundation, let's see how it will fit in TSDB world.

I still have a strong opinion that in database's world storage engine is a king, and horizontal scalability is a must for any modern data software.




вторник, 9 декабря 2014 г.

Semi-irregular Sysadmin Ninja's Github Digest (Vol. 17)

1. reptyr
Reparent a running program to a new terminal
https://github.com/nelhage/reptyr

Quite old tool made by @nelhage, it seems he is actively developing it again. It is really changes terminal for process.  "'reptyr PID' will grab the process with id PID and attach it to your current terminal. After attaching, the process will take input from and write output to the new terminal, including ^C and ^Z."
It is also quite interesting to know how it works - check this blog post if you are curious.

2. dockerana
Docker + Graphite + Graphana = Dockerana
https://github.com/dockerana/dockerana
It's exactly what it looks - Graphite + Graphana  packed in Docker container. Quite convenient.

3. seagull
Friendly Web UI to monitor docker daemon http://96.126.127.93:10086
https://github.com/tobegit3hub/seagull
Seagull is the best friend of docker which provides Web UI to monitor docker daemon. Demo site is down but screenshots looks nice. It seems that demo is working now.

4. Algorithms
Data Structures and Algorithms in Python
https://github.com/prakhar1989/Algorithms
Not very exciting stuff, but might be useful. Just as it says, it is collection of data structures and algorithms in Python.

5. pg_shard
PostgreSQL extension to scale out real-time reads and writes http://citusdata.com/docs/pg-shard
https://github.com/citusdata/pg_shard
Sharding helper extension for PostgreSQL. Nuff said, check docs.

6. peru
Maybe sometimes better than copy-paste.
https://github.com/buildinspace/peru
Ah, nice tool. Another approach of eternal problems of dependencies on your repos. Like "git submodules" but easier. Works with Mercurial and SVN too, not only with git. Demo gif below:

7. awesome-public-datasets
A awesome list of (large-scale) public datasets on the Internet. (On-going collection)
https://github.com/caesar0301/awesome-public-datasets
List of many public (but sometimes not free) datasets on Internet, for your fun and big data projects.

8. rocket
App Container runtime
https://github.com/coreos/rocket
CoreOS creates own container instead of Docker. Quite controversial decision, check their blog for explanation.

9. instavpn
the most user-friendly L2TP/IPsec VPN server
https://github.com/sockeye44/instavpn
Very user-friendly simple but secure VPN. Ubuntu, 512 MB RAM, curl -sS https://sockeye.cc/instavpn.sh | sudo bash, browse at http://IP-ADDRESS:8080 or use cli to setup.

10. shapeme
Evolve images using simulated annealinghttps://github.com/antirez/shapeme
Small toy from @antirez - it takes PNG and try to evolve bunch of triangles to copy it. Just for fun.

воскресенье, 16 ноября 2014 г.

Semi-irregular Sysadmin Ninja's Github Digest (Vol. 12)

Let's continue with review of most interesting Github projects.
N.B. this is vol. 12 - from quite old draft, I hope it's still relevant :)
Sorry for small comments, mostly projects are speak for themselves.

1. changelog
"What's changed in the last twenty minutes?"
https://github.com/prezi/changelog

The basic idea is that you'll send any event that has even a remote chance of causing problems to simple web service with REST interface. Later, when something goes wrong, you can quickly check what's changed in the last minutes / hours.
It's interface looks like this:



Very nice idea, indeed, especially with some graphing solution w/anomaly detection.
Clients for bash/python/java/etc. already exists.


2. db-readings
Readings in Databases
https://github.com/rxin/db-readings

A list of papers essential to understanding databases and building new data systems. Very nice reading for all sysadmins and developers. Only one problem exists - where I can find enough time for that????? :)

3. bitcoinbook
Mastering Bitcoin - Unlocking digital currencies - Early Release Draft
https://github.com/aantonop/bitcoinbook

Another "must-read" book for everyone who interested in digital currencies - it's about Bitcoin internals and technical realization. Early release of book is also available by O'Reilly.

4. github-cheat-sheet
A collection of cool hidden and not so hidden features of Git and GitHub
https://github.com/tiimgreen/github-cheat-sheet

Good stuff, even for experienced Git-hubbers.

5. Awesome Chef
A curated list of amazingly awesome Chef resources
https://github.com/obazoud/awesome-chef/

Where's our "Awesome Puppet"?

Ah, there it is:
6. Awesome Puppet
A curated list of amazingly awesome puppet resources
https://github.com/olindata/awesome-puppet


7. mcrouter
Mcrouter is a memcached protocol router for scaling memcached deployments.
https://github.com/facebook/mcrouter
Another nice piece of stuff from Facebook Engineering team. Must have if you're (still) using memcached on more than couple of servers.


8. masscan
TCP port scanner, spews SYN packets asynchronously, scanning entire Internet in under 5 minutes
https://github.com/robertdavidgraham/masscan


9. powa
PostgreSQL Workload Analyzer http://dalibo.github.io/powa
https://github.com/dalibo/powa


10. elasticsearch-HQ
Monitoring and Management Web Application for ElasticSearch instances and clusters.
https://github.com/royrusso/elasticsearch-HQ

Also nice tool for all ElasticSearch users.

воскресенье, 16 сентября 2012 г.

End of MySQL vs PostgreSQL arguments

Russian text is here.

Good article about MySQL vs Postgresql. Summary for lazy readers -
"MySQL is designed with the idea that applications provide logic and the database provides dumb storage of the application's state.  While this has changed a bit with the addition of user-defined functions and stored procedures, the overall design constrains MySQL primarily to this use case.  This is not necessarily a bad thing as, traditionally, software licensing costs and requirements have often required that even advanced database systems like Oracle are used in this way.  MySQL targets the "my app, my database" world and is usually sufficient for this, particularly when lowest common denominators are used to ensure portability.

PostgreSQL, on the other hand, is designed with the idea that the database itself is a modelling tool, and that the applications interact with it over an API defined in SQL.  Object-relational modelling advocates point out that often getting acceptable performance in complex situations requires an ability to put some forms of logic in the database and even tie this to data structures in the database.  In this model, the database itself is a development platform which exposes API's, and multiple applications may read or write data via these API's.  It is thus best seen as an advanced data modelling, storage, and centralization solution rather than as a simple application back-end.

These differences show, I think, that when PostgreSQL people complain that MySQL is not a "real database management system" and MySQL people dispute this that in fact the real difference is in definitions, and in this case the definitions are deceptively far apart.  Understanding those differences is, I think, the key to making an informed choice.