The Holy Java

Building the right thing, building it right, fast

Archive for the ‘Tools’ Category

IDEs and various other developer tools

Backup WD MyCloud to S3/Glacier with duplicity (build instructions included)

Posted by Jakub Holý on April 3, 2015

How to back up your precious files stored on the WD My Cloud NAS into S3 with the slow but low-cost storage class “Glacier”.

How does the backup work: duplicity does its job and uploads files to S3. The large data archives are recognized by S3 Lifecycle rules that we set up based on their prefix and moved to the Glacier storage class soon after upload. (It takes hours to restore something from Glacier but its cost is orders of magnitude lower than that of S3 itself). We leave metadata files in S3 so that duplicity can read them.

90% of this is based on and the WD build guide ( and the update at Kudos to the authors!

You will need to:

  1. Build duplicity and its dependencies (since WD Debian v04 switched to page size of 64kB, all pre-built binaries are unusable)
  2. Configure S3 to move the data files to Glacier after 0 days
  3. Create your backup script – see
  4. Schedule to run incremental backups regularly via Cron
  5. Preferably test restore manually

Read the rest of this entry »

Posted in Tools | Tagged: | 4 Comments »

AWS CloudWatch Alarms Too Noisy Due To Ignoring Missing Data in Averages

Posted by Jakub Holý on March 31, 2015

I want to know when our app starts getting slower so I sat up an alarm on the Latency metric of our ELB. According to the AWS Console, “This alarm will trigger when the blue line [average latency over the period of 15 min] goes above the red line [2 sec] for a duration of 45 minutes.” (I.e. it triggers if Latency > 2 for 3 consecutive period(s).) This is exactly what I need – except that it is a lie.

This night I got 8 alarm/ok notifications even though the average latency has never been over 2 sec for 45 minutes. The problem is that CloudWatch ignores null/missing data. So if you have a slow request at 3am and no other request comes until 4am, it will look at [slow, null, null, null] and trigger the alarm.

So I want to configure it to treat null as 0 and preferably to ignore latency if it only affected a single user. But there is no way to do this in CloudWatch.

Solution: I will likely need to run my own job that will read the metrics and produce a normalized, reasonable metric – replacing null / missing data with 0 and weight the average latency by the number of users in the period.

Posted in General, Tools | Tagged: , , | Leave a Comment »

A Usable Node.js REPL for Emacs

Posted by Jakub Holý on March 11, 2015

Being used to the excellent REPL in Clojure(Script), I was surprised to find out that Node.js REPL is somewhat weak and that its support in Emacs is not actively maintained. I anyway managed to get a usable REPL with these three components:

  1. The Emacs nodejs-repl package (nearly 2 years old)
  2. J. David Smith’s nodejs-repl-eval.el to be able to send code to the REPL (binding nodejs-repl-eval-dwim to C-x C-e so that I can execute the current sexp/region)
  3. My own extension of nodejs-repl-eval.el that takes care of escaping JS constructs that the REPL interprets in a special way

Regarding #3: The problem with the Node.js REPL is that valid JS code does not always behave correctly in the REPL. This is because: 1) _ is a special variable (the last result) while in code it is often used for the underscore/lodash library; 2) The REPL also interprets lines somewhat separately and tries to execute <dot><name> as a REPL command, breaking chained calls that start on a new line. My solution uses some RegExp magic to turn

var _ = require("lodash"); // #1a conflicting use of _
_.chain([1,2])             // #1b conflicting use of _
   .first()                // #2 interpreted as non-existing REPL command '.first'


var __ = require("lodash");  // #1a Notice the doubled _
__.chain([1,2]).             // #1b Notice the doubled _
   first().                  // #2 Notice the dot has moved to the previous line

when the code is being sent to the REPL

Posted in Languages, Tools | Tagged: , , | Comments Off

Connect Tunnelblick to VPN automatically after wake up

Posted by Jakub Holý on December 12, 2014

Need: Make sure that VPN is always running except when at work.

Partial solution: Make sure VPN is always running with “connect when computer starts” and using an AppleScript to connect after waking up from sleep. Disconnect manually when at work.

Future: Check the current location (wifi name? IP?) and do not connect when at work.

Read the rest of this entry »

Posted in Tools, Uncategorized | Tagged: , | Comments Off

Notes From CodeMesh 2014

Posted by Jakub Holý on November 3, 2014

My consise highlights from CodeMesh 2014.

Philip Potter has very good CodeMesh notes as well, as usually.

TODO: Check out the papers mentioned in the  NoSQL is Dead talk. (<- slides)

Tutorial: QuickCheck (John Hughes)

  • QC => Less code, more bugs found
  • QC tests are based on models of the system under test – with some kind of a simple/simplified state, legal commands, their preconditions, postconditions, and how they impact the state. The model is typically much smaller and simpler than the imple code.
  • QuickCheck CI ( – free on-line service for running CI tests for a GitHub project. Pros: You don’t need QC/Erlang locally to play with it, it provides history of tests (so you never loose a failed test case), it shows test coverage also for failed tests so you see which code you can ignore when looking for the cause.
  • See John’s GitHub repo with examples –
Shrinking (a.k.a. simplification)
  • Doesn’t just make the example shorter by leaving things out by tries a number of strategies to simplify the exmple, typically defined by the corresponding generators – f.ex. numbers are simplified to 0, lists to earlier elements (as in “(elements [3, 4, 5])”) etc.
  • You may implement your own shrinking strategies. Ex.: Replace a command with “sleep(some delay)” – so that we trigger errors due to timeouts. (A noop that just waits for a while is simpler than any op).
Bug discovery
  1. Run QC; assuming a test failed:
  2. Instead of diving into the implementation, use first QC to check your hypothesis of what constitutes “bad input” by excluding the presumed bad cases – f.ex. “it fails when input has 8 characters” => exclude tests with 8 and rerun, if you find new failures you know the hypothesis doesn’t cover all problems – and you will perhaps refine it to  “fails when it has a multiple of 8 chars etc. We thus learn more about the wrong behavior and its bounds. Assumption we want to verify: No (other) tests will fail.
  3. Do the opposite – focus on the problem, i.e. modify the test case to produce only “bad cases”. Assumption we want to verify: all tests will fail.
QC vs. example-based testing

QC code tends to be 3-6* smaller than implementation (partly thanks to the consiseness of Erlang) and fairly simple.

The case of Vovlo: 3k pages of specs, 20 kLOC QC, 1M LOC C implementations, found 200 bugs and 100 problems (contradictions, unclarities) in the specs. It took 2-3 years of working on it on and off.

Erlang dets storage race conditions: 100 LOC QC, 6 kLOC Erlang impl.

Testing stateful stuff

Invoke dfferent API calls (“commands”) until one of presumabely legal calls fails due to an accumulated corrupted state. This is an iterative process where we evolve our model of the system – commands, their preconditions (when they can be legally invoked), postconditions, and our repreentation of the state.

Ex.: Testing of a circular queue. Commads: push (legal on non-full queue), get (legal on non-empty), create new => generates sequences of new and pushs and gets.

Testing race conditions

Precondition: Run on a multicore PC or control the process scheduler.

  • There are many possible correct results (valid interleavings) of parallel actions => impractical to enumerate and thus to test with example-based tests
  • Correct result is such that we can order (interleave) the concurrently executed actions such that we get a sequential execution yielding the same result. F.ex. an incorrect implementation of a sequence number generator could return the same number to two concurrent calls – which is not possible if the calls were done sequentially.
Testing data structures

Map the DS to a simpler one and use that as the model – f.ex. a list for a tree (provided there is a to_list function for the tree).

Tutorial: Typed Clojure (Ambrose Bonnaire-Sergeant)

Note: The documentation (primarily introductory one) could be better

Defining types
  1. separately: (ann ..)
  2. around: wrap in (ann-form <defn> <type def.>)
  3. inside: use t/fn, t/defprotocol etc.
Gradual introduction of typed.clojure
  • wrap everything in (t/tc-ignore …)
  • for unchecked fns you depend on, add (ann ^:no-check somefn […])
  • If you stare at a type error, consider using contracts (prismatic/Schema or pre/post conds etc.)

Keynote: Complexity (Jessica Kerr, Dan North)

  • Always have tasks of all three types: research (=> surface conplexity), kaizen (cont. improvement, improvement of the imprv. process), coding – these 3 interlave the whole time
  • A team needs skills in a number of areas, it isn’t just coding – evaluation of biz valuedelivered, monitoring, programming, testing, deployment, DB, FS, networks, … .

Keynote: Tiny (Chad Fowler)

Keep things tiny to be efficient (tiny changes, tiny teams, tiny projects, …).

  • Research by armies and in SW dev [TODO: find the two slides / qsm, Scrum] shows that teams of max 5-6 work best
    • Teams of 7+ take considerably more time and thus money (5* more wrt. one study) to complete the same thing
    • => small, autonomous teams with separate responsabilities (decomosition, SRP etc. FTW!)
  • Human capacity to deal with others is limited – one company creates a new department whenever size exceeds 100
  • Big projects fail; Standish CHAOS report – only cca 10% larger projects succeed compared to nearly 80% of small ones (summed together: 39% succeed)
  • Note: 1 month is not a short iteration

Distributed Programming (Reid Draper)

RPC is broken

– it tries to pretend a remote call is same as local but:

  • what if the call never returns?
  • the connection breaks? (has the code been executed or not yet?)
  • what about serialization of arguments (file handles, DB conn.,…)

It ignores the special character of a remote code and the 8 fallacies of distributed progr.

Message passing

is batter than RPC. There is also less coupling as the receiver itself decides what code to call for a specific message.


The Bloom lang from the BOOM research project explores new, better ways of distributed programming. Currently implemented as a Ruby DSL.

From its page (highlight mine):

Traditional languages like Java and C are based on the von Neumann model, where a program counter steps through individual instructions in order. Distributed systems don’t work like that. Much of the pain in traditional distributed programming comes from this mismatch:  programmers are expected to bridge from an ordered programming model into a disordered reality that executes their code.  Bloom was designed to match–and exploit–the disorderly reality of distributed systems.  Bloom programmers write programs made up of unordered collections of statements, and are given constructs to impose order when needed.

Correctness testing of concurrent stuff
  • Unit testing unsuitable – there are just too many combinations of correct results and can only test the cases the dev can think of
  • => generate the tests – property-based testing / QuickCheck
  • PULSE – an addon to property-based testing that tries to trigger concurrency problems by using a scheduler that tries different interleavings of actions (randomly but repeatedly) [Erlang]
  • Simulation testnng – Simulant

Beware the effects of GC, page cache, cronjob (e.g. concurrently running backup), SW updates => running a simple load test for few mins is not enough.

Cheats & Liars: The Martial Art of Protocol Design (Pieter Hintjens)

Pieter is the brain behind AMQP, ZeroMQ and EdgeNet (protocols for anonymous, secure, peer-to-peer internet). He has shared great insights into designing good protocols, the dirty business surrounding protocols and standardization, and troll-proof organization of communities (as a self-roganizing, distributed team).

More: See Ch.6 The ØMQ Community in the online The ZeroMQ Guide – for C Developers. (He has also other interesting online or paid books.)

  • Protocol is a contract for working together
  • IT should be minimalistic and specific, name the participants, …
  • Protocols and their strandardization are prey to “psychopatic” organizations that want to hijack them for their own profit (by pushing changes that benefit them, taking over the standardization process, …) (Pieter has experienced it e.g. with AMQP; these trolls always show up). It’s advantegous to take control of a successful protocol so that you can make money off it or build stuff on it and sell that. Examples:
    • Microsoft MS Doc XML  – this “open” spec f.ex. reportedly defines that one functions works “as Word 95″
    • A company pushing changes that nobody else really understands, thus undermining compatibility of implementations
    • Pushing such changes that an implementor can claim compliance to the standard yet implement it so that his products only work with his products
    • Crazy/proprieetary protocol extensions, patenting/trademarking/copyrighting the spec (e.g. TM on Bluetooth)
  • Hijacking-safe protocol creation process (beware “predatory maliciousness”):
    • The specs is GPL => nobody can capture it (e.g. ZeroMQ)
    • The community has clear rules and deals with trolls by kicking them out
    • There is a good process for evolving the spec
  • How to spec i protocol?
    • Start with a very, very small and simple contract – only add things that you desperately need – e.g. ZeroMQ v1 had no versioning, security, metadata (versioning added in v2, metadata in v3, ecurity later). You don’t know what you really need until you try it. F.ex. even the original AMQP has 60-75% waste in it!!!!
    • Do very slow and gradual evolution
    • Layering is crucial – keep your protocol on one layer, only specify relevant things, leave the other layers for other specs so they can evolve and age in different speed; the more in a spec the earlier will st. be outdated (Pizza contract says nothing about the kitchen, f.ex.)
  • Community and cooperation (See the Ch.6 The ØMQ Community mentioned above.)
    • community needs clear rules to keep trolls away (and they always pop up)
    • don’t just absorb the damage trolls do, ban them
    • self-org., decentralized team

PureScript (Bodil Stokke)

Working with larger JavaScript apps (> 100 LOC :-)) is painful, primarily due to the lack of type checking and thus requiring one to take lot of care when chaning code so that nothing breaks at runtime. TypeScript is a possible solution but it still feels limited.

PureScript is very Haskell-like language compiled to JS. It is a pure functional lang, effects are performed only via the Effect Monad (Eff). It is pragmatic w.r.t. interoperability – it is possible to embedd JS code and just add a signature to it, the compiler will trust it.

Moreover, you can use property-basd testing with QuickCheck and Functional Reactive Programming with Bodil’s Signal library. Isn’t that wonderful?!

See the PureScript Is Magic code at GitHub, primarily the 150 LOC Main.purs.


Category theory notes:

  • Semigroup is a domain with a cumulative operation (e.g. ints with +)
  • Monoid (?) is a semigroup with a unit element, i.e. one where “element operation unit = element” as 0 for + or 1 for *.

Megacore, Megafast, Megacool (Kevin Hammond)

Interesting research project ParaPhrase for parallelization of code through automatic refactoring and application of any of supported topologies (Farm, Pipeline, Map, …) – and (in general the promises of automatizatio regarding fixing software development problems have have hugely underdelivered but still something might be possible). In some cases their solution managed to do in hours what a developer did in days.

Quote Bob Harper:

The only thing that works for parallelism is functional programming

PS: C++17 is going to have support for parallel and concurrent programming.

 Categories for the Working Programmer (Jeremy Gibbons)

An elementary intro into category theory in 10 points, yet I got immediately lost. It might be worth to study the slides and look for more resources at the author’s uni page and not-so-active blog Patterns in Functional Programming .

 NoSQL is Dead (Eric Redmond)

Main message: There is just too many differences between NoSQL DBs for this expression to be meaningful.

Lobby talks

Hacking people

I had an inpiring lunch chat with Chad and a Polish lady whose name I unfortunately don’t know. Their companies do fascinating stuff to leverage the potential of their humans – one has replaced top-down management wrt. projects with environment where there are clear objectives (increase monthly active users) and the freedom to come with ideas to contribute to them, recruit other people for the idea and, if successful, go implement it (while continually measuring against the objectives). Clearly enough it is not easy, some people have troubles trying to manage everything or doing what they believe in without checking the real implect on the objectives etc. but it already provides more value then before. This has been going on for just a few months so hopefully when it settles more we will hear about their experience.

The other company realized that people are different (wow! how could our industry ignore that?!) and started to go psychological profiling of employees to understand what type of team member they are – a driver, a worker, a critic who is always hunting for possible issues and problems etc. And they compose teams so that they have the right mix of different personalities to avoid both insurpasable conflicts and the risks of group-think.

I believe this is the future of our industry – really understand people and “hack” our organizations to leverage that for greater happiness and better results.


  • Jessica Kerr: Simulation of team work and the effect of (no) slack

    – what happens when you let your programmers crunch work without any slack time? And when you introduce slack? Jessica has made this Scala simulation to produce the results we would expect – much more even production in the slack case, lot of rework after deploying features in the non-slack version. Not at all scientific but very nice when you want to *show* your higher-ups what happens when you do the former or the latter. Some people hear much more to a visual stimuli (even though totally made to conform to the message you want to get across) than tons of theory.

  • Aphyr – [313] Strong consistency models – “strong consistency” is a much broader term than I expected and not all consistency models are so consistent :-) Check out especially this consistency family tree image.

Posted in Languages, Testing, Tools | Tagged: , , | Comments Off

Most interesting links of June ’14

Posted by Jakub Holý on June 30, 2014

Recommended Readings

  • The emperor’s new clothes were built with Node.js – I know sadly little about Node.js but this goes against the hype and is thus interesting. So what does Node.js give us? Performance 1-5x slower than Java [like Clojure] according to the Benchmarks Game (contrary to other benchmarks with the opposite result as mentioned in the comments), use of a single CPU/core on our multi-cpu, multi-core machines, callback hell. At the same time, there are good non-blocking servers available in other languages (Clojure’s http-kit, Vert.x, etc.) (Update: From the comments it seems that f.ex. the “callback hell” situation is geting better with 0.11, fibers and other things I do not know anything about. Also Sandro has a nice anti-comment (No. 36).)
    The Node.js Is Bad Ass Rock Star Tech 5 min video is a nice companion :)
  • The Expert (Short Comedy Sketch)  (7 min) – you’ve certainly seen this one but I had to put it here; a young engineer is hammered into being an “Of course I can do it, I am an expert” ‘expert/consultant’ during a business meeting. Maybe you too have experienced a dialog with the business where your true expert opinion was crushed by the business people’s insistence on their absurd requirements?
  • Reset The Net – Privacy Pack – privacy-enhancing apps for PC/mobile
  • The Dyslexic Programmer (via Kent Beck) – interesting read about quite a different way to percieve and think about code, the advantages of IDEs.
  • It’s Here: Docker 1.0 => more stable from now on
  • Kent Beck: Learning About TDD: The Purpose of #isTDDDead – what is the purpose and value of TDD? Where are the limits of its value? “I recognize that TDD loses value as tests take longer to run, as the number of possible faults per test failure increases, as tests become coupled to the implementation, and as tests lose fidelity with the production environment.
  • Failure & Cake: A Guide to Spotify’s Psychology of Success – want to be innovative and successfull? Learn to embrace failure, nurture the “growth mindset” (failure as opportunity to improve) rather than the “fixed mindset” (I do not learn and every failure shows I have no value). Read this if you want your org to be a better place to work!


  • LSD — The Problem-Solving Psychedelic – I never knew that drugs could be used to something positive, with an incredible effect. Are you stuck with a tech/design/art problem? Try LSD! :-)
  • The French are right: tear up public debt – most of it is illegitimate anyway – “Debt audits show that austerity is politically motivated to favour social elites. [..] 60% of French public debt is illegitimate” – not improving the lives of people but thos at power/rich. Time to reconsider this debt business and ways to make our system better?
  • Forbes: Why Financialization Has Run Amok – Wall Street is the kind and companies do everything to look better in its eyes – including giving up on opportunities. The might of the finance sector is destructive to our economy and distorts it, away from producing more value to making financial institutions richer, away from (value) creative activities to distributive ones. The article describes the problem and proposes a solution including limiting the size and leverage of banks, taxing financial transactions etc. Example of the effects: “[..] a cabal of senior IBM executives and the managers of some big investment firms got together and devised a five-year scheme—IBM’s Roadmap 2015—for increasing IBM’s earnings per share—and their own compensation—through measures that are not only increasing earnings per share but also steadily crippling IBM’s ability to innovate and compete [..]
  • Why Can’t We All Just Get Along? The Uncertain Biological Basis of Morality – very interesting criticism of “morality” that is mostly based on emotions and thus contradictory, a good argument for utilitarian morality [not that it hasn’t its own challenges]. According to the author, many conflicts are nor primarily due to divergent values but due to different interpretation of the reality and history (such as “who has right to this land?”). People suffer “[..] from a deep bias—a tendency to overestimate their team’s virtue, magnify their grievances, and do the reverse with their rivals.” “This is the way the brain works: you forget your sins (or never recognize them in the first place) and remember your grievances. [..] As a result, the antagonisms confronting you may seem mysterious, and you may be tempted to attribute them to an alien value system.” This leads to partial judgements that play very badly with another psychological feature – “Namely: the sense of justice—the intuition that good deeds should be rewarded and bad deeds should be punished.” “When you combine judgment that’s naturally biased with the belief that wrongdoers deserve to suffer, you wind up with situations like two people sharing the conviction that the other one deserves to suffer. Or two groups sharing that conviction. And the rest is history.” And “The most common explosive additive is the perception that relations between the groups are zero-sum—that one group’s win is the other group’s loss.” => “So maybe the first step toward salvation is to become more self-aware.
    When you’re in zero-sum mode and derogating your rival group, any of its values that seem different from yours may share in the derogation. Meanwhile, you’ll point to your own tribe’s distinctive, and clearly superior, values as a way of shoring up its solidarity. So outsiders may assume there’s a big argument over values. But that doesn’t mean values are the root of the problem.
    Those who choose not to act in the trolley dilemma[..] are just choosing to cause five deaths they won’t be blamed for rather than one death they would be blamed for. Not a profile in moral courage!

Clojure Corner

  • The Case for Clojure (video, 5 min) – a short video arguing for Clojure as a good solution language based on its simplicity, power, and fun factor. There are many claims and few facts (as dictated by the short length) but it might be interesting for somebody.
  • – cross-reference of many OSS Clojure projects – find all uses of a fn across the projects, all fns with a given name, all projects using ring, … . Search by fn, macro, var, ns, prj.
  • The Weird and Wonderful Characters of Clojure – ‘A reference collection of characters used in Clojure that are difficult to “google”.’


Posted in General, Languages, Tools, Top links of month | Tagged: , , , , , , , , | Comments Off

Most interesting links of May ’14

Posted by Jakub Holý on May 31, 2014

Recommended Readings

  • Monolith – from The Codeless Code – fables and koans for the SW engineer – the Monad monolth #Haskell #fun
  • http2 explained (pdf, 27 pages) – cons of http 1 (huge spec / no full impl., wasteful use of TCP <=> latency [x spriting, inlining, concatenation, sharding]) => make it less latency sensitive, fix pipelining (issue a req before previous one finished), stop the need for ever increasing # connections, remove/reduce optional parts of http. Http2 is binary; multiple “streams” over 1 connection => much less conns, faster data delivery; header/data compression; [predictive] resource pushing; . Inspired by SPDY. Chrome and Mozilla will only support it over TLS, yay! (see also Is TLS Fast Yet? [yes, it is]) Promise: faster, more responsive web pages & deprecation of http/1 workarounds => simplified web dev.


  • – crowd-sourced good code mentorship – get an exercise, implement it in any of the supported language(s), submit and get feedback, repeat; when finished, you too can comment the same excercise submitted by others while working on your next assignment. Languages include Clojure, JS, Scala, Python, Haskell, Go, Elixir, Java, and more.

Podcasts (FP & related)

  • Cognicast (also @ iTunes) – Clojure, FP, etc.
  • Functional Geekery (@ iTunes) – A podcast on Functional Programming, covering topics across multiple languages.
  • Mostly λazy…a Clojure podcast by Chas Emerick
  • Giant Robots Smashing into other Giant Robots – “a weekly technical podcast discussing development, design, and the business of software development”
  • Software Engineering Radio (@ iTunes) – “The goal is to be a lasting educational resource, not a newscast. Every two to four weeks, a new episode is published that covers all topics software engineering. Episodes are either tutorials on a specific topic, or an interview with a well-known expert from the software engineering world.”
  • EngineerVsDesigner – design insight (@ iTunes) – product design podcast – the latest digital design news, tips & tricks, Q&A, and an industry special guest


Clojure Corner


  • ownCloud – your own Dropbox/Google Drive, run on your server – sharing files between devices / PCs / web, syncing calendar and contacts, collaborative editing of documents (ODF)
  • Mailpile – “A modern, fast web-mail client with user-friendly encryption and privacy features.”, to be self-hosted on a PC, RaspberryPI, USB stick
  • Blackhole – role-based ssh proxy – an app that enables you to manage what users can ssh to what server as a particular user, from users’ point of view this is a ssh proxy; useful if many people need access to many servers but you do not want to add them all as users on those servers.
  • Wuala – Secure Cloud Storage – Backup. Sync. Share. Access Everywhere. – Dropbox alternative, secure by default
  • fb-flo – Facebook’s live-coding tool
  • – self-hosted Dropbox-like service with calendar and contact sync and more

Favourite Quotes

Posted in General, Languages, Tools, Top links of month | Tagged: , , , , , , , , , , | Comments Off

Graphite Shows Metrics But No Data – Troubleshooting

Posted by Jakub Holý on May 5, 2014

My Graphite has all the metrics I expect but shows no data for them. Communication between my app and Graphite clearly works otherwise the metrics would not have appeared in the list but why is there no data?

Update: Graphite data gotchas that got me

(These gotchas explain why I did not see any data.)

  1. Graphite shows aggregated, not raw data if the selected query period (24h by default) is greater than the retention period of the highest precision. F.ex. with the schema “1s:30m,1m:1d,5m:2y” you will see data at the 1s precision only if you select period less than or equal to the past 30 minutes. With the default one, you will see the 1-minute aggregates. This applies both to the UI and
  2. Aggregation drops data unless by default at least 50% of available data slots have values (xFilesFactor=0.5). I.e. if your app sends data at a rate more than twice slower than Graphite expects them, they will never show up in aggregates. F.ex. with the schema “1s:30m,1m:1d,5m:2y”  you must sends data at least 30 times within a minute for them to show in an aggregate.

I suppose that would show the raw data.

Lesson learned: Always send data to Graphite in *exactly* same rate as its highest resolution

As described above, if you send data less frequently than twice the highest precision (if 1s => send at least every 2s), aggregation will ignore the data, with the default xFilesFactor=0.5 (a.k.a. min 50% of values reqired factor). On the other hand, if you send data more frequently than the highest precision, only the last data point received in each of the highest precision periods is recorded, others ignored – that’s why f.ex. statsD flush period must = Graphite period.

Read the rest of this entry »

Posted in Tools | Tagged: , | Comments Off

How to create and run Gatling 2.0 tests

Posted by Jakub Holý on April 28, 2014

Getting up and running with Gatling perf. tests as I would like so I record this for my future reference.

0. Create a project:

$ mvn archetype:generate \

(The trailing “:” in the filter is important.)

1. Import to IntelliJ

In IntelliJ choose to import an object, instead of “from sources” select “from external model” and then Maven. You will also need to have the Scala plugin installed and, when imported, you will likely need to right-click on pom.xml and Maven – Reimport.

2. Record a simulation

  1. Run the src/test/scala/Recorder.scala (right-click – Run ‘Recorder’)
  2. Set the port it should listen on, f.ex. 8000 (maybe you also need to set port for HTTPS, f.ex. 8001), set the target app (perhaps localhost, <some port>, <some https/dummy port>)
  3. Optionally set the class name of the recorded simulation and the target package (the output goes to src/test/scala/<package>/<name>.scala)
  4. Click [Start !]
  5. Go to your browser and configure it to use the recorder as its HTTP[s] proxy
  6. Browse localhost:8000/your_app as you want for the test
  7. Click [Stop and save] in the Recorder UI

Read the rest of this entry »

Posted in Testing, Tools | Tagged: , | Comments Off

Ansible: Best practices for deriving host-level var from a group var

Posted by Jakub Holý on March 19, 2014

I have a cluster and a group variable (in test/staging/prod) holding the hostname of the cluster master. For each host, I want to derive a variable which is set to either “slave” or “master” and use it in templates.

This can be done with set_fact:

# group_vars/staging:


# roles/xxx/tasks/main.yml:
- name: Set default jboss_host_type
  set_fact: jboss_host_type=slave
- name: Set jboss_host_type to master if master
  set_fact: jboss_host_type=master
  when: jboss_master_host == inventory_hostname

We could simplify that by using more advanced Jinja:

# roles/xxx/tasks/main.yml:
- name: Set jboss_host_type var
  set_fact: jboss_host_type={{ 'master' if jboss_master_host == inventory_hostname else 'slave' }}

but it is preferred not to use logic coded in Jinja2.

// Ansible 1.5.3

Posted in Tools | Tagged: | Comments Off