Posted by Jakub Holý on March 31, 2015
I want to know when our app starts getting slower so I sat up an alarm on the Latency metric of our ELB. According to the AWS Console, “This alarm will trigger when the blue line [average latency over the period of 15 min] goes above the red line [2 sec] for a duration of 45 minutes.” (I.e. it triggers if Latency > 2 for 3 consecutive period(s).) This is exactly what I need – except that it is a lie.
This night I got 8 alarm/ok notifications even though the average latency has never been over 2 sec for 45 minutes. The problem is that CloudWatch ignores null/missing data. So if you have a slow request at 3am and no other request comes until 4am, it will look at [slow, null, null, null] and trigger the alarm.
So I want to configure it to treat null as 0 and preferably to ignore latency if it only affected a single user. But there is no way to do this in CloudWatch.
Solution: I will likely need to run my own job that will read the metrics and produce a normalized, reasonable metric – replacing null / missing data with 0 and weight the average latency by the number of users in the period.
Posted in General, Tools | Tagged: aws, monitoring, ops | Comments Off on AWS CloudWatch Alarms Too Noisy Due To Ignoring Missing Data in Averages
Posted by Jakub Holý on March 27, 2015
One of the annoying things with Jest is that while it enables you to run only a single test by using
it.only, it does not report this in any noticeable way. Thus you can end up in the same situation as we did, not running many tests without knowing it. (Oh yeah, if we only did review the code properly …).
This git pre-commit hook will fail when you introduce
it.only into the code:
Posted by Jakub Holý on March 17, 2015
Our systems always depend on other systems and services and thus may and will be subject to failures – network glitches, dropped connections, load spikes, deadlocks, slow or crashed subsystems. We will explore how to create robust systems that can sustain blows from its users, interconnecting networks, and supposedly allied systems yet carry on as well as possible, recovering quickly – instead of aggreviating these difficulties and turning them into an extended outage and potentially substiantial financial loss. In systems not designed for robustness, even a minor and transient failure tends to cause a chain reaction of failures, spreading destruction far and wide. Here you will learn how to avoid that with a few crucial yet simple stability patterns and the main antipatterns to be aware of. Based primarily on the book Release It! and Hystrix. (Presented at Iterate winter conference 2015; re-posted from blog.iterate.no.)
Read the rest of this entry »
Posted in SW development | Tagged: failure, ops, patterns | Comments Off on There will be failures – On systems that live through difficulties instead of turning them into a catastrophy
Posted by Jakub Holý on March 11, 2015
Being used to the excellent REPL in Clojure(Script), I was surprised to find out that Node.js REPL is somewhat weak and that its support in Emacs is not actively maintained. I anyway managed to get a usable REPL with these three components:
- The Emacs nodejs-repl package (nearly 2 years old)
- J. David Smith’s nodejs-repl-eval.el to be able to send code to the REPL (binding
C-x C-e so that I can execute the current sexp/region)
- My own extension of nodejs-repl-eval.el that takes care of escaping JS constructs that the REPL interprets in a special way
Regarding #3: The problem with the Node.js REPL is that valid JS code does not always behave correctly in the REPL. This is because: 1)
_ is a special variable (the last result) while in code it is often used for the underscore/lodash library; 2) The REPL also interprets lines somewhat separately and tries to execute
<dot><name> as a REPL command, breaking chained calls that start on a new line. My solution uses some RegExp magic to turn
var _ = require("lodash"); // #1a conflicting use of _
_.chain([1,2]) // #1b conflicting use of _
.first() // #2 interpreted as non-existing REPL command '.first'
var __ = require("lodash"); // #1a Notice the doubled _
__.chain([1,2]). // #1b Notice the doubled _
first(). // #2 Notice the dot has moved to the previous line
when the code is being sent to the REPL
Posted in Languages, Tools | Tagged: emacs, nodejs, productivity | Comments Off on A Usable Node.js REPL for Emacs