The Holy Java

Building the right thing, building it right, fast

Why do companies fail at adopting Functional Programming?

Posted by Jakub Holý on June 17, 2015

According to the NDC Oslo talk Lean and Functional Programming by Bryan Hunter, these are the reasons why companies fail to adopt FP:

  • They say “our developers aren’t smart enough” (to use F#, Erlang) [they should invest in their education!]
  • Culture of hiding problems => little incentive to adopt a paradigm that solves/prevents them if they’re invisible
  • Overburden => not time
  • Implementing changes (FP) without first proving them (PDCA) – blindly rewriting something in F#/… can fail; it’s better to have a value-proposition hypothesis and prove it with a limited experiment first
  • The prioritise short-term (e.g. fire-fighting) over long-term (removing the root causes of problems)

Tip for driving FP adoption: Find a pragmatist in pain – e.g. a business person experiencing problems that FP could have prevented.

Posted in Uncategorized | 1 Comment »

AWS API: Proper syntax for filtering by tag name and value (e.g. describeInstances)

Posted by Jakub Holý on June 11, 2015

It took me quite a while to figure out the right syntax for filtering instances by tag name and value in the AWS EC2 API’s describeInstances.

The documentation is not exactly crystal-clear to me:

  • tag:key=value – The key/value combination of a tag assigned to the resource, where tag:key is the tag’s key.

Anyway, here is the proper syntax, provided we are interested in the tag elasticbeanstalk:environment-name:

    var params = {
        Filters: [
            {
                Name: 'tag:elasticbeanstalk:environment-name',
                Values: ['mySuperApp']
            }
        ]
    };
    ec2.describeInstances(params);

So the name of the tag is embedded in the Name part and not, as I initially understood,
{ Name: 'tag', Values: ['elasticbeanstalk:environment-name=mySuperApp'] }

Credit: garnaat.

Posted in [Dev]Ops | Tagged: , | Comments Off on AWS API: Proper syntax for filtering by tag name and value (e.g. describeInstances)

Mounting an EBS volume to Docker on AWS Elastic Beanstal

Posted by Jakub Holý on June 2, 2015

Mounting an EBS volume to a Docker instance running on Amazon Elastic Beanstalk (EB) is surprisingly tricky. The good news is that it is possible.

I will describe how to automatically create and mount a new EBS volume (optionally based on a snapshot). If you would prefer to mount a specific, existing EBS volume, you should check out leg100’s docker-ebs-attach (using AWS API to mount the volume) that you can use either in a multi-container setup or just include the relevant parts in your own Dockerfile.

The problem with EBS volumes is that, if I am correct, a volume can only be mounted to a single EC2 instance – and thus doesn’t play well with EB’s autoscaling. That is why EB supports only creating and mounting a fresh volume for each instance.

Read the rest of this entry »

Posted in General | Tagged: , , , | Comments Off on Mounting an EBS volume to Docker on AWS Elastic Beanstal

OS X: Using scutils to discover whether/what a web proxy is in use

Posted by Jakub Holý on May 7, 2015

When looking for ways to discover whether a proxy is being used by OS X, you will be typically pointed to

networksetup -getwebproxy

However that does not always work – for example when using “Auto Proxy Discovery” and/or “Automatic Proxy Configuration” with a proxy.pac file. scutils --proxy seems to detect all these cases (though it cannot give you the proxy when using auto config, I suppose):
Read the rest of this entry »

Posted in General | Tagged: , , | Comments Off on OS X: Using scutils to discover whether/what a web proxy is in use

All-in-one Docker with Grafana, InfluxDB, and cloudwatch-to-graphite for AWS/Beanstalk monitoring

Posted by Jakub Holý on May 7, 2015

I have derived the Docker container docker-grafana-influxdb-cloudwatch that bundles Grafana dashboards, InfluxDB for metrics storage, and runs cloudwatch-to-graphite as a cron job to fetch selected metrics from AWS CloudWatch and feed them into the InfluxDB using its Graphite input plugin. It is configured so that you can run it in AWS Elastic Beanstalk (the main problem being that only a single port can be exposed – I therefore use Nginx to expose the InfluxDB API needed by Grafana at :80/db/).

Read the rest of this entry »

Posted in General | Tagged: , , , | Comments Off on All-in-one Docker with Grafana, InfluxDB, and cloudwatch-to-graphite for AWS/Beanstalk monitoring

Hack: Quickly Verify That All Your Mocha/Chai Tests Have Valid Assertions

Posted by Jakub Holý on May 6, 2015

Chai is a popular Node/browser assertion library. However – as everything – it has its flaws. An important flaw is that it performs checks on property access – and if you e.g. misspell the name of an assertion, it will be just ignored (for there is no way for Chai to know that you tried to access a non-existent property). This may be fixed in the future with ES6 proxies but so far you risk having tests that actually do not test anything. Though you should of course always develop your tests so that they initially fail and thus you know they actually assert something :-).

Anyway, there is a neat quick way to verify that all your tests have at least one valid assertion – simply replace expect with expect.not.
Read the rest of this entry »

Posted in Testing | Tagged: | Comments Off on Hack: Quickly Verify That All Your Mocha/Chai Tests Have Valid Assertions

My Highlights from Continuous Delivery and DevOps Conference 2015

Posted by Jakub Holý on April 30, 2015

The first Continuous Delivery and DevOps Conference in Oslo is over. It was nice to see so many people interested in the topic. I would have preferred more practical talks of the “how we did it” type over the “why” type but it was OK, though next year I would prefer flatMap. Here are my highlights:

  • Atmel is using a physical robot to plug and connect a particular configuration of circuit boards to test; your automated testing challenges cannot be greater than theirs!
  • Continuous Delivery decreases the risk of outage and time-to-recovery while enabling faster innovation, correlates with higher profits; No efficiency improvement will outperform cycle time reduction
  • Estimation pathologies; focus on value rather than costs
  • Stop talking about requirements, they are fake; they’re just beliefs about what may add value to customers. Use hypothesis instead!
  • Cisco: Most of the tools increasing productivity (and some innovation) were produced by engineers in their “spare” time; slack time is thus crucial
  • How does Cisco grow professionalism : optimise for the 10% best, not the 10% weakest developers; slack time; make everything visible; encourage code reviews but avoid making them mandatory; see the slide
  • CALMS: Culture, Automation, Lean, Measurement, Sharing. The pillars of devOps
  • Cisco invested a lot in crafting their build system, tailored test frameworks, and emulators to be able to get quick and quality feedback – because it pays off
    • “Make you own build system” says @olvemaudal at @CoDeOSL. IME this is inevitable for non-trivial projects, and a good investment.
  • Unleash: Feature Toggles server and Java/Node client by FINN.no
  • “They asked for a report while they actually need just a list of data, the result of a simple SQL query; have we listened to them, we would have wasted hours creating a report in the report framework with logos and all the crap.”

Slides:

Posted in General | Tagged: , | Comments Off on My Highlights from Continuous Delivery and DevOps Conference 2015

iTerm coprocess reporting result of (Mocha) tests run via nodemon

Posted by Jakub Holý on April 30, 2015

See this gist:

Read the rest of this entry »

Posted in Tools | Tagged: , | Comments Off on iTerm coprocess reporting result of (Mocha) tests run via nodemon

Backup WD MyCloud to S3/Glacier with duplicity (build instructions included)

Posted by Jakub Holý on April 3, 2015

How to back up your precious files stored on the WD My Cloud NAS into S3 with the slow but low-cost storage class “Glacier”.

How does the backup work: duplicity does its job and uploads files to S3. The large data archives are recognized by S3 Lifecycle rules that we set up based on their prefix and moved to the Glacier storage class soon after upload. (It takes hours to restore something from Glacier but its cost is orders of magnitude lower than that of S3 itself). We leave metadata files in S3 so that duplicity can read them.

90% of this is based on http://www.x2q.net/2013/02/24/howto-backup-wd-mybook-live-to-amazon-s3-and-glacier/ and the WD build guide (http://community.wd.com/t5/WD-My-Cloud/GUIDE-Building-packages-for-the-new-firmware-someone-tried-it/m-p/770653#M18650 and the update at http://community.wd.com/t5/WD-My-Cloud/GUIDE-Building-packages-for-the-new-firmware-someone-tried-it/m-p/841385#M27799). Kudos to the authors!

You will need to:

  1. Build duplicity and its dependencies (since WD Debian v04 switched to page size of 64kB, all pre-built binaries are unusable)
  2. Configure S3 to move the data files to Glacier after 0 days
  3. Create your backup script – see backup-pictures-to-s3.sh
  4. Schedule to run incremental backups regularly via Cron
  5. Preferably test restore manually

Read the rest of this entry »

Posted in Tools | Tagged: | 4 Comments »

AWS CloudWatch Alarms Too Noisy Due To Ignoring Missing Data in Averages

Posted by Jakub Holý on March 31, 2015

I want to know when our app starts getting slower so I sat up an alarm on the Latency metric of our ELB. According to the AWS Console, “This alarm will trigger when the blue line [average latency over the period of 15 min] goes above the red line [2 sec] for a duration of 45 minutes.” (I.e. it triggers if Latency > 2 for 3 consecutive period(s).) This is exactly what I need – except that it is a lie.

This night I got 8 alarm/ok notifications even though the average latency has never been over 2 sec for 45 minutes. The problem is that CloudWatch ignores null/missing data. So if you have a slow request at 3am and no other request comes until 4am, it will look at [slow, null, null, null] and trigger the alarm.

So I want to configure it to treat null as 0 and preferably to ignore latency if it only affected a single user. But there is no way to do this in CloudWatch.

Solution: I will likely need to run my own job that will read the metrics and produce a normalized, reasonable metric – replacing null / missing data with 0 and weight the average latency by the number of users in the period.

Posted in General, Tools | Tagged: , , | Comments Off on AWS CloudWatch Alarms Too Noisy Due To Ignoring Missing Data in Averages