The Holy Java

Building the right thing, building it right, fast

Archive for the ‘Languages’ Category

Topics related to a programming language and to these languages in general

Troubleshooting javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Posted by Jakub Holý on October 5, 2018

Re-published from the Telia Tech Blog.

The infamous Java exception javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure is hardly understandable to a mere mortal. What it wants to say is, most likely, something like this:

Sorry, none of the cryptographic protocols/versions and cipher suites is accepted both by the JVM and the server.

For instance the server requires a higher version of TLS than the (old) JVM supports or it requires stronger cipher suites than the JVM knows. You will now learn how to find out what is the case.

We will first find out what both the server and the JVM support and compare it to see where they disagree. Feel free to just skim through the outputs and return to them later after they were explained.

What does the server support?

We will use nmap for that (brew install nmap on OSX):

map --script ssl-enum-ciphers -p 443 my-server.example.com
Starting Nmap 7.70 ( https://nmap.org ) at 2018-10-05 00:54 CEST
Nmap scan report for my-server.example.com (127.0.0.1)
Host is up (0.031s latency).

PORT STATE SERVICE
443/tcp open https
| ssl-enum-ciphers:
| TLSv1.2:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
| compressors:
| NULL
| cipher preference: server
|_ least strength: A

Here we see that the server only supports TLS version 1.2 (ssl-enum-ciphers: TLSv1.2:) and the listed ciphers, such as TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA.

What does the JVM have on offer?

Now we will find out what the JVM supports (I did that through Clojure but you could have just as well used Java directly; notice the javax.net.debug property):

sh $ env -i java -Djavax.net.debug=ssl:handshake:verbose java -jar clojure-1.8.0.jar
Clojure 1.8.0
user=> (.connect (.openConnection (java.net.URL. "https://my-server.example.com/ping")))
;; ...
done seeding SecureRandom
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256
Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_128_CBC_SHA256
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
main, setSoTimeout(0) called
%% No cached client session
*** ClientHello, TLSv1
RandomCookie: GMT: 1521850374 bytes = { 121, 217, 101, 186, 111, 183, 47, 46, 159, 230, 139, 103, 7, 181, 250, 172, 113, 121, 4, 55, 122, 148, 111, 82, 87, 170, 70, 10 }
Session ID: {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_SHA, TLS_ECDH_ECDSA_WITH_RC4_128_SHA, TLS_ECDH_RSA_WITH_RC4_128_SHA, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_RC4_128_MD5, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods: { 0 }
Extension elliptic_curves, curve names: {secp256r1, sect163k1, sect163r2, secp192r1, secp224r1, sect233k1, sect233r1, sect283k1, sect283r1, secp384r1, sect409k1, sect409r1, secp521r1, sect571k1, sect571r1, secp160k1, secp160r1, secp160r2, sect163r1, secp192k1, sect193r1, sect193r2, secp224k1, sect239k1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension server_name, server_name: [host_name: my-server.example.com]
***
main, WRITE: TLSv1 Handshake, length = 175
main, READ: TLSv1 Alert, length = 2
main, RECV TLSv1 ALERT: fatal, handshake_failure
main, called closeSocket()
main, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
SSLHandshakeException Received fatal alert: handshake_failure sun.security.ssl.Alerts.getSSLException (Alerts.java:192)

Here we see that the JVM uses TLS version 1 (see *** ClientHello, TLSv1) and supports the listed Cipher Suites, including TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA.

What’s wrong?

Here we see that the server and JVM share exactly one cipher suite, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA. But they fail to agree on the TLS version, since the server requires v1.2 while the JVM only offers v1.

The solution

You can either configure the server to support a cipher suite and protocol version that the JVM has or teach JVM to use what the server wants. In my cases that was resolved by running java with -Dhttps.protocols=TLSv1.2 (alternatively, you could add all of SSLv3,TLSv1,TLSv1.1,TLSv1.2) as recommended by π at StackOverflow.

Sources

The troubleshooting technique comes from the article “SSLHandshakeException: Received fatal alert: handshake_failure due to no overlap in cipher suite
” by Atlassian. The observation that the server and JVM disagreed on the TLS version comes from my good colleague Neil.

Advertisements

Posted in General, Languages | Tagged: , , | Leave a Comment »

Experience: Awesome productivity with ClojureScript’s REPL

Posted by Jakub Holý on December 21, 2017

Re-posted from Telia’s tech blog.

What’s the deal with ClojureScript? How can you justify picking such a “niche” language? I have recently experienced a “wow” session, demonstrating the productivity gains of ClojureScript and the interactive development it enables thanks to its REPL. I would like to share the experience with you. (If you have never heard about it before – it is a modern, very well designed Lisp that compiles to JavaScript for frontend and backend development. It comes with a REPL that makes it possible to reload code changes and run code in the context of your live application, developing it while it is running.)

Read the rest of this entry »

Posted in Languages | Tagged: , , | Comments Off on Experience: Awesome productivity with ClojureScript’s REPL

Storytelling as a Vehicle of Change: Introducing ClojureScript for the Heart and Mind

Posted by Jakub Holý on October 7, 2015

People don’t really like changes yet change we must in this fast-developing world. How to introduce a change, or rather how to inspire people to embrace a change? That is one of the main questions of my professional life.

I have recently talked about Functional programming (FP) in JavaScript and compared it to ClojureScript, which was designed for FP. To my surprise the team proposed to give ClojureScript a try and we agreed to have a live coding session, implementing a new functionality in our internal part of our webshop using ClojureScript. But how to kindle this little flame of motivation to keep it going, despite hurdles that will certainly come? And here I got a few interesting ideas.

  1. An experienced speaker once recommended sharing personal experiences (even – or especially – if they make me vulnerable) as it is much easier for people to relate to them than to general statements.
  2. A Cognicast eposide mentioned storytelling as a great tool for introductory guides. We humans are natural storytellers, we think in stories and relate to them much more easily – so a story should be great also to communicate the value of a change.
  3. My ex-colleague Therese Ingebrigtsen gave an inspiring talk presenting some points from The Switch – mainly that we need to address the recipient’s minds with rational arguments, but also their hearts to involve their emotion (e.g. by drawing a picture of the new bright future), and that it is important to show a clear path forward.

Read the rest of this entry »

Posted in General, Languages | Tagged: , , | Comments Off on Storytelling as a Vehicle of Change: Introducing ClojureScript for the Heart and Mind

An answer to CircleCI’s “Why we’re no longer using Core.typed”

Posted by Jakub Holý on October 6, 2015

CircleCI has recently published a very useful post “Why we’re no longer using Core.typed” that raises some important concerns w.r.t. Typed Clojure that in their particular case led to the cost overweighting the benefits. CircleCI has a long and positive relation to Ambrose Bonnaire-Sergeant, the main author of core.typed, that has addressed their concerns in his recent Strange Loop talk “Typed Clojure: From Optional to Gradual Typing” (gradual typing is also explained in his 6/2015 blog post “Gradual typing for Clojure“). For the sake of searchability and those of us who prefer text to video, I would like to summarise the main points from the response (spiced with some thoughts of my own).

Read the rest of this entry »

Posted in Languages | Tagged: , | Comments Off on An answer to CircleCI’s “Why we’re no longer using Core.typed”

Refactoring & Type Errors in Clojure: Experience and Prevention

Posted by Jakub Holý on October 6, 2015

While refactoring a relatively simple Clojure code to use a map instead of a vector, I have wasted perhaps a few hours due to essentially type errors. I want to share the experience and my thoughts about possible solutions since I encounter this problem quite often. I should mention that it is quite likely that it is more a problem (an opportunity? :-)) with me rather than the language, namely with the way I write and (not) test it.

The core of the problem is that I write chains of transformations based on my sometimes flawed idea of what data I have at each stage. The challenge is that I cannot see what the data is and have to maintain a mental model while writing the code, and I suck at it. Evaluating the code in the REPL as I develop it helps somewhat but only when writing it – not when I decide to refactor it.

Read the rest of this entry »

Posted in Languages | Tagged: , , | Comments Off on Refactoring & Type Errors in Clojure: Experience and Prevention

Example: Functional Reactive Programming decisively beats Imperative on simplicity, length

Posted by Jakub Holý on June 17, 2015

@theburningmonk Yan Cui has a nice example demonstrating how Functional Reactive Programming [slides 185 – 206] (with Elm’s Signals) yields a much shorter and easier to understand (one you know FRP) code than an imperative code with mutations spread all over the code base.

The game

Use the Up and Down keys to move the platforms and thus bounce the ball from left to right and back:

screenshot-game

The imperative solution

Read the rest of this entry »

Posted in Languages | Tagged: , | Comments Off on Example: Functional Reactive Programming decisively beats Imperative on simplicity, length

A Usable Node.js REPL for Emacs

Posted by Jakub Holý on March 11, 2015

Being used to the excellent REPL in Clojure(Script), I was surprised to find out that Node.js REPL is somewhat weak and that its support in Emacs is not actively maintained. I anyway managed to get a usable REPL with these three components:

  1. The Emacs nodejs-repl package (nearly 2 years old)
  2. J. David Smith’s nodejs-repl-eval.el to be able to send code to the REPL (binding nodejs-repl-eval-dwim to C-x C-e so that I can execute the current sexp/region)
  3. My own extension of nodejs-repl-eval.el that takes care of escaping JS constructs that the REPL interprets in a special way

Regarding #3: The problem with the Node.js REPL is that valid JS code does not always behave correctly in the REPL. This is because: 1) _ is a special variable (the last result) while in code it is often used for the underscore/lodash library; 2) The REPL also interprets lines somewhat separately and tries to execute <dot><name> as a REPL command, breaking chained calls that start on a new line. My solution uses some RegExp magic to turn

var _ = require("lodash"); // #1a conflicting use of _
_.chain([1,2])             // #1b conflicting use of _
   .first()                // #2 interpreted as non-existing REPL command '.first'
   .value();

into

var __ = require("lodash");  // #1a Notice the doubled _
__.chain([1,2]).             // #1b Notice the doubled _
   first().                  // #2 Notice the dot has moved to the previous line
   value();

when the code is being sent to the REPL

Posted in Languages, Tools | Tagged: , , | Comments Off on A Usable Node.js REPL for Emacs

Running JavaScript Tests On a CI Server With Karma, Chrome And Fake X

Posted by Jakub Holý on January 13, 2015

So I want to run my JavaScript tests in a browser on our CI server. But the server has no graphical environment and the tests do not run under PhantomJS 1.x because it uses too old WebKit without ES5. The solution? Use a real browser and fake X via Xvfb. The browser I use is Chrome though Firefox would like work as well.

In code:
Read the rest of this entry »

Posted in Languages, Testing | Tagged: | Comments Off on Running JavaScript Tests On a CI Server With Karma, Chrome And Fake X

Notes From CodeMesh 2014

Posted by Jakub Holý on November 3, 2014

My consise highlights from CodeMesh 2014.

Philip Potter has very good CodeMesh notes as well, as usually.

TODO: Check out the papers mentioned in the  NoSQL is Dead talk. (<- slides)

Tutorial: QuickCheck (John Hughes)

General
  • QC => Less code, more bugs found
  • QC tests are based on models of the system under test – with some kind of a simple/simplified state, legal commands, their preconditions, postconditions, and how they impact the state. The model is typically much smaller and simpler than the imple code.
  • QuickCheck CI (quickcheck-ci.com) – free on-line service for running CI tests for a GitHub project. Pros: You don’t need QC/Erlang locally to play with it, it provides history of tests (so you never loose a failed test case), it shows test coverage also for failed tests so you see which code you can ignore when looking for the cause.
  • See John’s GitHub repo with examples – https://github.com/rjmh/
Shrinking (a.k.a. simplification)
  • Doesn’t just make the example shorter by leaving things out by tries a number of strategies to simplify the exmple, typically defined by the corresponding generators – f.ex. numbers are simplified to 0, lists to earlier elements (as in “(elements [3, 4, 5])”) etc.
  • You may implement your own shrinking strategies. Ex.: Replace a command with “sleep(some delay)” – so that we trigger errors due to timeouts. (A noop that just waits for a while is simpler than any op).
Bug discovery
  1. Run QC; assuming a test failed:
  2. Instead of diving into the implementation, use first QC to check your hypothesis of what constitutes “bad input” by excluding the presumed bad cases – f.ex. “it fails when input has 8 characters” => exclude tests with 8 and rerun, if you find new failures you know the hypothesis doesn’t cover all problems – and you will perhaps refine it to  “fails when it has a multiple of 8 chars etc. We thus learn more about the wrong behavior and its bounds. Assumption we want to verify: No (other) tests will fail.
  3. Do the opposite – focus on the problem, i.e. modify the test case to produce only “bad cases”. Assumption we want to verify: all tests will fail.
QC vs. example-based testing

QC code tends to be 3-6* smaller than implementation (partly thanks to the consiseness of Erlang) and fairly simple.

The case of Vovlo: 3k pages of specs, 20 kLOC QC, 1M LOC C implementations, found 200 bugs and 100 problems (contradictions, unclarities) in the specs. It took 2-3 years of working on it on and off.

Erlang dets storage race conditions: 100 LOC QC, 6 kLOC Erlang impl.

Testing stateful stuff

Invoke dfferent API calls (“commands”) until one of presumabely legal calls fails due to an accumulated corrupted state. This is an iterative process where we evolve our model of the system – commands, their preconditions (when they can be legally invoked), postconditions, and our repreentation of the state.

Ex.: Testing of a circular queue. Commads: push (legal on non-full queue), get (legal on non-empty), create new => generates sequences of new and pushs and gets.

Testing race conditions

Precondition: Run on a multicore PC or control the process scheduler.

  • There are many possible correct results (valid interleavings) of parallel actions => impractical to enumerate and thus to test with example-based tests
  • Correct result is such that we can order (interleave) the concurrently executed actions such that we get a sequential execution yielding the same result. F.ex. an incorrect implementation of a sequence number generator could return the same number to two concurrent calls – which is not possible if the calls were done sequentially.
Testing data structures

Map the DS to a simpler one and use that as the model – f.ex. a list for a tree (provided there is a to_list function for the tree).

Tutorial: Typed Clojure (Ambrose Bonnaire-Sergeant)

Note: The documentation (primarily introductory one) could be better

Resources
Defining types
  1. separately: (ann ..)
  2. around: wrap in (ann-form <defn> <type def.>)
  3. inside: use t/fn, t/defprotocol etc.
Gradual introduction of typed.clojure
  • wrap everything in (t/tc-ignore …)
  • for unchecked fns you depend on, add (ann ^:no-check somefn […])
  • If you stare at a type error, consider using contracts (prismatic/Schema or pre/post conds etc.)
Other

Keynote: Complexity (Jessica Kerr, Dan North)

  • Always have tasks of all three types: research (=> surface conplexity), kaizen (cont. improvement, improvement of the imprv. process), coding – these 3 interlave the whole time
  • A team needs skills in a number of areas, it isn’t just coding – evaluation of biz valuedelivered, monitoring, programming, testing, deployment, DB, FS, networks, … .

Keynote: Tiny (Chad Fowler)

Keep things tiny to be efficient (tiny changes, tiny teams, tiny projects, …).

  • Research by armies and in SW dev [TODO: find the two slides / qsm, Scrum] shows that teams of max 5-6 work best
    • Teams of 7+ take considerably more time and thus money (5* more wrt. one study) to complete the same thing
    • => small, autonomous teams with separate responsabilities (decomosition, SRP etc. FTW!)
  • Human capacity to deal with others is limited – one company creates a new department whenever size exceeds 100
  • Big projects fail; Standish CHAOS report – only cca 10% larger projects succeed compared to nearly 80% of small ones (summed together: 39% succeed)
  • Note: 1 month is not a short iteration

Distributed Programming (Reid Draper)

RPC is broken

– it tries to pretend a remote call is same as local but:

  • what if the call never returns?
  • the connection breaks? (has the code been executed or not yet?)
  • what about serialization of arguments (file handles, DB conn.,…)

It ignores the special character of a remote code and the 8 fallacies of distributed progr.

Message passing

is batter than RPC. There is also less coupling as the receiver itself decides what code to call for a specific message.

Bloom

The Bloom lang from the BOOM research project explores new, better ways of distributed programming. Currently implemented as a Ruby DSL.

From its page (highlight mine):

Traditional languages like Java and C are based on the von Neumann model, where a program counter steps through individual instructions in order. Distributed systems don’t work like that. Much of the pain in traditional distributed programming comes from this mismatch:  programmers are expected to bridge from an ordered programming model into a disordered reality that executes their code.  Bloom was designed to match–and exploit–the disorderly reality of distributed systems.  Bloom programmers write programs made up of unordered collections of statements, and are given constructs to impose order when needed.

Correctness testing of concurrent stuff
  • Unit testing unsuitable – there are just too many combinations of correct results and can only test the cases the dev can think of
  • => generate the tests – property-based testing / QuickCheck
  • PULSE – an addon to property-based testing that tries to trigger concurrency problems by using a scheduler that tries different interleavings of actions (randomly but repeatedly) [Erlang]
  • Simulation testnng – Simulant
Benchmarking

Beware the effects of GC, page cache, cronjob (e.g. concurrently running backup), SW updates => running a simple load test for few mins is not enough.

Cheats & Liars: The Martial Art of Protocol Design (Pieter Hintjens)

Pieter is the brain behind AMQP, ZeroMQ and EdgeNet (protocols for anonymous, secure, peer-to-peer internet). He has shared great insights into designing good protocols, the dirty business surrounding protocols and standardization, and troll-proof organization of communities (as a self-roganizing, distributed team).

More: See Ch.6 The ØMQ Community in the online The ZeroMQ Guide – for C Developers. (He has also other interesting online or paid books.)

  • Protocol is a contract for working together
  • IT should be minimalistic and specific, name the participants, …
  • Protocols and their strandardization are prey to “psychopatic” organizations that want to hijack them for their own profit (by pushing changes that benefit them, taking over the standardization process, …) (Pieter has experienced it e.g. with AMQP; these trolls always show up). It’s advantegous to take control of a successful protocol so that you can make money off it or build stuff on it and sell that. Examples:
    • Microsoft MS Doc XML  – this “open” spec f.ex. reportedly defines that one functions works “as Word 95”
    • A company pushing changes that nobody else really understands, thus undermining compatibility of implementations
    • Pushing such changes that an implementor can claim compliance to the standard yet implement it so that his products only work with his products
    • Crazy/proprieetary protocol extensions, patenting/trademarking/copyrighting the spec (e.g. TM on Bluetooth)
  • Hijacking-safe protocol creation process (beware “predatory maliciousness”):
    • The specs is GPL => nobody can capture it (e.g. ZeroMQ)
    • The community has clear rules and deals with trolls by kicking them out
    • There is a good process for evolving the spec
  • How to spec i protocol?
    • Start with a very, very small and simple contract – only add things that you desperately need – e.g. ZeroMQ v1 had no versioning, security, metadata (versioning added in v2, metadata in v3, ecurity later). You don’t know what you really need until you try it. F.ex. even the original AMQP has 60-75% waste in it!!!!
    • Do very slow and gradual evolution
    • Layering is crucial – keep your protocol on one layer, only specify relevant things, leave the other layers for other specs so they can evolve and age in different speed; the more in a spec the earlier will st. be outdated (Pizza contract says nothing about the kitchen, f.ex.)
  • Community and cooperation (See the Ch.6 The ØMQ Community mentioned above.)
    • community needs clear rules to keep trolls away (and they always pop up)
    • don’t just absorb the damage trolls do, ban them
    • self-org., decentralized team

PureScript (Bodil Stokke)

Working with larger JavaScript apps (> 100 LOC :-)) is painful, primarily due to the lack of type checking and thus requiring one to take lot of care when chaning code so that nothing breaks at runtime. TypeScript is a possible solution but it still feels limited.

PureScript is very Haskell-like language compiled to JS. It is a pure functional lang, effects are performed only via the Effect Monad (Eff). It is pragmatic w.r.t. interoperability – it is possible to embedd JS code and just add a signature to it, the compiler will trust it.

Moreover, you can use property-basd testing with QuickCheck and Functional Reactive Programming with Bodil’s Signal library. Isn’t that wonderful?!

See the PureScript Is Magic code at GitHub, primarily the 150 LOC Main.purs.

Category theory notes:

  • Semigroup is a domain with a cumulative operation (e.g. ints with +)
  • Monoid (?) is a semigroup with a unit element, i.e. one where “element operation unit = element” as 0 for + or 1 for *.

Megacore, Megafast, Megacool (Kevin Hammond)

Interesting research project ParaPhrase for parallelization of code through automatic refactoring and application of any of supported topologies (Farm, Pipeline, Map, …) – ParaPhrase-ict.eu and www.project-advance.eu (in general the promises of automatizatio regarding fixing software development problems have have hugely underdelivered but still something might be possible). In some cases their solution managed to do in hours what a developer did in days.

Quote Bob Harper:

The only thing that works for parallelism is functional programming

PS: C++17 is going to have support for parallel and concurrent programming.

 Categories for the Working Programmer (Jeremy Gibbons)

An elementary intro into category theory in 10 points, yet I got immediately lost. It might be worth to study the slides and look for more resources at the author’s uni page and not-so-active blog Patterns in Functional Programming .

 NoSQL is Dead (Eric Redmond)

Main message: There is just too many differences between NoSQL DBs for this expression to be meaningful.

Lobby talks

Hacking people

I had an inpiring lunch chat with Chad and a Polish lady whose name I unfortunately don’t know. Their companies do fascinating stuff to leverage the potential of their humans – one has replaced top-down management wrt. projects with environment where there are clear objectives (increase monthly active users) and the freedom to come with ideas to contribute to them, recruit other people for the idea and, if successful, go implement it (while continually measuring against the objectives). Clearly enough it is not easy, some people have troubles trying to manage everything or doing what they believe in without checking the real implect on the objectives etc. but it already provides more value then before. This has been going on for just a few months so hopefully when it settles more we will hear about their experience.

The other company realized that people are different (wow! how could our industry ignore that?!) and started to go psychological profiling of employees to understand what type of team member they are – a driver, a worker, a critic who is always hunting for possible issues and problems etc. And they compose teams so that they have the right mix of different personalities to avoid both insurpasable conflicts and the risks of group-think.

I believe this is the future of our industry – really understand people and “hack” our organizations to leverage that for greater happiness and better results.

Non-talks

  • Jessica Kerr: Simulation of team work and the effect of (no) slack

    – what happens when you let your programmers crunch work without any slack time? And when you introduce slack? Jessica has made this Scala simulation to produce the results we would expect – much more even production in the slack case, lot of rework after deploying features in the non-slack version. Not at all scientific but very nice when you want to *show* your higher-ups what happens when you do the former or the latter. Some people hear much more to a visual stimuli (even though totally made to conform to the message you want to get across) than tons of theory.

  • Aphyr – [313] Strong consistency models – “strong consistency” is a much broader term than I expected and not all consistency models are so consistent 🙂 Check out especially this consistency family tree image.

Posted in Languages, Testing, Tools | Tagged: , , | Comments Off on Notes From CodeMesh 2014

Most interesting links of July ’14

Posted by Jakub Holý on July 31, 2014

Recommended Readings

  • Video: The Unreasonable Effectiveness of Dynamic Typing for Practical Programs – a static-typing zealot turned friend of dynamic typing under the experience of real-world projects and problems shares thoughts about the limits of type systems (f.ex. both energy and torque are measured in N*m yet cannot be combined) and their cost: according to the Hanenberg’s experiment about static and dynamic typing => the time required to handle the time chacker > time to debug the errors that it would have caught. According to a review of issues at GitHub, only 2% of reported issues for JS, Clojure, Python, and Ruby are type errors and for a large, closed-source Python project type/name/attribute errors were 1%. “I have come to believe that tests are a much better investment [than static typing].” Rigorous type system/model => limited applicability (due to different needs) <=> modelling some things with types doesn’t cut it. “Are the costs of static typing offset by a few percent fewer defects? Is agility more important than reliability?” “Static types are anti-modular” – too a tight coupling. “Static type checking comes at the expense of complexity, brittleness and a tendency to monoliths.
    (Personally I miss static typing – but that is perhaps due to having relied on it for so long.)
  • ThoughtWorks Tech Radar July 2014 (pdf): f.ex. Ansible in Adapt, Masterless Chef/Puppet in Trial, Machine image as a build artifact: Trial, PostgreSQL for NoSQL: Trial, Adopt Dropwizard (Rest 4 Java), Go lang, Reactive Extensions across langs [JH: RxJava, RxJS, ..]; Asses Property-based (generative) testing, … . Other highlights: Mapbox (open-source mapping platform), OpenID Connect as a less complex and thus promising alternative to SAML/generic OAuth, Pacto/Pact for Consumer-Driven Contracts (contract => simulate consumers/stubb producers => test your REST clients against the contract so that the rest of tests can assume it is correct and use a stubbed client), Swagger for REST documentation.
  • The madness of layered architecture – a nice critique of over-designed “enterprise” apps, why that is a problem (SRP, cost of code, unclear where to do a change, ….), why it is different from the successful layered network stack of Ethernet/IP/TCP/… (because in an app, all layers are on the same level of abstraction); bottom line: do not add a layer unless you have a really good reason (hint: the advice of a consultant/speaker does not count as one)
  • Key Takeaway Points and Lessons Learned from QCon New York 2014 (viz @RiczWest) – “[..] deep insights into real-world architectures and state of the art software development practices, from a practioner’s perspective.” – architectures of Fb, Foursquare etc., continuous delivery, creating culture, real world functional programming, … .
  • Questioning the Lambda Architecture (J. Kreps of LinkedIn) – maintaining the same processing in two very different systems (one batch, one stream & real-time) is a maintenance nightmare => improve the RT/stream processing to handle re-processing and thus both (using e.g. Kafka to store the data and thus be able to re-play them)
  • Google: Checklist for mobile website improvement
  • Google Dataflow and the transition from batch to stream processing – G. Dataflow might not be a Hadoop killer due to requiring that the data are in the Google Cloud but the trend is clear, going away from batch processing to more stream-oriented processing with tools like Spark, Flume etc. that are faster thanks to using memory better and more flexible thanks to not being limited to the rigitd two-stage model of map-reduce. (Reportedly, Google – the one that made Map-Reduce popular – doesn’t use it anymore.)
  • OS X: Extract JDK to folder, without running installer

Society, economics, people

  • HBR: The Power of Meeting Your Employees’ Needs – people feel better, perform better, are more engaged and likely to stay longer (=> profitability) when 4 basic needs are met: physical [energy] renewal (=> give opportunity, encourage to take a nap or do whatever that helps), value – feeling of being valued by the company, ability to focus, purpose (i.e. serving something larger than ourselves). “What’s surprising about our survey’s results is how dramatically and positively getting these needs met is correlated with every variable that influences performance. It would be statistically significant if meeting a given need correlated with a rise of even one or two percentage points in a performance variable such as engagement, or retention. Instead, we found that meeting even one of the four core needs had a dramatic impact on every performance variable we studied. [..] when all four needs are met, the effect on engagement rises from 50% for one need, to 125%. Engagement, in turn, has been positively correlated with profitability. [..] employers with the most engaged employees were 22% more profitable than those with the least engaged employees.
    [..] those who were encouraged to take intermittent breaks reported they were 50% more engaged, more than twice as likely to stay with the company, and twice as healthy overall. Valuing and encouraging renewal requires no financial investment. What it does require is a willingness among leaders to test their longstanding assumption that that performance is best measured by the number of hours employees puts in – and the more continuous the better — rather than by the value they generate, however they choose to do their work.
  • The Pitchforks Are Coming… For Us Plutocrats – increasing inequality will eventually lead to the collapse of the sysem (at least so does teach the history). It is people – primarily the middle class – that are the source of the wealth of the society, they produce and also consume most. Thus it is necessary to support them …
  • Why the U.S. Corporate World Became ‘A Bull Market for Corruption’ – Enron, GM, Goldman Sachs, … – we hear more and more the names of large corporations in the context of negligence and misues of their customers and investors. It seems that leadership (in the lead by example sense) has died out as well as the feeling of responsibility when one wields power over her customers/investors/markets. Instead, we have the me-first and  money at any cost thinking. Organizations are designed to shield higher-ups from responsibility (meetings with no records…). High pay for short term gains, failure to punish high ranking people.
  • (US) This is what happened when I drove my Mercedes to pick up food stamps – the experience of life in poverty after dropping down from $125k to $25k/year in two months due to childbirth, real estate market crash, and loss of a job. “Using the coupons was even worse. The stares, the faux concern, the pity, the outrage — I hated it. [..] That’s the funny thing about being poor. Everyone has an opinion on it, and everyone feels entitled to share. [..] Poverty is a circumstance, not a value judgment. I still have to remind myself sometimes that I was my harshest critic. That the judgment of the disadvantaged comes not just from conservative politicians and Internet trolls. It came from me, even as I was living it.

Clojure Corner

  • Isomorphic Clojure[Script], part I – enjoying all the benefits of Single-Page Apps while avoiding their drawbacks (SEO, slower page load, accessibility etc.) – a SPA that can be pre-rendered by the server. Using Om/React, JDK8 with the Nashorn JS engine, core.async, Sente (bi-dirrectional HTTP/WS communication over core.async) and Clojure in the JVM, ClojureScript in Nashorn in the JVM, and ClojureScript in the browser. Example app: Omelette.
  • clj-crud: a relatively feature-complete example of a Clojure web (4/2014; GitHub) – using Component, Liberator (REST), Datascript + Quiescent (=> React.js), Enlive, Friend etc. including couple of unit-test and ui-test libraries
  • Conclujon: Acceptance testing tool (α), Clojure reimplementation of Concordion, a beautifully simple ADD tool
  • dynalint: human-friendly error messages during dev – Clojure typically provides little helpful and sometimes confusing error messages thrown from somewhere deep in the implementation, such as “Don’t know how to create ISeq from: java.lang.Long at clojure.lang.RT.seqFrom” while we want st. like “First argument to clojure.core/first must be seqable: 1 (instance of class java.lang.Long” – and that’s what Dynalint does. In the tradition of defensive programming, it adds checks and good error messages to Vars at runtime. You typically run it only during dev, triggering it from the REPL.
  • Grimoire (Reid McKenzie) – a more up-to-date replacement for ClojureDocs
  • Adam Bard’s Top Clojure Articles for beginners and intermediate Clojure devs – f.ex. Five Mistakes Clojure Newbies Make, Acceptable Error Handling in Clojure, Clojure Reducers for Mortals
  • J. Wilk: Isolating External Dependencies in Clojure – a nice overview of the options and their pros and cons – with-redefs, alter-var-root, Midje (using alter-var-root in a more controlled manner), higher-order-functions (#1!) etc.
  • philandstuff’s detailed notes from Euroclojure 2014

Tools/Libs

  • NixOS (via @bodil) – a new interesting “purely functional” Linux distribution – system configuration is fully declarative (think of Puppet/Chef) and it is always trivial to roll back, you can have multiple versions of a package, users can install non-global SW
  • InfluxDB – time series, metrics, and events DB that scales; contrary to Graphite it can store richer data than Graphite and its single value; additional highlights: authorization for individual data, roll-up/clean old data, https API. Written in Go.

Posted in General, Languages, Top links of month | Tagged: , , , , , , , , , | Comments Off on Most interesting links of July ’14