The Holy Java

Building the right thing, building it right, fast

Archive for the ‘General’ Category

NDC Oslo 2015: Talk notes, recommended talks (security, FP, etc.)

Posted by Jakub Holý on June 19, 2015

A great conference. A good deal of good talks.

To check later

Wednesday

Thursday

Friday

Keynote Data and Goliath ☆☆☆☆

Inspiring, slightly eye-opening!

Key points: We live in a society of ubiqutious surveillance. We need to share data to get great services yet we need to ensure personal privacy and security – finding the right balance will be a major dialog in the near future. Currently we share too much, too freely. Even high-tech, presumabely super secret surveillance tools and SW had their cousins presented at security conferences => anybody can build these, and many do. Either everybody gets to spy – or nobody does => build security in.

Check out Schneier’s book Data and Goliath and blog.

Lean and FP ☆☆

Nice intro to Lean and FP, some nice points (e.g. why do companies fail in adopting FP). A little new stuff for me; it wasn’t a waste of time but perhaps I could have seen a more valuable talk.

A tour of the language landscape ☆☆

It’s interesting to learn about various cool capabilities of other languages. But it is more useful as an inspiration, there is little you can start using in your daily work. The slides are worth going through.

Covered:

  • F#: Pipes (~ Clojure threading macro), Type Providers (e.g. get autocompletion, compile-time checks for browsing your Amazon S3 bucket), Unit-of-Measure (=> 2 + 2 fails), Statically Resolved Type Parameters
  • Clojure: Macros
  • Go: Implicit Interface Implementation (define an interface with a method; everything having the method autom. implements it, no need to tag those (which would be impossible for 3rd party code) => type safety and convenience)
  • Rust: Borrowed Pointers (autom. memory management without GC overhead – a piece of code owns a pointer => destroyed when the scope is left but can “borrow” it to code (of not longer scope) that it calls)
  • Erlang: Actor Model, Bit Syntax
  • Idris: Dependent Types, Uniqueness Types
  • Elm: Signals (reactive programming (Rx))

Also: Example: Functional Reactive Programming decisively beats Imperative on simplicity, length

Async and Streaming JavaScript, We’re All Doing it Wrong! ☆☆☆

Useful. Highlights limitations of / issues with Promises and (Node, WHATWG Streams). Demonstrates how Reactive Programming (e.g. RxJS) is a superior solution. About the coming inclusion of (some of) Rx in JS.

See NDC: Async and Streaming JavaScript, We’re All Doing it Wrong! (Promises, Streams, Rx)

Learning from Haskell ☆☆☆

Venkat Subramaniam is a skilled speaker. This was fun and enriching look at what we can learn from Haskell and hopefully use in our daily programming even if we use C#.

See list of the lessons learned.

50 Shades of AppSec ☆☆☆

Fun, scary. Examples of some hacks, surprisingly terrible security at many places, demo of tools that hack a site for you. Recommended as an eye opener to the importance of security.

Included:

  • Bishop:
    > Bishop is a vulnerability scanner that searches websites in the background while you browse, looking for exposed version control systems, misconfigured administrative tools, and more. With a whitelisting regex system, you can easily restrict this tool to hosts that you are authorized to scan.
  • Havij: Find & elverage SQL injection vulnerabilities automatically
  • Hire a hacker on-line: https://hackerslist.com/

Idioms for building distributed fault-tolerant applications w/ Elixir ☆☆

A light introduction to the Erlang VM and Elixir.

FP:

  • Explicit over implicit state
  • Transformation over mutation

Elixir is a distributed > concurrent > FP lang (in the order of importance).

Elixir goals: Compatibility, Extensibility => macros, Productivity.

  • Productivity: 1st class docs, tooling (ExUnit, IEx, Mix), hex packages

Demo: Agents, pattern matching, tools, supervision, streams (=> lazy computations), … .

Event Sourcing and DDD with F# ☆

After a brief introduction of E.S. (without going into why/when), the speaker presented E.S. as two functions (decide, evolve) and went on implementing a simple card game in F# using that. Cool: Own given-when-then test “framework” in just few lines of code. The talk concluded with demonstration of plugging in an actual Event Store server and load + save functions and using an Agent to hold the current state.

Command + [state]: (decide) -> event(s) -> store them, evolve the state
^--------------------------------------------/

decide: fn(state, command) -> Events
evolve: fn(initial state, state, Event) -> state

A little too low-level both on F# and even sourcing but OK to follow on the some-understanding level. My main lesson learned is that F# looks nice, concise, powerful. And FP is certainly a good match for E.S.

Phoenix – a framework for the modern web ☆☆

Elixir with Phoenix (and Plug) => Performance and Productivity.

The most frequent quote:

It’s just a function call

  • Request lifecycle: Endpoints -> Routers -> Controllers

Productivity

  • Middleware = just functions => compositions
  • Helpful errors (and web view); “if you can’t understand an error, it’s a bug”
  • Just mix in the comments generator to enable comments in the page and add the
    generated controller to your pipeline
  • mix ecto.migrate => data migrations
  • static assets with Brunch (Node; combine, minify, watch for changes, …)
  • ES6 transpiler included
  • live reload (html, css, ..)

Performance

  • Under-1ms response times
  • Views and templates: precompiled, fast, function calls. Views render templates (embedded elixir, HAML, or another). Thx to metaprogramming, render becomes just string concat. at runtime.
  • Ex.: Phoenix on RaspberryPi (16MB mem, 540req/s serving a simple HTML)
  • Robust concurrency model – extremely lightweight processes ( no global pauses; load balances on IO and CPU
  • Pattern-matched route dispatch, the VM is very fast at it (Note: Convenience fn to get route for a controller/… => metaprogramming posibilities)
  • Channels (Pub/Sub): trivial realtime communication (for connected devices) – WebSockets with minimal code Clients for the browser (fall-back to long-polling), for Objective-C, eventually Android. Embedded devices: CoAP instead of WS. Can use Redis, … . Synchronous0like communication possible; JS ex.: chan.push(..).receive("ok", okFn).receive("error", errFn).
  • Distributed cluster support (that’s what Erlang always did with multiple communicating switches)
  • Everything is a separate process => one channel crashing cannot harm other channels.

Ex.: Chat.Endpoint.broadcast("rooms:lobby", "new_msg", %{username: "me", body: "hello"}) => all connected clients receive it, no matter whether local or remote.

  • Erlang: Erlang handles 1/2 of the world’s telecom traffic. Reported 99.999999999% availability.
  • WhatsApp: 2M conn/server; 400M users, 30 engineers
  • Update your code while the app is running! Apps that run forever.

Awesome Elixir: A curated list of amazingly awesome Elixir and Erlang libraries, resources etc.

The Power and Practicalities of Immutability ☆☆☆☆

In F#, mutable is the keyword of shame.

Immutability (assignment-less programming where mutation is replaced by transformation) doesn’t remove mutation it just pushes it to lower-level code – the same we did with goto, which disappeared from our code but still is in the bytecode/asm/.. .

Assignment is to Functional Programming as goto is to Structural Programming.

How to program w/o assignments:

  • Recursion; cons:
  • It can blow up the stack (tail-cole optim.)
  • Some problems do not fit it naturally
  • Move mutation to lower level => build from a higher-level, safe building blocks;
    Ex.: for loop -> list.Select(e => e*2)

The problems

  • Mutability
  • Hard to reason about the code (especially if multiple variables)
  • Difficult to manage state transitions; concurrency difficult;
  • Imperative progr. (e.g. a for-loop)
  • I need to say both what and how to do (like a toddler – fun on the 1st day, not the other 18 years :-))
  • Mutating variables all the time

Benefits of immutability (and pure functions)

  • easy to reason, understand, explain, test
  • memoization
  • referential transparency (it’s possible to replace a call with its result) => optimizations (exec. reordering, running concurrently, memoization, ..)
  • immutability enables laziness (e.x: {1,2,3,4}.Where(e => e > 3).Where(e => e % 2 == 0).Select(double).firstOrDefault() – all the checks and the doubling are only executed 0-3 times)
  • immutability makes parallelization affordable

Notes

  • Examples in C#, F#
  • Ex.: sum, double a list, …
  • “Encapsulation” = “I don’t care” :)

How to turn software into competitive business advantage ☆☆☆☆

Gojko Adzic is my favourite speaker, with many good thoughts (and experiences).

Continuous delivery has huge side-effects. Exploit them and open new business opportunities.

Fast is good when there is human control over it – when business people have no say in what gets updated when, they loose control (and are unhappy).

  • Can we control C.D. or are we loosing control of it?
  • Ex.: Vehicle charger => possible fire: for Tesla it was just an over-night automatic software update contrary to GM that had to recall the cars.
  • Marketing side-effect of C.D.: No more big releases to have press conference about
  • C.D. is important for devs but we don’t want to piss off users
  • UX: CD (i.e. partial delivery) => user confusion

Gojko’s 3 rules

C.D. nay not …

  1. confuse users
  2. interrupt users’ work of sessions
  3. disrupt or prevent marketing initiatives

Deployment != Release

  • Release is a marketing event, biz decides.
  • We need to support running multiple versions of the SW in production (so current users are not interrupted, biz may decide when to switch over)
  • Problem: data. Ex. solution: add data (format) version field
  • Gentle deployment: Release a new version in || to the old one, enable (and motivate) users to switch at will while preserving the ability to go back (if they encounter any issues …)

Continuous Delivery without multi-versioning is irresponsible!

2 things to remember:

  • Decouple deployment and release
  • Give 2% of users a 100% solution, do not force-feed 100% users with 2% of scope

Modern architectural patterns for the cloud

Perhaps not a bad overview of various patterns (DB replication, sharding; CQRS, event sourcing, map reduce). Too little knew at start and then I lost focus :)


Get Unblocked (unleash your creativity) ☆☆☆

Fun, inspiring. Good tips for overcoming our internal barriers to creativity (and a happier life). Slides. Book: Banish Your Inner Critic.

The entire industry is broken – managers are from Mars, developers from Venuse. Micromanagement, open spaces / cubicles, time pressure, demanding clients, bureaucracy, … .

  • Face and ID your internal critic
  • Impostor syndrom (“I don’t deserve this, I am not good enough for it”)
  • Don’t compare yourself to other people
  • Perfectionism (focus on the product rather than the creative process)
    (Pottery exp.: 1/2 tries to make the best pot, 1/2 as many as possible =>
    the latter had more and many really good.)
  • Procrastination (feeds Perf. and v.v.)
  • Banish slave words from your vocabulary – Eliminate should/must/ought to/…
  • “Failure is only the opportunity to begin again more intelligently.” H. Ford
  • Stressed? Make it really bad on purpose => overcome the block of trying to be perfect.
  • Get into the Alpha brain waves mode (more relaxed than the normal beta)
  • Breath to re-focus
  • Lay down
  • Daydream / space out
  • Take shower
  • Get physical – movement allows the hemispheres to sync, …
  • Exercise discipline
  • Manage time (if stressed => no place for creativity)
  • Make “To DON’T” lists – what not to do (stop, delegate, …)
  • Say NO to distractions (see http://theheadphonesrule.com/)
  • Apps: Consider the RescueTime app => increase awarness; Mac: HeyFocus.com when on, stops you from going to distractive pages and apps; GetConcentrating.com; browser extensions, .. – see the slide
  • Single-task (instead of multi-tasking) – e.g. the Pomodoro Technique; especially useful when you do something you don’t like to do (during the 5 min break activate via a walk, laying down, having a look at the sky, …)
  • Do it with others – we come up with better / different things when working with others. Avoid sameness! => diverse environments. Share your ideas. Share mistakes. Amplify the ideas of others – try to help them to build them. Make your partned look good.
  • “Yes, and ..” technique (useful for meetings!) – don’t shut down others’ ideas – “yes, we can do this – and we can do this other thing”

Fear:

F.alse
E.vidence
A.ppearing
R.eal

(Or Fuck Everything And Run)

Watch:

A security testers toolkit ☆☆☆☆

Awesome! Scary! You have to see this! This is a real eye opener, demonstrating how easy hacking can be (especially with cooperating users via little social engineering).

You can only fight the way you practice – Miyamoto Musashi

Tools

  • Kali Linux
  • nmap (network mapper)
  • Veil fwrk – avoid anti-virus
  • Metasploit
  • BeEF – The Browser Exploitation Framework

Vulnerabilities

Kali Linux

  • Kali Linux VMs
  • Many hacking tools
  • Demo (see exploiting vsftpd):
  • Run against Metasploitable is an intentionally vulnerable Linux virtual machine
  • Zenmap: GUI for nmap
  • Metasploit – provides an interactive shell; need to know the exploit: use exploit/unix/ftp/vsftpd_234_backdoor; set RHOST ; exploit
  • Wifi hacking
  • Use a wifi card enabling monitoring, compatible with Kali – e.g. one of the Alpha cards
  • Aircrack-ng – WPA-PSK keys cracking program
  • airmon-ng => start the device in monitor mode, list devices around, …
  • WPS (Wi-Fi Protected Setup) is very crackable
  • PwnSTAR – soft access point – set up an AP, nmap scan clients, …
  • Armitage – GUI – if nmap, metasploit complicated, this is a nice, visual representation of the network and vulnerabilities on it; scan net -> scan a host -> “Attacks – Find Attacks” (Hail Mary – lunches 100s of exploits, not stealthy at all). Also a paid version Cobalt Strike, more powerful.
  • BeEF – hook into a website, take over a browser => detect extensions, send commands to them, metasploit, social eng. attacks, … Send fake flash/java/last pass update, petty theft – “facebook session timed out” popup (e.g. on any site with Fb integration), a Clippy help offer, … .
  • Veil – SW avoidance toolkit – generate payloads against machines, take them over – create f.ex. a “call home” .exe (connects to your metasploit server later command the app and take over the PC) (call it e.g. LastPassUpdate.exe, FlashUpdate.exe, …)
  • load espia; screengrab; keyscan_start; keyscan_dump; getsystem => elevate to local admin; load incognito; add_user XYZ myPsw; add_localgroup_user Administrators XYZ

Framework for Man-In-The-Middle attacks – => Bypassing HSTS (HTTP Strict Transport Security) with MITMf – downgrade the connection to HTTP
* The PC with MITMf makes itself to be the gateway for the victim PC
* Tip: http://wwww.google.com – can YOU spot the 4th w?; skandiabanken.no => webskandiabanken.no
* Only works if the user hasn’t accessed the page before in this browser, cookies and HSTS list are clear (wifi coffees & online banking = bad!)

Going beyond OWASP ☆☆

Walk through some past vulnerabilities in .NET. Some really nice hacks. Likely worth watching.

Few interesting examples:

  • HashDoS attack due to consistent hashing, putting all form fields starting with the same latter to the same bucket (common in other languages too)
  • When encrypting, also sign it so that when you get an encrypted stuff back, you’re sure you did indeed produce it (there was an endpoint allowing to check whether st. is properly encrypted => play with bytes and high math to encrypt st. w/o the key)
  • Misusing REST – constructing URLs from user input => make sure the input does not contain path (e.g. user = “./admin”), query strings, # chars, anything else affecting the path – basically 0-9a-Z
  • in Turkey, “interesting”.ToUpper() != “INTERESTING”; ppl make security decisions based on string comparison (.NET: string.Compare(a,b, ))
  • Unicode has “full width” chars: < (<); when put into SQL varchar, it becomes normal ” don’t use non-unicode columns to store Unicode
  • Common cryptographic mistakes: e.g. key / init.vector reuse (=> larger attack surface). Writing own crypto.
  • Hashing concat. strings – but hash(builtin+securely) == hash(built+insecurely) => do len(builtin) + hash(builtin) + len(securely) + hash(securely)
  • Cert pining – do; checking the thumbprint is fragile, check the signing chain, subject name too
  • Time-incons. hashing => timing attacks => don’t do early exit (when (mis)match found)
  • MVC in web apps with auto-binding data to objects assign even properties that normally are not in the form (but added by a hacker), e.g. ID – “over-binding”
  • Don’t trust file names from uploads
  • Zip bombs – tiny file which recursively uncompress; e.g. 42.zip (42kB) => 4.5 petabytes
  • XML bombs – e.g. Billion Laughs; preventable by turning of DTD Parsing

Making Hacking Child’s Play ☆

A fun talk though it contained only little new after being to Troy’s workshop and the other talk.

Knowledge is power! The guide to measure what matters. ☆

Essentially an intro into StatsD. I had to leave early.

Posted in General | Leave a Comment »

Notes from Troy Hunt’s Hack Yourself First workshop

Posted by Jakub Holý on June 17, 2015

Troy Hunt (@troyhunt, blog) had a great, very hands-on 2-day workshop about webapp security at NDC Oslo. Here are my notes.

Highlights – resources

Personal security and privacy

Site security

Breaches etc.

To follow

Read the rest of this entry »

Posted in General | Tagged: , | 1 Comment »

Mounting an EBS volume to Docker on AWS Elastic Beanstal

Posted by Jakub Holý on June 2, 2015

Mounting an EBS volume to a Docker instance running on Amazon Elastic Beanstalk (EB) is surprisingly tricky. The good news is that it is possible.

I will describe how to automatically create and mount a new EBS volume (optionally based on a snapshot). If you would prefer to mount a specific, existing EBS volume, you should check out leg100’s docker-ebs-attach (using AWS API to mount the volume) that you can use either in a multi-container setup or just include the relevant parts in your own Dockerfile.

The problem with EBS volumes is that, if I am correct, a volume can only be mounted to a single EC2 instance – and thus doesn’t play well with EB’s autoscaling. That is why EB supports only creating and mounting a fresh volume for each instance.

Read the rest of this entry »

Posted in General | Tagged: , , , | Comments Off on Mounting an EBS volume to Docker on AWS Elastic Beanstal

OS X: Using scutils to discover whether/what a web proxy is in use

Posted by Jakub Holý on May 7, 2015

When looking for ways to discover whether a proxy is being used by OS X, you will be typically pointed to

networksetup -getwebproxy

However that does not always work – for example when using “Auto Proxy Discovery” and/or “Automatic Proxy Configuration” with a proxy.pac file. scutils --proxy seems to detect all these cases (though it cannot give you the proxy when using auto config, I suppose):
Read the rest of this entry »

Posted in General | Tagged: , , | Comments Off on OS X: Using scutils to discover whether/what a web proxy is in use

All-in-one Docker with Grafana, InfluxDB, and cloudwatch-to-graphite for AWS/Beanstalk monitoring

Posted by Jakub Holý on May 7, 2015

I have derived the Docker container docker-grafana-influxdb-cloudwatch that bundles Grafana dashboards, InfluxDB for metrics storage, and runs cloudwatch-to-graphite as a cron job to fetch selected metrics from AWS CloudWatch and feed them into the InfluxDB using its Graphite input plugin. It is configured so that you can run it in AWS Elastic Beanstalk (the main problem being that only a single port can be exposed – I therefore use Nginx to expose the InfluxDB API needed by Grafana at :80/db/).

Read the rest of this entry »

Posted in General | Tagged: , , , | Comments Off on All-in-one Docker with Grafana, InfluxDB, and cloudwatch-to-graphite for AWS/Beanstalk monitoring

My Highlights from Continuous Delivery and DevOps Conference 2015

Posted by Jakub Holý on April 30, 2015

The first Continuous Delivery and DevOps Conference in Oslo is over. It was nice to see so many people interested in the topic. I would have preferred more practical talks of the “how we did it” type over the “why” type but it was OK, though next year I would prefer flatMap. Here are my highlights:

  • Atmel is using a physical robot to plug and connect a particular configuration of circuit boards to test; your automated testing challenges cannot be greater than theirs!
  • Continuous Delivery decreases the risk of outage and time-to-recovery while enabling faster innovation, correlates with higher profits; No efficiency improvement will outperform cycle time reduction
  • Estimation pathologies; focus on value rather than costs
  • Stop talking about requirements, they are fake; they’re just beliefs about what may add value to customers. Use hypothesis instead!
  • Cisco: Most of the tools increasing productivity (and some innovation) were produced by engineers in their “spare” time; slack time is thus crucial
  • How does Cisco grow professionalism : optimise for the 10% best, not the 10% weakest developers; slack time; make everything visible; encourage code reviews but avoid making them mandatory; see the slide
  • CALMS: Culture, Automation, Lean, Measurement, Sharing. The pillars of devOps
  • Cisco invested a lot in crafting their build system, tailored test frameworks, and emulators to be able to get quick and quality feedback – because it pays off
    • “Make you own build system” says @olvemaudal at @CoDeOSL. IME this is inevitable for non-trivial projects, and a good investment.
  • Unleash: Feature Toggles server and Java/Node client by FINN.no
  • “They asked for a report while they actually need just a list of data, the result of a simple SQL query; have we listened to them, we would have wasted hours creating a report in the report framework with logos and all the crap.”

Slides:

Posted in General | Tagged: , | Comments Off on My Highlights from Continuous Delivery and DevOps Conference 2015

AWS CloudWatch Alarms Too Noisy Due To Ignoring Missing Data in Averages

Posted by Jakub Holý on March 31, 2015

I want to know when our app starts getting slower so I sat up an alarm on the Latency metric of our ELB. According to the AWS Console, “This alarm will trigger when the blue line [average latency over the period of 15 min] goes above the red line [2 sec] for a duration of 45 minutes.” (I.e. it triggers if Latency > 2 for 3 consecutive period(s).) This is exactly what I need – except that it is a lie.

This night I got 8 alarm/ok notifications even though the average latency has never been over 2 sec for 45 minutes. The problem is that CloudWatch ignores null/missing data. So if you have a slow request at 3am and no other request comes until 4am, it will look at [slow, null, null, null] and trigger the alarm.

So I want to configure it to treat null as 0 and preferably to ignore latency if it only affected a single user. But there is no way to do this in CloudWatch.

Solution: I will likely need to run my own job that will read the metrics and produce a normalized, reasonable metric – replacing null / missing data with 0 and weight the average latency by the number of users in the period.

Posted in General, Tools | Tagged: , , | Comments Off on AWS CloudWatch Alarms Too Noisy Due To Ignoring Missing Data in Averages

Book Review & Digest: Capital In The Twenty-First Century

Posted by Jakub Holý on November 21, 2014

My key takeaways and highlights from the book. This is not an objective and consistent review.

Capital in the Twenty-First Century by Thomas Piketty is together with Thinking, Fast and Slow, the most important book on society I’ve ever read. And together with Rawls’ A Theory of Justice it has shaped my opinions on society and justice. All politicians and good people interested in politics and inequality should study it. It is unique in leveraging over 200 hundred years of data from different countries and using those as the base of the discussion of the evolution, laws, and future of capital and (in)justice in its distribution.

Key points:

  • Growth cannot be 4-5% forever. In the long term, 1-1.5% is more realstic (and still far more than in the past). Higher growth is typical of countries catching up to more advanced economies (such as Europe to US, UK after WW2, China to the West nowadays).
  • Return on capital, typically 4-5% (before taxes), is thus far higher than the growth of economy and salaries. The result is that the rich get ever richer, taking ever more of the national income. (Consequently, the poorer half has ever less of it.)
  • To keep democracy and have a stable society, this has to be adressed politically.

Read the rest of this entry »

Posted in General | Tagged: , , , | Comments Off on Book Review & Digest: Capital In The Twenty-First Century

Most interesting links of July ’14

Posted by Jakub Holý on July 31, 2014

Recommended Readings

  • Video: The Unreasonable Effectiveness of Dynamic Typing for Practical Programs – a static-typing zealot turned friend of dynamic typing under the experience of real-world projects and problems shares thoughts about the limits of type systems (f.ex. both energy and torque are measured in N*m yet cannot be combined) and their cost: according to the Hanenberg’s experiment about static and dynamic typing => the time required to handle the time chacker > time to debug the errors that it would have caught. According to a review of issues at GitHub, only 2% of reported issues for JS, Clojure, Python, and Ruby are type errors and for a large, closed-source Python project type/name/attribute errors were 1%. “I have come to believe that tests are a much better investment [than static typing].” Rigorous type system/model => limited applicability (due to different needs) <=> modelling some things with types doesn’t cut it. “Are the costs of static typing offset by a few percent fewer defects? Is agility more important than reliability?” “Static types are anti-modular” – too a tight coupling. “Static type checking comes at the expense of complexity, brittleness and a tendency to monoliths.
    (Personally I miss static typing – but that is perhaps due to having relied on it for so long.)
  • ThoughtWorks Tech Radar July 2014 (pdf): f.ex. Ansible in Adapt, Masterless Chef/Puppet in Trial, Machine image as a build artifact: Trial, PostgreSQL for NoSQL: Trial, Adopt Dropwizard (Rest 4 Java), Go lang, Reactive Extensions across langs [JH: RxJava, RxJS, ..]; Asses Property-based (generative) testing, … . Other highlights: Mapbox (open-source mapping platform), OpenID Connect as a less complex and thus promising alternative to SAML/generic OAuth, Pacto/Pact for Consumer-Driven Contracts (contract => simulate consumers/stubb producers => test your REST clients against the contract so that the rest of tests can assume it is correct and use a stubbed client), Swagger for REST documentation.
  • The madness of layered architecture – a nice critique of over-designed “enterprise” apps, why that is a problem (SRP, cost of code, unclear where to do a change, ….), why it is different from the successful layered network stack of Ethernet/IP/TCP/… (because in an app, all layers are on the same level of abstraction); bottom line: do not add a layer unless you have a really good reason (hint: the advice of a consultant/speaker does not count as one)
  • Key Takeaway Points and Lessons Learned from QCon New York 2014 (viz @RiczWest) – “[..] deep insights into real-world architectures and state of the art software development practices, from a practioner’s perspective.” – architectures of Fb, Foursquare etc., continuous delivery, creating culture, real world functional programming, … .
  • Questioning the Lambda Architecture (J. Kreps of LinkedIn) – maintaining the same processing in two very different systems (one batch, one stream & real-time) is a maintenance nightmare => improve the RT/stream processing to handle re-processing and thus both (using e.g. Kafka to store the data and thus be able to re-play them)
  • Google: Checklist for mobile website improvement
  • Google Dataflow and the transition from batch to stream processing – G. Dataflow might not be a Hadoop killer due to requiring that the data are in the Google Cloud but the trend is clear, going away from batch processing to more stream-oriented processing with tools like Spark, Flume etc. that are faster thanks to using memory better and more flexible thanks to not being limited to the rigitd two-stage model of map-reduce. (Reportedly, Google – the one that made Map-Reduce popular – doesn’t use it anymore.)
  • OS X: Extract JDK to folder, without running installer

Society, economics, people

  • HBR: The Power of Meeting Your Employees’ Needs – people feel better, perform better, are more engaged and likely to stay longer (=> profitability) when 4 basic needs are met: physical [energy] renewal (=> give opportunity, encourage to take a nap or do whatever that helps), value – feeling of being valued by the company, ability to focus, purpose (i.e. serving something larger than ourselves). “What’s surprising about our survey’s results is how dramatically and positively getting these needs met is correlated with every variable that influences performance. It would be statistically significant if meeting a given need correlated with a rise of even one or two percentage points in a performance variable such as engagement, or retention. Instead, we found that meeting even one of the four core needs had a dramatic impact on every performance variable we studied. [..] when all four needs are met, the effect on engagement rises from 50% for one need, to 125%. Engagement, in turn, has been positively correlated with profitability. [..] employers with the most engaged employees were 22% more profitable than those with the least engaged employees.
    [..] those who were encouraged to take intermittent breaks reported they were 50% more engaged, more than twice as likely to stay with the company, and twice as healthy overall. Valuing and encouraging renewal requires no financial investment. What it does require is a willingness among leaders to test their longstanding assumption that that performance is best measured by the number of hours employees puts in – and the more continuous the better — rather than by the value they generate, however they choose to do their work.
  • The Pitchforks Are Coming… For Us Plutocrats – increasing inequality will eventually lead to the collapse of the sysem (at least so does teach the history). It is people – primarily the middle class – that are the source of the wealth of the society, they produce and also consume most. Thus it is necessary to support them …
  • Why the U.S. Corporate World Became ‘A Bull Market for Corruption’ – Enron, GM, Goldman Sachs, … – we hear more and more the names of large corporations in the context of negligence and misues of their customers and investors. It seems that leadership (in the lead by example sense) has died out as well as the feeling of responsibility when one wields power over her customers/investors/markets. Instead, we have the me-first and  money at any cost thinking. Organizations are designed to shield higher-ups from responsibility (meetings with no records…). High pay for short term gains, failure to punish high ranking people.
  • (US) This is what happened when I drove my Mercedes to pick up food stamps – the experience of life in poverty after dropping down from $125k to $25k/year in two months due to childbirth, real estate market crash, and loss of a job. “Using the coupons was even worse. The stares, the faux concern, the pity, the outrage — I hated it. [..] That’s the funny thing about being poor. Everyone has an opinion on it, and everyone feels entitled to share. [..] Poverty is a circumstance, not a value judgment. I still have to remind myself sometimes that I was my harshest critic. That the judgment of the disadvantaged comes not just from conservative politicians and Internet trolls. It came from me, even as I was living it.

Clojure Corner

  • Isomorphic Clojure[Script], part I – enjoying all the benefits of Single-Page Apps while avoiding their drawbacks (SEO, slower page load, accessibility etc.) – a SPA that can be pre-rendered by the server. Using Om/React, JDK8 with the Nashorn JS engine, core.async, Sente (bi-dirrectional HTTP/WS communication over core.async) and Clojure in the JVM, ClojureScript in Nashorn in the JVM, and ClojureScript in the browser. Example app: Omelette.
  • clj-crud: a relatively feature-complete example of a Clojure web (4/2014; GitHub) – using Component, Liberator (REST), Datascript + Quiescent (=> React.js), Enlive, Friend etc. including couple of unit-test and ui-test libraries
  • Conclujon: Acceptance testing tool (α), Clojure reimplementation of Concordion, a beautifully simple ADD tool
  • dynalint: human-friendly error messages during dev – Clojure typically provides little helpful and sometimes confusing error messages thrown from somewhere deep in the implementation, such as “Don’t know how to create ISeq from: java.lang.Long at clojure.lang.RT.seqFrom” while we want st. like “First argument to clojure.core/first must be seqable: 1 (instance of class java.lang.Long” – and that’s what Dynalint does. In the tradition of defensive programming, it adds checks and good error messages to Vars at runtime. You typically run it only during dev, triggering it from the REPL.
  • Grimoire (Reid McKenzie) – a more up-to-date replacement for ClojureDocs
  • Adam Bard’s Top Clojure Articles for beginners and intermediate Clojure devs – f.ex. Five Mistakes Clojure Newbies Make, Acceptable Error Handling in Clojure, Clojure Reducers for Mortals
  • J. Wilk: Isolating External Dependencies in Clojure – a nice overview of the options and their pros and cons – with-redefs, alter-var-root, Midje (using alter-var-root in a more controlled manner), higher-order-functions (#1!) etc.
  • philandstuff’s detailed notes from Euroclojure 2014

Tools/Libs

  • NixOS (via @bodil) – a new interesting “purely functional” Linux distribution – system configuration is fully declarative (think of Puppet/Chef) and it is always trivial to roll back, you can have multiple versions of a package, users can install non-global SW
  • InfluxDB – time series, metrics, and events DB that scales; contrary to Graphite it can store richer data than Graphite and its single value; additional highlights: authorization for individual data, roll-up/clean old data, https API. Written in Go.

Posted in General, Languages, Top links of month | Tagged: , , , , , , , , , | Comments Off on Most interesting links of July ’14

Most interesting links of June ’14

Posted by Jakub Holý on June 30, 2014

Recommended Readings

  • The emperor’s new clothes were built with Node.js – I know sadly little about Node.js but this goes against the hype and is thus interesting. So what does Node.js give us? Performance 1-5x slower than Java [like Clojure] according to the Benchmarks Game (contrary to other benchmarks with the opposite result as mentioned in the comments), use of a single CPU/core on our multi-cpu, multi-core machines, callback hell. At the same time, there are good non-blocking servers available in other languages (Clojure’s http-kit, Vert.x, etc.) (Update: From the comments it seems that f.ex. the “callback hell” situation is geting better with 0.11, fibers and other things I do not know anything about. Also Sandro has a nice anti-comment (No. 36).)
    The Node.js Is Bad Ass Rock Star Tech 5 min video is a nice companion :)
  • The Expert (Short Comedy Sketch)  (7 min) – you’ve certainly seen this one but I had to put it here; a young engineer is hammered into being an “Of course I can do it, I am an expert” ‘expert/consultant’ during a business meeting. Maybe you too have experienced a dialog with the business where your true expert opinion was crushed by the business people’s insistence on their absurd requirements?
  • Reset The Net – Privacy Pack – privacy-enhancing apps for PC/mobile
  • The Dyslexic Programmer (via Kent Beck) – interesting read about quite a different way to percieve and think about code, the advantages of IDEs.
  • It’s Here: Docker 1.0 => more stable from now on
  • Kent Beck: Learning About TDD: The Purpose of #isTDDDead – what is the purpose and value of TDD? Where are the limits of its value? “I recognize that TDD loses value as tests take longer to run, as the number of possible faults per test failure increases, as tests become coupled to the implementation, and as tests lose fidelity with the production environment.
  • Failure & Cake: A Guide to Spotify’s Psychology of Success – want to be innovative and successfull? Learn to embrace failure, nurture the “growth mindset” (failure as opportunity to improve) rather than the “fixed mindset” (I do not learn and every failure shows I have no value). Read this if you want your org to be a better place to work!

Non-tech

  • LSD — The Problem-Solving Psychedelic – I never knew that drugs could be used to something positive, with an incredible effect. Are you stuck with a tech/design/art problem? Try LSD! :-)
  • The French are right: tear up public debt – most of it is illegitimate anyway – “Debt audits show that austerity is politically motivated to favour social elites. [..] 60% of French public debt is illegitimate” – not improving the lives of people but thos at power/rich. Time to reconsider this debt business and ways to make our system better?
  • Forbes: Why Financialization Has Run Amok – Wall Street is the kind and companies do everything to look better in its eyes – including giving up on opportunities. The might of the finance sector is destructive to our economy and distorts it, away from producing more value to making financial institutions richer, away from (value) creative activities to distributive ones. The article describes the problem and proposes a solution including limiting the size and leverage of banks, taxing financial transactions etc. Example of the effects: “[..] a cabal of senior IBM executives and the managers of some big investment firms got together and devised a five-year scheme—IBM’s Roadmap 2015—for increasing IBM’s earnings per share—and their own compensation—through measures that are not only increasing earnings per share but also steadily crippling IBM’s ability to innovate and compete [..]
  • Why Can’t We All Just Get Along? The Uncertain Biological Basis of Morality – very interesting criticism of “morality” that is mostly based on emotions and thus contradictory, a good argument for utilitarian morality [not that it hasn’t its own challenges]. According to the author, many conflicts are nor primarily due to divergent values but due to different interpretation of the reality and history (such as “who has right to this land?”). People suffer “[..] from a deep bias—a tendency to overestimate their team’s virtue, magnify their grievances, and do the reverse with their rivals.” “This is the way the brain works: you forget your sins (or never recognize them in the first place) and remember your grievances. [..] As a result, the antagonisms confronting you may seem mysterious, and you may be tempted to attribute them to an alien value system.” This leads to partial judgements that play very badly with another psychological feature – “Namely: the sense of justice—the intuition that good deeds should be rewarded and bad deeds should be punished.” “When you combine judgment that’s naturally biased with the belief that wrongdoers deserve to suffer, you wind up with situations like two people sharing the conviction that the other one deserves to suffer. Or two groups sharing that conviction. And the rest is history.” And “The most common explosive additive is the perception that relations between the groups are zero-sum—that one group’s win is the other group’s loss.” => “So maybe the first step toward salvation is to become more self-aware.
    When you’re in zero-sum mode and derogating your rival group, any of its values that seem different from yours may share in the derogation. Meanwhile, you’ll point to your own tribe’s distinctive, and clearly superior, values as a way of shoring up its solidarity. So outsiders may assume there’s a big argument over values. But that doesn’t mean values are the root of the problem.
    Those who choose not to act in the trolley dilemma[..] are just choosing to cause five deaths they won’t be blamed for rather than one death they would be blamed for. Not a profile in moral courage!

Clojure Corner

  • The Case for Clojure (video, 5 min) – a short video arguing for Clojure as a good solution language based on its simplicity, power, and fun factor. There are many claims and few facts (as dictated by the short length) but it might be interesting for somebody.
  • CrossClj.info – cross-reference of many OSS Clojure projects – find all uses of a fn across the projects, all fns with a given name, all projects using ring, … . Search by fn, macro, var, ns, prj.
  • The Weird and Wonderful Characters of Clojure – ‘A reference collection of characters used in Clojure that are difficult to “google”.’

Tools/Libs

Posted in General, Languages, Tools, Top links of month | Tagged: , , , , , , , , | Comments Off on Most interesting links of June ’14