The Holy Java

Building the right thing, building it right, fast

NDC Oslo 2015: Talk notes, recommended talks (security, FP, etc.)

Posted by Jakub Holý on June 19, 2015

A great conference. A good deal of good talks.

To check later

Wednesday

Thursday

Friday

Keynote Data and Goliath ☆☆☆☆

Inspiring, slightly eye-opening!

Key points: We live in a society of ubiqutious surveillance. We need to share data to get great services yet we need to ensure personal privacy and security – finding the right balance will be a major dialog in the near future. Currently we share too much, too freely. Even high-tech, presumabely super secret surveillance tools and SW had their cousins presented at security conferences => anybody can build these, and many do. Either everybody gets to spy – or nobody does => build security in.

Check out Schneier’s book Data and Goliath and blog.

Lean and FP ☆☆

Nice intro to Lean and FP, some nice points (e.g. why do companies fail in adopting FP). A little new stuff for me; it wasn’t a waste of time but perhaps I could have seen a more valuable talk.

A tour of the language landscape ☆☆

It’s interesting to learn about various cool capabilities of other languages. But it is more useful as an inspiration, there is little you can start using in your daily work. The slides are worth going through.

Covered:

  • F#: Pipes (~ Clojure threading macro), Type Providers (e.g. get autocompletion, compile-time checks for browsing your Amazon S3 bucket), Unit-of-Measure (=> 2 + 2 fails), Statically Resolved Type Parameters
  • Clojure: Macros
  • Go: Implicit Interface Implementation (define an interface with a method; everything having the method autom. implements it, no need to tag those (which would be impossible for 3rd party code) => type safety and convenience)
  • Rust: Borrowed Pointers (autom. memory management without GC overhead – a piece of code owns a pointer => destroyed when the scope is left but can “borrow” it to code (of not longer scope) that it calls)
  • Erlang: Actor Model, Bit Syntax
  • Idris: Dependent Types, Uniqueness Types
  • Elm: Signals (reactive programming (Rx))

Also: Example: Functional Reactive Programming decisively beats Imperative on simplicity, length

Async and Streaming JavaScript, We’re All Doing it Wrong! ☆☆☆

Useful. Highlights limitations of / issues with Promises and (Node, WHATWG Streams). Demonstrates how Reactive Programming (e.g. RxJS) is a superior solution. About the coming inclusion of (some of) Rx in JS.

See NDC: Async and Streaming JavaScript, We’re All Doing it Wrong! (Promises, Streams, Rx)

Learning from Haskell ☆☆☆

Venkat Subramaniam is a skilled speaker. This was fun and enriching look at what we can learn from Haskell and hopefully use in our daily programming even if we use C#.

See list of the lessons learned.

50 Shades of AppSec ☆☆☆

Fun, scary. Examples of some hacks, surprisingly terrible security at many places, demo of tools that hack a site for you. Recommended as an eye opener to the importance of security.

Included:

  • Bishop:
    > Bishop is a vulnerability scanner that searches websites in the background while you browse, looking for exposed version control systems, misconfigured administrative tools, and more. With a whitelisting regex system, you can easily restrict this tool to hosts that you are authorized to scan.
  • Havij: Find & elverage SQL injection vulnerabilities automatically
  • Hire a hacker on-line: https://hackerslist.com/

Idioms for building distributed fault-tolerant applications w/ Elixir ☆☆

A light introduction to the Erlang VM and Elixir.

FP:

  • Explicit over implicit state
  • Transformation over mutation

Elixir is a distributed > concurrent > FP lang (in the order of importance).

Elixir goals: Compatibility, Extensibility => macros, Productivity.

  • Productivity: 1st class docs, tooling (ExUnit, IEx, Mix), hex packages

Demo: Agents, pattern matching, tools, supervision, streams (=> lazy computations), … .

Event Sourcing and DDD with F# ☆

After a brief introduction of E.S. (without going into why/when), the speaker presented E.S. as two functions (decide, evolve) and went on implementing a simple card game in F# using that. Cool: Own given-when-then test “framework” in just few lines of code. The talk concluded with demonstration of plugging in an actual Event Store server and load + save functions and using an Agent to hold the current state.

Command + [state]: (decide) -> event(s) -> store them, evolve the state
^--------------------------------------------/

decide: fn(state, command) -> Events
evolve: fn(initial state, state, Event) -> state

A little too low-level both on F# and even sourcing but OK to follow on the some-understanding level. My main lesson learned is that F# looks nice, concise, powerful. And FP is certainly a good match for E.S.

Phoenix – a framework for the modern web ☆☆

Elixir with Phoenix (and Plug) => Performance and Productivity.

The most frequent quote:

It’s just a function call

  • Request lifecycle: Endpoints -> Routers -> Controllers

Productivity

  • Middleware = just functions => compositions
  • Helpful errors (and web view); “if you can’t understand an error, it’s a bug”
  • Just mix in the comments generator to enable comments in the page and add the
    generated controller to your pipeline
  • mix ecto.migrate => data migrations
  • static assets with Brunch (Node; combine, minify, watch for changes, …)
  • ES6 transpiler included
  • live reload (html, css, ..)

Performance

  • Under-1ms response times
  • Views and templates: precompiled, fast, function calls. Views render templates (embedded elixir, HAML, or another). Thx to metaprogramming, render becomes just string concat. at runtime.
  • Ex.: Phoenix on RaspberryPi (16MB mem, 540req/s serving a simple HTML)
  • Robust concurrency model – extremely lightweight processes ( no global pauses; load balances on IO and CPU
  • Pattern-matched route dispatch, the VM is very fast at it (Note: Convenience fn to get route for a controller/… => metaprogramming posibilities)
  • Channels (Pub/Sub): trivial realtime communication (for connected devices) – WebSockets with minimal code Clients for the browser (fall-back to long-polling), for Objective-C, eventually Android. Embedded devices: CoAP instead of WS. Can use Redis, … . Synchronous0like communication possible; JS ex.: chan.push(..).receive("ok", okFn).receive("error", errFn).
  • Distributed cluster support (that’s what Erlang always did with multiple communicating switches)
  • Everything is a separate process => one channel crashing cannot harm other channels.

Ex.: Chat.Endpoint.broadcast("rooms:lobby", "new_msg", %{username: "me", body: "hello"}) => all connected clients receive it, no matter whether local or remote.

  • Erlang: Erlang handles 1/2 of the world’s telecom traffic. Reported 99.999999999% availability.
  • WhatsApp: 2M conn/server; 400M users, 30 engineers
  • Update your code while the app is running! Apps that run forever.

Awesome Elixir: A curated list of amazingly awesome Elixir and Erlang libraries, resources etc.

The Power and Practicalities of Immutability ☆☆☆☆

In F#, mutable is the keyword of shame.

Immutability (assignment-less programming where mutation is replaced by transformation) doesn’t remove mutation it just pushes it to lower-level code – the same we did with goto, which disappeared from our code but still is in the bytecode/asm/.. .

Assignment is to Functional Programming as goto is to Structural Programming.

How to program w/o assignments:

  • Recursion; cons:
  • It can blow up the stack (tail-cole optim.)
  • Some problems do not fit it naturally
  • Move mutation to lower level => build from a higher-level, safe building blocks;
    Ex.: for loop -> list.Select(e => e*2)

The problems

  • Mutability
  • Hard to reason about the code (especially if multiple variables)
  • Difficult to manage state transitions; concurrency difficult;
  • Imperative progr. (e.g. a for-loop)
  • I need to say both what and how to do (like a toddler – fun on the 1st day, not the other 18 years :-))
  • Mutating variables all the time

Benefits of immutability (and pure functions)

  • easy to reason, understand, explain, test
  • memoization
  • referential transparency (it’s possible to replace a call with its result) => optimizations (exec. reordering, running concurrently, memoization, ..)
  • immutability enables laziness (e.x: {1,2,3,4}.Where(e => e > 3).Where(e => e % 2 == 0).Select(double).firstOrDefault() – all the checks and the doubling are only executed 0-3 times)
  • immutability makes parallelization affordable

Notes

  • Examples in C#, F#
  • Ex.: sum, double a list, …
  • “Encapsulation” = “I don’t care” :)

How to turn software into competitive business advantage ☆☆☆☆

Gojko Adzic is my favourite speaker, with many good thoughts (and experiences).

Continuous delivery has huge side-effects. Exploit them and open new business opportunities.

Fast is good when there is human control over it – when business people have no say in what gets updated when, they loose control (and are unhappy).

  • Can we control C.D. or are we loosing control of it?
  • Ex.: Vehicle charger => possible fire: for Tesla it was just an over-night automatic software update contrary to GM that had to recall the cars.
  • Marketing side-effect of C.D.: No more big releases to have press conference about
  • C.D. is important for devs but we don’t want to piss off users
  • UX: CD (i.e. partial delivery) => user confusion

Gojko’s 3 rules

C.D. nay not …

  1. confuse users
  2. interrupt users’ work of sessions
  3. disrupt or prevent marketing initiatives

Deployment != Release

  • Release is a marketing event, biz decides.
  • We need to support running multiple versions of the SW in production (so current users are not interrupted, biz may decide when to switch over)
  • Problem: data. Ex. solution: add data (format) version field
  • Gentle deployment: Release a new version in || to the old one, enable (and motivate) users to switch at will while preserving the ability to go back (if they encounter any issues …)

Continuous Delivery without multi-versioning is irresponsible!

2 things to remember:

  • Decouple deployment and release
  • Give 2% of users a 100% solution, do not force-feed 100% users with 2% of scope

Modern architectural patterns for the cloud

Perhaps not a bad overview of various patterns (DB replication, sharding; CQRS, event sourcing, map reduce). Too little knew at start and then I lost focus :)


Get Unblocked (unleash your creativity) ☆☆☆

Fun, inspiring. Good tips for overcoming our internal barriers to creativity (and a happier life). Slides. Book: Banish Your Inner Critic.

The entire industry is broken – managers are from Mars, developers from Venuse. Micromanagement, open spaces / cubicles, time pressure, demanding clients, bureaucracy, … .

  • Face and ID your internal critic
  • Impostor syndrom (“I don’t deserve this, I am not good enough for it”)
  • Don’t compare yourself to other people
  • Perfectionism (focus on the product rather than the creative process)
    (Pottery exp.: 1/2 tries to make the best pot, 1/2 as many as possible =>
    the latter had more and many really good.)
  • Procrastination (feeds Perf. and v.v.)
  • Banish slave words from your vocabulary – Eliminate should/must/ought to/…
  • “Failure is only the opportunity to begin again more intelligently.” H. Ford
  • Stressed? Make it really bad on purpose => overcome the block of trying to be perfect.
  • Get into the Alpha brain waves mode (more relaxed than the normal beta)
  • Breath to re-focus
  • Lay down
  • Daydream / space out
  • Take shower
  • Get physical – movement allows the hemispheres to sync, …
  • Exercise discipline
  • Manage time (if stressed => no place for creativity)
  • Make “To DON’T” lists – what not to do (stop, delegate, …)
  • Say NO to distractions (see http://theheadphonesrule.com/)
  • Apps: Consider the RescueTime app => increase awarness; Mac: HeyFocus.com when on, stops you from going to distractive pages and apps; GetConcentrating.com; browser extensions, .. – see the slide
  • Single-task (instead of multi-tasking) – e.g. the Pomodoro Technique; especially useful when you do something you don’t like to do (during the 5 min break activate via a walk, laying down, having a look at the sky, …)
  • Do it with others – we come up with better / different things when working with others. Avoid sameness! => diverse environments. Share your ideas. Share mistakes. Amplify the ideas of others – try to help them to build them. Make your partned look good.
  • “Yes, and ..” technique (useful for meetings!) – don’t shut down others’ ideas – “yes, we can do this – and we can do this other thing”

Fear:

F.alse
E.vidence
A.ppearing
R.eal

(Or Fuck Everything And Run)

Watch:

A security testers toolkit ☆☆☆☆

Awesome! Scary! You have to see this! This is a real eye opener, demonstrating how easy hacking can be (especially with cooperating users via little social engineering).

You can only fight the way you practice – Miyamoto Musashi

Tools

  • Kali Linux
  • nmap (network mapper)
  • Veil fwrk – avoid anti-virus
  • Metasploit
  • BeEF – The Browser Exploitation Framework

Vulnerabilities

Kali Linux

  • Kali Linux VMs
  • Many hacking tools
  • Demo (see exploiting vsftpd):
  • Run against Metasploitable is an intentionally vulnerable Linux virtual machine
  • Zenmap: GUI for nmap
  • Metasploit – provides an interactive shell; need to know the exploit: use exploit/unix/ftp/vsftpd_234_backdoor; set RHOST ; exploit
  • Wifi hacking
  • Use a wifi card enabling monitoring, compatible with Kali – e.g. one of the Alpha cards
  • Aircrack-ng – WPA-PSK keys cracking program
  • airmon-ng => start the device in monitor mode, list devices around, …
  • WPS (Wi-Fi Protected Setup) is very crackable
  • PwnSTAR – soft access point – set up an AP, nmap scan clients, …
  • Armitage – GUI – if nmap, metasploit complicated, this is a nice, visual representation of the network and vulnerabilities on it; scan net -> scan a host -> “Attacks – Find Attacks” (Hail Mary – lunches 100s of exploits, not stealthy at all). Also a paid version Cobalt Strike, more powerful.
  • BeEF – hook into a website, take over a browser => detect extensions, send commands to them, metasploit, social eng. attacks, … Send fake flash/java/last pass update, petty theft – “facebook session timed out” popup (e.g. on any site with Fb integration), a Clippy help offer, … .
  • Veil – SW avoidance toolkit – generate payloads against machines, take them over – create f.ex. a “call home” .exe (connects to your metasploit server later command the app and take over the PC) (call it e.g. LastPassUpdate.exe, FlashUpdate.exe, …)
  • load espia; screengrab; keyscan_start; keyscan_dump; getsystem => elevate to local admin; load incognito; add_user XYZ myPsw; add_localgroup_user Administrators XYZ

Framework for Man-In-The-Middle attacks – => Bypassing HSTS (HTTP Strict Transport Security) with MITMf – downgrade the connection to HTTP
* The PC with MITMf makes itself to be the gateway for the victim PC
* Tip: http://wwww.google.com – can YOU spot the 4th w?; skandiabanken.no => webskandiabanken.no
* Only works if the user hasn’t accessed the page before in this browser, cookies and HSTS list are clear (wifi coffees & online banking = bad!)

Going beyond OWASP ☆☆

Walk through some past vulnerabilities in .NET. Some really nice hacks. Likely worth watching.

Few interesting examples:

  • HashDoS attack due to consistent hashing, putting all form fields starting with the same latter to the same bucket (common in other languages too)
  • When encrypting, also sign it so that when you get an encrypted stuff back, you’re sure you did indeed produce it (there was an endpoint allowing to check whether st. is properly encrypted => play with bytes and high math to encrypt st. w/o the key)
  • Misusing REST – constructing URLs from user input => make sure the input does not contain path (e.g. user = “./admin”), query strings, # chars, anything else affecting the path – basically 0-9a-Z
  • in Turkey, “interesting”.ToUpper() != “INTERESTING”; ppl make security decisions based on string comparison (.NET: string.Compare(a,b, ))
  • Unicode has “full width” chars: < (<); when put into SQL varchar, it becomes normal ” don’t use non-unicode columns to store Unicode
  • Common cryptographic mistakes: e.g. key / init.vector reuse (=> larger attack surface). Writing own crypto.
  • Hashing concat. strings – but hash(builtin+securely) == hash(built+insecurely) => do len(builtin) + hash(builtin) + len(securely) + hash(securely)
  • Cert pining – do; checking the thumbprint is fragile, check the signing chain, subject name too
  • Time-incons. hashing => timing attacks => don’t do early exit (when (mis)match found)
  • MVC in web apps with auto-binding data to objects assign even properties that normally are not in the form (but added by a hacker), e.g. ID – “over-binding”
  • Don’t trust file names from uploads
  • Zip bombs – tiny file which recursively uncompress; e.g. 42.zip (42kB) => 4.5 petabytes
  • XML bombs – e.g. Billion Laughs; preventable by turning of DTD Parsing

Making Hacking Child’s Play ☆

A fun talk though it contained only little new after being to Troy’s workshop and the other talk.

Knowledge is power! The guide to measure what matters. ☆

Essentially an intro into StatsD. I had to leave early.

Posted in General | Leave a Comment »

Notes from Troy Hunt’s Hack Yourself First workshop

Posted by Jakub Holý on June 17, 2015

Troy Hunt (@troyhunt, blog) had a great, very hands-on 2-day workshop about webapp security at NDC Oslo. Here are my notes.

Highlights – resources

Personal security and privacy

Site security

Breaches etc.

To follow

Read the rest of this entry »

Posted in General | Tagged: , | 1 Comment »

Example: Functional Reactive Programming decisively beats Imperative on simplicity, length

Posted by Jakub Holý on June 17, 2015

@theburningmonk Yan Cui has a nice example demonstrating how Functional Reactive Programming [slides 185 – 206] (with Elm’s Signals) yields a much shorter and easier to understand (one you know FRP) code than an imperative code with mutations spread all over the code base.

The game

Use the Up and Down keys to move the platforms and thus bounce the ball from left to right and back:

screenshot-game

The imperative solution

Read the rest of this entry »

Posted in Languages | Tagged: , | 1 Comment »

NDC: Async and Streaming JavaScript, We’re All Doing it Wrong! (Promises, Streams, Rx)

Posted by Jakub Holý on June 17, 2015

By Matthew Podwysocki

Events

Lot of work (setup, remove listeners …), not composable.

Promises

No way to abort promise in progress. (Me: has to remember to check for errors: then(onOk, onError).)

No try-catch-finally; only try-catch.

Streams

Node: Stream 1 were terrible (pause/resume unusable, data sent before ready, …).

Read the rest of this entry »

Posted in Uncategorized | 1 Comment »

Why do companies fail at adopting Functional Programming?

Posted by Jakub Holý on June 17, 2015

According to the NDC Oslo talk Lean and Functional Programming by Bryan Hunter, these are the reasons why companies fail to adopt FP:

  • They say “our developers aren’t smart enough” (to use F#, Erlang) [they should invest in their education!]
  • Culture of hiding problems => little incentive to adopt a paradigm that solves/prevents them if they’re invisible
  • Overburden => not time
  • Implementing changes (FP) without first proving them (PDCA) – blindly rewriting something in F#/… can fail; it’s better to have a value-proposition hypothesis and prove it with a limited experiment first
  • The prioritise short-term (e.g. fire-fighting) over long-term (removing the root causes of problems)

Tip for driving FP adoption: Find a pragmatist in pain – e.g. a business person experiencing problems that FP could have prevented.

Posted in Uncategorized | 1 Comment »

AWS API: Proper syntax for filtering by tag name and value (e.g. describeInstances)

Posted by Jakub Holý on June 11, 2015

It took me quite a while to figure out the right syntax for filtering instances by tag name and value in the AWS EC2 API’s describeInstances.

The documentation is not exactly crystal-clear to me:

  • tag:key=value – The key/value combination of a tag assigned to the resource, where tag:key is the tag’s key.

Anyway, here is the proper syntax, provided we are interested in the tag elasticbeanstalk:environment-name:

    var params = {
        Filters: [
            {
                Name: 'tag:elasticbeanstalk:environment-name',
                Values: ['mySuperApp']
            }
        ]
    };
    ec2.describeInstances(params);

So the name of the tag is embedded in the Name part and not, as I initially understood,
{ Name: 'tag', Values: ['elasticbeanstalk:environment-name=mySuperApp'] }

Credit: garnaat.

Posted in [Dev]Ops | Tagged: , | Leave a Comment »

Mounting an EBS volume to Docker on AWS Elastic Beanstal

Posted by Jakub Holý on June 2, 2015

Mounting an EBS volume to a Docker instance running on Amazon Elastic Beanstalk (EB) is surprisingly tricky. The good news is that it is possible.

I will describe how to automatically create and mount a new EBS volume (optionally based on a snapshot). If you would prefer to mount a specific, existing EBS volume, you should check out leg100’s docker-ebs-attach (using AWS API to mount the volume) that you can use either in a multi-container setup or just include the relevant parts in your own Dockerfile.

The problem with EBS volumes is that, if I am correct, a volume can only be mounted to a single EC2 instance – and thus doesn’t play well with EB’s autoscaling. That is why EB supports only creating and mounting a fresh volume for each instance.

Read the rest of this entry »

Posted in General | Tagged: , , , | Leave a Comment »

OS X: Using scutils to discover whether/what a web proxy is in use

Posted by Jakub Holý on May 7, 2015

When looking for ways to discover whether a proxy is being used by OS X, you will be typically pointed to

networksetup -getwebproxy

However that does not always work – for example when using “Auto Proxy Discovery” and/or “Automatic Proxy Configuration” with a proxy.pac file. scutils --proxy seems to detect all these cases (though it cannot give you the proxy when using auto config, I suppose):
Read the rest of this entry »

Posted in General | Tagged: , , | Comments Off on OS X: Using scutils to discover whether/what a web proxy is in use

All-in-one Docker with Grafana, InfluxDB, and cloudwatch-to-graphite for AWS/Beanstalk monitoring

Posted by Jakub Holý on May 7, 2015

I have derived the Docker container docker-grafana-influxdb-cloudwatch that bundles Grafana dashboards, InfluxDB for metrics storage, and runs cloudwatch-to-graphite as a cron job to fetch selected metrics from AWS CloudWatch and feed them into the InfluxDB using its Graphite input plugin. It is configured so that you can run it in AWS Elastic Beanstalk (the main problem being that only a single port can be exposed – I therefore use Nginx to expose the InfluxDB API needed by Grafana at :80/db/).

Read the rest of this entry »

Posted in General | Tagged: , , , | Comments Off on All-in-one Docker with Grafana, InfluxDB, and cloudwatch-to-graphite for AWS/Beanstalk monitoring

Hack: Quickly Verify That All Your Mocha/Chai Tests Have Valid Assertions

Posted by Jakub Holý on May 6, 2015

Chai is a popular Node/browser assertion library. However – as everything – it has its flaws. An important flaw is that it performs checks on property access – and if you e.g. misspell the name of an assertion, it will be just ignored (for there is no way for Chai to know that you tried to access a non-existent property). This may be fixed in the future with ES6 proxies but so far you risk having tests that actually do not test anything. Though you should of course always develop your tests so that they initially fail and thus you know they actually assert something :-).

Anyway, there is a neat quick way to verify that all your tests have at least one valid assertion – simply replace expect with expect.not.
Read the rest of this entry »

Posted in Testing | Tagged: | Comments Off on Hack: Quickly Verify That All Your Mocha/Chai Tests Have Valid Assertions