Posts Tagged ‘performance’
Posted by Jakub Holý on July 22, 2015
By Michael T. Nygard, 2007, ISBN: 978-0-9787-3921-8
My digest and review of the book.
Of the books I have read, Release It! is the one I would require all “senior” developers to read (together with something like Architecting Enterprise Solutions: Patterns for High-Capability Internet-based Systems). Especially the first part on stability with its patterns and anti-patterns is a must read. Without knowing and applying them, we create systems that react to problems like a dry savannah to a burning match. I found also to next to last chapter, #17 Transparency, very valuable, especially the metrics and design of the OpsDB and observation practices.
One thing I have left out of the digest which is really worth reading are the war stories that introduce each section, they are really interesting, inspiring, and educational.
Stability x longevity bugs
Integration point = call to a DB, WS, … . Stability risk #1.
Read the rest of this entry »
Posted in [Dev]Ops | Tagged: architecture, book, ops, performance | Comments Off on Book Review & Digest: Release It! Design and Deploy Production-Ready Software
Posted by Jakub Holý on January 8, 2015
(Cross-posted from blog.iterate.no)
Digest of chapter 9 of the Continuous Delivery bible by Humble and Farley. See also the digest of ch 8: Automated Acceptance Testing.
(“cross-functional” might be better as they too are crucial for functionality)
- f.ex. security, usability, maintainability, auditability, configurability but especially capacity, throughput, performance
- performance = time to process 1 transaction (tx); throughput = #tx/a period; capacity = max throughput we can handle under a given load while maintaining acceptable response times.
- NFRs determine the architecture so we must define them early; all (ops, devs, testers, customer) should meet to estimate their impact (it costs to increase any of them and often they go contrary to each other, e.g. security and performance)
- das appropriate, you can either create specific stories for NFRs (e.g. capacity; => easier to prioritize explicitely) or/and add them as requirements to feature stories
- if poorly analyzed then they constrain thinking, lead to overdesign and inappropriate optimization
- only ever optimize based on facts, i.e. realistic measurements (if you don’t know it: developers are really terrible at guessing the source of performance problems)
A strategy to address capacity problems:
Read the rest of this entry »
Posted in Testing | Tagged: continuous_deployment, performance | Comments Off on Continuous Delivery Digest: Ch.9 Testing Non-Functional Requirements
Posted by Jakub Holý on April 28, 2014
Getting up and running with Gatling perf. tests as I would like so I record this for my future reference.
0. Create a project:
$ mvn archetype:generate \
(The trailing “:” in the filter is important.)
1. Import to IntelliJ
In IntelliJ choose to import an object, instead of “from sources” select “from external model” and then Maven. You will also need to have the Scala plugin installed and, when imported, you will likely need to right-click on pom.xml and Maven – Reimport.
2. Record a simulation
- Run the
src/test/scala/Recorder.scala (right-click – Run ‘Recorder’)
- Set the port it should listen on, f.ex. 8000 (maybe you also need to set port for HTTPS, f.ex. 8001), set the target app (perhaps localhost, <some port>, <some https/dummy port>)
- Optionally set the class name of the recorded simulation and the target package (the output goes to
- Click [Start !]
- Go to your browser and configure it to use the recorder as its HTTP[s] proxy
- Browse localhost:8000/your_app as you want for the test
- Click [Stop and save] in the Recorder UI
Read the rest of this entry »
Posted in Testing, Tools | Tagged: gatling, performance | Comments Off on How to create and run Gatling 2.0 tests
Posted by Jakub Holý on February 28, 2014
- Nathan Marz: Principles of Software Engineering, Part 1 – Nathan has worked with Big Data at Twitter and other places and really knows the perils or large, distributed, real-time systems and this post contains plenty of valuable advice for making robust, reliable SW. Main message: “there’s a lot of uncertainty in software engineering“; every SW operates correctly only for a certain range of inputs (including volume, HW it runs on, …) and you never control all of them so there always is an opportunity for failure; you can’t predict what inputs you will encounter in the wild. “[..] while software is deterministic, you can’t treat it as deterministic in any sort of practical sense if you want to build robust software.” “Making software robust is an iterative process: you build and test it as best you can, but inevitably in production you’ll discover new areas of the input space that lead to failure. Like rockets, it’s crucial to have excellent monitoring in place so that these issues can be diagnosed.“. From the content: Sources of uncertainty (bugs, humans, requirements, inputs, ..), Engineering for uncertainty (minimize dependencies, lessen % of cascading failure [JH: -> Hystrix], measure and monitor)
- Suffering-oriented programming is certainly also worth reading (summary: do not start with great designs; only start generalizing and creating libs when you have suffered enough from doing things more manually and thus learned the domain; make it possible > make it beautiful > make it fast, repeat)
- ThoughtWorks open-sources Go, continuous delivery platform – good bye, Jenkins! – better support for pipelines etc., see features and elementary concepts
- Cloud Design Patterns: Prescriptive Architecture Guidance for Cloud Applications (recommended by @markusbk so it must be good); Patterns: Cache-aside, Circuit Breaker, Compensating Transaction, Competing Consumers, Compute Resource Consolidation, Command and Query Responsibility Segregation (CQRS), Event Sourcing, External Configuration Store, Federated Identity, Gatekeeper, Health Endpoint Monitoring, Index Table, Leader Election, Materialized View, Pipes and Filters, Priority Queue, Queue-based Load Leveling, Retry, Runtime Reconfiguration, Scheduler Agent Supervisor, (data) Sharding, Static Content Hosting (-> CDN), Throttling, Valet Key.
Guidance topics: Asynchronous Messaging Primer, Autoscaling, Caching, Compute Partitioning, Data Consistency Primer, Data Partitioning, Data Replication and Synchronization, Instrumentation and Telemetry, Multiple Datacenter Deployment, Service Metering
- MOOC course Functional programming with Clojure at Uni of Helsinki – to get started you need, I suppose, follow the “Material and course content” – essentially read the text for each chapter, clone its repo, submit pull requests to get your work graded
- Jez Humble: The Case for Continuous Delivery – read to persuade manager about CD: “Still, many managers and executives remain unconvinced as to the benefits [of CD], and would like to know more about the economic drivers behind CD.” CD reduces waste: “[..]online controlled experiments (A/B tests) at Amazon. This approach created hundreds of millions of dollars of value[..],” reduces risks: “[..] Etsy, has a great presentation which describes how deploying more frequently improves the stability of web services.” CD makes development cheaper by reducing the cost of non-value-adding activities such as integration and testing. F.ex. HP got dev. costs down by 40%, dev cost/program by 78%
- Request Quest (via @jraregris) – entertaining and educational intractive quiz regarding what does (not) trigger a request in browsers and differences between them (and deviances from the standard) – img, script, css, etc.
- The REST Statelessness Constraint – a nice post about statelessness in REST if you, like me, don’t know REST so much in depth; highlights: Statelesness (and thus the requirement for clients to send their state with every request) is a trade-off crucial for web-scale and partially balanced by caching – while typical enterprise apps have different needs (more state, less scale) so REST isn’t a perfect match. Distinguish application (client-side) and server (resources) state. Using a DB to hold the state still violates the requirement. Use links to transfer some state (e.g. contain a link to fetch the next page of records in the response).
- CodeMesh 2013 presentations – good stuff! F.ex. Refactoring Functional Programs: Past and Future, Distribution, Scale and Flexibility with ZeroMQ, Deepak Giridharagopal on Puppet, Immutable Deployments, Analyzing Systems with PuppetDB, Francesco Cesarini and Viktor Klang on the Reactive Manifesto and more
- Cognitive Biases in Times of Uncertainty – people under pressure/stress start to focus on risks over gains and (very) short-term rather than long-term and thus also adopt 0-some mindset (i.e. if sb. else wins, I loose) => polarization into we x them and focus on getting as big piece of the cake possible at any price, now, dismissal of collaboration. With accelerating rate of change in the society due to technology, this is exactly what is happening. How to counter it? Create more positive narratives than the threat-based ones (views of the world), support them via short-term gains. Bottom line: each of us must work on spreading a more positive attitude to save us from bleak future.
- Book – Nathan Marz: Big Data – I dislike the big data hype (and, with passion, Hadoop) but would love to read this book; it presents a fresh look at big data processing, heavily inspired by functional programming. Nathan has plenty of experiences from Twitter and creating Storm and Cascalog (both in Clojure, btw.). Read ch 1: A new paradigm for big data.
- Facebook Engineering: The Mature Optimization Handbook (or go directly to the pdf, ePub, Mobi). If you get bored, jump directly to ch 5. Instrumentation.
Posted by Jakub Holý on September 30, 2013
- Stuff The Internet Says On Scalability For September 13, 2013 – a collection of interesting performance related articles with summaries (via @_dagi)
- Can you copy a culture? The NUMMI story (audio/transcript) – how the GM factory with the worst workforce has been turned around via a good application of Toyota Production System – “a truly inspiring story of human potential and how systems can be designed to bring the best or worst of of people.” And how GM failed to learn from it and to copy Toyota’s culture.
- The Reactive Manifesto – why to write reactive SW – “Reactive applications represent a balanced approach to addressing a wide range of contemporary challenges in software development. Building on an event-driven, message-based foundation, they provide the tools needed to ensure scalability and resilience. On top of this they support rich, responsive user interactions. We expect that a rapidly increasing number of systems will follow this blueprint in the years ahead.“
- NYT: Eiji Toyoda, Promoter of the Toyota Way and Engineer of Its Growth, Dies at 100 – learn about the life of one of the founders of lean thinking
- Gojko Adzic: How we solved our #1 product management problem – valuable experience of false assumptions, learning from users, and a much helpful UI remake: even if you build a product to scratch your itch, you have to test it with real users
- Don’t use Hadoop – your data isn’t that big – a great post about the downside of Hadoop and that there are much better options (large disks, large RAM, Pandas/R/Postgres) for data up to few TBs. “In addition to being more difficult to code for, Hadoop will also nearly always be slower than the simpler alternatives.”
- Gartner On Big Data: Everyone’s Doing It, No One Knows Why – golf talk / hype -driven initiatives FTW! “According to a recent Gartner report, 64% of enterprises surveyed indicate that they’re deploying or planning Big Data projects. Yet even more acknowledge that they still don’t know what to do with Big Data.”
- What makes Spark exciting – why it might be a good replacement for Hive/Hadoop, based on experiences with H/H: “Hive has served us well for quite a while now. […] That said, it has gotten to the point where Hive is more frequently invoked in negative contexts (“damn it, Hive”) than positive. (Primarily due to being hard to test, hard to extend.)” “We had heard about Spark, but did not start trying it until being so impressed by the Spark presentation at AWS re:Invent [..] that we wanted to learn more. [..] Spark, either intentionally or serendipitously, addresses both of Hive’s primary shortcomings, and turns them into huge strengths. (Easy to test, extend.) [..] I find the codebase small and very easy to read, [..] –which is a nice consolation compared to the daunting codebases of Hadoop and Hive.” Cons.: Spark is only pre-1.0, the author hasn’t yet tested it heavily.
- 10 Ways to Make Your Office Fun To Work In – because we spend there plenty of our time so why not have a pleasant/cosy, inspiring environment? Some tips: plants, not-your-boring-enteprprise-look-and-feel, open it to the nature (I want this!), design it as home, not office, provide play space (I am too into work to want to play but having a resting place for a nap is st. I’d love).
- Book: The Architecture of Open Source Applications (via @rmz) – learn by studying architectures of existing systems – “In these two books, the authors of four dozen open source applications explain how their software is structured, and why. What are each program’s major components? How do they interact? And what did their builders learn during their development?“
- Book: Seven Concurrency Models in Seven Weeks: When Threads Unravel – “how to exploit different parallel architectures to improve your code’s performance, scalability, and resilience” – threads & locks, actors, FP + immutability/futures/promisses, Software Transactional Memory etc., GPU, MapReduce on clusters, … (intro) Personally, I would prefer from theory to practice approach and mention of CSP (-> Go’s channels, core.async) and more.
- Books Amazon CEO requires his top execs to read (The Effective Executive by Peter Drucker, The Innovator’s Solution by Clayton Christensen, The Goal by Eliyahu Goldratt)
- Stanford engineers build computer using carbon nanotube technology (via
- NYT: The Banality of Systemic Evil – a good article about human tendency to “obey the system” thus potentially causing evil – and thus the need to resist the system, as heroic individuals such as Snowden, Hammond, Schwartz, Manning. See the famous Eichmann in Jerusalem for how “doing your job” can create evil – “[..] what happens when people play their “proper” roles within a system, following prescribed conduct with respect to that system, while remaining blind to the moral consequences of what the system was doing — or at least compartmentalizing and ignoring those consequences.” (Tip: The book Moral Mazes explores the ethics of decision making within several corporate bureaucracies => mid-managers rules of life: (1) never go around your boss, (2) tell the boss what she wants to hear, (3) drop what she wants dropped, (4) anticipate what the boss wants so that she doesn’t need to act as a boss to get it, (5) do not report something the boss does not want reported, cover it up; the the job & keep your mouth shut.) “The bureaucracy was telling him [Snowden] to shut up and move on (in accord with the five rules in “Moral Mazes”), but Snowden felt that doing so was morally wrong.” “[..] there can be no expectation that the system will act morally of its own accord. Systems are optimized for their own survival and preventing the system from doing evil may well require breaking with organizational niceties, protocols or laws.“
- Fairphone – “A seriously cool smartphone that puts social values first” (likely the only one not built by poorly paid workers and creating too much ecological burden), for just €325. You can see detailed cost breakdown, list of suppliers, specs, and essentially everything. This is, in my opinion, super cool! Go and read the story!
- Amazonica – “A comprehensive Clojure client for the entire Amazon AWS api.”
- Talk Ritz, The Missing Clojure Tooling (40min, 9/2013) – thanks to this I finally understood how to use Ritz but it still seems not to work well, f.ex. setting a breakpoint always reported “Set 0 breakpoints” (lein ritz/middleware 0.7.0, nrepl-ritz.el 0.7.1); according to callen, debug-repl is simpler and nicer if you only care about local vars and evaluation. To try ritz: use M-x nrepl-ritz-jack-in, then M-x nrepl-ritz-break-on-exception, exec. f.ex. “(/ 1 0)”. In the poped-up buffer, t or enter to show frame locals, e to eval a code in the context of the frame etc. If you managed to trigger the debug buffer through a breakpoint, the actions lists would contain STEP etc. (See fun. nrepl-ritz-line-breakpoint)
- C. Grand’s spreadmap – “library to turn Excel spreadsheets in persistent reactive associative structures” => access content via map functions; changing a value updates formula cells using it
- Alembic Reloads your Leiningen project.clj Dependencies – add a dependency to your project.clj w/o needing to restart your REPL (just call (alembic.still/load-project), provided you have it in your lein dependencies). Limitations: cannot remove deps or change versions.
- Defeating stack overflows – techniques for transforming mutually recursive calls etc. into something that won’t blow the stack – “Priming the pump” (memoize subresults first), core.async
- Google Groups: Clean Architecture for Functional Programming – How do the Clean Architecture and the Clean Code best practices apply to FP (Clojure/Haskell)? Some points: OOP isn’t worse than FP, only people do class-oriented programming instead; OO better e.g. for UIs, combining them (func. core, imperative shell) can be sometimes best. Some clean arch. patterns are actually more like functions – “Interactors and Presenters, for example, do not maintain any state of their own. Even those objects that do imply some kind of state, such as entities and gateways, keep that state hidden behind boundaries and present a functional interface instead.“
- night-vision: Handy, super light weight debugging utility – add it to your lein profile and then call
(night-vision.goggles/introspect-ns! '<name of ns>) and it will print each entry/exit of a function within the scope of the namespace with the argument/return values
- Nil Punning (Or Null Pointers Considered Not So Bad) – a great post about why nil in Clojure is not bad contrary to Java’s null (because it is actually an object, you can call functions on it, treat it as false/empty list/map/set, most core functions work on it)
Posted by Jakub Holý on July 31, 2013
This month focuses on languages and approaches (reactive programming, F#, Erlang, FP talks etc.), agile (need for speed, recommended books), Clojure/Linux/cloud tools and libs.
- The Need For Speed – the top 10 reasons for fast development flow (with time to market being one of the less important) – more learning, focus on the MVP, focus on the puprose/goal, happier customers/leadership, better quality (sic!), higher morale (I concur!), push for cotninuous improvement, “one of the only sustainable differentiators”; => “sense of urgency and motivation”; “[..] I continue to meet people and teams that not only move very slow, they don’t understand the relationship between speed and innovation, or speed and quality.”
- agile42 Summer Reading List 2013 – books recommanded by experienced people/agile experts – lot of interesting stuff! Topics: Communication and Coaching (f.ex. Practicing Nonviolent Communication), Business (How Great Companies Get Their Mojo from Maslow ,..), Learning From the Military, Agile and Technology (e.g. The Art of Action: How Leaders Close the Gaps between Plans, Actions and Results), Agile and Technology (f.ex. The people’s Scrum)
- Dan North: Are we nearly there yet? – optimize for time to business impact; SW dev as mountaineering (impossible to estimate correctly, many unknown details, dead ends, …); go fast – but sustainable; the tyranny of backglog (there are multiple paths to the top yet backlog defines only one; have you ever considerably changed it?) “Instead we could embrace the fact that today we always know more than we did yesterday, and that tomorrow we will know even more. We can take a fresh look up the mountain every time we pause to regroup, to plan.” => we ask 1) what gives us the shortest lead time to business impact? 2) what can help us to learn/invalidate more? 3) how to assure our stakeholders we are approaching the goal?
- Joel on Software: Software Inventory (7/2012) – a classical article about the evilness of software inventory (backlogs, issue trackers, undeployed features, …) ‘When I hear about product teams that regularly have “backlog grooming” sessions, in which they carefully waste a tiny amount of time and mental energy every day or every week thinking about every single feature which will never be implemented, I want to poke my eyes out.‘
- Job satisfaction self-test: Twelve questions that define a great place to work – check yourself how satisfied you are with your job (example questions: How well do I know what is expected of me? How often in the past seven days have I received recognition or feedback on my work? How much does the mission/purpose of the company make me feel like my work is important?)
- Coaching Anti-Patterns: Prescriptive Agile – a prescriptive coach “knows” what is “right” and forces it onto the client, without listening to her; instead, we should “Meet them where they are and leave them in a better place” => “[..] my first responsibility is to understand how and why they came to this practice. How did they come to this decision? What challenges does this approach address? What benefits are they optimizing for?” Worth remembering AND practicing :)
Languages, paradigms, approaches
- The Trouble with Erlang (or Erlang is a ghetto) – an objective criticism of Erlang by somebody who seems to be quite experienced with it; as I know very little about Erlang, it was interesting to learn about its weaknesses (no map/dict data structure, slow memory management, poor “JIT,” not usable for shared-state concurrency (contrary to e.g. Clojure), immutable state is not necessary and makes some things bad, inconsistent and ugly standard lib, …)
- Adventures in Multi Paradigm Programming – different programming paradigms/approaches re-implemented in Emacs Lisp – interesting 1) to see and compare these different approaches and 2) the flexibility of Lisp. Including iteration – Ruby’s map, Python’s list comprehension ([an_expression for x in list]), Scala’s default argument (_); search – Java’s for; arguments: direct, variadic (i.e. any number of args), named args; destructuring and pattern matching in CoffeScript/OCaml style; Haskell-like monads; objects with mixins; namespaces.
- Why bugs don’t like F# – no nulls, immutable data, strong type system, composition of small functions, asynchronous programming abstractions, higher-order functions over collections (no off-by-one), units of measure
- IBM high-fives Netflix open-source tools – it is interesting to see the spreading of Netflix’s open source tools for better cloud infrastructures; f.ex. “Karyon, is what Netflix calls the base container for applications and services built using the NetflixOSS ecosystem; Eureka is mid-tier load balancing; Hystrix controls interactions between myriad distributed services to nip cascading failures in the bud; and Ribbon is a Remote Procedure Call library.”
- ZeroMQ instead of HTTP, for internal services (with implementation in Clojure) – an interesting idea of using ZeroMQ – the sockets on steroids library – instead of HTTP in a way compatible with existing HTTP routing libs; advantages of ZeroMQ: automatic retrial (=> can restart the target service withou noticing), speed, reuse of a connection. The trick is to send a http-like structure (i.e. with method, uri, body) and pass that to Compojure or similar (update: there are now Clojure/core.async bindings for ZeroMQ)
- Joel on Software: Victory Lap for Ask Patents – killing a bad Microsoft patent request in 15 minutes – Ask Patents is a new StackExchange site that enables experts to look at SW patent requests and point to previous existing works that invalidate them; as Joel describes in his successful patent kill story, it is not difficult at all. Hopefully this will manage to really help the patent office and hit woul-be patent trolls hard! #victory
- The Dangers Of “Gamification” In Education by Kathy Sierra (a former game designer, a trainer of trainers at Sun, author of the Head First book series) – gamification is often regarded as something very desirable that will improve our lives; however, as Kathy discusses, it has also dark sides and, applied unappropriately, can actually decrease our intrinsic motivation (therefore it should be nearly never used in e.g. education)
- Choosing an OSS license doesn’t need to be scary (by GitHub) – a human-readable overview of OSS licenses; you should always assign a license to your GitHub account (Add A License can help with that; otherwise it is considered to be “all rights reserved” and you are not giving back to the community (I use the same as Clojure, Eclipse Public License)
- FunctionalTalks.org – “Brilliant people giving brilliant talks on functional programming” – f.ex. Wilkes Joiner: Functional Reactive Programming, Alexander Gounares: All your cores are belong to us, Katie Miller: Superhero monads, Bryan O’Sullivan: Running A Startup On Haskell, Rich Hickey: Introduction To Clojure, John A. De Goes: Building a Data Science Platform in Scala and many more.
- Types vs. Tests: An Epic Battle? – “Amanda Laucher and Paul Snively debate solving problems through types and tests using different approaches.” – can type system replace tests or vice versa? Interesting intro into the discussion for me. Using F#, Scala & more. Same claims: types don’t pay out so much for “small” codebases but scale better than tests. Types – Tests is a spectrum, not two single extremes. When a property should hold “for all,” a type would be a good match. Inductive types (Scala, Haskell?) can become quite complex, dependant types (as in Coq) would be much nicer [if I got that right].
- Paul Irish on Web Application Development Workflow (via M. Noddeland) – if you need to do some web development but are not up to date on the state of art, this might be useful – an overview of tools, utilities, services by a Googler and the person behind Modernizr, HTML5 Boilerplate, Yeoman etc. Including effective shell & dotfiles.GH, better ssh via .ssh/config and authorized_hosts, the all-in-one dev/build tool Yeoman with live reload, BrowserStack for testing, LocalTunnel to easily share anything running locally, Chrome Dev Tools support for SASS and testing devices (emulate touch events, screen sizes), JetBrains’ WebStorm, sharing tools via setapp.me. A genous idea to use GoogleAnalytics to track usage of features in a CLI app!
Other Interesting Stuff
Azul Systems’ high-performance JVM on the Vega architecture (from The Trouble with Erlang (or Erlang is a ghetto)) looks very interesting:
The other night I tweeted “If you’re looking for a language that gets multicore concurrency right, look at how Azul implemented Java on their Vega architecture” and I definitely stand by that. Azul is a company that got a lot of smart hardware and software people together and had them work on designing a custom system which would scale to hundreds of CPU cores (up to 768 of them), heaps that topped 500 GB (up to 768GB), and had the GC pause only 10-20ms at a time. The realtime performance characteristics Azul managed to eek out of their system lead them to often describe their GC as “pauseless”.
- Discussion: How core.async compares to agents, future and promise? – future/promise: 1 producer, 1 value, multiple consumers; agent: an unbounded queue of functions mutating a single value, with multiple producers and consumers (reading the latest value produced); channel: multiple 1:1 producers/consumers, i.e. a value can only be taken once from the channel, using a bounded queue (=> slow consumers can block fast producers). As mentioned elsewhere, channels is a relatively low-level abstraction and other things can be built on the top of it.
- Clojure Tradeoffs (design implications and why you should care) – perhaps not very unbiased but interesting anyway :) (shared-memory over other computing paradigms, i.e. message-passing, dynamic over static, speed over convenience, composition over IoC, …)
- Rich Hickey’s post introducing core.async with its Go-like channels as a better alternative to a collback hell (I know everybody has already read it but it is still an important link :))
- Tools etc.
- Faster Clojure Startup with Class Data Sharing – use JVM’s capability to include any classes in its boot image and include clojure in it
- lein-ancient – checks for outdated dependencies and plugins => run “lein ancient :all”
- lein-try – a Leiningen plugin that enables you to try a library in a REPL in the context of your project without having to add it to project.clj; simply run “lein try clj-time 0.5.1” and then in the REPL “(require ‘[clj-time.core :refer :all])” and e.g. “(date-time 1986 10 14)”
- Lemur: tool to launch a Hadoop job locally/on EMR from a job definition file + actions before/after
- Emacs: sexp fold/expand is very useful for exploring source code (hide all but the first lines of all top-level forms with hs-hide-all) – the built-in hs-minor-mode can hide/show all, or hide/show/toggle one but the keys for it are cumbersome; hideshow-org makes it possible to toggle hide/show with TAB, while preserving the original TAB behavior (it does the normal TAB first only only if nothing changes does it expand/fold); very useful!
- devdocs.io (via @palruud): “an all-in-one API documentation reader for [web] developers,” navigable via keyboard – JS, HTML, CSS, DOM, DOM events, jQuery, Underscore.js
- Kilim – a message-passing framework for Java that provides ultra-lightweight threads and facilities for fast, safe, zero-copy messaging between these threads.
- AssertJ – a library of assertions similar to fest-assert but providing a richer set of assertions (nicer API then fest-assert, according to a friend)
- NetflixOSS – Netflix, the online streaming gigant, has open-sourced many fascinating components of its cloud infrastructure such as Karyon, a blueprint for web-ready components with many features (monitoring,…), Genie/Hadoop as a Service, Servo for monitoring, Archaius for configuration management – too many to list. Check out Chris Fregly’s fluxcapacitor, a demo distributed application that uses many of the components
- Tools to keep a daemon running:
Posted by Jakub Holý on June 30, 2013
Agile, process, SW dev, people etc.
- Real Options—a Mindset – an intro into the Real Options approach, which has been quite a hot topic and a transformational way of thinking for a number of inspiring people (Dan North, Liz Keogh etc.). “Real Options help us to better make decisions and commitments with three simple principles: Options have value. Options expire. Never commit early unless you know why.” We can “pay” to keep our options open longer, i.e. to avoid commiting prematurely.
- Demystifying the CHAOS report’s claim of ~ 1/2 features being unused: the Standish Group’s CHAOS report has been often quoted for its “finding” that a large percentage of features in applications is never/rarely used. However this claim seems to have never been confirmed, their “research” is reportedly not very scientific and not publicly available for scrutiny. Critique by Laurent Bossavit (2013), Jorge Aranda’s Standish, the CHAOS report, and science. Thx to @smalltalk80 for pointing this out! However there is one research, Online Experimentation at Microsoft, that supports the claim, in a different context but the same problem applies to features: “Evaluating well-designed and executed experiments that were designed to improve a key metric, only about one-third were successful at improving the key metric!”
- Why Yammer believes the traditional engineering organizational structure is dead – small teams, small projects (2-10 people, 2-10 weeks), no separation into front/middle tier/backend team (=> communication, design obstacle); have instead people specializing in these areas and construct feature teams from them based on the actual needs; engineers, not managers do eng. decisions; all aligned via focusing on the same 3 key metrics. Small projects => constant sense of urgency (and excitement): Often very long projects cause engineers to lose track of the end goal. Think of it in terms of hiking: start fresh & excited, get tired and losing track of the goal, excited again at the end => cut out the middle part, keep them in the exciting state where they can measure progress and see it visually; it’s the only way to maintain urgency and morale. Focus: people alwasy work only at one (short) project at a time (there are special bug-fixing teams for maintenance tasks with people rotating in&out).
- Agile development is more culture than process – Why thinking of agile as culture and not just process explains resistance and difficulty in teaching and learning the approach – and should be taught so => 1. Underscore agile values that motivate practice; 2. Identify organization values that compete with agile values, conflict of values; 3. Be sensitive to culture shock.
- Mark Zuckerberg’s Letter to Investors: ‘The Hacker Way’ (quite long, you might want to read only “The Hacker Way” part at the end) – about Facebook’s “unique culture and management approach” – “Hackers believe that something can always be better, and that nothing is ever complete.” “Hackers try to build the best services over the long term by quickly releasing and learning from smaller iterations rather than trying to get everything right all at once.” “Instead of debating for days [..], hackers would rather just prototype something and see what works.” “Hacker culture is also extremely open and meritocratic.” “Many of our most successful products came out of hackathons, [..].” <=> five core values: Focus on Impact (focus on solving the most important problems, be good at finding the biggest problems to work on); Move Fast (“[..] if you never break anything, you’re probably not moving fast enough.”); Be Bold (“Building great things means taking risks.”); Be Open (=> effort to make as much info as possible visible to all); Build Social Value (“[..] Facebook exists to make the world more open and connected, and not just to build a company. “)
- Dave Nicolett: I know how to tie my shoes – on the difficulty of convincing people to try unfamiliar software development techniques – “People change the way they operate when they are experiencing some sort of inconvenience or negative feedback. As long as things are going along reasonably well, people don’t go out of their way to change the way they work.” (with few exceptions) You can learn to tie your shoes in a split second, but why to invest the effort? You’d need to set aside assumptions, suppress habits, practice. You can argument there are many inconveniences (bugs, criticism for slow delivery, …) but “Unfortunately, that’s all pretty normal, and most people in the software field are accustomed to it. They don’t see it as a problem that calls for them to change their practices. Most of them probably have a hard time visualizing a different reality.” => Maybe that’s the reason there’s been no satisfactory answer to the question of how to convince people to adopt different practices. We shouldn’t be trying to convince people to do anything. We should be helping people solve their problems and achieve their goals. If they are satisfied with the outcomes they achieve using their current methods, then there is no problem to solve.
- Kent Beck: Pace of Progress = Pace of Feedback – ‘”The pace of my progress is completely constrained by the pace of my feedback”. If I want to go faster, it’s hard to achieve by going faster. I can almost always optimize my feedback loop, though.’ “The second lesson from this episode is that it’s not just the duration of the feedback loop that matters, it’s also the quality. All week I was working in tiny little iterations. Without producing useful information, though, those iterations could be as small or as large as I liked, I was still just going to spin my wheels.” => “The next time I seem to be going slow, I’m going to look at my whole feedback loop–duration, quality and my ability to respond to the information.“
- What Google Has Learned About How to Hire People – interview results have no relation to actual performance on the job: “We looked at tens of thousands of interviews, and everyone who had done the interviews and what they scored the candidate, and how that person ultimately performed in their job. We found zero relationship. It’s a complete random mess.” “Instead, what works well are structured behavioral interviews, where you have a consistent rubric for how you assess people, [..]” ‘Behavioral interviewing also works — where you’re not giving someone a hypothetical, but you’re starting with a question like, “Give me an example of a time when you solved an analytically difficult problem.”’ Link to an interesting book, Hiring Geeks That Fit.
Cool tech stuff
- The Elixir language – Clojure + Ruby + Erlang – a functional meta-programming aware language built on top of the Erlang VM; a dynamic language with flexible syntax with macros support that leverages Erlang’s abilities to build concurrent, distributed, fault-tolerant applications with hot code upgrades. First-class support for pattern matching, polymorphism via protocols, etc. (via @bodil)
- Random Testing seems to be gaining popularity and looks very interesting; at NDC Oslo, John Hughes has presented how QuickCheck, which generates random sequences of API calls, has been successfully used to find bugs in the Riak DB and a file system that a human would never think of, and Stuart Halloway has presented simulation testing with Simulant, which runs predefined actions according to a probabilistic model (e.g. 100 traders, each having 1h mean time between trades and mean traded amount 100, the test runs for 4 simulated hours). Something worth exploring!
- Dmytro Navrotskyy’s collection of Frontend Development resources and learning materials for tools (grunt, unused css detection,..), best practices (Atomic Design, …), JS/CSS frameworks, typography, animation, visualization, useful on-line services, and many more (via Herman Schistad)
- The Secret To 10 Million Concurrent Connections – The Kernel Is The Problem, Not The Solution: To have really fast SW, you need to implement your own core services (FS, net driver (packet handling), thread scheduling, ..) tuned for your app. You need to be aware of the clock-time cost of cache misses, memory access etc.. Custom solutions are times faster than what the general OS kernel can offer. => “data plane oriented system” Core areas and solutions for them: packet scalability, multi-core scalability (locks are expensive), memory scalability.
- M. Fowler: EmbeddedDocument – a pattern for working with JSON flowing in/out of our services (REST <-> JSON-friendly DB) without unnecessary conversions but with good encapsulation; naive approach: json -> object graph -> (processing) -> json; “In many of these situtiations a better way to proceed is to keep the data in a JSONish form, but still wrap it with objects to coordinate manipulation.” – use a lib to parse the JSON into a generic structure (e.g. a structure of lists, and maps/dicts) and store in a field of an object defining methods that encapsulate it – f.ex. for an Order we could have a method returning the customer and another computing the cost, accessing the underlying generic structure. The user of the wrapper object doesn’t need to know/care about the underlying structure. “The sweet spot for an embedded document is when you’re providing the document in the same form that you get it from the data store, but still want to do some manipulation of that data. [..] The order object needs only a constructor and a method to return its JSON representaiton. On the other hand as you do more work on the data – more server side logic, transforming into different representations – then it’s worth considering whether it’s easier to turn the data into an object graph.”
- ThoughtWorks’ Approach To Big Data Analytics – an inspiring, brief read. Some really good points such as “It’s not about Data. It’s about Insight and Impact” => “focus on the questions you’d love to answer for your business” => “changing big data from a technological problem to a business solution.” Also “The value of data is only realised through insight. And insight is useless until it’s turned into action.” Measure the value you gain at each step. See Introducing Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing by Ken Collier
- Wired.com, Nassim Taleb: Beware the Big Errors of ‘Big Data’ – in big data, noise has much stronger effect and in a large enough dataset we will always find spurious (i.e. false) relationships => beware! “Well, if I generate (by simulation) a set of 200 variables — completely random and totally unrelated to each other — with about 1,000 data points for each, then it would be near impossible not to find in it a certain number of “significant” correlations of sorts. But these correlations would be entirely spurious.”
- A Taste of Salt: Like Puppet, Except It Doesn’t Suck – a deescription of Salt and the tools around by an enthusiastic user with deep experience with Puppet. Highlights: Light-weight communication over ZeroMQ, very active community, simplicity, configuration is YAML, Salt-cloud can spin instances in EC2/Openstack/…, Salt-virt does the same for virtual machines (KVM/Xen/…), Salt-vagrant, Salt-monitor (work in progess) can ask all the server for their stats. “Having stood up a number of different configuration management systems across a wide variety of environments, I’ve yet to find a solution that’s as rapid to deploy, simple to scale, or as well architected as Salt.”
- Trash Your Servers and Burn Your Code: Immutable Infrastructure and Disposable Components – leveraging the lectures of PF to have a stable infrastructure – instead of updating servers, throw them away and create a new one from scratch (requires virtualization/cloud); this is something that Netlfix is doing and also Comoyo is moving towards
- Robin Ward: AngularJS vs Ember – a nice overview of the different approaches of the two; the author is strongly pro-Ember, claiming that AngularJS is much closer to low-end libraris like Backbone/Knockout and that you will often need the additional features of Ember. The comments provide the right countrweight to the biased post and form thus a good whole together.
- Scala Productivity. A Survey of the Community – people (that asnwered) seem to be productive with Scala right from the start
- After Your Job Is Gone – an interesting essay on the future, which, according to the author, we can already see happening, when technology will take away most of our work and we will not need to work all day. Not very optimistic, though (the author predicts few reach and many poor people).
- Clojure Cup 2013, Sept 28-29 – create something cool with Clojure/ClojureScript within 48h and perhaps win a price! #fun
- Clojure use in the industry – examples at an e-mail forum – Netflix, Puppet Labs (e.g. PuppetDB), UBS (talk), Deutsche Bank (talk, some details), Citigroup (reportedly “the largest private sector deployment of Clojure to date,” 11/2012), getprismatic.com (with frontend moving to ClojureScript; -> Why Prismatic Goes Faster With Clojure), Roomkey.com (details in a Relevance podcast), MastodonC.com (big data), Trend Micro, Walmart, beanstalkapp.com, ReadyForZero.com (50kLoC), www.cognician.com (20kLoC), World Singles (13kLoC) and more… (another similar thread)
- Stuart Sierra’s My Clojure Workflow, Reloaded (6/2013) – mainly about reloading changes into REPL, working around things that are not reloaded/left over => restart the app from scratch after significant changes => the app as a transient object => no global state, careful management of resources, :dev profile with :source-paths to a dir with user.clj (autoloaded by repl, pre-loading useful stuff) and dev util deps
- Adam Bard’s walk-through useful Clojure libs – f.ex. clojure.[data.[csv xml json] inspector java.shell java.browse xml], tools.logging, clojure.core.[match logic typed contracts …]
- Juxt.pro: Jon Pither’s and Malcolm Sparks’ “network of experienced IT professionals who specialise in the Clojure programming language,” providing training, consulting, talks
- Anthony Grimes: The Clojure Community and Me (2011) – an exciting insight into the embracing and supportive Clojure community
- In Clojure-based Machine Learning: “Our backend is 99.4% coded in Clojure, and 66% of the team [of 3] had never programmed seriously in any Lisp, let alone Haskell or Prolog (heck, not even I (the remaining 33%) had actually tried anything non-mainstream for real in a big project!) Maybe some Ruby, and lots and lots of Java and C and C++. But they accepted the challenge after reading around and learning the basics, and 3 months later you couldn’t take Clojure from their prying hands.”
- J. Pither: TDD and Clojure – “If you were to create a shopping list of things you really want for your development experience then what would you put at the top?” => 1. rapid feedback on changes, 2. REPL (place to explore and to play with your code <=> TDD), 3. FP and Immutability (“FP and dynamic languages lead to a lot less code. There’s less ceremony, less modeling. Because you’re managing less code you do less large scale refactorings.” => TDD needed less), 4. Regression Tests (“It’s my current opinion that what you get left out of TDD once you have amazingly fast feedback and a REPL is regression testing.”)
- More beautiful and colorful git log, by Filipe Kiss (via @lcdutoit) You may also want to have a look at tig, which is a text-based UI for git with the default view similar to git log.
- jq (via lcdutoit) – sed/awk/grep for JSON – slice, filter, map, transform structured data
- SemanticMerge (Windows) – a Java-aware merge tool – free beta (I haven’t tried it)
- JetLang – a high performance java threading library for in-memory messaging, based upon Retlang (via @tastapod, used likely in a trading SW).
Agile [is] in NOT a process — it’s a philosophy.
– Joe Wroblewski in a comment to a blog post
Teach culture first, then process and techniques
– Jeff Patton in Agile development is more culture than process
If a candidate came telling me that s/he wanted to program only in, say, Java because that’s what s/he knows best and that s/he doesn’t really feel prepared or interested in learning and using, say, Clojure (or any other language, really), I wouldn’t hire her/him in a million years, no matter what language my project were using, and no matter how many thousands of candidates like this one I had at my disposal.
– José Antonio Ortega in Clojure-based Machine Learning
We shouldn’t be trying to convince people to do anything. We should be helping people solve their problems and achieve their goals. If they are satisfied with the outcomes they achieve using their current methods, then there is no problem to solve.
– Dave Nicolett in I know how to tie my shoes
Posted in General, Languages, Testing, Tools, Top links of month | Tagged: bigdata, clojure, DevOps, fp, frontend, Git, java, lean, performance, simulation, waste | Comments Off on Most interesting links of June ’13
Posted by Jakub Holý on June 16, 2013
I have finally managed to understand one of the most unusual databases of today, Datomic, and would like to share it with you. Thanks to Stuart Halloway and his workshop!
As we shall see shortly, Datomic is very different from the traditional RDBMS databases as well as the various NoSQL databases. It even isn’t a database – it is a database on top of a database. I couldn’t wrap my head around that until now. The key to the understanding of Datomic and its unique design and advantages is actually simple.
The mainstream databases (and languages) have been designed around the following constraints of 1970s:
- memory is expensive
- storage is expensive
- it is necessary to use dedicated, expensive machines
Datomic is essentially an exploration of what database we would have designed if we hadn’t these constraints. What design would we choose having gigabytes of RAM, networks with bandwidth and speed matching and exceeding harddisk access, the ability to spin and kill servers at a whim.
Read the rest of this entry »
Posted in Databases | Tagged: clojure, database, datomic, fp, performance | Comments Off on Making Sense Out of Datomic, The Revolutionary Non-NoSQL Database
Posted by Jakub Holý on April 1, 2013
The TechEmpower’s Web Framework Benchmark is quite interesting but the comments following it at HackerNews are even more so. That is at least the constructively critical ones that highlight many of the issues with benchmarks while also reminding us of their value. One could formulate the benchmark paradox:
Benchmarks are important for rational technological choices yet it is very hard if not impossible to perform them in a sensible way.
I would like to record here some of the important points, mainly as a future reference for myself for whenever I will be dealing with benchmarking.
Read the rest of this entry »
Posted in General, Testing | Tagged: benchmark, opinion, performance | Comments Off on The Value and Perils of Performance Benchmarks in the Wake of TechEmpower’s Web Framework Benchmark
Posted by Jakub Holý on January 31, 2013
- Dustin Marx: Significant Software Development Developments of 2012 – Groovy 2.0 with static typing, rise of Git[Hub], NoSQL, mobile development (iOS etc.), Scala and Typesafe stack 2.0, big data, HTML5, security (Java issues etc.), cloud, DevOps.
- 20 Kick-ass programming quotes – including Bill Gates’ “Measuring programming progress by lines of code is like measuring aircraft building progress by weight.”, B.W. Kernighan’s “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”, Martin Golding’s “Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.” (my favorite)
- How to Have a Year that Matters (via @gbrindusa) – do you want to just survive and collect possessions or do you want to make a difference? Some questions everybody should pose to him/herself.
- Expression Language Injection – security defect in applications using JSP EL that can sometimes leads to double evaluation of the expressions and thus makes it possible to execute data supplied by the user in request parameters etc. as expressions, affects e.g. unpatched Spring 2.x and 3.
- HN discussion about Scala 2.10 – compilation speed and whether it matters, comparison of the speed and type system with Haskell and OCaml, problems with incremental compilation (dependency cycles, fragile base class), some speed up tips such as factoring out subprojects, the pros and cons of implicits etc.
- Blog Mechanical Sympathy – interesting posts and performance tests regarding “writing software which works in harmony with the underlying hardware to gain great performance” such as Memory Access Patterns Are Important and Compact Off-Heap Structures/Tuples In Java.
- Neal Ford: Functional thinking: Why functional programming is on the rise – Why you should care about functional programming, even if you don’t plan to change languages any time soon – N. Ford explains the advantages of FP and why FP concepts are spreading into other languages (higher abstractions enabling focus on the results over steps and ceding control to the language, more reusability on a finer level (higher-order functions etc.), few generic data structures with many operations -> better composability, “new” and different tool such as lazy collections, shaping the language towards the problem instead of vice versa, aligning with trends such as immutability)
- Neal Ford: Java.next: The Java.next languages Leveraging Groovy, Scala, and Clojure in an increasingly polyglot world – a comparison of these languages with focus on what they are [not] suitable for, exploration of their paradigms (static vs. dynamic typing, imperative vs. functional)
- How to Completely Fail at BDD – a story of an enthusiastic developer who tried to make everyone’s life better by introducing automated BDD tests and failed due to differences in culture (and inability to change thinking from the traditional testing), a surprising lack of interest in the tool and learning how to write good tests: “Culturally, my current team just isn’t ready or interested in something like this.” Morale: It is hard to change people, good ideas are not enough.
- M. Feathers: Refactoring is Sloppy – refactoring is often prioritized out of regular development and refactoring sprints/stories aren’t popular due to past failures etc. An counter-intuitive way to get refactoring in is to imagine, during planning, what the code would need to be like to make it easy to implement a story. Then create a task for making it so before the story itself and assign it to somebody else then the story (to force a degree of scrutiny and communication). “Like anything else in process, this is medicine. It’s not meant to be ‘the way that people do things for all time’ [..]” – i.e. intended for use when you can’t fit refactoring in otherwise. It may also make the cost of the current bad code more visible. Read also the commits (f.ex. the mikado method case).
- Cyber-dojo: A great way to practice TDD together. Compare your read-green cycle and development over time with other teams. Purposefully minimalistic editor, a number of prepared tdd tasks.
- On the Dark Side of “Craftsmanship” – an interesting and provoking article. Some developers, the software labouers, want to get work done and go home, they haven’t the motivation and energy to continualy spend time improving themselves. There is nothing wrong with that and we shouldn’t disparge them because of that. We shouldn’t divide people into craftsmen and the bad ones. A summary of and response to the varied reactions follows up in More on “Craftsmanship”. The author is right that we can’t expect everybody to spend nights improving her/his programming skills. Still they should not produce code of poor quality (with few exceptions) since maintaining such code costs a lot. There should be time for enough quality in a 9-5 day and people should be provided with enough guidance and education to be able to write decent code. (Though I’m not sure how feasible it is, how much effort it takes to become an acceptable developer.) Does the increased cost of writing (an learning to write) good code overweight the cost of working with bad code? That is an eternal discussion.
Cloud, web, big data etc.
- Whom the Gods Would Destroy, They First Give Real-time Analytics (via Leon) – a very reasonable argument against real-time analytics: yes, we want real-time operational metrics but “analytics” only makes sense on a sensible amount of data (for the sake of statistical significance etc.) RT analytics could easily provide misguided results.
CAP Twelve Years Later: How the “Rules” Have Changed (tl;dr, via @_dagi) – an in-depth discussion of the CAP theorem and the simplification (2 out of 3) that it makes; there are many more nuances. By Eric Brewer, a professor of computer science at the University of California, Berkeley, and vice president of infrastructure at Google.
- ROCA: Resource-oriented Client Architecture – “A collection of simple recommendations for decent Web application frontends.” Server-side: true REST, no session state, working back/refresh etc. Client: semantic HTML independent of layout, progressive enhancement (usable with older browsers), usable without JS (all logic on the server) etc. Certainly not suitable for all types of apps but worthwile to consider the principles and compare them with your needs.
- Vaurien, the Chaos TCP Proxy (via @bsvingen) – an extensible proxy that you can control from your tests to simulate network failure or problems such as delays on 20% of the requests; great for testing how an application behaves when facing failures or difficulties with its dependencies. It supports the protocols tcp, http, redis, memcache.
- Wvanbergen’s request-log-analyzer for Apache, MySQL, PostgreSQL, Rails and more (via Zarko) – generates a performance report from a supported access log to point out requests that might need optimizing
- Working Effectively With iTerm2 (Mac) – good tips in the body and comments
A very good (though not very scientific) definition of project success applicable for distinguishing truly agile from process-driven projects:
[..] a project is successful if:
- Something was delivered and put to use
- The project members, sponsors and users are basically happy with the outcome of the project
– Johannes Brodwall in “How do we become Agile?” and why it doesn’t matter, inspired by Alistair Cockburn
(Notice there isn’t a single word about being “on time and budget”.)
Posted in General, Languages, Testing, Tools, Top links of month | Tagged: bigdata, clojure, cloud, fun, human, java, performance, SbE, scala, security, tdd, Testing | Comments Off on Most interesting links of January ’13