Becoming A Better Programmer Through The Study of Good And Bad Code & Design

Reading books about good design and good coding practices such as Clean Code is very helpful but it isn’t enough to become a good programmer. We need to see both good and bad code in practice, perhaps many times, to start to really understand and appreciate the principles and qualities of clean/good code. (And, of course, we must write code.) However our chances of encountering a noteworthy good or bad piece of code and realizing its qualities (or lack thereof) are limited and highly dependant on the project and people we work with. In an attempt to increase the chances and help other – especially junior – developers to encounter and evaluate more interesting pieces of code, I have started a new blog, Wonders of Code. When I encounter a code snippet lacking in some qualities, I re-implement it in a better way and publish both together with an analysis of their pros and cons and relating those to the  principles of readability, maintainability, and clean code in general.

This is an experiment and I hope to hear from the community if this is something that can really help people or not. I would also love to get contributions from other developers, to cover a broader range of opinions and examples. Comments and contributions are welcomed and appreciated!

Most interesting links of April ’13

Recommended Readings

The top top article

How To Survive a Ground-Up Rewrite Without Losing Your Sanity (recommended by Kent Beck) – sometimes you need to actually rewrite an important part of a system; here we learn about two such rewrites, one which went well and one that failed badly – and what are the important differences.

The pain of a rewrite: “it’s [a major rewrite] going to take insanely longer than you expect” – because: “there’s this endless series of weird crap encoded in the data in surprising ways” and it takes days to convert them, “It’s brutally hard to reduce scope” (you cannot drop features, edge cases), “There turn out to be these other system that use ‘your’ data”.

To succeed you need: 1) Determine clear business-visible wins to justify the effort that will be much higher than expected and to know when to give up / what to postpone; 2) Do it extremely incrementally (<->  Succession) – break it into a series of small, safe steps, each generating a business value and learning of its own thus enabling early and frequent economical tradeoffs (stop, shift priorities, …) – ex.: rewrite a single reports, migrate its data, switch customers to it, go on to the next one – complete slice of functionality => a more realistic estimate soon => reprioritisation; incrementalism requires you to be able to write data both to the old and new system, which is hard but always pays off: “Here’s what I’m going to say: always insert that dual-write layer. Always. It’s a minor, generally somewhat fixed cost that buys you an incredible amount of insurance.” 3) “Abandoning the Project Should Always Be on the Table” (<- known biz value, better estimate based on early feedback).

Some Specific Tactics: Shrink Ray FTW (a graph of how much has been already replaced => motivation), Engineer The Living Hell Out Of Your Migration Scripts (tests, robustness, error handling, restartability), If Your Data Doesn’t Look Weird, You’re Not Looking Hard Enough.

Methodology, agile, lean

• M. Fowler: The New Methodology – a good description of the rise of Agile, the motivation for it, the various Agile methodologies (XP, Lean, Scrum etc.) and what is required to be able to apply an agile approach. Main points: Agile is adaptive (vs. predictive) and relies heavily on people and their judgement and skills (vs. treating them as same, replacable units) – which also leads to the need of leadership instead of (command&control) management. Discusses unpredictability of requirements and scope, foolishness of separating design and implementation, difficulty of measurement of SW development, continuous improvement etc. Quotes: “However letting go of predictability doesn’t mean you have to revert to uncontrollable chaos. Instead you need a process that can give you control over an unpredictability. That’s what adaptivity is all about.”
• The Toyota concept of ‘respect for people’ – many state that they respect their workers but fail to really understand what it means; it is not about freedom of act, it is about a mutual respect, leveraging the strengths of each other: worker’s experience and insight and manager’s broader overview, as demonstrated by the problem-solving dialog and challenges (problem – root cause – solution – measure of success, the manager challenging the worker’s answers). Also a nice example how the evaluation of individual performance leads to a much worse system and high turnover compared to a whole-oriented company.
• Fixed Bid Agile Without Cognitive Dissonance – a refreshing take on fixed-scope projects and Agile; yes, they are bad but sometimes the client has no other choice so what best we can make out of it? The core advice: Agree “a pragmatic change management protocol (along with a contingency built into the pricing)” (push for lower initial requirements granularity, customer involvement, flexibility of functionality) => “you can gain significant agile benefits for clients who wouldn’t otherwise accept them”.
• Agile Atlas: Scrum – a good description of Scrum and its values, roles, artifacts, and activities

Learning, psychology, estimates

• How Developers Stop Learning: Rise of the Expert Beginner – sometimes you meet people with experience-indicating titles that are actually little competent, perhaps leading incompetent IT departments. Why? They, unchallenged by competent peers or broader IT community, came to believe that they are “experts” while actually being only little more advanced beginners, better than their beginner colleagues but still lacking any understanding of the big picture and the knowledge of what they do not know, trapped in the “unconscious incompetence” stage. The post explains this in a more detail and is followed up an explanation how it can lead to the rise of a mediocre SW group in “How Software Groups Rot: Legacy of the Expert Beginner“.
• Coding, Fast and Slow: Developers and the Psychology of Overconfidence (via @peterskeide) – why are we so bad at estimating (inherent complexity of SW vs. our overconfidence) and why it cannot be fixed. We can learn to somehow estimate tasks of few hours length (less complex, plenty of practice opportunities). The question is: “how you can your dev team generate a ton of value, even though you can not make meaningful long-term estimates?”
• Cognitive Overhead, Or Why Your Product Isn’t As Simple As You Think (via @JiriJerabek) – to make apps more accessible to users, we try to make them simple – but “simple” might be different from what you expect. The important thing is not less steps, less features, less elements, but lower cognitive overhead, i.e. “how many logical connections or jumps your brain has to make in order to understand or contextualize the thing you’re looking at.” Good examples of unexpectadly high / pleasantly low cognitive overhead, some tips, even suprising ones such as make people do more (to be more involved in the process – e.g. bump their phones), slow down your product.

Other

• Economies of Scala – a case for using Scala over Java, supported by data: many capable developers want to use it but there are few opportunities for them – and getting developers is one of the main challenges.
• A canonical Repository test – a nice standard way to test a “DAO”; highlights: use of  FEST assert 2 for clean and nice checks, no unimportant details in the test (f.ex. details of the test data hidden in randomPerson() and randomOder(Person)).
• How To Think Like An Engineer – some nice ideas such as: “Build A Simple First Version: With People, Not Code” – “Technology is not always the best solution, because technology is not always the simplest solution.”, i.e. don’t automate everything from the start (examples from Netflix, Amazon); “Rather than trying to do everything at once, break down the functions of your company into smaller goals.” – and focus at one at a time
• Economies Of Scale As A Service (do not mix up with Scala! )- an interesting description of the trend away from ownership to the rental of important resources (servers, manufacturing capabilities, personal cars, …) and the resulting changes in the society, business, and industry
• Troy Hunt: Our password hashing has no clothes (or the much shorter though biased How To Safely Store A Password) – MD5 and SHA are not safe enough due to brute-force attack enabled by GPUs, irrespective key size; it’s crucial to use hashing algorithms designed for passwords (and thus sufficiently slow) – f.ex. bcrypt, or PBKDF2 or the newer scrypt.
• Everything about Java 8 – a well-made summary of what should come in Java 8, based on the current state, discussing the finer points: static and default (non-static, overridable) methods on interfaces, lambdas (do I need to mentione that?!) and method references (String::valueOf, Object::toString, myVar::toString, ArrayList::new); good discussion of the various use cases and limitations of lambdas (capturing x non-c., ..); java.util.stream for functional operations on value streams (filter, map, reduce etc.); java.time inspired by Joda, more concurrency utilities (e.g. CompletableFuture for chaining futures); String.join (finally!), Optional ~ Scala’s Option & more; yummy!
• How To Keep Your Best Programmers – what motivates capable programmers to stay/leave? The author lists some common reasons and concludes that, ultimately, all are linked to the desire for autonomy, mastery, or purpose. However he goes further and proposes that, to keep talented devs, you must offer them an appealing narrative (regarding their actions and a result, related to autonomy/mastery/purpose) and reaffirm/update it frequently; ex.: “With the work that we’re giving you over the next few months, you’re going to become the foremost NoSQL expert in our organization.” “At any point, both you and the developers on your team should know their narratives.” – so that they will be “constant points of job satisfaction and purpose.”

Clojure Corner

• Clojure Data Analysis Cookbook review – “The book provides a collection of recipes for accomplishing common tasks associated with analyzing different types of data sets. It starts out by showing how to read data from a variety of sources such as JSON, CSV, and JDBC. [..] how to sanitize the collected data and sample large data sets. [..] a number of different strategies for processing it.” How to present them with ClojureScript and  NVD3 (D3.js components). “Some of the highlights include using the Clojure STM, parallel processing of the data, including useful tricks for partitioning, using reducers, and distributed processing with Hadoop and Casalog.”

Favorite Quotes

once again, trying to do it *and* do it right was too much all at once, resulting in little progress and little learning.

- Kent Beck’s tweet 2013-04-16

A true agile development process can be recognized by its continual evolution:

A project that begins using an adaptive process won’t have the same process a year later. Over time, the team will find what works for them, and alter the process to fit.

- Martin Fowler in The New Methodology

Book Review & Digest: Boyd: The Fighter Pilot Who Changed the Art of War (Relevant for IT/Business)

This book is about a great person, about change, about one of the largest bureaucracies and dysfunctional organizations, about projects gone astray, about warfare and its latest evolution. Many of the challenges and ideas that we encounter in the book are not limited to the military domain but apply also to business and IT. It is worth reading whether you are  a military person, somebody trying to push through a change, a business person, or interested in thinking and organizations.

The Value and Perils of Performance Benchmarks in the Wake of TechEmpower’s Web Framework Benchmark

The TechEmpower’s Web Framework Benchmark is quite interesting but the comments following it at HackerNews are even more so. That is at least the constructively critical ones that highlight many of the issues with benchmarks while also reminding us of their value. One could formulate the benchmark paradox:

Benchmarks are important for rational technological choices yet it is very hard if not impossible to perform them in a sensible way.

I would like to record here some of the important points, mainly as a future reference for myself for whenever I will be dealing with benchmarking.

Most interesting links of Mars ’13

Recommended Readings

A lot of stuff this month since I have finally got time to review some older articles. Quite a few articles by Fowler. Few really great (yet short) talks on agile & SW development.

Top

• Agile in a Nutshell (originally Agile Product Ownership in a Nutshell) by Henrik Kniberg – the best explanation of the agile development process ever, in just 15 minutes and with wonderful animation; every developer should see this. Some highlights: the most important task of product owner is to say NO so that backlog doesn’t grow infinitely; at start, the estimates of size and value will suck and that’s OK because the value is in the conversation, not in the numbers (that are anyway just relative); the goal is to maximize outcome (value), not output (# features). Compromises between short-term vs. long-term goals, knowledge vs. customer value building etc. Build the right thing (PO) x build it right (devs) x build it fast (SM). Technical debt x sustainable pace. As I said – you MUST see it.
• Martin Fowler: The Value of Software Design (talk, 22 min, from 0:45:00 til 1:07; Feb 2013) – a balanced argument for the value of good software design and internal code quality based on paying off by enabling us to keep our development speed. Discusses the DesignStaminaHypothesis (bad design => rapid decline of development speed), TechnicalDebt, TechnicalDebtQuadrant (Prudent x Reckless, Deliberate x Inadvertent), TradableQualityHypothesis. According to the experience of Fowler and others, the good design payoff point “it’s weeks, not months.”
• What Does It Take To Become A Grandmaster Developer? – great post about cognition and learning, valuable references, quotes from an interesting study of good vs. mediocre developers. We have mental capacity for ~7 chunks of information => great performers recognize patterns and see and understand thus higher-level chunks and have many “chunks” (patterns encountered previously) readily available. You need deliberate effort to learn more chunks – especially initially but you must always try to get out of your comfort zone to grow. Experienced collegues can help a lot in acending the learning curve.

Agile, organization, innovation, project management

• How to Prioritize a User Story Map – we all know that we should prioritize features by their value, risk, and lack of knowledge and that we should slice the features thin so that they fit into short iteration and can be deployed soon to produce feedback, right? Here we see a nice example of what happens if not done so and how to do feature slicing better.
• Bob Marshall: Rightshifting – according to the author, 80% of knowledge work organizations are very ineffective, wasting resources on non-value-adding activites; only few are effective, even fewer highly effective. Rightshifting is the attempt at shiting them to the right, towards higher effectiveness. Links to a few videos explaining it more. Related: Steve McConnell’s Business Case for Better Software Practices, referring to a study by SEI; “The actual distribution of software effectiveness is asymmetric. Most organizations perform much closer to the worst practice than to the best.” – the best performing 10 times better then the worst/average (productivity, speed, defects, value)
• On Antifragility in Systems and Organizational Architecture – introduces the concept of antifragility, based on Nassim Taleb’s book Antifragile that compares fragile, robust, and antifragile systems and organizational structures (which is also applicable to SW systems); robust = resists change (unless too large); antifragile: learn, adapt; closely related to DevOps and continous delivery
• M. Fowler: PurposeOfEstimation – many Agilist disdain estimation, this is a balanced view: “estimation is valuable when it helps you make a significant decision.” (F.ex. when deciding what we (don’t) have resources for or when in need of coordinating related activities.) It is evil when used as commitments that people are forced to stick to and blamed for not managing to do so. “Above all be wary of anyone who tells you they [estimates] are always needed, or never needed.” A. Ferguson: “[..] it is poor project management (whether by project managers or other team members) that results in a client who thinks estimates are fixed, or that raw estimates = actual effort/duration”.
• Ron Jeffries: Estimation is Evil – discusses the problems estimates can cause, issues with requirements gathering up front and their volatility, transparency and politics. Very valuable, highly recommended. See the “favorite quotes” at the bottom of this post. Also contains an interesting lesson learnt from the failed Chrysler C3 project: don’t try to build a grand new system to replace and fix the old one, fix one problem at a time – worth reading for this alone.
• Interview with Steve Blank: Why Big Companies Can’t Innovate – the 2013 list of the world’s 50 most innovative companies has only a few large, established firms (those that have built innovation into its DNA such as Apple and Google). Established companies are less innovative because they focus in their existing business model, have risk-aversion (while there are many failures on the way to a new business model); finally “the people who are best suited to search for new business models and conduct iterative experiments usually are not the same managers who succeed at running existing business units.” – and thus aren’t given the chance. “[..]  the process of starting a new business [..] is fundamentally different from running an existing one. So if you want your company to grow organically, then you need to organize your efforts around these differences.”

Architecture & Ops

• M. Fowler: Schemalessness + NoSQL and Consistency (20 + 20 min) two short, very good, balanced talks about NoSQL. He explains schemalessness and consistency and points out common misunderstanding about them so if you are into NoSQL, watch it.
• What Powers Instagram: Hundreds of Instances, Dozens of Technologies (2012) – interesting high-level overview of the Instagram infrastructure based on AWS and Python (25 XL instances running Django/Gunicorn behind ELB with 3 Nginxes, sharded PostgreSQL with streaming replication on 12 QXL mem instances with software raid and XFS to freeze when snapshoting, media in S3, Redis, Solr for geo-search, Memcached. Gearman for task queues, pyapns for notifications. Munin for monitoring.)
• The Netflix API Optimization Story – how Netflix redesigned its APIs to improve performance, reduce chattiness, and power product development and experimentation. The common REST API has become a development bottleneck and a lowest common denominator solution (w.r.t. supporting various clients). The main changes were: usage Hystrix for fault tolerance, each device team managing their own end-points in any JVM languges (primarily Groovy) and re-using common APIs (i.e. pushing some device-specific code to the server) => able to experiemnt more quickly, using the Functional Reactive Programming Model and asynchronous APIs (to abstract away thread-safety and parallel execution implementation details from the device teams so that code can execute sync. or async. without them needing to know).
• Me: Overview of current monitoring libs for Java – Netflix’ Servo, Yammer’s Metrics, JavaMelody, JavaSimon.
• Debug Servlets, or ‘HTTP Won; Use It’ – expose all debugging info of your services over HTTP – it makes debugging much simpler. We do a part of it and it really helps. Expose config (values, where they come from), logs, log configuration, JMX (setting it up otherwise not trivial), version, build number, git hash, server time (timezones tricky), metrics, stack dumps, app-specific status (Hadoop: live nodes, data size etc.). The author recommends JavaMelody to collect & visualize many common metrics. Not on security: Make sure to hide passwords and make the endpoints visible only internally. (Tip: consider Jolokia for exposing JMX over HTTP, see below.)
• JVM Crash/Core Dump Analysis – 3 common categories of  JVM crash causes (JVM/JIT/JNI) and how to recognize and troubleshoot them

Other

• How to lose wight in the browser:  The definitive front-end performance guide – a site by a number of experts from Twitter, Opera, Google, and other places with best practices for performant web sites (HTML, CSS, JS, jQuery, images). Ex.: styles up top, scripts down bottom; minify your html, css and JS; async script loading; combine css/JS files into one; cache array lengths while looping; use css sprites for icons.
• Luke Stevens: The harsh truth about HTML5′s structural semantics (part 1) – “HTML’s structural elements — article, section, nav and aside — are, at first glance, some of the easiest parts of the HTML5 specification to understand and implement. However, they’re actually some of the most poorly specified, poorly understood, and poorly implemented parts of HTML5.” Interesting: The “research” leading to their establishment was quite random, ignoring a crucial source of information (css IDs).
• Marco Emmanuel Patiño: Six non-technical books every programmer should read – 1. Team Geek: A Software Developer’s Guide to Working Well with Others (-> effective communication and collaboration), 2. The Pragmatic Programmer: From Journeyman to Master, 3. The Passionate Programmer: Creating a Remarkable Career in Software Development, 4. Clean Code: A Handbook of Agile Software Craftsmanship, 5. 97 Things Every Programmer Should Know: Collective Wisdom from the Experts (online), 6. Code Simplicity: The Fundamentals of Software.
  • Related: Top 5 Java programming books – Best of lot (actually 8) – 1) Head First Java, 2) Effective Java, 3) Thinking in Java, 4) Head First Design Pattern, 5) Concurrency Practice in Java, 6)Java performance, 7) Java Puzzlers, 8) Head First Object Oriented Analysis and Design.
• Humans as slaves of chemistry: America’s Real Criminal Element – Lead – a fascinating article about how whole nations can be seriously influenced by a single chemical substance. Aside of that it is also fascinating to observe how we tend to search for causes in our domain of expertise (police, sociologists, …) and of interest while denying other possible causes, no matter how strong are the proofs. If the facts presented are true, then the fivefold increase in serious crimes in (not only) America since 60s has been caused by the increase of lead in the environment (pushing many people over the edge of ocassional violent loss of control). How many social problems in the world have similar industrial causes? Are we careful enough with what we let into our air and bodies?

Languages

• newcoder.io: Learning more Python via projects – an excellent next step when you have learned Python syntax via LPHW or similar; in this tutorial series you will be building real-world apps while learning more of Python. You will play with, Data Visualization, APIs, Web Scraping, Networks, GUI.
• Brian McCallister: Go is PHP for the Backend – a very good explanation why you might want to use Go and that you have to first learn “the Go way” to avoid insanity, since it is very opinionated and different from what you might be used to. Some pros: “native code, UNIX friendly, higher level then C, lower level then Python or Ruby, garbage collected, strongly typed, good performance, good concurrency support, etc.”
• The Neophyte’s Guide to Scala 1 to 15 (list) – a good follow-up on the Cursera FP in Scala course, a series of blog posts exploring some topics more in depth. F.ex.: extracotrs (unapply, for pattern-matching), the broad applicability of pattern matching, pattern matching anonymous functions & partial functions #4, usiong Option idiomaticly #5, nice FP error handling with the Try type #6, Futures, etc. Higly recommended! Thx to Jakob Lind

Libs

• Jolokia is remote JMX with JSON over HTTP: a REST API bridged to JMX, with support for security, fine-grained access control, bulk operations. Especially useful if you either 1)  need to perform bulk operations (e.g. get multiple values) or 2) want to access them from something that doesn’t support JMX. JSON is in general very easy to use and navigate. You can install Jolokia as a WAR (or mebedd its Servlet), a JVM agent, or attach it on-the-fly to a running JVM.
• The Appeal and Controversy of ZeroMQ – why to use 0MQ? It is a messaging library that focuses on performance, decentralization and simplicity, solving some really hard problems (sending async. messages w/o locks, distribuing to specific subscribers) and providing a simple API. Main pros: decentralized (no central broker), many languages; cons: no security (but you can use it over SSH).

Talks

• Tim O’Reilly: Create More Value Than You Capture (30 min + questions) – build apps that matter, that change how we do things. Thinking just about money is bad. Try to make the society better, smart, create more value than you capture, solve important problems, help people. Ex. startups: Uber, Square, Code for America.
• TED: Bruce Feiler: Agile programming — for your family (20 min) – an inspirational talk, based on positive experience from multiple families, about applying the agile thinking and values to make our families happier by empowering the children (enlist them in their upbringing, deciding on goals, rewards, punishments), letting them know who they are, being adaptive, having regular “retrospectives” (that eventually become cherrished memories). Backed by research. Did you know that the #1 wish of children isn’t that parents spend more time with them but that they are less stressed?

Clojure Corner

• What’s new in Clojure 1.5.x – reducers, new threading macros (cond->, as->, some->, ..), various improvements, improved performance, erro messages, doc strings, bug fixes
• Stuart Sierra: On the Perils of Dynamic Scope – summary: don’t create macros like with-connection binding to a thread-local var; make all methods take the resource as a parameter – thus the user has the freedom to decide when to close the resource and isn’t limited to a single thread and can use lazy sequences
• Logic programming is overrated – core.logic is essentially only a complex DSL for doing an exhaustive search and there is already a nice, clean tool for that: the for comprehension. A logic puzzle can be much more clearly and also efficiently using for. But it is not completely useless – logic programming is good e.g. for running programs backwards, unification is important for writing type checkers, and the new constraint programming piece has good potential. Read also Logic Programming is Underrated, which provides a faster core.logic solution than for-comprehension and provides some pointers rgarding the practical usefulness of core.logic.
• Prismatic – Graph: Abstractions for Structured Computation – How to reduce the complexity overhead in large, real-world, FP systems by decoupling what is done from how it is executed. Graph is a Clojure library enabling a declarative way to describe how data flows between (mostly pure) functions => “It allows us to formalize the informal structure of good FP code, and enables higher-order abstractions over these structures that can help stamp out many persistent forms of complexity overhead.” By decoupling the description of how data flows and the actual execution, we can execute it in different ways (parallelized, with memoization, lazy/eager) and apply various interceptors (for logging etc.). See especially the part “Graph and complexity overhead.”
• Mike Anderson: Game development in Clojure : Alchemy 7DRL post-mortem (and the previous 7 daily updates, Alchemy @ GitHub) – an interesting report about game making in Clojure during 7 days, in as functional and immutable style as possible while keeping it sufficiently fast. How do you represent & handle statuf game objects, the world map, game state? The design of the game, what was easy and what hard with Clojure. Tl;dr: search it for “Some parting thoughts” (Clojure productive, immutability hard but pays off, prototype objects great, more typing would have helped). “Making everything immutable in Clojure is harder than it would have been in an OOP language like Java where everything can be encapsulated in mutable classes. In particular, the state update functions are tricky to make both correct and performant. The payoff is big however: in terms of the simplicity and effectiveness later on, and in the conceptual clarity being able to treat the entire game state as an immutable value”. Having REPL is a big win.
• Refactoring Java using Clojure with the Eclipse Java development tools (JDT) (operation on AST nodes, i.e. little too low level; the Eclipse Refactoring API might be better)
• Clojure at a Bank – [Our] Clojure Code Immaturity – experiences with going from Java to Clojure: 1) too few comments, too short names => hard to learn the code; 2) not knowing clojure.core well enough => reimplementing (if-let, juxt, …); 3) structure, comment, split up your namespaces well, navigating more complicated then in Java IDEs; 4) reasonably used Macros, Protocols, Defrecords payed off;
• Datomic for Five Year Olds – explaining the key characteristics of Datomic compared to relational and NoSQL DBs (schema, architecture, programmability/language); doesn’t go into details of how it works (e.g. how does Datomic determine what subset of the DB to cache locally and what if it is few GBs); Honey Badger’s 2012 talk Exploring Datomic: a database deconstructed explores the architecture and technical details much more

Tools

• Vagrant 1.1.0 is out (what’s new?), with support for VMWare Fusion and AWS VM backends in addition to VirtualBox – use the same config to create, provision, stop, destroy and connect to a virtual machine locally or in the cloud (with limited support for shared folders, I’d suppose). V. 1.1 is backwards compatible aside of plugins, upgrade to new config optional.
• Animated presentations: ArtRage (drawing program, also for iPad), Wacom Intuos 5 (drawing tablet), Screenflow (screen & audio capture) – used for the Agile in a Nutshell (Agile Product Ownership in a Nutshell) mentioned above
• ckjm — Chidamber and Kemerer Java Metrics (via Neal Ford) – a command-line tool (also Maven/Ant plugin) to compute some metrics, outputting text or XML for further processing; the metrics: WMC: Weighted methods per class (cyclomatic complexity of its methods), DIT: Depth of Inheritance Tree, NOC: Number of Children, CBO: Coupling between object classes, RFC: Response for a Class, LCOM: Lack of cohesion in methods, NPM: Number of Public Methods, Ca: afferent coupling.
• Bulletproof Demos: Record & Replay built into Chrome – ever got a failure while demonstrating a web app though it has worked moments ago? No more! You Chrome to record your requests and responses and let its cache handle them during the real demonstration. (Mac: stop Chrome, to record run open -a “Google Chrome” –args –record-mode, to replay run open -a “Google Chrome” –args –playback-mode. Linux: google-chrome –record-mode and –playback-mode. Win.: run chrome <arg>)
• UserTesting.com (via Ash Maurya, the author of Running Lean) – on-demand usability testing; they have a large base of test users, can select those matching your criteria and unleash them upon your site guided by a script your provide, watch videos of their actions while they verbalize their thinking process, recieve written answers from them, talk to them.
• MindMup.com – opensource, free mind-mapping in the cloud by Gojko Adzic & co., with main focus on productivity. Store private maps in Goolge Drive, support for mobile devices, keyboard shortcuts. No registration needed.

Favorite Quotes

Once we estimated a project to require 9 man-months but were later told that we do not understand a thing and it may not take more then 4. At the end it took over 25 and still wasn’t done.

- paraphrasing my collegue Kim Leskovski

On collecting requirements up front:

At the very beginning, we know less about our project than we’ll ever know again. This is the worst possible moment to be making firm decisions about what we “require.”

- Ron Jeffries in Estimation is Evil: Overcoming the Estimation Obsession

On the estimate of project delivery date at its initial phase:

It’s based on an unrealistic list of requirements, using weak estimates, made at the moment of maximum ignorance, by people who are always optimistic about their own abilities.
- ibid

On planning and requirements (the Chrysler’s C3 payroll project, having a payroll expert and a team familiar with the domain):

This was one of the best-planned projects I’ve ever seen, and even so, at least one third of the requirements were added, removed, or substantially changed.
- ibid

[..] a line of code is a liability, not an asset [..]
- Jez Humble in Why Software Development Methodologies Suck

Agile is not something you do, it is something you are.
- Huib Schoots in Creating my own flow
with personal kanban, Agile Record Feb 2013

Escaping the Zabbix UI pain: How to create a combined graph for a number of hosts using the Zabbix API

This post will answer two questions:

• How to display the same item, f.ex. Processor load, for a number of hosts on the same graph
• How to avoid getting crazy from all the slow clicking in the Zabbix UI by using its API

I will indicate how it could be done with plain HTTP POST and then show a solution using the Python library for accessing the Zabix API.

The problem we want to solve is to create a graph that plots the same item for a number of hosts that all are from the same Host group but not all hosts in the group should be included.

From Stateful Iteration in Python to Stateless Clojure

I have a piece of Python code that leverages a stateful object and was wondering how to reimplement it in the inherently stateless Clojure, without resorting to its facilities for managing state. It turns out to be simple and beautiful.

The core of the code is a for loop that transform a list of maps, while also using input from another iterator-like object, a Palette. The Palette has the method next() that remembers the last color used and returns the next one. Here it is:

class Palette:
    last_idx = 0
    colors = ["C04000", ...]

    def next(self):
        self.last_idx = (self.last_idx + 1) % len(self.colors)
        return self.colors[self.last_idx]

graph_items = []

for idx, item in enumerate(item_maps):
    graph_items.append({
            "itemid": item['itemid'],
            "color": palette.next(),
            "sortorder": idx
            })

In Clojure, we would rather have no such state (for reasons I won’t discuss now). However, when writing idiomatic Clojure, we actually don’t need to.

We will use the fact that map – the function that can transform collection items – can take not just one but any number of collections, iterating over all of them in parallel. Instead of using the stateful palette.next(), we will create an infinite sequence of (repeating) colors and pass it into map together with the items collection. We want also the index of the items so we pass in a third sequence, the range from 0 to infinity. The map function will be thus called with (the current index, the current item, the current color).

Using infinite sequences is quite common (and elegant) in Clojure.

This is what the code could look like:

;; An infinite sequence of the colors (starting
;; again with the 1st one, when all exhausted)
(def palette (cycle ["C04000" "..."]))

(defn make-graph-items [items]
  (map
   (fn [idx item color] {"itemid" (item "itemid"), "color" color, "sortorder" idx})
       (range) items palette))

(def graph-items (make-graph-items items))

Conclusion

Statelessness is not scary. It is fun.

Disclaimer: I am only learning both Python and Clojure.

Books Our Developers Should Read

Republished from blog.iterate.no with the permission of my co-author, Morten Berg, and later updated.

There are a few books that every developer in Iterate should read because they express what we believe in and are extremely valuable in themselves. The books chosen are generally and broadly useful and not tied to some too limited domain (contrary to e.g. Effective Java).  The list is kept as short as possible, about 4-5 books, and is revised regularly.

Why particularly these books, why lean and agile? Our people are primarily responsible for crafting solutions for our clients, for making sure that they use the customers’ limited resources efficiently to produce the maximal business value possible. However, according to our experience, it is never truly known upfront where that value lies. Software development is therefore inherently a learning and exploration process. A process that needs to be continually adjusted based on empirical feedback from the reality and on shifting conditions. This is what lean is about: eliminating waste, maximizing value by maximizing learning, making sure that the right product is built. We value software craftmanship and building things right – but building the right things is crucial.

Here are the books and why we believe they are so important.

Most interesting links of February ’13

Recommended Readings

• Øyvind Bakksjø: The inverse of IoC is Control – a well-founded criticism of the (over)use of inversion-of-control containers such as Spring and Guice. Bakksjø isn’t against dependency injection in general but he prefers to do it manually, instead of magically, in a main method, using Java instead of an obscure DSL/XML. The dependencies are typically known at compile time so why postpone assembling (and defect discovery) to runtime? Why hide how parts fit together into the non-transparent magic of IoC? He has many good points though I believe there are cases when some automation of the wiring process is valuable (think of scope-dependant beans, diff. deps in diff. environments etc.) Comment by B. Borud: “Explicit wiring you can read from start to finish beats vague declarative shit that you may be able to figure out if you pay close attention.” Check out also the comments at HN.
• J. B. Rainsberger: Demystifying the Dependency Inversion Principle – a very good explanation of the value of dependency injection in the terms of general good code principles (promoting abstractions and postponing determination of details to the latest moment/client)
• CodingHorror: Coding Without Comments (2008) – a very good explanation if how to treat comments (similar what Clean Code says): write code so that it is self-explanatory (using good method and variable names), if necessary, add comments that explain why (while the code shows what/how).
• Frustrations and Aspirations of a Software Craftsman – what makes us happy or unhappy on a project? (I could add some too.) Bad: bureaucracy, old/unfit technology, lack of autonomy and credibility, uninteresting domain, demotivated people, blaming and highly competitive environment, arrogant and unskilled people, … => it is mostly about people. Good: projects where there is passion, craftsmanship, friendship and trust.
• Jetty-9 goes fast with Mechanical Sympathy – interesting how the run-time behavior might differ from what we would expect and how knowing the hardware can improve performance. Here:  false sharing of a blocking queue’s head/tail pointers and locks (close => same CPU cache row => updating one invalidates the other), using trie backed directly by IO buffers for faster String lookups etc. Result (all microbenchmark disclaimers): jetty-9 out performs jetty-8 by 30% faster and 50% less YG garbage.

Czech

• Daniel Kolman: Jak správně pojmenovat test – a good description of different approaches to name and organize tests w.r.t. the experience of the developer – starting with “testWorks1″ through one method-one test with “testMyMethodName” through a more thoughtful naming scheme such as Roy Osherove’s UnitOfWork_StateUnderTest_ExpectedBehavior (e.g. ndexOf_containsSearchedString_returnsCorrectIndex) to tests as a living documentation: tests represent a list of features of the class, each name is a sentence having the class under test as its subject (e.g. [list] holdsItemsInTheOrderTheyWereAdded). Advantages of the latter: readability, documentation, it forces us to think what actually is still the responsability of the class and what isn’t. “Stačí názvům testů věnovat dostatečnou pozornost a ony vám samy napoví, kdy už toho třída dělá moc a měli byste se zamyslet, jak kód lépe rozdělit.”

Non-IT

• Why You Should Work With Someone You Hate – working with people we like and do not have conflicts with is nice but we are likely to have similar views and ideas and miss the broader picture. Working with somebody that drives you crazy while being able to respect each other is very valuable because it opens you to different views and forces you to really defend yours. Recommended!
• Kent Beck: Focusing Talks: Three Questions – some good advices for performing technical talks and avoiding putting too much into the talk. Ask 1) Who is the audience? – be as specific and narrow as possible to be able to tune and cut the content; “Better to reach one person than zero.” 2) What is my one startling sentence? – one that makes the audience take notice; “Picking one sentence, I encourage focus and empathy in myself. Now I have one person’s attention. What do I say?” 3) What would I like the audience to do? “‘To what purpose?’ helps me discard interesting-but-tangential points, to, as Peter Jackson put it, move the ring.” “The more clear and uncompromising I can be with the three questions, though, the easier all the rest of it becomes.” “Talk about less. And less. Less than that.”

Clojure Corner

• Clojure’s Reader is Unsafe (for untrusted strings) – don’t use read-string to read untrusted strings – even if you set *read-eval* to false, it can still be persuaded to execute arbitrary code (hopefully fixed in Clojure 1.5). The right, save way is to use clojure.tools.reader.edn/read-string.

Tools

• Android: Clock to use when giving presentations – presentation countdown apps for Android
• Netflix open-sourced its Chaos Monkey and the whole SymianArmy

Most interesting links of January ’13

Recommended Readings

Various

• Dustin Marx: Significant Software Development Developments of 2012 – Groovy 2.0 with static typing, rise of Git[Hub], NoSQL, mobile development (iOS etc.), Scala and Typesafe stack 2.0, big data, HTML5, security (Java issues etc.), cloud, DevOps.
• 20 Kick-ass programming quotes – including Bill Gates’ “Measuring programming progress by lines of code is like measuring aircraft building progress by weight.”,  B.W. Kernighan’s “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”, Martin Golding’s “Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.” (my favorite)
• How to Have a Year that Matters (via @gbrindusa) – do you want to just survive and collect possessions or do you want to make a difference? Some questions everybody should pose to him/herself.
• Expression Language Injection – security defect in applications using JSP EL that can sometimes leads to double evaluation of the expressions and thus makes it possible to execute data supplied by the user in request parameters etc. as expressions, affects e.g. unpatched Spring 2.x and 3.

Languages etc.

• HN discussion about Scala 2.10 – compilation speed and whether it matters, comparison of the speed and type system with Haskell and OCaml, problems with incremental compilation (dependency cycles, fragile base class), some speed up tips such as factoring out subprojects, the pros and cons of implicits etc.
• Blog Mechanical Sympathy – interesting posts and performance tests regarding “writing software which works in harmony with the underlying hardware to gain great performance” such as Memory Access Patterns Are Important and Compact Off-Heap Structures/Tuples In Java.
• Neal Ford: Functional thinking: Why functional programming is on the rise – Why you should care about functional programming, even if you don’t plan to change languages any time soon – N. Ford explains the advantages of FP and why FP concepts are spreading into other languages (higher abstractions enabling focus on the results over steps and ceding control to the language, more reusability on a finer level (higher-order functions etc.), few generic data structures with many operations -> better composability, “new” and different tool such as lazy collections, shaping the language towards the problem instead of vice versa, aligning with trends such as immutability)
• Neal Ford: Java.next: The Java.next languages Leveraging Groovy, Scala, and Clojure in an increasingly polyglot world – a comparison of these languages with focus on what they are [not] suitable for, exploration of their paradigms (static vs. dynamic typing, imperative vs. functional)

SW development

• How to Completely Fail at BDD – a story of an enthusiastic developer who tried to make everyone’s life better by introducing automated BDD tests and failed due to differences in culture (and inability to change thinking from the traditional testing), a surprising lack of interest in the tool and learning how to write good tests: “Culturally, my current team just isn’t ready or interested in something like this.” Morale: It is hard to change people, good ideas are not enough.
• M. Feathers: Refactoring is Sloppy – refactoring is often prioritized out of regular development and refactoring sprints/stories aren’t popular due to past failures etc. An counter-intuitive way to get refactoring in is to imagine, during planning, what the code would need to be like to make it easy to implement a story. Then create a task for making it so before the story itself and assign it to somebody else then the story (to force a degree of scrutiny and communication). “Like anything else in process, this is medicine.  It’s not meant to be ‘the way that people do things for all time’ [..]” – i.e. intended for use when you can’t fit refactoring in otherwise. It may also make the cost of the current bad code more visible. Read also the commits (f.ex. the mikado method case).
• Cyber-dojo: A great way to practice TDD together. Compare your read-green cycle and development over time with other teams. Purposefully minimalistic editor, a number of prepared tdd tasks.
• On the Dark Side of “Craftsmanship” – an interesting and provoking article. Some developers, the software labouers, want to get work done and go home, they haven’t the motivation and energy to continualy spend time improving themselves. There is nothing wrong with that and we shouldn’t disparge them because of that. We shouldn’t divide people into craftsmen and the bad ones. A summary of and response to the varied reactions follows up in More on “Craftsmanship”. The author is right that we can’t expect everybody to spend nights improving her/his programming skills. Still they should not produce code of poor quality (with few exceptions) since maintaining such code costs a lot. There should be time for enough quality in a 9-5 day and people should be provided with enough guidance and education to be able to write decent code. (Though I’m not sure how feasible it is, how much effort it takes to become an acceptable developer.) Does the increased cost of writing (an learning to write) good code overweight the cost of working with bad code? That is an eternal discussion.

Cloud, web, big data etc.

• Whom the Gods Would Destroy, They First Give Real-time Analytics (via Leon) – a very reasonable argument against real-time analytics: yes, we want real-time operational metrics but “analytics” only makes sense on a sensible amount of data (for the sake of statistical significance etc.) RT analytics could easily provide misguided results.
  CAP Twelve Years Later: How the “Rules” Have Changed (tl;dr, via @_dagi) – an in-depth discussion of the CAP theorem and the simplification (2 out of 3) that it makes; there are many more nuances. By Eric Brewer, a professor of computer science at the University of California, Berkeley, and vice president of infrastructure at Google.
• ROCA: Resource-oriented Client Architecture – “A collection of simple recommendations for decent Web application frontends.” Server-side: true REST, no session state, working back/refresh etc. Client: semantic HTML independent of layout, progressive enhancement (usable with older browsers), usable without JS (all logic on the server) etc. Certainly not suitable for all types of apps but worthwile to consider the principles and compare them with your needs.

Clojure Corner

• Distributed Actors in Clojure (5/2012) – a discussion of options for Akka-like stuff in Clojure. Akka is great but “interfacing to Akka from Clojure is not nice, and certainly not idiomatic”. Though there have been some new Akka-Clojure libs since that (akka-clojure, okku).

Tools

• Vaurien, the Chaos TCP Proxy (via @bsvingen) – an extensible proxy that you can control from your tests to simulate network failure or problems such as delays on 20% of the requests; great for testing how an application behaves when facing failures or difficulties with its dependencies. It supports the protocols tcp, http, redis, memcache.
• Wvanbergen’s request-log-analyzer for Apache, MySQL, PostgreSQL, Rails and more (via Zarko) – generates a performance report from a supported access log to point out requests that might need optimizing
• Working Effectively With iTerm2 (Mac) – good tips in the body and comments

Favorite Quotes

A very good (though not very scientific) definition of project success applicable for distinguishing truly agile from process-driven projects:

[..] a project is successful if:

• Something was delivered and put to use
• The project members, sponsors and users are basically happy with the outcome of the project

- Johannes Brodwall in “How do we become Agile?” and why it doesn’t matter, inspired by Alistair Cockburn

(Notice there isn’t a single word about being “on time and budget”.)