Most interesting links of January ’13

Recommended Readings

Various

• Dustin Marx: Significant Software Development Developments of 2012 – Groovy 2.0 with static typing, rise of Git[Hub], NoSQL, mobile development (iOS etc.), Scala and Typesafe stack 2.0, big data, HTML5, security (Java issues etc.), cloud, DevOps.
• 20 Kick-ass programming quotes – including Bill Gates’ “Measuring programming progress by lines of code is like measuring aircraft building progress by weight.”,  B.W. Kernighan’s “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”, Martin Golding’s “Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.” (my favorite)
• How to Have a Year that Matters (via @gbrindusa) – do you want to just survive and collect possessions or do you want to make a difference? Some questions everybody should pose to him/herself.
• Expression Language Injection – security defect in applications using JSP EL that can sometimes leads to double evaluation of the expressions and thus makes it possible to execute data supplied by the user in request parameters etc. as expressions, affects e.g. unpatched Spring 2.x and 3.

Languages etc.

• HN discussion about Scala 2.10 – compilation speed and whether it matters, comparison of the speed and type system with Haskell and OCaml, problems with incremental compilation (dependency cycles, fragile base class), some speed up tips such as factoring out subprojects, the pros and cons of implicits etc.
• Blog Mechanical Sympathy – interesting posts and performance tests regarding “writing software which works in harmony with the underlying hardware to gain great performance” such as Memory Access Patterns Are Important and Compact Off-Heap Structures/Tuples In Java.
• Neal Ford: Functional thinking: Why functional programming is on the rise – Why you should care about functional programming, even if you don’t plan to change languages any time soon – N. Ford explains the advantages of FP and why FP concepts are spreading into other languages (higher abstractions enabling focus on the results over steps and ceding control to the language, more reusability on a finer level (higher-order functions etc.), few generic data structures with many operations -> better composability, “new” and different tool such as lazy collections, shaping the language towards the problem instead of vice versa, aligning with trends such as immutability)
• Neal Ford: Java.next: The Java.next languages Leveraging Groovy, Scala, and Clojure in an increasingly polyglot world – a comparison of these languages with focus on what they are [not] suitable for, exploration of their paradigms (static vs. dynamic typing, imperative vs. functional)

SW development

• How to Completely Fail at BDD – a story of an enthusiastic developer who tried to make everyone’s life better by introducing automated BDD tests and failed due to differences in culture (and inability to change thinking from the traditional testing), a surprising lack of interest in the tool and learning how to write good tests: “Culturally, my current team just isn’t ready or interested in something like this.” Morale: It is hard to change people, good ideas are not enough.
• M. Feathers: Refactoring is Sloppy – refactoring is often prioritized out of regular development and refactoring sprints/stories aren’t popular due to past failures etc. An counter-intuitive way to get refactoring in is to imagine, during planning, what the code would need to be like to make it easy to implement a story. Then create a task for making it so before the story itself and assign it to somebody else then the story (to force a degree of scrutiny and communication). “Like anything else in process, this is medicine.  It’s not meant to be ‘the way that people do things for all time’ [..]” – i.e. intended for use when you can’t fit refactoring in otherwise. It may also make the cost of the current bad code more visible. Read also the commits (f.ex. the mikado method case).
• Cyber-dojo: A great way to practice TDD together. Compare your read-green cycle and development over time with other teams. Purposefully minimalistic editor, a number of prepared tdd tasks.
• On the Dark Side of “Craftsmanship” – an interesting and provoking article. Some developers, the software labouers, want to get work done and go home, they haven’t the motivation and energy to continualy spend time improving themselves. There is nothing wrong with that and we shouldn’t disparge them because of that. We shouldn’t divide people into craftsmen and the bad ones. A summary of and response to the varied reactions follows up in More on “Craftsmanship”. The author is right that we can’t expect everybody to spend nights improving her/his programming skills. Still they should not produce code of poor quality (with few exceptions) since maintaining such code costs a lot. There should be time for enough quality in a 9-5 day and people should be provided with enough guidance and education to be able to write decent code. (Though I’m not sure how feasible it is, how much effort it takes to become an acceptable developer.) Does the increased cost of writing (an learning to write) good code overweight the cost of working with bad code? That is an eternal discussion.

Cloud, web, big data etc.

• Whom the Gods Would Destroy, They First Give Real-time Analytics (via Leon) – a very reasonable argument against real-time analytics: yes, we want real-time operational metrics but “analytics” only makes sense on a sensible amount of data (for the sake of statistical significance etc.) RT analytics could easily provide misguided results.
  CAP Twelve Years Later: How the “Rules” Have Changed (tl;dr, via @_dagi) – an in-depth discussion of the CAP theorem and the simplification (2 out of 3) that it makes; there are many more nuances. By Eric Brewer, a professor of computer science at the University of California, Berkeley, and vice president of infrastructure at Google.
• ROCA: Resource-oriented Client Architecture – “A collection of simple recommendations for decent Web application frontends.” Server-side: true REST, no session state, working back/refresh etc. Client: semantic HTML independent of layout, progressive enhancement (usable with older browsers), usable without JS (all logic on the server) etc. Certainly not suitable for all types of apps but worthwile to consider the principles and compare them with your needs.

Clojure Corner

• Distributed Actors in Clojure (5/2012) – a discussion of options for Akka-like stuff in Clojure. Akka is great but “interfacing to Akka from Clojure is not nice, and certainly not idiomatic”. Though there have been some new Akka-Clojure libs since that (akka-clojure, okku).

Tools

• Vaurien, the Chaos TCP Proxy (via @bsvingen) – an extensible proxy that you can control from your tests to simulate network failure or problems such as delays on 20% of the requests; great for testing how an application behaves when facing failures or difficulties with its dependencies. It supports the protocols tcp, http, redis, memcache.
• Wvanbergen’s request-log-analyzer for Apache, MySQL, PostgreSQL, Rails and more (via Zarko) – generates a performance report from a supported access log to point out requests that might need optimizing
• Working Effectively With iTerm2 (Mac) – good tips in the body and comments

Favorite Quotes

A very good (though not very scientific) definition of project success applicable for distinguishing truly agile from process-driven projects:

[..] a project is successful if:

• Something was delivered and put to use
• The project members, sponsors and users are basically happy with the outcome of the project

- Johannes Brodwall in “How do we become Agile?” and why it doesn’t matter, inspired by Alistair Cockburn

(Notice there isn’t a single word about being “on time and budget”.)

Do You Know Why You Are Testing?! (On The Principles Underlying TDD)

Kent Beck in his recent post Functional TDD: A Clash of Cultures summarizes well the key principles and benefits that underlie test-driven development. I think it is really worthwhile becoming aware of and thinking over these foundation stones of TDD (and testing in general). Knowing them enables you to apply TDD in the most effective way with respect to a particular context to gain the maximum of these benefits. People that do not really understand the value of their tests and TDD tend to write hard to maintain tests of limited (and occasionally even negative) value. Are you one of them?

Most interesting links of September ’12

Recommended Readings

• Johannes Brodwall: This dependency injection madness must end! – it’s very valuable to hear well-founded arguments against any popular belief and Dependency Injection is one of these. “I have started disliking the consequence of this strategy very much: All coupling in my system becomes implicit and harder to understand. I have instead reverted to using design patterns like the Singleton pattern, but with a slight twist.”
• Computer Programmers Learn Tough Lesson in Sharing – WSJ.com – A balanced presentation of pair-programming including both benefits and issues. A key point: It is a skill that must be learned (to respect the other one, give her space, be aware of how your behavior is perceived by her, …).
• Kent Beck: Functional TDD: A Clash of Cultures – TDD has been developed for object-oriented languages and applying it to a functional language with strong type brings interesting challenges. Also a good summary of the benefits of TDD: double checking of the logic (by the implementation and the, preferably quite different, test), solution decomposition (focus on part of the problem, once solve be sure it stays solved),  automatic checking of correctness, outside in design (API first, implementation after that). Plus the pleasant experience of the continuous cycle of tension (failing test) – relief (green).
• Paul Callaghan: Thinking Functionally with Haskell: Types? Tests? We Need a New Word – Powerful type systems eliminate possibility of defects thus venturing into the domain of testing – what can they offer and where the new border and symbiosis between types and tests will be?
• Tales from the Ops Side: Black Friday – an interesting and exciting view into the life of operations engineers one day when all went wrong. Key learnings: Many interdependant components are difficult to reason about; good monitoring and communication are crucial. The post refers to an interesting concept of Recovery-Oriented Computing, i.e. failures are inevitable and their prediction is nearly impossible thus we must focus on making the systems able to survive failures (e.g. vi damage containment, automatic fault detection, component-level restartability).
• Groovy: The road map for the popular JVM language – why was Groovy created (as Java companion focused on productivity), key changes in Groovy 2.0 (more suport for static typing, Java 7, modularity with speed as a side-effect) and in the future Groovy 3.0 (invokedynamic everywhere, more Groovy written in itself).
• Martin Fowler: Key Points from NoSQL Distilled – an overview of why NoSQL, data models, distribution models, consistency, map-reduce, polyglot persistence, criteries for choosing a database.
• You’re a Top Developer! – a surprising hypothesis that “90% of all developers never read a programming blog, never have any side projects to learn something new, and never spend any time or effort outside work hours to improve”. However I haven’t seen any data to back that up (the author only quotes  Peopleware) and the author doesn’t propose any explanation for the fact. I’d really like to know if it is true and why it is so.

Business & Agile

• Experimentation Is The New Planning – “You have no idea what’s going to happen to your industry. That’s why you build your organization into an engine of possibility.” We need “to  continually develop options and explore possibilities” to survive in the ever-changing conditions. Successful strategies emerge from the many ongoing experiments. However, “For emergent strategy to be successful, there must be enough autonomy, freedom, and slack in the system for people and resources to connect in a peer-to-peer way”.
• The Buy-vs-Build Shift (part 1) – Buy to reduce risk of failure (however true agile development – with frequent deliveries and feedback-driven direction – may be cheaper and more importantly can tailor the product to the actual needs) and to avoid inefficiecy of development (but it doesn’t need to be so with agile). “[..] in projects with long cycle times (years) there is a tendency for the business to be somewhat speculative and request all functionality that they can think of [..] With prioritised iterative delivery the business can halt a project when all features that are actually needed have been completed. [..] it does reduce the amount of features that are implemented, and based on my experience, quite substantially so.” Today’s development with e.g. TDD, powerful IDEs supporting automated refactoring, powerful development/production machine, the all-knowing Internet may be much more efficient.
• European entrepreneurs – Les misérables – A good analysis of why it is much more difficult to be an entrepreneur in Europe than in USA (the strong negative impact of a business failure, lack of local investors, cost of firing people) and the decline of European entrepreneurship since 19th century/WW1.
• U.S. Government Accountability Office: Effective Practices and Federal Challenges in Applying Agile Methods – US government considers agile effective; description of the useful practices and of challenges

Tools

• Command line tools for the Google Data APIs – create calendar tasks with reminders, list [today's] task; edit a doc in a local editor (vim, …), get/upload docs; send photos to Picasa, … .

Clojure Corner

• 10-minute Emacs for Clojure – getting started with Emacs for Clojure – install & config & basic usage for Emacs newbies (though no REPL integration yet)
• Keep IT Simply Simple: First month @Runa Inc. – Clojure shop in Silicon Valley – brief post about using Clojure in the wild. Some points: TDD works splendidly; frameworks are not necessary; Clojure can be really fast (<= type hinting, memoziation, performant data structures + occasional Java code)
• Blackstag: Why Clojure? The author describes the set of reasons that have led him to Clojure – and those that actually made him stick with it. “[..] what I like the most about Clojure is that it brings together the best of what many languages have to offer while not forcing it all upon me and, in doing so, has provided a good balance between power and flexibility.” “With Clojure I accomplish more and have found a greater sense of happiness with the work I am doing.”

Programming Like Kent Beck

Republished from blog.iterate.no with the permission of my co-authors Stig Bergestad and Krzysztof Grodzicki.

Three of us, namely Stig, Krzysztof, and Jakub, have had the pleasure of spending a week with Kent Beck during Iterate Code Camp 2012, working together on a project and learning programming best practices. We would like to share the valuable lessons that we have learnt and that made us better programmers (or so we would like to think at least).

Principles for Creating Maintainable and Evolvable Tests

Having [automated] unit/integration/functional/… tests is great but it is too easy for them to become a hindrance, making any change to the system painful and slow – up to the point where you throw them away. How to avoid this curse of rigid tests, too brittle, too intertwined, too coupled to the implementation details? Surely following the principles of clean code not only for production code but also for tests will help but is it enough? No, it is not. Based on a discussion on our recent course with Kent Beck, I think that the following three principles below are important to have decoupled, easy to evolve tests:

1. Tests tell a story
2. True unit tests + decoupled higher-level integration tests (-> Mike Cohn’s Layers of the Test Automation Pyramid)
3. More functional composition of the processing

Never Mix Public and Private Unit Tests! (Decoupling Tests from Implementation Details)

It seems to me that developers often do not really distinguish between the various types of unit tests they should be writing and thus mix things that should not be mixed, leading to difficult to maintain and hard to evolve test code. The dimension of unit test categorization I feel especially important here is the level of coupling to the unit under test and its internals. We should be constantly aware of what kind of test we are writing, what we are trying to achieve with the test, and thus which means are justifiable or, on the contrary, not suitable for that kind of test. In other words, we should always know whether we’re writing a Public Test or a Private Test and never ever mix the two. Why not and what actually are these two kinds of tests? Read on! (And if you agree or disagree, don’t hesitate to share your opinion.)

Most interesting links of September

Recommended Readings

• J. Yip: It’s Not Just Standing Up: Patterns for Daily Standup Meetings - it isn’t easy to make stand-up meetings short, focused, energizing, and centered around continuous improvements and team spirit. This description of an example good standup, the meeting’s goals, and especially the “patterns” and “bad smells” can be pretty useful to get and keep on the track towards a brighter future. TBD: standup goals: GIFTs, team spirit, appreciation where we go and are.
• M. Poppendieck: Don’t Separate Design from Implementation – according to Mary, (detailed) requirements – being it in the form of (backlog) user stories or any other – represent actually a design of the system, which shouldn’t be done by the amateur product owner/business analyst but by professionals, meaning the developers, based on high-level goals and clear specification of the desired business value. She writes about a project that her factory outsourced and which she could have designed but didn’t – yet it succeeded even though there were no detailed requirements. I’ve also read and unfortunately lost an interesting answer where the author argues that that is only possible if the developers are really experienced in the field. I tend to agree more with Mary though it is of course a question what “high” and “low” level goals/requirements are. But undeniably users/analysts tend to propose solutions disguised as requirements while often missing the technical insight to see possible other and better solutions. We also cannot expect the developers to produce a great SW if the true goals, needs, and business values behind the requested “features” aren’t clearly communicated to them. (The best example – lost source again – is where a developer proposes to the client a simple process change that will solve the problem without writing a single line of code.)
• Mike Cohn: The Forgotten Layer of the Test Automation Pyramid – three levels of testing with increasing number of tests: UI/Service/Unit (or end-to-end instead of UI), each requiring a different approach. Unit tests are best because a failure points directly to its source (with higher level tests you don’t immediately know the cause). The higher in the pyramid, the less tests we should have (e.g. because of their redundancy). It’s important not to forget the middle, service layer – unit tests are too low-level, UI tests too difficult and brittle. Also Gojko in Specification by Examples says that acceptance/BDD tests should run mainly at the service layer because of the UI level issues.
  “Although automated unit testing is wonderful, it can cover only so much of an application’s testing needs. Without service-level testing to fill the gap between unit and user interface testing, all other testing ends up being performed through the user interface, resulting in tests that are expensive to run, expensive to write, and brittle.” [Emphasis JH.]
• Technical Debt and the Lean Startup – Paul Dyson remarks that while quality is an essential concern for projects in established environments, in the case of lean startups the primary goal is to find out whether a product is viable and what it should be like and thus it’s reasonable to accept much higher technical debt by not spending too much time on ensuring scalability, de-duplication etc. – only when the product proves viable should we start to care for its long-evity by emphasizing the quality. But one thing can never miss and that is good test suite because this is the crucial factor that makes letter payment of the technical debt possible without ruining oneself.
• Coding dojo – Real time coding competition with Extreme Startup – an inspiring report about a coding dojo lead by Johannes Brodwall in Bergen’s JUG, the task being the implementation of a server that can respond to questions send over HTTP (that’s all participants know at the beginning – they learn the rest during the iterations)
• Using Code Katas to Improve Programming Skills – why to use code katas + links to different proposed katas
• Kent Beck: Don’t Cross the Beams: Avoiding Interference Between Horizontal and Vertical Refactorings - when to do depth-first (more exploratory) refactoring and when to extend it into breadth (i.e. apply it to several similar objects)

Learning Clojure (maybe not so interesting for those not learning the language)

• Phil Calçado: My Experience With TDD In Clojure (via planetclojure) – nice example of how to decompose a task in functional programming to make it easy to test (via Midje), including useful testing-related links and a discussion of side-effect isolation and the building blocks of functional programs, i.e. function composition using combinators (i.e. functions producing functions)
• How to learn Clojure effectively (via planetclojure) – a very good description of how the task at  4Clojure (though I prefer Clojure koans) should be solved to benefit one’s learning the most plus some general tips on functional thinking
• Clojure open source projects for learning how to code it

Most interesting links of June

Recommanded Readings

• Empirical Studies Show Test Driven Development Improves Quality – brief summary of two researche papers comparing TDD/non-TDD, one paper with 1 case study from IBM and 3 from Microsoft (get PDF; conclusion: The pre-release defect density of the four products, measured as defects per thousand lines of code, decreased between 40% and 90% relative to the projects that did not use TDD. The teams’ management reported subjectively a 15–35% increase in initial development time for the teams using TDD, though the teams agreed that this was offset by reduced maintenance costs.), one older summarizing 13 case studies (conclusion: TDD seems to improve software quality, [...] there were indications that TDD does not necessarily decrease the developer productivity or extend the project leadtimes: In some cases, significant productivity improvements were achieved [...] However, in both of those studies the quality was improved.).
• Quantum computers are reality: World’s first commercial quantum computer sold to Lockheed Martin – this means that hard and NP-hard problems become solvable or at least much easier to solve (this means incredibly lot, just think about optimization problems, scientific computations and simulations, …) and that current security measures are becoming obsolete. It has “only” 128 qubits, which sounds small but is a big thing (competitive regime for quantum comp. is about 100).
• Fit co-author says acceptance testing costs more then it is worth (i.e. there are better alternatives) – it’s always good to learn from failures; the author, James Shore, has several pleas against A.T.: “[..] customers (a) weren’t interested in doing that, and (b) often couldn’t understand and didn’t trust tests that were written by others. [..] Furthermore, acceptance testing tools are almost invariably used to create end-to-end integration tests, which are slow and brittle. Fit works best for targeted tests that describe the domain, but that’s not how it’s used. Also, tools like Fit [JH: which is based on HTML] don’t work with refactoring tools.“, summarized: non-participation of customers and maintenance burden. Gojko Adzic opposes that we can avoid the pitfalls while retaining the benefits – I guess his Specification by Example book is quite lot about this, as well as the last month mentioned post Top 10 reasons why teams fail with AT

Favourite Quotes

Unit tests, [..], are so coupled to the low-level API that it is often hard for the developers to avoid the trap of proving that the solution works in a particular way, rather than asserting that is solves a particular problem.

- from the book Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation, published in an InformIT article

Most interesting links of January

I’m moving to Norway and have thus little time for reading but still I’ve stumbled upon something really interesting.

• Bob C. Martin (the Clean Code guy): The Transformation Priority Premise – with TDD the code goes through a series of transformations (similar to refactorings but they change the behaviour) that “move the code from a specific form to a more generic form” (from “return 0;” to “return myFunc();”). “It also appears that these transformations have a preferred order based on complexity.” (nothing => simple code with null; null => constant; …). “It is the premise of this blog that if tests are chosen and implemented in this preferred order of transformations, then TDD impasses will be reduced or eliminated.“
  Also the article explains nicely why the practice of writting simple, wrong (i.e. too specific) code is good. 
  • Uncle Bob’s TDD excercises (katas): Prime Factors kata (as .ppt), The Bowling Game Kata (as .ppt)
• 10 Tips for Proper Application Logging – should be rather called 10 best practices for (enterprise Java) logging
• Kanban Development Oversimplified – finally an explanation of Kanban for SW development I could understand, also thanks to the comparison with other agile approaches

Most interesting links of November

This month has been quite interesting, among others I ‘ve picked up several blogs by Adam Bien. I really like his brief, practical and insightful posts

Java, Jave EE, architecture etc.

• EJB 3.1 + Hessian = (Almost) Perfect Binary Remoting – “With hessian it is very easy to expose existing Java-interfaces with almost no overhead. Hessian is also extremely (better than IIOP and far better than SOAP) fast and scalable.” See also the discussion regarding a “DynamicHessianServlet” (which would be more comfortable then creating a new servlet for each service).
• Binary XML – Fast Infoset performance results – FI is the name of a standard for binary XML encoding and also of its OSS implementation; according to these measurements (with default settings, compared to Xerces 2.7.1), on average: The FI SAX parser  is about 5 times faster than the Xerces SAX parser, the FI DOM serializer (using default settings) is 25% to 30% faster than the Xerces DOM XMLSerializer, and FI documents are 40% to 60% smaller than XML documents (i.e. it can provide modest to good compression). And: “The use of external vocabularies can be a very effective way to increase the efficiency of parsing, serializing and size at the expense of the fast infoset documents no longer being self-describing … .”
  For the interested ones: FI is built on ASN.1, a standard for describing data structures in a way that is independent of machine architecture and implementation language, it also includes a selection on different binary encoding rules, e.g. DER (triplets tag id – length – value).
• EJB 3.1 And REST – The Lightweight Hybrid – Why to use @Stateless – there have been recently discussion about Spring vs. EJB and whether @Stateless is of any use. According to this article, the added value of EJB (though some are provided also by Spring) include injection capabilities, transactions, single threading model (for why that is good see #2 on Why I like EJB 3.0/3.1), visibility in JMX, concurrency restriction via thread/bean pools.
• Why Service Isn’t A ServiceFacade, But ServiceFacade Is Sometimes A Service…
• Experiences from migrating Hyperic 4.5 from EJB/JBoss to Spring/Tomcat, what good has it brought – simplified unit and integration tests, simplified code thanks to Jdbc/JmsTemplate, … (My comment: EJB 3.1 would certainly bring at least some of the advantages too.)

Performance – large JVM heaps are very much feasible

• Tuning the IBM JVM for large heaps – tuning 64b IBM JVM 6 with 100GB heap to have acceptable GC times (~ 1/2s as opposed to 25s with the default settings)
• BigMemory: Heap Envy – there have been recently a lot of fuss about Terracotta’s new BigMemory, a  GC-resistent many GB cache space (using NIO byte buffer). This post discusses it’s disadvantages and compares it with a solution based on ConcurrentHashMap, comming to the conclusion that the old good ConcurrentHashMap outperforms BigMemory considerably.

Other

• Mind-breaking TDD exercise – Thought-provoking TDD exercise at the Software Craftsmanship conference – a TDD exercise with strict rules requiring to only write implementation in the test itself and then refactor it out as needed. I’d strongly suggest that you try to do it by yourself. There is a more detailed report from a tic-tac-toe version and a bit about implementing a message interceptor in this way. An interesting observation: “differences from a pre-conceived design and the one that emerges from code are considerable, the latter being a lot more elegant and better suited to describe a problem.“
• To code quickly, you must quit coding (Pomodoro etc.) – why Pomodoro technique (concentrate 25m on work rejecting interruptions, then take a break) and similar ones work
• Colorize Maven output in Linux/bash via filtering of Maven’s mvn command output using sed. (Thanks to Alda for the link.)