Posted by Jakub Holý on October 31, 2013
- Google engineers insist 20% time is not dead—it’s just turned into 120% time – it is interesting to see how has this evolved; “I have done many engineering/coding 20% projects and other non-engineering projects, with probably 20-40% producing “real” results (which over 7 years I think has been more than worth it for the company). But these projects are generally not rewarded.” [highlight mine]
- The Worst Daily Scrum Ever – a story whose bad part is a too common reality; if energy is low, nobody asks for / offers help, and people only report status / plans then you are doing the daily scrum wrong and should stop now (but it also documents a nice example of a good, effective scrum)
- Why Responsive Design is a Waste of Time – a refreshingly critical take on responsive design; the author now aknowledges that it is sometimes worth the pain but the points are still valid – responsive design requires (lot of) extra work, the attempt to create a one-size-fits-all site of course adds considerable complexity (having two separate simple frontends might be better than one that is too complex), also many sites are good enough as they are (especially taking into account the capabilities of mobile browsers)
- How to lose $172,222 a second for 45 minutes – i.e. your bugs are likely not so serious after all A financial company screwed big and ended up bankrupt. The cause? Chaotic DevOps, not removing old unused code, reusing a feature flag instead of creating a new one, lack of monitoring. The story in short: They deployed new trading code but failed to notice that it has not been deployed to one of the 8 servers; due to the flag reuse, the old, 10 years unused code has been activated instead. Due to the lack of monitoring they did not notice the cause, tried to roll back while leaving the flag enabled thus effectively activating the bad code on all the servers. => have proper automated and self-checking deployments, delete old code, do not repurpose old switches.
- 40 Inappropriate Actions to Take Against an Unlocked (Windows) PC – good tips for promoting security and having fun at the same time; I shall keep this at hand
- How to go about ‘proving’ why dynamically typed languages are better – a cultivated and interesting discussion; as argueed, thinking in this direction is itself wrong and in different contexts, different languages will be more appropriate. I also like Phil Lord’s “Programming is a highly fashion-centric occupation for any number of reasons.” and “For me, the main advantage is that you are not forced to build complex hierarchies just to support the type system ([..]), and that having only a few abstractions makes it worthwhile adding lots of functions operating over them.” and L. Petit’s “IMHO, the question is irrelevant. It implicitly assumes that statically typed vs dynamically typed is a black / white choice, and that either ‘static wins over dynamic’ or ‘dynamic wins over static’ will be a true statement whatever the context.” Also a good observation that types are only a subset of function contract enforcement and one of possible implementations.
- The Failure of Governmental IT (Learnings From HealthCare.gov) – links to a few really good articles about the problems with governmental IT in general and my summary of them
- Inside the Arctic Circle, Where Your Facebook Data Lives – the designs of data centers used to be proprietary secrets until Fb developed its own and open-sourced them, enabling many Asian manufactures to start creating cheaper datacenters and thus started a revolution in this domain. Facebook’s data centers are not general purpose but suitable ot the kind of work they need, but it is still widely applicable. Cool to see how they use natural conditions to get energy needs down and make HW that fits best their needs – that is what I call innovation!
- Academia.edu (via @RiczWest) – a rich source of free research papers – just register as an independant researcher; also lean/agile/systems thinking and other interesting topics
+ Username +
- Johannes Brodwall: A canonical web test – a simple web app end-to-end smoke test – using an embedded Jetty, a test DB (preferably in-memory), WebDriver to test it (simple: browser.get(“/people”), assertThat(browser.findElement(<person id>.contains(<person’s name>)); simple, nice, useful
- LearnGitBranching – an online game to learn branching & rebase in git; use the menu in the lower-right corner to navigate between the levels etc. You can also execute commands “show goal”, “hint”, “level” to navigate around; pretty cool and great for learning the command line commands
Society & people
Not a typical topic I share here but really worth it this time.
- The ocean is broken – a saddening story worth reading to learn what does your tuna sandwitch cost and where does all the plastic we use end up. From a sailing trip from Melbourne to US where there were plenty of fish (and birds) 10 years ago – and 2 this year, killed to a noticable degree by huge fishing ships that catch tuna – and kill and throw away all the other “junk” fish. Nowadays fish are replaced by plastic and other waste that actually prevents usage of the engine unless somebody can watch for dangerous nets and ropes leftovers. Earth, where are you falling to?
- The Guardian: Why have young people in Japan stopped having sex? – sad and interesting to observe what happens when the system is set up so that people “can’t be bothered” to have inter-sexual relationships, partnership, and children. Japan needs a good deal of systems thinking to fix its broken society where women do not want children because it would cost them their career and neither men nor women are willing to subjects themselves to the social pressure and demands associated with relationships.
- The Guardian: 29 million people enslaved, says first global index on slavery – welcome to the 21st century! The leading slave countries are India (14M), China (3M), Pakistan (2M). Also, slaves are building the world cup stadion in Qatar.
- They’re Taking Over! – how we managed to destroy sea ecosystems and helped the now unstoppable return of jellyfish – Jellyfish are evidently very veried and extemely resilient and have been hold at bay only by rather complex ecosystems that we managed to destabilize so much that Jellyfish are on their way back to ruling all the sees again (destroying the rests of the ecosystems – i.e. fish – on the way); a sad future for the sea, Earht, and us
- LightTable 0.5.9 got elementary paredit commands
- Amazonica: A comprehensive Clojure client for the entire Amazon AWS api – best with a REPL!
- Clojure Understood: the Rush Hour Platform – application of the design best practices promoted in the Clojure community (separation of concerns, simplicity, …) on a project – “highly accurate vehicle traffic simulations”; I have only started reading it but it looks highly interesting
- Clojure content at InfoQ – articles, news, interviews, presentations etc.
- A bitter taste [of EuroClojure] – we as a community must embrace diversity and stop fostering our egos by mocking other than our holy editor (and, I would add, by mocking other languages and in general mocking whatever); respect and open minds, please!
- Garden – Clojure alternative to scss/less – still needs time to mature and gain tooling support but it is cool that it exists
- C. Emerick’s Austin, the ClojureScript REPL over nREPL – 1) Start nREPL (via lein repl, from your editor…), 2.a) Execute austin’s exec to start a ClojureScript REPL in it, backed by headless PhantomJS or a real browser – or, alternatively, 2.b) create a C.S. REPL connected to your webapp, as described in the browser-connected-repl-sample. You should watch this 8 min demo.
- WhiteHat Aviator – A Safer Web Browser – WhiteHat, a well-known security company, has released a browser that aims at improving privacy by preventing user tracking (f.ex. but not sending referral URL) and blocking ads even at the cost of occassional slight discomfort, i.e. something that the mainstream browsers are not interested in. So far for OS X only.
- EnvPane – a preference pane for environment variables for Mac OS X 10.8 (Mountain Lion) – set env. vars for GUI/terminal apps, no need to log out upon change
Weinberg: Bureaucracy is what we do when we no longer remember why we are doing it
- via Ben Simo, no source specified so it may be fake but anyway it is valid
Posted in General, Languages, Testing, Top links of month | Tagged: aws, clojure, ClojureScript, design, earth, ecology, Git, innovation, privacy, science, scrum, security | 4 Comments »
Posted by Jakub Holý on April 30, 2013
The top top article
How To Survive a Ground-Up Rewrite Without Losing Your Sanity (recommended by Kent Beck) – sometimes you need to actually rewrite an important part of a system; here we learn about two such rewrites, one which went well and one that failed badly – and what are the important differences.
The pain of a rewrite: “it’s [a major rewrite] going to take insanely longer than you expect” – because: “there’s this endless series of weird crap encoded in the data in surprising ways” and it takes days to convert them, “It’s brutally hard to reduce scope” (you cannot drop features, edge cases), “There turn out to be these other system that use ‘your’ data”.
To succeed you need: 1) Determine clear business-visible wins to justify the effort that will be much higher than expected and to know when to give up / what to postpone; 2) Do it extremely incrementally (<-> Succession) – break it into a series of small, safe steps, each generating a business value and learning of its own thus enabling early and frequent economical tradeoffs (stop, shift priorities, …) – ex.: rewrite a single reports, migrate its data, switch customers to it, go on to the next one – complete slice of functionality => a more realistic estimate soon => reprioritisation; incrementalism requires you to be able to write data both to the old and new system, which is hard but always pays off: “Here’s what I’m going to say: always insert that dual-write layer. Always. It’s a minor, generally somewhat fixed cost that buys you an incredible amount of insurance.” 3) “Abandoning the Project Should Always Be on the Table” (<- known biz value, better estimate based on early feedback).
Some Specific Tactics: Shrink Ray FTW (a graph of how much has been already replaced => motivation), Engineer The Living Hell Out Of Your Migration Scripts (tests, robustness, error handling, restartability), If Your Data Doesn’t Look Weird, You’re Not Looking Hard Enough.
Methodology, agile, lean
- M. Fowler: The New Methodology – a good description of the rise of Agile, the motivation for it, the various Agile methodologies (XP, Lean, Scrum etc.) and what is required to be able to apply an agile approach. Main points: Agile is adaptive (vs. predictive) and relies heavily on people and their judgement and skills (vs. treating them as same, replacable units) – which also leads to the need of leadership instead of (command&control) management. Discusses unpredictability of requirements and scope, foolishness of separating design and implementation, difficulty of measurement of SW development, continuous improvement etc. Quotes: “However letting go of predictability doesn’t mean you have to revert to uncontrollable chaos. Instead you need a process that can give you control over an unpredictability. That’s what adaptivity is all about.”
- The Toyota concept of ‘respect for people’ – many state that they respect their workers but fail to really understand what it means; it is not about freedom of act, it is about a mutual respect, leveraging the strengths of each other: worker’s experience and insight and manager’s broader overview, as demonstrated by the problem-solving dialog and challenges (problem – root cause – solution – measure of success, the manager challenging the worker’s answers). Also a nice example how the evaluation of individual performance leads to a much worse system and high turnover compared to a whole-oriented company.
- Fixed Bid Agile Without Cognitive Dissonance – a refreshing take on fixed-scope projects and Agile; yes, they are bad but sometimes the client has no other choice so what best we can make out of it? The core advice: Agree “a pragmatic change management protocol (along with a contingency built into the pricing)” (push for lower initial requirements granularity, customer involvement, flexibility of functionality) => “you can gain significant agile benefits for clients who wouldn’t otherwise accept them”.
- Agile Atlas: Scrum – a good description of Scrum and its values, roles, artifacts, and activities
Learning, psychology, estimates
- How Developers Stop Learning: Rise of the Expert Beginner – sometimes you meet people with experience-indicating titles that are actually little competent, perhaps leading incompetent IT departments. Why? They, unchallenged by competent peers or broader IT community, came to believe that they are “experts” while actually being only little more advanced beginners, better than their beginner colleagues but still lacking any understanding of the big picture and the knowledge of what they do not know, trapped in the “unconscious incompetence” stage. The post explains this in a more detail and is followed up an explanation how it can lead to the rise of a mediocre SW group in “How Software Groups Rot: Legacy of the Expert Beginner“.
- Coding, Fast and Slow: Developers and the Psychology of Overconfidence (via @peterskeide) – why are we so bad at estimating (inherent complexity of SW vs. our overconfidence) and why it cannot be fixed. We can learn to somehow estimate tasks of few hours length (less complex, plenty of practice opportunities). The question is: “how you can your dev team generate a ton of value, even though you can not make meaningful long-term estimates?”
- Cognitive Overhead, Or Why Your Product Isn’t As Simple As You Think (via @JiriJerabek) – to make apps more accessible to users, we try to make them simple – but “simple” might be different from what you expect. The important thing is not less steps, less features, less elements, but lower cognitive overhead, i.e. “how many logical connections or jumps your brain has to make in order to understand or contextualize the thing you’re looking at.” Good examples of unexpectadly high / pleasantly low cognitive overhead, some tips, even suprising ones such as make people do more (to be more involved in the process – e.g. bump their phones), slow down your product.
- Economies of Scala – a case for using Scala over Java, supported by data: many capable developers want to use it but there are few opportunities for them – and getting developers is one of the main challenges.
- A canonical Repository test – a nice standard way to test a “DAO”; highlights: use of FEST assert 2 for clean and nice checks, no unimportant details in the test (f.ex. details of the test data hidden in randomPerson() and randomOder(Person)).
- How To Think Like An Engineer – some nice ideas such as: “Build A Simple First Version: With People, Not Code” – “Technology is not always the best solution, because technology is not always the simplest solution.”, i.e. don’t automate everything from the start (examples from Netflix, Amazon); “Rather than trying to do everything at once, break down the functions of your company into smaller goals.” – and focus at one at a time
- Economies Of Scale As A Service (do not mix up with Scala! :-))- an interesting description of the trend away from ownership to the rental of important resources (servers, manufacturing capabilities, personal cars, …) and the resulting changes in the society, business, and industry
- Troy Hunt: Our password hashing has no clothes (or the much shorter though biased How To Safely Store A Password) – MD5 and SHA are not safe enough due to brute-force attack enabled by GPUs, irrespective key size; it’s crucial to use hashing algorithms designed for passwords (and thus sufficiently slow) – f.ex. bcrypt, or PBKDF2 or the newer scrypt.
- Everything about Java 8 – a well-made summary of what should come in Java 8, based on the current state, discussing the finer points: static and default (non-static, overridable) methods on interfaces, lambdas (do I need to mentione that?!) and method references (String::valueOf, Object::toString, myVar::toString, ArrayList::new); good discussion of the various use cases and limitations of lambdas (capturing x non-c., ..); java.util.stream for functional operations on value streams (filter, map, reduce etc.); java.time inspired by Joda, more concurrency utilities (e.g. CompletableFuture for chaining futures); String.join (finally!), Optional ~ Scala’s Option & more; yummy!
- How To Keep Your Best Programmers – what motivates capable programmers to stay/leave? The author lists some common reasons and concludes that, ultimately, all are linked to the desire for autonomy, mastery, or purpose. However he goes further and proposes that, to keep talented devs, you must offer them an appealing narrative (regarding their actions and a result, related to autonomy/mastery/purpose) and reaffirm/update it frequently; ex.: “With the work that we’re giving you over the next few months, you’re going to become the foremost NoSQL expert in our organization.” “At any point, both you and the developers on your team should know their narratives.” – so that they will be “constant points of job satisfaction and purpose.”
- Clojure Data Analysis Cookbook review – “The book provides a collection of recipes for accomplishing common tasks associated with analyzing different types of data sets. It starts out by showing how to read data from a variety of sources such as JSON, CSV, and JDBC. [..] how to sanitize the collected data and sample large data sets. [..] a number of different strategies for processing it.” How to present them with ClojureScript and NVD3 (D3.js components). “Some of the highlights include using the Clojure STM, parallel processing of the data, including useful tricks for partitioning, using reducers, and distributed processing with Hadoop and Casalog.”
once again, trying to do it *and* do it right was too much all at once, resulting in little progress and little learning.
- Kent Beck’s tweet 2013-04-16
A true agile development process can be recognized by its continual evolution:
A project that begins using an adaptive process won’t have the same process a year later. Over time, the team will find what works for them, and alter the process to fit.
- Martin Fowler in The New Methodology
Posted in General, SW development, Testing, Top links of month | Tagged: agile, bigdata, clojure, human, learning, scrum | Comments Off
Posted by Jakub Holý on May 31, 2012
This was a rich month, bringing some hope for ORM, providing a peep-hole into the bright and awesome future with in-browser video and other cool web-stuff presented at WebRebels 2012 and IDEs providing immediate feedback and visualisation. There were valuable articles about simplicity and quality in software and good talks about the lean startup (i.e. enabling innovation) and other topics.
- M.Fowler: ORM Hate – Why ORM is actually a good solution – a very valuable article where Fowler opposes the popular trend of criticising Object-Relational Mappers such as Hibernate. Yes, using an ORM is difficult and a leaky abstraction – but that’s because the problem of mapping from a rich in-memory object model to a relational store is inherently difficult (and you need to do it with or w/o an ORM tool) and because those last 10-20% of DB access require human intelligence. If you can avoid the need for ORM by using the relational model also in memory or by using a NoSQL database with a data model that fits your in-memory model then it’s great to do so but often you can’t and then using ORM is the best solution. You certainly don’t want to code your own “lightweight” ORM.
- Communicating Sequential Processes: Theory for reasoning about concurrent, interacting processes – an inspirational reading about a much better way to do concurrency than Java threads; “… [CSP] is a language for describing patterns of interaction between concurrent objects. It is supported by an elegant, mathematical theory, a set of proof tools, and an extensive literature.” The beauty is that thanks to the theory behind, you can actually reason about the interactions and verify their correctness, contrary to the feared mess of Java threads. CSP is broadly similar to the popular Actors model and is implemented in Occam while it also influenced Erlang’s concurrency model and Go. The library JSCP brings it to Java. I guess we’re better of using Actors due to their popularity and maturity though the mathematical backing of CSP with the potential of formal proofs of correctness is indeed attractive. Any of the two is better than using threads directly because:
The monitor-threads model provided by Java, whilst easy to understand, proves very difficult to apply safely in any system above a modest level of complexity. One problem is that monitor methods are tightly interdependent, so that their semantics compose in non-trivial ways [...]
- Rich Hickey introduces the Reducers library: simplicity in practice – a beautiful example of simplifying something by taking appart all the unrelated but mingled concerns and focus only on those really needed. Whether you’re interested in Clojure or not, you should read the beginning of the post where Hickey explains how the current collection functions based on first (returns 1st element) and rest (returns the remaining ones) mix too many things (ordering, output representation, etc.) and how this “new super-generalized and minimal abstraction for collections” avoids that and thus provides e.g. for doing things in parallel and composing transformation without producing intermediate collections. Beautiful! (PS: I’ve blogged about more examples of pursuing simplicity & gaining power.)
- M. Fowler: Cannot Measure Productivity – a thoughful discussion of why the productivity of programmers is hard/impossible to measure (i.e. you should concentrate on measuring other, more useful metrics) “[..] false measures only make things worse.”
- Gojko Adzic: Redefining software quality – an obligatory read that introduces a holistic view of SW quality and the quality pyramid. The key idea is that there are multiple, vertically organized facets of quality and once a more basic facet is saturated, you should move and and concentrate on the next facet and level of quality. The quality pyramid: Deployable & functionally OK > Performant & secure > Usable > Useful > Successful. Once a particular level is satisfied, it is wasteful to put more effort into it and you’ll bring much more value to the customer by focusing on the next higher level. Gojko: “Yet from what I see most software teams invest, build and test only at the lowest two levels, gold-plating things without a way to explain why that is bad.”
- Is Pair Programming for Me? – the author, who claims to have taught pair programming to 200+500 people, points out that pair programming is a skill that must be (consciously) learned, or actually a number of inter-personal skills. He also describes the cycle people go through when learning it, including a temporary downswing in productivity and negative view of pairing (therefore people should do it at least for 3-4 weeks to overcome the problems and gain the benefits).
- Bret Victor: Inventing on Principle (55 min, see at least the first 5 min) – very inspiring! Victor firmly believes that “creators need an immediate connection to what they create” and demonstrates how this can be achieve when coding image rendering, a game, an algorithm, when designing a circuit. After watching it for few minutes you will think: How could we have been working with such crappy tools without realizing how limited they are? Fortunately people started to apply the idea of an immediate connection between code and the result, f.ex. in LightTable and Bikeshed‘s IDE. On the other hand, there is an evidence that this may be too hard with the current programming languages.
- Eric Ries: Evangelizing for the Lean Startup – entertaining and enriching introduction into an approach for bringing innovation to life – f.ex. in startups – withou failing unnecessary, demonstrated on the example of the author’s failed and successful startup. Many innovators fail because they don’t realize that their key challenge is that they don’t now neither the problem (who are our customers and what they need) nor the solution (the product to satisfy the need) and thus what they need to do is to experiment and learn in the shortest cycles possible. If you wander what the buzz about lean startup is or how to build innovations, this is the ultimite source you should watch. The video has 1h but the first 20-30 min will give you a sufficient overview. The key points summarized by the Iterate lean guru Anders Haugeto:
- There is only one way to measure progress: Progress == The amount of things you have learned from your real customers
- Hence, you need to work a continuous loop to build, measure and learn as fast as possible. Typical iterations, like sprints, are too long, hence inefficient
- Until you have an established product, even recognized engineering practices like TDD, sprints, refactoring, and all the XP-stuff are less important than this feedback cycle
- Even the perfect agile method is nothing, if you’re using it to build the wrong product: How can you know you are heading in the right direction?
Links to Keep
- E. King: Maximizing the Value of Your Stand-up – interesting techniques to try out at your stand-ups – Speed Scrum, Pass-the-Conch Scrum (passing a token randomly to define the order), Time-Box Scrum, Challenge Scrum (the team may ask 1 question each presenter), Impediments-Only Scrum, Award Scrum (reward for best articulation of his/her information), Business Value-Focused Scrum, No-Board Scrum, Whiteboard Scrum, Buddy Scrum (report for sb. else)
- puppet-lint – check code style of your Puppet files
- Guard – cross-platform tool that can watch for file changes and execute actions (“guards”) when a file is changed, useful e.g. to execute tests/style checks only on the files being modified. Includes support for many testing/checking tools and multiple notification means such as Growl.
- ThreadLogic – Thread dump analyzer that understands common patterns found in application servers and enabling the definition of custom patterns. Supports Sun, IBM, and JRockit.
- Dumbster – mock SMTP server for unit testing (start in @Before, get sent messages in the test, stop afterwards)
Kai Thomas Gilb, in a talk proposal for JavaZone 2012:
Accurate estimation is impossible for complex technical projects, but keeping to agreed budgets, and deadlines is achievable by using feedback and change.
- StackOverflow: Clojure Performance Benchmarks – links to various discussions and benchmarks (beware that older results and facts are likely to be outdated). And of course you must keep in mind that 1) benchmark only measure what they measure, e.g. the outcomes cannot be generalized and that 2) benchmark results aren’t relevant for your problem unless you’re doing exactly the kind of operations being benchmarked (e.g. who cares that X is 100 ms slower if your code spends 1s waiting for a XML file download?) (Craig Andera had a pretty good experience report from webapp performance testing including what (not) to do)
- A good wrapup of the EuroClojure conference by Deon Garrett. Such a pity I missed it!
- Goldberg (at GitHub) – Johann Sebastian Bach’s Goldberg Variations in Overtone by @ctford; using Overtone and Clojure to build up mathematical and functional definitions of canons. Deon Garrett: “Go right now and download the code from Chris’ talk. If you don’t know Clojure, use this as an excuse to learn it – it’s that good.”
Rich Hickey interviewed by M. Fogus:
Reducing incidental complexity is a primary focus of Clojure, and you could dig into how it does that in every area.
In particular, the use of objects to represent simple informational data is almost criminal in its generation of per-piece-of-information micro-languages, i.e. the class methods, versus far more powerful, declarative, and generic methods like relational algebra. Inventing a class with its own interface to hold a piece of information is like inventing a new language to write every short story. This is anti-reuse, and, I think, results in an explosion of code in typical OO applications.
Posted in Databases, General, Testing, Tools, Top links of month | Tagged: agile, clojure, concurrency, design, html5, leanstartup, ORM, scrum, smtp, trends | Comments Off
Posted by Jakub Holý on November 7, 2011
A review of and extract from Agile Project Management With Scrum by Ken Schwaber, Microsoft Press 2003, ISBN 0-7356-1993-X.
The book is basically a set of case studies about Scrum that show how to implement the individual aspects of Scrum, what are the common pitfalls and how to avoid them, and help to understand its mantra of “the art of the possible” and how to adapt Scrum to various situations. It’s very easy to read thanks to the case studies being brief and organized by topics (team, product owner, …). I’d absolutely recommend it as a third book in this domain, after a general introduction into the lean thinking (Implementing Lean Software Development – From Concept to Cash by M. & T. Poppendieck is great for that) and an introduction into Scrum itself. Scrum is not just a set of practices, it requires an essential shift in thinking. Thus it is not enough to learn about the practices – you have to learn, understand, and accept the principles behind. This book will hopefully help you to refine your understanding of these principles.
This extract contains the quotes and observations that I find the most interesting. It tries by no means to be objective or representative, a different person with a different experience and background would certainly pick different ones. Thus its value for others than myself is rather limited but it may perhaps serve as an inspiration to read the book. My all favourite quotes are in italics. Read the rest of this entry »
Posted in General | Tagged: agile, book, review, scrum | Comments Off
Posted by Jakub Holý on September 30, 2011
- J. Yip: It’s Not Just Standing Up: Patterns for Daily Standup Meetings - it isn’t easy to make stand-up meetings short, focused, energizing, and centered around continuous improvements and team spirit. This description of an example good standup, the meeting’s goals, and especially the “patterns” and “bad smells” can be pretty useful to get and keep on the track towards a brighter future. TBD: standup goals: GIFTs, team spirit, appreciation where we go and are.
- M. Poppendieck: Don’t Separate Design from Implementation – according to Mary, (detailed) requirements – being it in the form of (backlog) user stories or any other – represent actually a design of the system, which shouldn’t be done by the amateur product owner/business analyst but by professionals, meaning the developers, based on high-level goals and clear specification of the desired business value. She writes about a project that her factory outsourced and which she could have designed but didn’t – yet it succeeded even though there were no detailed requirements. I’ve also read and unfortunately lost an interesting answer where the author argues that that is only possible if the developers are really experienced in the field. I tend to agree more with Mary though it is of course a question what “high” and “low” level goals/requirements are. But undeniably users/analysts tend to propose solutions disguised as requirements while often missing the technical insight to see possible other and better solutions. We also cannot expect the developers to produce a great SW if the true goals, needs, and business values behind the requested “features” aren’t clearly communicated to them. (The best example – lost source again – is where a developer proposes to the client a simple process change that will solve the problem without writing a single line of code.)
- Mike Cohn: The Forgotten Layer of the Test Automation Pyramid – three levels of testing with increasing number of tests: UI/Service/Unit (or end-to-end instead of UI), each requiring a different approach. Unit tests are best because a failure points directly to its source (with higher level tests you don’t immediately know the cause). The higher in the pyramid, the less tests we should have (e.g. because of their redundancy). It’s important not to forget the middle, service layer – unit tests are too low-level, UI tests too difficult and brittle. Also Gojko in Specification by Examples says that acceptance/BDD tests should run mainly at the service layer because of the UI level issues.
“Although automated unit testing is wonderful, it can cover only so much of an application’s testing needs. Without service-level testing to fill the gap between unit and user interface testing, all other testing ends up being performed through the user interface, resulting in tests that are expensive to run, expensive to write, and brittle.” [Emphasis JH.]
- Technical Debt and the Lean Startup – Paul Dyson remarks that while quality is an essential concern for projects in established environments, in the case of lean startups the primary goal is to find out whether a product is viable and what it should be like and thus it’s reasonable to accept much higher technical debt by not spending too much time on ensuring scalability, de-duplication etc. – only when the product proves viable should we start to care for its long-evity by emphasizing the quality. But one thing can never miss and that is good test suite because this is the crucial factor that makes letter payment of the technical debt possible without ruining oneself.
- Coding dojo – Real time coding competition with Extreme Startup – an inspiring report about a coding dojo lead by Johannes Brodwall in Bergen’s JUG, the task being the implementation of a server that can respond to questions send over HTTP (that’s all participants know at the beginning – they learn the rest during the iterations)
- Using Code Katas to Improve Programming Skills – why to use code katas + links to different proposed katas
- Kent Beck: Don’t Cross the Beams: Avoiding Interference Between Horizontal and Vertical Refactorings - when to do depth-first (more exploratory) refactoring and when to extend it into breadth (i.e. apply it to several similar objects)
Learning Clojure (maybe not so interesting for those not learning the language)
- Phil Calçado: My Experience With TDD In Clojure (via planetclojure) – nice example of how to decompose a task in functional programming to make it easy to test (via Midje), including useful testing-related links and a discussion of side-effect isolation and the building blocks of functional programs, i.e. function composition using combinators (i.e. functions producing functions)
- How to learn Clojure effectively (via planetclojure) – a very good description of how the task at 4Clojure (though I prefer Clojure koans) should be solved to benefit one’s learning the most plus some general tips on functional thinking
- Clojure open source projects for learning how to code it
Posted in General, Testing, Top links of month | Tagged: agile, CleanCode, clojure, lean, refactoring, scrum, tdd | 1 Comment »
Posted by Jakub Holý on July 31, 2011
- Martin Fowler, M. Mason: Why not to use feature branches and prefer feature toggles instead, when branches can actually be used (video, 12min) – feature branches are pretty common yet they are a hindrance for a good and stable development pace due to “merging hells”. With trusted developers, feature toggles are a much better choice.
- M. Fowler: The LMAX Architecture – Martin describes the innovative and paradigm shaking architecture of the high-performance, high-volume financial trading platform LMAX. The platform can handle 6 million orders per second – using only a single java thread and commodity hardware. I highly recommend the article for two reasons: First, it crashes the common view that to handle such volumes you need multithreading. Second, for the rigorous, scientific approach used to arrive to this architecture. The key enablers are: 1) The main processing component does no blocking operations (I/O), those are done outside (in other threads). 2) There is no database – the state of the processor can be recreated by replaying the (persistent) input events. 3) To get further from 10k to 100k TPS they “just” wrote good code - well-factored, small methods (=> Hotspot more efficient, CPU can cache better). 4) To gain another multitude they implemented more efficient, cache-friendlier collections. All that was done based on evidence, enabled by thorough performance testing. 5) The processor and input/output components communicate without locking, using a shared (cyclic) array, where each of them operates on sum range of indexes and no element can ever be written by more than one component. Their internal range indexes do ever only increase so it is safe to read them without synchronization (at worst you will get old, lower value). The developers also tried Agents but found them in conflict with modern CPUs for their require context switch leading to emptying of the fast CPU caches.
Updated: Martin has published the post titled Memory Image which discusses the LMAX approach to persistence in a more general way.
- S. Mancuso: Working with legacy code with the goal of continual quality improvement – this was quite interesting for me as our team is in the same situation and arrived to quite similar approach. According to the author, the basic rule is “always first write tests for the piece code to be changed,” even though it takes so much time – he justifies it saying “when people think we are spending too much time to write a feature because we were writing tests for the existing code first, they are rarely considering the time spend elsewhere .. more time is created [lost] when bugs are found and the QA phase needs to be extended”. But it is also important to remember when to stop with refactoring to get also to creating business value and the rule for that is that quality improvements are done only with focus on a particular task. I like one of the concluding sentences: “Constantly increasing the quality level in a legacy system can make a massive difference in the amount of time and money spend on that project.”
- Uncle Bob: The Land that Scrum Forgot – Scrum makes it possible to be very productive at the beginning but to be able to keep the productivity and continue meeting the expectations that are thus created we need to concentrate on some essential technical practices and code quality. Because otherwise we create a mess of growing complexity – the ultimate killer of productivity. Uncle Bob advices us what practices and how to apply to attain both high, sustainable productivity and (as required for it) high code quality. It’s essential to realize that people do what they are incented to do and thus we must measure and reward both going fast and staying clean.
How do we measure quality? There is no perfect measure but we can build on the available established metrics – coverage, # (new) tests, # defects, size of tests (~ size of production code, 5-20 lines per method), test speed, cyclomatic complexity, function (< 20) and class (< 500) sizes, Brathwaite Correlation (> 2), dependency metrics (no cycles, in the direction abstraction).
The practices that enable us to stay clean include among others TDD, using tools like Chceckstyle, FindBugs to find problems and duplication, implementing Continuous Integration.
- Getting Started: Testing Concurrent Java Code – very good and worthy overview of tools for checking and testing of concurrent code with links to valuable resources. The author mentions among others FindBugs, concurrent test coverage (critical sections examined by multiple threads) measurement with IBM’s ConTest, multithreaded testing with ConTest (randomly tries to create thread interleaving situations; trial version – contact the authors for the full one) and MultithreadedTC (which divides time into “ticks” and enables you to fine-configure the interactions)
- The top 9+7 things every programmer or architect should know – quite good selection of nine, respectively 7 things from the famous (on-line available) books 97 Things every programmer/architect should know.
- Beans and noses – J. Spool reveals the First Rule of Consultancy: “No matter how much you try, you can’t stop people from sticking beans up their nose.” In other words, sometimes your clients decide to do some very unwise thing and no amount of reasoning can discourage them from that (quite understandably, as already the way they got to this decision defies logic). “The only thing I can do in a beans-and-noses situation is wait. Wait until the bean is in its final resting place.” Then you ask the person how it is working for him and “… if sticking a bean deep into their nostril doesn’t meet the very high expectations they’d had, I can now start talking alternative approaches to reaching those expectations.” Already before you can actually ask them about their expectations, in some cases (50:50) this discussion can lead them to realize they could achieve them with an alternative, less painful approach. Now, if you have read up to this point, you clearly have enough time, so go an read the article because it’s really orth it!
Posted in Languages, Testing, Tools, Top links of month | Tagged: architecture, continuous_deployment, java, Jenkins, legacy, quality, scrum | Comments Off