Posted by Jakub Holý on October 6, 2015
While refactoring a relatively simple Clojure code to use a map instead of a vector, I have wasted perhaps a few hours due to essentially type errors. I want to share the experience and my thoughts about possible solutions since I encounter this problem quite often. I should mention that it is quite likely that it is more a problem (an opportunity? :-)) with me rather than the language, namely with the way I write and (not) test it.
The core of the problem is that I write chains of transformations based on my sometimes flawed idea of what data I have at each stage. The challenge is that I cannot see what the data is and have to maintain a mental model while writing the code, and I suck at it. Evaluating the code in the REPL as I develop it helps somewhat but only when writing it – not when I decide to refactor it.
Read the rest of this entry »
Posted in Languages | Tagged: clojure, ClojureScript, refactoring | Comments Off on Refactoring & Type Errors in Clojure: Experience and Prevention
Posted by Jakub Holý on January 14, 2013
Re-published from blog.iterate.no.
Our code has been broken for weeks. Compiler errors, failing tests, incorrect behavior plagued our team. Why? Because we have been struck by a Blind Frog Leap. By doing multiple concurrent changes to a key component in the hope of improving it, we have leaped far away from its ugly but stable and working state into the marshes of brokenness. Our best intentions have brought havoc upon us, something expected to be a few man-days work has paralized us for over a month until the changes were finally reverted (for the time being).
Lessons learned: Avoid Frog Leaps. Follow instead Kent Beck’s strategy of Sprinting Centipede – proceed in small, safe steps, that don’t break the code. Deploy it to production often, preferably daily, to force yourself to really small and really safe changes. Do not change multiple unrelated things at the same time. Don’t assume that you know how the code works. Don’t assume that your intended change is a simple one. Test thoroughly (and don’t trust your test suite overly). Let the computer give you feedback and hard facts about your changes – by running tests, by executing the code, by running the code in production.
Read the rest of this entry »
Posted in General, SW development | Tagged: best practices, design, opinion, refactoring | Comments Off on The Sprinting Centipede Strategy: How to Improve Software Without Breaking It
Posted by Jakub Holý on January 31, 2012
- Jeff Sutherland: Powerful Strategy for Defect Prevention: Improve the Quality of Your Product – “A classic paper from IBM shows how they systematically reduced defects by analyzing root cause. The cost of implementing this practice is less than the cost of fixing defects that you will have if you do not implement it so it should always be implemented.” – categorize defects by type, severity, component, when introduced; 80% of them will originate in 20% of the code; apply prioritized automated testing (solve always the largest problem first). “In three months, one of our venture companies cut a 4-6 week deployment cycle to 2 weeks with only 120 tests.”
- Ebook draft: Beheading the Software Beast – Relentless restructurings with The Mikado Method (foreword by T. Poppendieck) – the book introduces the Mikado Method for organized, always-staying-green (large-scale) refactorings, especially useful for legacy systems, shows it on a real-world example (30 pages!), discusses various application restructuring techniques, provides practical guidelines for dealing with different sizes of refactorings and teams, discusses in depth technical debt and more. To sum it up in three words: Check it out!
- Daily Routine of a 4 Hour Programmer (well, it’s actually about 4h of focused programming + some hours of the rest) – a very interesting reading with some inspiring ideas. We should all find some time to follow up the field, to reflect on our day and learn from it (kaizen)
- The Agile Testing Quadrants – understanding the different types of tests, their purpose and relation by slicing them by the axis “business facing x technology facing” and the axis “supporting the team x critiquing the product” => unit tests x functional tests x exploratory testing x performance testing (and other). It helps to understand what should be automated, what needs to be manual and helps not to forget all the dimensions of testing.
- Adam Bien: Can stateful Java EE apps scale? – What does “stateless” really mean? “Stateless only means, that the entire state is stored in the database and has to synchronized on every request.” “I start the development of non-trivial (>CRUD) applications with Gateway / PDOs [JH: stateful EJBs exposing JPA entities] and measure the performance and memory consumption continuously.” Some general tips: Don’t split your web server and servlet container, don’t use session replication.
- Brian Tarbox: Just-In-Time Logging – How to remove 90% of worthless logs while still getting detailed logs for cases that matters – the solution is to (1) only add logs for a particular “transaction” with the system into a runtime structure and (2) flush it to the log only if the transaction fails or st. else significant happens with it. The blog also proposes a possible implementation in detail.
- DZone’s Top 10 NoSQL Articles of 2011
- DZone’s Top 5 DevOps Articles of 2011
- Test Driven Infrastructure with Vagrant, Puppet and Guard – this is interesting for me for I’m using Vagrant and Puppet on my project to create and share development environments or their parts and applying test-first approach to it seems interesting as do also the tools, rspec-puppet, cucumber-puppet and Guard (events triggered by file changes) and referenced articels.
- 5+1 Sonar Plugins you must not miss (2012 version) – Timeline Plugin (with Google Visualization Annotated TimeLine), Useless Code Plugin, SIG Maintainability Model Plugin (metrics Analysability, Changeability, Stability, Testability), Quality Index Plugin (1-number health indicator), Technical Debt Plugin
Links to Keep
- ClojureScript One Guide – “ClojureScript One shows you how to use ClojureScript to build single-page, single-language applications in a productive, effective and fun way.”
- Asynchronous workflows in Clojure – true asynchronous (non-blocking) network access in Clojure with Netty/the Lamina project.
- Clojure 2011 Year in Review – a list with important events in the Clojure sphere with links to details – C. 1.3.0, ClojureScript, logic programming with core.logic, clojure-contrib restructuring, birth of 4Clojure and Avout.
- Clojure Atlas – interesting project (alpha version) presenting Clojure documentation in the form of interactive graph of related concepts and functions; it’s far from perfection but I like the concept and consider paying those ~ $25 for the 1.3.0 version when its out (however, the demo is free and it might become open-sourced in 2012)
Posted in General, Languages, Testing, Tools, Top links of month | Tagged: book, clojure, DevOps, java, lean, legacy, logging, performance, quality, refactoring, Testing | Comments Off on Most interesting links of January ’12
Posted by Jakub Holý on November 21, 2011
To learn how complex your code base really is and how much effort a particular refactoring might require compared to the initial expectations, follow these steps:
- Schedule git reset --hard; git clean -fd to run in 1 hour (e.g. via cron)
- Do the refactoring
- “WT*?! All my changes disappeared?!” – this experience indicates the end of the refactoring :-)
- Go for a walk or something and think about what you have learned about the code, its complexity, the refactoring
- Repeat regularly, f. ex. once every week or two – thus you’ll improve your ability to direct the refactoring so that you learn as much as possible during the short time
Read the rest of this entry »
Posted in General | Tagged: agile, exercise, learning, legacy, refactoring | Comments Off on Refactoring Spikes as a Learning Tool and How a Scheduled Git Reset Can Help
Posted by Jakub Holý on September 30, 2011
- J. Yip: It’s Not Just Standing Up: Patterns for Daily Standup Meetings – it isn’t easy to make stand-up meetings short, focused, energizing, and centered around continuous improvements and team spirit. This description of an example good standup, the meeting’s goals, and especially the “patterns” and “bad smells” can be pretty useful to get and keep on the track towards a brighter future. TBD: standup goals: GIFTs, team spirit, appreciation where we go and are.
- M. Poppendieck: Don’t Separate Design from Implementation – according to Mary, (detailed) requirements – being it in the form of (backlog) user stories or any other – represent actually a design of the system, which shouldn’t be done by the amateur product owner/business analyst but by professionals, meaning the developers, based on high-level goals and clear specification of the desired business value. She writes about a project that her factory outsourced and which she could have designed but didn’t – yet it succeeded even though there were no detailed requirements. I’ve also read and unfortunately lost an interesting answer where the author argues that that is only possible if the developers are really experienced in the field. I tend to agree more with Mary though it is of course a question what “high” and “low” level goals/requirements are. But undeniably users/analysts tend to propose solutions disguised as requirements while often missing the technical insight to see possible other and better solutions. We also cannot expect the developers to produce a great SW if the true goals, needs, and business values behind the requested “features” aren’t clearly communicated to them. (The best example – lost source again – is where a developer proposes to the client a simple process change that will solve the problem without writing a single line of code.)
- Mike Cohn: The Forgotten Layer of the Test Automation Pyramid – three levels of testing with increasing number of tests: UI/Service/Unit (or end-to-end instead of UI), each requiring a different approach. Unit tests are best because a failure points directly to its source (with higher level tests you don’t immediately know the cause). The higher in the pyramid, the less tests we should have (e.g. because of their redundancy). It’s important not to forget the middle, service layer – unit tests are too low-level, UI tests too difficult and brittle. Also Gojko in Specification by Examples says that acceptance/BDD tests should run mainly at the service layer because of the UI level issues.
“Although automated unit testing is wonderful, it can cover only so much of an application’s testing needs. Without service-level testing to fill the gap between unit and user interface testing, all other testing ends up being performed through the user interface, resulting in tests that are expensive to run, expensive to write, and brittle.” [Emphasis JH.]
- Technical Debt and the Lean Startup – Paul Dyson remarks that while quality is an essential concern for projects in established environments, in the case of lean startups the primary goal is to find out whether a product is viable and what it should be like and thus it’s reasonable to accept much higher technical debt by not spending too much time on ensuring scalability, de-duplication etc. – only when the product proves viable should we start to care for its long-evity by emphasizing the quality. But one thing can never miss and that is good test suite because this is the crucial factor that makes letter payment of the technical debt possible without ruining oneself.
- Coding dojo – Real time coding competition with Extreme Startup – an inspiring report about a coding dojo lead by Johannes Brodwall in Bergen’s JUG, the task being the implementation of a server that can respond to questions send over HTTP (that’s all participants know at the beginning – they learn the rest during the iterations)
- Using Code Katas to Improve Programming Skills – why to use code katas + links to different proposed katas
- Kent Beck: Don’t Cross the Beams: Avoiding Interference Between Horizontal and Vertical Refactorings – when to do depth-first (more exploratory) refactoring and when to extend it into breadth (i.e. apply it to several similar objects)
Learning Clojure (maybe not so interesting for those not learning the language)
- Phil Calçado: My Experience With TDD In Clojure (via planetclojure) – nice example of how to decompose a task in functional programming to make it easy to test (via Midje), including useful testing-related links and a discussion of side-effect isolation and the building blocks of functional programs, i.e. function composition using combinators (i.e. functions producing functions)
- How to learn Clojure effectively (via planetclojure) – a very good description of how the task at 4Clojure (though I prefer Clojure koans) should be solved to benefit one’s learning the most plus some general tips on functional thinking
- Clojure open source projects for learning how to code it
Posted in General, Testing, Top links of month | Tagged: agile, CleanCode, clojure, lean, refactoring, scrum, tdd | 1 Comment »
Posted by Jakub Holý on August 31, 2011
- Martin Fowler on the problem of software patents – “… while patents (even software patents) are a good idea in principle, in practice they have turned into an unmitigated disaster and would be better scrapped.”
- Discovering Hidden Design, Michael Feathers – When refactoring complex code towards a better design with clearer separation of concerns, it may be sometimes worthwhile to factor out a more-less separated cluster of functionality even if it doesn’t do just one thing (and this it is difficult to find a descriptive name for it). Comparing the cost and benefit of this and an “ideal” refactoring (where we get to single-responsibility factors), this one may prove better.
- Martin Fowler: Tradable Quality Hypothesis – Martin argues that we must claim that quality in software development is not tradable (even though we know that certain tradeoffs can be done). The reason is that people are used to quality (in food, clothing, …) being pretty “tradable” and so it is very hard to persuade them that in the case of software development it is tradable much less (or not at all). And once your manager and customers view quality as tradable, you are doomed. They will force you to trade it for time, features, … in a proportion that will harm the project (because, as already mentioned, in SW it is much less tradable then in other domains).
- Are estimates worthless?& Magne’s response – interesting discussion of the value and cost of estimation and its role in contracting w.r.t. trust – a nice addition to the discussion: Agile not suitable for governmental IT?.
- Generate Test Data with DataFactory – nice java library that generate “random” values of different types and optionally satisfying some constraints – f.ex. first/last name (using built-in or custom list), date (within a range, w.r.t. another date, …), address (cities, streets etc.), email, random text/word/characters, number. Available at GitHub.
Posted in General, Testing, Tools, Top links of month | Tagged: agile, design, quality, refactoring, Testing | Comments Off on Most interesting links of August
Posted by Jakub Holý on May 31, 2011
Acceptance testing / Specification by example:
- Gojko Adzic: Anatomy of a good acceptance test – an example of refactoring a bad acceptance test into a good one – good for learning about pitfalls and how a good one should look like
- Gojko: Top 10 reasons why teams fail with Acceptance Testing – acceptance testing is great and brings lot of value but must not be underestimated; some of the problems are bad collaboration, focusing on “how” instead of “what,” confusing AT with full regression tests. Brief, worth reading.
- Specification by Example: a love story (go directly to the PDF with the story): A nice, made-up story of going from low-level, workflow-based Selenium tests through similar Cucumber ones to true BDD tests describidng clearly what, not how – very well shows the point of specification by example and how it should (and should not) look like
(Enterprise) Java best practices:
- Clean code, clean logs: 10 brief posts on logging best-practices – nothing really new here for me but in total it is a very good overview that every developer should know
- Make Large Scale Changes Incrementally with Branch By Abstraction – Continuous integration doesn’t work well with branches but as this article shows, you can manage even large-scale refactorings without branches using “branch by abstraction,” an approach reminding me of Fowler’s “strangler application” (an incremental replacement of a legacy system). The idea is: 1. Create an abstraction over the part of code to be changed; 2. Refactor the code to use it; 3. Implement new functionality using the new way / step by step move old functionality too, the abstraction layer delegating either to the new or old implementation … . It may be more work but: 1) your software is always working and deliverable; 2) (side-effect) in the end it will be more decoupled
- John Wiegley’s Git from the bottom upp (31p, Git 22.214.171.124, PDF) – a useful explanation of the fundamentals of Git, i.e. how it is constructed and how it works, which makes it much easier to understand how to use it properly (recommended by Pål R.). Reading the The Git Parable first may be a good idea for an easy introduction into the fundamentals, though absolutely not necessary. This document introduces very well the important Git concepts (blob, index, commit, commit names such as branches, reflog) and how they cooperate to provide the rich set of functionality it has. It also explains well the value and usage of rebase. Among others I’ve appreciated the tip to use checkout, branch -m <new-branch> master, branch -D instead of the much more dangerous reset –hard and the tip to use stash / stash apply to create daily backups of your working tree in the reflog (with clearing it with ‘git reflog expire –expire=30.days refs/stash‘ instead of stash clear). Also git diff/log master..[HEAD] for reviewing work done in the current branch and and git diff/log ..master for checking the changes since the last merge/rebase after a fetch are interesting.
- The secret power of bookmarklets – bookmarklets are an indispensable tool for every developer who works with web applications (to fill in test data, speed up log in, …), yet I’m sometimes surprised by meeting people who don’t know or use them; this blog explains them nicely, links to some useful ones and some useful tools for building them
- (*****) Implementing Lean Software Development: From Concept to Cash by Mary Poppendieck, Tom Poppendieck – A great introduction into lean thinking (the values and principles it is build upon), clearly communicated with the help of “war stories”. I absolutely recommend it to anybody interested in lean/agile.
- (**** ) Agile Project Management with Scrum (Microsoft Professional) by Ken Schwaber – Even though you can’t understand Scrum without experiencing it, this book full of war stories will help you to avoid many Scrum implementation pitfalls and to understand its mantra of “the art of the possible” and will show you how to adapt Scrum to various situations. It’s very easy to read thanks to its format of brief case studies organized by topics (team, product owner, …).
Favourite Quotes of the Month
@unclebobmartin: Cleaning code does NOT take time. NOT cleaning code does take time.
Posted by Jakub Holý on April 16, 2011
I’m preparing a coding dojo for my colleges at Iterate where we will try to collectively refactor the “legacy” Hudson/Jenkins, especially Hudson.java, to something more testable, using the Mikado Method. I’ve got the idea after reading Gojko Adzic’s blog on how terrible the code is and after discovering the Mikado Method by a chance. Since a long time I’m interested in code quality and since recently especially in improving the quality of legacy applications, where “legacy” means a terrible code base and likely insufficient tests. As consultants we often have to deal with such application and with improving their state into something easier and cheaper to maintain and evolve. Therefore such a collective practice is a good thing.
The Mikado Method
The Mikado Method, which the authors describe as “a tool for large-scale refactorings”, serves two purposes: Read the rest of this entry »
Posted in General, Languages | Tagged: book, java, legacy, quality, refactoring | 1 Comment »
Posted by Jakub Holý on November 25, 2010
I had high expectations for Martin Fowler’s Refactoring (1999/2002) but it turned out that both me and the book are too old. It had some interesting parts, but the main one – the refactoring catalog itself – had little new for me because I already know most of the refactorings and the description of steps how to perform them safely is nowadays essentially useless as they’re already automatically and safely performed by our IDEs.
I’ve enjoyed chapter 1 with a nice example of how bad code is turned into a nice one via a series of refactorings and I’d recommend it to any beginning developer. For others than me also chapter 2 may be useful, it explains why and when to refactor, how it impacts your development speed and how to justify it to the manager. I’d skip chapter 3 – bad smells in the code – and read instead of it the Uncle Bob’s lovely Clean Code , which is a great justification and basis for refactoring anyway.
In the catalog I’ve appreciated the description of some refactorings for various reasons, such as 6.4 Replace Temp with Query, 8.14 Replace Type Code with Subclasses, 9.7 Introduce Null Object, 9.8 Introduce Assertion. Of course also the other ones are good but they are just too familiar to me and my IDE to draw my attention anymore.
Chapter 12, Big Refactorings, is quite interesting, especially Tease Apart Inheritance.
Finally there are some good advices in Putting It All Together by Kent Beck, such as that it’s good to know when to stop and to be able to resist the temptation to refactor too much at once.
I’ve also appreciated the “war stories”, especially regarding performance tuning, where once again it’s demonstrated that a guess (however founded) is incomparable to hard evidence.
If you do refactorings daily, perhaps skip the book. If not, read chapter 1, perhaps browse through 2 and 12, check whether anything catches your eye in the refactoring catalog table of content, read Kent’s closing chapter and return the book to the local museum.
Posted in Languages | Tagged: book, java, refactoring, review | Comments Off on Book review: Refactoring by Martin Fowler
Posted by Jakub Holý on July 9, 2010
The last month’s list of interesting articles is a bit shorter due to off-line holidays, which I enjoyed much more then reading articles :-)
Posted in Languages, Top links of month | Tagged: html5, java, refactoring | Comments Off on Most interesting links of June