Posts Tagged ‘development’
Posted by Jakub Holý on February 28, 2014
- Nathan Marz: Principles of Software Engineering, Part 1 – Nathan has worked with Big Data at Twitter and other places and really knows the perils or large, distributed, real-time systems and this post contains plenty of valuable advice for making robust, reliable SW. Main message: “there’s a lot of uncertainty in software engineering“; every SW operates correctly only for a certain range of inputs (including volume, HW it runs on, …) and you never control all of them so there always is an opportunity for failure; you can’t predict what inputs you will encounter in the wild. “[..] while software is deterministic, you can’t treat it as deterministic in any sort of practical sense if you want to build robust software.” “Making software robust is an iterative process: you build and test it as best you can, but inevitably in production you’ll discover new areas of the input space that lead to failure. Like rockets, it’s crucial to have excellent monitoring in place so that these issues can be diagnosed.“. From the content: Sources of uncertainty (bugs, humans, requirements, inputs, ..), Engineering for uncertainty (minimize dependencies, lessen % of cascading failure [JH: -> Hystrix], measure and monitor)
- Suffering-oriented programming is certainly also worth reading (summary: do not start with great designs; only start generalizing and creating libs when you have suffered enough from doing things more manually and thus learned the domain; make it possible > make it beautiful > make it fast, repeat)
- ThoughtWorks open-sources Go, continuous delivery platform – good bye, Jenkins! – better support for pipelines etc., see features and elementary concepts
- Cloud Design Patterns: Prescriptive Architecture Guidance for Cloud Applications (recommended by @markusbk so it must be good); Patterns: Cache-aside, Circuit Breaker, Compensating Transaction, Competing Consumers, Compute Resource Consolidation, Command and Query Responsibility Segregation (CQRS), Event Sourcing, External Configuration Store, Federated Identity, Gatekeeper, Health Endpoint Monitoring, Index Table, Leader Election, Materialized View, Pipes and Filters, Priority Queue, Queue-based Load Leveling, Retry, Runtime Reconfiguration, Scheduler Agent Supervisor, (data) Sharding, Static Content Hosting (-> CDN), Throttling, Valet Key.
Guidance topics: Asynchronous Messaging Primer, Autoscaling, Caching, Compute Partitioning, Data Consistency Primer, Data Partitioning, Data Replication and Synchronization, Instrumentation and Telemetry, Multiple Datacenter Deployment, Service Metering
- MOOC course Functional programming with Clojure at Uni of Helsinki – to get started you need, I suppose, follow the “Material and course content” – essentially read the text for each chapter, clone its repo, submit pull requests to get your work graded
- Jez Humble: The Case for Continuous Delivery – read to persuade manager about CD: “Still, many managers and executives remain unconvinced as to the benefits [of CD], and would like to know more about the economic drivers behind CD.” CD reduces waste: “[..]online controlled experiments (A/B tests) at Amazon. This approach created hundreds of millions of dollars of value[..],” reduces risks: “[..] Etsy, has a great presentation which describes how deploying more frequently improves the stability of web services.” CD makes development cheaper by reducing the cost of non-value-adding activities such as integration and testing. F.ex. HP got dev. costs down by 40%, dev cost/program by 78%
- Request Quest (via @jraregris) – entertaining and educational intractive quiz regarding what does (not) trigger a request in browsers and differences between them (and deviances from the standard) – img, script, css, etc.
- The REST Statelessness Constraint – a nice post about statelessness in REST if you, like me, don’t know REST so much in depth; highlights: Statelesness (and thus the requirement for clients to send their state with every request) is a trade-off crucial for web-scale and partially balanced by caching – while typical enterprise apps have different needs (more state, less scale) so REST isn’t a perfect match. Distinguish application (client-side) and server (resources) state. Using a DB to hold the state still violates the requirement. Use links to transfer some state (e.g. contain a link to fetch the next page of records in the response).
- CodeMesh 2013 presentations – good stuff! F.ex. Refactoring Functional Programs: Past and Future, Distribution, Scale and Flexibility with ZeroMQ, Deepak Giridharagopal on Puppet, Immutable Deployments, Analyzing Systems with PuppetDB, Francesco Cesarini and Viktor Klang on the Reactive Manifesto and more
- Cognitive Biases in Times of Uncertainty – people under pressure/stress start to focus on risks over gains and (very) short-term rather than long-term and thus also adopt 0-some mindset (i.e. if sb. else wins, I loose) => polarization into we x them and focus on getting as big piece of the cake possible at any price, now, dismissal of collaboration. With accelerating rate of change in the society due to technology, this is exactly what is happening. How to counter it? Create more positive narratives than the threat-based ones (views of the world), support them via short-term gains. Bottom line: each of us must work on spreading a more positive attitude to save us from bleak future.
- Book – Nathan Marz: Big Data – I dislike the big data hype (and, with passion, Hadoop) but would love to read this book; it presents a fresh look at big data processing, heavily inspired by functional programming. Nathan has plenty of experiences from Twitter and creating Storm and Cascalog (both in Clojure, btw.). Read ch 1: A new paradigm for big data.
- Facebook Engineering: The Mature Optimization Handbook (or go directly to the pdf, ePub, Mobi). If you get bored, jump directly to ch 5. Instrumentation.
Posted by Jakub Holý on February 2, 2014
(Cross-posted from blog.iterate.no.)
Technical debt is not the only monster we have to fight – it has a hidden evil twin, as pointed out by Niklas Björnerstedt: Competence Debt. The rope of ignorance that binds our hands and suffocates us by fear so that we don’t dare to change the system. Technical debt makes change difficult because the structure of the system does not support it. Competence debt makes change difficult because we do not know the system well enough, what & where to change and what impacts a change may have.
There is an often neglected tool at our fingertips that might help us fight competence debt. Its name is – behold – JavaDoc. An example from practice: I have returned to a client after two years and needed to understand the functionality of a part of the system. And, to my surprise, I found my own JavaDoc providing exactly the answer I was looking for. A colleague of mine mentioned that I should get the award for the “most documenting developer”. But I don’t do it for fun or just to help my bad memory and to be nice to my colleagues. It is an important contribution to the fight against the ever growing hydra of legacy code. Next time you code, try to remember that you are not typing code, but fighting. As every fight, it is hard – but do not give up or the enemy will prevail.
Side note: Writing JavaDoc that helps yet is not too verbose and not too likely to get outdated soon is hard. Getting the right balance – neither too little nor too much, focusing on the why and the broader context and relationships instead of the changing implementation etc. is difficult. But it is worth it. Help yourself, help your fellow colleagues, strike the hydra. Write good JavaDoc.
PS: This post is about competence debt even though the title mentions technical debt, sorry for the confusion (even though they are two sides of the same coin). And “good enough” documentation, though important, is not the sole remedy, as well as developers’ lack of knowledge is not the sole cause of competence debt. Also, Niklas provides a more in-depth review of the technical & competence debt terms in Misunderstanding technical debt (tech. debt as evolving understanding [but not code] and crappy code). He wrote also A deeper look at Competence debt.
Posted in General | Tagged: development, legacy | Comments Off on A Secret Weapon Against Technical Debt
Posted by Jakub Holý on December 6, 2013
If we knew exactly what code needs to be written, what needs to be done and how it can be done, we would need very little time to write it. It is the discovery of the knowledge what to build and how to build it that takes all the time. Yet non-developers usually see it as an unacceptable waste to write and then throw away – and perhaps rewrite – code.
Read the rest of this entry »
Posted in SW development | Tagged: development, opinion | Comments Off on Code Is Cheap, It’s Knowledge Discovery That Costs
Posted by Jakub Holý on March 31, 2013
A lot of stuff this month since I have finally got time to review some older articles. Quite a few articles by Fowler. Few really great (yet short) talks on agile & SW development.
- Agile in a Nutshell (originally Agile Product Ownership in a Nutshell) by Henrik Kniberg – the best explanation of the agile development process ever, in just 15 minutes and with wonderful animation; every developer should see this. Some highlights: the most important task of product owner is to say NO so that backlog doesn’t grow infinitely; at start, the estimates of size and value will suck and that’s OK because the value is in the conversation, not in the numbers (that are anyway just relative); the goal is to maximize outcome (value), not output (# features). Compromises between short-term vs. long-term goals, knowledge vs. customer value building etc. Build the right thing (PO) x build it right (devs) x build it fast (SM). Technical debt x sustainable pace. As I said – you MUST see it.
- Martin Fowler: The Value of Software Design (talk, 22 min, from 0:45:00 til 1:07; Feb 2013) – a balanced argument for the value of good software design and internal code quality based on paying off by enabling us to keep our development speed. Discusses the DesignStaminaHypothesis (bad design => rapid decline of development speed), TechnicalDebt, TechnicalDebtQuadrant (Prudent x Reckless, Deliberate x Inadvertent), TradableQualityHypothesis. According to the experience of Fowler and others, the good design payoff point “it’s weeks, not months.”
- What Does It Take To Become A Grandmaster Developer? – great post about cognition and learning, valuable references, quotes from an interesting study of good vs. mediocre developers. We have mental capacity for ~7 chunks of information => great performers recognize patterns and see and understand thus higher-level chunks and have many “chunks” (patterns encountered previously) readily available. You need deliberate effort to learn more chunks – especially initially but you must always try to get out of your comfort zone to grow. Experienced collegues can help a lot in acending the learning curve.
Agile, organization, innovation, project management
- How to Prioritize a User Story Map – we all know that we should prioritize features by their value, risk, and lack of knowledge and that we should slice the features thin so that they fit into short iteration and can be deployed soon to produce feedback, right? Here we see a nice example of what happens if not done so and how to do feature slicing better.
- Bob Marshall: Rightshifting – according to the author, 80% of knowledge work organizations are very ineffective, wasting resources on non-value-adding activites; only few are effective, even fewer highly effective. Rightshifting is the attempt at shiting them to the right, towards higher effectiveness. Links to a few videos explaining it more. Related: Steve McConnell’s Business Case for Better Software Practices, referring to a study by SEI; “The actual distribution of software effectiveness is asymmetric. Most organizations perform much closer to the worst practice than to the best.” – the best performing 10 times better then the worst/average (productivity, speed, defects, value)
- On Antifragility in Systems and Organizational Architecture – introduces the concept of antifragility, based on Nassim Taleb’s book Antifragile that compares fragile, robust, and antifragile systems and organizational structures (which is also applicable to SW systems); robust = resists change (unless too large); antifragile: learn, adapt; closely related to DevOps and continous delivery
- M. Fowler: PurposeOfEstimation – many Agilist disdain estimation, this is a balanced view: “estimation is valuable when it helps you make a significant decision.” (F.ex. when deciding what we (don’t) have resources for or when in need of coordinating related activities.) It is evil when used as commitments that people are forced to stick to and blamed for not managing to do so. “Above all be wary of anyone who tells you they [estimates] are always needed, or never needed.” A. Ferguson: “[..] it is poor project management (whether by project managers or other team members) that results in a client who thinks estimates are fixed, or that raw estimates = actual effort/duration”.
- Ron Jeffries: Estimation is Evil – discusses the problems estimates can cause, issues with requirements gathering up front and their volatility, transparency and politics. Very valuable, highly recommended. See the “favorite quotes” at the bottom of this post. Also contains an interesting lesson learnt from the failed Chrysler C3 project: don’t try to build a grand new system to replace and fix the old one, fix one problem at a time – worth reading for this alone.
- Interview with Steve Blank: Why Big Companies Can’t Innovate – the 2013 list of the world’s 50 most innovative companies has only a few large, established firms (those that have built innovation into its DNA such as Apple and Google). Established companies are less innovative because they focus in their existing business model, have risk-aversion (while there are many failures on the way to a new business model); finally “the people who are best suited to search for new business models and conduct iterative experiments usually are not the same managers who succeed at running existing business units.” – and thus aren’t given the chance. “[..] the process of starting a new business [..] is fundamentally different from running an existing one. So if you want your company to grow organically, then you need to organize your efforts around these differences.”
Architecture & Ops
- M. Fowler: Schemalessness + NoSQL and Consistency (20 + 20 min) two short, very good, balanced talks about NoSQL. He explains schemalessness and consistency and points out common misunderstanding about them so if you are into NoSQL, watch it.
- What Powers Instagram: Hundreds of Instances, Dozens of Technologies (2012) – interesting high-level overview of the Instagram infrastructure based on AWS and Python (25 XL instances running Django/Gunicorn behind ELB with 3 Nginxes, sharded PostgreSQL with streaming replication on 12 QXL mem instances with software raid and XFS to freeze when snapshoting, media in S3, Redis, Solr for geo-search, Memcached. Gearman for task queues, pyapns for notifications. Munin for monitoring.)
- The Netflix API Optimization Story – how Netflix redesigned its APIs to improve performance, reduce chattiness, and power product development and experimentation. The common REST API has become a development bottleneck and a lowest common denominator solution (w.r.t. supporting various clients). The main changes were: usage Hystrix for fault tolerance, each device team managing their own end-points in any JVM languges (primarily Groovy) and re-using common APIs (i.e. pushing some device-specific code to the server) => able to experiemnt more quickly, using the Functional Reactive Programming Model and asynchronous APIs (to abstract away thread-safety and parallel execution implementation details from the device teams so that code can execute sync. or async. without them needing to know).
- Me: Overview of current monitoring libs for Java – Netflix’ Servo, Yammer’s Metrics, JavaMelody, JavaSimon.
- Debug Servlets, or ‘HTTP Won; Use It’ – expose all debugging info of your services over HTTP – it makes debugging much simpler. We do a part of it and it really helps. Expose config (values, where they come from), logs, log configuration, JMX (setting it up otherwise not trivial), version, build number, git hash, server time (timezones tricky), metrics, stack dumps, app-specific status (Hadoop: live nodes, data size etc.). The author recommends JavaMelody to collect & visualize many common metrics. Not on security: Make sure to hide passwords and make the endpoints visible only internally. (Tip: consider Jolokia for exposing JMX over HTTP, see below.)
- JVM Crash/Core Dump Analysis – 3 common categories of JVM crash causes (JVM/JIT/JNI) and how to recognize and troubleshoot them
- How to lose wight in the browser: The definitive front-end performance guide – a site by a number of experts from Twitter, Opera, Google, and other places with best practices for performant web sites (HTML, CSS, JS, jQuery, images). Ex.: styles up top, scripts down bottom; minify your html, css and JS; async script loading; combine css/JS files into one; cache array lengths while looping; use css sprites for icons.
- Luke Stevens: The harsh truth about HTML5′s structural semantics (part 1) – “HTML’s structural elements — article, section, nav and aside — are, at first glance, some of the easiest parts of the HTML5 specification to understand and implement. However, they’re actually some of the most poorly specified, poorly understood, and poorly implemented parts of HTML5.” Interesting: The “research” leading to their establishment was quite random, ignoring a crucial source of information (css IDs).
- Marco Emmanuel Patiño: Six non-technical books every programmer should read – 1. Team Geek: A Software Developer’s Guide to Working Well with Others (-> effective communication and collaboration), 2. The Pragmatic Programmer: From Journeyman to Master, 3. The Passionate Programmer: Creating a Remarkable Career in Software Development, 4. Clean Code: A Handbook of Agile Software Craftsmanship, 5. 97 Things Every Programmer Should Know: Collective Wisdom from the Experts (online), 6. Code Simplicity: The Fundamentals of Software.
- Related: Top 5 Java programming books – Best of lot (actually 8) – 1) Head First Java, 2) Effective Java, 3) Thinking in Java, 4) Head First Design Pattern, 5) Concurrency Practice in Java, 6)Java performance, 7) Java Puzzlers, 8) Head First Object Oriented Analysis and Design.
- Humans as slaves of chemistry: America’s Real Criminal Element – Lead – a fascinating article about how whole nations can be seriously influenced by a single chemical substance. Aside of that it is also fascinating to observe how we tend to search for causes in our domain of expertise (police, sociologists, …) and of interest while denying other possible causes, no matter how strong are the proofs. If the facts presented are true, then the fivefold increase in serious crimes in (not only) America since 60s has been caused by the increase of lead in the environment (pushing many people over the edge of ocassional violent loss of control). How many social problems in the world have similar industrial causes? Are we careful enough with what we let into our air and bodies?
- newcoder.io: Learning more Python via projects – an excellent next step when you have learned Python syntax via LPHW or similar; in this tutorial series you will be building real-world apps while learning more of Python. You will play with, Data Visualization, APIs, Web Scraping, Networks, GUI.
- Brian McCallister: Go is PHP for the Backend – a very good explanation why you might want to use Go and that you have to first learn “the Go way” to avoid insanity, since it is very opinionated and different from what you might be used to. Some pros: “native code, UNIX friendly, higher level then C, lower level then Python or Ruby, garbage collected, strongly typed, good performance, good concurrency support, etc.”
- The Neophyte’s Guide to Scala 1 to 15 (list) – a good follow-up on the Cursera FP in Scala course, a series of blog posts exploring some topics more in depth. F.ex.: extracotrs (unapply, for pattern-matching), the broad applicability of pattern matching, pattern matching anonymous functions & partial functions #4, usiong Option idiomaticly #5, nice FP error handling with the Try type #6, Futures, etc. Higly recommended! Thx to Jakob Lind
- Jolokia is remote JMX with JSON over HTTP: a REST API bridged to JMX, with support for security, fine-grained access control, bulk operations. Especially useful if you either 1) need to perform bulk operations (e.g. get multiple values) or 2) want to access them from something that doesn’t support JMX. JSON is in general very easy to use and navigate. You can install Jolokia as a WAR (or mebedd its Servlet), a JVM agent, or attach it on-the-fly to a running JVM.
- The Appeal and Controversy of ZeroMQ – why to use 0MQ? It is a messaging library that focuses on performance, decentralization and simplicity, solving some really hard problems (sending async. messages w/o locks, distribuing to specific subscribers) and providing a simple API. Main pros: decentralized (no central broker), many languages; cons: no security (but you can use it over SSH).
- Tim O’Reilly: Create More Value Than You Capture (30 min + questions) – build apps that matter, that change how we do things. Thinking just about money is bad. Try to make the society better, smart, create more value than you capture, solve important problems, help people. Ex. startups: Uber, Square, Code for America.
- TED: Bruce Feiler: Agile programming — for your family (20 min) – an inspirational talk, based on positive experience from multiple families, about applying the agile thinking and values to make our families happier by empowering the children (enlist them in their upbringing, deciding on goals, rewards, punishments), letting them know who they are, being adaptive, having regular “retrospectives” (that eventually become cherrished memories). Backed by research. Did you know that the #1 wish of children isn’t that parents spend more time with them but that they are less stressed?
- What’s new in Clojure 1.5.x – reducers, new threading macros (cond->, as->, some->, ..), various improvements, improved performance, erro messages, doc strings, bug fixes
- Stuart Sierra: On the Perils of Dynamic Scope – summary: don’t create macros like with-connection binding to a thread-local var; make all methods take the resource as a parameter – thus the user has the freedom to decide when to close the resource and isn’t limited to a single thread and can use lazy sequences
- Logic programming is overrated – core.logic is essentially only a complex DSL for doing an exhaustive search and there is already a nice, clean tool for that: the for comprehension. A logic puzzle can be much more clearly and also efficiently using for. But it is not completely useless – logic programming is good e.g. for running programs backwards, unification is important for writing type checkers, and the new constraint programming piece has good potential. Read also Logic Programming is Underrated, which provides a faster core.logic solution than for-comprehension and provides some pointers rgarding the practical usefulness of core.logic.
- Prismatic – Graph: Abstractions for Structured Computation – How to reduce the complexity overhead in large, real-world, FP systems by decoupling what is done from how it is executed. Graph is a Clojure library enabling a declarative way to describe how data flows between (mostly pure) functions => “It allows us to formalize the informal structure of good FP code, and enables higher-order abstractions over these structures that can help stamp out many persistent forms of complexity overhead.” By decoupling the description of how data flows and the actual execution, we can execute it in different ways (parallelized, with memoization, lazy/eager) and apply various interceptors (for logging etc.). See especially the part “Graph and complexity overhead.”
- Mike Anderson: Game development in Clojure : Alchemy 7DRL post-mortem (and the previous 7 daily updates, Alchemy @ GitHub) – an interesting report about game making in Clojure during 7 days, in as functional and immutable style as possible while keeping it sufficiently fast. How do you represent & handle statuf game objects, the world map, game state? The design of the game, what was easy and what hard with Clojure. Tl;dr: search it for “Some parting thoughts” (Clojure productive, immutability hard but pays off, prototype objects great, more typing would have helped). “Making everything immutable in Clojure is harder than it would have been in an OOP language like Java where everything can be encapsulated in mutable classes. In particular, the state update functions are tricky to make both correct and performant. The payoff is big however: in terms of the simplicity and effectiveness later on, and in the conceptual clarity being able to treat the entire game state as an immutable value”. Having REPL is a big win.
- Refactoring Java using Clojure with the Eclipse Java development tools (JDT) (operation on AST nodes, i.e. little too low level; the Eclipse Refactoring API might be better)
- Clojure at a Bank – [Our] Clojure Code Immaturity – experiences with going from Java to Clojure: 1) too few comments, too short names => hard to learn the code; 2) not knowing clojure.core well enough => reimplementing (if-let, juxt, …); 3) structure, comment, split up your namespaces well, navigating more complicated then in Java IDEs; 4) reasonably used Macros, Protocols, Defrecords payed off;
- Datomic for Five Year Olds – explaining the key characteristics of Datomic compared to relational and NoSQL DBs (schema, architecture, programmability/language); doesn’t go into details of how it works (e.g. how does Datomic determine what subset of the DB to cache locally and what if it is few GBs); Honey Badger’s 2012 talk Exploring Datomic: a database deconstructed explores the architecture and technical details much more
- Vagrant 1.1.0 is out (what’s new?), with support for VMWare Fusion and AWS VM backends in addition to VirtualBox – use the same config to create, provision, stop, destroy and connect to a virtual machine locally or in the cloud (with limited support for shared folders, I’d suppose). V. 1.1 is backwards compatible aside of plugins, upgrade to new config optional.
- Animated presentations: ArtRage (drawing program, also for iPad), Wacom Intuos 5 (drawing tablet), Screenflow (screen & audio capture) – used for the Agile in a Nutshell (Agile Product Ownership in a Nutshell) mentioned above
- ckjm — Chidamber and Kemerer Java Metrics (via Neal Ford) – a command-line tool (also Maven/Ant plugin) to compute some metrics, outputting text or XML for further processing; the metrics: WMC: Weighted methods per class (cyclomatic complexity of its methods), DIT: Depth of Inheritance Tree, NOC: Number of Children, CBO: Coupling between object classes, RFC: Response for a Class, LCOM: Lack of cohesion in methods, NPM: Number of Public Methods, Ca: afferent coupling.
- Bulletproof Demos: Record & Replay built into Chrome – ever got a failure while demonstrating a web app though it has worked moments ago? No more! You Chrome to record your requests and responses and let its cache handle them during the real demonstration. (Mac: stop Chrome, to record run open -a “Google Chrome” –args –record-mode, to replay run open -a “Google Chrome” –args –playback-mode. Linux: google-chrome –record-mode and –playback-mode. Win.: run chrome <arg>)
- UserTesting.com (via Ash Maurya, the author of Running Lean) – on-demand usability testing; they have a large base of test users, can select those matching your criteria and unleash them upon your site guided by a script your provide, watch videos of their actions while they verbalize their thinking process, recieve written answers from them, talk to them.
- MindMup.com – opensource, free mind-mapping in the cloud by Gojko Adzic & co., with main focus on productivity. Store private maps in Goolge Drive, support for mobile devices, keyboard shortcuts. No registration needed.
Once we estimated a project to require 9 man-months but were later told that we do not understand a thing and it may not take more then 4. At the end it took over 25 and still wasn’t done.
– paraphrasing my collegue Kim Leskovski
On collecting requirements up front:
At the very beginning, we know less about our project than we’ll ever know again. This is the worst possible moment to be making firm decisions about what we “require.”
– Ron Jeffries in Estimation is Evil: Overcoming the Estimation Obsession
On the estimate of project delivery date at its initial phase:
It’s based on an unrealistic list of requirements, using weak estimates, made at the moment of maximum ignorance, by people who are always optimistic about their own abilities.
On planning and requirements (the Chrysler’s C3 payroll project, having a payroll expert and a team familiar with the domain):
This was one of the best-planned projects I’ve ever seen, and even so, at least one third of the requirements were added, removed, or substantially changed.
[..] a line of code is a liability, not an asset [..]
– Jez Humble in Why Software Development Methodologies Suck
Agile is not something you do, it is something you are.
– Huib Schoots in Creating my own flow
with personal kanban, Agile Record Feb 2013
Posted in General, SW development, Tools, Top links of month | Tagged: agile, aws, book, CleanCode, clojure, cloud, design, development, DevOps, estimation, html5, human, leanstartup, learning, methodology, presentation, python | Comments Off on Most interesting links of Mars ’13
Posted by Jakub Holý on January 5, 2013
I spent one afternoon adding a much needed feature to our application. Now I have been waiting for several days for various people to review and approve it. And I have just realized how tiring it is and how much energy it takes from me.
To create something and get it out into production at once (plus minus) is fun and really motivates me to do and try stuff, it is a great feeling to see it immediately affecting the users and processes. And a quick feedback (from the users and the behavior of the application) – while still being engaged for the thing and having all the context in my mind – makes it easy and fun to fix and improve it, leading to a better result faster. Waiting, on the other hand, is exhausting and depressing. I usually start working on something else, forget the context, lose the interest (it is not easy to be fired up for two things at the same time) etc.
Therefore, if you care for happy developers and better products, make sure that your time from code to production is as short as possible. Look at your process, eliminate all delays, make it smooth.
PS: I am not saying that code reviews are bad. They are great. They just shouldn’t be a source of delay. For example if you can pair-program then you get an instant code review.
You might enjoy also other posts on effective development.
Posted in General | Tagged: agile, development, opinion | Comments Off on Fast Code To Production Cycle Matters: For Pleasure, Productivity, Profit
Posted by Jakub Holý on December 31, 2012
- Kent Beck: When Worse Is Better: Incrementally Escaping Local Maxima – Kent reintroduces his Sprinting Centipede strategy (“reduce the cost of each change as much as possible so as to enable small changes to be chained together nearly continuously” => “From the outside it is clear that big changes are happening, even though from the inside it’s clear that no individual change is large or risky.”) and advices how to deal with situations where improvements have reached a local maxima by making the design temporarily intentionally worse (f.ex. by inlining all the ugly methods or writing to both the old and the new data store); strongly recommended
- Related: Efficient Incremental Change – transmute risk into time by doing small, safe steps, then optimize your ability to make these steps quickly and thus being able to achieve large changes
- Researchers: It is not profitable to outsource development – the Scandinavian research organisation SINTEF ICT has studied the effects of outsourcing and discovered that often it is more expensive than in-country development due to hidden costs caused by worse communication and cultural differences (f.ex. Indians tend not to ask questions and work based on their, often incomplete, understanding) and very high people turn-over; even after the true cost is discovered, companies irrationally stay there. However it is possible to succeed, in some cases.
- Bjørn Borud: Tractor pulling and software engineering – very valuable and pragmatic advices on producing good software (i.e. avoiding accumulating so much crap that the software just stops progressing). Don’t think only about the happy path. Simplify. Write for other developers, i.e. avoid too “smart” solutions, test & document, dp actually think about design and its implication w.r.t performance etc. Awake the scientist in you: “Do things because you know they work, not because it happens to be the hip thing to do.”
(Note: I see the good intention behind “design for the weakest programmer you can think of” but plase don’t take it too far! Software should be primarily simple, not necessarily easy.
- Know your feedback loop – why and how to optimize it – to succeed, we need to learn faster; the only way to do that is to optimize our feedback loops, i.e. shorten the path our assumptions travel before they are (in)validated, whether about our code, business functionality, or the whole project idea. Conscise, valuable.
- Code quality is the least important reason to pair program – the author argues, based on his experience, other benefits of pair programming are more important than code quality: “[..] the most important reasons why we pair: it contributes to an amazing company culture, it’s the best way to bring new developers up to speed, and it provides a great way to share knowledge across the development team.”
- You Can’t Refactor Your Way Out of Every Problem – refactoring can’t help you if the design is fundamentally wrong, you need to rewrite it; know when it can or cannot help and act accordingly (related to how much design is needed upfront since some design decision cannot be reverted/improved upon)
- Josh Bloch: Java – the good, bad and ugly parts (video, 15 min); summary: right design decisions (VM, GC, threads, dynamic linking, OOP, static typing, exceptions, …), some bad details (signed byte, lossy long-> double, == doesn’t cal .equals, ability to call overriden methods from constructors, …); Mr. Bloch has also given a longer talk examining the evolution of Java from 1.0 to 1.7 in The Evolution of Java: Past, Present, and Future.
- True Scala complexity – a thoughtful criticism of the complexity of Scala, based on code samples; “[it is true that] Scala is a language with a smaller number of orthogonal features than found in many other languages. […] However, the problem is that each feature has substantial depth, intersecting in numerous ways that are riddled with nuances, limitations, exceptions and rules to remember. It doesn’t take long to bump into these edges, of which there are many.”; however, its possible to avoid many of the problems mentioned by resorting to less smart, more clumsy and verbose Java-like code; also, the author still likes Scala.
- Scala or Java? Exploring myths and facts (3/2012) – a balanced view of Scala’s strengths and weaknesses; “[..] the same features that makes Scala expressive can also lead to performance problems and complexity. This article details where this balance needs to be considered.” Topics: productivity, complexity, concurrency support, language extensibility, Java interoperability, quality of tooling, speed, backward compatibility. Plenty of useful links.
Big data & Cloud:
- Dean Wampler’s slides from Beyond Map Reduce – 1) Hadoop Map Reduce is the EJB 2 of big data but there are better APIs such as Cascading with Scala/Clojure wrappers; there are also “alternative” solutions like Spark and Storm; 2) functional/relational programming with simple data structures (lists, sets, maps etc.) is much more suitable for big data than OOP (for we do mostly stateless data transformations)
- Apache HBase vs Apache Cassandra – comparison sheet – if you want to decide between the two
- Optimizing MongoDB on AWS – 20 min talk about the current state of the art. Simplicity: Mongo AMIs by 10gen, Cloudformation template etc. Stability & perf.: new storage options – EBS with provisioned IOPS volumes (high I/O) + EBS Optimized Instances (dedicated throughput to EBS), High IO instances (hi1.4xlarge – SSD)); comparison of throughput (number of operations, MBs) of these storages; tips for filesystem config. Scalability: scale horizontally and vertically, shrink as needed.
- Getting Real About Distributed System Reliability by Jay Kreps, the author of the Voldemort DB: distributed software is NOT somehow innately reliable; a common mistake is to consider only probability of independent failures but failures typically are dependent (e.g. network problems affect the whole data center, not a single machine); the theoretical reliability “[..] is an upper bound on reliability but one that you could never, never approach in practice”; “For example Google has a fantastic paper that gives empirical numbers on system failures in Bigtable and GFS and reports empirical data on groups of failures that show rates several orders of magnitude higher than the independence assumption would predict. This is what one of the best system and operations teams in the world can get: your numbers may be far worse.” The new systems are far less mature (=> mor bugs, worse monitoring, less experience) and thus less reliable (it takes a decade for a FS to become mature, likely similar here). Distributed systems are of course more complex to configure and operate. “I have come around to the view that the real core difficulty of these systems is operations, not architecture or design.” Some nice examples of failures.
- Talks To Help You Become A Better Front-End Engineer In 2013 (tl;dr) – topics such as mobile web development, modern web devel. workflow, current/upcoming featrues of CSS3, ECMAScript 6, CSS preprocessors (LESS etc.), how to write maintainable JS, modular CSS, responsive design, JS debugging, offline webapps, CSS profiling and speed tracer, JS testing
- On Being A Senior Engineer – valuable insights into what makes an engineer “senior” (i.e. mature; from the field of web operations but applies to IT in general): mature engineers seek out constructive criticism of their designs, understand the non-technical areas of how they are perceived (=> assertive, nice to work with etc.), understand that not all of their projects are filled with rockstar-on-stage work, anticipate the impact of their code (on resource usage, others’ ability to understand & extend it etc.), lift the skills and expertise of those around them, make their trade-offs explicit when making decisions, do not practice “Cover Your Ass Engineering,” are able to view the project from another person’s (stakeholder’s) perspective, are aware of cognitive biases (such as the Planning Fallacy), practice egoless programming, know the importance of (sometimes irrational) feelings people have.
- Polymorphism in Clojure – Tim Ewald’s 1h live coding talk at Øredev conference introducing mechanisms for polymorphism (and Java interoperability) in Clojure and explaining well the different use cases for them. Included: why records, protocols & polymorphism with them (shapes, area => open, not explicit switch) (also good for Java interop.: interfaces), reify, multimethods.
- Stuart Sierra: Thinking in Data (1h talk) – Sierra introduces data-oriented programming, i.e. programming with generic, immutable data structures (such as maps), pure functions, and isolated side-effects. Some other points: Records are an optimization, only for perforamnce (rarely) or polymorphism (ot often); the case for composable functions; testing using simulations (generative testing) etc.; visualization of state & process
Tools & Libs
- Netflix’ Hysterix: library to make distributed systems more resilitent by preventing a single slow/failing dependency from causing resource (thread etc.) exhaustion etc. by wrapping external calls in a separate thread with clear timeouts and support for fallbacks, with good monitoring etc. Read “Problem Definition” on the page to understand the problem it tries to solve.
if you build something that is fundamentally broken it isn’t really interesting that you followed the plan or you followed some methodology — the thing you built is fundamentally broken.
– Bjørn Borud, Chief Architect at Comoyo.no, in an email 12/2012
The root of the Toyota Way is to be dissatisfied with the status quo; you have to ask constantly, “Why are we doing this?”
– Katsuaki Watanabe, Tyota President 2005 – 2009 (from the talk Deliberate Practice)
Posted in General, Languages, Tools, Top links of month | Tagged: aws, bigdata, clojure, cloud, development, java, kent beck, methodology, outsourcing, scala, scaling, webapp, xp | Comments Off on Most interesting links of December ’12
Posted by Jakub Holý on September 14, 2012
The best code I have ever written is the code I never wrote. It works exactly as intended. There are no bugs, ever. It doesn’t increase complexity of the application. Other people don’t need to struggle to understand it. It gets never outdated.
Therefore don’t write code unless you really have to.
You might enjoy also other posts on effective development.
Posted in General | Tagged: development, opinion | 2 Comments »
Posted by Jakub Holý on August 13, 2012
Real World Java EE Night Hacks – Dissecting the Business Tier, Adam Bien, 2011, ISBN 9780557078325.
I highly recommend this very thin and down-to-the-earth-practical book to everybody interested in back-end Java (a basic knowledge of servlets, Java ORM, and REST might be useful). The book evolves around a small but realistic project (X-Ray), which we follow from the inception through couple of iterations til the end. Bien shows us how lean Java EE can be, how to profit from the functionality that it offers, and how these pieces of functionality fit together to deliver something useful. He actually introduces a complete Java EE development environment including continuous integration (CI), code quality analysis, functional and stress testing.
Some of the things that I appreciate most in the book is that we can follow author’s decision process with respect to implementation options (such as SOAP vs. REST vs. Hessian etc., a REST library vs. sockets, or when (not) to use asynchronous processing) and that we can see the evolution of the design from an initial version that failed through cycles of growing and refactoring and eventually introducing new technologies and patterns (events, configuration over convention) to support new and increased requirements. Read the rest of this entry »
Posted in General, j2ee, Languages | Tagged: book, development, java, javaEE, review | Comments Off on Recommended Book: Real World Java EE Night Hacks by Adam Bien
Posted by Jakub Holý on January 9, 2012
I’m taking part in a course of Specification by Example, lead by Gojko Adzic. Here I want to summarize the key things I’ve learned in the first day of this entertaining and fruitful course thanks to both Gojko and my co-participants.
If you haven’t heard about Specification by Example (SbE) before (really?!), then you need know that its main concern is ensuring that you build the right thing (complimentary to building the thing right), which is achieved by specifying functionality collaboratively with business users, testers, and developers, clarifying and nailing them with key examples, and finally, where it is worth the effort, automating checks of those examples to get not only automated acceptance tests but, more importantly, a “living documentation” of what the system does that never gets out of date. Best to read the key ideas described by Gojko himself or the SbE Wikipedia page. Read the rest of this entry »
Posted in General, Testing | Tagged: development, methodology, SbE | Comments Off on Key Lessons from the Specification by Example Course, Day 1
Posted by Jakub Holý on November 22, 2011
This post is inspired by Kent Beck’s excellent talk at JavaZone 2011 titled Software G Forces: The Effects of Acceleration where he describes how the development process, practices and partly the whole organization change and/or have to change as you go from annual to monthly to weekly, daily, hourly deployments. I’d like to summarize some of the points he made and use that as a ground for arguing that more frequent deployments are (in general) better.
I’d highly recommend you to watch his presentation as I will only reproduce parts of it (and as they are out of their original context, they might well not represent exactly what Kent wanted to communicate).
Kent argues that as you deploy more and more frequently, many things have to change including the business side of the software. What is a best practice for one of these speeds becomes an impediment for another one. With more frequent deployments teams have to progress towards the following practices, while leaving some other practices behind:
Read the rest of this entry »
Posted in General | Tagged: agile, continuous_deployment, development, DevOps, opinion | 2 Comments »