Archive for the ‘General’ Category
Posted by Jakub Holý on November 30, 2013
Some interesting topics this time despite me spending lot of time on the Principles of Reactive Programming course: Java x Node.js, REST x other future-proof architectures, scary legacy code. Of course, also plenty of Clojure.
People, organizations, teams, development:
- Chris Argyris (1923-2013): An Appreciation – Thinkers 50 – recently departed Ch. Argyris is a person whose work you should know, if a bit interested in learning and organizations and how they (dis)function; and since we all work in organizations and want our work to be pleasant, this means all of us. We all want to work in orgs that do double-loop learning, i.e. they actually evovle as they learn. “Argyris argued that organizations depend fundamentally on people and that personal development is and can be related to work.” Now stop and go read it!
- Bob Marshall: The Antimatter Principle – “the only principle we need for becoming wildly effective at collaborative knowledge work” – can be summarized as “attend to folks’ needs” (importantly, including your own) => find out what people actually need, interpret their behavior (including anger, seemingly irrational or stupid requests etc.) in terms of needs; mastering this will make you excell in communication and effective work. Read the post to find out more.
- It’s a state of mind: some practical indicators on doing agile vs. being agile – are you agile or are just “doing agile”? Read on ti find out, if you dare! F.ex. “Non Agile teams have a process that slows the review of the changes.” Cocnlusion: “An Agile mindset is just that – a state of mind, a set of values. Its a constant questioning, and an opening up to possibilities. Its a predisposition to produce great things.“
- Johannes Brodwall: Humble architects – how to avoid being an architect that does more harm than good, as so many out there? Some tips: Don’t assume stupidity, Be aware that you might be wrong, Be careful with technology (i.e. simplicity beats everything; applies so much to us developers too!), Consistency isn’t as important as you think (or beware context), Tactical reuse in across systems is suboptimization (i.e. reuse has a cost), separate between (coding) rules and dogma (i.e. is that way unsafe, incomprehensible, or just a heresy w.r.t. a dogma?) Very valuable insights into creating good technical solutions and teams that work.
- Liz Keogh’s The Dream Team Nightmare: a Review – a very good review of this adventure-style book about coaching “agile” teams through (around?) common pitfalls, provides a good base for deciding whether you shall read the book (Liz essentially says yes)
- Fibonacci Kittens: One Idea One Commit – a short story of coming from biannual releases to frequent release of individual features; I link to this primarily to spread optimism, if this company managed it then, perhaps, we other can too?
- The Eternal Struggle Between Business and Programmers – “Business: More features! Now! Programmers: More refactoring! Now!” How can we resolve this eternal conflict of needs? This post reveals how the two parties can find a common ground and mutual understanding (beware, everybody must give up on something) and thus work together rather than against each other.
Coding, architecture, legacy
- Why the future is NOT RESTful – always refreshing to read something against the mainstream; “REST is not fit for the next generation of smart client applications because it has not been designed for smart clients.” According to the author, a smart client app stack needs: “1. persistence (storage and query), 2. documents/orm (conversion to tree-like datastructures), 3. data authorization (once authenticated), 4. pub/sub (websocket communications), 5. client db (client-side caching and querying), 6. templating (presentation level)” Meteor.js has nearly all but #3 thanks to mongodb (1+2), dpp (4), mongo on the client (5), spark (6). The author considers a similar but Clojure-based stack (with Datomic, Angular etc.) and looks at authorization possibilities. “Secured, personalised, CRUD operations are the future to a more simplified web.” We may agree or not but it certainly is worth reading.
- Michael Feathers (of Working Effectively With Legacy fame): Unconditional Programming – “Over and over again, I find that better code has fewer if-statements, fewer switches, and fewer loops. Often this happens because developers are using languages with better abstractions. [..] The problem with control structures is that they often make it easy to modify code in bad ways.” Includes a nice example of replacing if-else with much more readable and safer code.
- The Quality of Embedded Software, or the Mess Has Happened – an interesting and scary read about terrible spaghetti code (and hardware) that makes some Toyotas to accelerate when the driver tries to break; 11,000 global variables, the key function showing so high cyclomatix complexity that “makes it impossible not only to test but even maintain this program in any way.” Then 80k violations of the car industry coding standard cannot surprise. And a safety control that does not work. Interesting that a great manufacturer may have so terrible IT (and Toyota isn’t the only one).
- The string type is broken – the String type is M. Feathers’ favourite example of a leaky abstraction – most languages fail to process/split/count less common Unicode characters properly, the fact that String is implemented as a series of bytes leaks through (UTF-16 langs like Java); worth reading to be aware of the limitations
- Why I’m Productive In Clojure – some interesting points about simplicity, consciousness, interactive development, power without overwhelming fatures, etc. “With it [Clojure] I can always easily derive a solution to a particular problem from a small set of general patterns. [..] However, the number of ways that these concepts can be combined to solve all manner of problems appears to be inexhaustible.“
- Node.js at PayPal – PayPal is switching from Java to Node.js (among others to promote language consistency) and, as a part of that, has implemented the same app in Node and Java; results: Node was done earlier, had less code, performed better (though, as Daniel Kvasnička pointed out, “Comparing Node.js and servlet-based archs is not fair… compare Node with
@vertx_project and you’ll get a whole different story ;)”)
- IBM: Developing mobile apps with Node.js and MongoDB, Part 1: A team’s methods and results – also IBM has implemented the same (REST) app once with Java and DB2, once with Node and Mongo where Node+Mongo required less work and performed better; one of the great wins was having JSON as a native structure everywhere instead of transforming from/to it so Mongo is, in my opinion, an important factor in this particular case
- Dynamics of Programming: Benefits of Scala in CS1 – reasons for and experiences with using Scala in an introductory computer science course, worth reading; some of the advantages over Java are consistency (.asInt on String and Double vs. casting/parsing, no “primitive” types), REPL with time inference good for learning, functional style enables focus on what rather than how; quite persuasive arguments
- Stuart Sierra’s Component – a library for making it easier to implement Stuart’s reloadable workflow; a component is something that can be started, stopped, and depend on other components so that it is easier to do interactive REPL development
- Logan Linn: Clojure/conj 2013 – a pretty good overview of the conference
- Caribou – “the kernel of usefulness that has emerged from years of this basic practice“- a new Clojure web framework – seems to be interesting
- Results of the 2013 State of Clojure & ClojureScript survey and drill-down into what features people want – the most interesting fact is how many more participants use Clojure in production than last year and perhaps also the relatively wide adoption of Datomic among the respondents. Light Table has become the 3rd most popular dev. env., after Emacs and Vim. Some of the most mentioned language features requested were types (=> core.typed, Prismatic’s Schema), better error reporting (=> slingshot, dire, clj-stacktrace, io.aviso:pretty, etc.), debuger (though progress is being made)
- Book: Clojure High Performance Programming
- Improving Clojure Feedback : Stack Traces – making Clojure stacktraces more usable by filtering out noise and linking to relevant content – io.aviso:pretty, io.aviso:twixt
- Clojure Dev discussion: Hashing strategies – Executive summary – “In Clojure, however, it is far more common than in Java to use longs, vectors, sets, maps and compound objects comprised of those components (e.g., a map from vectors of longs to sets) as keys in other hash maps. It appears that Java’s hash strategy is not well-tuned for this kind of usage. Clojure’s hashing for longs, vectors, sets, and maps each suffer from some weaknesses that can multiply together to create a crippling number of collisions.” Ex.: An implementation of N-Queens took forever, spending most time on hash collisions. But be calm, smart Clojurians are working on a solution.
- Datomic Pro Starter Edition – Datomic with all storages, Datomic Console, a year of updates, full Datomic programming model; limitations: no HA transactor, no integrated memcached, max 2 peers and 1 transactor
- AirPair.com – a new site that enables developers to get help from other devs via remote pairing, code review, code mentoring etc. – a good opportunity to get help / help others (and earn something); I haven’t tried it but it sounds pretty interesting
MongoDB web stacks
- Meteor: JS frontend + MongoDB backend with changes in the DB pushed live to the clients, i.e. MongoDB is used both as the “application server” and storage. It seems great for apps where users need to collaborate in real-time with each other, certainly great for quick proof of concepts etc.; worth checking out; it also comes with free (at least for start?) hosting so really good for prototyping – “an open-source platform for building top-quality web apps in a fraction of the time.” The intro screencast will give you a good overview (10 min).
- Mean.io – MEAN (Mongo, Express, Angular, Node) stack Boilerplate – frontend, backend and storage using the same language and some of the most popular technologies (not that popular = best fit for you :)); it seems to be very new but since it just glues together 4 popular and documented technologies, that should not be an obstacle. There is an intro on the MongoDB blog.
- BusyBox – reportedly a better POSIX env for Windows than gow, Cygwin, et al.
[..] no organization should exist unless it is “of service” to its employees, its customers, its community.
- @Tom_Peters 28/11
I hope you’ll agree that there is a certain amount of irony involved in having to write repetitive code
- Dmitri Sotnikov in Why I’m Productive In Clojure
Happy teams are productive teams but:
Morale is 95% a function of the prevailing system (the way the work works). Which in turn is a function of the prevailing collective mindset
- @flowchainsensei Nov 10th
Posted in General, Languages, Top links of month | Tagged: agile, book, clojure, continuous_deployment, diy, legacy, mongodb, nodejs, SbE, scala, security | Leave a Comment »
Posted by Jakub Holý on October 31, 2013
- Google engineers insist 20% time is not dead—it’s just turned into 120% time – it is interesting to see how has this evolved; “I have done many engineering/coding 20% projects and other non-engineering projects, with probably 20-40% producing “real” results (which over 7 years I think has been more than worth it for the company). But these projects are generally not rewarded.” [highlight mine]
- The Worst Daily Scrum Ever – a story whose bad part is a too common reality; if energy is low, nobody asks for / offers help, and people only report status / plans then you are doing the daily scrum wrong and should stop now (but it also documents a nice example of a good, effective scrum)
- Why Responsive Design is a Waste of Time – a refreshingly critical take on responsive design; the author now aknowledges that it is sometimes worth the pain but the points are still valid – responsive design requires (lot of) extra work, the attempt to create a one-size-fits-all site of course adds considerable complexity (having two separate simple frontends might be better than one that is too complex), also many sites are good enough as they are (especially taking into account the capabilities of mobile browsers)
- How to lose $172,222 a second for 45 minutes – i.e. your bugs are likely not so serious after all A financial company screwed big and ended up bankrupt. The cause? Chaotic DevOps, not removing old unused code, reusing a feature flag instead of creating a new one, lack of monitoring. The story in short: They deployed new trading code but failed to notice that it has not been deployed to one of the 8 servers; due to the flag reuse, the old, 10 years unused code has been activated instead. Due to the lack of monitoring they did not notice the cause, tried to roll back while leaving the flag enabled thus effectively activating the bad code on all the servers. => have proper automated and self-checking deployments, delete old code, do not repurpose old switches.
- 40 Inappropriate Actions to Take Against an Unlocked (Windows) PC – good tips for promoting security and having fun at the same time; I shall keep this at hand
- How to go about ‘proving’ why dynamically typed languages are better – a cultivated and interesting discussion; as argueed, thinking in this direction is itself wrong and in different contexts, different languages will be more appropriate. I also like Phil Lord’s “Programming is a highly fashion-centric occupation for any number of reasons.” and “For me, the main advantage is that you are not forced to build complex hierarchies just to support the type system ([..]), and that having only a few abstractions makes it worthwhile adding lots of functions operating over them.” and L. Petit’s “IMHO, the question is irrelevant. It implicitly assumes that statically typed vs dynamically typed is a black / white choice, and that either ‘static wins over dynamic’ or ‘dynamic wins over static’ will be a true statement whatever the context.” Also a good observation that types are only a subset of function contract enforcement and one of possible implementations.
- The Failure of Governmental IT (Learnings From HealthCare.gov) – links to a few really good articles about the problems with governmental IT in general and my summary of them
- Inside the Arctic Circle, Where Your Facebook Data Lives – the designs of data centers used to be proprietary secrets until Fb developed its own and open-sourced them, enabling many Asian manufactures to start creating cheaper datacenters and thus started a revolution in this domain. Facebook’s data centers are not general purpose but suitable ot the kind of work they need, but it is still widely applicable. Cool to see how they use natural conditions to get energy needs down and make HW that fits best their needs – that is what I call innovation!
- Academia.edu (via @RiczWest) – a rich source of free research papers – just register as an independant researcher; also lean/agile/systems thinking and other interesting topics
+ Username +
- Johannes Brodwall: A canonical web test – a simple web app end-to-end smoke test – using an embedded Jetty, a test DB (preferably in-memory), WebDriver to test it (simple: browser.get(“/people”), assertThat(browser.findElement(<person id>.contains(<person’s name>)); simple, nice, useful
- LearnGitBranching – an online game to learn branching & rebase in git; use the menu in the lower-right corner to navigate between the levels etc. You can also execute commands “show goal”, “hint”, “level” to navigate around; pretty cool and great for learning the command line commands
Society & people
Not a typical topic I share here but really worth it this time.
- The ocean is broken – a saddening story worth reading to learn what does your tuna sandwitch cost and where does all the plastic we use end up. From a sailing trip from Melbourne to US where there were plenty of fish (and birds) 10 years ago – and 2 this year, killed to a noticable degree by huge fishing ships that catch tuna – and kill and throw away all the other “junk” fish. Nowadays fish are replaced by plastic and other waste that actually prevents usage of the engine unless somebody can watch for dangerous nets and ropes leftovers. Earth, where are you falling to?
- The Guardian: Why have young people in Japan stopped having sex? – sad and interesting to observe what happens when the system is set up so that people “can’t be bothered” to have inter-sexual relationships, partnership, and children. Japan needs a good deal of systems thinking to fix its broken society where women do not want children because it would cost them their career and neither men nor women are willing to subjects themselves to the social pressure and demands associated with relationships.
- The Guardian: 29 million people enslaved, says first global index on slavery – welcome to the 21st century! The leading slave countries are India (14M), China (3M), Pakistan (2M). Also, slaves are building the world cup stadion in Qatar.
- They’re Taking Over! – how we managed to destroy sea ecosystems and helped the now unstoppable return of jellyfish – Jellyfish are evidently very veried and extemely resilient and have been hold at bay only by rather complex ecosystems that we managed to destabilize so much that Jellyfish are on their way back to ruling all the sees again (destroying the rests of the ecosystems – i.e. fish – on the way); a sad future for the sea, Earht, and us
- LightTable 0.5.9 got elementary paredit commands
- Amazonica: A comprehensive Clojure client for the entire Amazon AWS api – best with a REPL!
- Clojure Understood: the Rush Hour Platform – application of the design best practices promoted in the Clojure community (separation of concerns, simplicity, …) on a project – “highly accurate vehicle traffic simulations”; I have only started reading it but it looks highly interesting
- Clojure content at InfoQ – articles, news, interviews, presentations etc.
- A bitter taste [of EuroClojure] – we as a community must embrace diversity and stop fostering our egos by mocking other than our holy editor (and, I would add, by mocking other languages and in general mocking whatever); respect and open minds, please!
- Garden – Clojure alternative to scss/less – still needs time to mature and gain tooling support but it is cool that it exists
- C. Emerick’s Austin, the ClojureScript REPL over nREPL – 1) Start nREPL (via lein repl, from your editor…), 2.a) Execute austin’s exec to start a ClojureScript REPL in it, backed by headless PhantomJS or a real browser – or, alternatively, 2.b) create a C.S. REPL connected to your webapp, as described in the browser-connected-repl-sample. You should watch this 8 min demo.
- WhiteHat Aviator – A Safer Web Browser – WhiteHat, a well-known security company, has released a browser that aims at improving privacy by preventing user tracking (f.ex. but not sending referral URL) and blocking ads even at the cost of occassional slight discomfort, i.e. something that the mainstream browsers are not interested in. So far for OS X only.
- EnvPane – a preference pane for environment variables for Mac OS X 10.8 (Mountain Lion) – set env. vars for GUI/terminal apps, no need to log out upon change
Weinberg: Bureaucracy is what we do when we no longer remember why we are doing it
- via Ben Simo, no source specified so it may be fake but anyway it is valid
Posted in General, Languages, Testing, Top links of month | Tagged: aws, clojure, ClojureScript, design, earth, ecology, Git, innovation, privacy, science, scrum, security | 4 Comments »
Posted by Jakub Holý on October 28, 2013
The failure of Healthcare.Gov has been discussed a lot but the main causes of the failure are unrelated to the project or to technology and apply similarly to other governments / large projects.
According to some reasonable articles, the main problems were:
- Byzantine procurement process – those with the best lawyers and experience, not the most capable ones get the job
- Size – the project is “too important to fail” – and thus also too big to succeed; +- 55 contractors creating SW with fixed date, integrating hundreds of insurence providers and some 36 states and various agencies; big-bang style of development and deployment
- Fragmented responsibility and lack of ability – no one both with enough knowledge and power responsible for the whole project (and lack of the best talent in government IT in general), responsibility spread across tens of contractors and officials likely driven by cover-my-ass motivation (e.g. the procurement officer interested in selecting the cheapest offer that checks all the checkboxes instead of the best one – because who can fire her/him for doing that?)
- Niagara Falls of waterfall development, constrained by rules and bureaucracy to immobility – extensive legislation, rules, and security requirements together with a fear/blame-driven organization or not good for agile approaches
BTW, according to L. Hart (below), 0 federal projects over $5M were delivered on time, only 6.4% of those over $10M have succeeded and full 40% of such projects were canceled. So, under those circumstances, Healthcare.Gov is actually a small miracle.
By the way, the Government Digital Service team in the UK has become “recently” famous for bringing effective IT to the government. It is interesting to read about the UK DS strategy, based on delivery – frequent, iterative, repeatedly successfull.
@flowchainsensei and @timoreilly for the links.
Posted in General | Tagged: agile, government, management | Comments Off
Posted by Jakub Holý on September 30, 2013
- Stuff The Internet Says On Scalability For September 13, 2013 – a collection of interesting performance related articles with summaries (via @_dagi)
- Can you copy a culture? The NUMMI story (audio/transcript) – how the GM factory with the worst workforce has been turned around via a good application of Toyota Production System – “a truly inspiring story of human potential and how systems can be designed to bring the best or worst of of people.” And how GM failed to learn from it and to copy Toyota’s culture.
- The Reactive Manifesto – why to write reactive SW – “Reactive applications represent a balanced approach to addressing a wide range of contemporary challenges in software development. Building on an event-driven, message-based foundation, they provide the tools needed to ensure scalability and resilience. On top of this they support rich, responsive user interactions. We expect that a rapidly increasing number of systems will follow this blueprint in the years ahead.“
- NYT: Eiji Toyoda, Promoter of the Toyota Way and Engineer of Its Growth, Dies at 100 – learn about the life of one of the founders of lean thinking
- Gojko Adzic: How we solved our #1 product management problem – valuable experience of false assumptions, learning from users, and a much helpful UI remake: even if you build a product to scratch your itch, you have to test it with real users
- Don’t use Hadoop – your data isn’t that big – a great post about the downside of Hadoop and that there are much better options (large disks, large RAM, Pandas/R/Postgres) for data up to few TBs. “In addition to being more difficult to code for, Hadoop will also nearly always be slower than the simpler alternatives.”
- Gartner On Big Data: Everyone’s Doing It, No One Knows Why – golf talk / hype -driven initiatives FTW! “According to a recent Gartner report, 64% of enterprises surveyed indicate that they’re deploying or planning Big Data projects. Yet even more acknowledge that they still don’t know what to do with Big Data.”
- What makes Spark exciting – why it might be a good replacement for Hive/Hadoop, based on experiences with H/H: “Hive has served us well for quite a while now. [...] That said, it has gotten to the point where Hive is more frequently invoked in negative contexts (“damn it, Hive”) than positive. (Primarily due to being hard to test, hard to extend.)” “We had heard about Spark, but did not start trying it until being so impressed by the Spark presentation at AWS re:Invent [..] that we wanted to learn more. [..] Spark, either intentionally or serendipitously, addresses both of Hive’s primary shortcomings, and turns them into huge strengths. (Easy to test, extend.) [..] I find the codebase small and very easy to read, [..] –which is a nice consolation compared to the daunting codebases of Hadoop and Hive.” Cons.: Spark is only pre-1.0, the author hasn’t yet tested it heavily.
- 10 Ways to Make Your Office Fun To Work In – because we spend there plenty of our time so why not have a pleasant/cosy, inspiring environment? Some tips: plants, not-your-boring-enteprprise-look-and-feel, open it to the nature (I want this!), design it as home, not office, provide play space (I am too into work to want to play but having a resting place for a nap is st. I’d love).
- Book: The Architecture of Open Source Applications (via @rmz) – learn by studying architectures of existing systems – “In these two books, the authors of four dozen open source applications explain how their software is structured, and why. What are each program’s major components? How do they interact? And what did their builders learn during their development?“
- Book: Seven Concurrency Models in Seven Weeks: When Threads Unravel – “how to exploit different parallel architectures to improve your code’s performance, scalability, and resilience” – threads & locks, actors, FP + immutability/futures/promisses, Software Transactional Memory etc., GPU, MapReduce on clusters, … (intro) Personally, I would prefer from theory to practice approach and mention of CSP (-> Go’s channels, core.async) and more.
- Books Amazon CEO requires his top execs to read (The Effective Executive by Peter Drucker, The Innovator’s Solution by Clayton Christensen, The Goal by Eliyahu Goldratt)
- Stanford engineers build computer using carbon nanotube technology (via
- NYT: The Banality of Systemic Evil – a good article about human tendency to “obey the system” thus potentially causing evil – and thus the need to resist the system, as heroic individuals such as Snowden, Hammond, Schwartz, Manning. See the famous Eichmann in Jerusalem for how “doing your job” can create evil – “[..] what happens when people play their “proper” roles within a system, following prescribed conduct with respect to that system, while remaining blind to the moral consequences of what the system was doing — or at least compartmentalizing and ignoring those consequences.” (Tip: The book Moral Mazes explores the ethics of decision making within several corporate bureaucracies => mid-managers rules of life: (1) never go around your boss, (2) tell the boss what she wants to hear, (3) drop what she wants dropped, (4) anticipate what the boss wants so that she doesn’t need to act as a boss to get it, (5) do not report something the boss does not want reported, cover it up; the the job & keep your mouth shut.) “The bureaucracy was telling him [Snowden] to shut up and move on (in accord with the five rules in “Moral Mazes”), but Snowden felt that doing so was morally wrong.” “[..] there can be no expectation that the system will act morally of its own accord. Systems are optimized for their own survival and preventing the system from doing evil may well require breaking with organizational niceties, protocols or laws.“
- Fairphone – “A seriously cool smartphone that puts social values first” (likely the only one not built by poorly paid workers and creating too much ecological burden), for just €325. You can see detailed cost breakdown, list of suppliers, specs, and essentially everything. This is, in my opinion, super cool! Go and read the story!
- Amazonica – “A comprehensive Clojure client for the entire Amazon AWS api.”
- Talk Ritz, The Missing Clojure Tooling (40min, 9/2013) – thanks to this I finally understood how to use Ritz but it still seems not to work well, f.ex. setting a breakpoint always reported “Set 0 breakpoints” (lein ritz/middleware 0.7.0, nrepl-ritz.el 0.7.1); according to callen, debug-repl is simpler and nicer if you only care about local vars and evaluation. To try ritz: use M-x nrepl-ritz-jack-in, then M-x nrepl-ritz-break-on-exception, exec. f.ex. “(/ 1 0)”. In the poped-up buffer, t or enter to show frame locals, e to eval a code in the context of the frame etc. If you managed to trigger the debug buffer through a breakpoint, the actions lists would contain STEP etc. (See fun. nrepl-ritz-line-breakpoint)
- C. Grand’s spreadmap – “library to turn Excel spreadsheets in persistent reactive associative structures” => access content via map functions; changing a value updates formula cells using it
- Alembic Reloads your Leiningen project.clj Dependencies – add a dependency to your project.clj w/o needing to restart your REPL (just call (alembic.still/load-project), provided you have it in your lein dependencies). Limitations: cannot remove deps or change versions.
- Defeating stack overflows – techniques for transforming mutually recursive calls etc. into something that won’t blow the stack – “Priming the pump” (memoize subresults first), core.async
- Google Groups: Clean Architecture for Functional Programming – How do the Clean Architecture and the Clean Code best practices apply to FP (Clojure/Haskell)? Some points: OOP isn’t worse than FP, only people do class-oriented programming instead; OO better e.g. for UIs, combining them (func. core, imperative shell) can be sometimes best. Some clean arch. patterns are actually more like functions – “Interactors and Presenters, for example, do not maintain any state of their own. Even those objects that do imply some kind of state, such as entities and gateways, keep that state hidden behind boundaries and present a functional interface instead.“
- night-vision: Handy, super light weight debugging utility – add it to your lein profile and then call
(night-vision.goggles/introspect-ns! '<name of ns>) and it will print each entry/exit of a function within the scope of the namespace with the argument/return values
- Nil Punning (Or Null Pointers Considered Not So Bad) – a great post about why nil in Clojure is not bad contrary to Java’s null (because it is actually an object, you can call functions on it, treat it as false/empty list/map/set, most core functions work on it)
Posted by Jakub Holý on September 5, 2013
Use case: Deploy a new version of a webapp so that all new users are sent to the new
version while users with open sessions continue using the previous version
(so that they don’t loose their precious session state). Users of the new version
can explicitely ask for the previous version in the case that it doesn’t work as expected and vice versa.
Benefits: Get new features to users that need them as soon as possible without affecting
anybody negatively and without risking that a defect will prevent users from achieving their goal
(thanks to being able to fall back to the previous version).
Read the rest of this entry »
Posted in General | Tagged: continuous_deployment, DevOps, java | Comments Off
Posted by Jakub Holý on August 31, 2013
Sorry folks, this month it will be very brief. I have many more great stuff in the queue but haven’t managed to write it down yet. Next month will be heavy
- Why Software Projects are Terrible and How Not To Fix Them – many teams are not ready to embrace new/better software practices, primarly for two reasons: 1) most of them are nonintuitive (f.ex. adding more people will slow dev down) and need to be sold through a high hierarchy of managament – but people/managers/organizations don’t really care, it takes years for good/bad practices to have an impact, which is not relevant “now.” 2) Businss objectives change too quickly and SW is blamed for not delivering. Based on evaluating many failed projects. Conclusion: Choose carefully people/organizations your work with. Avoid blame-driven ones. Quote on middle managers: “He has to put more developers on the project, call a meeting and yell at people, and other arbitrary bad ideas. Not because he thinks those will solve the problem. In fact, managers often do this in spite of the fact that they know it’s bad. Because that’s what will convince upper management that they’re doing their best.” “In the vast majority of failed projects I’ve been called to looked at, the managers have not read one book on software engineering.“
Data & Analytics
- Big Data: Kafka for uSwitch’s Event Pipeline – a better alternative to log files – use LinkedIn’s Kafka for messaging, have MR jobs to import latest messages into Hadoop/HDFS. The advantage of Kafka is that it persists the messages for a period of time so it is easy to batch-import and even re-import them. The uSwitch’s talk Users As Data explains the downsides of log files. LinkedIn’s Camus is a tool for importing messages from Kafka to HDFS.
- Realtime Analytics with Storm and Hadoop (at Twitter; presentation deck) – pre-aggregate some data into a read-only, random read DB such as ElephantDB, Voldemort, or Manhattan. For newer data use Storm and aggregated data in a read-write, big-data DB such as HBase, Riak, or Cassandra. For stuff that cannot be pre-aggregated you might use Storm’s Distributed RPC.
- The Unified Logging Infrastructure for Data Analytics at Twitter – a paper from late 2012 that presents “Twitter’s production logging infrastructure and its evolution from application-speciﬁc logging to a uni- ﬁed “client events” log format, where messages are captured in common, well-formatted, ﬂexible Thrift messages” – with the benefit of “s streamlined log collection and data analysis”.
- Development and Deployment at Facebook (Kent Beck et. al., 8/2013, 13p paper) – “More than one billion users log in to Facebook at least once a month to connect and share content with each other. Among other activities, these users upload over 2.5 billion content items every day. In this article we describe the development and deployment of the software that supports all this activity, focusing on the site’s primary codebase for the Web front-end.“
- One of the most valuable talks I’ve seen, in just 18 min: The Progress Principle – about the disengagement crisis and motivation at work by Teresa Amabile at TEDx Atlanta (via @thovden). Disengagement from work is increasing, at all age and salary levels, and leads to unhappy people, low productivity, huge financial losses. Based on analysing diaries of 12k participants, the single most important engaging and motivating factor is making progress in a meaningful work (including small wins). A culture of management by fear and punishment for failure creates disengagement and can crush even an innovative, profitable, praised company in a few years. Everybody, though especially the management, creates the culture through their everyday, small actions. If everybody focuses on catalysing progress and supporting their fellow humans through good and bad times, engagement and success will follow. Remove progress inhibitors, nourish the human spirit (acknowledge what we humans value, encourage people). Yet of the managers asked, very few knew of the significance of making progress (or, I can assume, of supporting people and making them happy(er) and the impact of our inner work life (perceptions, emotions, etc.) on our productivity and creativity). The study included two seemingly similar, successfull companies, one with great engagement, another with a new management that managed to destroy the engagement and thus eventually the company. Actions to take: catalyse progress, celebrate wins, encourage and support your colleagues.
- Wonderful Clojure Cheatsheet 1.5 with tooltips showing the doc and summary of information available at clojuredocs.org (other Clj versions), by Andy Fingerhut
- Chas Emerick’s Clojure type selection flowchart to help you decide whether to use a map, a record, reify, proxy, gen-class, or deftype. (Reify and proxy don’t produce a class but just an instance of an anonymous class; proxy can extend a base class, reify cannot. gen-class produces a class visible from Java and can extend Java classes. …)
- Docker.io – pack, ship and run any application (and its dependencies) as a lightweight container, i.e. essentially “a VM without the overhead of a VM,” using linux containers (chroot on steroids with resource limits via control groups) see reports of some uses such as Java app deployment, desktop virtualization, automatic app deployment in GitHub commit. Docker also supports evolving the containers over time, i.e. deploying new version, by pushing just diffs so it’s low-overhead. You can build a container (include files, SW, forward ports, …) using a Dockerfile. See dotScale 2013 – Solomon Hykes – Why we built Docker for an intro (20 min).
- Packer.io – tool for building pre-configured VM images for different platforms (EC2, VirtualBox, …), remotely similar to Netflix’s Aminator. See Immutable Servers With Packer and Puppet for an example use case.
- Ubuntu-build Vagrant boxes at cloud-images.ubuntu.com/vagrant/
- SlimerJS – PhantomJS-compatible headless browser engine based on Firefox/Gecko (well, it is not fully headless yet but that is planned; the main focus now is full compatibility with PhantomJS’ API) (Both work with CasperJS for navigational steps/testing.)
- localtunnel – instantly show locally running webapp/server to the rest of the world (gem install localtunnel, localtunnel <port to share>, => share the url returned, e.g. http://xyz.localtunnel.com) – I haven’t tried it but it looks simple and very convenient
- Logstash + Kibana (via @mortenberg): take control of your logs – while Logstash can collect (from multiple servers/services), parse (over 100 built-in patterns), store, index, search your logs, Kibana is a web interface to seach them, view them in realtime (based on a query) etc. See this Logstash slides (9/2012) and an overview of Kibana’s powers. PS: Logstash can also compute metrics and send them to graphite etc. It is typically used with ElasticSearch.
- ncdu is an interactive, command-like disk usage browser that shows a list of directories sorted by size shown in human-friendly units, you can navigate with arrows and enter and i to show the current dir/file info, d to delete it, q to quit; check out this article about ncdu with screenshots and ncdu man page. Install via Apt etc., run f.ex. with ncdu -x / .
- vagrant-cachier – Vagrant plugin for caching apt/yum/.. packages locally, thus speeding up destroy+up
Posted by Jakub Holý on July 31, 2013
This month focuses on languages and approaches (reactive programming, F#, Erlang, FP talks etc.), agile (need for speed, recommended books), Clojure/Linux/cloud tools and libs.
- The Need For Speed – the top 10 reasons for fast development flow (with time to market being one of the less important) – more learning, focus on the MVP, focus on the puprose/goal, happier customers/leadership, better quality (sic!), higher morale (I concur!), push for cotninuous improvement, “one of the only sustainable differentiators”; => “sense of urgency and motivation”; “[..] I continue to meet people and teams that not only move very slow, they don’t understand the relationship between speed and innovation, or speed and quality.”
- agile42 Summer Reading List 2013 – books recommanded by experienced people/agile experts – lot of interesting stuff! Topics: Communication and Coaching (f.ex. Practicing Nonviolent Communication), Business (How Great Companies Get Their Mojo from Maslow ,..), Learning From the Military, Agile and Technology (e.g. The Art of Action: How Leaders Close the Gaps between Plans, Actions and Results), Agile and Technology (f.ex. The people’s Scrum)
- Dan North: Are we nearly there yet? – optimize for time to business impact; SW dev as mountaineering (impossible to estimate correctly, many unknown details, dead ends, …); go fast – but sustainable; the tyranny of backglog (there are multiple paths to the top yet backlog defines only one; have you ever considerably changed it?) “Instead we could embrace the fact that today we always know more than we did yesterday, and that tomorrow we will know even more. We can take a fresh look up the mountain every time we pause to regroup, to plan.” => we ask 1) what gives us the shortest lead time to business impact? 2) what can help us to learn/invalidate more? 3) how to assure our stakeholders we are approaching the goal?
- Joel on Software: Software Inventory (7/2012) – a classical article about the evilness of software inventory (backlogs, issue trackers, undeployed features, …) ‘When I hear about product teams that regularly have “backlog grooming” sessions, in which they carefully waste a tiny amount of time and mental energy every day or every week thinking about every single feature which will never be implemented, I want to poke my eyes out.‘
- Job satisfaction self-test: Twelve questions that define a great place to work – check yourself how satisfied you are with your job (example questions: How well do I know what is expected of me? How often in the past seven days have I received recognition or feedback on my work? How much does the mission/purpose of the company make me feel like my work is important?)
- Coaching Anti-Patterns: Prescriptive Agile – a prescriptive coach “knows” what is “right” and forces it onto the client, without listening to her; instead, we should “Meet them where they are and leave them in a better place” => “[..] my first responsibility is to understand how and why they came to this practice. How did they come to this decision? What challenges does this approach address? What benefits are they optimizing for?” Worth remembering AND practicing
Languages, paradigms, approaches
- The Trouble with Erlang (or Erlang is a ghetto) - an objective criticism of Erlang by somebody who seems to be quite experienced with it; as I know very little about Erlang, it was interesting to learn about its weaknesses (no map/dict data structure, slow memory management, poor “JIT,” not usable for shared-state concurrency (contrary to e.g. Clojure), immutable state is not necessary and makes some things bad, inconsistent and ugly standard lib, …)
- Adventures in Multi Paradigm Programming – different programming paradigms/approaches re-implemented in Emacs Lisp – interesting 1) to see and compare these different approaches and 2) the flexibility of Lisp. Including iteration – Ruby’s map, Python’s list comprehension ([an_expression for x in list]), Scala’s default argument (_); search – Java’s for; arguments: direct, variadic (i.e. any number of args), named args; destructuring and pattern matching in CoffeScript/OCaml style; Haskell-like monads; objects with mixins; namespaces.
- Why bugs don’t like F# – no nulls, immutable data, strong type system, composition of small functions, asynchronous programming abstractions, higher-order functions over collections (no off-by-one), units of measure
- IBM high-fives Netflix open-source tools – it is interesting to see the spreading of Netflix’s open source tools for better cloud infrastructures; f.ex. “Karyon, is what Netflix calls the base container for applications and services built using the NetflixOSS ecosystem; Eureka is mid-tier load balancing; Hystrix controls interactions between myriad distributed services to nip cascading failures in the bud; and Ribbon is a Remote Procedure Call library.”
- ZeroMQ instead of HTTP, for internal services (with implementation in Clojure) – an interesting idea of using ZeroMQ – the sockets on steroids library – instead of HTTP in a way compatible with existing HTTP routing libs; advantages of ZeroMQ: automatic retrial (=> can restart the target service withou noticing), speed, reuse of a connection. The trick is to send a http-like structure (i.e. with method, uri, body) and pass that to Compojure or similar (update: there are now Clojure/core.async bindings for ZeroMQ)
- Joel on Software: Victory Lap for Ask Patents – killing a bad Microsoft patent request in 15 minutes – Ask Patents is a new StackExchange site that enables experts to look at SW patent requests and point to previous existing works that invalidate them; as Joel describes in his successful patent kill story, it is not difficult at all. Hopefully this will manage to really help the patent office and hit woul-be patent trolls hard! #victory
- The Dangers Of “Gamification” In Education by Kathy Sierra (a former game designer, a trainer of trainers at Sun, author of the Head First book series) – gamification is often regarded as something very desirable that will improve our lives; however, as Kathy discusses, it has also dark sides and, applied unappropriately, can actually decrease our intrinsic motivation (therefore it should be nearly never used in e.g. education)
- Choosing an OSS license doesn’t need to be scary (by GitHub) – a human-readable overview of OSS licenses; you should always assign a license to your GitHub account (Add A License can help with that; otherwise it is considered to be “all rights reserved” and you are not giving back to the community (I use the same as Clojure, Eclipse Public License)
- FunctionalTalks.org – “Brilliant people giving brilliant talks on functional programming” – f.ex. Wilkes Joiner: Functional Reactive Programming, Alexander Gounares: All your cores are belong to us, Katie Miller: Superhero monads, Bryan O’Sullivan: Running A Startup On Haskell, Rich Hickey: Introduction To Clojure, John A. De Goes: Building a Data Science Platform in Scala and many more.
- Types vs. Tests: An Epic Battle? – “Amanda Laucher and Paul Snively debate solving problems through types and tests using different approaches.” – can type system replace tests or vice versa? Interesting intro into the discussion for me. Using F#, Scala & more. Same claims: types don’t pay out so much for “small” codebases but scale better than tests. Types – Tests is a spectrum, not two single extremes. When a property should hold “for all,” a type would be a good match. Inductive types (Scala, Haskell?) can become quite complex, dependant types (as in Coq) would be much nicer [if I got that right].
- Paul Irish on Web Application Development Workflow (via M. Noddeland) – if you need to do some web development but are not up to date on the state of art, this might be useful – an overview of tools, utilities, services by a Googler and the person behind Modernizr, HTML5 Boilerplate, Yeoman etc. Including effective shell & dotfiles.GH, better ssh via .ssh/config and authorized_hosts, the all-in-one dev/build tool Yeoman with live reload, BrowserStack for testing, LocalTunnel to easily share anything running locally, Chrome Dev Tools support for SASS and testing devices (emulate touch events, screen sizes), JetBrains’ WebStorm, sharing tools via setapp.me. A genous idea to use GoogleAnalytics to track usage of features in a CLI app!
Other Interesting Stuff
Azul Systems’ high-performance JVM on the Vega architecture (from The Trouble with Erlang (or Erlang is a ghetto)) looks very interesting:
The other night I tweeted “If you’re looking for a language that gets multicore concurrency right, look at how Azul implemented Java on their Vega architecture” and I definitely stand by that. Azul is a company that got a lot of smart hardware and software people together and had them work on designing a custom system which would scale to hundreds of CPU cores (up to 768 of them), heaps that topped 500 GB (up to 768GB), and had the GC pause only 10-20ms at a time. The realtime performance characteristics Azul managed to eek out of their system lead them to often describe their GC as “pauseless”.
- Discussion: How core.async compares to agents, future and promise? – future/promise: 1 producer, 1 value, multiple consumers; agent: an unbounded queue of functions mutating a single value, with multiple producers and consumers (reading the latest value produced); channel: multiple 1:1 producers/consumers, i.e. a value can only be taken once from the channel, using a bounded queue (=> slow consumers can block fast producers). As mentioned elsewhere, channels is a relatively low-level abstraction and other things can be built on the top of it.
- Clojure Tradeoffs (design implications and why you should care) – perhaps not very unbiased but interesting anyway (shared-memory over other computing paradigms, i.e. message-passing, dynamic over static, speed over convenience, composition over IoC, …)
- Rich Hickey’s post introducing core.async with its Go-like channels as a better alternative to a collback hell (I know everybody has already read it but it is still an important link :))
- Tools etc.
- Faster Clojure Startup with Class Data Sharing – use JVM’s capability to include any classes in its boot image and include clojure in it
- lein-ancient – checks for outdated dependencies and plugins => run “lein ancient :all”
- lein-try – a Leiningen plugin that enables you to try a library in a REPL in the context of your project without having to add it to project.clj; simply run “lein try clj-time 0.5.1″ and then in the REPL “(require ‘[clj-time.core :refer :all])” and e.g. “(date-time 1986 10 14)”
- Lemur: tool to launch a Hadoop job locally/on EMR from a job definition file + actions before/after
- Emacs: sexp fold/expand is very useful for exploring source code (hide all but the first lines of all top-level forms with hs-hide-all) – the built-in hs-minor-mode can hide/show all, or hide/show/toggle one but the keys for it are cumbersome; hideshow-org makes it possible to toggle hide/show with TAB, while preserving the original TAB behavior (it does the normal TAB first only only if nothing changes does it expand/fold); very useful!
- devdocs.io (via @palruud): “an all-in-one API documentation reader for [web] developers,” navigable via keyboard – JS, HTML, CSS, DOM, DOM events, jQuery, Underscore.js
- Kilim – a message-passing framework for Java that provides ultra-lightweight threads and facilities for fast, safe, zero-copy messaging between these threads.
- AssertJ – a library of assertions similar to fest-assert but providing a richer set of assertions (nicer API then fest-assert, according to a friend)
- NetflixOSS – Netflix, the online streaming gigant, has open-sourced many fascinating components of its cloud infrastructure such as Karyon, a blueprint for web-ready components with many features (monitoring,…), Genie/Hadoop as a Service, Servo for monitoring, Archaius for configuration management – too many to list. Check out Chris Fregly’s fluxcapacitor, a demo distributed application that uses many of the components
- Tools to keep a daemon running:
Posted by Jakub Holý on July 27, 2013
Running a Java/Clojure app as a daemon on Linux used to be hard but is pretty simple with Ubuntu Upstart (docs). The short story:
- Create an all-in one uberjar via “lein with-profile production ring uberjar” (using the lein-ring plugin; a simple lein uberjar would suffice for an app with a
- Create an upstart <service name>.conf file in /etc/init/
- Run sudo start/stop/status <service name>
And of course it works with Puppet too.
Read the rest of this entry »
Posted in General | Tagged: clojure, DevOps, linux | Comments Off
Posted by Jakub Holý on July 23, 2013
Setting up the new Google Universal Analytics (still in beta) is not completely obvious. You normally won’t be able to send events from localhost and it will claim that “Tracking Not Installed.” Here are some tips how to use Analytics from localhost and test it.
Read the rest of this entry »
Posted in General | Tagged: analytics, frontend, google, troubleshooting, web | Comments Off
Posted by Jakub Holý on July 12, 2013
Incanter 1.5.1 doesn’t support logarithmic axes, fortunately it is easy to add one manually.
Update: Pushed improved version to Incanter.
This is how our final code will look like:
;; core and charts are the incanter namespaces
(defn plot-power 
(let [fun #(Math/pow 10 %)
y-axis (log-axis :label "log(x)")
chart (charts/function-plot fun 0 5)]
(set-axis chart :y y-axis)
(core/view chart :window-title "LogAxis Test: Incanter fun plot")))
Read the rest of this entry »
Posted in General | Tagged: clojure, data, incanter | Comments Off