Posts Tagged ‘DevOps’
Posted by Jakub Holý on April 30, 2015
The first Continuous Delivery and DevOps Conference in Oslo is over. It was nice to see so many people interested in the topic. I would have preferred more practical talks of the “how we did it” type over the “why” type but it was OK, though next year I would prefer flatMap. Here are my highlights:
- Atmel is using a physical robot to plug and connect a particular configuration of circuit boards to test; your automated testing challenges cannot be greater than theirs!
- Continuous Delivery decreases the risk of outage and time-to-recovery while enabling faster innovation, correlates with higher profits; No efficiency improvement will outperform cycle time reduction
- Estimation pathologies; focus on value rather than costs
- Stop talking about requirements, they are fake; they’re just beliefs about what may add value to customers. Use hypothesis instead!
- Finn: Most of the tools increasing productivity (and some innovation) were produced by engineers in their “spare” time; slack time is thus crucial
- How does grow professionalism : optimise for the 10% best, not the 10% weakest developers; slack time; make everything visible; encourage code reviews but avoid making them mandatory; see the slide
- CALMS: Culture, Automation, Lean, Measurement, Sharing. The pillars of devOps
- Cisco invested a lot in crafting their build system, tailored test frameworks, and emulators to be able to get quick and quality feedback – because it pays off
- “Make you own build system” says @olvemaudal at @CoDeOSL. IME this is inevitable for non-trivial projects, and a good investment.
- Unleash: Feature Toggles server and Java/Node client by FINN.no
- “They asked for a report while they actually need just a list of data, the result of a simple SQL query; have we listened to them, we would have wasted hours creating a report in the report framework with logos and all the crap.”
Posted in General | Tagged: continuous_deployment, DevOps | Leave a Comment »
Posted by Jakub Holý on May 31, 2014
- Monolith – from The Codeless Code – fables and koans for the SW engineer – the Monad monolth #Haskell #fun
- http2 explained (pdf, 27 pages) – cons of http 1 (huge spec / no full impl., wasteful use of TCP <=> latency [x spriting, inlining, concatenation, sharding]) => make it less latency sensitive, fix pipelining (issue a req before previous one finished), stop the need for ever increasing # connections, remove/reduce optional parts of http. Http2 is binary; multiple “streams” over 1 connection => much less conns, faster data delivery; header/data compression; [predictive] resource pushing; . Inspired by SPDY. Chrome and Mozilla will only support it over TLS, yay! (see also Is TLS Fast Yet? [yes, it is]) Promise: faster, more responsive web pages & deprecation of http/1 workarounds => simplified web dev.
- exercism.io – crowd-sourced good code mentorship – get an exercise, implement it in any of the supported language(s), submit and get feedback, repeat; when finished, you too can comment the same excercise submitted by others while working on your next assignment. Languages include Clojure, JS, Scala, Python, Haskell, Go, Elixir, Java, and more.
Podcasts (FP & related)
- Cognicast (also @ iTunes) – Clojure, FP, etc.
- Functional Geekery (@ iTunes) – A podcast on Functional Programming, covering topics across multiple languages.
- Mostly λazy…a Clojure podcast by Chas Emerick
- Giant Robots Smashing into other Giant Robots – “a weekly technical podcast discussing development, design, and the business of software development”
- Software Engineering Radio (@ iTunes) – “The goal is to be a lasting educational resource, not a newscast. Every two to four weeks, a new episode is published that covers all topics software engineering. Episodes are either tutorials on a specific topic, or an interview with a well-known expert from the software engineering world.”
- EngineerVsDesigner – design insight (@ iTunes) – product design podcast – the latest digital design news, tips & tricks, Q&A, and an industry special guest
- core.async walkthrough for practicioners (12/2013; original title “Core.Async”) – Tim Baldrige 40m talk from Conj is a must-watch for anybody that wants to understand and use core.async. Examples from and beyond the talk are at GitHub. Highlights: While take! takes a callback, <!! wraps that into a promise so that it blocks and returns the value. Thread is similar to future, running the code asynchronously, but returns a channel instead of a promise. So does go, only it does not use real threads but simulated light-weight ones via a state machine and parking. Tim shows the different buffers (normal, dropping = earliest values kept, sliding = latest valus kept). Other topics: alts!!, …
Related vids by Tim: Core.Async Channel Internals, Core Async Go Macro Internals – Part I, Core Async Go Macros – Part II
- Clojure, Leiningen and Functional Programming for Java developers– implementing a queue in a functional way
- ownCloud – your own Dropbox/Google Drive, run on your server – sharing files between devices / PCs / web, syncing calendar and contacts, collaborative editing of documents (ODF)
- Mailpile – “A modern, fast web-mail client with user-friendly encryption and privacy features.”, to be self-hosted on a PC, RaspberryPI, USB stick
- Blackhole – role-based ssh proxy – an app that enables you to manage what users can ssh to what server as a particular user, from users’ point of view this is a ssh proxy; useful if many people need access to many servers but you do not want to add them all as users on those servers.
- Wuala – Secure Cloud Storage – Backup. Sync. Share. Access Everywhere. – Dropbox alternative, secure by default
- fb-flo – Facebook’s live-coding tool
- owncloud.org – self-hosted Dropbox-like service with calendar and contact sync and more
Posted in General, Languages, Tools, Top links of month | Tagged: clojure, cloud, DevOps, e-mail, haskell, mentoring, podcast, privacy, security, sharing, ssh | Comments Off on Most interesting links of May ’14
Posted by Jakub Holý on March 5, 2014
Ansible 1.5 has partial support for recursive copy of files:
- the synchronize module, using rsync
- cons: does not support group=, owner=
- -C and –diff – it does not print diff of the files changed; when running ansible with -v, it will print output of rsync’s –itemize-changes, i.e., for each changed file/dir, something like “<f.st…… conf/httpd.conf\n” (< = file uploaded, s = size changed, t = timestamp changed, . = this attribute has not been changed)
- the copy module
- -C –diff – it only reports “changed” without naming the changed files or showing diffs (unless there is only one changed file)
- the local_action module, used to run rsync manually (essentially the same as synchronize but more control)
- So the only way to do a kind of recursive copy with working diff is to use
copy with with_fileglob for each directory and subdirectory :-(
To check differences (without diff) manually:
rsync -e ssh -vrnc --itemize-changes source/dir myuser@myserver:/opt/dest/
where v= verbose, r = recursive, n = dry-run, c = check based on checksum, not timestamp+size; a typical output for a changed file is
<fcsT...... httpd.conf (< = to be uploaded, f = it is a file, c = checksum differ, s = size differ, T = timestamp would be updated).
Posted in Tools | Tagged: ansible, DevOps | Comments Off on Recursive Copy In Ansible 1.5 And –diff
Posted by Jakub Holý on March 4, 2014
Few tips for troubleshooting Ansible, based on my brief experiences with Ansible 1.4 (read: do not rely on this info too much).
Run ansible-playbook in the verbose mode
ansible-playbook -vvvv ... will provide you with plenty of details of what is going on. (Notice that additional v:s, starting from none, add more detail.)
Check out Ansible sources and use the ./hacking/test-module script – see Developing Modules.
Read the rest of this entry »
Posted in Tools | Tagged: ansible, DevOps | Comments Off on Ansible Troubleshooting Tips
Posted by Jakub Holý on February 24, 2014
It is tricky to manage JBoss with a provisioner such as Puppet or Ansible because its
domain.xml contains not only rather static configuration but also sections that change quite often such as
deployments. So how can we manage the static parts of
domain.xml with f.ex. Ansible while still enabling developers to deploy at will via
jboss-cli (and thus changing the
<deployments> sections of the file)? Here is one possible solution, based on extracting the sections from the current file and merging them into the template.
Read the rest of this entry »
Posted in General | Tagged: ansible, DevOps, jboss | Comments Off on Handling Deployments When Provisioning JBoss domain.xml (With Ansible)
Posted by Jakub Holý on February 16, 2014
I have created a small project to demonstrate some features of Ansible, the new DevOps hotness, including Vagrant VMs for running Ansible and for testing the configuration. Either go straight to
or continue reading the copy & paste here.
This project has three things of interest:
- A non-trivial Ansible configuration that demonstrates couple of useful features and tricks
- A Vagrant/VirtualBox virtual machine with Ansible & co. to make it easy to run it (even on Windows)
- Another VM that can be used to test the configuration
And of course all the plumbing that makes them work together. It might be therefore a good base for Ansible projects of your own.
Read the rest of this entry »
Posted in Tools | Tagged: ansible, DevOps | Comments Off on Demonstration of Ansible Features With Control & Test VMs
Posted by Jakub Holý on December 31, 2013
- HBR: Want to Build Resilience? Kill the Complexity – a highly interesting, thought provoking article relevant both to technology in particular and the society in general; f.ex.: more security features are bad for they make us behave less safely (risk compensation) and are more fragile w.r.t. unexpected events. “Complexity is a clear and present danger to both firms and the global financial system: it makes both much harder to manage, govern, audit, regulate and support effectively in times of crisis. [..] Combine complex, Robust-Yet-Fragile systems, risk-compensating human psyches, and risk-homeostatic organizational cultures, and you inevitably get catastrophes of all kinds: meltdowns, economic crises, and the like.” The solution to future financial crisis is primarily not more regulation but simplification of the system – to make it easier to police, tougher to game. We also need to decrease interconnectednes (of banks etc.), one of the primary sources of complexity. Also a great example of US Army combatting complex, high-risk situations by employing “devil’s advocates / professional skeptics” trained to help “avoid the perils of overconfidence, strategic brittleness, and groupthink. The goal is to respectfully help leaders in complex situations unearth untested assumptions, consider alternative interpretations and “think like the other”“.
- The Dark Side of Technology – technologies provide great opportunities – but also risks we should be aware of – they create a world of mounting performance pressure for all of us (individuals, companies, states), accelerate the rate of change, increasing uncertanity (=> risk of Taleb’s black swans). “All of this mounting pressure has an understandable but very dangerous consequence. It draws out and intensifies certain cognitive biases [..]” – magnify our perception of risk, shrink our time horizons, foster a more and more reactive approach to the world, the “if you win, I will lose” view, erode our ability to trust anyone – and “combined effect of these cognitive biases increases the temptation to use these new digital infrastructures in a dysfunctional way: surveillance and control in all aspects of our economic, social and political life.” => “significantly increase[d] the likelihood of an economic, social and political backlash, driven by an unholy alliance between those who have power today and those who have achieved some modest degree of income and success.”
Complexity theory: the more connected a system is, the more vulnerable it becomes to cascades of disruptive information/action.
- What Do Government Agencies Have Against 23andMe, Uber, and Airbnb? – innovative startups do not fit into established rules and thus bureaucrats do not know how to handle them and resort to their favourite weapon: saying no, i.e. enforcing rules that harm them (f.ex. France recently passed a law that requires Uber etc. drivers to wait 15 min before picking up a customer so that established taxi services have it easier; wot?!)
- Nonviolent communication in action – wonderful stories about NVC being applied in difficult situations with a great success
- Quora: Michael Wolfe’s answer to Engineering Management: Why are software development task estimations regularly off by a factor of 2-3? – a wonderful story explaining to a layman why estimation is hard, on the example of a hike from SF to LA
- Johannes Brodwall: Why I stopped using Spring [IoC] – worth to read criticism of Spring by a respected and experienced architect and developer; summary – dependency injection is good bug “magical” frameworks decrease understandability and encourage unnecessarily complex code => smaller code, , easier to navigate and understand and easier to test
- Misunderstanding technical debt – a brief discussion of the various forms of tech. debt (crappy code x misaligned design and problem domain x competence debt)
- Tension and Flaws Before Health Website Crash – surprising lack of understanding and tensions between the government and contractors on HealthCare.gov – “a huge gap between the administration’s grand hopes and the practicalities of building a website that could function on opening day” – also terribly decision making, shifting requirements (what news!), management’s lack of decision power, CGI’s blame-shifting. A nice horror story. The former head knew that they should “greatly simplify the site’s functions” – but the current head wasn’t able to “talk them out of it”.
- The Log: What every software engineer should know about real-time data’s unifying abstraction – logs are everywhere and especially important in distributed apps – DB logs, append-only logs, transaction logs – “You can’t fully understand databases, NoSQL stores, key value stores, replication, paxos, hadoop, version control, or almost any software system without understanding logs” – I have only read a small part but it looks useful
- What I Wish I Knew When Learning Haskell tl;dr
- Better Than Unit Tests – a good overview of testing approaches beyond unit tests – including “Automated Contract Testing” (ability to define a contract for a web service, use it to test it and to simulate it; see Internet of Strings for more info), Property-based Testing (test generic properties using random data/calls as with Quickcheck), Fault Injection (run on multiple VMs, simulate network failures), Simulation Testing as with Simulant.
- Use #NoEstimates to create options and deliver value reliably – a brief post with an example of an estimation-based vs. no-estimates project (i.e. more focus on delivering early, discovery)
- How Google Sold Its Engineers on Management – managers may be useful after all :-); a report about Google’s research into management and subsequent (sometimes radical) improvements in management style/skills and people satisfaction; I love that Google hasn’t HR but “people ops”
- Roy Osherove: Technical Disobedience – take nothing for granted, don’t let the system/process stop you, be creative about finding ways to improve your team’s productivity; there always is a way. Nice examples.
- Uncle Bob: Extreme Programming, a Reflection – a reflection on changes in the past ~ 14 years since XP that have seen many of the “extreme” practices becoming mainstream
- The Anti-Meeting Manifesto – essentially a checklist and tips for limitting meetings to minimum
- Ethics Daily – a freuently published collection of ethics-related links, articles, talks etc.
- British Library uploads one million public domain images to the net for remix and reuse – from 17th-19th century books, available at Flicker; it asks for help categorizing and documenting them, metadata of the images at GitHub (with links to Flicker); great project, interesting pics
- The Software Engineers’ Oath – lets bring ethics back to our daily (work) lives – respect the knowledge of others, use technology for good (!!!), keep learning, writing code for people, not ashamed to admit lack of knowledge, respect for privacy, obligation to make lifes of humans better, …
- David Pollak: Some musings on Scala and Clojure by a long time Scala dude (46 min) – a subjective but balanced comparison of Scala and Clojure and their strengths/weaknesses by the author of the Scala Lift framework (doing Scala since 2006, Clojure since 2013)
- Apache Sirona – a new monitoring tool in the Apache incubator – “a simple but extensible monitoring solution for Java applications” with support for HTTP, JDBC, JAX-RS, CDI, ehcache, with data published e.g. to Graphite or Square Cube. It is still very new.
- GenieJS – Ctrl-Space to popup a command-prompt for your web page, inspired by Alfred (type ‘ to see all possible commands)
A good #agile team considers their backlog inaccurate. It is merely a list of assumptions that must be tested & refined by shipping product
– @mick maguire 12/10
Ada Lovelace (1st program), Grace Hopper (1st compiler), Adele Goldberg (1st OO language), why would anyone think women aren’t in computing?
– @Dan North 12/11
There will always be a shortage of talented, self-motivated creative professionals who will unquestioningly follow orders.
– @Thomas K Nilsson 12/7
Estimation paradox = If something unpredictable happens, predict how long it will take to fix it
– me 12/7
IT systems can be inspired by AK-47 a.k.a. Kalashnikov. The rifle was purposefully designed to be simple and to be tolerant to imperfections in most parts; as a result, it required essentially no maintenance and was extremely reliable.
– summarized from Roman Pichlík’s Odkaz Michaila Kalašnikova softwarovému vývoji
Posted by Jakub Holý on September 5, 2013
Use case: Deploy a new version of a webapp so that all new users are sent to the new
version while users with open sessions continue using the previous version
(so that they don’t loose their precious session state). Users of the new version
can explicitely ask for the previous version in the case that it doesn’t work as expected and vice versa.
Benefits: Get new features to users that need them as soon as possible without affecting
anybody negatively and without risking that a defect will prevent users from achieving their goal
(thanks to being able to fall back to the previous version).
Read the rest of this entry »
Posted in General | Tagged: continuous_deployment, DevOps, java | Comments Off on Webapp Blue-Green Deployment Without Breaking Sessions/With Fallback With HAProxy
Posted by Jakub Holý on September 3, 2013
Are you using Puppet in the client-server setup and want to test the configuration for a particular node without actually changing it? You can do that by fooling Puppet Master into believing that a Vagrant virtual machine (VM) is that node and applying it there. The process is simple: you essentially only need to get the nodes’ cert/private key and supply them to Puppet (and likely make sure that the hostname puppet can be resolved from within the VM). Let’s see it in detail.
Read the rest of this entry »
Posted in Testing | Tagged: DevOps, puppet, vagrant | Comments Off on Test Puppet config of an existing node using Puppet Master via Vagrant
Posted by Jakub Holý on August 31, 2013
Sorry folks, this month it will be very brief. I have many more great stuff in the queue but haven’t managed to write it down yet. Next month will be heavy :-)
- Why Software Projects are Terrible and How Not To Fix Them – many teams are not ready to embrace new/better software practices, primarly for two reasons: 1) most of them are nonintuitive (f.ex. adding more people will slow dev down) and need to be sold through a high hierarchy of managament – but people/managers/organizations don’t really care, it takes years for good/bad practices to have an impact, which is not relevant “now.” 2) Businss objectives change too quickly and SW is blamed for not delivering. Based on evaluating many failed projects. Conclusion: Choose carefully people/organizations your work with. Avoid blame-driven ones. Quote on middle managers: “He has to put more developers on the project, call a meeting and yell at people, and other arbitrary bad ideas. Not because he thinks those will solve the problem. In fact, managers often do this in spite of the fact that they know it’s bad. Because that’s what will convince upper management that they’re doing their best.” “In the vast majority of failed projects I’ve been called to looked at, the managers have not read one book on software engineering.“
Data & Analytics
- Big Data: Kafka for uSwitch’s Event Pipeline – a better alternative to log files – use LinkedIn’s Kafka for messaging, have MR jobs to import latest messages into Hadoop/HDFS. The advantage of Kafka is that it persists the messages for a period of time so it is easy to batch-import and even re-import them. The uSwitch’s talk Users As Data explains the downsides of log files. LinkedIn’s Camus is a tool for importing messages from Kafka to HDFS.
- Realtime Analytics with Storm and Hadoop (at Twitter; presentation deck) – pre-aggregate some data into a read-only, random read DB such as ElephantDB, Voldemort, or Manhattan. For newer data use Storm and aggregated data in a read-write, big-data DB such as HBase, Riak, or Cassandra. For stuff that cannot be pre-aggregated you might use Storm’s Distributed RPC.
- The Unified Logging Infrastructure for Data Analytics at Twitter – a paper from late 2012 that presents “Twitter’s production logging infrastructure and its evolution from application-speciﬁc logging to a uni- ﬁed “client events” log format, where messages are captured in common, well-formatted, ﬂexible Thrift messages” – with the benefit of “s streamlined log collection and data analysis”.
- Development and Deployment at Facebook (Kent Beck et. al., 8/2013, 13p paper) – “More than one billion users log in to Facebook at least once a month to connect and share content with each other. Among other activities, these users upload over 2.5 billion content items every day. In this article we describe the development and deployment of the software that supports all this activity, focusing on the site’s primary codebase for the Web front-end.“
- One of the most valuable talks I’ve seen, in just 18 min: The Progress Principle – about the disengagement crisis and motivation at work by Teresa Amabile at TEDx Atlanta (via @thovden). Disengagement from work is increasing, at all age and salary levels, and leads to unhappy people, low productivity, huge financial losses. Based on analysing diaries of 12k participants, the single most important engaging and motivating factor is making progress in a meaningful work (including small wins). A culture of management by fear and punishment for failure creates disengagement and can crush even an innovative, profitable, praised company in a few years. Everybody, though especially the management, creates the culture through their everyday, small actions. If everybody focuses on catalysing progress and supporting their fellow humans through good and bad times, engagement and success will follow. Remove progress inhibitors, nourish the human spirit (acknowledge what we humans value, encourage people). Yet of the managers asked, very few knew of the significance of making progress (or, I can assume, of supporting people and making them happy(er) and the impact of our inner work life (perceptions, emotions, etc.) on our productivity and creativity). The study included two seemingly similar, successfull companies, one with great engagement, another with a new management that managed to destroy the engagement and thus eventually the company. Actions to take: catalyse progress, celebrate wins, encourage and support your colleagues.
- Wonderful Clojure Cheatsheet 1.5 with tooltips showing the doc and summary of information available at clojuredocs.org (other Clj versions), by Andy Fingerhut
- Chas Emerick’s Clojure type selection flowchart to help you decide whether to use a map, a record, reify, proxy, gen-class, or deftype. (Reify and proxy don’t produce a class but just an instance of an anonymous class; proxy can extend a base class, reify cannot. gen-class produces a class visible from Java and can extend Java classes. …)
- Docker.io – pack, ship and run any application (and its dependencies) as a lightweight container, i.e. essentially “a VM without the overhead of a VM,” using linux containers (chroot on steroids with resource limits via control groups) see reports of some uses such as Java app deployment, desktop virtualization, automatic app deployment in GitHub commit. Docker also supports evolving the containers over time, i.e. deploying new version, by pushing just diffs so it’s low-overhead. You can build a container (include files, SW, forward ports, …) using a Dockerfile. See dotScale 2013 – Solomon Hykes – Why we built Docker for an intro (20 min).
- Packer.io – tool for building pre-configured VM images for different platforms (EC2, VirtualBox, …), remotely similar to Netflix’s Aminator. See Immutable Servers With Packer and Puppet for an example use case.
- Ubuntu-build Vagrant boxes at cloud-images.ubuntu.com/vagrant/
- SlimerJS – PhantomJS-compatible headless browser engine based on Firefox/Gecko (well, it is not fully headless yet but that is planned; the main focus now is full compatibility with PhantomJS’ API) (Both work with CasperJS for navigational steps/testing.)
- localtunnel – instantly show locally running webapp/server to the rest of the world (gem install localtunnel, localtunnel <port to share>, => share the url returned, e.g. http://xyz.localtunnel.com) – I haven’t tried it but it looks simple and very convenient
- Logstash + Kibana (via @mortenberg): take control of your logs – while Logstash can collect (from multiple servers/services), parse (over 100 built-in patterns), store, index, search your logs, Kibana is a web interface to seach them, view them in realtime (based on a query) etc. See this Logstash slides (9/2012) and an overview of Kibana’s powers. PS: Logstash can also compute metrics and send them to graphite etc. It is typically used with ElasticSearch.
- ncdu is an interactive, command-like disk usage browser that shows a list of directories sorted by size shown in human-friendly units, you can navigate with arrows and enter and i to show the current dir/file info, d to delete it, q to quit; check out this article about ncdu with screenshots and ncdu man page. Install via Apt etc., run f.ex. with ncdu -x / .
- vagrant-cachier – Vagrant plugin for caching apt/yum/.. packages locally, thus speeding up destroy+up