The Parallel Change technique is intended to make it possible to change code in a small, save steps by first adding the new way of doing things (without breaking the old one; “expand”), then switching over to the new way (“migrate”), and finally removing the old way (“contract”, i.e. make smaller). Here is an example of it applied in practice to refactor code producing a large JSON that contains a dictionary of addresses at one place and refers to them by their keys at other places. The goal is to rename the key. (We can’t use simple search & replace for reasons.)
Archive for the ‘General’ Category
Posted by Jakub Holý on February 3, 2017
Posted by Jakub Holý on November 27, 2015
Our team has struggled with slow calls to the back-end, resulting in unpleasant, user-perceivable delays. While a direct (HTTP) call to a backend REST service took around 50ms, our median time was around 300ms (while using HTTPS and a proxy between us and the service).
We have just decreased that time to median of 80ms by making sure to keep the connections alive and reusing them, which in Node.js can be achieved via using an https.agent and setting its keepAlive: true (see the Node TLS documentation).
PayPal has a couple of additional useful tips in their 4/2014 post Outbound SSL Performance in Node.js, mainly:
- Disable expensive SSL ciphers (if you don’t need their strength)
- Enable SSL session resume, if supported by the server, for shorter handshakes – the StrongLoop post “How-to Improve Node.js HTTPS Server Performance” explains how to enable SSL session resume
- Keep Alive
The article SSL handshake latency and HTTPS optimizations (via Victor Danell) explains the ± 3.5* higher cost of SSL due to the 3 roundtrips need for the handshake (+ key generation time) and shows how to use curl to time connections and their SSL parts, as well as how to use OpenSSL and Tcpdump to learn even more about it.
See also IsTlsFastYet.com for a lot of valuable information, benchmarks etc.
(See the articles linked to above for examples)
- openssl s_client
- pathchar by the traceroute author, intended to help to “find the bandwidth, delay, average queue and loss rate of every hop between any source & destination”; there is also pchar, based on it
Posted by Jakub Holý on October 7, 2015
People don’t really like changes yet change we must in this fast-developing world. How to introduce a change, or rather how to inspire people to embrace a change? That is one of the main questions of my professional life.
- An experienced speaker once recommended sharing personal experiences (even – or especially – if they make me vulnerable) as it is much easier for people to relate to them than to general statements.
- A Cognicast eposide mentioned storytelling as a great tool for introductory guides. We humans are natural storytellers, we think in stories and relate to them much more easily – so a story should be great also to communicate the value of a change.
- My ex-colleague Therese Ingebrigtsen gave an inspiring talk presenting some points from The Switch – mainly that we need to address the recipient’s minds with rational arguments, but also their hearts to involve their emotion (e.g. by drawing a picture of the new bright future), and that it is important to show a clear path forward.
Posted by Jakub Holý on June 19, 2015
A great conference. A good deal of good talks.
To (perhaps) check later
- Practical CSS tips & tricks for backend developers – really useful tips! video
- No Estimates, Let’s Explore the Possibilities video
- Form with Function: Adding Behavior with CSS – recommended by a friend (modal dialogs, tab switching, …) video
- ? Anti-fragile and feedback. Trying to make up for the failures of “agile.” video
- ? Designing and Programming Accessible Website and App UIs video
- ? Declarative REST: State Machines for the Web video
- Continuous Delivery for Architects – Neal Ford video
- ? High Performance in the Critical Rendering Path – how to make pages to load fast video
- This is Water – Neal Ford – an excursion into a strange, fantastical world with things like immutable database server, phoenix machines, and lambdas. video
- Securing Web APIs – Patterns & Anti-Patterns video
- ? Functional Data – event sourcing & FP video
- ? Taking other peoples money: A guide to online payments video
- ? Running Docker and Containers in Development and Production video
- Not Even Close: The State of Computer Security
- 595 billions income – untouched by human hands video
- Boosting security with HTTP headers video
- The rest of ReST – we’ll look at the challenges of building usable real-world ReST APIs: Hypertext Application Language (HAL), HTTP Patch, … video
- ? How do you scale a logging infrastructure to accept a billion messages a day? – DB -> ELK -> ELK + Kafka video
- ? Learning Client Hypermedia from the Ground Up – how to move specific knowledge of 1) addresses, 2) inputs, and 3) workflow out of the client app and place it into the message => a more robust, adaptable, and resilient client video
- ? Make it Faster – Lessons Learned from Benchmarking NoSQL on the AWS Cloud – best practices for performing database benchmarking on the AWS cloud & how to get more speed and efficiency in your production workloads video
- ? Crafting Evolvable Web API Representations – like structuring for evolution, sizing for optimum caching, the different ways to include metadata, … video
- ? Mob Programming, A Whole Team Approach video
- ? Removing barriers – JetBrains’s good and bad expericences with minimizing management video
Keynote Data and Goliath ☆☆☆☆
Posted in General | Comments Off on NDC Oslo 2015: Talk notes, recommended talks (security, FP, etc.)
Posted by Jakub Holý on June 17, 2015
Highlights – resources
Personal security and privacy
- https://www.entropay.com/ – a Prepaid Virtual Visa Card
- mailinator.com – tmp email
- f-secure VPN
- https://www.netsparker.com/ – scan a site for issues (insecure cookies, framework disclosure, SQL injection, …) (lot of $k)
- https://report-uri.io/ – get reports when CSP rules violated; also displays CSP headers for a site in a human-friendly way
- https://securityheaders.io/ check quality of headers wrt security
- free SSL – http://www.startssl.com/, https://www.cloudflare.com/ (also provides web app firewall and other protections) ;
- SSL quality check: https://www.ssllabs.com/ssltest/
- https://letsencrypt.org/ – free, automated, open Certificate Authority (Linux Found., Mozilla)
- HSTS Preload – tell Chrome, FF that your site should only be ever loaded over HTTPS – https://hstspreload.appspot.com/
- https://twitter.com/jmgosney – one of ppl behind http://passwordscon.org . http://password-hashing.net experts panel. Team Hashcat.
- ! http://krebsonsecurity.com/
- ! http://www.troyhunt.com/
- ! https://www.schneier.com/
- ! https://twitter.com/mikko (of F-Secure) also great [TED] talks
- kevin mitnick (jailed for hacking; twitter, books)
Posted by Jakub Holý on June 2, 2015
Mounting an EBS volume to a Docker instance running on Amazon Elastic Beanstalk (EB) is surprisingly tricky. The good news is that it is possible.
I will describe how to automatically create and mount a new EBS volume (optionally based on a snapshot). If you would prefer to mount a specific, existing EBS volume, you should check out leg100’s docker-ebs-attach (using AWS API to mount the volume) that you can use either in a multi-container setup or just include the relevant parts in your own Dockerfile.
The problem with EBS volumes is that, if I am correct, a volume can only be mounted to a single EC2 instance – and thus doesn’t play well with EB’s autoscaling. That is why EB supports only creating and mounting a fresh volume for each instance.
Posted by Jakub Holý on May 7, 2015
When looking for ways to discover whether a proxy is being used by OS X, you will be typically pointed to
However that does not always work – for example when using “Auto Proxy Discovery” and/or “Automatic Proxy Configuration” with a proxy.pac file.
scutils --proxy seems to detect all these cases (though it cannot give you the proxy when using auto config, I suppose):
Read the rest of this entry »
Posted by Jakub Holý on May 7, 2015
I have derived the Docker container docker-grafana-influxdb-cloudwatch that bundles Grafana dashboards, InfluxDB for metrics storage, and runs cloudwatch-to-graphite as a cron job to fetch selected metrics from AWS CloudWatch and feed them into the InfluxDB using its Graphite input plugin. It is configured so that you can run it in AWS Elastic Beanstalk (the main problem being that only a single port can be exposed – I therefore use Nginx to expose the InfluxDB API needed by Grafana at :80/db/).
Posted by Jakub Holý on April 30, 2015
The first Continuous Delivery and DevOps Conference in Oslo is over. It was nice to see so many people interested in the topic. I would have preferred more practical talks of the “how we did it” type over the “why” type but it was OK, though next year I would prefer flatMap. Here are my highlights:
- Atmel is using a physical robot to plug and connect a particular configuration of circuit boards to test; your automated testing challenges cannot be greater than theirs!
- Continuous Delivery decreases the risk of outage and time-to-recovery while enabling faster innovation, correlates with higher profits; No efficiency improvement will outperform cycle time reduction
- Estimation pathologies; focus on value rather than costs
- Stop talking about requirements, they are fake; they’re just beliefs about what may add value to customers. Use hypothesis instead!
- Cisco: Most of the tools increasing productivity (and some innovation) were produced by engineers in their “spare” time; slack time is thus crucial
- How does Cisco grow professionalism : optimise for the 10% best, not the 10% weakest developers; slack time; make everything visible; encourage code reviews but avoid making them mandatory; see the slide
- CALMS: Culture, Automation, Lean, Measurement, Sharing. The pillars of devOps
- Cisco invested a lot in crafting their build system, tailored test frameworks, and emulators to be able to get quick and quality feedback – because it pays off
- “Make you own build system” says @olvemaudal at @CoDeOSL. IME this is inevitable for non-trivial projects, and a good investment.
- Unleash: Feature Toggles server and Java/Node client by FINN.no
- “They asked for a report while they actually need just a list of data, the result of a simple SQL query; have we listened to them, we would have wasted hours creating a report in the report framework with logos and all the crap.”
Posted by Jakub Holý on March 31, 2015
I want to know when our app starts getting slower so I sat up an alarm on the Latency metric of our ELB. According to the AWS Console, “This alarm will trigger when the blue line [average latency over the period of 15 min] goes above the red line [2 sec] for a duration of 45 minutes.” (I.e. it triggers if Latency > 2 for 3 consecutive period(s).) This is exactly what I need – except that it is a lie.
This night I got 8 alarm/ok notifications even though the average latency has never been over 2 sec for 45 minutes. The problem is that CloudWatch ignores null/missing data. So if you have a slow request at 3am and no other request comes until 4am, it will look at [slow, null, null, null] and trigger the alarm.
So I want to configure it to treat null as 0 and preferably to ignore latency if it only affected a single user. But there is no way to do this in CloudWatch.
Solution: I will likely need to run my own job that will read the metrics and produce a normalized, reasonable metric – replacing null / missing data with 0 and weight the average latency by the number of users in the period.