Archive for the ‘[Dev]Ops’ Category
Posted by Jakub Holý on March 14, 2018
Cross-posted from Telia’s Tech Blog.
We have had our mission-critical webapp running on AWS Elastic Beanstalk for three years and have been extremely happy with it. However we have now outgrown it and move to a manually managed infrastructure and CodeDeploy.
AWS Beanstalk provides you with lot of bang for the buck and enables you to get up and running in no time:
- Simple, no-downtime deployment and automatic roll-back based on user-provided health-check (either one subset of nodes at a time or blue-green deployment)
- Autoscaling
- Managed updates – security fixes and other improvements installed automatically
- Built-in HTTP Proxy with caching in front of your application
- Monitoring dashboard with alerting and access to logs without the need for SSH
- A list of past versions & ability to roll-back
- Support for many runtimes (Java, Node.js, Docker to name just a few)
So if you need a solid, state-of-the-art infrastructure for a web-scale application and you don’t have lot of time and/or skill to build one on AWS on your own, I absolutely recommend Beanstalk.
Read the rest of this entry »
Posted in [Dev]Ops | Tagged: aws, DevOps | 2 Comments »
Posted by Jakub Holý on March 14, 2018
Cross-posted from Telia’s Tech Blog
We use Amazon Web Services (AWS) heavily and are in the process of migrating towards infrastructure-as-code, i.e. creating a textual description of the desired infrastructure in a Domain-Specific Language and letting the tool create and update the infrastructure.
We are lucky enough to have some of the leading Terraform experts in our organisation so they lay out the path and we follow it. We are at an initial stage and everything is thus “work in progress” and far from perfect, therefore it is important to judge leniently. Yet I think I have gain enough experience trying to apply Terraform both now and in the past to speak about some of the (current?) limitations and disadvantages and to consider alternatives.
Read the rest of this entry »
Posted in [Dev]Ops | Tagged: aws, Terraform | Comments Off on Pains with Terraform (perhaps use Sceptre next time?)
Posted by Jakub Holý on October 1, 2015
These 2 magical lines will protect your upstream server from possible overload of many users try to access the same in cached or expired content:
proxy_cache_use_stale updating timeout; # Serve the cached version even when outdated while refreshing it
proxy_cache_lock on; # Only one req is allowed to load/refresh the item, others wait / get the stale one
You can verify this using Shopify’s Toxiproxy.
❤ Nginx
Posted in [Dev]Ops | Comments Off on Nginx: Protecting upstream from overload on cache miss
Posted by Jakub Holý on July 30, 2015
Gor is a great utility for replicating (a subset of) production traffic to a staging/test environment. Running it on AWS Elastic Beanstalk (EB) has some challenges, mainly that it doesn’t support running as a daemon and that there isn’t any documentation/examples for doing this. Well, here is a solution:
Read the rest of this entry »
Posted in [Dev]Ops | Tagged: aws | Comments Off on Running Gor, the HTTP traffic replayer, as a service on AWS Elastic Beanstalk
Posted by Jakub Holý on July 30, 2015
If you are adding a service entry to your .ebextensions/
config to run a service in AWS Elastic Beanstalk and it fails with either “Could not enable service [..]” or “Could not disable service [..]” (based on the value of ensureRunning
), make sure that the service init.d file supports chkconfig, i.e. contains the comments it looks for.
Posted in [Dev]Ops | Tagged: aws | 1 Comment »
Posted by Jakub Holý on July 29, 2015
Our webshop, nettbutikk.netcom.no, runs on AWS Elastic Beanstalk and we use .ebextensions/
to customize the environment. I have been just trying to get Gor running on our leader production instance to replay some traffic to our staging environment so that we get a much richer feedback from it. However the container_command
I used caused the instance to time out and trash the environment, against all reason. The documentation doesn’t help and troubleshooting this is hard due to lack of feedback and time-consuming. Luckily I have arrived to a solution.
Read the rest of this entry »
Posted in [Dev]Ops | Tagged: aws | Comments Off on Fixing a mysterious .ebextensions command time out (AWS Elastic Beanstalk)
Posted by Jakub Holý on July 29, 2015
Philipp Garbe describes how to pass environment variables that you want to keep private to a public Docker instance run on Amazon Web Services (beanstalk or ECS) in his post How to Run HuBot in Docker on AWS EC2 Container Services – Part 3. The trick is:
- Put them into an
env.sh
file that you can source on S3 (and allow the appropriate EC2 IAM role to access it)
- As a part of your startup CMD, run
aws s3 cp
to fetch and then source it
Here is his example of the CMD from a Dockerfile:
CMD ["/bin/sh", "-c", "aws s3 cp --region eu-west-1 s3://your-bucket/env.sh .; . ./env.sh; bin/hubot --adapter slack"]
See the full source code in his GitHub repo. Thanks for sharing, Phillipp!
Posted in [Dev]Ops | Tagged: aws, Docker | Comments Off on AWS: Passing private configuration to a Docker container (via S3)
Posted by Jakub Holý on July 22, 2015
By Michael T. Nygard, 2007, ISBN: 978-0-9787-3921-8
My digest and review of the book.
Review
Of the books I have read, Release It! is the one I would require all “senior” developers to read (together with something like Architecting Enterprise Solutions: Patterns for High-Capability Internet-based Systems). Especially the first part on stability with its patterns and anti-patterns is a must read. Without knowing and applying them, we create systems that react to problems like a dry savannah to a burning match. I found also to next to last chapter, #17 Transparency, very valuable, especially the metrics and design of the OpsDB and observation practices.
One thing I have left out of the digest which is really worth reading are the war stories that introduce each section, they are really interesting, inspiring, and educational.
Extra Links
Stability
Stability x longevity bugs

Selected (anti)patterns
Stability antipatterns
Integration points
Integration point = call to a DB, WS, … . Stability risk #1.
Read the rest of this entry »
Posted in [Dev]Ops | Tagged: architecture, book, ops, performance | Comments Off on Book Review & Digest: Release It! Design and Deploy Production-Ready Software
Posted by Jakub Holý on June 11, 2015
It took me quite a while to figure out the right syntax for filtering instances by tag name and value in the AWS EC2 API’s describeInstances.
The documentation is not exactly crystal-clear to me:
tag
:key=value – The key/value combination of a tag assigned to the resource, where tag
:key is the tag’s key.
Anyway, here is the proper syntax, provided we are interested in the tag elasticbeanstalk:environment-name:
var params = {
Filters: [
{
Name: 'tag:elasticbeanstalk:environment-name',
Values: ['mySuperApp']
}
]
};
ec2.describeInstances(params);
So the name of the tag is embedded in the Name part and not, as I initially understood,
{ Name: 'tag', Values: ['elasticbeanstalk:environment-name=mySuperApp'] }
Credit: garnaat.
Posted in [Dev]Ops | Tagged: aws, ops | Comments Off on AWS API: Proper syntax for filtering by tag name and value (e.g. describeInstances)