Why we love AWS Beanstalk but are leaving it anyway

Cross-posted from Telia’s Tech Blog.

We have had our mission-critical webapp running on AWS Elastic Beanstalk for three years and have been extremely happy with it. However we have now outgrown it and move to a manually managed infrastructure and CodeDeploy.

AWS Beanstalk provides you with lot of bang for the buck and enables you to get up and running in no time:

• Simple, no-downtime deployment and automatic roll-back based on user-provided health-check (either one subset of nodes at a time or blue-green deployment)
• Autoscaling
• Managed updates – security fixes and other improvements installed automatically
• Built-in HTTP Proxy with caching in front of your application
• Monitoring dashboard with alerting and access to logs without the need for SSH
• A list of past versions & ability to roll-back
• Support for many runtimes (Java, Node.js, Docker to name just a few)

So if you need a solid, state-of-the-art infrastructure for a web-scale application and you don’t have lot of time and/or skill to build one on AWS on your own, I absolutely recommend Beanstalk.

Advertisements

Pains with Terraform (perhaps use Sceptre next time?)

Cross-posted from Telia’s Tech Blog

We use Amazon Web Services (AWS) heavily and are in the process of migrating towards infrastructure-as-code, i.e. creating a textual description of the desired infrastructure in a Domain-Specific Language and letting the tool create and update the infrastructure.

We are lucky enough to have some of the leading Terraform experts in our organisation so they lay out the path and we follow it. We are at an initial stage and everything is thus “work in progress” and far from perfect, therefore it is important to judge leniently. Yet I think I have gain enough experience trying to apply Terraform both now and in the past to speak about some of the (current?) limitations and disadvantages and to consider alternatives.

Nginx: Protecting upstream from overload on cache miss

These 2 magical lines will protect your upstream server from possible overload of many users try to access the same in cached or expired content: 

proxy_cache_use_stale updating timeout; # Serve the cached version even when outdated while refreshing it
proxy_cache_lock on; # Only one req is allowed to load/refresh the item, others wait / get the stale one 

You can verify this using Shopify’s Toxiproxy. 

❤ Nginx

Running Gor, the HTTP traffic replayer, as a service on AWS Elastic Beanstalk

Gor is a great utility for replicating (a subset of) production traffic to a staging/test environment. Running it on AWS Elastic Beanstalk (EB) has some challenges, mainly that it doesn’t support running as a daemon and that there isn’t any documentation/examples for doing this. Well, here is a solution:

AWS ebextensions: Avoiding “Could not enable service” (or .. disable ..)

If you are adding a service entry to your .ebextensions/ config to run a service in AWS Elastic Beanstalk and it fails with either “Could not enable service [..]” or “Could not disable service [..]” (based on the value of ensureRunning), make sure that the service init.d file supports chkconfig, i.e. contains the comments it looks for.

Fixing a mysterious .ebextensions command time out (AWS Elastic Beanstalk)

Our webshop, nettbutikk.netcom.no, runs on AWS Elastic Beanstalk and we use .ebextensions/ to customize the environment. I have been just trying to get Gor running on our leader production instance to replay some traffic to our staging environment so that we get a much richer feedback from it. However the container_command I used caused the instance to time out and trash the environment, against all reason. The documentation doesn’t help and troubleshooting this is hard due to lack of feedback and time-consuming. Luckily I have arrived to a solution.

AWS: Passing private configuration to a Docker container (via S3)

Philipp Garbe describes how to pass environment variables that you want to keep private to a public Docker instance run on Amazon Web Services (beanstalk or ECS) in his post How to Run HuBot in Docker on AWS EC2 Container Services – Part 3. The trick is:

1. Put them into an env.sh file that you can source on S3 (and allow the appropriate EC2 IAM role to access it)
2. As a part of your startup CMD, run aws s3 cp to fetch and then source it

Here is his example of the CMD from a Dockerfile:

CMD ["/bin/sh", "-c", "aws s3 cp --region eu-west-1 s3://your-bucket/env.sh .; . ./env.sh; bin/hubot --adapter slack"]

See the full source code in his GitHub repo. Thanks for sharing, Phillipp!

Book Review & Digest: Release It! Design and Deploy Production-Ready Software

By Michael T. Nygard, 2007, ISBN: 978-0-9787-3921-8

My digest and review of the book.

Review

Of the books I have read, Release It! is the one I would require all “senior” developers to read (together with something like Architecting Enterprise Solutions: Patterns for High-Capability Internet-based Systems). Especially the first part on stability with its patterns and anti-patterns is a must read. Without knowing and applying them, we create systems that react to problems like a dry savannah to a burning match. I found also to next to last chapter, #17 Transparency, very valuable, especially the metrics and design of the OpsDB and observation practices.

One thing I have left out of the digest which is really worth reading are the war stories that introduce each section, they are really interesting, inspiring, and educational.

Extra Links

• Release It! slides: http://gotocon.com/dl/jaoo-sydney-2009/slides/MichaelT.Nygard_FailureComesInFlavoursPart2.pdf
• https://github.com/Netflix/Hystrix/
• Netflix’s Dependency Command talks about using circuit breakers and a thread pool limit http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html
• MSDN Guidance for Cloud Applications: Design Patterns – Circuit Breaker and many more highly useful patterns

Stability

Stability x longevity bugs

Selected (anti)patterns

Stability antipatterns

Integration points

Integration point = call to a DB, WS, … . Stability risk #1.

AWS API: Proper syntax for filtering by tag name and value (e.g. describeInstances)

It took me quite a while to figure out the right syntax for filtering instances by tag name and value in the AWS EC2 API’s describeInstances.

The documentation is not exactly crystal-clear to me:

• tag:key=value – The key/value combination of a tag assigned to the resource, where tag:key is the tag’s key.

Anyway, here is the proper syntax, provided we are interested in the tag elasticbeanstalk:environment-name:

    var params = {
        Filters: [
            {
                Name: 'tag:elasticbeanstalk:environment-name',
                Values: ['mySuperApp']
            }
        ]
    };
    ec2.describeInstances(params);

So the name of the tag is embedded in the Name part and not, as I initially understood,
{ Name: 'tag', Values: ['elasticbeanstalk:environment-name=mySuperApp'] }

Credit: garnaat.