Nginx: Protecting upstream from overload on cache miss

These 2 magical lines will protect your upstream server from possible overload of many users try to access the same in cached or expired content: 

proxy_cache_use_stale updating timeout; # Serve the cached version even when outdated while refreshing it
proxy_cache_lock on; # Only one req is allowed to load/refresh the item, others wait / get the stale one 

You can verify this using Shopify’s Toxiproxy. 

❤ Nginx

Running Gor, the HTTP traffic replayer, as a service on AWS Elastic Beanstalk

Gor is a great utility for replicating (a subset of) production traffic to a staging/test environment. Running it on AWS Elastic Beanstalk (EB) has some challenges, mainly that it doesn’t support running as a daemon and that there isn’t any documentation/examples for doing this. Well, here is a solution:

AWS ebextensions: Avoiding “Could not enable service” (or .. disable ..)

If you are adding a service entry to your .ebextensions/ config to run a service in AWS Elastic Beanstalk and it fails with either “Could not enable service [..]” or “Could not disable service [..]” (based on the value of ensureRunning), make sure that the service init.d file supports chkconfig, i.e. contains the comments it looks for.

Fixing a mysterious .ebextensions command time out (AWS Elastic Beanstalk)

Our webshop, nettbutikk.netcom.no, runs on AWS Elastic Beanstalk and we use .ebextensions/ to customize the environment. I have been just trying to get Gor running on our leader production instance to replay some traffic to our staging environment so that we get a much richer feedback from it. However the container_command I used caused the instance to time out and trash the environment, against all reason. The documentation doesn’t help and troubleshooting this is hard due to lack of feedback and time-consuming. Luckily I have arrived to a solution.

AWS: Passing private configuration to a Docker container (via S3)

Philipp Garbe describes how to pass environment variables that you want to keep private to a public Docker instance run on Amazon Web Services (beanstalk or ECS) in his post How to Run HuBot in Docker on AWS EC2 Container Services – Part 3. The trick is:

1. Put them into an env.sh file that you can source on S3 (and allow the appropriate EC2 IAM role to access it)
2. As a part of your startup CMD, run aws s3 cp to fetch and then source it

Here is his example of the CMD from a Dockerfile:

CMD ["/bin/sh", "-c", "aws s3 cp --region eu-west-1 s3://your-bucket/env.sh .; . ./env.sh; bin/hubot --adapter slack"]

See the full source code in his GitHub repo. Thanks for sharing, Phillipp!

Book Review & Digest: Release It! Design and Deploy Production-Ready Software

By Michael T. Nygard, 2007, ISBN: 978-0-9787-3921-8

My digest and review of the book.

Review

Of the books I have read, Release It! is the one I would require all “senior” developers to read (together with something like Architecting Enterprise Solutions: Patterns for High-Capability Internet-based Systems). Especially the first part on stability with its patterns and anti-patterns is a must read. Without knowing and applying them, we create systems that react to problems like a dry savannah to a burning match. I found also to next to last chapter, #17 Transparency, very valuable, especially the metrics and design of the OpsDB and observation practices.

One thing I have left out of the digest which is really worth reading are the war stories that introduce each section, they are really interesting, inspiring, and educational.

Extra Links

• Release It! slides: http://gotocon.com/dl/jaoo-sydney-2009/slides/MichaelT.Nygard_FailureComesInFlavoursPart2.pdf
• https://github.com/Netflix/Hystrix/
• Netflix’s Dependency Command talks about using circuit breakers and a thread pool limit http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html
• MSDN Guidance for Cloud Applications: Design Patterns – Circuit Breaker and many more highly useful patterns

Stability

Stability x longevity bugs

Selected (anti)patterns

Stability antipatterns

Integration points

Integration point = call to a DB, WS, … . Stability risk #1.

AWS API: Proper syntax for filtering by tag name and value (e.g. describeInstances)

It took me quite a while to figure out the right syntax for filtering instances by tag name and value in the AWS EC2 API’s describeInstances.

The documentation is not exactly crystal-clear to me:

• tag:key=value – The key/value combination of a tag assigned to the resource, where tag:key is the tag’s key.

Anyway, here is the proper syntax, provided we are interested in the tag elasticbeanstalk:environment-name:

    var params = {
        Filters: [
            {
                Name: 'tag:elasticbeanstalk:environment-name',
                Values: ['mySuperApp']
            }
        ]
    };
    ec2.describeInstances(params);

So the name of the tag is embedded in the Name part and not, as I initially understood,
{ Name: 'tag', Values: ['elasticbeanstalk:environment-name=mySuperApp'] }

Credit: garnaat.