Fixing a mysterious .ebextensions command time out (AWS Elastic Beanstalk)
Posted by Jakub Holý on July 29, 2015
Our webshop, nettbutikk.netcom.no, runs on AWS Elastic Beanstalk and we use
.ebextensions/ to customize the environment. I have been just trying to get Gor running on our leader production instance to replay some traffic to our staging environment so that we get a much richer feedback from it. However the
container_command I used caused the instance to time out and trash the environment, against all reason. The documentation doesn’t help and troubleshooting this is hard due to lack of feedback and time-consuming. Luckily I have arrived to a solution.
This is the working solution:
(The S3 private bucket access is a story of its own, requiring and addition of AWS::CloudFormation::Authentication and a change of the bucket policy.)
The key points are starting
gor in the background with
& and, the magic ingredient that took me so long to figure out, the redirection of the command’s output to
/dev/null (the redirection inside the script likely doesn’t need to be there with respect to this problem; I have it because I don’t want any output to accumulate on the disk).
I do not know if I need to redirect both stdin and stderr of
/opt/gor-in-background and why I need to do it but without it I got the infamous
[time N+1] INFO Command execution completed on all instances. Summary: [Successful: 1, TimedOut: 1].
[time N] WARN The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own): [i-1e35c2b3].
and the instance continued to time out even when I tried to re-deploy a working version and never managed to deliver logs.
Troubleshooting tip: Clone the target env a few times and use those to test changes multiple times and multiple changes in parallel to speed up the process.
If you (container) command leads to a time out, try to redirect its stdout and/or stderr to
Thanks to João Abrantes for the redirection idea!
Sorry, the comment form is closed at this time.