AmazonCloudWatchAsyncClient setting maximum concurrent HTTP connections / throttling

I was looking at ways to throttle the Amazon CloudWatch Async Client from making a lot of concurrent connections simultaneously as we in our company monitor AWS system of lot of customers which means the number of metrics being fetched reach thousands easily. which leads to network throttling/packet drops/rejection of requests by AWS.

Turns out there is a way to do this which was not apparent at first as I was looking at the API of  AmazonCloudWatchAsyncClient. It is present as a property of ClientConfiguration class and the way to use it is as follows.

AWSCredentials credentials = new BasicAWSCredentials(obj.getString("AWS_ACCESS_KEY"),obj.getString("AWS_SECRET_KEY")); 
ClientConfiguration config = new ClientConfiguration(); 
config.setMaxConnections(1); // This is done to create fixed number of connections per client
AmazonCloudWatchAsyncClient client = new AmazonCloudWatchAsyncClient(credentials);
client.setConfiguration(config);

Varnish infinite redirect loop for naked domain redirect to www

You might face an infinite redirect loop with varnish if your backend server does a redirect for a naked domain or for that matter any domain.

Problem is that varnish by default uses a combination of hostname  and url  to create a cache key. This becomes a problem when your varnish server's hostname is not the same as the served domain (which is the case most of the times).

This leads to varnish caching a 301/302 redirect in itself. So if I hit example.com and my backend throws a 301 to www.example.com the key generated in varnish would be hostname_url and not HOST_url. So now if I hit www.example.com I am provided a 302 to the same url as lookup does not take into account the HOST which is different for example.com and www.example.com. To overcome this update the VCL to include HOST in the cache key. I have added req.http.X-Forwarded-Prot also to the hash to overcome https redirect that we face with ELB and varnish when the SSL terminates at ELB and the backend has been coded to do a redirect from non secure url (http) to secure urls (https).

sub vcl_hash {
    hash_data(req.http.host);
    hash_data(req.url);
    hash_data(req.http.X-Forwarded-Proto);
    return(hash);
}

If you are facing infinite redirect loop for https pages behind ELB read this.

Why NFS for code sharing is a bad idea on production machines

One of our customers was facing slowness on their website at peak loads. The architecture looked something like this:

1) LAMP stack.

2) A single NFS server hosting all the static content as well as PHP code.

3) 15 application servers hosted behind a load balancer and the NFS server mounted on all of these.

When we started debugging we found that the CPU load on the app server was never high even at peak loads. But the CPU load on NFS server would be very high at those times.

So we suspected NFS to be an issue but were not very sure because we were using APC with PHP and apc.stat was 0 which means that if the opcode cache is present in the APC, Apache would not do a look up in the file system for that file. If the above is true then once the APC opcode cache is warmed up (at peak loads it should be), then why are we seeing slowness in the site and high CPU load on NFS server at peak loads.

We used a linux utility called strace to trace all the system calls that were being made by  apache processes. We attached strace to one of the apache processes and found that it was doing hell lot of stat and lstat which are Linux system calls to find out if a file has changed or not. Which means that even after making apc.stat =0 the system was still doing lookup for PHP files. Strange.

Turns out it has been clearly mentioned in the APC documentation that APC does a look up for files irrespective of stat status if the file has been included with a relative path (and not absolute path). Most of the includes in the code were relative, which means apc.stat=0 did not help us :( .

Even if the look ups are happening isn't NFS supposed to cache the files at each client ?

Turns out NFS does not cache the file rather caches just the file meta-data (that too for 3 secs by default, which can be changed). The reason for caching the meta-data (called file attribute cache) is performance so that client does not need to make frequent network calls to just do a stat or meta-data lookup. The reason this cache has finite time period is to avoid staleness which could have disastrous effects in a shared environment. There are ways of caching the files too using fscache but is not recommended in a dynamic environment.

So the lessons we learnt are these:

1) Never share code on NFS , never ever, ever. 

2) Use NFS for just sharing the static content.

3) Never ever write to a file shared over NFS. For eg many applications have debug logs. If this log is shared then you can imagine how many network calls need to be made to write logs in the request scope.

After doing the above the response time of the application at peak loads reduced by 10X and down time became history. We were able to run the site with half the number of app servers.

The problem with shared code is that, the load eventually  goes down to the NFS and the app servers just act as dumb terminals. At peak loads you cannot add more servers as it would slow down the environment even more.  Its like putting another straw in a coke bottle which already had 10 straws drawing form it.

AWS JAVA client examples for ELB and RDS metrics (Asynchronous)

When I started working on the AWS JAVA client for fetching the ELB and RDS metrics, I could not find many examples on the internet. So here I am putting down some sample code snippets.

So first you need to create a Client object

AWSCredentials credentials = new BasicAWSCredentials(AWS_ACCESS_KEY,AWS_SECRET_KEY);
AmazonCloudWatchAsyncClient client = new AmazonCloudWatchAsyncClient(credentials);

ELB:

Now creating an init object . Note that we are getting an aggregation of 5 mins. Which means that we are asking for one data point every five minutes and since time range is 5 mins, we would get exactly one data point every time we invoke this method

private static GetMetricStatisticsRequest initializeRequestObject(AmazonCloudWatchAsyncClient client){ 
        GetMetricStatisticsRequest request   = new GetMetricStatisticsRequest();
         
        request.setPeriod(60*5); // 5 minutes 
         
        request.setNamespace("AWS/ELB"); 
         
        List<Dimension> dims      = new ArrayList<Dimension>(); 
        Dimension dim       = new Dimension(); 
        dim.setName("LoadBalancerName");  
        dim.setValue("NAME");  
        dims.add(dim); 
         
        Date end = new Date(); 
        request.setEndTime(end); 

        // Back up 5 minutes 
        Date beg = new Date(end.getTime() - 5*60*1000); 
        request.setStartTime(beg); 
        request.setDimensions(dims); 
        return request; 
    } 

Now fetching data for individual metrics:

ELB 4XX:

public static void get5MinELB4XX(AmazonCloudWatchAsyncClient client){ 
        client.setEndpoint("monitoring.ap-southeast-1.amazonaws.com"); // endpoints are listed at http://docs.aws.amazon.com/general/latest/gr/rande.html
        GetMetricStatisticsRequest request = initializeRequestObject(client); 
         
        request.setMetricName("HTTPCode_ELB_4XX"); 
        request.setUnit(StandardUnit.Count); 
         
        List<String> stats  = new ArrayList<String>(); 
        stats.add("Sum"); 
        request.setStatistics(stats); 
         
        client.getMetricStatisticsAsync(request, new AsyncHandler<GetMetricStatisticsRequest, GetMetricStatisticsResult>() { 
             
            @Override 
            public void onSuccess(GetMetricStatisticsRequest arg0,
                    GetMetricStatisticsResult arg1) { 
                List<Datapoint> data = arg1.getDatapoints(); 
                // Do something with this data here 
            } 
             
            @Override 
            public void onError(Exception arg0) {
                // log an error 
            } 
        }); 
        return; 
         
    }

ELB Latency:

public static void get5MinLatency(AmazonCloudWatchAsyncClient client){ 
        client.setEndpoint("monitoring.ap-southeast-1.amazonaws.com"); // endpoints are listed at http://docs.aws.amazon.com/general/latest/gr/rande.html
        GetMetricStatisticsRequest request = initializeRequestObject(client); 
         
        request.setMetricName("Latency"); 
        request.setUnit(StandardUnit.Seconds); 
         
        List<String> stats     = new ArrayList<String>(); 
        stats.add("Average"); 
        stats.add("Maximum"); 
        stats.add("Minimum"); 
        request.setStatistics(stats); 
         
        client.getMetricStatisticsAsync(request, new AsyncHandler<GetMetricStatisticsRequest, GetMetricStatisticsResult>() { 
             
            @Override 
            public void onSuccess(GetMetricStatisticsRequest arg0,
                    GetMetricStatisticsResult arg1) { 
                List<Datapoint> data = arg1.getDatapoints(); 
                // Do something with this data here  
            } 
             
            @Override 
            public void onError(Exception arg0) {
                 // log an error  
            } 
        }); 
        return; 
    }

ELB Request Count:

public static void get5MinRequestCount(AmazonCloudWatchAsyncClient client){ 
        client.setEndpoint("monitoring.ap-southeast-1.amazonaws.com"); // endpoints are listed at http://docs.aws.amazon.com/general/latest/gr/rande.html
        GetMetricStatisticsRequest request = initializeRequestObject(client); 
         
        request.setMetricName("RequestCount"); 
        request.setUnit(StandardUnit.Count); 
         
        List<String> stats = new ArrayList<String>(); 
        stats.add("Sum"); 
        request.setStatistics(stats); 
         
        client.getMetricStatisticsAsync(request, new AsyncHandler<GetMetricStatisticsRequest, GetMetricStatisticsResult>() { 
             
            @Override 
            public void onSuccess(GetMetricStatisticsRequest arg0,
                    GetMetricStatisticsResult arg1) { 
                List<Datapoint> data = arg1.getDatapoints(); 
                // Do something with this data here  
                 
            } 
             
            @Override 
            public void onError(Exception arg0) {
                // log an error  
            } 
        }); 
        return; 
    }

RDS:

CPU Utilization:

public static void get5MinCPUUtilization(AmazonCloudWatchAsyncClient client){ 
        client.setEndpoint("monitoring.ap-southeast-1.amazonaws.com"); // endpoints are listed at http://docs.aws.amazon.com/general/latest/gr/rande.html
        GetMetricStatisticsRequest request = initializeRequestObject(client); 
         
        request.setMetricName("CPUUtilization"); 
        request.setUnit(StandardUnit.Percent); 
         
        List<String> stats     = new ArrayList<String>(); 
        stats.add("Average"); 
        stats.add("Maximum"); 
        stats.add("Minimum"); 
        request.setStatistics(stats); 
         
        client.getMetricStatisticsAsync(request, new AsyncHandler<GetMetricStatisticsRequest, GetMetricStatisticsResult>() { 
             
            @Override 
            public void onSuccess(GetMetricStatisticsRequest arg0,
                    GetMetricStatisticsResult arg1) { 
                List<Datapoint> data = arg1.getDatapoints(); 
                // Do something with this data here  
            } 
             
            @Override 
            public void onError(Exception arg0) {
                // log an error 
            } 
        }); 
        return; 
    }

Similarly for other metrics of RDS you just need to chose the correct metric name and its unit.

Building Real Time Analytics Engine (counters) with Redis

We were assigned a task of building a REAL TIME analytics API . The data to be analyzed were simple counters for multiple events that happen across the website. For eg how many times a particular listing appeared in the search results, how many times the listing page was viewed, how many times Click to Reveal was pressed, etc.

The problem is simple when you have a finite set of listing. But in our case we had around a million listings and each listing had around 5 events and we had to keep data for each listing for an year.

The first approach would be to use a NoSQL database like Mongodb. To be honest, mongodb for just maintaining the counters seemed like an over kill. Other reasons for not using Mongodb were:

1. The write throughput would be a cause for concern given the collection level lock in Mongodb.
2. No support for transactions. It becomes very import in case of an analytics API, as we cannot afford to update one counter and then fail to update the other.
3. No support for atomic operation on a document. So there was no way to synchronize the counter increments or decrements.
4. No native support for incrementing/decrementing counters.

So we decided to look for something else and came across Redis (kidding, I always wanted to use Redis, was just looking for an excuse :) ).

The features that tipped the scale in Redis' favor were.

1. Its crazy fast as the whole db is in memory.
2. Transaction are supported.
3. Native support for atomic operations.
4. Native support for incrementing/decrementing counters.

Two very nice libraries exist for redis analytics namely bitmapist and Minuteman . These make use of Redis bit operations which are really awesome and helps us fit humongous data into a few MBs of memory. But these did not work for us as they work for non repeating data (i.e an event for a user is captured only once). Its binary in nature which does not fit with our counters requirement.

So we implemented our own solution and here are some of my learning from the same.

1) Design for failure: You cannot afford to have an analytics API down. So have a file backed fallback in case of datastore (Redis) being down. Have multiple app servers behind a load balancer which helps in making hot releases without bringing the site down. Write a script which replays these logs when your datastore is up.

2) All non counter data is written to a file. We did not know what to do with this additional data, so just persisted it in a file and pushed it periodically to AWS S3.

3)  Enable AOF for Redis with a flush frequency of 1 sec. This is a must if you dont want to lose data on reboot of Redis :). The AOF file should be periodically pushed to S3 for higher durability.

4) Have RDB snaphots atleats twice a day and push those to S3.

5) When the data size becomes few GBs Redis takes a few minutes (yes minutes) to start as it runs the whole AOF file on startup. The AOF file is nothing but a list of all the DMLs that have happened and it does so serially. Redis would throw an exception until this AOF has been replayed completely. This is the reason why we need a file backed fallback when something like this happens.

6) Use Redis Hashes as they are superbly optimized for space (with high time complexity but this trade off doesnt hurt much).

7) Redis is single threaded. So adding more CPUs is of no use. Throw as much memory as you can at it.

8) No redis library supports transactions (they cannot actually) when you run Redis cluster as its not possible to have a transaction across multiple datastores. Redis cluster works the same way as memcache (i.e data is sharded on key hashes).

9) Needless to say use the smallest possible key:value pairs as even a single "_" in 1M keys adds a couple of hundred MBs to memory footprint.

The request flow of our application is something like this:

Hope this helps someone to make a decision about using Redis for analytics.

   

HTTPS redirection with ELB and Varnish (infinite redirect loop)

Yesterday I was trying to deploy Varnish cache in front of my app server and was faced with the issue of infinite redirect loop for some pages. This problem happened when I tried to use HTTPS for pages which are not supposed to be.

The problem is like this:

I am terminating SSL at the ELB, behind that is the Varnish and behind that is the app server. Its a standard practice that whenever we hit the app server with HTTPS scheme for a page which is not supposed to be HTTPS it throws a 302 with the same page url , only difference being HTTP instead of HTTPS.

With SSL termination at ELB we can get the original scheme using a header which ELB sets called X-Forwarded-Proto . So my app was making use of this and doing a redirection from HTTPS -> HTTP. Everything was good.

The only problem was the Varnish cache caching 302. So this is what happened:

1) User hit https://example.com/myurl.
2) Varnish sends request to the backend.
3) Backend send a 302 redirect to http://example.com/myurl.
4) The response (302) gets cached in varnish

Now the problem is since scheme is not a part of the varnish hash key so effectively the mapping inside varnish is:

myurl : 302  irrespective of the scheme (https/http). So from now even if the user hit http://example.com/myurl he would always get a 302 to the same url. Infinite redirect loop.

Solution is very simple just make scheme part of the vcl_hash like this:

sub vcl_hash {
hash_data(req.url);
hash_data(req.http.X-Forwarded-Proto);
return(hash);
}


Here scheme (X-Forwarded-Proto) has been made a part of the key. So the internal mapping in varnish would like like:

myurl_http : Actual content
myurl:https: 302 redirect

Problem solved.

If you are facing infinite redirect loop for naked domain redirects (example.com -> www.example.com) or otherwise, read this.  

Multi bit rate video streaming with Cloudfront and Flow player

Sometime back I was trying to get multi-bit rate streaming working with Cloudfront, but could not find a working sample online. The reason why you need this is that all your users do not have the same bandwidth and even for those with high bandwidth the ACTUAL bandwidth might fluctuate as your packets have to travel over the internet which your ISP  does not control.

The way this works with Cloudfront is that you need to have the same video encoded at different bit rates present in your S3 bucket. Flowplayer (or any other player supporting multi bit rate), keeps switching between the different bit rate videos depending upon the ACTUAL bandwidth. The users would definitely feel the quality of video changing but that is better than showing the dreaded "Buffering" wheel :)

So first thing that you need is your video encoded at different bit rates. I used ffmpeg to do the same, but you are free to try something else.

 ffmpeg -i <input_file> -b <bit_rate> -r 30 -ab 384k -ar 44100 -s 624x256 -ac 2 -y sample_2200k.flv

bit_rate : This is the target bit rate, example : 2200k
output_file : This is the output file name, example : sample_2200k.flv

In this example lets create two files one with 200 kbps and one with 1000 kbps bit rate. Lets name them multi_bit_rate_200k.flv and multi_bit_rate_1000k.flv .

Now create a bucket (lets call it videostreamingakash) in S3 and put both the files there.

We now need to create a streaming distribution for serving these video. Go here for steps to create the same. While creating the distribution pick the bucket that we had created earlier as the origin.

Now we need to have an HTML file with the following content:

<html>
<body>
<div class="box black"><a
    href=""
    style="display:block;width:640px;height:360px;"
    id="player">
</a></body>
</html>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js" type="text/javascript" charset="utf-8"></script>
<script src="flowplayer-3.2.11.min.js"></script>
<script>
flowplayer("player", "http://releases.flowplayer.org/swf/flowplayer-3.2.16.swf", {
    clip: {

       urlResolvers: 'bwcheck',
       provider: 'rtmp',
       autoPlay: false,
       scaling: 'fit',

       // available bitrates and the corresponding files. We specify also the video width
       // here, so that the player does not use a too large file. It switches to a
       // file/stream with larger dimensions when going fullscreen if the available bandwidth permits.
       bitrates: [
         {
            url: "flv:multi_bit_rate_200k", width: 720, bitrate: 200,
            // this is the default bitrate, the playback kicks off with this and after that
            // Quality Of Service monitoring adjusts to the most appropriate bitrate
            isDefault: true
         },
         { url: "flv:multi_bit_rate_1000k", width: 720, bitrate: 1000 }
       ]
    },
    plugins: {

        // bandwidth check plugin
        bwcheck: {
            url: "flowplayer.bwcheck-3.2.12.swf",

            // CloudFront uses Adobe FMS servers
            serverType: 'fms',

            // we use dynamic switching, the appropriate bitrate is switched on the fly
            dynamic: true,

            netConnectionUrl: 'rtmp://s3ht5aoild7wia.cloudfront.net/cfx/st',

            // show the selected file in the content box. This is not used in real installations.
            onStreamSwitchBegin: function (newItem, currentItem) {alert("here");
               $f().getPlugin('content').setHtml("Will switch to: " + newItem.streamName +
                " from " + currentItem.streamName);
   $('#stream_data').text("Will switch to: " + newItem.streamName +
                " from " + currentItem.streamName);
            },
            onStreamSwitch: function (newItem) {
               $f().getPlugin('content').setHtml("Switched to: " + newItem.streamName);
               $('#stream_data').text("Switched to: " + newItem.streamName);
            }
        },

        // RTMP streaming plugin
        rtmp: {
            url: "http://releases.flowplayer.org/swf/flowplayer.rtmp-3.2.12.swf",
            netConnectionUrl: 'rtmp://s3ht5aoild7wia.cloudfront.net/cfx/st'        },

        // a content box so that we can see the selected video dimensions. This is not used in real
        // installations.
        content: {
            url: "http://releases.flowplayer.org/swf/flowplayer.content-3.2.8.swf",
            top: 0, left: 0, width: 250, height: 150,
            backgroundColor: 'transparent', backgroundGradient: 'none', border: 0,
            textDecoration: 'outline',
            style: {
                body: {
                    fontSize: 14,
                    fontFamily: 'Arial',
                    textAlign: 'center',
                    color: '#ffffff'
                }
            }
        }
    }
});
</script> 

Replace  netConnectionUrl with you STREAMING distribution. Also you need to download flowplayer-3.2.11.min.js from the flow player website to make the above work.

If you look at  the block:

  bitrates: [

         {

            url: "flv:multi_bit_rate_200k", width: 720, bitrate: 200,

            // this is the default bitrate, the playback kicks off with this and after that

            // Quality Of Service monitoring adjusts to the most appropriate bitrate

            isDefault: true

         },

         { url: "flv:multi_bit_rate_1000k", width: 720, bitrate: 1000 }

       

We have declared the two videos with their bitrates. Flowplayer uses this for switching the streams.

Just open the HTML in the browser of your choice and you are done :) .