Jaspersoft with AWS Redshift Experince, Learnings and Problems

This was my first experience with AWS Redshift and Jaspersoft. Both the technologies are good and easy to use but can sometimes throw up issues which are difficult to decode/fix . Here are some of the things that I faced/discovered using both.

Redshift:

1. Has a decent performance when the queries that you are making do not have joins in them. For eg if there is one table which has 1 Bn rows and another which has just 50k rows and you do a join on a column which is not a sort key, then the query may take anywhere between 5-10min which for me is a no go for a dashboard (a user would not wait for 5 mins for a chart to load). This is because Redshift is very disk intensive and the caching of blocks in memory in Redshift is still not up to the mark so it has to read a lot of blocks on disk for every query.
2. Always try to denormalize data as much as possible. Joins are a crime. It will suck the life out of you and the query.
3. Concurrency is not upto the mark in redshift. The performance goes down exponentially with every new parallel execution.
4. A LIKE match would almost always make the query slow (no surprise here as even an OLTP DB would do the same). But the degradation is significant.
5. We have a query which when run makes the whole cluster unresponsive :) . We still dont know the reason but it happens. Again no clue why.
6. We used to run VACCUM everyday after importing data into the cluster, but on numerous occasion we saw that the whole cluster would hang because VACCUM stalled on one of the nodes. One has to do a cluster reboot to fix this. Also the command does not time out so it may hang your cluster (READ/WRITE) for almost a day if not rebooted. So we moved this command to just once a week so that we do not hit this problem that often.
7. Always make use of the SORT key in your queries. It works like a charm.
8. Build aggregates for the charts as time taken by queries is not predictable when run on huge tables. We build aggregates for all the charts and dont run queries on big tables at run time.

Jaspersoft:
It also has its fair share of issues but most of them are due to Redshift :). The following is for HTML 5 reports which we built using the web wizard and NOT through report designer.

1. CREATE VIEW may throw an error or may not load at all when redshift is having performance issues or there is a lot of load on redshift, still dont know the reason why but it happens.
2. If you change the chart type in VIEW , it may not reflect in the REPORT. Recreate the report.
3. There is no support for having a single axis chart with multiple measures with tooltip showing all the measures together. I found a work around and is present here
4. There is no API to clear the Query Cache. I tried a lot of command line tools like phantomjs to script it but it did not work as every page has an executionKey and jaspersoft was smart enough to know that it was a scripted attempt :( . I had to write a selenium test case to do the same. As the cache is in memory and NOT DB I guess one would have to modify the JAVA code. It uses EH Cache for caching the queries.
5. One Good thing about jaspersoft is that even if you leave a report without it completely loading the query still works in the background and populates the cache. This helped me in automating cache warming by simulating user clicks (selenium) without waiting for the page to load.
6. There is no eager/ preemptive caching of queries and is on demand. So one has to write a selenium test case to warm up the cache. This is a much needed thing for redshift which is not good with concurrent queries.

Hope this will help somebody.

Jaspersoft Shared/Common Tooltip for non mutli-axis/single axis graphs (HTML5)

We were facing this problem wherein we needed four measures to be shown on the same graph (line graph). By default jaspersoft has two chart types, single axis and multi axis. The single axis graph shows only one y axis and one has to hover/move from one measure to the other to view the data in the tooltip.

This was very inconvenient for the end users as they were not able to see all the measures in a single tooltip for the same point on the x axis (date in our case) making it difficult for them to compare the data for all the measures on the graph.

We tried using multi axis chart which by default has a shared tooltip, but shows myltiple y axis which we did not need  as it was useless for us as  one, all measures were comparable to each other and two, it gave  a very wrong impression to the end user as a graph with a very low value could appear over a graph with a very high values due to difference in scales. This created a lot of confusion.

We tried all the forums and blogs but did not get an answer so decided to get our hands dirty. Turns out the fix/change is very easy (it took me 3 days though).

For those who do not know jaspersoft uses highcharts as the charting library for dynamic (HTML5) charts. The highcharts API has a property called "shared" for "tooltip". If we enable this the tooltip becomes shared. So we just needed to find the js file where it was being set and we found it :) .

Go to file scripts/adhoc/highchart.datamapper.js, method name "getCommonSeriesGeneralOptions", line number 342 and change

options.tooltip.shared = HDM.isDualOrMultiAxisChart(extraOptions.chartState.chartType);
to
options.tooltip.shared = true;

Beware, this will make the tooltip shared for all the graphs. If you want more granularity then u need to create another method similar to  HDM.isDualOrMultiAxisChart() and return true or false accordingly.

AWS JAVA client examples for Auto Scaling metrics (Asynchronous)

Below are few code snippets for gathering Auto Scaling metrics from CloudWatch using the AWS Java Async Client (AmazonCloudWatchAsyncClient). Its very similar to the other code snippets I have shared. The only thing that took me almost a day to discover was the namespace which according to the documentation should be "AWS/AutoScaling" but what actually worked for me was "AWS/EC2"

:(

As always first create the client:

AWSCredentials credentials = new BasicAWSCredentials(obj.getString("AWS_ACCESS_KEY"),obj.getString("AWS_SECRET_KEY")); 
ClientConfiguration config = new ClientConfiguration(); 
config.setMaxConnections(1); // This is done to create fixed number of connections per client
AmazonCloudWatchAsyncClient client = new AmazonCloudWatchAsyncClient(credentials);
client.setConfiguration(config);

Now a utility method to initialize the request object:

private static GetMetricStatisticsRequest initializeRequestObject(AmazonCloudWatchAsyncClient client,JSONObject groupDetails){ 
    GetMetricStatisticsRequest request   = new GetMetricStatisticsRequest();
     
    request.setPeriod(60*5); // 5 minutes 
     
    request.setNamespace("AWS/EC2"); 
         
    List<Dimension> dims  = new ArrayList<Dimension>(); 
    Dimension dim  = new Dimension(); 
    dim.setName("AutoScalingGroupName"); 
    dim.setValue(groupDetails.getString("NAME")); 
    dims.add(dim); 
     
    Date end = new Date(); 
    request.setEndTime(end); 
    // Back up 5 minutes 
    Date beg = new Date(end.getTime() - 10*60*1000); 
    request.setStartTime(beg); 
    request.setDimensions(dims); 
    return request; 
}

Lets gather some metrics now:

    public static void get5MinCPUUtilization(AmazonCloudWatchAsyncClient client, final JSONObject groupDetails, final String clientName){ 
        client.setEndpoint(groupDetails.getString("END_POINT")); 
        GetMetricStatisticsRequest request = initializeRequestObject(client, groupDetails); 
         
        request.setMetricName("CPUUtilization"); 
        request.setUnit(StandardUnit.Percent); 
         
        List<String> stats = new ArrayList<String>(); 
        stats.add("Average"); 
        stats.add("Maximum"); 
        stats.add("Minimum"); 
        request.setStatistics(stats); 
         
        client.getMetricStatisticsAsync(request, new AsyncHandler<GetMetricStatisticsRequest, GetMetricStatisticsResult>() { 
             
            @Override 
            public void onSuccess(GetMetricStatisticsRequest arg0,
                    GetMetricStatisticsResult arg1) { 
                List<Datapoint> data = arg1.getDatapoints(); 
                Double avg = data.size() > 0 ? data.get(0).getAverage() : 0.0; 
                Double min = data.size() > 0 ? data.get(0).getMinimum() : 0.0; 
                Double max = data.size() > 0 ? data.get(0).getMaximum() : 0.0; 
                 
            } 
             
            @Override 
            public void onError(Exception arg0) {
                 
            } 
        }); 
        return; 
    } 
     
    public static void get5MinDiskReadOps(AmazonCloudWatchAsyncClient client, final JSONObject groupDetails, final String clientName){ 
        client.setEndpoint(groupDetails.getString("END_POINT")); 
        GetMetricStatisticsRequest request = initializeRequestObject(client, groupDetails); 
         
        request.setMetricName("DiskReadOps"); 
        request.setUnit(StandardUnit.Count); 
         
        List<String> stats = new ArrayList<String>(); 
        stats.add("Average"); 
        stats.add("Maximum"); 
        stats.add("Minimum"); 
        request.setStatistics(stats); 
         
        client.getMetricStatisticsAsync(request, new AsyncHandler<GetMetricStatisticsRequest, GetMetricStatisticsResult>() { 
             
            @Override 
            public void onSuccess(GetMetricStatisticsRequest arg0,
                    GetMetricStatisticsResult arg1) { 
                List<Datapoint> data = arg1.getDatapoints(); 
                Double avg = data.size() > 0 ? data.get(0).getAverage() : 0.0; 
                Double min = data.size() > 0 ? data.get(0).getMinimum() : 0.0; 
                Double max = data.size() > 0 ? data.get(0).getMaximum() : 0.0; 
                 
            } 
             
            @Override 
            public void onError(Exception arg0) {
            } 
        }); 
        return; 
    } 
     
    public static void get5MinStatusCheckFailed(AmazonCloudWatchAsyncClient client, final JSONObject groupDetails, final String clientName){ 
        client.setEndpoint(groupDetails.getString("END_POINT")); 
        GetMetricStatisticsRequest request = initializeRequestObject(client, groupDetails); 
         
        request.setMetricName("StatusCheckFailed"); 
        request.setUnit(StandardUnit.Count); 
         
        List<String> stats = new ArrayList<String>(); 
        stats.add("Average"); 
        stats.add("Maximum"); 
        stats.add("Minimum"); 
        request.setStatistics(stats); 
         
        client.getMetricStatisticsAsync(request, new AsyncHandler<GetMetricStatisticsRequest, GetMetricStatisticsResult>() { 
             
            @Override 
            public void onSuccess(GetMetricStatisticsRequest arg0,
                    GetMetricStatisticsResult arg1) { 
                List<Datapoint> data = arg1.getDatapoints(); 
                Double avg = data.size() > 0 ? data.get(0).getAverage() : 0.0; 
                Double min = data.size() > 0 ? data.get(0).getMinimum() : 0.0; 
                Double max = data.size() > 0 ? data.get(0).getMaximum() : 0.0; 
                 
            } 
             
            @Override 
            public void onError(Exception arg0) {
            } 
        }); 
        return; 
    } 
     
    public static void get5MinDiskWriteOps(AmazonCloudWatchAsyncClient client, final JSONObject groupDetails, final String clientName){ 
        client.setEndpoint(groupDetails.getString("END_POINT")); 
        GetMetricStatisticsRequest request = initializeRequestObject(client, groupDetails); 
         
        request.setMetricName("DiskWriteOps"); 
        request.setUnit(StandardUnit.Count); 
         
        List<String> stats = new ArrayList<String>(); 
        stats.add("Average"); 
        stats.add("Maximum"); 
        stats.add("Minimum"); 
        request.setStatistics(stats); 
         
        client.getMetricStatisticsAsync(request, new AsyncHandler<GetMetricStatisticsRequest, GetMetricStatisticsResult>() { 
             
            @Override 
            public void onSuccess(GetMetricStatisticsRequest arg0,
                    GetMetricStatisticsResult arg1) { 
                List<Datapoint> data = arg1.getDatapoints(); 
                Double avg = data.size() > 0 ? data.get(0).getAverage() : 0.0; 
                Double min = data.size() > 0 ? data.get(0).getMinimum() : 0.0; 
                Double max = data.size() > 0 ? data.get(0).getMaximum() : 0.0; 
                 
            } 
             
            @Override 
            public void onError(Exception arg0) {
            } 
        }); 
        return; 
    } 
     
    public static void get5MinNetworkOutBytes(AmazonCloudWatchAsyncClient client, final JSONObject groupDetails, final String clientName){ 
        client.setEndpoint(groupDetails.getString("END_POINT")); 
        GetMetricStatisticsRequest request = initializeRequestObject(client, groupDetails); 
         
        request.setMetricName("NetworkOut"); 
        request.setUnit(StandardUnit.Bytes); 
         
        List<String> stats = new ArrayList<String>(); 
        stats.add("Average"); 
        stats.add("Maximum"); 
        stats.add("Minimum"); 
        request.setStatistics(stats); 
         
        client.getMetricStatisticsAsync(request, new AsyncHandler<GetMetricStatisticsRequest, GetMetricStatisticsResult>() { 
             
            @Override 
            public void onSuccess(GetMetricStatisticsRequest arg0,
                    GetMetricStatisticsResult arg1) { 
                List<Datapoint> data = arg1.getDatapoints(); 
                Double avg = data.size() > 0 ? data.get(0).getAverage() : 0.0; 
                Double min = data.size() > 0 ? data.get(0).getMinimum() : 0.0; 
                Double max = data.size() > 0 ? data.get(0).getMaximum() : 0.0; 
                 
            } 
             
            @Override 
            public void onError(Exception arg0) {
            } 
        }); 
        return; 
    } 
     
    public static void get5MinNetworkInBytes(AmazonCloudWatchAsyncClient client, final JSONObject groupDetails, final String clientName){ 
        client.setEndpoint(groupDetails.getString("END_POINT")); 
        GetMetricStatisticsRequest request = initializeRequestObject(client, groupDetails); 
         
        request.setMetricName("NetworkIn"); 
        request.setUnit(StandardUnit.Bytes); 
         
        List<String> stats = new ArrayList<String>(); 
        stats.add("Average"); 
        stats.add("Maximum"); 
        stats.add("Minimum"); 
        request.setStatistics(stats); 
         
        client.getMetricStatisticsAsync(request, new AsyncHandler<GetMetricStatisticsRequest, GetMetricStatisticsResult>() { 
             
            @Override 
            public void onSuccess(GetMetricStatisticsRequest arg0,
                    GetMetricStatisticsResult arg1) { 
                List<Datapoint> data = arg1.getDatapoints(); 
                Double avg = data.size() > 0 ? data.get(0).getAverage() : 0.0; 
                Double min = data.size() > 0 ? data.get(0).getMinimum() : 0.0; 
                Double max = data.size() > 0 ? data.get(0).getMaximum() : 0.0; 
                 
            } 
             
            @Override 
            public void onError(Exception arg0) {
                log.error("Could not get Autoscaling data for " + groupDetails.getString("NAME") + " for client "+ clientName,arg0); 
                NotificationMail.sendMail("Could not get Autoscaling data for " + groupDetails.getString("NAME") + " for client "+ clientName, "AutoScaling data could not be read"); 
            } 
        }); 
        return; 
    } 

For some more examples (ELB and RDS metrics) go here

Cloud Based (AWS) Elastic Jmeter Load Testing Application (SWARM)

In this age of internet its imperative for any web based application to benchmark itself for high concurrency. As AWS Advanced Technology partners our work includes helping enterprises/start ups embrace AWS for their production as well as testing workloads. Few questions that people have are

1) Is AWS  scalable ? 

2) How many requests/min can an EC2 instance serve ? 

3) What instance class should I choose for my my application ?

4) How many instances should I chose for my application ?

5) Does Auto Scaling actually work ? 

Turns out there are no simple answers to these question as these are very subjective in nature and vary from one application to the other. The only way to test this is by doing a load test.

Jmeter is almost an industry standard for load testing. We can run a test with desired concurrency and duration and write our own test cases through the GUI provided with it. It can provide a summary in the form a RAW log file (JTL) or a table or a graph.

All this is good when you want to run a load test from one machine, but what if you want to run load test from multiple machines ? How would you aggregate the data across multiple machines ?

You must be wondering why would we need to run load test from multiple machines and why not from one machine only ? 

Some things that I have learnt from my experience are :

1) The test should always be run in a distributed nature. When running concurrent connections from a single machine one could easily reach the network/IO limit of a single machine which would add to response time which would not be correct.

2) Since Jmeter creates multiple concurrent threads, the more the threads more would be the CPU contention which would add to the response time incorrectly.

3) You cannot target requests/unit time for your load test as its a function of number of concurrent threads and the server response time.

4) You can simulate only concurrency with jmeter. For example if you select 100 threads then Jmeter would make sure that there are 100 concurrent requests at any given time. Also Jmeter reuses these threads for maximum performance.

5) When doing load testing for an application behind ELB make sure that either ELB is pre warmed (details here) or you use ramp up. Please note that this is required only when the concurrency you are testing for is very high (there are no numbers shared by AWS). To know whether you are reaching the limits of ELB look for ELB 5XX value in the cloudwatch for your ELB.

6) To know which part of your stack is the bottleneck, use a profiler. My favorite is New Relic. It has plugins for almost all softwares.

To try out our product please visit https://swarm.minjar.com/ . 

AmazonCloudWatchAsyncClient setting maximum concurrent HTTP connections / throttling

I was looking at ways to throttle the Amazon CloudWatch Async Client from making a lot of concurrent connections simultaneously as we in our company monitor AWS system of lot of customers which means the number of metrics being fetched reach thousands easily. which leads to network throttling/packet drops/rejection of requests by AWS.

Turns out there is a way to do this which was not apparent at first as I was looking at the API of  AmazonCloudWatchAsyncClient. It is present as a property of ClientConfiguration class and the way to use it is as follows.

AWSCredentials credentials = new BasicAWSCredentials(obj.getString("AWS_ACCESS_KEY"),obj.getString("AWS_SECRET_KEY")); 
ClientConfiguration config = new ClientConfiguration(); 
config.setMaxConnections(1); // This is done to create fixed number of connections per client
AmazonCloudWatchAsyncClient client = new AmazonCloudWatchAsyncClient(credentials);
client.setConfiguration(config);

Varnish infinite redirect loop for naked domain redirect to www

You might face an infinite redirect loop with varnish if your backend server does a redirect for a naked domain or for that matter any domain.

Problem is that varnish by default uses a combination of hostname  and url  to create a cache key. This becomes a problem when your varnish server's hostname is not the same as the served domain (which is the case most of the times).

This leads to varnish caching a 301/302 redirect in itself. So if I hit example.com and my backend throws a 301 to www.example.com the key generated in varnish would be hostname_url and not HOST_url. So now if I hit www.example.com I am provided a 302 to the same url as lookup does not take into account the HOST which is different for example.com and www.example.com. To overcome this update the VCL to include HOST in the cache key. I have added req.http.X-Forwarded-Prot also to the hash to overcome https redirect that we face with ELB and varnish when the SSL terminates at ELB and the backend has been coded to do a redirect from non secure url (http) to secure urls (https).

sub vcl_hash {
    hash_data(req.http.host);
    hash_data(req.url);
    hash_data(req.http.X-Forwarded-Proto);
    return(hash);
}

If you are facing infinite redirect loop for https pages behind ELB read this.

Why NFS for code sharing is a bad idea on production machines

One of our customers was facing slowness on their website at peak loads. The architecture looked something like this:

1) LAMP stack.

2) A single NFS server hosting all the static content as well as PHP code.

3) 15 application servers hosted behind a load balancer and the NFS server mounted on all of these.

When we started debugging we found that the CPU load on the app server was never high even at peak loads. But the CPU load on NFS server would be very high at those times.

So we suspected NFS to be an issue but were not very sure because we were using APC with PHP and apc.stat was 0 which means that if the opcode cache is present in the APC, Apache would not do a look up in the file system for that file. If the above is true then once the APC opcode cache is warmed up (at peak loads it should be), then why are we seeing slowness in the site and high CPU load on NFS server at peak loads.

We used a linux utility called strace to trace all the system calls that were being made by  apache processes. We attached strace to one of the apache processes and found that it was doing hell lot of stat and lstat which are Linux system calls to find out if a file has changed or not. Which means that even after making apc.stat =0 the system was still doing lookup for PHP files. Strange.

Turns out it has been clearly mentioned in the APC documentation that APC does a look up for files irrespective of stat status if the file has been included with a relative path (and not absolute path). Most of the includes in the code were relative, which means apc.stat=0 did not help us :( .

Even if the look ups are happening isn't NFS supposed to cache the files at each client ?

Turns out NFS does not cache the file rather caches just the file meta-data (that too for 3 secs by default, which can be changed). The reason for caching the meta-data (called file attribute cache) is performance so that client does not need to make frequent network calls to just do a stat or meta-data lookup. The reason this cache has finite time period is to avoid staleness which could have disastrous effects in a shared environment. There are ways of caching the files too using fscache but is not recommended in a dynamic environment.

So the lessons we learnt are these:

1) Never share code on NFS , never ever, ever. 

2) Use NFS for just sharing the static content.

3) Never ever write to a file shared over NFS. For eg many applications have debug logs. If this log is shared then you can imagine how many network calls need to be made to write logs in the request scope.

After doing the above the response time of the application at peak loads reduced by 10X and down time became history. We were able to run the site with half the number of app servers.

The problem with shared code is that, the load eventually  goes down to the NFS and the app servers just act as dumb terminals. At peak loads you cannot add more servers as it would slow down the environment even more.  Its like putting another straw in a coke bottle which already had 10 straws drawing form it.