Complete idiot's guide to Debugging

Time and again I have learnt much to my displeasure that I'm not efficient while debugging. Most often than otherwise, I would just miss the point and keep 'searching' for the issues. Sitting back I found one fundamental flaw in my approach to debugging a problem that there was no structure in my method! Instead of narrowing the cause-effect scenarios I was simply 'searching for the needle in the hay stack'. This would obviously take more time than expected and would lead to sheer wastage of my resources.

Learning about the scientific method of experimentation I found that solution of problems too complex for common sense to solve are achieved by mixture of inductive and deductive techniques.

The inductive logic demands us to start observing the 'system' and arrive at general conclusions. Make a mental note of your observations and try finding explanations to them chances are you would hit few observations which are not clearly understood in the given context, some thing which is kind of a 'mystery' and you find it tough to explain it the first time, hold on and do not get distracted collect information. Having done that now try to break your thoughts and write them under following heads;

1. Problem statement: The biggest mistake to avoid here is to write too much, things which you are not sure of.. just an assumption. Rather, it is much better to state the problem straight. It might sound dumb in he beginning but would save from making dumb efforts based on incorrect assumptions later ..
2. Hypotheses to he problem; typically list down possible reasons for the failure
3. Experiments to test your hypotheses: The most important mistake to avoid here is that the experiments must be designed to test the hypotheses nothing more nothing less. This step would help you narrow down your solution/problem area.
4. Expected outcomes will help you draw a base line for your experimentations.
5. Actual outcomes must be compared and contrasted with the expected outcome for rigour in the testing process. Any deviations must be carefully studied and isolated to ensure that the anomaly is not because of some mistakes/incorrect measurements this will save one from drawing incorrect conclusions.
6. Conclusion, here again care must be taken to state no more than the experiment proves/disproves.

By asking the right questions, conducting the right tests and drawing the right conclusions one might drill down the hierarchical structure of the problem to isolate the point of failure and fix it so that the system no longer fails at this point.

One might need to-fro between the two logical techniques to reach a plausible solution. Need to put this learning in practice now...

Debugging

This is funny, this is not funny for me.

During this debugging session, I'm continuously guilt ridden for the technical failure is delaying the delivery and it was increasingly time consuming because I needed to question each of my assumptions, that sucks!

After my stress-profile-code-profile-stress cycles I did however learned a bit more about various JVM GC tuning options. Also, that in an overzealous attempt to gain customer delight the code was doing far more than required. I moved a step back and thinking from the usability perspective the problem immediately visible. Thanks to right encapsulation none of the interfaces are breaking and the affected code module could be refactored to fix the performance issue at hand. Further, tweaking the data structure to collect data for generating the performance reports helped us achieve performance gains. Now, we are able to do more with less. The trick is finding the application of technology that makes the most sense for solving your problem at hand.

Looks like my build is complete, let me profile my code again..

Java: Heap crunch

Profile-code-deploy-load was all that I did yesterday and yes it did improve the system performance and technically it did survived the brutal four hour load test without causing OOOE (out of memory exception). The primary hypothesis under which I'm working is that their the certain long living references have retained a sizable of memory. But, I'm sure I'm not exactly able to find the root cause for the system degradation under load. The important observations are;

• We are loading the system using a pattern where we ramp up the load in x minutes and maintain the peak load for y hours and hen ramp down to zero in x minutes.
• The heap usage percentage follows the similar pattern during ramp up, but although the load has plateaued the heap usage percentages continues go higher, while ramp down it eases a bit but never follows the decrease in load and never actually reaches zero or minimum even though there is no activity in the application.
  
• CPU times in GC increases almost linearly albeit with a low gradient and decreases to zero after load completes.

I plan to study GC logs today to understand the system characteristics and fine tune the JVM fit for our purpose and I am sill not sure if I have a memory leak... more later.

Java: Memory leak

Around two years back, when I encountered an article mentioning memory leak in Java, it was little surprising for me and may be author knew about his audience and went ahead to provide one demo program. It surely made me aware of the issue and certain anti-patterns to avoid. Then, I encountered my very own first Java memory leak, and that I my skills are not adequate to handle them. Here, in the current case, when I first observed this phenomenon I went ahead and explicitly 'nulled' my references, which are other wise supposed to go out of scope any ways and hence eligible for garbage collection, but to my frustration this has only marginally helped as the changes have only reduced the rate of memory leak and has not eliminated it. Now I plan to work to work associated anti-patterns like that weak references or references to small living objects held by long living objects etc. Will keep you posted.

Just like that...

We Indians have a tendency to generalize thing so much so that it hardly takes any effort from our side to paint the world in one color. Often as an extension to this exercise to tend o extract particular characteristics of people from a city, and I quite believe that any self respecting engineer from a decent engineering college in India must have had great fun using them to describe their friends. Almost every where you can find 'Baba', 'Guru', 'Ziddi Jat', 'Sirfira Sardar', 'Bihari', 'Taant', 'Gulti', 'Kannadiga', 'Khatta', 'Ghatan'... and the list goes on and on... while I was siting wih some of my senior friends we happened to discuss the changing times.. the arch typical '... those were the day, ...' and one of my senior friend was quite amused and interested to learn the diversity, he then told us that back then during his times such richness and variety was unheard of and as a result of which he would often be referred to as 'Madrasi', now that was real bad as this could be the worst form of identity crisis... he complained that all those from 'North India' used to refer any one down south with the same term and he would never be able to take that... and in order to have revenge on them he would paint them (Northies) as 'Bihari' and that had far reaching affect on them and upon complain he would highlight the similar difference to them... and that probably sowed the seed to heightened verbosity currently in use... seems like this is what is referred to as evolution of language ;)

Disclaimer: With malice to no one... have fun, n'joy!

Failed to connect to remote VM. Connection refused. Connection refused: connect

A lot of time was wasted to resolve the above mentioned error, while I was trying to connect to a remote server using my eclipse IDE in debug mode. Here is the solution;

1. Compile your application using '-g' option. This will generate all the debugging related information, including local variables. By default, only line number and source file information is generated.
2. Launch a new Java Virtual Machine (VM) with the main class of the application to be debugged. For e.g. Launch a new Java Virtual Machine (VM) with the main class of the application to be debugged. For e.g. $> gt; java -Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket, server=y, suspend=n, address=8000

Hope that helps.

Are YOU Game?

... and the Lord said, 'let their be light...' yeah and in that moment of enlightenment I could now very clearly see some anti-patterns in my behavior. Let us try and understand the same. Let's say you are pretty very well educated, you have a decent job, you are considered a 'potential' candidate to handle most challenging assignments in your organization and you dream big. But.. and a big one there to be precise, more often than not, you hit a big wall somewhere along the way. You can get something going, but it does not go very far. We have no idea how many times we end up stuck in second gear for the rest of our lives. While we may be “good enough” we could be so much better.

Yes, I wanted to talk about 'Scalability' as much as this concept is borrowed from Software Engineering I plan to adopt it in my day to day life.

"Scalability is a desirable property of a system, a network, or a process, which indicates its ability to either handle growing amounts of work in a graceful manner, or to be readily enlarged.[1] For example, it can refer to the capability of a system to increase total throughput under an increased load when resources (typically hardware) are added. An analogous meaning is implied when the word is used in a commercial context, where scalability of a company implies that the underlying business model offers the potential for economic growth within the company.

An algorithm, design, networking protocol, program, or other system is said to scale if it is suitably efficient and practical when applied to large situations (e.g. a large input data set or large number of participating nodes in the case of a distributed system). If the design fails when the quantity increases then it does not scale." - Source: wikipedia

Now scalability has multiple facets to understand;

• 'Scale Out' - Diversify your horizon of understanding, often I have observed that concepts from seemingly unrelated subjects, an example in case the recent rage with Web 2.0 where concepts from Sociology and Technology is gainfully employed to conduct 'Viral Marketing' for businesses. "Tell-A-Friend" widget from Pramati which you see on my blog is niche area platform product operating in this space. It definitely helps to find/create new opportunities if you can apply your diverse knowledge to add value to the society at large. Another way to scale out is to play multiple roles in your team to gain first hand understanding of issues at hand. It may be an enriching experience if you can mix-match-employ your experience with solving engineering problems and handling customer situations.

• 'Scale Up' - Based upon your analysis of the situation you feel it just right to add extra effort.. walk that extra mile to achieve 'customer delight'. [here, customer is overloaded for both the internal and external customer behavior]

• 'Scale Down' - Last but not the least, it is important for us to identify and 'scale down' things which are not working. Things which might become too big a problem while one is trying to scale...
  

With that understanding of issues with me.. I want to ask myself if I'm Game enough to leave my footprint on the third rock from Sun? Game enough to tackle the challenging technology? Game enough to get my game on the planet. I believe I can.. Come on! double up now... faster, higher, stronger.. be a sport :)

Why slow?

Hmm.. so good that you are able to monitor response times of the application components but then how does that helps tuning the application... do not count activity.. show me the productivity.. so here I'm able to monitor my code and generate some couple of reports.. but then they didn't really helped me understand performance bottlenecks.. I would run them again and again to aggregate more reports!! No amount of diligence seemed to work.. imagine I was not able to find problems.. that reminds me of the adage 'bin mange moti mile aur mange mile na bheekh' ... I had understood there is sure shot knowledge gap from technical and business perspective... after all we are still far from creating tools which would scan and identify and fix code on their own. There are just so many options for every refactoring which one intends o make in the code.. one needs to weigh your options before making code change les it should fall under its own weight... Ok, I digressed because even while conducting my own research I chanced to read interesing articles which would give me a peek ino the future about gainfully utlizing concepts from autonomic computing to application monitoring as an aid to debugging applications... but 'Show me the results' was driving me crazy.. but then as the luck would have it a visit to the the local book shop helped me discover "Bitter Java" and as prophecized by "The Alchemist" when you one truly starts liking something the entire universe conspires to make it happen.. seemed so much true .. he "bitter ejb" chapter seems to have been written just for me a quick read with reports immediately helped me identify the 'chatty' interface between our application and the database.. the server frequent roundtrips are consuming far more time over the wire than conducting any productive work... steriotypically such problems occur when we make far more granular requests than is required or may be executing queries in loops... it helps to find initial symptoms of the problem at the macro level.. such an information can be suitably used in other low performing use cases.. bugs come and go, but I'm accumulating debugging skills ... I'm already feeling the need to study "Refactoring: Martin Fowler".. may be I should get back to reading.. more later

I love Johnny Walker...

I love it for the simple reason that it provides me with one of the very core ideology I have decided to follow in my life. It helps me take rough with the smooth and always keeps me brimming with confidence. It helps me overcome difficulties and avoid my indulgence with petty attractions. It helps me take change in my stride. It helps me keep walking...

Performance Monitoring Utility

My current assignment required me to create a light weight utility to conduct application monitoring during production and development phases to identify potential performance bottlenecks. The utility which I created is now ready and is under Beta. Key rationale to create this utility were;

1. Create an unobtrusive (or minimal) way of monitoring the application, although in its current state it does not uses aspect technologies and thus requires us to manually inject code to those applicaion components which we want to instrument.
2. The utility should be real lightweight so that it could be used at all times to monitor application performance and system health.
3. Simple to use. Client code should be able to use the utility in an easy to use way. Figuratively speaking, just place your probes and desired parts of your application and assign a logically comprehend able name to it, that's all!
4. User should be able set monitoring preferences.
5. Reporting monitoring results.
6. Currently the utility monitors the execution times alone but can easily be extended to monitor other metrics like memory, cpu etc.

Design Considerations:

To improve simplicity, reduce verbosity and hide inner working of the monitor code a facade to the monitor library is provided so all hat client code needs to do is;

public SomeObject theMethodUnderInstrumentation(){
try{
MonitorFacade facade = MonitorFactory.getFacade();
if(facade.isMonitorEnabled()){
facade.recordExecutionStart("someLogicalContextName");
}
// do some time consuming task here
}finally{
if(facade.isMonitorEnabled()){
facade.recordExecutionStop("someLogicalContextName");
}
}
}

This is all that is required by the developer to instrument it's code. Simple isn't it!! The above becomes transparent to the developer in case we use aspects to define our pointcuts to inject a similar code as above.

It is a common observation that application performance tuning exercise requires an inter disciplinary approach and much is needed to understand the system performance in the right context. It was therefore important to understand the complete execution path under observation. To make this possible Composite pattern is implemented. The intent of composite is to compose objects into tree structures to represent part-whole hierarchies. The call tree is captured which displays the function execution paths that were traversed in the profiled application. The root of the tree is the entry point into the application or the component. Each function node lists all the functions it called and performance data about those function calls. So to put it crudely composite pattern helped me create the tree structure of application components under observation.

Now that we are ready with the basic infrastructure needed to monitor he code performance at all times the next piece of challenge is to create automated load test and as any one with similar experience will immediately identify that creating automated test cases is not that big a deal but maintaining them over a period of time is.. my next task involves around ways to put intelligence into our load test scripts... I woke up pretty early today .... ;)

Tip: Sort a map by values

A handy tip to sort map by values.

//:~
package samples.core.java;

import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;

public class SortMapByValueExample {
public static void main(String[] args) {
Map map = new TreeMap();
map.put("1", "100");
map.put("10", "50");
map.put("100", "40");
map.put("1000", "10000");

List keys = new ArrayList(map.keySet());
ByValueComparator bvc = new SortMapByValueExample().new ByValueComparator(
map);
Collections.sort(keys, bvc);
Iterator iterator = keys.iterator();
while (iterator.hasNext()) {
String key = (String) iterator.next();
System.out.println(key + "-" + map.get(key));
}
}

private class ByValueComparator implements Comparator {
private Map map;

public ByValueComparator(Map map) {
this.map = map;
}

public int compare(Object o1, Object o2) {
int integer = Integer.parseInt((String) this.map.get((String) o1));
int otherInteger = Integer.parseInt((String) this.map
.get((String) o2));
return -1
* (new Integer(integer)
.compareTo(new Integer(otherInteger)));
}
}
}

Performance monitoring - creating home grown utilities

One of my current focus area is performance engineering, which although is an umbrella concept in itself pretty much sums the need to achieve the non-funcional requirement defined for the software solution. You may find a neat introduction the concept here. As much as it is important from software engineering perspective, talking of it during design session is looked upon as 'over-engineering' and is grossly ignored saying 'It will be one nice problem to solve, but we are not there yet...' probably, I understand the product manager's need to hit the market ASAP.

As they say, 'Dates in the calendar are closer than they appear', we normally start receiving complaints in form of poor user experience, loss of revenue to the competition blah, blah.. so now we have the 'nice' problem actually waiting for us to solve. Frankly, to start with there are no straight answers or atleast I do not have one. What do we do now? Where do we start? Which are the ideal candidates for performance tuning exercise? Here, is what I would do to track the problem. Place an extremely light weight activity monitoring utility in production at all times because I deem it necessary to even capture the usage of the application by the real life users and not us software engineers enmasked as pseudo users [During one of my earlier assignments, the users of the application would simply close the browser window wih out logging out, because the application didn't contained his private data and just that the users didn' like to wait for the application to log out... resulting in numerous open sessions...]

There are plenty of open source and commercially available tools in the market, still if you feel the need to create a home grown solution for some reason, here is the simple idea that I'm working on now to collect metrics from different parts of the application performance and use it for the quantitative analysis of the application.

Provide a ThreadLocal storage to 'StatisticsCollector' which would provide probes to be kept at key parts of the application to record statistics, to start with I'm only collecting execution times. Create a new object each time a new thread is spawned and keep adding metrics for the components with their metrics at collection points and them to the tree data structure which represents the call hierarchy. Finally, once the response is committed for the thread persist the data and reset thread local variables. I have primarily identified following collection points viz. the servlet filter, servlet do*(), JSPs, transactions, cache hits, jdbc layer while they are less in number they can pretty much indicate the slowest moving layer in my application. This information will form the input o iterative process of profiling-tuning-monitoring to gain performance improvement upon completion of each cycle.

Buckle up - Enterprise Edition

Time and again people ask me, if I still think I made the right decision to cut short my 'coveted' foreign assignment to return India because I felt it 'did not add value'... well let time keep tab on that. My story is that... whatever role life gives you play it big [this one is lifted from Smirnoff's ad, we can even learn from a Daru company ;)].

We[Indian IT industry] started real small with back office support work but soon gained confidence of the world to provide medium to large scale system enhancements from where we incrementally we created value to achieve customer delight through our efficiency and professionalism. It is no wonder that today IT captains are trusted partners for very large scale system transformation exercises. In essence, we have incrementally moved up the value chain.

Now, it is important for us to move up to the next step where in we need to conceptualize and develop 'product(s)'. Definitely, this would be a much riskier business. But, then are there any options for us but to grow better. The world will truly respect our technical prowess if we 'create' new concepts and 'implement' them... and can there be better times than these. Here is why I think these are the best of time to go deep...

• Under the current scheme of affairs techies prefer to stay home because much of the quality work will be done off-shore and more importantly an increased focus on domestic projects gives them an opportunity to solve 'their' own problems. This tends to improve employee job satisfaction as it allows them to play 'lead' roles.
• Working on domestic projects will create more avenues where we can find generic 'patterns' to be created as 'products'.
• Investing into products will make employee confident about the longevity of vision of the organization leaders.
• Confidence breeds success.

NASSCOM is equally exhilarated to display the abounding capacity of Indian software companies representing products conceptualized, designed and made in India. It has come up with a logo as displayed above. You may wish to read more here

It is all in mind and we can definitely make a difference by shaping our reactions to the events in life.