Just dance

Yes, I'm watching the screening rounds on Star Plus.. the candidates are from as diverse a background as one might only imagine.. their energy and sheer enthuthiasm is infectious. I don't have words to express my feelings, however, that's not the point of this piece... but the fact that it is important to be an artist in your profession be it dancing or software engineering.. practice your craft

Leveling the Threat Matrix

Almost serendipitously, I discovered some simple sentences which when put together does makes sense to me...

• Initiative and Responsibility: Very few of us take initiative, while this is an important trait of young, energetic people and should be highly appreciated. More often than not the same people lack responsibility or step back when an avalanche of things start coming one's way.
• Action and Control: Now is better than never, although never is almost always better than "now". One needs to be in control of things... evolutionary, step-by-step improvements keeps one in control and can help steer the direction of future coarse of actions in a more informed manner.

Simply put one of the reasons we come under stress for all the wrong reasons after all the right intentions is the eerie feeling og the lack of control... I told you, some times the universe conspires to help me discover the effective habits... it was only this time I thought it could be useful to you too...

Few observations on my blog readers

... ho ho ho the big brother is watching you!! well not really, but yes as a trained statistician and a practicing software engineer I love to observe "things", a collection of things... and try to create an empirical model of their behavior pattern.. a blue print.. and study them to put them into my knowledge stack that may or may not be used immediately.


My findings;

• It adds to your credibility if you provide references in your write ups at proper places. It only proves that the write has done his/her home work.
• However, merely providing the hyper-link.. like to learn more read here will not necessary be followed by your reader because
• that may break his train of thought
• you might risk to loose your audience

The way I see it is that it would serve the reader as well as the writer by providing the central idea around the hyper-link for the more inquisitive reader to follow without loosing his focus.

My two cents. Get leaner!!

Innovation - Generating Ideas: The monkey's view

Not again ..not another hyperbolic session on INNOVATION but these days almost every one in my social circle at least hints about innovation. Not so surprisingly it is a constant endeavour of our species to be introspective and improve continuously, so much so that historians have even classified our cultural heritage in terms of the innovations that have distinguished them viz: the stone age, iron age.. Similarly, the role of innovation at enterprises is pretty much tied with their sustainability and even existence.

 

The dictionary defines the term in a rather half baked manner (blasphemous?!!) while it does appreciates creating the "new thing" it safely discounts the efforts gone into continuous improvement of existing "thing". Sadly the incremental or the continuous improvements made with the existing "things" meets the same fate and goes unnoticed at our enterprises.  

Also, we tend to focus more on the fruits of innovation, that is, the "product" itself and do tend to take the process part for granted and much worse "a necessary evil" at least in my software industry. It is only timely that much work of art is already published (think continuous delivery) but they are yet to make to the mainstream howsoever "agile" the enterprises may like to call themselves.

I understand by now must have already read my mind and must be wondering if I'm really going to add any value... some of the ways I think can make a good start can be summarized as under;

• Become a producer of content.. in my limited scope of understanding I feel most of us do not write or express ourselves or take stand for the fear of being judged. Let's change that and start putting our thoughts in black and white here on blogs, comment on others blogs.. connect. Not many of our experienced leaders write. Writing is such a powerful medium that one can reach out to whole bunch of people at one time which is boon for our time-pressed industry leaders.
• In one of my earlier post I discussed the challenges I faced to share knowledge. Some of the ways sharing knowledge can help is to come out of our stove pipe modes and satiate our higher needs of self actualization. There is this nice article which covers the subject in greater depth and also provides the relevant background knowledge. You will do good to read it for yourself.
• Management determines the organizational climate. It can send a clear message through rewards, mentor employees to participate in open source software products, participation at technical conferences, taking engineers to customer meetings, sales team to participate in test cycles...
• The above mentioned set of activities create and encourage the cross-pollination of ideas which can create magical results.. you never know when that bulb glows in someone's head, the EUREKA moment!!
• Support innovators as per Seth Godin if you don't support and nurture them it simply puts spanner into the innovation engine even before it gained any considerable momentum.

 Innovation is critical to our growth and generating ideas is critical to innovation itself while thinkers must understand their responsibilities have just begun. Here I made an attempt to showcase some of the problems which should be addressed to create value added services to serve our society.

Parallel Search

Problem:

Write a Java class that allows parallel search in an array of integer. It provides the following static method:

public static int parallelSearch(int[ ] a , int numThreads)

This method creates as many threads as specified by numThreads, divides the array a into that many parts, and gives each thread a part of the array to search for sequentially. If any thread finds x, then it returns an index i such that A [ i ] = x. Otherwise, the method returns -1.

Solution:

package org.zero.concurrent.chap01;

import java.util.Arrays;

public class ParallelSearch {
    private static int index = -1;

    public static int parallelSearch(int search, int[] in, int numThreads) {
        // partition
        int partitionSize = in.length / numThreads;
        Thread[] threads = new Thread[numThreads];
        int end = 0;
        int begin = 0;
        // search
        for (int i = 0; i < numThreads; i++) {
            end = begin + partitionSize;
            if (i == numThreads - 1 || end > in.length) {
                end = in.length;
            }
            Search target = new Search(begin, end, in, search);
            System.out.println(target);
            threads[i] = new Thread(target);
            threads[i].start();
            System.out.println(threads[i].getName());
            begin = end;
        }
        return index;
    }

    public static void main(String[] args) {
        int[] a = new int[100];
        for (int i = 0; i < a.length; i++) {
            a[i] = (int) (Math.random() * 10);
        }
        int parallelSearch = parallelSearch(2, a, 7);
        if (-1 == parallelSearch) {
            System.out.println("not found");
        } else {
            System.out.println("found at: " + parallelSearch);
        }
    }

    private static class Search implements Runnable {
        int begin;
        int end;
        int[] a;
        int x;

        public Search(int begin, int end, int[] a, int x) {
            super();
            if (end < begin) {
                throw new IllegalStateException();
            }
            this.begin = begin;
            this.end = end;
            this.a = a;
            this.x = x;
        }

        @Override
        public void run() {
            for (int i = begin; i < end; i++) {
                if (x == a[i]) {
                    index = i;
                    System.out.println(i + " "
                            + Thread.currentThread().getName());
                    // break;
                }
            }
        }

        @Override
        public String toString() {
            return "Search [a=" + Arrays.toString(a) + ", length=" + a.length
                    + ", begin=" + begin + ", end=" + end + ", x=" + x + "]";
        }

    }
}

P.S: Any suggestion to improve the code is welcome.

Product Versioning: Embedding packaging information

Almost serendipitously, I discovered the Java Package class today. One must wonder how possibly this is going to make a difference (aka increase their geek quotient or coolness factor). Well the beauty lies in the details.

Problem Statement: You commit your code to the cvs and after going through the complete lifecycle experience your code finally sees the light of the day, much to your chagrin that the users discovered some bug, even after all those unit testing and that pragmatic ranting ;) But, then you are the rock star developer who had already discovered the problem and fixed it :D but how do you know if user is not using some older version of your package???

Solution: You can embed the information in the manifest file at build time which could be read by exploding the jar, simple!! NO, most of your users wouldn't (shouldn't) know it.It would be really nice if you can print this star-studded information at the beginning of the code execution. You can make this as the first line of your log file or may be a separate file which could be used for bug reporting, options are open.. How do you do it?

Step 1: Create an annotation.

package org.zero;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Target(ElementType.PACKAGE)
@Retention(RetentionPolicy.RUNTIME)
public @interface MyVersionAnnotation {

    String version();

    String revision();

    String date();

    String user();

    String url();
}

Step 2: Generate a class package-info.java

@MyVersionAnnotation(date = "2010-11-26", revision = "11", url = "http://onjava.com/pub/a/onjava/2004/04/21/declarative.html?page=3", user = "nitin", version = "123")
package org.zero;

Step 3: Create a class to access this information;

/**
 * This class finds the package info for mypackage and the MyVersionAnnotation
 * information.
 */
public class PackageDemo {
    private static Package myPackage;
    private static MyVersionAnnotation version;

    static {
        myPackage = MyVersionAnnotation.class.getPackage();
        version = myPackage.getAnnotation(MyVersionAnnotation.class);
    }

    /**
     * Get the meta-data for the mypackage package.
     *
     * @return
     */
    static Package getPackage() {
        return myPackage;
    }

    /**
     * Get the mypackage version.
     *
     * @return the mypackage version string, eg. "0.6.3-dev"
     */
    public static String getVersion() {
        return version != null ? version.version() : "Unknown";
    }

    /**
     * Get the subversion revision number for the root directory
     *
     * @return the revision number, eg. "451451"
     */
    public static String getRevision() {
        return version != null ? version.revision() : "Unknown";
    }

    /**
     * The date that mypackage was compiled.
     *
     * @return the compilation date in unix date format
     */
    public static String getDate() {
        return version != null ? version.date() : "Unknown";
    }

    /**
     * The user that compiled mypackage.
     *
     * @return the username of the user
     */
    public static String getUser() {
        return version != null ? version.user() : "Unknown";
    }

    /**
     * Get the subversion URL for the root mypackage directory.
     */
    public static String getUrl() {
        return version != null ? version.url() : "Unknown";
    }

    /**
     * Returns the buildVersion which includes version, revision, user and date.
     */
    public static String getBuildVersion() {
        return PackageDemo.getVersion() + " from " + PackageDemo.getRevision()
                + " by " + PackageDemo.getUser() + " on "
                + PackageDemo.getDate();
    }

    public static void main(String[] args) {
        System.out.println("mypackage " + getVersion());
        System.out.println("Subversion " + getUrl() + " -r " + getRevision());
        System.out.println("Compiled by " + getUser() + " on " + getDate());
    }
}

PS: Source code courtesy org.apache.hadoop.util.VersionInfo

Race etiquettes

Read some them here,
http://runnersforlife.com/page/race-etiquette

Talking common sense

'Common sense is quite uncommon' is what I was told whenever I failed to cut my way through the chaos. As much as, I was frustrated with my failure to learn from my failures. I equally wondered if there is a way to master the technique. Today, I wish to share my learnings with my readers.

'Common sense is a combination of experience, training, humility, wit and intelligence'. Wow! now that sounds like a well balanced equation with known variables. Doesn't that excites you to take control of participating variables and work on your weaker areas to improve your gross wisdom quotient! Let's work on that too;

• Experience is nothing but the accumulation of knowledge or skill that results from direct participation in events or activities. Simply put, just do it will alone help learn things in small digestible chunks. Fear of failure is the greatest impediment to learning. Make it your friend!! Adapt a fail fast approach. trust me it always helps and at least you end up enriching your experience.
• Training refers to the acquisition of knowledge, skills, and competencies as a result of the teaching. But, where do I start. Look around yourself. Be relevant to your eco-system. Learning something which is immediately useful keeps you motivated and provides avenues to 'Apply Yourself'. Remember no one learns or appreciates swimming by reading the book!!
• Pride eats up your brain, being humble means taking good and bad in your strides and accept them as part of our lives. It always helps to balance your emotions [I know I'm being preachy, I need a lot ground work here.. but then whose blog is this ;) ]
• When in trouble use your humor! It sheds the unnecessary weight from your shoulder and helps you relax. It's very important. Enjoy! every moment.
• Intelligence is you ability to comprehend, to profit from your experience which now looks like product of the healthy concoction of the above ingredients.. the 'executable knowledge' !!

There you have it! the prescription to work on your common sense. Happy learning. You may now join you hands for a big round of applause... Have fun!!

Ride to Kudremukha

It's been so long that I last went out on a ride. Then, there these set of really enthusiastic guys who infected me with the travel bug. Get Off Your Ass!! was the war cry for us and we simply carried our back-packs and hi the road straight .. late night driving, after a really long week was proving to be physically taxing on us and we decided to call it the day at Arasikere, starting Saturday morning we visited Halebidu, Belur on our way and they took a detour to Chikmanglur and then through Ghas we reached Horanadu, little did we knew that there is a majestic temple of Goddess Annapurna. The place was really peaceful, one may just go there and stay over the weekend. It was blissful to walk in the clouds and funny to wach clouds enter our rooms and pour water on our belongings. Trust me we were drenched to core, so much so, that even our spirits must have also satiated their thirsts. We slept like Dogs, no idea when did the dawn broke on us and after one quick shower we headed towards the temple. After seeking the blessings, we hit the road again and after loosing our way to he place we went through the ghats uphill to Kudremukh, KIOCL colony and then to Lakya Dam. It was just plain fun to drive in the rains and watch the sight of nature's beauty along coffee estates, many of which just could not captured, lest we must loose our camera to heavy rains.

On our return trip, we reached Hassan town and then took the road to Bangalore. Journey was quite eventful and we have a whole lot of fun stories to share.. I miss my old gang.. God knows .. kiski nazar lag gayi :|

Keep walking..

Find the pictures here

Maps: To, From

Training for long distance running

Suddenly something! No, way.

I was bitten by this bug during my school days. Just that, I lack discipline and often my initial enthusiasm causes more harm than necessary. Often my strong will to complete certain distance over powers my muscular strength :)

Basically, this marathon thingy is more than just adding miles to my legs, I must admit it is highly rejuvenating, some how I feel very good after half an hour of running. It even helps me fight emotional voids being created in me.

I pledge to complete my Sunfeast half marathon next year.

You may take tips from the references below. Do share your tips.

References:

• http://www.runnersworld.com/
• http://marathontraining.com/

Hadoop application packaging

Job jar must be packaged as below;

job.jar
|--META-INF
|----MANIFEST.INF
|------Main-Class: x.y.z.Main
|--lib
|---- commons-lang.jar Note: Place your dependent jars inside lib directory
|--org.zero
|---- application classes here

Archiving large number of small files into small number of large files

A small file is one which is significantly smaller than the HDFS block size (default 64MB).

We have a lot of data feeds in the range of 2MB per day, storing each as a separate file is non-optimal.

The problem is that HDFS can't handle lots of files, because, every file, directory and block in HDFS is represented as an object in the namenode's memory, each of which occupies 150 bytes. So for 10 million files, each using a block, would use about 3 gigabytes of memory. Scaling up much beyond this level is a problem with current hardware. Certainly a billion files is not feasible.

Furthermore, HDFS is not geared up to efficiently accessing small files: it is primarily designed for streaming access of large files. Reading through small files normally causes lots of seeks and lots of hopping from datanode to datanode to retrieve each small file, all of which is an inefficient data access pattern.

Also, HDFS does not supports appends (follow http://www.cloudera.com/blog/2009/07/file-appends-in-hdfs/).

Known options are;

1. Load data to Hbase table and periodically export them to files for long term storage. Some thing like we have product log for a particular date/timestamp against the content of the files stored as plain text in Hbase table.
2. Alternatively, we can treat these files as pieces of the larger logical file and incrementally consolidate additions to a newer file. That is, file x was archived on day zero, the next day new records are available to be archived. We will rename the existing file to let's say x.bkp and then execute a mapreduce job to read the content from the exiting file and the new file to the file x.
3. Apache Chukwa solves the similar problem of distributed data collection and archival for log processing. We can also take inspiration from their and provide our custom solution to suit our requirements, if needed.

One wish

को‌ई गाता मैं सो जाता

संस्कृति के विस्त्रित सागर मे
सपनो कि नौका के अंदर
दुख सुख कि लहरों मे उठ गिर
बहता जाता, मैं सो जाता ।

आँखों मे भरकर प्यार अमर
आशीष हथेली मे भरकर
को‌ई मेरा सिर गोदी मे रख
सहलाता, मैं सो जाता ।

मेरे जीवन का खाराजल
मेरे जीवन का हालाहल
को‌ई अपने स्वर मे मधुमय कर
बरसाता मैं सो जाता ।

को‌ई गाता मैं सो जाता
मैं सो जाता
मैं सो जाता

 - हरिवंशराय बच्चन