Thursday, March 4, 2010

How does the data flows when a job is submitted to Hadoop?

Based on the discussion here, typically the data flow is like this:
  1. Client submits a job description to the JobTracker. 
  2. JobTracker figures out block locations for the input file(s) by talking to HDFS NameNode. 
  3. JobTracker creates a job description file in HDFS which will be read by the nodes to copy over the job's code etc. 
  4. JobTracker starts map tasks on the slaves (TaskTrackers) with the appropriate data blocks. 
  5. After running, maps create intermediate output files on those slaves. These are not in HDFS, they're in some temporary storage used by MapReduce. 
  6. JobTracker starts reduces on a series of slaves, which copy over the appropriate map outputs, apply the reduce function, and write the outputs to HDFS (one output file per reducer). 
  7. Some logs for the job may also be put into HDFS by the JobTracker.
However, there is a big caveat, which is that the map and reduce tasks run arbitrary code. It is not unusual to have a map that opens a second HDFS file to read some information (e.g. for doing a join of a small table against a big file). If you use Hadoop Streaming or Pipes to write a job in Python, Ruby, C, etc, then you are launching arbitrary processes which may also access external resources in this manner. Some people also read/write to DBs (e.g. MySQL) from their tasks.

Sunday, February 21, 2010

Rampyari ki khoj

What's the big deal, one might ask? Yes, I'm looking for a car and while hunting around I found a wealth of information is available about brands and products available in the market, but one finds zero information about the dealers in the local market.

Market is full of inexperienced, uneducated, incompetent salesmen who are ready grok 'the best' car to you in no time without understanding my needs or expectations. They simply seem to be oblivious about the competition and live a ,monopolistic universe of their own brand, to such an extent that a mere mention about the competition makes them turn blue.

Further, talking to old timers, I realized that your dream car may soon become a nightmare of sorts based upon your service providers, about which you may not have any reliable information. After all any machine needs skilled maintenance and I need some one whom I can trust for the pre-defined level of quality of service in lieu of the amount I spend on the same. The need for rating dealers and service stations is both timely and important.

In the mean while in order to live in the more 'real' world I soon would have to make my choices...

Tuesday, February 16, 2010

Evolving programming paradigms

The free lunch is over and it simply means that;
  • Our applications may no longer be able to automatically seek benefits of hardware up-gradations, we will have to plan, design and make room for concurrency.
  • As a developer we need to hone our skills to make good use of hyper-threading, multi-core CPUs and caching.
  • Do not get confused with marketing slogans about new found technologies. Seldom, does a technology becomes mature too fast to become main-stream, generally it is old technologies which increasingly useful with time and better understanding of its developers that can gainfully be used to solve important problems.
  • Because, our applications do not automatically gain performance with newer hardware solving performance problems and performance tuning will become an important activity.
  • Start preparing now.

Urban Area: Community or a Marketplace

'Mumbai for Mumbaikars' might bring bad taste to your mouth or a sense of ownership based upon your political affiliations. Whatever be your reaction it is however increasingly becoming important for us to delve deeper to try and understand the core issues lest we must reject them as pure nuisance.
From a pure theoretical perspective the urban population can be broadly divided into two sections;
  • Community dwellers, who inhabits a particular place and encourage symbiotic relationship between various members of the community. They largely like to take the ownership of the place for its general well being and associated identity. Such people have high tolerance towards problems as result of scarcity of resources.
  • Market dwellers, who look at an urban area as a place where market operates and they themselves form links into the demand and supply chains. Such people have high preference for quality of life.
The stated difference in their value system creates competition resulting in clashes.

What can be done about it? well .. I'm not too sure will it help if market dwellers contribute back to the community of which they are part off to avoid being  tagged as parasites and community ensures that market operates in safety  thus creating  a conducive environment to attract better talent!!

Part of the answer also requires us to understand the larger picture and avoid such local focus. Will it not help if we have more balanced growth in .. sounds wishful.

Share your mind, who knows our ideas are heard and we do reach a resolution.. let us start from ground zero again..

Tuesday, February 9, 2010

Mar Java, Mit Java..

... and with European Commission's nod the last hope of keeping the Sun shining too went away, while the whole software and business community is still evaluating the takeover deal, I must express that I feel bad about the whole thing.. there is something which is not generating positive vibes... although, some of you might discount me for being emotional, I have at least one objective reason to support my point in case. Old timers will clearly be able to spot the difference that post acquisition all the Sun domains have been distastefully defaced and simply reflects the tyranny, greed and callous attitude of the raiders.

I, seriously hope similar fate is not vetted out to the open source initiatives.

Sunday, February 7, 2010

In bad taste...

Reality television is truly a reality today and one can no longer ignore them. There popularity can be attributed primarily because it also involves its viewers. I used to love 'Boogie-Woogie' for their plain ingenuity and honesty in bringing recognition to dancing talent. The show was conducted wonderfully well.

With time however, the production value of reality shows have increased dramatically but I'm sorry to say that they leave everyone demoralized. Particularly shows like Roadies and Splitsvilla are truly disgusting.

My whole point is that such shows are increasingly creating a 'public acceptance' for foul and abusive language in public which is one very dangerous trend. They must mend their ways or else criminal action be taken against them.

Thursday, February 4, 2010

Running Java in debug mode using Apache Ant

Configure your Java task as given below;



<java classname="org.zero.Main" fork="true" failonerror="true">
            <sysproperty key="DEBUG" value="true" />
            <jvmarg value="-Xdebug" />
            <jvmarg value="-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8999" />
            <jvmarg value="-Xnoagent" />
            <jvmarg value="-Djava.compiler=none" />
            <arg value="${project.root}/conf/preferences.xml" />
            <classpath refid="project.classpath" />
        </java>

Further,  configure your eclipse debug remote application to listen to the specified port (8999 in this case).

Also, add Thread.sleep(1000); as the first line in your main method.

Execute Java task and try connecting from eclipse, you may need to increase the thread sleep time.

Saturday, January 30, 2010

Bug fix: Eclipse springsource on UBuntu 9.10

...so u have upgraded to UBuntu 9.10 and you are really impressed with its new features and you cannot control your excitement to check out your development environment and launch eclipse .. but oops! you are not able to create a new project.. none of the buttons seem to work chances are you are also suffering from the same bug follow the link for the fix  

This issue is resolved in eclipse 3.5.2 M2 release. Hope that helps.

Friday, October 16, 2009

Who created God?


Who created God? or Is it our own invention? It is an age-old question that has plagued all those who like to think about the big questions.

Just last fortnight, I visited Puri, Orissa for the pilgrimage to Jagannath temple. As much as it was spiritually "satisfying" for my parents I was irritated with the chaos and mayhem created by 'pandas' there. I was wondering if indeed visitors are able to offer their prayers in peace. It is after all every Hindus long cherished wish to do their 'char-dham-yatra' but was that the only purpose or the reason lies in creating another money spinning business in the name of God? Is God our own creation? may be I don't know and neither do I have the patience or time or inclination or competence to read and understand the philosphical content of all the books on the subject. But, here was this craftsman on the Grand Road - famous for the 'Rathyatra' conducted here every year, focussed on giving final touches to idols. His frentic fingure movement was a feast to watch and I was still wondering if he created the God!! Shy of me taking his photos he asked me 'Babu, humari photo kyun le rahe ho?' I told him, I never saw any one creating God to which he replies 'Babu, murti nahin banayenge to ghar kaise chalayenge aur bhagwan to haathon mein hai to khana khila raha hai.. humara to bass itna hi kaam hai'

Soon I realised why do we say 'Apna haath Jagannath'. I guess it is very important to be focussed and do our duty religiously.. rest shall follow. I guess I learn some thing from him.

Saturday, August 15, 2009

Maya tumko itna bhi samajh mein nahin aaya!!

Dalit ki beti has done it again... this time she harps back on Manuwadis for targeting her for being partial.. why is it that congress is not targeted for creating 'brand' by naming every little thing after Nehru-Gandhi family member... Read more

I believe instead of crying foul, she can take a lesson here and help creating lets say an IT park in the name of Dr. Ambedkar (hmm.. that will not help BSP) or Kansiram (hmm.. that will help her) Maya Pragat Sangadak Sansthan etc as a matter of fact it doesn't matter if you are trying to bring token benefit for your party provided it brings considerable tangible benefit to people and help them improve their living... get creative Maya you can earn much more by helping people create wealth.. and you can really be in good history books.

Friday, July 24, 2009

Finally spotted the 'Tiger'

Once upon a time long, long, long ago SCJP 5 was newly released and it was a proud moment in my life to secure a distinction in the same. I felt like I 'Tamed the Tiger' never ask how did I feel working on versions 1.4 and 1.3! all these years. Never mind, the 'Tiger' will soon be a sunset technology from Sun's stable... never mind 'Mustang' is galloping faster and faster... Today I spotted the 'Tiger'. I'm happy with the opportunities at hand to experiment... looking forward to play with the beast...

Tuesday, July 21, 2009

School chalein hum...

Coming from a modest middle-class background I understand and appreciate the importance of education in securing one's future. Not that people from other background don't appreciate education or that careers in other fields do not exist. But, pragmatically speaking education can help you achieve sustainable living and it is crucial in managing one's self esteem, specially for those from related backgrounds. Probably, that sets the pretext for the heightened stress that parents and children undergo to secure better grades, which has even led to suicides etc in some really unfortunate cases.

The recent education policy aims to abolish school board exams quoting the need to remove [reduce?] the stress for school going children. A welcome step? Well, depends where are you coming from.. In case you feel it is a welcome step because it help reduce stress I'm not with you. My question is who are you trying to help? are you saying you can make their life a bed of roses? they are not going to have any stress? ever? May be your new system allows them to pass certain public exam 'without failing' but what next? are they going to spend their lives at school alone, studying? I believe, No.. some time or the other they need to face the industry, the real world.. believe me all that school work will look like a cake walk...

But, in case you do believe that abolishing public exams at school level is welcome step because now students do not have to study to get grades. Yes, as much as it sounds strange it is good. It is important to understand that education is not some material object [read grades/marks] that one gets upon completion of education. The real education is the state of mind. It is that great heritage of common sense that has been brought down to us over the ages. Now, with that material motivation removed students can now focus on learning rather than getting good grades, which makes them seeker. The motivation to learn is now from within, there are no socially defined pressures.

I understand what many of us might go through as a result of this void created in our lives. There are no 'goals' which might distract our 'normal' student and he may eventually loose his interest in studies and compromises on a lesser education ... which is good isn't it?!! think about it!!... he was not liking it any ways and at best would have remained a mediocre all his life even after a 'highly-rated-degree'. Only those who really appreciate their learning will continue thus saving a lot of resources.

In either case there are certain risk factors which need to be mitigated to avoid chaos:
  • The 'normal' student who settled for a lesser education as a result of his ever increasing committment to his 'other-than-education' activities should be able to leave his education without any penalties
  • He should be able to find an employment opportunity fit to his calibre.
  • He should be able to return to his education in case he gets frustrated with his 'boring' work to seek creativity to gain theoretical understanding of THE World around him.
Learning should be fun.

Monday, July 13, 2009

Imagining Female Foeticide and Gay marriages and Prostitution

All along I was tagged as a happy-go-lucky person who would go out of his way to have fun.. for that one moment of 'nirmal anand' and the best compliment to me was when some called me a 'stress buster' and further asked what's your source of energy? how do you do that? In reply to which I would say I'm happy as I'm gay!!! and then everyone would burst out to laughter.. I dare say that again and people would take it really seariously .. thanks to the recent turn of events in our society. Recent court judgements have 'legalised' the gay/lesbian marriages leading to nationwide celebrations for their newly acquired "freedom". I was wondering what's the real cause for their inclination to the unantural carnalness... Is that their utmost frustration in finding the right girl? Is it because of acute gender imbalance in their community? Is it THE result of female foeticide in our country? I mean I'm also not in favor of moral poilicing but then isn't it banal and disgusting or may be I don't understand... so what's next? Let's legalize prostitution and earn more money for governement coffers..

The whole point is such act of recognition will only encourage impressionable minds to follow the next 'IN' thing to prove them to be 'COOL' and the by the time they will realize their mistakes it will be to late...