Cheat Sheet: A Conceptual Framework for System Fault Tolerance

refer http://www.sei.cmu.edu/reports/92tr033.pdf

Cheat Sheet: What every software engineer should know about logs

refer https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

Checking Framework Vulnerabilities Using Maven Plugin

Chances are your application depends on certain open source libraries. Typically these libraries are not updated frequently for different reasons stability et al. However, it makes sense to be notified of security issues in our dependencies so that one can take corrective action. 

Would it not be so good if our build fails when a new vulnerability is discovered in one or more of our dependencies.. Enter Mavendependency check plugin which identifies project dependencies and checks if there are any known, publicly disclosed, vulnerabilities. 

How to use

<plugin>

<groupId>org.owasp</groupId>

<artifactId>dependency-check-maven</artifactId>

<version>1.2.9</version>

<executions>

<execution>

<goals>

<goal>check</goal>

</goals>

</execution>

</executions>

</plugin>

 

It is recommended to read the associated document to understand the working of this plugin.

Performance Note: First time execution may take longer but all future checks should take only few seconds if executed once in few days.

Interpret Page Fault Metrics

A page fault occurs when a program requests an address on a page that is not in the current set of memory resident pages.  What happens when a page fault occurs is that the thread that experienced the page fault is put into a Wait state while the operating system finds the specific page on disk and restores it to physical memory. It is important to distinguish between minor/soft and major/hard page fault;

• Minor - occurs when the page is resident at an alternate location in the memory. It may happen becasue the page is no longer part of the working set but not yet moved to disk or it was resident in memory as result of prefetch operation
• Major - occurs when the page is not located in physical memory or in the memory mapped files created by the process.

Why bother?

• Poor Latency - You may ignore minor faults however, major faults can be detrimental to your application performance in the presence of insufficient physical memory and excessive hard fault and as such needs to be fixed immediately.
• Poor CPU utilization - as a direct result of thrashing

What next?

• Increase physical memory - this might be the easy one to start with, although, if you already own a large real estate chances you need to go back to design room as this might just delay the problem
• Reduce overall memory usage - think right data type, de-duplication, effective (de)serialization
• Improve memory locality - think about your choice of algorithm based on data access pattern to reduce page fault

JVM Pressure - Context Switching Overhead

Context Switching (CS) is a valuable service provided by the underlying Operating System. It prevents greedy processes from CPU hogging, time-shares CPU between multiple threads/tasks/processes to create an illusion of continuous progress. However, the suspension of the first process and scheduling of the second one requires the kernel to store the state of the first process and load the state of the second one. The overhead and time required for the same is referred to as context switching overhead.

Why bother?

• With increasing context switching the probability of context data available in CPU's cache is small and therefore needs to be fetched from main memory which is way more costly operation than fetching the same data from CPU cache. Further, the currently resident data in CPU cache needs to be committed to RAM. Together, these two operation per context switch make the problem grave.

• Increases the general user space processing. A typical user space that requires instructions and data from CPU L1/L2 cache will now need to fetch the same data from main memory.

• Increases latency for the process as explained above.

How to interpret data?

CS can be classified as below;

• Voluntary - context switch can be triggered by the process making itself unrunnable, such as by waiting for an I/O or synchronization operation to complete. Thus, this value can be used to infer frequency of blocking calls in your application process. 
• Hints: think non blocking, asynchronous, reactive modes
• Pre-emptive/Non-Voluntary - signify processes contending for the CPU and having to be switched out even though their task had not completed. Simply put, there are far too many threads than necessary.
• Hints: reduce threads, use thread pools, reduce task granularity

Exception Handling

Application Logging

Pass maven properties to maven-antrun-plugin

Problem:

You want to bundle a your application jar files with other software artifacts but as maven names your jar file as magic-app-${project.version}.jar your maven-antrun-plugin fails to pick it up on its own.

Solution:
This was a persistent irritant as every time we upgraded our applications' version we needed to manually enter the same version name to build.properties and only then we can package latest jars for release.

After exploring certain unsatisfactory solution like;

• using-maven-to-output-the-version-number-to-a-text-file
• how-can-i-get-maven-to-write-a-property-into-a-property-file-at-package-time

Finally, got some hints from;

• maven-antrun-plugin examples 

Update your pom as shown below to export maven properties to ant build at run time.

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-antrun-plugin</artifactId>
        <version>1.7</version>
        <executions>
          <execution>
            <phase>package</phase>
            <configuration>
              <target>
                <property name="release.version" value="${project.version}" />                <ant antfile="build.xml" />
              </target>
            </configuration>
            <goals>
              <goal>run</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

 Hope that helps.

SBAs - Search Based Applications

Googling for the above term (as of this day) does not returns quality results. However, if you search for Search Based Applications (SBAs) you will hit a wikipedia entry and certain other articles which have used the similar term. You will even find a book by the same name on Amazon. To sum, it simply means that my thought process is cutting edge ;) ...come on, this is my blog, I can brag!

But, I'm not kidding, the idea in itself is very simple... just use right data-structure(inverted indexes) for read-intensive jobs and it dramatically simplifies your application architecture and brings God sent agility to your deliveries. Let me try to explain that...

In traditional database-centered-applications data stored in relational databases can only be retrieved through SQLs, which is a good thing as it standardizes the information access. However, let us try to contrast them with SBAs;

• If the desired information is too complicated to derive from the existing data, think query complexity, frequency for the requested information, scalability, while SBAs can process both structured and natural language queries and then provide faceted navigation which allows users to slice-n-dice data in innumerable ways.
• If the needed query does not exists and as such users will need to wait for the completion of next sprint while in the other case if you have indexed the data, the information is mostly available. Think fast time to market, high agility!
• Forget mulitple tables, think about a scenario where in you need collate information from multiple data sources read emails, blog entry, cvs, pdf, office documents, simply put if you can groke text SBAs can digest them to seek intelligent information later.

As if it was not enough motivation for us to experiment more with SBAs here is some more;

• SBAs are more fault tolerant to user queries and can make spelling suggestions, return results for synonymous queries
• You can integrate your SBA with some statistical based system to identify popular entries or make recommendations using clustering and collaborative filtering
• You can even filter results which are not suitable for user think access permissions, adult content, exclusively for club members

Mind you all this talk is not just about 'search text box' generated results the usage can be more implicit based on user's context which may be driven by application specific use cases like content browsing.

Today I've just scratched the surface, more later..

Do let me know your queries and feedback.

REALSEX

REALSEX a.k.a REAL eState EXchange

Oh! so you finally ended up being inquisitive.. You Peeping Tom!! Never mind, I was just experimenting with the idea of putting some thing scandalous as the blog title to attract readers.. Gotch Ya! :)

This blog entry is actually about my new found big ticket Startup !dea. Well for those of you who know me might frown.. isn't it? You see for past few days I'm observing that there is rising trend of this particular keyword being thrown upon me from all sides to such an extent that if you ask me to summarize my day today in one word I would say "START-UP".

Allow me to explain, I learnt a tonne about start-up, venture capitalist, incubation, mentor, angel investors at break-fast and then later in the day, all the way to my office I came to learn about Karnataka Government's initiative to bring fresh investments in the state through smiling faces of our CM on roadside-makeshift-hoardings, as if that was not enough my kiddo-colleague declared his next job title is going to be CEO of his new found organization. Wow!

Well I'm trying to scratch the personal itch of identifying a spot for constructing my house in Bangalore, but I discovered that not only it is painstakingly difficult identify properties which are up for sale it is all the more difficult to verify and validate all the meta-data (read ownership, water source, sanitation, future prospects etc.) associated with the property to ascertain its purchase worthiness. One would have to run pillar to post to collect minimal data (read land records) to seek loan from banks however other information as I mentioned above is at best adhoc in nature and is definitely not available at one window.

Real estate portals in its current form cater to information need for availability of properties and related news about Govt. policies (taxation etc) and proposals (read upcoming metro or an IT park) which might affect the property prices in the region. However, it is all left to the user to make intelligent decisions without really providing an avenue to collate this information in a digestible format, so that it becomes difficult for dummies like me to correlate all the information and generate some sort of index which I may use as pointers to make purchase decision say should I go Bangalore North because future lies with 'Aerotropolis' or should I go South because "Kanakpura Road is the new Jaynagar!!"

Would it not be better if we had a single-window-trusted-source-of-information where in we can seek answers to our queries?!

Let us now think about developing one such user platform in a phased manner;

• Aggregate data available from dependable public sources like rainfall patterns, vegetation, soil type, depth of water table, civic amenities and put them on map after geo-encoding.
• Create 'event-listeners' to collect data from events like news feeds, user forums, govt plans and establish intelligent correlation with above mentioned slow changing data to determine actionable information. For e.g. Govt. approved a new metro station at 'X' now if you overlap this information water availability in the area on a map it becomes pretty useful isn't it!
• Create property index. 
• Rank them for different purposes and ensure environmental sustainability in your recommendations.
• Cultivate a community of users, information seekers, who would like to learn from your applications' knowledge-base.
• Create avenues for these user's to provide feedback/contribute to the community. Gamify it.
• Allow users to publish/search individual property.
• Motivate users to provide meta-data, like do you have rain-water harvesting in your premises, give them more points for that .. improves value on property index.
• Suggest corrective actions to improve property ranking, may trigger improvements at a macro -level. Like if citizens in a particular locality identify that improving cleanliness in their locality will fetch better returns for their investments, might as well improve general hygiene in the entire locality.
• ...

Simply put the platform is required to ensure high quality governance of the eco-system so created.

Now the impatient among you might already want to return from the inception point and see money. OK, let me try my hands in explaining the business plan as well.

• Free for all, always!
• Multilevel premium charges for activities like identifying (organic recommendation i.e. no promotions) potential properties, generating reports based on fact (advisories) on identified properties etc.
• Advertisements
• Service providers say plumbers or builders
• Classifieds

Was it too big for an elevator pitch? Nevertheless, it feels good to see light at the end of the tunnel. I feel like God! .. err I must sleep now as I have a job to keep tomorrow in lieu of 'Bharat Bandh'

Do share your feedback, because it is important.

Mongodb Paging Item Reader

Here, is a sample..

package org.zero.index.batch.read;

import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.item.database.AbstractPagingItemReader;
import org.springframework.data.mongodb.core.MongoOperations;
import org.springframework.util.Assert;

public class MongodbPagingItemReader<T> extends AbstractPagingItemReader<T> {
  private MongoOperations mongoOperations;
  private QueryContext<T> queryContext;
  private boolean strict;

  private static final Logger LOG = LoggerFactory
      .getLogger(MongodbPagingItemReader.class);

  @Override
  protected void doReadPage() {
    if (results == null) {
      results = new CopyOnWriteArrayList<T>();
    } else {
      results.clear();
    }
    final List<T> result = mongoOperations.find(
        queryContext.getQuery().limit(getPageSize()).skip(getPage()),
        queryContext.getEntityClassName(), queryContext.getCollectionName());
    if ((getPage() == 0) && result.isEmpty()) {
      if (strict) {
        throw new IllegalStateException(
            "No matching documents found (reader is in 'strict' mode)");
      }
      LOG.warn("No matching documents found");
      return;
    }
    results.addAll(result);
  }

  @Override
  protected void doJumpToPage(final int itemIndex) {
    // do nothing
  }

  /**
   * @param mongoOperations
   *          the mongoOperations to set
   */
  public void setMongoOperations(final MongoOperations mongoOperations) {
    this.mongoOperations = mongoOperations;
  }

  /**
   * @param queryContext
   *          the queryContext to set
   */
  public void setQueryContext(final QueryContext<T> queryContext) {
    this.queryContext = queryContext;
  }

  /**
   * @param strict
   */
  public void setStrict(final boolean strict) {
    this.strict = strict;
  }

  @Override
  public void afterPropertiesSet() throws Exception {
    super.afterPropertiesSet();
    Assert.notNull(mongoOperations, "mongoOperations is null");
    Assert.notNull(queryContext, "queryContext is null");
  }
}

Hausla Buland

जैसे कंधे पे इक दोस्त का हाथ हो 
जैसे  लफ्ज़ो पे दिल की हर एक बात हो
जैसे आखोँ से चिंता की चिलमन हटे
जैसे मिट जाए हर ग़म कुछ इतना घटे
जैसे राहोँ में सपनो की कलियाँ खिले
जैसे दिल में उजालो के दरिया बहे
जैसे तन-मन कोई गीत गाने लगे
जैसे सोई हुई हिम्मत अंगडाई ले
जैसे जीना ख़ुशी की कहानी लगे
जैसे खुल जाए रस्ते अब तक थे बंद

हौसला हो बुलंद
हौसला हो बुलंद
हौसला हो बुलंद
हौसला जैसे आस्मां से हो बुलंद
हौसला हो बुलंद
हौसला हो बुलंद
हौसला हो बुलंद
हौसला जैसे आस्मां से हो बुलंद
हौसला हो बुलंद
हौसला हो बुलंद
हौसला हो बुलंद

हो बुलंद.....

कवि -- श्री जावेद अख्तर 


You like to grab your own copy here

Spring Managed DI for Quartz Jobs With Dependencies

Problem Statement:

Quartz scheduler (v 2.1.3) triggers a job which has its own dependencies which needs to be managed using Spring (3.1.1.RELEASE+)

Solution:

The Job

public class HttpDealsDownloadJob implements Job {

// NO DEPENDENCIES HERE

  @Override

  public void execute(final JobExecutionContext context)

      throws JobExecutionException {

    final JobDataMap jdm = context.getJobDetail().getJobDataMap();
// EXTRACT YOUR DEPENDENCY FROM JOB DATA MAP

    final HttpFileDownloader delegate = (HttpFileDownloader) jdm

        .get("job.delegate");

    delegate.apply(context.getJobDetail().getJobDataMap().getWrappedMap());

  }

}

Spring Context

<bean id="mySchedulerFactoryBean"

class="org.springframework.scheduling.quartz.SchedulerFactoryBean">

<property name="configLocation" value="file:quartz.properties" />

<property name="jobFactory" ref="springBeanJobFactory" />

<property name="triggers">

<list>

<ref bean="myCronTriggerFactoryBean" />

</list>

</property>

</bean>

<bean id="springBeanJobFactory"

class="org.springframework.scheduling.quartz.SpringBeanJobFactory" />

<bean id="myCronTriggerFactoryBean"

class="org.springframework.scheduling.quartz.CronTriggerFactoryBean">

<property name="cronExpression" value="${my.cron.expression}" />

<property name="jobDetail" ref="myJobDetailFactoryBean" />

</bean>

<bean id="myJobDetailFactoryBean"

class="org.springframework.scheduling.quartz.JobDetailFactoryBean">

<property name="jobDataAsMap">

<map>

<entry key="source.url" value="${my.deals.url}" />

<entry key="job.delegate" value-ref="httpFileDownloader" />

</map>

</property>

<property name="jobClass"

value="org.zero.gb.acequia.scheduler.HttpDealsDownloadJob" />
<!-- INJECT APPLICATION CONTEXT -->

<property name="applicationContextJobDataKey" value="appicationContext" />

</bean>
<!-- THIS IS THE CLASS WHICH ACTUALLY DOES THE WORK -->

<bean id="httpFileDownloader" class="org.zero.acequia.integration.http.HttpFileDownloader">

             <!-- your dependencies here -->

     </bean>