Saturday, January 11, 2014

Java 8 lambda with combiner

Lambda expression and functional programming are advanced features provided by Java 8. Here I would like to explain why and how the combiner is used in a practical example.

        
public class Person {

    String name;
    int age;
    Sex sex;
    .....


        Person p1 = new Person("Shaozhen", 5, male);
        Person p2 = new Person("Mike", 10, male);
        Person p3 = new Person("Darin", 20, male);
        Person p4 = new Person("Alice", 25, female);
        Person p5 = new Person("Mark", 30, male);

        List<person> persons = new ArrayList<person>();
        persons.add(p1);
        persons.add(p2);
        persons.add(p3);
        persons.add(p4);
        persons.add(p5);


The above code initializes a person array with five persons that person has name, age and sex attributes. Now the problem is to use lambda/function [1] to find the average age of male persons. In order to show the usage of combiner we are using a custom IntCustomer.
 
class Averager implements IntConsumer
{
    private int total = 0;
    private int count = 0;

    public double average() {
        return count > 0 ? ((double) total)/count : 0;
    }

    public void accept(int i) {
        total += i; count++;
    }
    public void combine(Averager other) {
        
    }
}

Then we can use lambda to filter the array and calculate the average.
 
    persons.stream().filter(p -> p.getSex() == male).map(p -> p.getAge()).collect(Averager::new, Averager::accept, Averager::combine).average()
The above code is to get the collection's stream (laziness since the filter operation, we do not want the mapping function to apply to all the items). collect method basically evaluates the lazy expression with arguments of method references. Averager::new, how to initialize the return object, Averager::accept, how to accumulate the item for each item. The implementation of the Average is straightforward. Whenever scan a new item, person's age accumulated into the total variable and count variable plus 1. Then total divided by average resulted the average age.

 However, assuming the persons array is a huge array and the computer has multiple cores. How the program takes advantages of parallel processing of multi core and advance the processing speed? Since Java 7 there is a fork/join [2] paradigm that split the workload recursively and aggregate the results. Java 8 Stream provides a way to process the collections in parallel with fork/join.
    persons.parallelStream().filter(p -> p.getSex() == male).map(p -> p.getAge()).collect(Averager::new, Averager::accept, Averager::combine).average();
The above code uses parallelStream to mark the processing way as fork/join. However, After running the program, we got a wrong result which is 5. The reason is that we have not implemented the combiner method. So when combine method applies, it always do nothing and conserves the left argument result, which is the first male person's age as 5. Now let's implement the combine method and get the correct result.
    
    public void combine(Averager other) {
        total += other.total;
        count += other.count;
    }

Along with above discussion, the keyword of Java 8 is about lambda, function programming, stream, fork/join. The combiner here is very similar with map/reduce's [3] reduce step.

Reference:

  1. Lambda tutorial
  2. fork/join tutorial
  3. Map Reduce

Tuesday, October 15, 2013

Cloudfoundry makes continuous integration easy

Cloudfoundry works as a PAAS that makes dev opts work much easier for continuous integration.

Without a good PAAS, setting up a stable and reusable build environment requires a lot of efforts. For example, the build requires a tomcat container setup, the Jenkin slave machine needs to avoid the tomcat port conflict, the web application has other service dependencies for testing, needs profiles to separate the different environments and injecting to your pom.xml and so on.

I will describe an experiment of using cloudfoundry and maven to do a clean integration test.

Prerequisite:

  • Register a cloudfoundry account https://console.run.pivotal.io/register with a trial account or you can create a light weight cloudfoundry by using https://github.com/cloudfoundry/bosh-lite (Recommended since you can get better understanding cloudfoundry principles and components)

Setup the POM (example as https://github.com/datianshi/spring-mvc-service with cloudfoundry branch):
  • Add cloudfoundry credentials to your maven settings.xml
<servers>
        <server>
          <id>mycloudfoundry-instance</id>
          <username>${username}</username>
          <password>${password}<password>
        </server>
</servers>


  • Add cloudfoundry maven plugin (cf-maven-plugin) for pre-integration-test and post-integration-test
<plugin>
	<groupId>org.cloudfoundry</groupId>
	<artifactId>cf-maven-plugin</artifactId>
	<version>1.0.0.BUILD-SNAPSHOT</version>
	<configuration>
                    <server>mycloudfoundry-instance</server>
                    <target>http://10.244.0.14:9022</target>
                    <org>myorg</org>
                    <space>myspace</space>
                    <appname>${app}</appname>
                    <url>${app}.10.244.0.254.xip.io</url>
                    <memory>256</memory>
                    <buildpack>git://github.com/datianshi/java-buildpack.git</buildpack>
	</configuration>
	<executions>
		    <execution>
			  <id>pre-integration</id>
			  <phase>pre-integration-test</phase>
			  <goals>
			  	<goal>push</goal>
			  </goals>
			  </execution>
		    <execution>
			  <id>post-integration</id>
			  <phase>post-integration-test</phase>
			  <goals>
			  	<goal>delete</goal>
			  </goals>
	            </execution>			  	
	</executions>
</plugin>

The above plugin has two executions, one is set as pre-integration-test life cycle with a push goal to set up a tomcat environment. The application can be configured with appname, url and so on. What does it actually happen during push inside of cloud foundry? In short story, the cf client side upload the war and give that to the cloudfoundry (specified in the target url), and cloudfoundry is smart enough to create a ready tomcat container based on the build packs, url(it has internal DNS server) and app name. For the interest readers, please refer cloudfoundry architecture and how-applications-are-staged




  • Add maven fail safe plugin to run your integration tests.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
        <skip>true</skip>
</configuration>
<executions>
        <execution>
                <id>integration-tests</id>
                <goals>
                        <goal>integration-test</goal>
                        <goal>verify</goal>
                </goals>
                <configuration>
                        <skip>false</skip>
                        <includes>
                                <include>**/*IntegrationTest.java</include>
                        </includes>
                        <systemPropertyVariables>
                                <integrationUrl>http://${app}.10.244.0.254.xip.io</integrationUrl>
                        </systemPropertyVariables>
                </configuration>
        </execution>
</executions>
</plugin>

Since we parameterized the app name, we could just pass in the url to our test and the test knows which url to perform test. Because of the existence of router and dns server in cloudfoundry, we would not worry port configuration/collision for tomcat, since every jenkin build we just pass in different app name (E.g. build number), then the build executes concurrently.

To run the build: mvn clean install -Dapp=test

Cloudfoundry is also flexible to integrate with other application containers (Controlled by buildpack). Another feature called services (As your application dependencies E.g. mysql, redis....) could be configured and deployed to cloudfoundry environment so your application could be easily bind to during push with minimum configuration. For the readers interested with cloudfoundry deployment and release management, please refer bosh





Tuesday, July 30, 2013

Junit multiple users functionalities with TestRule interface

Recently I was trying to do some performance testing integrated with build. There are some use cases that I want to simulate multi users(Threads) behavior inside my junit test cases to have multiple threads executing the tests concurrently.

Then TestRule, a built in Junit interface which brought my attention, is easy and powerful to customize your test code behaviors.

TestRule
As I understand, the TestRule is an interface which developer can inject into test classes, which is similar to the aspectj Around point cut.

The code below implements TestRule and return an inner class as CustomStatement that injects some code before and after the testing code (base.evaluate())

public class SimpleTestRule implements TestRule{
 
 public Statement apply(Statement base, Description description) {
  return new CustomStatement(base);
 }
 
 private class CustomStatement extends Statement{
  private Statement base;
  
  public CustomStatement(Statement base) {
   this.base = base;
  }

  @Override
  public void evaluate() throws Throwable {
   //Before test execution
   System.out.println("Before Test Running");
   //Test execution
   base.evaluate();
   //After test execution
   System.out.println("After Test Running");
   
  }
  
 }
}

Use the above Custom Test Rule
public class SimpleTestRuleTest {
 
 @Rule
 public TestRule rule = new SimpleTestRule();
 
 @Test
 public void test(){
  System.out.println("My test is running");
 }
}

Code execution Result:

Before Test Running
My test is running
After Test Running


Multiple User TestRule

After understanding the above simple test rule, we can start create a more complicate rule to support multiple threading of tests.

The following code wrapped the test execution code (base.evaluate()) into an independent thread.

public class MultiUserTestRule implements TestRule{
 
 private final int numOfUsers;
 private final ExecutorService executor;
 
 public MultiUserTestRule(int poolSize, int numOfUsers){
  this.numOfUsers = numOfUsers;
  executor = Executors.newFixedThreadPool(poolSize);
 }

 public Statement apply(Statement base, Description description) {
  return new CustomStatement(base);
 }
 
 private class CustomStatement extends Statement{
  
  private Statement base;
  
  public CustomStatement(Statement base) {
   this.base = base;
  }

  @Override
  public void evaluate() throws Throwable {
   for(int i=0; i< numOfUsers; i++){
    executor.execute(new Thread(new Runnable() {
     
     public void run() {
      try {
       base.evaluate();
      } catch (Throwable e) {
       throw new IllegalArgumentException(e);
      }
     }
    }));
   }
  }
  
 }

}
Use the above test Rule
public class MultiUserTestRuleTest {
 /**
  * The thread pool size is 10
  * There are 5 users run the tests
  */
 @Rule
 public TestRule rule = new MultiUserTestRule(10, 5);
 
 @Test
 public void test(){
  System.out.println(Thread.currentThread().getName() + " is running the test");
 }
}
The above test case is trying to print out each current thread name for demo purpose.
The result is:
pool-1-thread-1 is running the test
pool-1-thread-4 is running the test
pool-1-thread-5 is running the test
pool-1-thread-2 is running the test
pool-1-thread-3 is running the test

All the above code can be found here: https://github.com/datianshi/multiuser-junit

Tuesday, July 16, 2013

Spring MVC testing with session scoped beans

Spring MVC test is a new feature from Spring 3.2 that could help easy function/integration testing with the controller without bring up a real server. However, when controller autowiring  a session scoped bean and some controller request mapping methods have operations on the bean, how to make assertion on those operations becomes tricky. Here I will propose an approach to test against it.


Assume there is a session scoped bean configured as below:

@Component
@Scope(value= "session", proxyMode = ScopedProxyMode.TARGET_CLASS)
public class SessionScopeBean {
 
 private String value;

 public String getValue() {
  return value;
 }

 public void setValue(String value) {
  this.value = value;
 }
 
 
}
There is a controller autowired with the session scoped bean
@Controller
public class HomeController {
 
 @Autowired
 private SessionScopeBean sessionBean;
  
 @RequestMapping(value = "/sessionScope/{value}")
 public String test(@PathVariable String value){
  sessionBean.setValue(value);
  return "home";
 }
 
}
The above controller modify the value field of a session scope bean, then how would we retrieve that and assert the appropriate value in the session. Couple of tips here:


  • Spring saves the session scope bean as a session attribute "scopedTarget.${idOfTheBean}"
  • Spring mvc test provides request().sessionAttribute("attr", hamcrestMatcher) to assert the session
  • hamcrest matcher HasPropertyWithValue can be used to inject into the above expression


@RunWith(SpringJUnit4ClassRunner.class)
@WebAppConfiguration
@ContextConfiguration("classpath:test-appContext.xml")
public class ControllerTest {
 
 private MockMvc mockMvc;
 
 @Autowired
 private WebApplicationContext wac;
 
 
 
 @Before
 public void setup(){
  this.mockMvc = MockMvcBuilders.webAppContextSetup(this.wac).build();
 }
 
 @Test
 public void test() throws Exception{
  mockMvc.perform(get("/sessionScope/abc")).
   andExpect(view().name("home")).
   andExpect(request().sessionAttribute("scopedTarget.sessionScopeBean", hasProperty("value", is("abc"))));
 }
}
The above tests shows how the session attribute being asserted

Cheers

Sunday, March 10, 2013

Major Countries Military Action Principle

Found from a Chinese version:
1,美国,想打谁就打谁;
2,英国,美国打谁我打谁;
3,俄罗斯,谁骂我我打谁;
4,日本,谁打我我让美国打谁;
5,朝鲜,谁让我不痛快我就打韩国;
 6,韩国,谁打我我和美国一块军演吓唬谁;
7,中国,谁打我我就骂谁。

 Translate to English:
1. US, I will beat whoever I want.
2. UK, I will beat whoever US wants to beat.
3. Russia, I will beat whoever condemns me.
4. Japan, I will let US beat whoever beats me.
 5. North Korea, I will beat South Korea if anyone makes me unhappy.
6. South Korea, I will have military exercises with US whenever someone beats me.
7. China, I will condemn whoever beats me.

Thursday, March 7, 2013

How does IncompatibleClassChangeError happen (with sample code)



Introduction

When different teams working separately producing libraries consumed by each other ,you may run into a problem with the java.lang.IncompatibleClassChangeError exception.

What does this exception mean?

IncompatibleClassChangeError means the class in the classloader is not the same format as the one compiled. (class vs interface)

How does this happened?


E.G.  
  • Your application "app" with dependency jar b in class path.
  • A is your code in app which has a reference with class B and compiled with b.
  • The class B in b-beta version was refactored to an interface
  • When you run your application, the code is running and load your dependency with b-beta.
  • Since the B as an interface not same as the one compiled as a class. Bang!~ The application throws the exception.

Sample code with Maven



I made an example to demonstrate how this happened. The code sample can be found here:
https://github.com/datianshi/maven-inCompatibleClassChangeError

The five repos are: version1, version2, framework1, framework2, app

You can run maven clean install with the above sequences in your local box

Version 1 and Version2 have the same group id and artifact id, while the version is not the same (Here in order to simulate multiple version with different code locally, I created two repo for the same group and artifact). version1 has an interface Person, while version2 has a class Person.

Framework1 depends on version1 Person, while Framework2 depends on version2 Person.

The app depends on both framework1 and framework2. When maven loads the dependencies, it has to resolve the conflicts for version1 and version2, because they are having the same artifact and group. Maven uses "nearest win" policy to resolve the conflicts, then version1 wins.

When the app loads Framework2, it was looking for class Person, but the classloader contains an interface Person. The application failed like below.

mvn exec: java inside of app folder.


Caused by: java.lang.IncompatibleClassChangeError: Found interface org.freelance.incomp.core.Person, but class was expected
at org.freelance.version2.SayPerson.sayPerson(SayPerson.java:9)
at org.freelance.incomp.app.App.main(App.java:12)













Tuesday, October 23, 2012

Create your own maven integration test plugin

Did you ever user maven tomcat plugin to perform your integration test. Start your embedded tomcat server and do the integration test and then shut down the server. There is another popular cargo project doing the similar things.

What if you want to develop your own integration test with your minimal dependency and code effort. This article may be a place for you to start.

Generate the plugin project by using maven plugin archetype

mvn archetype:create-from-project \
-DartifactId=echo-maven-plugin \
-DarchetypeGroupId=org.apache.maven.archetypes \
-DarchetypeArtifactId=maven-archetype-mojo

This maven archetype will generate a project with a default mojo, that we can
start development with. In this example I am going to use a socket echo server to demo the integration-test.

The Mojo which is an interface with a method execute() is the core execution path for maven plugins. Basically each goal attached with a Mojo. And the Mojo class can be annotated to express the goals, phases and properties.

Here is our start goal mojo StartMojo.java

The annotation of @goal tells maven which Mojo will be used during execution. The @phase will tell maven default phase to execute if no explicit phase configured.

This class will run the Echo Server in a backend thread listening on a server
socket. Then we create a StopMojo.java



This mojo has a stop goal and will shut down the echo server.
Similarly, we can have a run goal in case we want the echo server to run in the main execution rather than the backend thread. So it will block the maven execution for manual testing.




how are we going to use the plugin? We need to build the plugin by "maven clean install" to install the plugin into the local repository.

Now we can use the plugin to run the integration test. I created another maven project called integration-test. Here is how we configure the plugin usage



Then we create the integration test code. IntegrationTest.java



We need to exclude the test during the test lifecycle, since during the "test" life cycle the server is not up ready (because we start the server at pre-integration-test life cycle, which is after "test" life cycle happening).


Then we will include the test in integration-test lifecycle by using the fail-safe plugin. The reason we are using the fail-safe plugin for integration test is to make sure the build continue when the test failed, so the post-integration-test is triggered and the server can be elegantly shutdown.



If you want a standalone server running triggered by your maven plugin. Just run "maven echo:run", then the socket is running and waiting for connection.

This above example source code can be found here:
Source Code

References:
Guide of maven plugin development

Maven fail safe plugin

Maven life cycle

mojo-api-specification