CraftConf 2014 – Way of the Agile Developer

So…yes…indeed we all need or want to learn the “Way of the Agile Developer“, don’t we?

CraftConf - AgileManifesto

It seems to be quite a trend nowadays to act fast, build quality software, meet marketing expectations, build up mammoths and of course get decent piece of ham in terms of revenue for you business.

It surely sounds easy with Agile…BUT IT’S NOT!!!

These things were pointed out very well (in my opinion and the opinion of my colleagues) by some of my new gurus – Gojko Adzic(slides and stream) and Dan North(slides and stream). Many other speakers enforced the main ideas shared by these guys, but at least for us they proved to be the best fit.

So to summarize what you’ll learn if you dare to watch the streams:

  • agile is all about Flexible Scope – if you have no flexibility where is the agility???
  • if you are agile it doesn’t mean you can predict everything – at least 3 dimensions are unpredictable – time, humans, and local
  • make sure you take time to get mastery
  • variation – seek out new ideas and try some new things
  • survivability – do it on a scale where failure is survivable
  • selection – seek out feedback and learn from mistakes

I’ll dare you to take a look over the slides and watch the streams and I’ll guarantee it will be worth your time.

Thank you Dan North and Gojko Adzic, you’ve surely put some cornerstones in my head!

CraftConf 2014 – Review Intro

Several months have passed since my first international conference – CraftConf 2014 Budapest, a nice conference about “Software Craftsmanship”. To be honest I had no idea on what to expect from such a conference so close from home. Was it worth it? Who were the speakers? Was it well organised?

And the answer to all the questions above is a big YES! YES it was totally worth it, YES there were lots of top notch speakers, YES it was well organised!

In the next few articles I will do my best to stress out what I’ve found out there, what speakers are my gurus now and why I will do my best to be there at CraftConf 2015.

MongoDB and Spring-Data II

And here comes the follow-up for the previous article regarding Mongo, Java and Spring-Data.

The previous topic brought us to the point where we have an initial setup and we are ready to give it a go to our webapp.

The first thing that you need before achieving this is setting up the Home Page:

And the HomeController:

The snippets of code above define the two things that we need most in our webapp, the starting page and the URL of that page. In this case the controller will handle how and when the homepage should be loaded. Since you have a viewfilter defined (there is an example of how to achieve this in the previous article) you can access the homepage jsp by simply returning the name of the view as a string such as return “home”;

To test your web application you can install the VMware vFabric plugin and drag and drop your project in the Servers view in STS.

Then if you surf in your browser at http://localhost:8080/springmvc/ you should see the page bellow (if by any chance you can’t see, please let me know and I will gladly help you):

helloworld

By now you should have the minimum required setup of a web application built with Spring MVC so it is time to get going with the Mongo required configurations.

If you have everything configured as in the examples above and in the previous article, the next steps should be straight-forward.

First of all setup the MongoConfig bean as above. This bean will configure the mongo template with the URL of the mongo instance, the database associated with the given config and the package in which it should scan for @Document annotated classes.

NOTE: If you have multiple databases configured you need to have multiple configuration classes!!!

Then you need to add your domain object that will map over a Mongo collection (traditionally called table), so that you have a place for storing documents (traditionally records/rows/etc.). That is why the class is annotated with @Document (as seen in the example bellow) and the fields are annotated with @Field.

And of course the last step of this topic is persisting the entity into the database:

The classes that handle document persisting are named repositories. Such classes need to be annotated with @Repository and are used as a wrapper over the mongoTemplate (in this case but it can be any other underlying database client that’s handled by spring-data).

So by now you should have a running instance of a SpringMVC app that can store records into a NoSQL databases – MongoDB. To test your application you can hit the URL specified above and then check your local instance of MongoDB. Refresh the page several times and you should see several records inserted.

NOTE: I will not cover mongoDB setup and installation at this time, so for the time being just head on at the MongoDB official website and follow the steps listed there.

In the next chapter we will go in-depth with the Spring-Data tutorial; since I will show you how to get ready to process BigData with a MongoDB, so we will start talking about Map-Reduce.

MongoDB and Spring-Data I

Since I’ve touched this topic in a previous post. It’s about time i show you some steps required in effectively using MongoDB.

First of all MongoDB offers seamless integration with a large set of programming languages, through a series of clients built for them. But in the case of this tutorial I will be talking about Java.

Minimum requirements:

  • java JDK-1.6 minimum
  • eclipse or spring tool suite (or whatever IDE you prefer here)
  • tomcat 6+ (older versions might not work as expected, and of course if you prefer other containers, feel free to use them)
  • Maven 3.x (should work with 2.x but still i would rather go with the latest and greatest)

The rest of the dependencies will be written bellow with a sample project from A-Z.

If you have STS installed you must setup a Spring MVC Project and make sure your pom.xml contains the following:

The configuration above should enable minimum usage of the Spring infrastructure required to implement a Spring MVC project with MongoDB integration.

The second step would be to configure the web.xml as bellow:

After writing the above in your web.xml you need to configure the servlet/servlets that will handle your requests.

As it can be seen in the web.xml above, the name of the servlet is appServlet, thus you must add a file under WEB-INF, either directly in a file following the naming convention appServlet-context.xml, either as shown above in a folder named appServlet containing a servlet-context.xml.

The content of the servlet-context.xml can be seen bellow:

The definition above will define what Spring Beans will be loaded in the context of your servlet. However i will not cover the contents of how and why we should use beans and dependency injection – maybe in a follow-up article.

All you need to know is that the configuration above sets the Spring MVC Project as annotation aware, defines the main folder for resources that will be used by our app (css files, javascript files, property files, etc.), defines a view filter (capable of mapping jsp or html files so that they are accessible from the Spring Controllers) and sets the component scan to a specific base-package (the component scan handles auto-wiring beans from the specified packages so that we don’t have to initialize them by hand).

Probably the last step in Part 1 and the most important if you are thorough and organized, is setting up the root-context.xml or as it is commonly known the applicationContext.xml. This context file has the sole purpose of storing configurations or beans initialization that should be used in all your servlets so that you avoid loading the same bean multiple time in each of them.

For the test application we are building the minimum required setup for now is:

This is it for setting up the initial project, however in the second part of the article i will talk more about preparing your application for Mongo Development.

MongoDB

MongoDB is part of the NoSQL generation of databases. As described on kkovacs.eu the best use case for it is:

    If you need dynamic queries. If you prefer to define indexes, not map/reduce functions. If you need good performance on a big DB. If you wanted CouchDB, but your data changes too much, filling up disks.

For most things that you would do with MySQL or PostgreSQL, but having predefined columns really holds you back.

I agree with most of the things written above, well most but one thing: mongodb and map/reduce.

The use case above was described somewhere in Q1-Q2 this year(2013). Of course MongoDB had a lot of limitations:

  • slow map/reduce
  • newly introduce Javascript engine(should provide at least some parallelism in terms of jobs/operations on the same collection)
  • slight improve in performance

But this was just the first major release with the new Javascript engine. I’ve tested a Map/Reduce on a collection that was growing around 1 million documents a day. On version 2.2 which is described in the article it took about one hour to achieve this task with around 30 million existing records.

A simple migration to MongoDB 2.4.5 earlier this year drastically reduced the time to around 15 minutes top. But still this wasn’t enough. One of our requirements was to provide analytics at a fast pace, i would say almost “real time” (in terms of map/reduce real time is relative, in our case it can be anywhere from 5 to 15 minutes).

With the volume of data increasing at a fast rate I had to see if there is a better way of achieving this…and after 2 weeks i finally came up with a solution. After searching the World Wide Web like crazy i managed to work out a solution. I will talk more about it in 2 future blog posts which describe the two mechanisms that i had to build for this solution.

So stay tuned for more info this week 😉

Upload CommonsMultipartFile with jUnit and Spring-Test

One of my recent endeavors was to upload a file from a unit test without mocking the process and to simulate the behavior of a form submission.

Of course the web is full of solutions on how to achieve this if you actually have a form…but if the form is gone…well good luck there.

I’ve tried my luck on stackoverflow and unfortunately all i got was a NullPointerException:

Oh and how pleased i was to write the content above and try around 20 scenarios all with the same result. And then out of nowhere i found a post somewhere on the web which eventually led me to the solution i was seeking:

So guess what, writing simple stupid code actually works 🙂 and all the mumbo-jumbo code above was reduce to 3 lines of code that achieve the proper result. I also found out that by implementing the solution above, Spring will deserialize the FileSystemResource to a CommonsMultiPartFile without having to do any extra work 🙂

Google Custom Search API

Quite recently i needed to find a way to extract a lot of information from the World Wide Web. I was working for my masters final paper and needed a lot of file in SQL format so that i may process them and extract information.

The problem was in how should i get an amount of data that big? Well after searching a bit i found the answer: Google Custom Search API.

I’ve searched for examples on how to use the API but unfortunately most were not complete or using deprecated versions of the API. Also you can parse the output in JSON format but in the newer versions of the API you have to implement a JSON factory to map it to my Java code. Well i wanted something much faster and with little more flexibility since i only wanted to extract the URL’s and next page from the output JSON.

So to start using the seach api, first of all you need to secure some prerequisites:

So in the example above i was looking for filetype:sql, meaning all results should be files in SQL format.

The next step needed is a HTTP connection towards the URL we construct above:

We open a connection on the search URL, tell the connection to expect results in JSON format and since the Google Search API is using the REST template, we must also specify the REST method expected; in this case  of “GET” type.

After we have a connection opened what we need next is to establish a buffered data stream and start processing the data:

By this point we have the result of the search query buffered. The next step would be to extract the information we need from the buffered input stream. And since we won’t be using a JSON Factory we need to process the output line by line. A thing that can be achieved like in the snippet below:

In this snippet what I’ve did was to parse the stream line by line and search for 2 significant groups for me: the “totalResults” and “link” groups in the output JSON. The first one is needed to compute the page i am on, and the second to extract the information i need. As you can see on the last line of the second If, I add the link to a list of strings.

If you want to go further than the first page of search results you need to wrap the code above in a method with the following signature:

And after wrapping the method you need the snippet below which increments a counter with 10 items (google only allows 10 items a page in the search result, thus you need exactly the maximum free of 100 api calls to extract 10 pages of information) and after that i call the method recursively.

And now to test the code above you can use:

This was the final step. Normally if you properly configure a custom search engine and get an API key from Google the above code should return you a list of strings containing the URLs from the results.

Thank you and i hope it helps you.