I have made a lot of progress on my Google Summer of Code Project.

First, I've completed the code for the generation of the directory structure for the forms. This is done using the form id, which is the name with all illegal characters removed. The only valid characters left are alpha-numeric and periods. Let me tell you, the regular expression to replace all of those characters was hell to write. But, thanks to the help of Jason Davis, I was able to make it much more readable and easier digest. You've gotta see it to get an idea of how bad it was, so here goes:

   /**
     * Removes all illegal characters, then removes all spaces and makes it all lower-case.
     *
     * @param str the form name
     * @return a viable form id.
     */
    public static String formNameToFormId(String str) {
        return str.replaceAll("[\\$\\(\\)%`~!@#&;:\\^=\\+\\?
\"\\*\\\\////\\\\{\\}'\\]\\[\\|\\s+]", "").toLowerCase();
   
    }

That is pretty much hell on earth to read, and it was hell on earth to type. Here's the *MUCH* simpler version:

   /**
     * Removes all illegal characters, then removes all spaces and makes it all lower-case.
     *
     * @param str the form name
     * @return a viable form id.
     */
    public static String formNameToFormId(String str) {
        return str.replaceAll("[^a-zA-Z0-9.]", "").toLowerCase();

    }

Note, you can clearly see that i'm negating that entire sequence. It does the SAME thing, but is MUCH more readable. That reads: "replace everything that isn't a letter from A to Z *OR* a to z(lower case letters) *OR* from 0 through 9 *OR* a period. Whereas the above just excluded the invalid characters explicitly but was a MONSTER of a regular expression.

Secondly, metadata persistence is written in, I'm using xstream. It's a great tool for XML serialization.

That's it. Oh, by the way, these are the things I earmarked TWO WEEKS FOR! Now, I push domain class model processing up, will work on that tomorrow.

So, it's time for an update. This week got off to a sketchy start, but it's gained momentum. Let me enumerate what I have done thus far with the project. First, I've added AJAX using jquery. I figured I would use jquery mainly because it provided painless AJAX goodness. Prior to even thinking of using jquery, I wrote my own AJAX code using a tutorial that I found while googling. I had to tweak it a bit, but for the most part, it seemed very standard. However, after I wrote the AJAX equivalent code, it didn't feel too elegant. So, first I'm going to show the AJAX code I wrote; then I'll show you the jquery version; finally, I'm going to show you the servlet which handles the AJAX on the server-side.

First, the AJAX I wrote:

function AjaxValidation(url, callback) {
    var req = init()
    req.onreadystatechange = processRequest

    function init() {
        if (window.XMLHttpRequest) {
            return new XMLHttpRequest()
        } else if (window.ActiveXObject) {
            return new ActiveXObject("Mircosoft.XMLHTTP")
        }
    }

    function processRequest() {
        if (req.readyState == 4) {
            if (req.status == 200) {
                if (callback) {
                    chkSyntaxCallBack(req.responseXML)
                }
            }
        }
    }

    this.doGet = function() {
        req.open("GET", url, true)
        req.send(null)
    }
}
function chkSyntaxCallBack(responseXML) {
    var res = responseXML.getElementsByTagName("result")[0].firstChild.nodeValue
    document.getElementById("out").innerHTML = res
}

function checkSyntax() {
    var target = document.getElementById("groovyModel")
    var url = "${pageContext.request.contextPath}/moduleServlet/groovyforms/createGroovyForm?groovyModel=" + escape(target.value)
    var ajax = new AjaxValidation(url, chkSyntaxCallBack)
    ajax.doGet()
}

I warned you that it wasn't too elegant. Understanding this isn't too hard. Here is the call sequence: init() -> doGet() -> processRequest() -> chkSyntaxCallBack(). Not that bad. Now let's see the jquery version.

$(window).ready(function () {
    $("#groovyModel").bind("blur", function () {
        $.ajax({
            type: 'POST',
            data: { groovyModel: $("#groovyModel").val() } ,
            url: "${pageContext.request.contextPath}/moduleServlet/groovyforms/createGroovyForm" ,
            cache: false ,
            success: function(data) {
                var res = $(data).find("result").text()
                $("#out").html(res)

            }
        })
    });
})

Okay, that's much better. A few things are still happening here. When the window is finished loading, I bind my textarea element which has the CSS id of "groovyModel" to the blur event (lost focus). Then you see the AJAX. Now this is very straight forward. We're using the POST method, we're sending whatever value is inside of the textarea at the time the event is fired, we're posting to a servlet, not going to cache, and when we're done, it's printed to the screen. Very straight forward.

Now, like I said, this is all backed on the server-side by a servlet, which is written in Groovy. So here we go:

/**
 * The contents of this file are subject to the OpenMRS Public License
 * Version 1.0 (the "License") you may not use this file except in
 * compliance with the License. You may obtain a copy of the License at
 * http://license.openmrs.org
 *
 * Software distributed under the License is distributed on an "AS IS"
 * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the
 * License for the specific language governing rights and limitations
 * under the License.
 *
 * Copyright (C) OpenMRS, LLC.  All Rights Reserved.
 */
package org.openmrs.module.groovyforms.web

import javax.servlet.ServletException
import javax.servlet.http.HttpServlet
import javax.servlet.http.HttpServletRequest
import javax.servlet.http.HttpServletResponse
import org.apache.commons.logging.LogFactory
import org.codehaus.groovy.control.CompilationFailedException

class CreateGroovyFormServlet extends HttpServlet {
  def classLoader
  static final def log = LogFactory.getLog(CreateGroovyFormServlet.class)
  private static final long serialVersionUID = 066373513262051L

  @Override
  protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
    def generateTemplate = request.getParameter("template")
    def generateController = request.getParameter("controller")
    def finalMarkup = request.getParameter("markup")
    def clazz = URLDecoder.decode(request.getParameter("groovyModel"))
    def name = request.getParameter("formName")
    def version = request.getParameter("version")
    def res = this.checkSyntax(clazz)
    if (clazz) {
      if (checkSyntax(clazz)) {
        response.contentType = "text/xml"
        response.setHeader "Cache-Conrol", "no-cache"
        response.writer.write "\n\t$res\n"
      } else {
        response.contentType = "text/xml"
        response.setHeader "Cache-Control", "no-cache"
        response.writer.write "true"
      }
    } else {
      response.contentType = "text/xml"
      response.setHeader "Cache-Control", "no-cache"
      response.writer.write "\n\tPlease fill in the Form Model\n"
    }
  }

  @Override
  protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
    doGet(request, response)
  }

  @Override
  void init() throws ServletException {
    if (log.infoEnabled)
      log.info("Initializing...")
    classLoader = getClassLoader()
  }

  def getClassLoader() {
    def gcl = new GroovyClassLoader(this.getClass().getClassLoader())
    gcl
  }

/**
 * This method is used to relay errors to the user
 * @param clazz the class
 * @return the exception message or null if it was successful
 */
  def checkSyntax(clazz) {
    def sb = new StringBuilder()
    sb << "import org.openmrs.*\n\n\n"
    sb << clazz
    def res = null
    try {
      getClassLoader().parseClass(sb.toString())

    } catch (CompilationFailedException e) {
      res = "Exception: ${e.message}"
    }
    res

  }

/**
 * Check if it is result groovy code.
 * @param clazz the class
 * @return whether or not it is result groovy code
 */
  def isValidGroovy(clazz) {
    def sb = new StringBuilder()
    sb << "import org.openmrs.*\n\n\n"
    sb << clazz
    try {
      getClassLoader().parseClass(sb.toString())
    } catch (CompilationFailedException e) {
      return false
    }
    return true
  }
}

This servlet contains a lot of utility methods. One compiles, one initializes/returns the GroovyClassLoader, and of course doGet(), doPost() and init(). doGet() and doPost() both do the same thing, with doPost() simply delegating to doGet(). The code should be reasonably easy to understand. checkSyntax() returns null if it was parsed cleanly, otherwise it returns the exception message, the stack track wouldn't be useful in my case. It returns an XML tag <result> with "true" if it was successful, the exception message if it was not, and a message stating that the field must be filled in if it's empty or just not passed in. I implicitly import org.openmrs.* to allow for easier access to the OpenMRS domain model classes. The parent classloader for the GroovyClassLoader is set the servlet container's classloader. This gives me access to the classpath when loading Groovy classes.

I still have a bit to do, templating needs to be written in, for the most part it's done. Just have a few problems I'm facing, but I'll get through it. Too many people are relying on me succeeding. I feel that this project could help a lot of people, so i feel pressure to succeed. I still need to write in the "Edit" functionality of the "Manage Groovy Forms" page.

I'm definately making progress. More updates to come, that's for sure.

Okay, week 1 is nearing an end.I accomplished alot of things.

First, I completed the meta-data storage code. I wrote some tests, which revealed some bugs that needed fixing, and they were. I am happy to say, that the metadata storage works!

Writing the test was difficult due to the fact I store the forms in the system using a static List in my container class. I wound up updating to JUnit 4 to get at the @BeforeClass annotation so that the only one set of forms gets loaded into the container class. This helped me greatly. Additionally, the @AfterClass annotation was handy for cleaning up from the tests.

Secondly, Writing the code for interrogating the model was a piece of cake, thanks to groovy. All code for interrogating the model has 50 lines! That includes a model class I wrote to hold the field types and field names. I'll show you, but before I do, I should note that return statements are optional, and the final statement will be returned.

package org.openmrs.module.groovyforms.util

import java.lang.reflect.Field
import org.openmrs.module.groovyforms.metadata.model.GroovyFormsDomainModel

/*
 * Utility class containing methods for class interrogation.
 */
class GroovyFormsClassUtil {
    /**
     * Interrogates the class for all declared fields and
     * stores the type and name in a container class
     * @param the {@link Class#getCanonicalName() canonical name of the class}
     * @see Class#getCanonicalName()
     * @return a reference to a container class containing the type
     */
    static def getModel(fields) {
        def domainModel = new GroovyFormsDomainModel()
        def names = domainModel.fieldNames.&add;
        def types = domainModel.fieldTypes.&add;
        def f = fields.each {Field field ->
            names field.name
            types field.type.canonicalName
        }
        domainModel
    }
}

With groovy, it's so easy and concise (as you can see). Let's explain what's going on. First, I pass in the Field array I get from Field.getDeclaredFields(). Now, groovy adds methods onto the standard JDK classes, one of those methods is a method named each() which takes a closure. Now, back to the point, I pass a Field into the closure. That closure is executed for each element (in this case, a Field). Then I add the name, and the type to a List stored in a container class, now that container class is also written in groovy!

package org.openmrs.module.groovyforms.metadata.model

/*
 * Ths class holds information about the properties of the model. 
 */
class GroovyFormsDomainModel {

    /**
     * The field names
     */
    def fieldNames = []

    /**
     * The field types
     */
    def fieldTypes = []

}

Now, the fields are Lists, not arrays. Anyways, I've gotten off on a tangent here, so let me get back on track.

What I accomplished:


Week 1:

Code to generate the directory structure, serialization of metadata back/forth between XML and POJOs (Plain Old Java Objects), Wrote tests to ensure everything works in that regard. Additionally, I wrote the code to interrogate the domain model which I will generate the forms from.

Next Week

Write up the templates for the view/controller and write code to do the generation of the view/controller. Write some tests to ensure everything generates correctly.

Unrelated to that, my stored-value card from google came today. It feels nice to be $500 richer! This is going to be the best summer, I'm already having fun doing this. It's amazing seeing the whole project evolve into something amazing.

Apologies for not blogging as much as I should. I've focused on getting what needed to be done, done.

Tasks that have been completed thus far with a target milestone of using a mock Form, enable persistence of Questions and Values to the database:

• Mock out a report form using the domain classes
• Using the mocked up schema, generate a simple report form
• Design the SQL Schema (for just FacilityDataValue and FacilityDataQuestion)
  
• Write Hibernate Mapping files (for just FacilityDataValue and FacilityDataQuestion)
  
• Write Data Access layer (for FacilityDataValue and FacilityDataQuestion)
  
• Write Service layer (for FacilityDataValue and FacilityDataQuestion)
• Refactor the rendering logic to use the JSP and write EL function(s) to check types using instanceof.
  
• Allow a simple the mocked form from Week 1 to save the question answers.

Tasks that are are in progress, soon to be finished with a target milestone of removing the code used to mock up everything from the first few weeks; ability to use the saved schemas for rendering the report form:

• Design SQL Schema for the rest of the domain classes
  
• Write mappings for the rest of the domain classes
• Support loading the previously saved values for a form/startdate/enddate/location into a page for viewing or editing
  
• Write methods to save the rest of the domain classes in the data access layer
  
• Write methods to save the rest of the domain in the service layer

Now that I have summarized work completed and in progress, let's explain the overall design:

1. FacilityDataFormSchema serves as the overall representation of the report form in the system.
2. FacilityDataFormSection is simply that, sections on the form, e.g., monitoring equipment status, stock status of vaccinations, number of people vaccinated, etc.
3. FacilityDataFormQuestion holds metadata regarding a question.
4. FacilityDataQuestion is the question itself; it specifies the datatype; it is subclassed for each question datatype; if not subclassed, then the question is considered to be "freetext" -- in other words: just simply a text-based question.
   
1. CodedQuestion is a question that has a coded answer. This too is subclassed for each coded question datatype.
   
1. StockQuestion is exactly as the name says, to track stock of items such as vaccinations. The coded answers are: "not_stocked_out","stocked_out","expired","not_applicable"
   
2. BooleanCodedQuestion is a simple "yes","no","not applicable" type of thing; e.g., "Was there mobile clinic today?"
2. NumericQuestion is a question which has a numeric answer, e.g., "Number of Adults Vaccinated."
   
5. FacilityDataValue is what holds the values entered in the report forms for each question.
6. FacilityDataReportFormData is a non-persisted class used for retrieving the answers for a specific report for a specified period.

Hopefully this makes up for my lack of updates.

For the second year in a row, I was accepted into Google Summer of Code!!! I will be working with OpenMRS on the Facility Data Module.

My project will focus on developing tools for collecting and ultimately generating reports for aggregate data from external sources.

No, not the cleaning product! The "Web 2.0" kind of AJAX! Now where will this goodness be placed? The pages for creating the forms and managing the forms.

First off, I used Direct Web Remoting (DWR) to provide a link between my backend java codebase and the front-end javascript! It's quite amazing. You can choose to publish/unpublish the forms with the tick of a checkbox and DWR will magically set the published attribute on the form to the proper state. I then decided to go one step furthur, and provide a mechanism to backup the forms via xstream to XML. Now, with the XML files, two files are maintained, one individual XML file for each form, and one global one, which makes up the model for the entire system. The rationale for having the XML file for each form is that you can just zip up the form folder, and unzip it into another OpenMRS install and you're good to go. Adding the ability to re-generate the XML files via the management interface is especially handle so that if you make a quick change on one system, then want to install it onto another you can.

Now, DWR will be used for creating the forms as well. This will be done mostly to easily relay errors, such as syntax errors in the Controller and the Model of the groovy form. Don't fear, I'm going to go overboard with AJAX! It'll only be used up until the point of where templates are generated. After that point, a Servlet will take over the bulk of the work (I am undecided on the Servlet bit.).

Now what's next? Several things need to happen.

First, creating the form:

1. We need to validate that form fields are not empty; also the input must be checked for validity.
   
2. The model's syntax needs to be compiled and syntax errors need to be relayed to the user.
   
3. We need to get the fields of the class and store them.
4. We need to generate templates like: <%= textArea(...); %> -- They will not be run it through the template engine immediately until the user has confirmed what they want,
5. After the view has been tweaked -- it is run through the template engine; at this point we will also run the controller through and generate that and present it at the same time the final view is presented.
6. The user will be presented with the HTML of the view and Groovy controller and given a final opportunity to tweak it before it is saved to the system.
   
7. The form is added to the system -- by default the form is not published.
   
8. Finally, to publish the form so that it can be used to enter data, they go to the management screen and tick the checkbox.
   
9. The form is now ready to be used for data entry.

Pictures are worth a thousand words so here we go: One and Two.

First one contains one form which is published, the rest are not. The second, no forms are published.

The user interface is intuitive and easy to use. It's a great design, of course I'm biased since It was was me who designed it.

So, I was working on my summer of code project, and I have groovy code along side my java code. I needed to add the groovyc ant task. While i assumed that groovy-all-1.5.6.jar, (note the all) contained everything that was needed, guess what folks, it doesn't! I spent the greater part of the evening tracking down this problem, and then i added the ant-1.7.0.jar and ant-launcher.jar jars, after that suddenly everything worked. That begs the question though, why isn't everything included in the groovy jar?

The problem: NoClassDefFoundError on MatchingTask class.


I figured I would put this out there for others in case they too experience what I did.

Today was the official start of Google Summer of Code 2008 Hopefully everybody is finding it a pleasant experience. I have yet to start coding as today was a US Holiday (Memorial Day).

The project plan for the summer has been finalized. I decided to set two-week blocks, rather than do a weekly thing. I chose this, as it's extremely flexible and can scale items either back or forward.

The project plan can be found here. A Google Doc which my mentor and I used to plan the project can be found here. Finally, See my general plan written as a previous blog post.

Best of luck to all the summer of code students!

I haven't really blogged since I last announced that I was accepted into Google Summer of Code. I have a phone meeting with my mentor to bang out a project plan, I more or less have a general idea of how I want things to go, and he does too. Overall, it should be a very successful summer.

I've been enjoying the Community Bonding period. I have made a lot of friends. My fellow summer of coders are amazing. I planned a potential meet-up for google soc'ers from NY, NJ and CT (to be announced formally when it is more concrete).

Now, onto the fun part, telling you all about my project and how I plan on doing it. I'm gonna break this down. There's a lot of things that need to happen. A lot of data needs to be passed between the front and back ends.

Firstly, Direct Web Remoting (DWR) will be used to grab all the information the users enter into the form and forwards it to the backend whose job it is to generate the form, and then relay it back to the user showing them the result, then saving it to the system. DWR will also be used to provide some AJAX magic.

Then, after I have the data, I need to use reflection on the Form Model and based on the data type, using a templating engine, such as Velocity generate HTML form components.

String, etc: text field or text area; Using some Groovy ExpandoMetaClass magic will aid in deciding if we use a text field or text area.

List: drop-select select box

Boolean: radio or checkbox with some Groovy ExpandoMetaClass magic, i'll add a property which will help aid in deciding which one to use.

After the form is generated, a controller will be generated with default checks that the fields are not empty. This will also be done using Velocity for templating.

Once I have everything generated, it will be saved on the file system someplace with an associated XML file containing all metadata(title,version, description,etc) which will be read in by the system when showing which forms are in the system.

As posted earlier that OpenMRS is participating in Google Summer of Code as a mentoring open source project. Working with OpenMRS will be highly rewarding knowing that you are single-handedly affecting the lives of people in developing nations ravaged by HIV/AIDS. If you are a student (undergaduate/graduate) you are eligible! So go apply for summer of code with OpenMRS! Student applications opened up yesterday and will be accepted up until Monday, March 31^st 2008.

Stop by #openmrs on irc.freenode.org for more information -- ask for docpaul or burke.