It's been a while since I last updated on the progress of the Facility Data Module. So without farther adieu here we go:

• The relation between the FacilityDataFormSchema and FacilityDataFormSection is now Many to Many meaning that sections can now be re-used between several form schemas (or reports).
• It now runs without errors on OpenMRS 1.5.x
  

The first item was a big problem that was a sore spot for this project since it wasn't too intuitive, ideally a user should be able to re-use a section. The second item was a freebie since the errors were likely all due to the incorrect modeling.

What's next on my list is the following:

1. Allow for sections to be re-ordered within the schema (this is currently not possible and is a show-stopper.
   
2. Fix up the calendar management pages -- those are horribly ugly, but this is after all a first pass, so that is to be expected right?
3. Currently, it does not validate inputs for Numeric data types -- It need to.
   
4. There still needs to be the ability to analyze the data that exists in the system to submit to funding sources; those pages need to be implemented but that is phase two of the project.
   
5. There are likely other sore spots that exist that I do not see at this point and time that will be fixed at a later date.

That is all for now.

I have made a lot of progress on my Google Summer of Code Project.

First, I've completed the code for the generation of the directory structure for the forms. This is done using the form id, which is the name with all illegal characters removed. The only valid characters left are alpha-numeric and periods. Let me tell you, the regular expression to replace all of those characters was hell to write. But, thanks to the help of Jason Davis, I was able to make it much more readable and easier digest. You've gotta see it to get an idea of how bad it was, so here goes:

   /**
     * Removes all illegal characters, then removes all spaces and makes it all lower-case.
     *
     * @param str the form name
     * @return a viable form id.
     */
    public static String formNameToFormId(String str) {
        return str.replaceAll("[\\$\\(\\)%`~!@#&;:\\^=\\+\\?
\"\\*\\\\////\\\\{\\}'\\]\\[\\|\\s+]", "").toLowerCase();
   
    }

That is pretty much hell on earth to read, and it was hell on earth to type. Here's the *MUCH* simpler version:

   /**
     * Removes all illegal characters, then removes all spaces and makes it all lower-case.
     *
     * @param str the form name
     * @return a viable form id.
     */
    public static String formNameToFormId(String str) {
        return str.replaceAll("[^a-zA-Z0-9.]", "").toLowerCase();

    }

Note, you can clearly see that i'm negating that entire sequence. It does the SAME thing, but is MUCH more readable. That reads: "replace everything that isn't a letter from A to Z *OR* a to z(lower case letters) *OR* from 0 through 9 *OR* a period. Whereas the above just excluded the invalid characters explicitly but was a MONSTER of a regular expression.

Secondly, metadata persistence is written in, I'm using xstream. It's a great tool for XML serialization.

That's it. Oh, by the way, these are the things I earmarked TWO WEEKS FOR! Now, I push domain class model processing up, will work on that tomorrow.

So, it's time for an update. This week got off to a sketchy start, but it's gained momentum. Let me enumerate what I have done thus far with the project. First, I've added AJAX using jquery. I figured I would use jquery mainly because it provided painless AJAX goodness. Prior to even thinking of using jquery, I wrote my own AJAX code using a tutorial that I found while googling. I had to tweak it a bit, but for the most part, it seemed very standard. However, after I wrote the AJAX equivalent code, it didn't feel too elegant. So, first I'm going to show the AJAX code I wrote; then I'll show you the jquery version; finally, I'm going to show you the servlet which handles the AJAX on the server-side.

First, the AJAX I wrote:

function AjaxValidation(url, callback) {
    var req = init()
    req.onreadystatechange = processRequest

    function init() {
        if (window.XMLHttpRequest) {
            return new XMLHttpRequest()
        } else if (window.ActiveXObject) {
            return new ActiveXObject("Mircosoft.XMLHTTP")
        }
    }

    function processRequest() {
        if (req.readyState == 4) {
            if (req.status == 200) {
                if (callback) {
                    chkSyntaxCallBack(req.responseXML)
                }
            }
        }
    }

    this.doGet = function() {
        req.open("GET", url, true)
        req.send(null)
    }
}
function chkSyntaxCallBack(responseXML) {
    var res = responseXML.getElementsByTagName("result")[0].firstChild.nodeValue
    document.getElementById("out").innerHTML = res
}

function checkSyntax() {
    var target = document.getElementById("groovyModel")
    var url = "${pageContext.request.contextPath}/moduleServlet/groovyforms/createGroovyForm?groovyModel=" + escape(target.value)
    var ajax = new AjaxValidation(url, chkSyntaxCallBack)
    ajax.doGet()
}

I warned you that it wasn't too elegant. Understanding this isn't too hard. Here is the call sequence: init() -> doGet() -> processRequest() -> chkSyntaxCallBack(). Not that bad. Now let's see the jquery version.

$(window).ready(function () {
    $("#groovyModel").bind("blur", function () {
        $.ajax({
            type: 'POST',
            data: { groovyModel: $("#groovyModel").val() } ,
            url: "${pageContext.request.contextPath}/moduleServlet/groovyforms/createGroovyForm" ,
            cache: false ,
            success: function(data) {
                var res = $(data).find("result").text()
                $("#out").html(res)

            }
        })
    });
})

Okay, that's much better. A few things are still happening here. When the window is finished loading, I bind my textarea element which has the CSS id of "groovyModel" to the blur event (lost focus). Then you see the AJAX. Now this is very straight forward. We're using the POST method, we're sending whatever value is inside of the textarea at the time the event is fired, we're posting to a servlet, not going to cache, and when we're done, it's printed to the screen. Very straight forward.

Now, like I said, this is all backed on the server-side by a servlet, which is written in Groovy. So here we go:

/**
 * The contents of this file are subject to the OpenMRS Public License
 * Version 1.0 (the "License") you may not use this file except in
 * compliance with the License. You may obtain a copy of the License at
 * http://license.openmrs.org
 *
 * Software distributed under the License is distributed on an "AS IS"
 * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the
 * License for the specific language governing rights and limitations
 * under the License.
 *
 * Copyright (C) OpenMRS, LLC.  All Rights Reserved.
 */
package org.openmrs.module.groovyforms.web

import javax.servlet.ServletException
import javax.servlet.http.HttpServlet
import javax.servlet.http.HttpServletRequest
import javax.servlet.http.HttpServletResponse
import org.apache.commons.logging.LogFactory
import org.codehaus.groovy.control.CompilationFailedException

class CreateGroovyFormServlet extends HttpServlet {
  def classLoader
  static final def log = LogFactory.getLog(CreateGroovyFormServlet.class)
  private static final long serialVersionUID = 066373513262051L

  @Override
  protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
    def generateTemplate = request.getParameter("template")
    def generateController = request.getParameter("controller")
    def finalMarkup = request.getParameter("markup")
    def clazz = URLDecoder.decode(request.getParameter("groovyModel"))
    def name = request.getParameter("formName")
    def version = request.getParameter("version")
    def res = this.checkSyntax(clazz)
    if (clazz) {
      if (checkSyntax(clazz)) {
        response.contentType = "text/xml"
        response.setHeader "Cache-Conrol", "no-cache"
        response.writer.write "\n\t$res\n"
      } else {
        response.contentType = "text/xml"
        response.setHeader "Cache-Control", "no-cache"
        response.writer.write "true"
      }
    } else {
      response.contentType = "text/xml"
      response.setHeader "Cache-Control", "no-cache"
      response.writer.write "\n\tPlease fill in the Form Model\n"
    }
  }

  @Override
  protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
    doGet(request, response)
  }

  @Override
  void init() throws ServletException {
    if (log.infoEnabled)
      log.info("Initializing...")
    classLoader = getClassLoader()
  }

  def getClassLoader() {
    def gcl = new GroovyClassLoader(this.getClass().getClassLoader())
    gcl
  }

/**
 * This method is used to relay errors to the user
 * @param clazz the class
 * @return the exception message or null if it was successful
 */
  def checkSyntax(clazz) {
    def sb = new StringBuilder()
    sb << "import org.openmrs.*\n\n\n"
    sb << clazz
    def res = null
    try {
      getClassLoader().parseClass(sb.toString())

    } catch (CompilationFailedException e) {
      res = "Exception: ${e.message}"
    }
    res

  }

/**
 * Check if it is result groovy code.
 * @param clazz the class
 * @return whether or not it is result groovy code
 */
  def isValidGroovy(clazz) {
    def sb = new StringBuilder()
    sb << "import org.openmrs.*\n\n\n"
    sb << clazz
    try {
      getClassLoader().parseClass(sb.toString())
    } catch (CompilationFailedException e) {
      return false
    }
    return true
  }
}

This servlet contains a lot of utility methods. One compiles, one initializes/returns the GroovyClassLoader, and of course doGet(), doPost() and init(). doGet() and doPost() both do the same thing, with doPost() simply delegating to doGet(). The code should be reasonably easy to understand. checkSyntax() returns null if it was parsed cleanly, otherwise it returns the exception message, the stack track wouldn't be useful in my case. It returns an XML tag <result> with "true" if it was successful, the exception message if it was not, and a message stating that the field must be filled in if it's empty or just not passed in. I implicitly import org.openmrs.* to allow for easier access to the OpenMRS domain model classes. The parent classloader for the GroovyClassLoader is set the servlet container's classloader. This gives me access to the classpath when loading Groovy classes.

I still have a bit to do, templating needs to be written in, for the most part it's done. Just have a few problems I'm facing, but I'll get through it. Too many people are relying on me succeeding. I feel that this project could help a lot of people, so i feel pressure to succeed. I still need to write in the "Edit" functionality of the "Manage Groovy Forms" page.

I'm definately making progress. More updates to come, that's for sure.

Okay, week 1 is nearing an end.I accomplished alot of things.

First, I completed the meta-data storage code. I wrote some tests, which revealed some bugs that needed fixing, and they were. I am happy to say, that the metadata storage works!

Writing the test was difficult due to the fact I store the forms in the system using a static List in my container class. I wound up updating to JUnit 4 to get at the @BeforeClass annotation so that the only one set of forms gets loaded into the container class. This helped me greatly. Additionally, the @AfterClass annotation was handy for cleaning up from the tests.

Secondly, Writing the code for interrogating the model was a piece of cake, thanks to groovy. All code for interrogating the model has 50 lines! That includes a model class I wrote to hold the field types and field names. I'll show you, but before I do, I should note that return statements are optional, and the final statement will be returned.

package org.openmrs.module.groovyforms.util

import java.lang.reflect.Field
import org.openmrs.module.groovyforms.metadata.model.GroovyFormsDomainModel

/*
 * Utility class containing methods for class interrogation.
 */
class GroovyFormsClassUtil {
    /**
     * Interrogates the class for all declared fields and
     * stores the type and name in a container class
     * @param the {@link Class#getCanonicalName() canonical name of the class}
     * @see Class#getCanonicalName()
     * @return a reference to a container class containing the type
     */
    static def getModel(fields) {
        def domainModel = new GroovyFormsDomainModel()
        def names = domainModel.fieldNames.&add;
        def types = domainModel.fieldTypes.&add;
        def f = fields.each {Field field ->
            names field.name
            types field.type.canonicalName
        }
        domainModel
    }
}

With groovy, it's so easy and concise (as you can see). Let's explain what's going on. First, I pass in the Field array I get from Field.getDeclaredFields(). Now, groovy adds methods onto the standard JDK classes, one of those methods is a method named each() which takes a closure. Now, back to the point, I pass a Field into the closure. That closure is executed for each element (in this case, a Field). Then I add the name, and the type to a List stored in a container class, now that container class is also written in groovy!

package org.openmrs.module.groovyforms.metadata.model

/*
 * Ths class holds information about the properties of the model. 
 */
class GroovyFormsDomainModel {

    /**
     * The field names
     */
    def fieldNames = []

    /**
     * The field types
     */
    def fieldTypes = []

}

Now, the fields are Lists, not arrays. Anyways, I've gotten off on a tangent here, so let me get back on track.

What I accomplished:


Week 1:

Code to generate the directory structure, serialization of metadata back/forth between XML and POJOs (Plain Old Java Objects), Wrote tests to ensure everything works in that regard. Additionally, I wrote the code to interrogate the domain model which I will generate the forms from.

Next Week

Write up the templates for the view/controller and write code to do the generation of the view/controller. Write some tests to ensure everything generates correctly.

Unrelated to that, my stored-value card from google came today. It feels nice to be $500 richer! This is going to be the best summer, I'm already having fun doing this. It's amazing seeing the whole project evolve into something amazing.

Google Summer of Code 2009 officially ended on August 17, 2009 at 19:00 UTC (12:00 PDT). I had an amazing time this summer. While stressful at times, it was well worth it.

I would like to thank the following people for my success this summer:

My mentor Mike Seaton for helping me throughout various misunderstandings surrounding the project's requirements. I probably made him work more than he had to. I attribute the success this summer to that one factor. This man is quite brilliant and knows his stuff.

My backup mentor Darius Jazayeri, for stepping in and helping at times and providing guidance when necessary. He is truly a brilliant man.

Ben Wolfe for answering any questions I had. He didn't have to help, but he did. I am very grateful.

Paul Biondich and Burke Mamlin for believing in me and giving me a second chance this summer. Did I mention these guys are doctors? Burke, an internist and Paul a pediatrician. Sorry for clumping you guys together.

Finally, without Leslie Hawthorn, Cat Allman, and Ellen Ko from Google, this program would not exist. They deserve a huge thank you. These three women are amazing. Thank you so much!!

So we're in the home stretch folks! 13 days to go until the "Pencils Down" date! Hope everybody's project is going well. Here's how mine is going:

• Reports can be saved and reloaded (with all relevant data filled out (fields etc -- reports can be viewed in editable or "view only" mode).
  
• Each component of the report (Questions, Form Questions(allows a question to exist on multiple reports), and Sections can be created in isolation
• A page to view the status of a report (complete, partially complete or incomplete) is done for reports that are to be entered daily in a monthly calendar view; monthly still needs to be completed.

Overall, i feel the project is progressing well -- I expect to finish either on time, or early. (Let's hope for early!)

Cheers!

Apologies for not blogging as much as I should. I've focused on getting what needed to be done, done.

Tasks that have been completed thus far with a target milestone of using a mock Form, enable persistence of Questions and Values to the database:

• Mock out a report form using the domain classes
• Using the mocked up schema, generate a simple report form
• Design the SQL Schema (for just FacilityDataValue and FacilityDataQuestion)
  
• Write Hibernate Mapping files (for just FacilityDataValue and FacilityDataQuestion)
  
• Write Data Access layer (for FacilityDataValue and FacilityDataQuestion)
  
• Write Service layer (for FacilityDataValue and FacilityDataQuestion)
• Refactor the rendering logic to use the JSP and write EL function(s) to check types using instanceof.
  
• Allow a simple the mocked form from Week 1 to save the question answers.

Tasks that are are in progress, soon to be finished with a target milestone of removing the code used to mock up everything from the first few weeks; ability to use the saved schemas for rendering the report form:

• Design SQL Schema for the rest of the domain classes
  
• Write mappings for the rest of the domain classes
• Support loading the previously saved values for a form/startdate/enddate/location into a page for viewing or editing
  
• Write methods to save the rest of the domain classes in the data access layer
  
• Write methods to save the rest of the domain in the service layer

Now that I have summarized work completed and in progress, let's explain the overall design:

1. FacilityDataFormSchema serves as the overall representation of the report form in the system.
2. FacilityDataFormSection is simply that, sections on the form, e.g., monitoring equipment status, stock status of vaccinations, number of people vaccinated, etc.
3. FacilityDataFormQuestion holds metadata regarding a question.
4. FacilityDataQuestion is the question itself; it specifies the datatype; it is subclassed for each question datatype; if not subclassed, then the question is considered to be "freetext" -- in other words: just simply a text-based question.
   
1. CodedQuestion is a question that has a coded answer. This too is subclassed for each coded question datatype.
   
1. StockQuestion is exactly as the name says, to track stock of items such as vaccinations. The coded answers are: "not_stocked_out","stocked_out","expired","not_applicable"
   
2. BooleanCodedQuestion is a simple "yes","no","not applicable" type of thing; e.g., "Was there mobile clinic today?"
2. NumericQuestion is a question which has a numeric answer, e.g., "Number of Adults Vaccinated."
   
5. FacilityDataValue is what holds the values entered in the report forms for each question.
6. FacilityDataReportFormData is a non-persisted class used for retrieving the answers for a specific report for a specified period.

Hopefully this makes up for my lack of updates.

Progress has been slow and steady. I'm on track to finish however! What's done:

• Reports can be entered, saved, and the values entered will be reloaded into the proper form fields. If edited and if the values have changed -- the old values are voided and new values are saved (retaining the old values in the database for auditing purposes).

What's left:

• Management pages See this and this for what it will ultimately look like.
• Still need to get the "View Only" pages working; in theory it should work. The "View Only" will be the answers but no form fields present.
  
• The UI to create the report, questions, sections, etc.
  

Ultimately, I think this shouldn't be too bad.

This is gonna be short, sweet and to the point:

Week 1:

1. I wrote a mock report form with 10 questions (screenshots in a later post)
2. I then wrote up some code to render it approriately for each question type.
   

Week 2 (still ongoing):

1. Designed the SQL schema, and wrote the service/database layer classes for 2 of the classes.
2. Wrote in the functionality to save the report data to the database.

Will explain the design at a later date.

Ciao!