As suggested by Gsoc organizers today is the *suggested* pencil-down date. In my new updates I included the Synchronization maintenance tasks as the last part of the project .It’s divided into three parts : Sync search , sync journal and imports archiving and managing some sync timeouts from the sync admin ui.
Now you can search for sync records according to their display name and you can view or edit them from the results list obtained.
For the journal archiving task , I created a new class that can zip up the contents of a folder into a .zip file using Adler32 checksum . The class org.openmrs.serialization.ZipPackage
compress the sync journal or import up to 30% . For example the contents of %APPDATA%\OpenMRS\journal are archived into %APPDATA%\OpenMRS\archive\journal\journal_2008_08_09_22_26_00_288.zip and the contents of the original folder are cleaned up .

The screen shot of the maintenance page is presented here :

As the last step of this project I am going to check everything in the codes like the log messages,result messages from Spring,to check if there is anything missing or any misunderstanding message, to add comments for developers guidance ;I have also to update the sync administrator helping screen and refine the project wiki page for future reference.

Thanks

These days I am implementing the final part of my soc project . One of the features in the last commit was the ability of the Sync Administrator to edit the Sync record items payload .To that end I created a method in the DWRSynchronizationService.java that gets the sync Item content and then sent it to the web client .

public String getSyncItemContent(String guid, String key) {
    String content = "";
    Collection<SyncItem> itemList;
    if (guid != null && guid != "" && key != null && key != "") {
        itemList = Context.getSynchronizationService().getSyncRecord(guid).getItems();
        for (SyncItem item : itemList) {
            if (item.getKey().getKeyValue().equals(key))
            content = item.getContent();
        }
    }
    return content;
}

After creating that function, I have to configure it in dwr.xml

<create creator="new" javascript="DWRSynchronizationService">
<param name="class" value="org.openmrs.web.dwr.DWRSynchronizationService" />
<include method="testConnection"/>
<include method="syncToParent"/>
<include method="getSyncItemContent"/>
<include method="setSyncItemContent"/>
</create>

The content String got in synchronizationViewRecordList.jsp (via Ajax) is just in mxl tags format and it looks like :

<org.openmrs.ConceptName><guid type="string">f6b29ecf-b15e-102b-ac59-58f4c1e745a2</guid>
<concept type="org.openmrs.Concept">f5aa80e9-b15e-102b-ac59-58f4c1e745a2</concept>
<description type="string">edited A disease caused by a lack of vitamin C and characterized by
 spongy gums, loosening of the teeth, and bleeding into the skin and mucous membranes.</description>
<locale type="locale">en</locale><name type="string">VITAMIN C DEFICIENCY</name>
<dateCreated type="timestamp">2004-06-01T00:00:00.000+0200</dateCreated><shortName type="string"></shortName>
<creator type="org.openmrs.User">fc193cd8-b15e-102b-ac59-58f4c1e745a2</creator></org.openmrs.ConceptName>

It was quite useful to have that payload in a single text field on the Admin UI so that the Sync Admin can edit it like that ,but it could result in a sync failure if the admin alters the tags structure or changes some crucial data like the GUIDs . So we decided to parse it with javaScript and display the payload values in different text fields where the Admin can edit and save them . To save that content we have to rebuild it from html tags .

The parsing code is here:

  function showContentResult(result) {
    DWRUtil.setValue("loadContent", "");
    domParser = new DOMParser();
    xmlDocument = domParser.parseFromString(result,'application/xml');
    root=xmlDocument.documentElement;
    rootElt=root.nodeName;
    cn=root.childNodes;
    newHTML="<form id='contentForm' action='' method=''><table width='100%' border='0' cellspacing='0' cellpadding='0'>";
    for(i=0;i<cn.length;i++){
      newHTML+="<tr class='syncTr' ";
      if(i%2==0)newHTML+="bgcolor='#C4D9D9'";
      else newHTML+="bgcolor='#E7EFEF'";
      newHTML+="><td><strong>"+cn[i].nodeName+"</strong></td><td>"+cn[i].attributes[0].value+"</td><td   height='30'>
<input id='field'"+i+" type='text' value='";
      if(cn[i].childNodes.length==1)
      newHTML+=cn[i].childNodes[0].nodeValue;
      newHTML+="' size='50' /></td></tr>";
    }
    newHTML+="<tr><td>&nbsp;</td><td>&nbsp;</td><td   height='35' align='left' valign='bottom'><input type='button' name='saveButton'
   value='  Save  ' onclick='setItemContent();'/></td></tr>";
    newHTML+="</table></form>";
    document.getElementById("contents").innerHTML = newHTML;

  }
  function getItemContent() {
    DWRUtil.setValue("loadContent","Loading Item payload ...");
    DWRSynchronizationService.getSyncItemContent(currentGuid, currentKey,showContentResult);

  }

And the screen shot is here:

An other task implemented was to perform standard search for sync records.This is actually implemented according to the record name.

In the next step , I have to finalize all maintenance tasks and update the help screen .I hope that will be the end of the soc , and our codes should get in production quickly .

So far my project was reviewed and so far you come up to get my updates.Yes, last Thursday I get in the OpenMRS developers conference call.I was a little bit late and hopefully my mentor had come in advance and he had talked about my project and the work I already achieved at the time.This summer of code was not exactly a Summer holiday like some other students. In fact I am doing it in parallel with the school classes and other small assignments.

Back to the code review , I am unable to explain the feelings i had at the review time. My skype connection was so bad at campus (where I live) so late I decided to go to an internet cafe for better connection ; it was not the best idea to attend a conference in a public place like that but it was my last chance to get my project reviewed .Really I got a little bit frustrated for making a public speech online to some *software experts* located in different places of the globe.The content of my review was just a summary of what has been made to the Sync Admin UI and the next trip in the advance of the project.The developers suggested everything concerning the project have to be fixed before the end of the GSOC so that we should come up with something really productive.

As a description of my midterm goals you can refer to the wiki page of the project as well as in the last updates of this blog . For this next term of the project I have to Implement mainly the Sync maintenance tasks as well as refining the tasks useful to the pages achieved in the last part of the project . To that end, I ended up by bringing up the Sync Item payload to the Sync Record View/Edit screen via DWR/Ajax .
Right now it’s in text format with tags but you can edit and save it ;that seems even useful for the Sync administrator at the time .The  problem with this is a possible data corruption if the administrator try to edit some sensitive information like IDs,GUIDs,or tag names or if he alters the tag structure in the payload.So I will have to parse the payload as long as  it’s loaded from DWR then put it in some text fields where the user have option to edit only the text tag content.To save the edited payload, I will  have to rebuild the content from the fields and save them via the same channel (DWR/Ajax).

An other thing that needs to be fixed is to update of the project wiki page for future public reference ; hopefully I will come up with everything working properly and my updates are going to turn to regular.

It’s now time for midterm evaluation. The time has been running so quick so that I couldn’t imagine how it ends up to this.I have almost achieved a part  of the work expected  and the project screen shots are available on the project wiki page. I am planning for a conference call with the OpenMRS core developers for code review.I think they should give me more ideas about the next step project orientation if I pass the midterm evaluation . The current source codes have been updated with small css changes on the user interface . Hope I will pass the evaluation , and you should still updated for the next project development.

- Antoine

Yesterday I got a code branch created for my project. And I deployed my local codes to the new repository branch  http://trac.openmrs.org/browser/openmrs/branches/data-synchronization-admin-ui .Most of the codes there are just merged from the current OpenMRS Data Synchronization project. The codes I’ve  written are in the web\WEB-INF\view\admin\synchronization and src\web\org\openmrs\web\controller\synchronization directories .Here are current goals of the  project:

*User after login to the Admin screen find a Synchronization section links: >>Settings >>Manage Synchronization >>Sync Statistics >>View Sync Changes

On the settings page you can setup sync properties such as Enable/Disable sync , Setup Scheduled sync,setup servers(Child and Parent) and select Items to synchronize. On the Manage sync page you can Synchronize manually via file  or  via Web, View Sync Records details ;there are also an option to reset as new any failed to sync record or just to remove it from the list just as a not supposed to synchronize type (in order to unblock other sync records when failure is due to one or some items in the sync records list).

The Statistics page give you the number of sync records by category:New,Synchronized,Failed etc.You can also browse sync records by category(status) .You can choose an interval of timestamps to visualize what happened at the sync process.

The View Sync Changes page is merged from the history page and it’s now paged when the records number exceeds a given number.

Thanks to the mentors for their review,

Antoine.

Great weekend !Rwanda’s Amavubi Stars [Wasps]-3:1-Morocco’s Atlas Lions . It was Saturday at Nyamirambo Stadium (Kigali) near my coding place for GSOC so I stopped coding then went to watch the match,even H.E. and Madam were there,so exciting !This was a day after I went to one of OpenMRS implementation site by PIH at Rwinkwavu. It’s in a rural area 3hours of bus from Kigali. I spent several hours with Chase who is an OpenMRS programmer working with PIH Rwanda.He is really cool and he introduced to me the OpenMRS system integrated into two clinics Rwinkwavu and Kirehe. I visited the patient registration office,data entry team and the server room.The system generates patient identification numbers and paper form sheets to be filled in by medical staff and patients, then saved in the system data model by an entry team.At server side there is actually data synchronization between two server sites Rwinkwavu and Kirehe.

About coding issue, I ‘m implementing some Spring controllers for my project UI.It was a little bit hard for me to integrate the spring bean binding methods for getting dates and other data from UI to controller.The  screenshots of my project are right now monitored on the wiki page ,and hopefully I will be done with the deliverables just before the issue date.

An other issue in OpenMRS world is the SA conference, some mentors working busy and the IRC a little bit cold.

I started the second week of GSOCoding with the idea that I must achieve my project as soon as possible. So I started to design a *settings* page for the Sync project.By this time , the second semester have started so I need more time for revising assignments while implementing my GSoC project.There was also this EURO and some local volleyball memorial games,but I decided to think more on my GSoC project than others.I remember there is also an East-Africa Google gadget competition that needs also some posting.So Busy!

I’ve been talking to my mentor who advised to be more open (favor for public discussion) in an open source project.I also faced many internet disconnections so I was afraid of breaking my communication scheme with the mentor.with the help of my mentor I set up two openmrs installation running on the same Tomcat (@my laptoph http://localhost:8080). That simplifies Synchronization testing when I am offline ,so i don’t have to handle two computers as early.To that end,here are some changes I’ve made on the deployment files to create a new web-app : openmrs_child

-Create a new Database with MySQL scripts: drop database if exists openmrs_child;
create database openmrs_child default charset utf8;
use openmrs_child;drop user test_child;
create user test_child identified by ‘test_child’;
grant all on openmrs_child.* to test_child; After run 1.3.0-schema-with-core-and-demo-data.sqlopenmrs_1.1.0-add-synchronization.sql and openmrs_1.1.0-add-guids.sql scripts and its done.

-Change properties at the build.properties file so that:webapp.name=openmrs_child
webapp.display.name=OpenMRS_child.

-Create system env. variable called: OPENMRS_CHILD_BUILD_PROPERTIES_FILE and OPENMRS_CHILD_RUNTIME_PROPERTIES_FILE that links to the %APPDATA%\OpenMRS\OpenMRS_child-build.properties file and %APPDATA%\OpenMRS\OpenMRS_child-runtime.properties respectively.

-Override properties there and in the ant build file (build.xml).Just in 5 minutes of setup & installation I got my two installation running . After doing some synchronization configuration I got my two DB synchronizing bidirectionally. I was a half done.

Later I implemented almost in parallel two testing pages for synchronization,one for settings and one for control,the main ideas was from the lastly build Spring MVC controllers for Sync branch. I think it’s even easier to build controllers for a Service Oriented API . For Example when I want to get a list of Sync records I simply call List<SyncRecord> recordList = Context.getSynchronizationService().getSyncRecords();Thanks to the API developers ! I made a change on how to turn on/off Synchronization via the *settings* page without interacting *manually* with the runtime properties file . I can also deal with the DWRSynchronizationService to synchronize via Ajax .For the next steps I have to implement the ability of selecting which items to synchronize (default is now to *select all*) and also be able to edit Sync Items, i.e. refreshing failed ones;then start sync reporting , i.e. analytics.

An other thing is that I upgraded my Netbeans IDE 6.1 with Eclipse Project Importer Module, so most of times I am coding OpenMRS under Netbeans, I mostly like its JavaScript editing capabilities, but  Eclipse still great  for its *automatic* imports .

Today I ended up by having my two installations connected successfully.But they still problems and bugs that I have to track tomorrow . I talked to my mentor (Maros) who introduced to me how to make tickets for bugs found . I already faced some bugs when creating a new patient in the 1.2.0 OpenMRS release and the other one about guid fields set as not null and the default value is null which result in JDBCException or SQLException and I am ready to open a ticket for handling it . Problem with the GSOC credit card names (XXXX + GSOC 2008 STUDENT)don’t match my real ID Names , so the bank refused take funds out off the card (while I was planning to extend my laptop RAM at least to 1.5GB with that 1st Google funds)and there is no ATM for Mastercard in Kigali; heard other Gsoc guys facing same problems with the PayPal trick.

My first week of gsoc went without writing a lot of code for the Openmrs Data Synchronization. At least I tried to get familiar with the code in the branch there, there is a big work already done on the project , so I have to design the user interface in order to setup , control and monitor the Synchronization process between two OpenMRS server installations : PARENT and CHILD.

A parent should have many CHILD(s) The actual synchronization process goes bi-directional means that any new , update or delete transaction made on the Child model is handled as a Sync Item hold in the Sync Journal waiting for transmission to the PARENT Server . When the PARENT receives the Sync record (made of Sync Items) from the Child , It get updated and then deliver a Sync report to the Child and then the Child get updated and before Synchronization PARENT and CHILD should have an identical Database Model. The Synchronization process involves many tricks as Sync Data Structures , Model Changes tracking within SynchronizationHibernateInterceptor algorithms , Data Serialization (Xml), Transmission , Deserialization and Updates .The main difference between the Openmrs trunk data model and the Sync data model was the addition of guid unique keys to every table (i.e. Synchonizable object/class) being synchronized .This enable Server to handle records from different Openmrs systems even having the same id (PK). An other trick to be done by Sync developers is to design algorithms for Conflict resolution within Sync updates and an adaptive data transmission algorithm (I think there are tickets for that).

In order to test Synchronization, I’ve made two OpenMRS installations at my work office .There is a desktop installation working as PARENT server and my laptop is set as the Child server. At the beginning I got some problems with the Sync branch deployment as I checked it out from the repository . I couldn’t even get the welcome page , there was a runtime error about the Anonymous Role login ,so I tried to change some login settings at org.openmrs.api.contect.UserContext.getAnonymousRole() and I was logged with a SUPERUSER_ROLE (”System Developer”). After I found the Synchronization was Disabled then I set a runtime property synchronization.sync_status to =enabled but the bug still there (Error to parse the runtime property synchronization.sync_status),after I decided to change the default return value at org.openmrs.synchronization.SyncUtil.getSyncStatus() to ENABLED_CONTINUE_ON_ERROR, but right now I found the I left the default return value as it was DISABLED_SYNC , then I set the runtime property synchronization.sync_status with an enum value in the org.openmrs.synchronization.SyncStatusState (ENABLED_CONTINUE_ON_ERROR) and there was no more parsing error .An other issue was with the guid field in the database .The addition of guid column in the Sync database was so crucial for Items exchange between different servers DB but in the openmrs_1.1.0-add-guids.sql script set guid as a Not Null value which result in JDBCException (guid field doesn’t have a default value) when creating new Items or Updating others in the DB without setting the guid. That exception prevented me neigther to create new data in the DB nor to update(edit) present ones. I was even unable to create a new Child Server @PARENT because the Child account is like adding a new User to the System .I think the solution for this is to create guid field with a Not Null default value like “emremremremremremremremremremremremr” . For my testing I decided to remove the script setting guid as not null value.Then It works fine for adding users, but it doesn’t for adding Patients. I still have problems with Sync Web transmission because after setting a Child the History page throws an Error , I still have more to track on those errors,(it’s even helpful to know more crucial lines in the code ) , while implementing my design plans.

I think my project is not too challenging and I hope I should be done early , but my testing will be always depending on the sync engines build by my mentors Maros and Christian during the in Data Synchronization project .I ‘ve read things about the Spring MVC framework used there and I already have my own JSP-JSLT page for testing and accessing Sync data in the DB via controllers . I will of course build more Controllers and Viewers , and more still to come …

*Ibyiza biri Imbere !*(Better Tomorrow)

Before this “Code up” date starts,it was a community bonding period .I was supposed to get know more about the organization (OPENMRS), have the coding environment set up and running on my machine and get familiar with the project code . I think the biggest challenge was to get my project running . Even after checking out the source code from the repository via Subversion, there was so many settings to run in MySql , Tomcat , System Variables change some login privileges and roles in the code and others. I think this may be a common challenge in Open source development when you are disconnected from the online community , and it’s hard to find someone else to help you .

During the community bonding period I had a Conference call with my mentor Maros and sometimes logged into the OpenMRS IRC just for some technical assistance and see what is going on with the other guys .Today I got introduced to e new OpenMRS developer who is going to work in Rwanda for PIH Mr. Chase . I hope I should have a visit there in Rwinkwavu and Kirehe and see the performance of OpenMRS , especially the Synchronization process while implementing the Admin UI of it .This should be encouraging for me because 1)

OpenMRS Synchronization Administration User Interface , that is my first open source project. I just made the project description on the wiki http://openmrs.org/wiki/Data_Synchronization_Admin_UI_Project. During this first week I have to deal with the Implementation Plans while reading more on different Apis and Frameworks on which OpenMRS relies . Talking about Data Synchronization , here is a help message that gives an understanding of the process:

What is Synchronization?

Synchronization is when you have multiple sites using OpenMRS, and you want them to copy data to each other. In this situation, one site has to act as the “Parent”, and one acts as the “Child”. The Parent is always the computer or site that is most central. Any site can choose only one Parent. However, a Parent can have many Child sites/servers.

What is the difference between “Synchronization” vs. “Generic Import” and “Generic Export”?

Synchronization means that the machines copying data to each other keep track of what has or hasn’t been sent. Generically importing and exporting allows you to export patient data from your system, and then import it to another system, but the 2 systems do not keep track of what data has been passed.

” How do I use Synchronization?

There are 2 different ways to use Synchronization, depending on your connectivity. You can:

1. Synchronize via File/Disk
2. Synchronize via Web

Synchronizing via File/Disk means that the Child server will copy all of its new data to a file, which you can save on a disk. That disk can then be carried to a Parent server at a different site, and then the file can be uploaded on the Parent server. In this situation, the Parent server generates a response to be carried back to the Child. This technique allows you to synchronize without Internet connectivity.

Synchronizing via the Web allows you to simply click a button to send data to the Parent, receive the response, and process it. The technique is useful if there is connectivity between the Parent and Child.”

Because the Sync process involves a Parent and Child , I plan to have two OpenMRS installations on different networked machines for testing Sync via web .

For any other issue in concern please feel free to feedback me.

Antoine