I thought it would be nice to post some screenshots of the current state of the cohort builder.

After defining a cohort, the default patient list view is displayed:

This shows the cohort patients separated according to gender and age. Clicking a pie slice filters the cohort to include only the corresponding patients:



That's about it for this view. The Concept view offers some more functionalities...



The bars are clickable and, just as the Age/Gender chart, clicking filters the data shown. In this case, clicking the [277-314] range, produces the following:



For numeric concepts, there is also the option to view a Patients vs Values chart. This plots the values for every patient in the cohort and can be useful for detecting critical patients. When available, the critical values for the concept are also shown:



Both types of charts (Frequency and Values) allow zooming into specific areas, by clicking the reddish band on the X axis and defining a zoom "window":


Resulting in:


And that's what's been keeping me busy in a glance... Some other remarks:

• Browser back button works, to step back after zooming in, even though one never actually leaves the cohort builder page. Though I love AJAX, the fact that it breaks the back button is possibly one of its most annoying aspects. Luckily there are several solutions to it, and after some research and tweaking, this one did the trick.

And what I guess will keep me busy for some time:

• Buttons or links to go back to previous view or go back to original cohort.
  
• The option of using the most recent, the least recent or all of the observations is still under development.
  
• There is a code refactoring in progress, to aggregate values directly on the database instead of in Java in order to improve performance. This implies massive refactoring in DWR services, the JSP cohort portlet and in the Flash charting library as well, so I imagine this will take quite some time...
  

I'll post some more technical details once the code is a bit more stable.

This was a great hack-a-thon.  We tackled a lot of different issues for the logic service.  Unfortunately, a number of these will require additional effort to be completed. We knocked out some of the simpler tickets: Rules tokens need to be case insensitive (#1104) Encounter datasource expansion (#1105) Creating an Operand interface (#1106) We made some decisions (like not moving [...]

OpenMRS was featured today on Public Radio International’s program, The World. Hamish, Paul, and James Arbaugh were interviewed. Listen to the segment

(A) We're experiencing out of memory errors in Tomcat with 100,000 patients in OpenMRS and 512 MB allocated to Tomcat. The first blocking run (blocking on postal code) appears to be completing; the out of memory error appears to occur during the analysis phase (either during random sampling or during Expectation Maximization) of the second blocking run (blocking on SSN).

Memory snapshots from the profiler reveal that a large amount of memory is allocated for the MySQL database connection. Large result sets are not being released and closed. Even though we are attempting to close the result set, it remains open, we believe because unidentified resources are still accessing the result set.

James has updated the code to address releasing the result set, and Win is testing the revision on his computer
If this code update doesn't resolve the issue, we'll re-examine the profiler output for other places where memory use could be further optimized.

Update 10/22/2008: The new code successfully ran with no out-of-memory errors observed with the new code. Therefore


(B) In some cases, the duplicate report listing may be large. So rather than displaying the report in a web browser (which is not designed to display large amounts of line-list data), it may be more practical to output the duplicate report directly to a file. Consequently, we're in the process of modifying the module to accommodate this. Win is implementing a more robust reporting work flow using AJAX. The user will be able to initiate a report, navigate away from the administrator interface, return to check status, and when the report is complete, a link to that report is displayed.

We’ve implemented a lock-out feature so that once a report is started, no reports can be initiated until the current report is complete.


(C) Currently we have been unable to run the “Yourkit” java profiler on Linux. We are using Windows to profile memory usage. If this issue further hinders progress, we will need to address the barrier we face to run Yourkit on Linux.

OpenMRS 1.3.2 has been released. This is a bug fix and security release over the current 1.3.1 release. This upgrade is strongly recommended.

See the the release notes for more information.

1.3.2 can be downloaded from http://download.openmrs.org.

We modified our tactics for matching because the matching process was taking longer to complete than we hoped.

We believe the performance issues relate to the myriad Hibernate queries that are required to create blocks of potential pairs. For example, if the patient table contains 4,500 unique SSN's, then 4,500 different Hibernate queries must be called to create potential pairs where SSN's match.

To minimize the number of Hibernate queries, the batch de-duplication process first examines the following tables: person, patient, patient_identifiers and person_attributes. A "flat", non-normalized table is then created with all fields from the above 4 tables.

All further analyses and scoring are performed against the flat, non-normalized table.

Recent timing test found that 10,000 patients could be extracted and stored in the the flat table in about 20 minutes, a rate of about 9 patients extracted from OpenMRS per second.

Once extracted from OpenMRS, analyzing and scoring the flat table took about 20 seconds. (SSN was the blocking variable and email/SSN were the include variables)

Next Steps Include:

1. Verify that the module can handle multiple blocking runs, and will join the multiple runs appropriately for the human readable report.

2. Verify that the current patient extraction process execution time increases linearly with the number of patients. We need to load 20-40,000 patients into OpenMRS and measure how long it takes to extract patients into a flat table. If time is not linear, we will need to consider other optimizations.

3. Create blocking runs that use neither patient_identifiers table nor the person_attributes table and measure how long these take to complete. I suspect that the "stacked" nature of these tables impacts efficiency.

4. Examine which specific tasks in the data extraction process are taking up time. To do so, we discussed the following approach:

• Comment out PatientToRecord method. Comment out SQL INSERT statement to load record into flat table. Measure how long it takes to iterate thru all Patients.
• Access only 1 or 2 properties in PatientToRecord. Comment out SQL INSERT statement to load record into flat table. Measure how long it takes to iterate thru all Patients.
• Fully execute the PatientToRecord method. Comment out SQL INSERT statement to load record into flat table. Measure how long it takes to iterate thru all Patients.

5. Because the de-duplication process will need to run when an OpenMRS system is not heavily loaded, it will likely need to be scheduled. We need to explore implementing a scheduling component.

6. We need to create two separate modules for each distinct linkage process: One for the batch duplication use-case, and one for the real-time matching use case (NBS). We envision creating a common package of linkage utilities that can be re-used in both modules.

The program officially ended on August 18^th. I'm awaiting the arrival of my t-shirt!!

Now, I didn't quite have time to finish my project prior to that date since my uncle suddenly passed away on August 10^th.

The project itself is about 80 percent completed. All that is left is to get the rendering working and finish up the management page to edit the form metadata as it appears in the system.

My time is now limited since school started up again for me, so as much time as I can I'm going to devote to finishing up this project.

I'd like to thank the following:

• Burke Mamlin for helping me and answering all the questions I had and guiding me when I was lost.



• Ben Wolfe for putting up with me and helping me when Burke was MIA.



• OpenMRS and its wonderful worldwide network of developers for making this project into what it has become.



• Paul Biondich for encouraging me (along with Burke) to apply for summer of code.



• Last, but certainly not least, Leslie Hawthorn for managing this program and every problem that popped up. She must have super human powers or something.

The list can only be so long, I feel like I'm accepting an academy award here. It was a great experience, and I will definately continue to maintain my project in whatever free time I can find.

Software Distribution is an essential part of Software Development and can sometimes be the first impression that can make or break the user's opinion about a software. We, as software programmers forget the importance of easy distribution and easy installation of software that we develop. We do not understand the problems that a new computer user or a non-programmer may face. And I experienced this first hand about 2 weeks back, when a physician friend of mine heard that I was working on OpenMRS.

I was lucky enough to work on OpenMRS this summer and learnt a lot more about Medical Informatics during this period than I expected. Hearing this, my friend wanted to install OpenMRS at his clinic which already used Tally (hehe... isn't that innovative??) for storing patient records, observations and prescriptions. He practices at Kolkata, visits different hospitals and sometimes the patients he attended at a hospital come to his clinic. When I told him that OpenMRS was a webapp, he got all excited and I narrated him all the features that OpenMRS could provide and help him manage his patients better through the web, only if he could host OpenMRS from his clinic. I'll skip the other interesting parts and his extra-terrestrial expressions ;-), since we are actually talking about software distribution.

So then came the day when I was about to leave office and he was in his clinic trying to install OpenMRS. It was Independence Day and the clinic was closed but he was excited to experience the new-age medical informatics :-)) When I first got his call he had downloaded the Windows Installer. I was pretty sure it was for an older version and hence told him to instead download the OpenMRS Appliance, which is a VM Image that can be run from one of the virtual machine softwares. Yaw Anokwa made this wonderful Virtual Image with Ubuntu + All Necessary Stuff (tomcat, mysql, demo data) and OpenMRS running. You just have to have VirtualBox or VMPlayer or VmWare Workstation and load the VMimage and wait for Ubuntu to start. It is simple, fast and safe to play with... But for novice users, I just realized it wasn't easy enough. My friend installed VirtualBox and loaded the image. It booted fine, but the network wasn't working and OpenMRS webapp could not be reached from the Windows host. After being on call for close to an hour, we just couldn't make the networking work!! I advised him to install VMPlayer instead and run the image. This time everything ran fine, but some changes had to be made in the Norton 360 Firewall. He kept complaining that Windows XP was punishably slow and then I realized that his 512Mb wasn't enough to virtualize :-( ... So we were back to where it all started!! The Windows Installer that OpenMRS distributes is based on Bitrock. He first tried the OpenMRS 1.1 Installer, but it is an older version that hasn't been upgraded for a year or so... Everything installed fine and he was happy to use it, but it didn't have the features I talked about that were added in newer releases of OpenMRS. I walked him through the manual installation and finally we managed to get OpenMRS up-and-running at 2am in the morning and he having spent about 8hrs on it. Last week when I asked him, he still wasn't using OpenMRS for his clinic and hospital. May be the first experience made him bitter!!

With that episode in my mind, I pledged him that within the next month or so I'll give him and easy to install setup and he'll be happy using OpenMRS. And that's when began my chase to find an easy to use, cross-platform installer framework. OpenMRS has lots of implementations on different platforms (Windows, Linux and Mac) and hence I wanted the installer to be cross-platform. At my office, we generally use Windows Installer or NSIS for making installers. But those are only for Windows. These 2 frameworks are so simple and extensible to use that I was thinking if there was something similar and cross-platform, I could make an OpenMRS Installer in an hour. But sadly, that wasn't the case... I tried a variety of installer frameworks, but couldn't find any of them as simple as NSIS or Windows Installer (msi). The following are the installer frameworks I tried working on:

Installer Framework	Short Description	Problems
1.) Antigen	Antigen (Ant Installer Generator) is a tool to take an Ant build script, combine it with a GUI and wrap it up as an executable jar file. Its primary purpose is to create powerful graphical installers from Ant scripts.	Couldn't get it to execute ant-calls at lots of places. Didn't work in openSuSE 11.0 due to some incomplete ant configurations. Hasn't been updated in a long time
2.) IzPack	IzPack-generated installers require Java. They are simple, efficient and fast to use. Simple executable deployment is best done through IzPack.	Isn't very powerful. Good for simple image deployment, but isn't highly configurable and powerful.
3.) OpenInstaller	A newer cross-platform installer framework that is completely customizable and written in Java. Glassfish uses this installer framework.	Not much documentation. Complex to implement and doesn't look native on all platforms
4.) Netbeans Installer (nbi)	A completely customizable and powerful installer framework. Configuration Logic is written in Java and can be used to do anything and everything that Java programs can do.	Old documentation. Requires some effort to get up and running with all the scripts.

So finally, I decided to work on using the Netbeans Installer. Netbeans Installer already has components like Tomcat, MySQL, Glassfish, OpenESB and their deployment scripts. And I thought it will simplify my effort... Dmitry Lipin of Sun Microsystems, the lead developer of the NBI team has been of great help over the past weeks and has helped a lot in explaining about nbi... While I was building the installer, 2 other colleagues of mine got interested in OpenMRS and have helped build some parts and want to contribute to OpenMRS code in a larger way!!

I have successfully been able to build an Installer/Uninstaller that can deploy Tomcat/Glassfish, MySQL and the OpenMRS web application on Windows, Linux, OSX, Solaris. The demo data set, JRE/JDK and starting the respective servers are yet to be completed.

Update: The OpenMRS Windows Installer based on Bitrock has been upgraded to install the latest version of OpenMRS. Is it useful for the OpenMRS community to have a cross-platform installer?? Or do the Windows guys only need an Installer ??

1) Implemented: Display additional matching variables in the web based report so the user can better evaluate the nature/quality of the match

2) Implemented: Create a flat file output containing (at least) all blocking and include variables, if not additional variables

3) Implemented: Because different blocking runs may have different include variables, we need to be able to include the union of all fields used across all blocking runs

4) Testing: We need to test the de-dupe module with different data sets. We need to test both for A) performance (how long does it take to match larger numbers of patients), and B) bugs -- what issues will we run into with different data?

Results to date: An out of memory error occurs with an OpenMRS patient table containing 10,000 patients (likely a Tomcat memory error)

• Had increased Tomcat memory allocation to 1024M, no change
• We may need to use a software profiler to see what processes are using memory
• Question: are match results objects (MR's) stored in memory? MR's should not be stored in memory -- to minimize memory use, each MR should be handled and written to a file, etc.