Diseases and Disasters (Dis2): Riff’s Performance

A couple of years ago, I introduced Riff; a hybrid (event-based and indicator-based) disease surveillance platform, at the International Society for Disease Surveillance (ISDS) [the original concept is fully described here - more up-to-date information can be found on InSTEDD's website here].  Riff was designed to streamline the collaboration between human experts dealing with diverse streams of information and enabling them with machine learning algorithms that can learn quickly and accurately classify the information for detection, prediction and response to health-related events (such as disease outbreaks or pandemics).  On Jan 17^th 2010, the Thomson Reuters Foundation used Riff [after prior adoption in their EIS system; an Emergency Information Service for survivors of natural disasters (early work can be found in Nico’s Blog here)] to launch a first-of-its kind, free disaster-information service for the people of Port Au Prince, Haiti. This allowed survivors of Haiti's earthquake to receive critical information by text message directly to their phones, free of charge.

Earlier this year; April 2010, I setup a Riff space; Diseases and Disasters (or Dis²), for providing timely situation awareness from credible and reliable [gold standard of a sort] online reports on diseases and disasters in the world provided primarily by BioCaster and HealthMap. This already tagged and verified information provided Riff’s classifier with a great training opportunity that I had to monitor closely and correct at the beginning for both features (e.g., condition, type of disease transmission, severity, etc.) and geo-location of where the event actually occurred as shown here:

Conditions Tracked in Dis2 Proportional to their Coverage since April 19, 2010

%Conditions Tracked in Dis2 since April 19, 2010

Location of Conditions Tracked in Dis2 Proportional to their Coverage since April 19, 2010

Location (Heatmap) of Conditions Tracked in Dis2 since April 19, 2010

In collaboration with the Humanitarian FOSS Project; supported by a group of computing faculty and open source proponents at Trinity College, Wesleyan University, and Connecticut College (Open Source ALPACA Light Parsing And Classifying Application (ALPACA) and Open Source e-dop for Disease Ontology Prediction for Riff), we developed a Support Vector Machines (or SVM) for automatic feature extraction, data classification and tagging. Riff’s classifier performed incredibly well in the Dis²collaborative space with very little training at the outset; 83% (95% CI: 81-85%) True Positive rate as shown here:

Riff's Performance in the Dis2 Collaborative Space (True Positive and False Negative Ratios)

Riff's Performance in the Dis2 Collaborative Space (True Positive and False Negative Ratios)

Riff is an Open Source Project, you can download its source code here and help further enhance its performance. 

Acknowledgment

• Nicolás di Tada and Eduardo (Ed) Jezierski: InSTEDD, Palo Alto, CA, USA
• Nigel Collier: BioCaster - National Institute of Informatics and PRESTO Japan Science and Technology (JST) Corporation, Tokyo, Japan
• John Brownstein and Clark Freifeld: HealthMap - Children’s Hospital Informatics Program at Harvard-Massachusetts Institute of Technology, Division of Health Sciences and Technology, Division of Emergency Medicine, Children’s Hospital Boston, Boston, Massachusetts, USA
• Humanitarian FOSS Project

Riff: A Social Network and Collaborative Platform for Public Health Disease Surveillance

From back of the napkin concept, to initial prototype and production, and finally to implementation, here is the story of Riff...
 

View more presentations from Taha Kass-Hout, MD, MS.

Dengue Fever in Florida (in the US?)

An outbreak in Key West, Fla., has re-emerged in April 2010. Historically, dengue was in Florida in the early 1900, but locally-acquired cases had not been confirmed since then until August 2009.  The recent Florida outbreak might have gone unnoticed if it hadn't been for an astute physician in upstate New York who identified the first case in Rochester, N.Y., involving a patient who had just returned from a week-long visit to Key West.

In the US, dengue fever has rarely been an issue outside of the Texas-Mexico border region, but currently it is estimated that more than half of the US population live in places which contain one or both of mosquito species capable of transmitting the dengue fever virus creating conditions more favorable for an outbreak.  Dengue is now [starting last year] a reportable disease in the US due to its increasing threat.

Dengue fever is characterized by agonizing aching in the bones, joints and muscles, a pounding headache, pain behind the eyes, a high fever and a classic rash. There is no cure or vaccine, only preventative and supportive care. 

You can collaborate around Dengue on Riff here

2009, AMIA Spring Congress

Last week, I presented Evolve; InSTEDD's Global Early Warning and Response System, at the 2009, American Medical Informatics Association (AMIA) Spring Congress. The conference took place at the Walt Disney World Swan, May 28th–30th, in Orlando, Florida, USA. Here is the presentation:

View more OpenOffice presentations from Taha Kass-hout.

Related Links:

• Influenza A(H1N1) Media Hype: Mid-March 2009 thru May 19, 2009
  
• Tracking A(H1N1) using Evolve
  
• Low volume Evolve announcements on Twitter
• Extremely Affordable Health Innovations
  
• MBDS ICT and Technology Forum
  
• Best Poster Award for Improving Public Health Investigation and Response at the Seventh Annual International Society for Disease Surveillance Conference
• Collaborative Analytics and Environment for Linking Early Event Detection to an Effective Response

Influenza A(H1N1) Media Hype: Mid-March 2009 thru May 19, 2009

Data source: A(H1N1) Evolve Collaborative Workspace

Note: Tag cloud was created on Many Eyes © IBM

Note: Tag cloud was created on Many Eyes © IBM

Note: Map was created on Many Eyes © IBM

Related Links:

• Tracking A(H1N1) using Evolve
  
• Low volume Evolve announcements on Twitter
• Extremely Affordable Health Innovations
  
• MBDS ICT and Technology Forum
  
• Best Poster Award for Improving Public Health Investigation and Response at the Seventh Annual International Society for Disease Surveillance Conference
• Collaborative Analytics and Environment for Linking Early Event Detection to an Effective Response

Tracking A(H1N1) using Riff

Last week, InSTEDD's CEO; Dr. Eric Rasmussen, blogged about the Riff workspace we stood up earlier to further aid experts and responders collaborating around emerging reports related to the 2009 A(H1N1) pandemic influenza. To-date, there's been massive news coverage around the event, but in order to make sense of it all, a group of experts (with a backgrounds in public health, international relations, diplomacy, social work, and emergency response) volunteered their time to collaborate around the various streams of information (listed below).



One by-product of this ongoing effort is that the information is appropriately tagged and geo-located. The A(H1N1) workspace allows you to subscribe to all the information or to a filter of your own. The tags include information beyond just a disease category, symptom, or syndrome, but we also tried to capture other important information; such as policy issues (e.g., vaccination, school closure, travel advisory, etc.). However, our primary goal is not to become another information source, rather to be able to provide a good situational awareness of the event in order to respond effectively. We are trying to address the following problems that are inherent in the current early detection systems:

• Classic problem: too much data, not enough information. Why aren’t the key indicators noticed earlier?
• Noisy data—low reliability—
• Need to keep the human in the loop (a lot of this is still an art).
• Sources not always obvious - we saw emerging sources of information, like the citizen reporting over a Google map (in the past much attention was paid to sources like Internet search queries (e.g., Eysenbach, Ginsberg, Polgreen, Hulth, and Cooper), over the counter medications sales (e.g., Wagner/RODS Lab), absenteeism (e.g., Wagner/RODS Lab), as opposed to ER chief complaints or routine disease surveillance hierarchical systems).
• Threat profile keeps changing- is not known for something like a SARS/SARI (e.g., Swine Flu or other things like it).
• Political and Organizational Boundaries (Note that we weren't able to perfectly communicate the first indications of SARS, or the outbreaks of H5N1 in China, and we saw that happen again with the H1N1 Swine outbreak).
  
• Need to set up the need for shared collaboration spaces and geographic distribution. Not only do we need human experts in the loop, but they need to share the hunches and concerns. Discussion needs to identify communities of interest. Different specialists are involved based on the nature of the threat.

I particularly applaud the effort by HealthMappers (Dr. Brownstein, et al) for quickly putting together an A(H1N1) mashup (or the New England Journal of Medicine HealthMap) which tracks the cases (confirmed, suspect, dead, or ruled out) alongside with the informal sources that HealthMap continuously monitors and moderates. HealthMap and BioCaster also setup moderated tweets on Twitter that are timely, reliable and of high quality.

Information Sources:

1. EISS Weekly Electronic Bulletin
2. Canada - FluWatch
3. USA - Centers for Disease Control and Prevention - FluView
   
4. Health Information for International Travel The Yellow Book
5. Moreover Public Health News
6. BBC Outbreak News
7. WHO Outbreak News
8. CDC Flu Updates
9. WHO Latest news on the avian influenza situation in humans around the world
10. EID Podcasts
11. EID Journal
12. Recent Outbreaks and Incidents
13. CDC Emergency Preparedness & Response
14. Google Outbreak News
15. Flu Stop with CDC
16. ProMED Mail (including ProMED MBDS)
17. FDA Twitter Feed
18. CDC MMWR
19. Eurosurveillance
20. Clinician Outreach and Communication Activity
21. Y! Health Cold & Flu News
22. CNN Health News
23. VitalStats
24. HEDDS Surveillance News
25. CDC en Español
26. Public Health Matters
27. WDIN Disease Map Digest
28. CDC Travel Notices
29. Citizen reporting using Google Map
30. Various Twitter feeds (including HealthMap, BioCaster, EpiSpider, and Veratect)
31. Google Insights for Search
    

The Challenge Ahead:

While all this is important, I'd like to emphasize the fact that the best data is still coming from old-fashioned shoe leather epidemiology. We have to be true to ourselves and remember that, with all the information, new tools, open networks of collaborators, etc., we still missed the early indication(s) of the A(H1N1) outbreak. I remember when I was in the trenches of SARS back in 2003, we didn't have the breadth and depth of the information nor the tools we currently have. This calls for an action to rethink our strategies around early detection especially for emerging infectious diseases...

Related Links:

• Low volume Riff announcements on Twitter
• Extremely Affordable Health Innovations
  
• MBDS ICT and Technology Forum
  
• Best Poster Award for Improving Public Health Investigation and Response at the Seventh Annual International Society for Disease Surveillance Conference
• Collaborative Analytics and Environment for Linking Early Event Detection to an Effective Response
  
  

Open Collaboration Experiment around Emergency Alerts, Disasters, Market Recalls, and More...

We set up two open collaboration spaces around up-to-date notifications and alerts on the following:

- Emergency Preparedness & Response alerts, Recent Outbreaks and Incidents, Clinician Outreach and Communication Activity (COCA), and Public Service Announcements for Hurricanes from US CDC

- Recalls, market withdrawals and safety alerts from the the U.S. Food and Drug Administration (FDA)

These spaces offer a suite of collaborative features, including: commenting, tagging (adding your own keywords), mapping (user generated and automated), relating multiple alerts, searching and filtering, specifying a time window, adding attachments, subscribing (currently in the form of a web-friendly format also known as RSS, but in the very near future we will support email subscription), and more. The two spaces are also equipped with an intelligent process (referred to as machine-learning algorithm) that "learns" from inputs provided by the users (e.g., adding a keyword (tag), or correcting the mapping of an item). This intelligent process quickly and accurately learns from users inputs (with 95% confidence based on previous tests) and starts suggesting tags as well as correcting itself and offering even better results over time (by simply accepting or rejecting a suggested tag or an automatically mapped location).

Have you an alert or item you like to share with the community? You can easily contribute that alert by clicking the "Add Item" feature.

We invite you to try it and help us spread the good word by inviting others as well. We look forward to your feedback and comments in order to make this tool as useful as it can possibly be. If you have thoughts or questions you can contact us at info@instedd.org or email me directly at: kasshout@instedd.org.

World Health-Related Events

Data is aggregated and moderated on routinely basis using InSTEDD's Collaborative Early Event Detection and Response space. Sources of information include the following (in alphabetical order):

- BBC Outbreak News

- BioCaster Open Source Ontology

- CNN Health News

- EpiSpider

- Eurosurveillance

- Google Outbreak News

- HealthMap (Global Disease Alert Map)

- HEDDS Surveillance News

- Moreover Public Health News

- ProMED News (including MBDS)

- Veratect Corporation

- WDIN Disease Map Digest

- WHO Latest news on the avian influenza situation in humans around the world

- WHO Outbreak News

- Y! Health Cold & Flu News

Related Links:

• Collaborative Analytics and Environment for Linking Early Event Detection to an Effective Response
• Best Poster Award for Improving Public Health Investigation and Response at the Seventh Annual ISDS Conference
  

Note: Tag cloud was created on Many Eyes © IBM (click here for a live version)

Best Poster Award for Improving Public Health Investigation and Response at the Seventh Annual ISDS Conference

Earlier this month, I presented our vision for an Integrated Global Early Warning and Response System at the Seventh Annual International Society for Disease Surveillance Conference, December 3-5, 2008 at the Raliegh Conference Civic Center. We received the most votes in the "Best Poster" award category for the "Improving Public Health Investigation and Response" track. You can also download the article abstract from here.

Related Links:

• Collaborative Analytics and Environment for Linking Early Event Detection to an Effective Response
• Next steps for Riff
• Humanitarian FOSS (HFOSS) Project Summer Institute 2008
• ProMED

Epi Info™ and Mesh4x Prototype Demonstration with US CDC

On Thursday, December 18th, 2008, we gave a joint theater-style demonstration of synchronizing Epi Info™ Data using Mesh4x with the Division of Integrated Surveillance Systems and Services (NCPHI/DISSS) at US CDC.

[The Epi Info™ team: David Nitschke (lead) (left), Roger Mir (middle) and Mark Berndt (right). According to our imaginary scenario (where we extended the sample Oswego outbreak from 1940), David is the NY State epidemiologist, Roger is the Oneida county Medical Officer, and Mark is the CDC epidemiologist]

Here is a presentation and script scenario which walks you through the scenario step-by-step. You can download the latest Mesh4x tool from here, which also includes the sample data. [If you do not already have a copy of Epi Info™, you can download it from here]

Epi Info™ ─ Mesh4x Synchronization Prototype

View SlideShare presentation or Upload your own. (tags: technology david)

Oswego in the Cloud: Scenario Script

View more documents from Taha Kass-Hout.

During this proof-of-concept, we showed the utility of a Mesh4x tool for synchronizing Epi Info™ data over the cloud (web) and SMS (Please see my previous blog in which I introduced this effort: "Empowering Epidemiologists to Share Information, Anytime, Anywhere: Epi Info™ and Mesh4x"). Epi Info™ is now available as an Open Source project (Please see Official US CDC MMWR release notice and the most recent Government Health IT article: CDC takes its epidemiological software open source).

[The Epi Info™ ─ Mesh4x Synchronization Tool]

[Synchronization over over the Amazon EC2/S3 cloud (State’s available online data in the scenario)]

The Synchronization over SMS using cell phones provides great potential for sharing data among field epidemiologists conducting investigations in areas with limited resources and infrastructure, especially in austere conditions (e.g., during or after disasters).

[Synchronization over SMS]

As part of the scenario, we showed how to share maps across various investigators (in the scenario Oneida county and the neighboring counties). This is especially true as the epidemiologic investigation in underway, that data is shared in aggregate forms; such as a map with a few pins, before further collaboration. During the demonstration, we used Google Earth as the viualization tool to show the various cases (Ill) and no cases are distributed across space (accurately geocodesd) and time (through a time slider). We integrated Google geocoder with the Mesh4x tool to automatically geocode the sample physical addresses and we provided means in the Mesh4x tool to automatically generate and synchronize the maps across the various counties.

Here I show the maps before synchronization of data (one map for Oneida county (highlighted in pink) on the left and the rest of the counties on the right). After the synchronization was completed, both maps were identical as the counties now have similar data.

After the demonstration, we identified with US CDC a high priority list for next steps, including:

1. Preview/Accept/Reject or "Undo" & Conflict Resolution
2. Schema/view update and propagation
3. Mesh-based authentication & authorization
4. Specify multiple tables to sync
5. SMS-to-cloud and back
6. Client to define mesh, feeds, mappings
7. Privacy & Signatures

[Brainstorming priorities with the Epi Info™ team (from left to right): Roger, Mark and David. Ed facilitated the discussion as you see him going through the projected list]

DISSS is currently seeking a project to continue development of this functionality pending the availability of resources. I'll keep you posted!

We are very grateful for the time and expertise US CDC offered us during this exercise and we wish to further enhance our tools and platform as a result of this effort. I want to personally thank the Epi Info™ team: David Nitschke (lead), Roger Mir and Mark Berndt, and US CDC National Center for Public Health Informatics (or NCPHI) leadership team: Dr. Leslie (Les) Lenert (Director) and Enrique Nieves (DISSS Division Director (Acting)).

[The NCPHI leadership team: Les (left) and Enrique (right)]

I also want to acknowledge my colleagues at InSTEDD who worked really hard over the past six weeks to put this together and to see it succeed, including work done at very early odd morning hours: Juan Marcelo Tondato, Daniel Cazzulino, Pablo M. Cibraro, and Eduardo (Ed) Jezierski.

Finally, on Behalf of InSTEDDers, I want to wish you Happy Holidays and a Happy New Year!

Useful Links:

• Oswego in the Cloud: Scenario Script
  
• US CDC Recommends Mesh4x to Synchronize Data using Epi Info™
  
• Empowering Epidemiologists to Share Information, Anytime, Anywhere: Epi Info™ and Mesh4x
  
• Epi Info™ current website on CDC.gov (available for download Epi Info™ 3.5.1 (last accessed May 16th, 2009))
• Epi Info™ User Forum
  
• Epi Info™ Community Edition (CE)
  
• Epi Info™ Friends Group on Google (Restricted to invited members. To become a member, contact the group owner at andy (dot) dean (at) gmail (dot) com)
• Epi Info™ in Italy
  
• Epi Info™ in Brazil (Portuguese)
  
• Epi Info™ in Spain
  
• InSTEDD Mesh4X
  
• InSTEDD Mesh4x Discussion Group
• Juan Marcelo Tondato's Blog on Mesh4x
• Eduardo Jezierski's Blog on Mesh4x
• Taha Kass-Hout's Blog on Mesh4x
• Daniel Cazzulino's Blog on Mesh Architecture
  
• Pablo M. Cibraro's Blog on Mesh Architecture
• Public Health Grid (PHGrid) - Research & Development

Empowering Epidemiologists to Share Information, Anytime, Anywhere: Epi Info™ and Mesh4x

I’m leading a collaborative project with US CDC to establish a proof of concept demonstrating the potential to synchronize data in disparate Epi Info™ installations over the cloud and Short Message Service (SMS) text messages using the tools and libraries of the Mesh4X project. The Epi Info™ team includes: David Nitschke (lead), Roger Mir and Mark Berndt. The InSTEDD team includes: Juan Marcelo Tondato, Daniel Cazzulino, Pablo M. Cibraro, and Eduardo (Ed) Jezierski.

[The Epi Info™ team: Roger Mir (left) and David Nitschke (lead) (right)]

US CDC Epi Info™ is a suite of tools for use by public health professionals in conducting outbreak investigations, managing databases for public health surveillance, and general database and statistics applications. With Epi Info™, physicians, nurses, epidemiologists, and other public health and medical workers can rapidly develop a questionnaire, customize the data entry and validation process, enter and analyze data. Epi Info™ offers adaptability to changing requirements, growing demands, and innovative and scalable public health solutions. Its language and localization features make it portable for national and international missions and events. And, it’s FREE, so developing countries with limited resources can also employ its power. Epi Info™ is now available as an Open Source project [Please see Official US CDC MMWR release notice].

While Epi Info™ is widely used around the world, its implementations have been limited to discrete stand alone applications with no collaborative, peer-to-peer exchange of data or internet connectivity. Data are exported and sent as discrete packets (databases or spreadsheets) to collaborating centers where the data are merged and analyzed. This is a time consuming process and presents a significant limitation especially during an outbreak investigation. This was an opportunity for us to work collaboratively with the US CDC National Center for Public Health Informatics (or NCPHI), directed by Dr. Leslie (Les) Lenert, and demonstrate the value of Mesh4x to meet this challenge.

Mesh4X is a light-weight synchronization platform developed by InSTEDD which provides libraries, tools and applications to simplify interoperability of different applications and services. Ed just posted a blog on the progress of Mesh4x and its various and interesting properties. During this prototype, we developed an adapter for Epi Info™ enabling near real-time data synchronization using the fastest available technology (Internet, wireless Internet, satellite communication, SMS, or flash drive/pen drive) that we believe to be of significant value to the public health community at large.

An example scenario where Mesh4x would be useful is in an outbreak investigation. Many epidemiologists are familiar with the food borne outbreak in Oswego, New York, U.S.A. on April 18^th, 1940. In this outbreak, 75 of the 80 people known to have been present at the pot-luck church supper. A survey was created and interviews were conducted with participants to determine the source of the contamination. While the Oswego study focused on a single region, the significant value of data synchronization can be seen by expanding this scenario to where interviews and data entry are conducted in different localities. Therefore, we recreated the outbreak as if the Oswego church supper was attended by residents of the Oswego county and four other neighboring counties: Jefferson, Lewis, Oneida, and Wayne. In this hypothesized scenario, we imagined two epidemiologists are investigating this outbreak; one investigating the outbreak in Oneida county and the other investigating the other counties. Prior to synchronization, Oneida county had inconclusive results on the cause of the outbreak (baked ham and Vanilla ice cream). After data synchronization, both investigators had a clear picture of the spread of the illness over space and time and concluded the actual source of the outbreak to be from the Vanilla Ice Cream prepared the night before the church supper on April the 18^th. During this scenario, we also demonstrated synchronizing Google Earth maps between localities. We will be demonstrating the solution at the US CDC offices in Atlanta, GA next Thursday December 18^th, I’ll keep you posted!

[Roger and David working on the User Interface]

Related Links:

• US CDC Recommends Mesh4x to Synchronize Data using Epi Info™
  
• CDC/InSTEDD Collaboration blog
• Epi Info™ current website on CDC.gov (available for download Epi Info™ 3.5.1 (last accessed May 16th, 2009))
• Epi Info™ User Forum
  
• Epi Info™ Community Edition (CE)
  
• Epi Info™ Friends Group on Google (Restricted to invited members. To become a member, contact the group owner at andy.dean@gmail.com)
• Epi Info™ in Italy
  
• Epi Info™ in Brazil (Portuguese)
  
• Epi Info™ in Spain
  
• InSTEDD Mesh4X
  
• InSTEDD Mesh4x Discussion Group
• Juan Marcelo Tondato's Blog on Mesh4x
• Eduardo Jezierski's Blog on Mesh4x
• Taha Kass-Hout's Blog on Mesh4x
• Daniel Cazzulino's Blog on Mesh Architecture
  
• Pablo M. Cibraro's Blog on Mesh Architecture
• Public Health Grid (PHGrid) - Research & Development