Mobile Health Apps Evaluation Dataset

The primary dataset can be downloaded by clicking the link for data in the following format:

This is the publication associated with this dataset: Singh K, Drouin K, Newmark LP, et al. Many Mobile Health Apps Target High-Need, High-Cost Populations, But Gaps Remain. Health Aff. 2016;35(12):2310-2318. doi:10.1377/hlthaff.2016.0578.

What do the variables mean?

Note: Variable names containing spaces or punctuation had these removed to minimize compatability issues when importing for statistical analysis.

General

App ID: This is a unique identifier we assigned to each app. AppIDs ending in b were originally discovered as being present on both the Android and iOS app stores. Some of these may have been ultimately reviewed on only one of the 2 operating systems (OSes) if the app was pulled from one of the two app stores or if the app was non-functional on one of the 2 OSes.

Permission To Name: We asked app developers for permission to name their apps in our study. For more details, see the methods in the original manuscript.

y - the developer gave us permission to name their app

n - the developer did not give us permission to name their app

App Name: This is the name of the app as entered by the reviewer. Though we did not report the developer in the dataset, the names of the developers for each of the named apps are available in the appendix accompanying the manuscript.

Reviewer: Is the reviewer a clinician or non-clinician?

Options are: clinician_reviewer or nonclinician_reviewer.

Platform: This is the type of device on which the app was reviewed. Options are: iPhone, iPad or iPad Mini, Android phone, and Android tablet.

OS: This is the operating system on which the app was reviewed. This was derived automatically from the platform. Options are: iOS, Android, and Both.

Which populations is the app targeting?

Instructions to rater: Answer this question based on the app’s description in the app store and the app’s website. If the app is targeted at caregivers, try to determine the vulnerable populations who those individuals care for. Check ALL that apply. Some apps are very broad (like patient health records) and may be targeted at all conditions. For these types of apps, check only “All of the above.”

Options are: Asthma or COPD, Arthritis, Cancer, Chronic kidney disease, Cirrhosis or liver disease, Coronary artery disease or congestive heart failure, Dementia or mild cognitive impairment, Depression or bipolar disorder, Diabetes mellitus, Elderly, Hypertension, Obesity, Pain, Smoking, alcohol, or drug abuse, Stroke, All of the above, None of the above. The reason that None of the above was included as a response was to ensure that the question was not accidentally missed – at least one selection was required.

How would you best categorize the app developer?

Options are: Individual, For-profit company, Medical professional society, Other non-profit organization, and Government agency.

What kinds of support does the developer offer?

Instructions to rater: You may need to check the app store page and app website to determine the answer.

Seelct all that apply. Options are: E-mail, Phone, Live chat, Text messaging/SMS, None of the above. The reason that None of the above was included as a response was to ensure that the question was not accidentally missed – at least one selection was required.

Was an appropriate clinical expert involved in app development or qualitycontrol?

Instructions to rater: Answer this ONLY based on the app, app store page, and website. If you cannot determine this, choose No clinical expert involved.

Options are: Clinical expert involved and No clinical expert involved

Were patients involved in app development or quality control?

Instructions to rater: Answer this ONLY based on the app, app store page, and website. If you cannot determine this, choose No patients involved.

Options are: Patients involved and No patients involved

Does the app make reference to disease-specific guidelines?

Instructions to rater: Guidelines, broadly defined, may refer to professional society guidelines, review articles, or primary literature. If you cannot determine, please answer No.

Options are: Yes and No.

Does the app claim to be compliant with the FDA guidelines surrounding regulation of mobile medical apps?

Instructions to rater: Not all apps are required to be registered as medical devices but should acknowledge their standing with the FDA guidelines. If you cannot determine this, answer No.

Options are: Yes and No.

App Functionality

Does the app require a login?

Instructions to rater: Select yes if the app has no or minimal functionality without users creating an account. Otherwise, select no.

Does the app have in-app advertising?

Options are: Yes and No.

If you answered Yes to the prior question, is it possible to upgrade to an ad-free version?

Options are: Yes, No, and Not applicable (answered “No” to prior question)

Is the app tethered to a healthcare system?

Instructions to rater: Does the app require patients to be receiving care from a participating clinic or hospital in order to use the app?

Can the app sync directly with a peripheral device?

Instructions to rater: Examples include wearables (e.g. Fitbit) and other devices (e.g. thermometers, glucometers, and scales). If transferring data between a peripheral device and the app is a highly manual process (exporting a CSV file and reimporting a CSV file), then this does not count (answer No).

If the app is an iOS app, is it compatible with HealthKit?

Instructions to rater: This info should be on the app page description on the app store.

Options are: Yes, No, Not applicable (Android app)

If the app is an Android app, is it compatible with Google Fit?

Options are: Yes, No, Not applicable (iOS app)

Privacy and Security

Does the app have a privacy policy?

Instructions to rater: You may need to look on app page on the app store and on the app’s website. If you cannot find one, answer No.

Does the app developer provide user information to others individually or in aggregate?

Instructions to rater: This should be addressed in the privacy policy. Again, this may require looking at the app’s website.

Options are: Yes, individual information, Yes, in aggregate only, No, Not stated.

Does the app claim to meet the standard of HIPAA compliance?

Instructions to rater: You may need to look on app page on the app store and on app website. If you cannot determine this, answer No.

Patient Engagement

In what ways does the app engage patients?

Select all that apply. Options are: Provides educational information, Instructs patients (app provides guidance or advice based on information you provide it), Records information, Displays patient’s health information, Reminds or alerts patients, Enables communication of information with clinician, Enables communication of information with family (e.g. caregiver), None of the above. The reason that None of the above was included as a response was to ensure that the question was not accidentally missed – at least one selection was required.

Does the app reward the user for engaging with the app or achieving health goals?

Instructions to rater: Rewards could be in the form of points, games, rankings, or social media shares (e.g., app posting achievements to Facebook).

Options are: Yes and No.

Does the app engage users through social media? If so, how?

Options are: Public (e.g. Twitter, Facebook, Pinterest, etc), Private (e.g. app connects you to other patients through its own social media platform), Through a web-based forum, None of the above. The reason that None of the above was included as a response was to ensure that the question was not accidentally missed – at least one selection was required.

Does the app appropriately warn users, caregivers, or clinicians when dangerous information is entered?

Instructions to rater: As an example, does the app respond appropriately to very low or high blood sugars or very depressed/suicidal mood? The warning should reflect a sense of urgency that matches the degree of danger.

Options are: Yes and No.

How does the app handle communication or sharing of information with CAREGIVERS?

Instructions to rater: If the only way for a caregiver to access patient data is to login using the patient’s username/password, this does not count as data sharing.

Options are: Caregiver has access to information using separate login, Patient can e-mail information to caregiver using app, Patient can text information to caregiver using app, None of the above. The reason that None of the above was included as a response was to ensure that the question was not accidentally missed – at least one selection was required.

How does the app handle communication or sharing of information with CLINICIANS?

Instructions to rater: If the only way for a clinician to access patient data is to login using the patient’s username/password, this does not count as data sharing.

Options are: Clinician has access to information using separate login, Patient can e-mail information to clinician using app, Patient can text information to clinician using app, Patient can enter information directly into electronic health record or patient portal using app, None of the above. The reason that None of the above was included as a response was to ensure that the question was not accidentally missed – at least one selection was required.

System Usability Scale

The System Usability Scale (SUS) is a simple, ten-item scale giving a global view of subjective assessments of usability. In adapting this scale for evaluating app usability, we replaced “system” with “app.” For each question, option ranges from 1 (Strongly disagree) to 5 (Strongly agree).

Note: This question was completed only by non-clinician reviewers.

I think that I would like to use this app frequently
I found the app unnecessarily complex
I thought the app was easy to use
I think that I would need the support of a technical person to be able to use this app
I found the various functions in this app were well integrated
I thought there was too much in consistency in this app
I would imagine that most people would learn to use this app very quickly
I found the app very cumbersome to use
I felt very confident using the app
I needed to learn a lot of things before I could get going with this app

Costs: As an example, if the assigned app is free but a “Pro” version is available for $4.99 with the option of a subscription to premium content for $10/month, enter 0 in the first question below, 4.99 in the second question, and 120 in the last question.

Net Promoter Score

How likely is it that you would recommend this app to a friend or colleague?

Note: This question was completed only by clinician reviewers.

Options range from 0 (not likely at all) to 10 (extremely likely).

Consumer Ratings

iOS Avg Rating: This is the mean rating of all versions of an app as abstracted from the app store.

iOS Num Ratings: The is the total number of ratings for all versions of an app as abstracted from the app store.

Android Avg Rating: This is the mean rating of all versions of an app as abstracted from the app store.

Android Num Ratings: The is the total number of ratings for all versions of an app as abstracted from the app store.

Android Num Downloads: The is the total number of downloads for all versions of an app as abstracted from the app store.

Costs

iOS ONLY: Price to download the app (in dollars)

Instructions to rater: If the app is free, enter 0. This is price to download the app only (not including in-app purchases/subscriptions). Enter a NUMBER only (no dollar sign).

iOS ONLY: Price to purchase ALL in-app purchases (in dollars)

Instructions to rater: If no in-app purchases, enter 0. If there are multiple in-app purchases, please add up their cost to get this number. Enter a NUMBER only (no dollar sign).

iOS ONLY: Annual price to purchase a subscription (in dollars)

Instructions to rater: If no subscriptions are offered, enter 0. If monthly subscription cost is available, multiply by 12 to obtain annual price. Enter a NUMBER only (no dollar sign).

ANDROID ONLY: Price to download the app (in dollars)

Instructions to rater: If the app is free, enter 0. This is price to download the app only (not including in-app purchases/subscriptions). Enter a NUMBER only (no dollar sign).

ANDROID ONLY: Price to purchase ALL in-app purchases (in dollars)

Instructions to rater: If no in-app purchases, enter 0. If there are multiple in-app purchases, please add up their cost to get this number. Enter a NUMBER only (no dollar sign).

ANDROID ONLY: Annual price to purchase a subscription (in dollars)

Instructions to rater: If no subscriptions are offered, enter 0. If monthly subscription cost is available, multiply by 12 to obtain annual price. Enter a NUMBER only (no dollar sign)