| General attributes of the data and data system (meta data): Description of data characteristics (e.g., sources of data, types of facilities contributing data, types of patients, and period covered in the data) and the systems used to access and use the data |
1a. Who collects, organizes, and provides the data and what processes are used in doing this? |
Narrative description and schematic
Qualitative assessment of strengths and limitations |
| 1b. What types of organizations or entities contribute the data? |
List of organization or entity types with description (e.g., hospitals and health systems, health information exchanges, ambulatory care providers, FQHCs, insurance companies, employers, laboratories, pharmacies, and state and local health departments)
Number of each type of organization |
| 1c. From what types of health care settings are data available? |
List of health care provider, facility, or setting types from which data are provided (e.g., inpatients, emergency department, outpatient day surgery, outpatients or ambulatory care, telehealth visits, long-term care patients, and postacute care)
Number of each type of setting in most recent years |
| 1d. What types of data are available? |
List of data types (e.g., EHRs and EMRs, discharge summaries, pharmacy or laboratory information management systems, medical or pharmacy insurance claims, hospital administrative data, surveillance systems, and surveys)
Number of patients, visits, or events captured in each type of data |
| 1e. For what time period is the data available? |
List of time periods |
| 1f. How is patient privacy and data security ensured? |
Narrative description
Qualitative assessment of strengths and limitations |
| 1g. What are monetary costs associated with acquisition and use of the data? |
List of costs |
| 1h. Is a data use agreement required and what are, if any, salient restrictions related to data use and information dissemination? |
Narrative description |
| 2a. What data platforms and applications can be used or are needed to access, manipulate, and analyze the data (if applicable)? |
List of the data platforms and applications with description |
| 2b. What salient characteristics of the data platform, including associated software applications and tools, potentially impact (positively or negatively) users’ ability to conduct data manipulation and analysis? |
Narrative description
Schematics or screenshots of the data platforms and applications
Qualitative assessment of strengths and limitations |
| 2c. What is the architecture and digital volume of the data? |
Description of data architecture and schema
Schematics or screenshots of the data platforms and applications
Size of the data sets in terms of digital volume (number of bytes)
Qualitative assessment of strengths and limitations |
| 3. What information is available in the data (what are the data elements) and how is the data organized (e.g., what are the sub-data sets or tables and how are they relatable)? |
Tabular information on sub-data sets or data tables and data elements included in each
Narrative describing the relation or linkability between the tables |
| Data coverage, representativeness, inclusion, and equity: The geographic, population, provider, and health visit coverage and representativeness of the data and how well the data address principles of health equity |
4a. What is the geographic coverage of the data? |
Number of different types of providers (e.g., hospitals and ambulatory care providers) contributing to the data and related resources, nationally and by region, state, county, or zip code
Number of different types of health care visits or events (e.g., ambulatory care, emergency department visits, hospitalizations, laboratory tests, and prescriptions filled) captured, nationally and by region, state, county, or zip code |
| 4b. What is the representativeness of data in terms of the population, types of health facilities and providers, types of health care visits, and other salient health care–related events captured in the data? |
Percentage of different types of existing providers (e.g., hospitals and ambulatory care providers) contributing to the data and related resources, nationally and by region, state, county, or zip code
Percentage of different types of incident or prevalent health care visits or events (e.g., ambulatory care, emergency department visits, hospitalizations, laboratory tests, and prescriptions filled) captured, nationally and by region, state, county, or zip code
Comparative distributions of persons’ or patients’ age group, gender, payor type, and other salient characteristics between events captured in the data and incident or prevalent events in the area or facilities from which the data are derived |
| 4c. How well do the data enable analysis and understanding of factors affecting health equity? |
List of data elements for persons’ race, ethnicity, language, rural or urban location, income, education, poverty status, disability status, sexual orientation, social vulnerability index, and other salient social determinants of health
Number of persons, patients, or events by race, ethnicity, language, rural or urban location, income, education, poverty status, disability status, sexual orientation, social vulnerability index, and other salient social determinants of health, nationally and by region, state, county, or zip code
Comparative distributions of persons’ or patients’ race, ethnicity, language, rural or urban location, income, education, poverty status, disability status, sexual orientation, social vulnerability index, and other salient social determinants of health between events captured in the data and incident or prevalent events in the area or facilities from which the data are derived, nationally and by region, state, county, or zip code |
| Data standardization and quality: Use of standardized data formats and the quality of the data in terms of completeness and validity |
5a. To what extent do the data conform with data standardization principles and recommendations? |
List of key data elements and their definitions with assessment whether they meet recommended standards (e.g., standards established by ONC, OMB, CMS, or OMOP)
Brief narrative summary of data standardization |
| 5b. To what extent is information complete for data elements? |
Frequencies for key data elements, including observations with missing data |
| 5c. To what extent are data values valid values? |
Percentage of observations that have a valid (feasible) value for key data elements |
| 5d. To what extent are data values accurate values? |
Percentage of observations that have accurate values for key data elements (to the extent information is available) |
| 5e. To what extent are data values precise values? |
Percentage of data elements that contain exact quantitative values (as opposed to value ranges), where applicable |
| 5f. To what extent are there duplicate observations in the data? |
Tabular presentation of percentage of patients with duplicate observations and average number of duplicate observations per patient, by data table |
| Data period, periodicity, and recency: Period covered by the data, frequency of data updates, and lag time in the data |
6a. What is the periodicity by which updated data are made available? |
Periodicity of data updates in terms of weeks, months, or years |
| 6b. What is the timeliness of data updates? |
Lags (if any are observed) between scheduled data updates and when the data are actually updated |
| 6c. What is the recency of the data? |
Average, minimum, and maximum difference between date of service for the health care event and when data are available for analysis |
| Versatility of the data: Ability to create unduplicated patient and visit-level observations, analyze patient data longitudinally, develop and apply specific case definitions, and link or integrate the data with other data and related resources |
7. To what extent can data be linked together to form complete and unduplicated individual patient and visit or event-level information? |
Percentage of observations that can be linked to form unique and unduplicated patient health care visits or events
Percentage of health care visits or events for which information available in the data can be linked at the visit or event level |
| 8. To what extent can the data be linked together to form longitudinal patient-level information? |
Proportion of observations that can be linked to form unique unduplicated patient-level information for longitudinal analysis
Percentage of patients for which information available in the data can be linked at the patient level |
| 9. To what extent can the data be linked together or integrated with other data? |
Description of how the data can be linked to other or outside data |
| 10. How well can specific case or event definitions (e.g., COVID-19 inpatient admission, COVID-19 disease severity levels, and COVID-19 vaccination) be formed when analyzing the data? |
List of case or event definitions successfully applied by the data users when exploring and analyzing the data
Narrative summary of feedback from the data users about their experience in developing and applying specific case or event definitions |
| Utility of the data: How well the data can help to address research, evaluation, and programmatic issues of importance |
11a. Can the data and related resources be used successfully to address selected potential research, evaluation, and programmatic questions important to the researcher, program, or organization? |
List of use cases (e.g., topics or research questions) in published and gray literature
List of unpublished use cases (research, evaluation, and programmatic questions) for which data were analyzed and provided informative results |
| 11b. What is a broader range of important research, evaluation, and programmatic issues or questions that are important to the researcher, program, or organization that the data and related resources can potentially help to answer? |
As reported by the data users: list of research, evaluation, and programmatic questions for which the data potentially can be used to effectively address |
| 11c. What are the potential benefits that can be gained through this data when considering other data resources that are also available to the researcher, program, or organization (e.g., does the data and related resources help address issues or questions that are not addressable or not as well addressable using other data)? |
Narrative summary developed by evaluator based on feedback from the data users, other information gathered through the evaluation, and the evaluator’s own assessment |
| 12a. Do the data have information on charges, costs, and expenditures and how specific or itemized are these? |
List of cost-related data elements, their descriptions, and sources they are derived from |
| 12b. What are the sources of cost-related data (e.g., information based on pre- or postadjudicated claims)? |
List of cost-related data elements, their descriptions, and sources they are derived from |
| 12c. What is the potentially utility of the data for conducting cost-estimate, cost-effectiveness, or cost-benefit analysis? |
Narrative summary developed by evaluator based on feedback from data analyst feedback, other information gathered through the evaluation, and the evaluators own assessment |
| 13. Can the data be used to address various potential research and evaluation issues? |
Can the data be used to address the following:
• Monitor health status among populations
• Rapidly identify occurrence of disease and associated comorbidities and outcomes
• Identify and measure risk factors for diseases and outcomes
• Track diverse groups of patients along the care continuum
• Track the history of disease and health status across the lifespan
• Assess persons’ access to and use of health care services and the continuity of care
• Assess persons’ eligibility for and use of preventive services
• Assess alignment to treatment guidelines for various diseases and conditions
• Assess treatment strategies and outcomes
• Describe the impact of social determinants on health status and outcomes
• Support the development of comprehensive population health registries
• Identify early signals of emerging or novel diseases or events (e.g., symptom clusters) of high concern
• Assess population-level disease burden, burden on the health care infrastructure, or both
• Assess cost of care and cost-effectiveness of patient care and preventive care strategies
• Assess genetic characteristics of and variations among causal pathogens
• Assess biological and laboratory markers associated with disease state, severity, and outcomes
• Assess persons’ health-related knowledge, attitudes, and practices |
| Usability of the data and related resources: How easily and effectively the data can be used (i.e., the capability for and level of difficulty in manipulating and analyzing the data for specific purposes using the data platform) |
14a. How do salient characteristics of the data and related resources, including data platform and associated software applications and tools, potentially impact (positively or negatively) users’ ability to conduct data manipulation and analysis? |
As reported by the data users: data manipulation and analysis related advantages and limitations associated with the data, data platform, and associated software applications and programming language needs |
| 14b. How easily are the data accessible and analyzable through the data platform (if applicable)? |
As reported by the data users: activities that were conducted successfully using the platform
As reported by the data users: the level of user friendliness of the platform and associated applications
Descriptions of any problems encountered by the data users when accessing and using the data through the platform
Summary of strengths and weaknesses and usability of the platform and associated applications reported by the data users |
| 15. What is the availability and quality of data-related and the data platform–related documentation, technical support, and training? |
Qualitative summary of the data users’ feedback on the extent to which the documentation, trainings, technical support, and tutorial videos are useful |
| Adaptability of the data and related resources: Ability to change or adapt the data and the mechanisms for accessing and using the data for changing research, evaluation, and program needs |
16. To what extent is it possible to obtain changes to the data (e.g., additions or changes to data elements included in the data sets and how data elements are defined) to meet changing analysis needs? |
Narrative based on information obtained from the data supplier |
| 17. Can changes be made with regard to functionalities of the data platform and how data are accessed, manipulated, and analyzed (if applicable)? |
Narrative based on information obtained from the data supplier |
| Stability of the data: How consistent is the availability of the overall data and data elements over time and with regard to how the data elements are formed and defined |
18a. How constant are the number and types of organization, facilities, and providers providing data across the years for which data are available? |
Difference over time in the number and types of organizations, facilities, and providers providing data |
| 18b. How constant are the data elements across the years for which data are available? |
Difference over time in the data elements available in the data sets |
| 18c. How constant are the definitions of the data elements (including response categories) across the years for which data are available? |
Difference over time in how key data elements are defined and what the respective response categories are |
