We present a process to evaluate the continuing development of All Payer Claims Databases (APCDs) using a collaborative evaluation process. Project teams enhanced the Utah APCD with improved analytic capacity to provide online pricing and cost-transparency reports to support health care reform in Utah. Our program evaluation efforts added key methods and tools, building on recommendations in the APCD Development Manual to provide evaluation data facilitating improvements [1] . These additions included a Collaborative Evaluation Model, logic models, and development and use of best practices as measures. Stakeholders found that the added use of best practices, logic models, and frequent feedback to practitioners facilitated the project’s success. Since the Collaborative Evaluation Model served a structural purpose, it was transparent to the project teams.
APCDs currently operate in thirteen states, with five more being implemented [
We present an evaluation process that stakeholders viewed favorably. Stakeholders, such as employers, the public, insurers, and payers, all participate in healthcare decisions. All are influential drivers in cost reduction [
Utah has had an APCD since September 2009, with 21 health insurance carrier plans submitting enrollment, pharmacy, and medical file data as early as 2007 [
The All Payer Claims Database Development Manual (2015) suggests that several components are essential for APCD development [
Before the Cycle III grant award, the OHCS had a limited ability to undertake needed analyses to disseminate meaningful information, including price transparency, to facilitate healthcare reform efforts. While OHCS hosted the APCD in SQL databases―where authorized users can access data files and perform analyses using statistical software such as SAS or Stata―OHCS was dependent upon vendors to produce standard reports. For example, the APCD did not incorporate information on insurance premiums, rates, benefits, risk adjustment, or quality metrics. The lack of these data reduced the utility of the available information to inform consumer choices about healthcare services.
We used a Model for Collaborative Evaluation (MCE) built upon six precepts utilized in unity with stakeholders throughout the project: 1) Identify the situation; 2) Clarify expectations; 3) Establish a shared commitment; 4) Ensure open communication; 5) Encourage best practices; and 6) Follow specific guidelines [
To strengthen our evaluation model, we also utilized community-based participatory research (CBPR) principles as described in Sandoval et al. 2011 [
We emphasized context, group dynamic processes, the APCD as an intervention, and outcomes in our work.
other important aspects of the context of the evaluation we were undertaking. We also used a process in which there was equity of stakeholder groups and participants. The concept of equity was critical to establish bi-directional communication essential to the collaborative evaluation approach. To iteratively develop the APCD, we focused on the improvements needed to produce required results such as creating data to assist providers in reducing the cost of healthcare, assisting the Utah Insurance Department with insurance rate review, and empowering consumers to make better decisions about healthcare expenses. Last, we focused on the desired outcomes in terms of being able to use the APCD to provide data to fulfill important uses cases determined by our stakeholders. Important uses cases include developing asthma measures, assessing falls in the elderly, and increasing price transparency for maternity episodes of care.
We developed the process of change statement as well as our evaluation plan, including our evaluation questions presented in
Initial Evaluation Questions | Are best practices in large-scale Health IT being used in the project? |
---|---|
Are best practices in use for administrative data, and continuous quality improvement used in the project? | |
Are best practices in IT development used? | |
Is documentation provided, and standards being used? | |
Is the information in the report/data extract from the APCD useful for the stakeholders for key use cases? |
other methods used in the evaluation were developed in partnership with the project team and key stakeholders.
During the development of the evaluation, we considered the recommendations made in the APCD manual as listed in
Technical Manual Category | Technical Manual Component | Cycle III Evaluation Plan |
---|---|---|
Engagement | Develop use cases | Iteratively and collaboratively develop use casesa |
Identify data needs | Collaboratively engage each development team | |
Articulate APCD goals | Develop logic models for each stakeholder groupa | |
Develop logic model for entire APCD projecta | ||
Identify and engage stakeholders | Regularly engage stakeholders | |
Clarify expectations | ||
Establish a shared commitment | ||
Governance | Receive authorizing legislation | Determine whether an IRB or exemption is necessarya |
Describe data collection and release rules | Use and track best practices in Healthcare Privacy and Security | |
Participate with governing board | Meet regularly with Steering Committee | |
Meet with governing stakeholders individually | ||
Funding | Clarify funding and budget | Participate in grant application developmenta |
Technical Build | Evaluate that data releases and stages support use cases | Iteratively evaluate staged data release with stakeholders |
Gather and track needed data elements as they support the use cases | ||
Describe core data elements and format | Develop selection criteria matrix for technical design | |
Use quality assurance | Use and track best practices in data quality | |
Use continuous quality improvement | ||
Develop data submission manuals | Gather and track submission manuals to improve quality control | |
Analysis and Applications Development | Gather and develop data policy principles | Use and track best practices in state-regulated software development life cycles |
Utilize a technical advisory group | Regularly meet with technical teams and leaders | |
Use and track best practices in large-scale Health IT implementation | ||
Describe data use and release | Gather and track data release documentation | |
Feedback Loops and Continuous Engagement | Foster inclusiveness of all groups | Ensure open communication |
Engage stakeholder groups individually | ||
Utilize transparent and open process | Make evaluation plans and materials available to stakeholders | |
Manage stakeholder expectations | Provide methods for anonymous feedbacka | |
Iteratively and collaboratively refine logic modelsa | ||
Continuously evaluate if project is on course | Regularly interview project management | |
Regularly review and refine evaluation plan |
aAdditional refinement executed by the Evaluation Team to improve upon what is specified in the APCD Development Manual based on our program evaluation approach.
added key methods and tools, building on recommendations in the APCD Development Manual, to provide evaluation data facilitating improvements discussed in the Cycle III grant [
We conducted meetings to establish collaboratively use cases for effective evaluation of the APCD, using an iterative approach. In early 2014, we first met with the Principal Investigator who provided names of key informants for our collaborative efforts. We then helped plan and facilitate a UDOH stakeholder brainstorming session. We prepared topics of interest for the session including questions about data sources and needs, analytic tools, reports, and documents. In this session, we led an open but guided discussion to generate ideas from various teams in the Utah Department of Health. Some questions that we asked to direct the discussion were:
・ What kinds of reports do you want to have?
・ What data would you want in a de-identified data set?
・ What data sources do you typically use?
・ What statistical or analytic tools do you use and envision needing?
・ Is there anything else you would like to share with us about how the APCD can be used to support your work?
Based on suggestions from the event, we developed a large set of use cases that we shared with all project groups. Project teams continue to prioritize and iteratively develop the use cases.
During Years 1 and 2, we conducted 49 meetings and exchanged numerous emails with various stakeholders―including project leaders, development teams, other state APCD teams, consumer engagement groups, and individual key informants―to collaboratively establish the use cases and determine what we should use to evaluate the APCD effectively. We reflect the results at the bottom of
In our program evaluation process, we used additional techniques that built upon the APCD Development Manual [
In accordance with the collaborative evaluation model, we worked with internal stakeholders, including Cycle III collaborators, UDOH staff, and external stakeholders, to develop the model to evaluate the overall project and the vision for evaluation [
We shared the draft evaluation plan with stakeholders in late 2013, shortly after initiation of the Cycle III grant, as part of a collaborative development of the final plan. Collaborative evaluation was a new concept to most project leaders and members so, throughout the first half of 2014, we met frequently with various stakeholder groups to present formalized work plans, metrics, and logic models. We followed our dissemination efforts with an initial survey to evaluate our process at the end of Year 1. We surveyed the leads for each aim regarding use of collaborative evaluation as part of our plan development. We followed this with a second survey in Year 2 to determine how well we accomplished this. The Year 2 survey asked whether the collaborative and iterative engagement of the evaluation team contributed to the overall success of the project and development of the use cases. We administered a second survey to gather feedback about our evaluation activities in Year 2. This survey included three instruments with different questions tailored to specific project groups based on work assignments. We based questions on a five-item Likert scale assessing statements about the evaluation plan and its contribution to the APCD project. Response choices were Strongly Agree, Somewhat Agree, Neither Agree nor Disagree, Somewhat Disagree, Strongly Disagree. All responses were anonymous.
We developed the evaluation questions in
We used logic models as a bridge to understanding for the program teams associated with all aims of the work. We developed an overall logic model for the project (Appendix 1) as well as one for each aim. Especially at the beginning of the project, the logic models helped to identify critical inputs, activities and participation (outputs) to achieve the overall project outcomes as well as the interconnectivity of each part of the project. We used logic models because they provide an overview of critical elements and because they facilitate use of project management techniques by program staff for each project aim. As part of the Year 2 survey, we created two questions to gather feedback from project managers who worked with the evaluation team to design and distribute logic models throughout the APCD project.
We identified relevant best practices in large-scale health information technology (HIT) development, administrative data use and quality, healthcare information security and privacy, and Solutions Development Life Cycle (SDLC). We identified best practices by an initial literature search or by consulting an active UDOH Department of Technology Services internal policy. For each aspect of large-scale HIT, we searched its name and best practice. We analyzed the assessment of best practices in the top scientific and gray literature search results for each category. We undertook literature searches to identify best practices for data quality and healthcare information security and privacy. We based SDLC best practices on an internal UDOH policy. We shared best practices with each project team and iteratively developed the practices through 56 meetings and nine email communications during which we revised the practices 48 times. Final versions of best practices were used annually to assess team performances. Each team’s work was assessed with a designation of partial use, ongoing use, and complete use at year’s end. We shared assessments with project teams and revised them for accuracy as necessary.
We assessed five use cases to evaluate the success of APCD data application (
Evaluation Area | Evaluation Question and Use Case |
---|---|
Use of Best Practices | Are best practices in large-scale Health IT being used in the project? |
Are best practices in administrative data use, data quality and continuous quality improvement used with regard to the data used for the use cases? | |
Are best practices in Healthcare Information Security and Privacy being used? | |
Are best practices for software development life cycles (SDLC) being used? | |
Use Cases | Determine asthma incidence and control. |
Use of APCD to capture tumor markers for the SEER Registry. | |
Use of APCD to support UID Effective Rate Review. | |
Undertake opioid surveillance. | |
What is the cost of maternity in Utah? |
with multiple stakeholders to best represent actual users of APCD data, such as internal UDOH researchers and staff, external collaborators, and consumers in the general public. Once the APCD implementation had been completed, we selected and interviewed key informants to represent each of these stakeholder groups to obtain feedback about their experience with the APCD in fulfilling their requests and use cases. Interviewees included epidemiologists, professional and academic researchers, informaticists, consumers, and patient advocates.
We carried out the interviews in a semi-structured format, obtaining consent before each interview. While we left the interviews as open-ended as possible, we used research questions to guide our discussion with informants. These research questions were:
1) What did the stakeholder express about:
a) Requesting data?
b) Applying APCD data to a use case?
c) Utility/value of APCD use?
d) Quality of the data?
e) Effectiveness of the staff?
f) Ease of use of the data?
g) Security of the data?
2) What other issues were communicated from the stakeholder?
Once we had completed our interviews, we used a qualitative analysis on our gathered responses. Summarize snippets of feedback can be found in Appendix 1. Overall, feedback about the APCD was very positive. All of our interviewees were successful at utilizing APCD data for their needs, and expressed positive results. The most common difficulty expressed by APCD users was the initially using the APCD data, as those without claims data experience had a higher learning curve than those that had previously used claims data in other applications. However, all interviewees were able to overcome difficulties along the way, largely due to positive interaction and assistance from UDOH staff. Based on interviewee responses, this commitment to assistance from APCD staff was critical to user success. It should be noted, however, that we were only able to evaluate four of the five use cases, as we were unable to evaluate the use of the APCD to support effective rate review due to time constraints and technical barriers for the stakeholder.
In addition to our findings, there has been further support of successful application of the improved APCD. On December 15, 2016, a community showcase was held to describe the success of a variety of use cases that had utilized APCD data [
We gathered feedback about the evaluation process with surveys. We used survey results to clarify the goals of the collaborative evaluation model by sharing the formalized evaluation plan including the process of change and evaluation process framework. We began with project leadership, subsequently met with the various stakeholder groups, and then provided updates on our progress throughout Year 2.
The three surveys focused on HIT, logic models, and collaborative evaluation (Tables 4-6). Our logic models were well received but there was confusion regarding the evaluation team’s role and questions about the utility of the collaborative evaluation model. Satisfaction with the model and understanding of its use were mixed. Half of the respondents found the process useful, one quarter were ambivalent and one quarter found the process intrusive. Satisfaction with the model and understanding of its use likewise varied: 50% found the model useful, 25% were ambivalent, and 25% found the model intrusive.
The majority of respondents specifically felt the evaluation plan positively contributed to the Cycle III grant project, while a minority felt neutral and an
Survey/Question | Strongly Agree | Agree | Neutral | Somewhat Disagree | Strongly Disagree |
---|---|---|---|---|---|
Best Practices | |||||
Q1. HIT | 3 | 2 | 0 | 0 | 0 |
Q2. Data quality | 4 | 1 | 0 | 0 | 0 |
Q3. SDLC | 2 | 1 | 2 | 0 | 0 |
Q4. Privacy & Security | 3 | 2 | 0 | 0 | 0 |
Total: | 12 | 6 | 2 | 0 | 0 |
Survey/Question | Strongly Agree | Agree | Neutral | Somewhat Disagree | Strongly Disagree |
---|---|---|---|---|---|
Model for Collaborative Evaluation | |||||
Q1. Overall success | 1 | 4 | 3 | 0 | 0 |
Q2. Use cases | 4 | 1 | 2 | 1 | 0 |
Total: | 5 | 5 | 5 | 1 | 0 |
Survey/Question | Strongly Agree | Agree | Neutral | Somewhat Disagree | Strongly Disagree |
---|---|---|---|---|---|
Logic Models | |||||
Q1. Understand tasks | 1 | 2 | 0 | 0 | 0 |
Q2. Understand goals | 1 | 2 | 0 | 0 | 0 |
Total: | 2 | 4 | 0 | 0 | 0 |
even smaller minority felt that the plan did not positively contribute. This feedback indicated the need for ongoing communication about evaluation goals. Several respondents expressed that the logic model and collaborative model provide “valuable guidance” for “what tasks/activities need attention”. Others reported being unclear about the role of the evaluation team and did not appear to understand the actual focus of the evaluation. They felt the team should be “focusing on an evaluation of the impact of the project”, although discussions emphasized the focus on the plan’s processes.
Engagement with our collaborative stakeholders remained positive throughout Year 2, and we again prepared a survey to gather feedback. We designed three surveys in Year 2 to gather information from specific groups based on their work, increasing the potential for all respondents to provide valid responses. Survey questions ranged from use of the collaborative model and the evaluation team’s role in the project, to some of our more visible contributions, such as best practices tracking and logic models.
Feedback from our Year 2 survey yielded overwhelmingly positive results and showed that our stakeholders had increased confidence in our collaborative efforts. The majority of respondents specifically felt the evaluation plan positively contributed to the Cycle III project, while a minority felt neutral and an even smaller minority felt that the plan did not positively contribute. This was a noticeable improvement from the previous year’s survey, demonstrating improvement in sharing and utilizing the collaborative evaluation model in the APCD development project. We developed our evaluation plan through 71 communications with 16 project groups. Responses were overwhelming positive, with 90% of all responses falling under “Strongly Agree” or “Agree”. The remaining 10% were neutral, which all pertained to the use of best practices in SDLC. Respondents felt that the most valuable best practices were those in data quality (80% “Strongly Agree”; 20% “Agree”).
Out of 13 people offered the survey, eight responded. Respondents had varied opinions of the collaborative model’s usefulness. In the initial survey, there was confusion about the model and its role. However, the second year survey showed an overall improvement in stakeholders’ views and understanding of the collaborative evaluation, as the majority of responses (63%) fell into the “Agree” and “Strongly Agree” categories. Interestingly, the question with the most similar responses in all surveys was whether the collaborative and iterative approach facilitated the development of use cases, with most (63%) agreeing or strongly agreeing and one respondent somewhat disagreeing.
Of the six people offered the survey, three responded. We asked if logic models improved the teams’ understanding of Cycle III tasks and goals. This survey reconfirmed that overall reception of logic models was positive. All responses were positive, with two answering “Agree” and one answering “Strongly Agree”. Additionally, one respondent commented that the logic models could have benefited from another update at the beginning of Year 2.
In addition to discussion during program management and other project meetings, communication about the logic models occurred in 21 emails. All four logic models were modified twice during the grant to reflect progress toward completion. The models identify critical inputs, activities and participation (outputs) to achieve the overall project outcomes as well as the interconnectivity of each part of the project. Logic models were used because they not only provide an overview of critical elements but also facilitate use of project management techniques by program staff in each project Aim.
During Years 1 and 2, we evaluated the use of best practices components to improve claims data quality, large-scale HIT development, healthcare information security and privacy, and software development. As reflected in
UDOH obtained funds from the Cycle III rate review grant to further develop the APCD [
Year 1 | Year 2 | Year 3 | ||||
---|---|---|---|---|---|---|
Partial | Ongoing | Completed | Ongoing | Completed | Completed | |
Data Quality | 0% (0) | 37% (7) | 63% (12) | 37% (7) | 63% (12) | 100% (19) |
HIT Development | 14% (4) | 43% (12) | 43% (12) | 43% (12) | 57% (16) | 100% (28) |
Privacy and Security | 5% (1) | 21% (4) | 74% (14) | 26% (5) | 74% (14) | 100% (19) |
Software Development | 10% (4) | 50% (20) | 40% (16) | 55% (22) | 45% (18) | 100% (40) |
Totals: | 8% (9) | 41% (43) | 51% (54) | 43% (46) | 57% (60) | 100% (106) |
which is new for our stakeholders.
Framework elements such as best practices are foundational to the support of use cases and are essential to accomplishing the project’s objectives. As noted in the APCD Development Manual, obtaining funding for further development is important. Collaborative program evaluation provided additional data, processes, and value to facilitate successful completion of the project. The evaluation process will benefit stakeholders by improving online pricing and cost transparency reports for consumers, employers, researchers, and the public in Utah.
The additional methods used in Utah may be beneficial for other states developing an APCD, especially if use cases of high value and HIT (such as analytic and other software) are being developed and used. The collaborative approach requires a significant number of meetings with stakeholders and this may not be feasible for all projects. Most important, best practices can play an important role in helping high risk, large scale HIT projects achieve success.
A collaborative program evaluation approach, including the use of best practices in developing and implementing enhancement for APCDs, builds on the foundation provided by the APCD Development Manual [
The use of a collaborative approach in this APCD evaluation included key methods and tools, recommendations from the APCD Development Manual, the use of a collaborative evaluation model, logic models, and development and use of best practices as measures. Stakeholders felt that our transparent evaluation efforts facilitated the project’s success.
This publication is funded by CMS Grants to States to Support Health Insurance Rate Review and Increase Transparency in Health Care Pricing, Cycle III (Grant Number: 1 PRPPR140059-01-00), through a subcontract with Utah Department of Health Office of Health Care Statistics.
No conflict of interest is declared by the authors.
Waiver for human subject research by IRB # 00070963.
Garvin, J.H., Cardwell, J.H., Doing-Harris, K., Bolton, D., Snow, L.A., Hawley, C.W. and Xu, W. (2018) Collaborative Evaluation for the Utah All Payer Claims Database Capacity Enhancement. Technology and Investment, 9, 91-108. https://doi.org/10.4236/ti.2018.92007