Watch on Demand

First Name
Last Name
Country (Citizenship)
Opt-In to Future Emails
Thank you!
Error - something went wrong!

How to Visualize Data to Increase Trial Performance and Efficiencies

August 8, 2016 Heather Bilinski

Identifying hidden patterns in patient behavior can help you run an efficient and successful clinical trial. With powerful new visualization tools for reporting and analysis, CRF Health's TrialManager 6.0 makes it easier than ever to gain new insights into patient behavior and react in real-time to key trial information.

Watch this on-demand webinar to see a live demo on how you can leverage the NEW TrialManager 6.0 reporting tool to make your trial data more actionable, lower monitoring costs by pinpointing lower-performing sites, and better manage your eCOA study in real-time.

Full Transcript



Hello everyone on the call and thank you for joining us today. This is another CRF Health webinar. Today’s topic is how can you use data visualization to increase your trail performance and efficiencies. And we’ll be talking about some of the new reporting capabilities that are available in TrialManager Version 6.

We’ve asked one of our product managers, Miko Pietila, to join us today to present this session to you, having worked on this product for quite a long time. A little bit of background on Miko, he has over 15 years of experience in product management within health and enterprise mobility. And as a product manager here at CRF Health, he focuses almost exclusively on making our reporting and data analysis tools better for our customers. Miko joins us with quite a lot of experience from BlackBerry as well as Nokia, and he has a lot of experience deploying new data solutions both in corporate and consumer sectors.

So welcome, Miko. And we’ll let you take it from here please.


Yeah, thank you for the introduction and thank you everybody, for joining the webinar today.

So we have divided today’s agenda to the three different areas. And the first two topics we discuss are the trends that impact the data monitoring needs in the eCOA trials operations, based on our experience, and also what are the specific codes for data reporting and visualization in those eCOA studies. Then we also in the third part want to focus on demonstrating some real-life use cases, how the new data visualization technologies can be used to increase the efficiencies in these trials. And in this webinar, we are mainly focused on how the study teams and site personnel can increase the operational efficiencies to run these trials. So that’s kind of the scope of the users when we are presenting these examples to you today.

And before we jump to the actual content, we wanted to do a quick poll and ask you what type of tools you use to monitor and collect the data in your studies currently. That will help us to focus the presentation to the most relevant parts. So here you have five different options, going from the most basic ones—Excel spreadsheet types—to the third-party reporting tools, different statistical tools or business analytics tools. Or, if you are using the CRO-provided analysis tools, the eCOA vendor provided portals, like we will show one example to you today, or if you are not measuring at all. So please cover what are you currently using, and we’ll see the results in a few seconds.

Interestingly, it’s quite well spread over the different areas, though there seemed to be a trend there at the beginning. But it looks like it’s fairly equally spread from the very basic tools like Excel spreadsheets to the more advanced tools, business analytics tools, so eCOA vendor-provided portals. But CRO-provided analysis tools seems to be the top one here. So thank you for your input. I think that we will be able to provide you probably some new ideas during the next few slides and then on the actual demos.


So then, moving to the actual content on the webinar. So when we’ve been designing the key metrics for the data reporting, we decided at first to do the survey to better understand what are the key challenges our customers and sponsors in the clinical trials need to solve in these eCOA trials. And there were three key focus areas that were consistently stated in this survey, where we actually surveyed nineteen different CRF customers. And the first challenge that was consistently highlighted as one of the top three, was the data quality and accuracy. And this was about the challenge and importance of monitoring the data quality and proactively identifying and resolving the different structural issues with that data and with the quality, such as the duplicate subject IDs, duplicate data entries, missing data, and so on. So we’ll touch that topic today in a little bit more detail. And the second challenge that was highlighted there was the patient compliance. And our customers in this survey and in the different projects have emphasized the need to have very good dashboard type of summaries about the compliance that really allows them to identify the issues with the patient compliance and in real time follow up with these patients so that they can solve these root causes that are leading to the non-compliance, so they can take preventative actions earlier on in the process. And then the third key challenge there was the need to monitor the patient behavior and then use those insights in either improving the data quality or the quantity. And one example of monitoring the patient behavior is, for example, if they enter the data retrospectively or on time. It’s a fairly simple metric, but it gives interesting insight how you can optimize both the data quality and quantity. And I will demonstrate to you later on how we have used these types of insights in the actual studies. And so in this webinar we'll be looking at several examples of how the data reporting and visualization can be used in solving these challenges. And it's all about the operational efficiencies and how to achieve that in the clinical trials that is here highlighted as category number four.

So moving forward about the trends that are impacting data monitoring in these clinical trials. So we’ve been seeing different trends there. Here are five examples. We’ll expand on these and others later one. But the first trend in eCOA studies is really the increasing complexity of the data that is collected. So there’ve been different medical devices, such as glucose measuring devices and spirometers, being used in clinical trials already for many years. And now they have more and more new medical devices and wearables that are becoming part of the studies, and those are increasing both the amount and the complexity of the data that we are collecting. And that’s one of the trends that increases the importance of visualizing that data, so that even if there is a lot of data, complex data, both the study teams and site personnel need to be able to quickly draw conclusions of what actions they need to take based on that large amount of data. And as this amount of complexity in the data grows, it also increases the importance to actively monitor the data and the quality of the data on a daily basis. And the issues of the data need to be identified very quickly, and users need to be able to react faster in resolving those issues. So it’s no longer sufficient to monitor the compliance on a monthly basis, on a weekly basis. But more often our customers need to have the tools and the visualization that allow them to interact on the following day if they see issue, for example, on the compliance or on the quality of the data. And then the other trend that we are seeing increasingly is the need for the custom reports for the different therapeutic areas and for the different therapeutic areas’ specific metrics. And related to this area, I will show you some examples as part of the demo section. And then lastly there is also the trend from a study operations perspective, there is a need to decrease the study set-up times. And at that point where the instruments or questionnaires are defined for the study, and also when the data collection needs and the reporting needs are defined. So because if this, it’s very important to have standard reports available out of the box during that study set-up time. So the focus there during the week or days that are available for defining reports can be allocated to these custom study-specific reports instead of re-doing all the new standard and basic reports.


So still one more slide about the trends that we are seeing here. And this chart, I will walk you through that. It’s based on the client survey that we did with our partner TriTiCon Consulting to get a more objective view. And it’s based on responses from several people in seven different large pharmaceutical companies. And how to read this chart, is that this shows which items they highlighted to be the most important from a data monitoring perspective and from the operational efficiency perspective in the eCOA studies. So in this one we had 18 individuals who were decision makers related to the eCOA studies and the data monitoring of those, either eCOA or ePRO managers, data management leads from these companies. And I have highlighted here in this chart actually on the red, I have highlighted items that are the most commonly highlighted as a key differentiators or essentials. So even more important than the other ones, by these participants. And like you can see, they’re essential ePRO monitoring, there were three different items there, all of them were highlighted as the key differentiators or essential from the study operations perspective. The first one there in the central ePRO monitoring section is about actually facilitating that central data collection monitoring and then issue resolution. And then the second one that was highlighted there is that it’s critical to be proactively able to identify these issues. So to have the proper tools to do that. Then the third one was operational issue mitigation. So similar type of items that were highlighted in the previous survey. And then, on the active data management, the key item that was highlighted as the most important was the ability to monitor the data trends related to the quality, related to the compliance, and related to the user, like actual patient behavior. And then when we move to the right on the chart, the other two items that were among the top six or seven items on this chart were the data dashboard summaries. So they needed to have that summary views where they can very quickly pinpoint the key metrics that require their attention and require actions to be taken. And then, there was also then the back end where issue resolution goes more to the data management side, so what is the process whereby these different issues will get resolved on the back end side.

So these were the key trends that basically came up in these two different surveys with our customers. And it was really about finding out the data visualization can be used to improve the data quality, compliance, and quantity, and overall the goal was to improve the operational efficiencies in these studies.


Next, I will move to define then, what are the key goals and objectives in using the different new data recording and data visualization technologies to address these different data trends and different needs that our customers have been highlighting. And there are actually four key objectives that come up in our discussions, with the sponsors, with the CROs, and so on. And the first one of them is the need to make that trial data actionable. So again, there’s a huge amount of data there, but if it’s presented in a tabular text-based format it’s very difficult for the users to draw conclusions, what actions they need to take on that. I will talk in more detail how we can use data visualizations to make that data actionable. Then the second one is the ability to explore that data from multiple dimensions. So traditionally, the data recording has been very static, so that the users have a large number of different reports. But now there is increasing need to be able to actually do some analysis on those, kind of ad hoc analysis and exploration of those reports. So drilling from the top level where they can identify issues to one level down or two levels down where they can understand the root cause of what’s causing the issue. The third one is the need to focus on the most relevant, especially when we work with our partners at the site level, so with the site personnel. We all know that those people are very busy in their everyday work, and they don’t have a data management background. So it’s important that they will be seeing the information that is relevant to their specific role in the study. And then the last one there is that by having the best practice reports, a kind of standard set of reports ready out the box in the study set-up, you don’t only save time but you also save money through that. And that has to be compliant then with the study statistic custom reports. And also you’ll quickly see visualizations on all of these issues, and then we will move to the demo part so you can see them working and how the data interactivity and exploration works in practice.

So first, on making the data actionable, the data visualizations. On the left hand side  you see the traditional tabular-based SQL database tables that were traditionally used on the data reporting. And we were not different. This has also been used in the CRF Health TrialManager software. The challenge there is that, when you have a huge number of data—you may have thousands of subjects participating in the study, then there’ll be different types of medical devices, increasingly complex instruments—it’s very difficult to take and draw any conclusions of what actions I need to take based on that data. So when we move that same information to the more visualized data reporting, there are key benefits and the key goal is to allow the user at one glance to determine, do they need to take some actions and in which area they need to take actions then. And if they need to do those, then they can drill into the details—who are the patients they need to be contacting, and so on.

On the second one, I mentioned about the increasing demand for us to allow the users—the study personnel or site personnel—to drill into this data. So here I have an example about the compliance metrics. So on the left hand side, you actually see the compliance on the country level. So then by clicking the name of the country, doing one click, the user can go and see the compliance in that specific country at the site level. They can identify that in this case the site there at the top that is highlighted, the bar that is highlighted in red, that the compliance is below 50. And then they can further drill into that data so that they can see the compliance in the specific site at the patient level, and then they can also see more patient-specific data—when they have received training, and when they have sent the data last time. So by that they understand if there is maybe some issue with the training—that the site needs to do more training—and then they contact the actual patients, they have all this information available to find the solution, how they will meet the compliance requirements.


So the third key objective there that I’m interested in is focusing on the most relevant metrics. So here you can see on the lefthand side, you can see one example about the study team dashboard view, which you will see on the actual demo in a few minutes. But then you see on the right hand side, the site team dashboard view. So obviously there are very different types of needs that the users have in the study teams. They need to have an overall view about the study, across all the patients. And then they need to also monitor the different sites and the performance of the sites in different areas. And then on the right hand side on the site team dashboard view, I will be showing you some examples, like what are the key metrics for them to follow and monitor. But the main point here is, like I mentioned, that the users will see only the relevant information, their specific role in the study, so that the time that they use for the data monitoring, they can use it efficiently. And obviously they are asked, if it’s a tool, if it’s an eCOA provider who is providing this, other type of providers, they need to have a very easy way to configure these different roles for the users. So on the top left hand corner you see there just a quick screenshot, what example roles can be assigned to these studies and to the reports these users will be seeing.

And the fourth goal, and the second-last one before we go to the demo, was how we can use data visualization to combine the best practice reports with the custom reports. So we’ve been now working for quite a long time with different partners to build a robust library of best practice report templates that need to be available out of the box at the beginning of the study. So they need to be standardized so that they are relevant and valid for all of the studies. So they can be different dashboards, like I showed in the previous slide and I’m showing in the demo. And there are different standard templates for compliance, different subject or patient info standardized sheets, inconsistencies that relate to the data quality—so finding out about duplicate subjects, duplicate data entries, so on. But it’s important to have that available, but then when we go to the custom reports, there are different types of therapeutic areas, and they have different types of needs for those reports. I will be talking more about diabetes today—that’s there in the middle—and I’m showing you some examples. But on the left hand side there’s you have an example report about the pain scale in the oncology study. So there, the goal is that when there is a pain scale from 0 to 10, with this report the users can very quickly identify if the pain levels have been increasing or decreasing to the certain set threshold that requires some level of action from the study team. On the right hand side you can see an example about the vaccine-specific therapeutic area reports. It’s fairly small text, but there is basically a summary based on a one-week period about the systematic symptoms that we’re seeing on this vaccine study, from the irritability that the medicine was causing to the high level of fever and so on.


And with all of these goals is to really support the users to react to this data that we are collecting in real time. So that’s really the key benefit, for example, compared to the paper-based studies, that now when we are doing electronic studies, eCOA studies. And if we have a proper data visualization, data reporting tools, the users can react to it in real time. And within our customers, reacting in real time often means that they are following the compliance or some stopping factor items in the study of course on a daily basis, and they can contact the patient within 24 hours after they see the report, and then react to it in real time. And this is really the key benefit of eCOA when it’s combined with appropriate data visualization-based reporting.

So at this point, let’s move ahead. And this is the second part that I will do before we go and see the actual product demo and some use cases, how we are using these different data visualization technologies. So on this one, we would be interested in getting your view, which are the most important metrics that you will be monitoring in your clinical trials, whether it’s compliance, eligibility, data quality, safety, or data issue resolution. So if you can take a few seconds and just quickly highlight here what are the key metrics from your perspective.

So I’ll move to the results now. Now there is a little bit more variation here. So the two top ones are here, quite consistent with what we are seeing in our studies that we’ve been doing with our customers. So the compliance, and then the data quality and inconsistencies. And in many studies also what we are seeing, the eligibility is of course the critical part at the beginning of the study. And often we have custom reports created to the eligibility part. And then, monitoring certain safety and symptoms and stopping factors for the study that may require them temporarily to stop the study and adjust it, have also showed up here having a little bit, a smaller percentage. But this actually aligns quite well with what we are seeing when we are running these eCOA studies and the data reporting for those with our customers.

We’ll move to the demo part now, and I will be showing there, actually four different use cases and show how you can use the data visualizations to resolve these issues in a study.

The first one is analyzing the compliance across the multiple dimensions, and showing how users can explore the data with the new technologies. The second one is focused on just highlighting with the same tool, how you can focus on the most relevant metrics. The third one, focused on the quality and the DCF. And then the fourth one will be the case study, a little case study example, how to use the actionable insights on the user behavior to optimize the data quality and data quantity in the study. And there I’ll be using the diabetes example.


So I’m using the CRF Health TrialManager 6.0 trial management software for this demo. And here on the screen, as a first example, you can see the study team dashboard. And I’ll be using this to show you how you can actually analyze the compliance in different levels. But first, the purpose of this example report is to highlight the key items for the study team, what they want to monitor during the study. So you can see here, with my mouse on the left, that the column on the lefthand side is focused on summarizing the number of subjects that are participating in this study. Here is the total number, 258, and then the same information represented in a graphical format, when the different users were enrolling to this study. In the middle, you can see the overall compliance in this study, 80%. And in the graph below this number you can see the compliance by different countries in this study. Then, on the right-hand side, you can see as a quality metric the number of active DCFs at the top—so data clarification forms—and there below that you can see the graph, what are the main categories and how many DCFs there are in each of those categories. And so the focus on these compliance metrics here in the middle.

So if I move my mouse and want to look at the compliance in Germany, so just by clicking it once, you can see that when I click Germany graph, and it’s selected that country, immediately also the number of subjects, below lefthand side, and the active DCF number on the right hand side were updated to reflect what is the situation in that specific country, in Germany. So it was just by one click, I moved from the global level compliance to the country-level compliance. Then if I, here by clicking this green button, confirm the selection, then it will show the compliance in Germany in these different sites. The numbers on the left-hand side of the chart, those are the site numbers. So here we can see that site number 1458 has the lowest compliance at 74%. And also when we use this type of visualized summaries, it’s often very powerful to use simple but very insightful color schemes. So for example in this study, it was defined that the compliance was somewhere between 50% and 80%. That was in the yellow area. If it was below 50% it was highlighted in red. And if it’s above 80% it’s green. And this type of colour-based coding can be customized for each of these reports based on the needs of the specific study team.

So then we further drill into the specific sites, so now we can see after clicking the site number 1458 that there are 20 subjects. And we want to look at the DCF situation, we also see in this same dashboard after one click how many DCFs that are active, and then also we see the DCF types there on the right. And then, if we want to further drill into the patient-level compliance, what we can do, we can select different reports for that from the top drop-down menu. So if I now go to the site details report, then we can see the compliance during the last seven days in that specific site at the patient level. And here, just related to the tool that I’m using, I want to highlight that here on the top lefthand corner, you can see these selections, the filters that I have selected. So this can be also unselected from here. And then you notice that through these—in these tools, using these tools and technologies, these filters actually stay when we move from the one report to another one. So here, at the patient level then, when we look at the compliance, we can for example the study team or site personnel can see that in this specific site, there are two patients who haven’t reported any data during the last seven days. So if they want to further drill into these specific patients, they can select them, maybe select a few more, and then on the data table at the bottom they can see when these subjects were trained, when they were sending the data last time. So they can contact them proactively.


So the new technologies enable also really combining the data on the actual graphs, that are visualized and that allow the users to really quickly understand where they need to take actions, to the tables that can be linked so that they can see further details if there are some details that can only be presented in the table format or in a tabular format.

So then, this was an example about drilling from the country level to the site level and patient level, compliance. And as the next short example, I will be showing you from here like how that dashboard view could be different for the site. So like we’ve talked about how it’s important that the site personnel can focus on the most relevant metrics. So in this example, in this diabetes-specific study, the metrics that were selected to the site dashboard were the summary in the table format in this case here on the left hand side, about the subjects that are not sending data or haven’t been sending data recently. Then the compliance, that very often is one of the top metrics that are highlighted in the dashboard. And then these dashboards can be also customized based on the study. So in this case, one of the key metrics that was monitored was the hypoglycemic events and if those were received or not. And that’s then the example here on the left hand side, how these dashboards can be customized. And again, about the example, how the data tables can be combined in the individual graphs, is that when the user will select these subjects that haven’t been sending data, then they can immediately see what has been the compliance over the whole time period of the study for those patients and also what are they reporting related to hypoglycemic events. So just a quick example about focusing on the most relevant metrics.

Then I will show you quickly how these technologies can be used to monitor the data clarification forms. And this is the data clarification form summary report. And in this case, here on the right hand side, you can see the number of DCFs based on the different categories, so you can very quickly identify that the duplicate subject has been the main driver of DCFs for these studies. And then changing subject information and so on. And here below, on the bottom right hand corner, there you can see the number of DCFs per site. And on the bottom left hand corner, the DCF site performance, how many days it took for the sites to complete the DCFs. And then on the top left hand corner you can see the status of the DCFS—have they been waiting for the approval, or are they under work, are they actually being processed by CRF or the eCOA provider, or what’s the status for them.

So here, for example, the study team can very quickly identify that in these sites that have the largest number of DCFs, how quickly they are performing on resolving those DCFs. So the problem with site number 1458 seems to be in this case that it took them 30 days actually to approve the DCFs, compared to, for example another site—1841—that only took less than half of that time. So a quick demonstration how they can measure how quickly the sites are performing regarding solving their DCFs and data quality issues.


And then the last example is about diabetes. So in this example here, for the study team, it was critical that they got a detailed root cause analysis on all of the hypoglycemic events that were reported. And how it works, so that every user, they filled the first report that was measuring the glucose levels. And if it was actually above the certain threshold, from the system there was another hypo event report that was sent to the user. And that report had the questions that allowed the study team to understand what were the root causes, like the possible reasons that were causing the higher-than-expected blood glucose level. So in this one chart, they are seeing the total number of the reported low ratings of the blood glucose, and then on the red there, they have a number how many of those were unreported. Overall it was 24%. And then here at the bottom, they have the same information by countries and by the sites in each individual country. So by using this analysis, they were actually able to contact the specific sites and provide them further information if for some reason a subject in the sites were not filling out that follow-up hypoglycemic event report that was critical to comply with the study protocol and get the details that were needed. So example about the diabetes-specific report.

And another quick example—I will not drill in for the sake of the time—but another metric that has been used in several studies is the measure how timely the patients are reporting their meal events. Are they doing it retrospectively—that in this case would be fine if it was 3 hours or more after the actual event—or are they doing it on time. So in this case, for example, it was an interesting finding for the study team that actually on average there was roughly 25% of the events were reported retrospectively. And here they saw the same information by countries, and then they saw also how many retrospective entries were done one day after two days after three days after. And as it was for this study, it was important that as often the paper studies it happens that the patients actually fill the questionnaires just before their site visit in a car in a parking lot. From a data quality perspective it was determined in this study that they needed to actually narrow down the amount of time that was allowed for the user to fill the questionnaire, and it was determined that they actually need to narrow it down to just a 12-hour period so that they get accurate enough data from the users that are reporting related to this meal event.

So this was the last example, and this about gaining actionable insights on user behavior on this diabetes case. And actually, I will now pass it back to Naor.



Yeah, so now is the time for you to let us know if you have any questions. Feel free to use the chat window to ask any questions of Miko regarding what you’ve seen or the software, and we’ll have some time for Q&A.

[Q&A section begins at 44:49]

About the Author

Heather Bilinski

Associate Director of Marketing at Signant Health

More Content by Heather Bilinski
Previous Video
How to Calculate the True Cost of Paper When Considering eCOA
How to Calculate the True Cost of Paper When Considering eCOA

Some clinical trial sponsors fear the cost of eCOA will be prohibitive. We unpack the expense of paper data...

Next Video
SCDM Presents: Managing eCOA Data - Principles and Best Practices
SCDM Presents: Managing eCOA Data - Principles and Best Practices

At the invitation of SCDM, CRF Health reveals common issues related to data management. Presenters draw on ...