Watch on Demand

First Name
Last Name
Country (Citizenship)
Opt-In to Future Emails
Thank you!
Error - something went wrong!

eCOA Essentials: 1 Hour to Higher Quality Data

April 22, 2016 Heather Bilinski

The industry's shift from paper to electronic Clinical Outcome Assessments (eCOA) in clinical trials is happening at dizzying speeds.

Trials that eliminate paper in favor of eCOA regularly benefit from improved compliance, higher data quality, and trial results that are available more quickly than with paper.

Isn't now the right time to increase patient compliance and data quality in your trials? Learn the essentials of eCOA in 1 hour or less.

Full Transcript



Hello, and good morning and good afternoon to everybody that’s joining us today. My name is Naor, I’m going to be your moderator for today’s webinar session. And we’re very excited that you’re taking some time out of your day to join us. So today’s webinar, believe it or not, is our most popular webinar we’ve done in the last year. It’s about eCOA essentials, and our goal is to introduce you to the technology of eCOA in one hour or less, hopefully. So very introductory, very high-level. We’ve had a lot of requests from clients who wanted to teach some of their newer team members about eCOA.

And today’s webinar speaker is Michael Wolgast, who is from CRF Health. He is our Director of Client Relations. He’s based in San Diego, California, so a lot of his clients are on the west coast. And he’s responsible for relationship management for some of our key accounts. And he’s been with us for just about 11 years and in the pharmaceutical industry for about 20. So I’m going to let Michael take you through, and we’ll catch up at the Q&A at the end of the webinar. Go ahead Michael.


Thank you very much Naor. I’d like to tell everybody good morning, good afternoon. We’ll start going through this.  Let’s take a look at the agenda. So what we’re going to do today is we’re going to look at some of the challenges of paper-based studies. We’re going to introduce eCOA and where it fits into the data stream. We’re going to look at drivers for change and regulators, and their demand for accurate data. We’ll look at benefits of eCOA for different stakeholders. And we’ll look at a cost comparison of paper-based versus eCOA from a sample study that we did. And then we’ll look at the values and benefits of eCOA.

So some of the challenges of paper-based studies. Even though every clinical trial strives to capture high-quality patient data, there are some challenges that paper faces. Looking at low patient compliance, or misrepresented patient compliance. The data quality is not always great. We get delayed locks to the database as we’re chasing the paper, trying to get it in-house so that we can get it into the databasedatabase, so that we can then lock that database. And then we’ve got a whole issue with the lack of data integrity. If we look at the right side of the slide, we’re going to look at some examples, answers that are illegible. Whether we’re doing data entry in-house, at the sponsor site, or that data is being one at the investigator site, if the writing is not legible, whoever is doing that data entry is going to do a certain amount of interpretative guessing. And that’s never good. At times with paper you end up with just partially completed answers or parts of the questionnaire that are completed. And that is equally bad. We get incorrect filling of sections. Patients may read the directions but not understand them or just not read the directions. And so they’ll take a single-answer question and provide multiple answers to it, or take a multiple-answer question and provide a single answer to it. It’s just a matter of making sure that they’re answering questions correctly. And then the last example that we’ve got here is completing questionnaires at a time other than the one specified by the protocol. So if they’re only supposed to be completing their questionnaire in the evening, but they decide it’s more convenient for them to fill it out in the morning, or as stated here, they’re not filling it out at all until they get to the investigator site, and they fill it out while they’re sitting in their car in the parking lot. I’ve had the experience of looking out in the waiting room and seeing patients furiously filling out questionnaires while they sit waiting for their appointment.


So the challenges to the site staff by paper is  that the data itself is passive. It must sit there on the paper until it’s entered into the system either at the site itself—possibly with the EDC system—or sent to the sponsor site and is entered by data entry operators at the sponsor site. And that’s not good for anybody at that point in time. That data is useless, it doesn’t serve any purpose other than to have potential of doing wrong. The interpretation of that data by data management may be difficult due to handwritten observations, again the illegibility of it. And sometimes data is missing or incomplete. So chances are they lost the last page of the questionnaire or they just skipped a page either by accident or because they didn’t want to answer those questions or they were in a hurry and just didn’t feel like they needed to. So paper allows a whole lot of freedom for the patients, but it also makes for poor data quality.

So we ask ourselves what is COA, what is eCOA? And what we’re looking at here is an FDA definition. “Clinical Outcome Assessment measures a patient’s systems, overall mental state, or the effects of a disease or condition on how the patient functions.” When we apply “electronic” to that, what we’re doing is, we’re taking that same definition, but we’re employing technology like smartphones, tablets, personal computers to allow the patients, allow the clinicians or caregivers for that patient to directly report outcomes for that patient. It allows for highly accurate data, giving a better understanding of what the patient experiences. And ultimately, this simplifies the path to approval. So employing an electronic solution to the clinical outcome assessments allows for a smoother move of data or a study from enrolling its first patient to getting approval for that drug.

We’ve got three types of clinical outcome assessments or COAs. We’ve got patient reported outcomes, which is what we hear about most. And this reports the status of the patient’s health condition. And that comes directly from the patient. We’ve got clinician-reported outcomes or ClinROs, and these are where healthcare professionals observe the patient and record the data.  ClinROs are employed actually quite often, not necessarily as much as patient reported outcomes, but more and more we're seeing these clinician-reported outcomes being employed using primarily tablets at the clinical site. And then the last one that we’ve got here is the observer reported outcomes, or ObsROs, which is where data is reported by an observer who may not necessarily be a healthcare professional. This could be a patient’s parents, a caregiver, a spouse. Oftentimes, if there are studies for infants, the patient’s parents, the actual parent of the subject that’s participating in the study will complete questionnaires for the infant. Also with patients that may be experiencing some impairment, such as Alzheimer’s patients, will employ an ObsRO solution to get the data that’s needed.


So where does eCOA fit in the clinical trial data stream? eCOA data goes along with investigator collected diagnostics data, such as lab data, any type of diagnostic data that may be collected by the investigator. It’s supplemental at times to these more tangible data sets. Sometimes eCOA questionnaires are more subjective, which we’re asking how the patient is feeling. More quality of life questionnaires. And because of this, they can be very useful when it comes to comparative efficacy. So for example, two drugs may do something similar in terms of physical benefit. But if one of them clearly shows the patient feels better, or their quality of life is improved by using the drug, this can potentially be used for labelling claims. So being able to get good clean data on quality of life is certainly a benefit over doing a paper questionnaire using the same questionnaire.

So influencers within the industry and patients are really driving the change to electronic. Regulatory authorities like the US Food and Drug Administration and the European Medicines Agency, or EMA, are looking for data that is really more attributable and much more reliable than what they’re finding on paper. Studies of comparisons between electronic data and paper data have shown a huge spread in compliance with patients, where the electronic data is 80% or higher compliance and the paper data can be as low as 18% compliance, even if it’s being reported by the patient that they’re being 80% compliant. So the FDA and EMA are really interested in having data that is not only completed by the patient,  but completed by the patient when they’re supposed to be completing it. As we mentioned earlier, not in the parking lot, not in the waiting room, not at the beginning of the week for all the entries that need to be made for that week.

Industry associations such as ISPOR, ISOQOL, DIA, the ePRO Consortium, are all interested in making sure that the data is accurate, is easy to use, and speeds up the drug approval process. And just trying to simplify the process for everybody involved, from the patient all the way up to the regulators. And moving to an electronic format definitely helps.

PRO experts are really interested in getting the accurate voice of the patient.  So if they’re developing questionnaires that ask about the various issues that patients are encountering or are looking for more objective responses to questions, if those questionnaires are not being administered the way that they should be, then the PRO experts aren’t getting the information that they’re really expecting to get. So it’s very important to them that they get an accurate voice of the patient coming through on their questionnaires.

And then finally, clinical trial teams are looking for efficiency within a clinical trial. They’re looking for cost savings. They’re looking for ease of use. And when you start reducing data entry costs, shipping costs, and start reducing time from the time data is created on a questionnaire to the time it’s available to the team, that always makes life certainly easier and more efficient. And also, if there’s more data you can monitor remotely, you can actually see a benefit in your monitoring times. So your incidence of having to send monitors out to a site can be reduced by using an electronic system that is able to capture data remotely.


So as I indicated earlier, regulators demand accurate data. The FDA refers to this as the ALCOA principle. That’s data that is Attributable, Legible, Contemporaneous, Original, and Accurate. And we’ll go through the definitions of these.

So as far as attributable, it’s a question of can the data be linked to its source. It should be attributable to the subject who observed and recorded the data, as well as traceable to the source of the data itself. It’s also true of the clinical observers that may be recording data or any third-party observer that’s completing an ObsRO, for example. So in creating the systems, what we create is unique PIN codes for the patients, and we give them the opportunity to change that PIN code to a PIN code of their choosing. So it’s only known to them. The system itself does date and time stamps, so any time data is entered, you know precisely when that data was entered. And we also have an audit trail on everything that is entered into the system, as well as anything that may be modified through a data correction. So everything is time stamped, everything is fully audit trailed, and access is limited by the use of PIN codes and patient identifiers, or identifiers for whoever may be accessing the system, who has authorization to access that system.

Legible, is the data easy to read. We’re not dealing with tiny writing. We’re not dealing with writing that has been crossed out, written over, or is written by somebody who has very poor penmanship. The entries are tight or they are standard response options. So if we’re allowing free text, we do have searchable keywords that are available that pop up on the screen, very similar to text messaging that allow subjects or clinical observers to fill in free text entries. But for the most part we’re dealing with questions that have standardized choices or code lists, and the subject or observer will select one of those items. And so it takes out any of the guesswork that may go into interpreting legibility.

Contemporaneous, has the data been recorded at the time the observation was made? We can program into the system reminders and alarms that prompt the subject to enter data at the right time. But the other thing that we can do is we can program in windows to allow the subject to only answer a question during a certain time. And if they try to answer that question outside of that window, they’ll be prompted to return to the system at the correct time to make that entry. We can make those windows as hard or as soft as are required by that particular study team or for that particular protocol, where a subject logs in in the morning for an evening questionnaire and it gives them a message that tells them, this questionnaire should be filled out in the evening, do you want to continue. And if they say yes, then it allows them in. We can also program a message in that says, this questionnaire must only be filled out between the hours of 6:00 and 10:00, please log back in at that time. And when the subject taps on the tag, it will shut the device down and they’ll be prompted with an alarm sometime between 6:00 and 10:00 to return to the questionnaire and complete it at that time.


So we can also control or eliminate retrospective data entry. So with paper, what you’re going to see is that patients are filling out their questionnaire in the week, they get to Tuesday, and their child had a baseball game. They usually fill out their questionnaire during the hours of 4:00 and 6:00. And the baseball game happened to fall between those hours, they left their questionnaire at home, by the time they got home they needed to complete dinner and completely forgot that they were supposed to fill out a questionnaire for that time. On paper they can go into the questionnaire and fill it out as if it was Tuesday afternoon at 4:00 in the afternoon, just like they did on Monday or Sunday or the Saturday previous. And you would never know the difference. With electronic, we can actually eliminate that possibility that they’re going to fill out a questionnaire on a day that is not represented by when they're actually filling it out. Or we can limit that. And what has been found is that recall periods definitely start to be reduced or disintegrate beyond 24 hours. People’s true recollection of what happens becomes worse and worse beyond that 24-hour window. So we can actually allow retrospective data entry as far back as you’d like. But you can try to limit it to that 24-hour recall period, maybe go out as far as 48 hours. But you can control or eliminate that retrospective data entry possibility much more cleanly and effectively than you would on paper.

Original, is the data the first and most accurate and reliable recording of the data? So PRO data is captured directly from the subject in electronic form. This eliminates the need for source data verification, you don’t have to look from the questionnaire that the patient completed on paper and compare that to what was entered into the EDC system. Or that paper questionnaire does not need to be collected and shipped back to the place that data entry is going to be done. The devices that we capture PRO data on, whether it’s a handheld device or a tablet device, are considered a transient source. What happens with that data is it’s captured on the device itself, and that electronic data stream is actually sent to our servers, and it’s stored on our servers in the database. At that point in time, when that transfer happens, the data is no longer available on the device itself, on the hand-held device or the tablet, and is only available in the database. And at that point in time, the data in the database is the source. So it eliminates some of the intermediate stages of the data, reduces the amount of paper that needs to be retained for the standard retention period, cuts down on storage, cuts down on having to ship that paper to the data entry location or to a storage location. And what you end up with is the original data as recorded by the observer or by the patient.


And lastly, accurate, does the final report accurately describe the study conduct and the reported results? So we can build in logic and provide real-time edit checks to prevent incomplete or inconsistent data entry. We can require that patients answer every single question, or we can allow them to skip certain questions. Oftentimes, we’re asked to allow patients to skip questions that are in relation to sexual function or to money, things that make patients a little uncomfortable answering. And if it’s not a critical question to study, those are things that we can program them to allow the subject to skip. But if you have a questionnaire that has a fairly complicated path, such as if you answer yes to question three, please skip to question seven, oftentimes you’re going to get answers to questions four, five, and six. And what do you do with that data? Do you record it? Do you skip it? How do you handle that data? And that’s an answer that you need to provide, usually up front, before you start collecting data, how are you going to deal with outliers. We can build in logic that will automatically skip questions four, five, and six if they answer yes to question three. So it will automatically navigate for the patient in that situation.

You can also make sure that patients only see the questionnaires that they need to see. So if there is a particular questionnaires that need to be seen at a particular visit, we will only present those questions, and they won’t have to decide which questions to answer and which questions not to answer, depending on the circumstances of the visit.  Also,  the sites will not need to make that decision when a subject comes in for a visit, to decide which questionnaires to hand a patient to complete and which ones not. It’s programmed in ahead of time time, so that the only questionnaires that the patient sees are the questionnaires that they need to respond to.

So we’ll look at stakeholder benefits. For investigative sites, the devices themselves guide the site staff to follow visit procedures according to protocol. Very similar to what I was just talking about. If visit two only requires two of the questionnaires and visit three requires a whole battery of eight questionnaires that are required by the protocol, as long as the site selects the proper visit, the proper questionnaires will be presented to the patient. The site doesn’t have to make any decisions on which ones to provide the patient and which ones not to provide the patient. And the patient doesn’t need to make any decisions on what questions they need to answer and what questions they don’t. The system will do all of the navigating for them and provide them with only the questionnaires that they need to answer.

The system also can be programmed to automate scoring for any scales that are done. So if you have an item such as a Mayo score of an ulcerative colitis study for example, that Mayo score can be calculated within the system to allow patients to answer the questions for the clinical observer to answer their assessment as well as any other outside assessment such as lab values or, in this case, possibly a colonoscopy score. And that overall score for that patient will be calculated or can be calculated within the system. So it takes some of the pressure off of the sites to complete that scoring properly and accurately and reduces the amount of monitoring that needs to be done, either possibly at the time of randomization or moving a patient through different phases of the study.

We can also automate visit scheduling. So we’ve had instances where patients had scheduled visits that they needed to attend. And I’m sure you’ve had this experience where a patient missed their visit because they had no reminder of it. And what we’ve developed is an ability to schedule appointments within our system and provide reminders to patients, either on the handheld device that they’re using, that has been distributed as part of the study, or through text messages or emails to them that are generated through our system that allow patients to know that a visit has been scheduled at a particular time and particular date, or that a visit has been updated or changed to a particular time or particular date. And this allows for greater compliance to protocol, greater compliance to attending site visits, and therefore better compliance to completing questionnaires.


And then the last thing is clinician-reported outcomes being moved to an electronic format, minimize the volume of monitored documents. So there’s less possibility that the site will lose questionnaires, that the questionnaire will not be completed when it should be completed, and there’s less burden on the site to maintain that paper and maintain that schedule.

As far as patients, patients are more engaged, which increases compliance and retention on the study. When subjects are given a device that interacts with them, that records their answers, but not only that, provides them with reminders that their questionnaire is due, or they have some other action that they need to perform in regard to a study, they’re more engaged with the study overall.

It also allows for data to be available to the sites in near-real time. So if a patient answers a questionnaire at 3:00 in the afternoon, let’s say they complete it at 3:05 in the afternoon and they send data immediately. That data, as soon as it hits our servers, will be available to the site to view. So at approximately 3:05 and 30 seconds, after that data has been sent, the site will be able to review that data. And if there’s anything critical in there—for example, if you’ve got a suicidality scale in there and the patient is exhibiting behaviours that are critical, alerts can be programmed into the system to notify the site that this patient has answered in a certain way. And the site at that point in time can contact the patient, can work with the patient and do what needs to be done under those circumstances.

Patient privacy and discretion with fewer eyes on the source documents. This can be very important to certain patients, and certainly with certain studies. If you’re doing a sexual dysfunction study for example, patients can be a little bit unclear on their responses. And if they know that there’s not an army of people that are looking at this data, from the site all the way down to the person that enters the data and then analyzes the data, the chances of getting more accurate data and more honest data is increased significantly. So it’s a comfort to the patient to know that the information that they’re providing is confidential.

And then finally what we have here is that the method of collecting this can be adapted to fit special needs of various patient populations. Patients that have vision difficulties such as older patients that may not be able to read small print, we can increase the size of the print on the device itself to accommodate these patients, or reduce the number of buttons on a screen so that we can increase the size of those. We can do the same thing for dexterity. Parkinson’s patients, for example, who have a certain tremor, may not have the dexterity to tap on a particular small button, and we can increase the size of those. So it allows us to provide design considerations for patients that may have special needs.


And then, monitors. Monitors can reduce their site monitoring load. Reviewing data remotely through a web portal of data that has been submitted so far allows them to look at this data remotely, allows them to look from their office or possibly even a home office. They can look pretty much anywhere where they’ve got an internet connection. So knowing that monitors travel a lot, they can actually review data from anywhere globally. And it allows them to reduce the number of site visits that they would ordinarily do to monitor this data. It allows them to do a lot of this monitoring remotely, allows them to suggest changes, or—best yet—it allows them to identify potential issues that individual sites may be having, either with compliance to the system, or just how they work with their patients and how they set patients up. And so instead of waiting six weeks or longer for the next monitoring visit to identify that a site is completing the wrong questionnaire or is not completing questionnaires at all, the monitors can identify these problems much earlier than they would if there was a paper system and allow for a correction to that to happen almost immediately.

And then the final point that we’ve got here is reduced paper usage. I know from my days working at the site, we were using three-part patient report forms, which may actually be dating me on this, but I know that there is a significant amount of paper that was used to collect patient information, either CRF data, or even questionnaires. And so if we can reduce that paper usage, it’s better for all of us. But the other thing is that monitors won’t be either having to tote that paper around, or be shipping that paper to another location, which actually increases the risk of lost data.

And then, our final group here is data management and statistics. The data that they receive in data management and statistics is going to be cleaner. It’s going to lead to faster database locks. Items such as visual analog scales that you see measured by hand, no longer need to be measured by hand. What happens on devices is that instead of counting by inches, you count by pixels, which gives us more accurate data, it gives us more consistent data, and allows either data managers or site coordinators or anybody who is interpreting that score on a visual analog scale the time to do other things than measure visual analog scales.

Electronic DCFs. DCFs are data correction forms. And this allows items such as demographic data and data that is entered by the site to be updated through the system. And instead of sending paper queries around, everything is done electronically, very similar to what’s done using EDC. And so it allows for a faster turnaround on corrections, and allows for a more efficient system for everybody involved.

And then finally, as I was indicating earlier, data is available in the database in near-real time. So just like the monitors are able to monitor the data remotely and be able to identify issues at the site, data management can do the same thing and identify issues that may be happening at the site or trends amongst the data. With some systems, they can actually download an export of the whole database and review that data on their own system remotely. It allows them to possibly identify trends—either good trends or bad trends—within the data to allow corrections to how that data is collected. So if we’re finding that patients are skipping a key question because we’ve allowed them to skip questions, if they’re skipping one of the primary efficacy values, then we may need to make adjustments to the system at that time and require that question to be answered. So allowing data management to review these things is definitely important, and being able to see it in real time actually speeds up that process.


So looking at the return on investment, we put together a test study. It included 20 sites, 180 patients that were screened, it is in five countries, six languages, it was a 17-month study with two diary entries per day. And what we found is something that was somewhat remarkable, which is actually the difference in cost between an electronic system and a paper system. And if you look at the diagram to the right, the top portion of that, you can see that most of our costs are incurred on an electronic system up front. It’s in programming the system, creating the screens, programming the edit tracks, making sure that the system works appropriately, and loading the database. Whereas with paper, you’ve created a paper document, you may need to have it printed, and build the specific facilities for the database to store that data. But generally the up front cost on paper is significantly less. But where the difference comes in is in the maintenance period, because with the electronic data, it to a certain extent manages itself. The data is collected by the patient. If you’ve got any questions on how that patient is collecting that data, you can address that very early, because you see the results very early, as opposed to a paper system where the data comes in in dribs and drabs, particularly early on, and if there are any issues, you may not identify that for weeks or months after those issues have been established within the system. And you reduce the amount of monitoring of that data that happens, so you don’t have monitors going out to the site to monitor and collect that data, it’s already been placed in the system, so it’s already been collected. And the monitors can review that data remotely and make any adjustments that possibly need to be made, or instruct the site.

So the other thing that this allows is scalability. It allows a system to be built that accommodates 80 patients. Or that same system can accommodate 8000 patients. It doesn’t matter. The only thing that would change in that case is possibly the amount of hardware that you need. But the other thing that it allows is the re-use. So hardware that’s purchased for one study could be reused on another study. It’s generally durable and can be reused on another study simply with just reprogramming, taking off the old protocol that we had installed on it and installing new protocol. So it’s a fairly simple, efficient process.

And then the final thing that we see here, and this is illustrated in the bottom part of the image to the right. And that is the cost of paper, in this example, comes to three times greater than what you’re finding with electronic. Again, most of the cost for electronic is in setup. But when it comes down to the final cost, you’re getting a much cheaper cost for maintenance, and you’re also getting a reduced cost for database lock, as most of the data is already present and you’re not chasing paper. And you’re not working at the last minute to clean data.


So as we start to wind down here, we’re looking at how eCOA delivers value to patients, sites and study teams. As we’ve mentioned earlier, eCOA improves the quality of the data. It makes the data attributable to the patients, attributable to the time point that it was recorded in. It optimizes the management of the clinical trial itself, so it allows you to manage visits, it allows you to manage how the data is being collected. If there’s any issues with data collection, those can be identified very early on, and the system adjusting to accommodate that.

The regulatory agencies are really pushing hard for data to be collected electronically. Whether it’s eCOA data or CRF data, they’re really pushing hard for a move to electronic versions of things to reduce the amount of paper, to reduce the amount of inefficiencies that are found in the system, and allow for a quicker move from drug discovery to approval into use. And so it benefits the regulatory agencies, it benefits the companies that are developing these compounds, and it benefits the patients who will get use from these, and possibly get relief from their illness.

It reduces the burden on patients, on sites, on study teams. Patients won’t be required to carry around paper diaries, they won’t be required to remember when they need to fill it out, the system will aid them with it. The sites will be able to monitor the patients remotely much more closely for compliance or possibly urgent situations that may require their attention.

And then finally, study teams. Study teams will be able to monitor data and monitor sites remotely to make sure that they’re following the protocol and completing things the way that they should be completing things.

And then finally, higher quality outcomes really equate to lower overall study costs. So eCOA definitely brings these benefits of higher quality outcomes to allow study teams to focus on other items within the study such as patient enrolment and the things that are important that they need to make sure that they stay on their timeline and make sure that they stay within their protocol, which overall is going to reduce the amount of study costs.


All right, thanks for that. As Michael is sorting through the quite huge variety of questions that we’re receiving, just want to remind you that we do have an extensive resource portal on our website, and that is resources.crfhealth.com. These are just some of the few things that you can get on there. We have something like over 60 pre-recorded webinars. So feel free to dig in, it’s all available free and instantly.

[Q&A section begins at 48:29]

About the Author

Heather Bilinski

Associate Director of Marketing at Signant Health

More Content by Heather Bilinski
Previous Video
Innovations in e-Consent: Reducing Regulatory Risk While Improving Participant Comprehension
Innovations in e-Consent: Reducing Regulatory Risk While Improving Participant Comprehension

As regulatory agencies push for more effective informed consent processes in clinical trials, eConsent can ...

Next Video
Winning Strategies for Successful eCOA Implementation
Winning Strategies for Successful eCOA Implementation

Drawing on 20+ years of experience in clinical research and ePRO/eCOA implementation, Scottie Kern provides...