Watch on Demand

First Name
Last Name
Country (Citizenship)
Opt-In to Future Emails
Thank you!
Error - something went wrong!

How to Calculate the True Cost of Paper When Considering eCOA

September 16, 2016

Clinical trials using electronic Clinical Outcome Assessments (eCOA) benefit from compliance rates 300% higher than paper-based studies - yet some sponsors hesitate to adopt eCOA due to a misconception that paper diaries cost less.  

We compare the often hidden costs of paper data collection vs. eCOA to help you decide which is best for your next trial.

Full Transcript


Welcome. Thanks for joining us at CRF Health. Today’s webinar is all about calculating the true cost of paper when you're considering eCOA, and I would also say when you’re trying to justify your investment in eCOA. It’s a question that we get asked quite a lot, surprisingly. And there’s some misinformation in the marketplace, so we thought we would help you set the record straight by giving you some of the calculations that we typically use when we consult with a client.

Joining me today is Jery Grupp, who is one our senior-most directors at CRF Health. Jery is someone who is very very good at presenting this kind of material because he’s been in the industry for over 25 years, and he’s also been in positions on the sponsor side where he’s been responsible directly for technology implementation. And of course, in doing that work, a lot has been relating to cost and comparing potential ROI and evaluating risk. So welcome, Jery, and thanks for joining us.

And with that, I’m going to pass it along to Jery to get started with the agenda. And you will have time for questions, so as they come up please feel free to enter them into the chat box, and we’ll make sure that we have time at the end to answer your questions. Jery.


Terrific. Thanks, Naor. Good morning everyone. I’m very pleased to be speaking about clinical outcomes assessments today and explore the cost differentiation when comparing eCOA to paper.

We got a lot to do on our agenda today, we have a number of items. We’ll talk about the end of the clinical paper chain and the rise of eCOA, what the barriers to eCOA adoption may be or may be perceived to be. We’ll walk through a hypothetical about a mid-sized global trial and talk about the cost drivers there. Then we’ll compare the results of that analysis with what a similar eCOA implementation would be. We’ll talk about some additional benefits for eCOA that may not be seen or demonstrated specifically in dollar value. We’ll touch on regulatory risks and what the impact on time-to-market, and finally we’ll wrap up with a long-term ROI using eCOA.

So there is a growing demand for clinical outcomes assessments today. These are being driven largely by, not only industry associations and sponsors’ leagues, but the clinical teams and regulatory authorities are also looking for additional COA, clinical outcomes assessments, input. There’s also a strong drive around clinical outcomes assessments for expanded opportunities with regard to reimbursement by documenting the quality of outcomes and superiority of products, and in general the user perceptions and subject perceptions around your products. There was a time obviously when trials were driven almost entirely by objective measures. Is your tumor progression in remission, changes in lab values. While those days aren’t gone, there is certainly a new direction around clinical outcomes assessments and the perception of caregivers, the patients, and the clinician. Historically these were done on paper. The paper comes at a high cost, and it’s not just dollars, but it’s also the energy and the workload for both sponsors and sites, and we should add, for the subjects. And we’ll talk about those costs on upcoming slides.

Now I’m going to talk about eCOA, I’m going to use that term broadly today. And in our conversation, we’re going to use that as an umbrella term to cover a number of different methodologies. I’ve listed them here. You’ll see patient reported outcomes, often commonly called diaries. And diaries, which are patients reporting their experience or answering questions. It also includes observer reported outcomes, clinician-reported outcomes. But as I say, today we’re going to group all of those under the broad term eCOA as we progress our study.

So we talked early on in our slides that there is increased demand. But is eCOA adoption also rising. And it should be, and it is. But one of the challenges that we’re encountering is people looking at paper. So let’s talk about what happens with paper. One of the challenges with paper is historically a low compliance. Compliance rates of around 30% are common as reported by both research and journals. But that 30% in and of itself is still suspect. Is the data collected contemporaneously, and is it accurate? An objective study was done with a control group using a paper diary comparing to eCOA. And the paper diary reveals that only 11% of the entries were completed on time and contemporaneously. We have seen the use of eCOA drive compliance to as high as 90% or even higher. And with the system recording the time of entries, you can document that the entries weren’t made while the subject was sitting in the parking lot preparing for their visit.


All the entries made on paper eventually need to be transcribed into a system somewhere. At one time this was done by double data entry at the sponsor, but the time delays between entry of the data and collection of the paper at the sites, and ultimately the entry into the system at the sponsor, those delays were dramatic. Following that, there was a push that the sites would be doing the data entry from the paper. But that’s no longer doing double data entry, and so now have the site personnel doing low-value-added tasks instead of the more critical tasks around monitoring and patient safety are a problem. The transfer of responsibility certainly has a negative impact on the data quality, the data reliability and integrity, and the traceability, to say nothing of the perception at the sites of taking on yet another task. So this leaves us with a question. Forty percent of the trials with a COA component are using eCOA. Why aren’t more?

So let’s look at why more, and look at the barriers to eCOA adoption. There are a couple of reasons and we’ll bring some of them to the front here.

Some people look at and see the up-front costs as a huge barrier. Some people quite frankly just aren’t familiar with eCOA and don’t know to ask. Others know about eCOA but they have a fear of unknown and perceived risk. They’re not sure what it takes, and they’re afraid to enter into it. And some are scared off by the perception of additional planning required. I’m often thinking back to the old days when we were children and The Tortoise and the Hare. As children we all thought the hare dashing off in the race with no planning or forethought would win, but we were wrong. Similarly here, a well-planned eCOA, while it may seem to have a higher burden—much like the tortoise planning his race—ultimately will prevail. And we’ll show you that today. The perception of a higher start-up cost translates often into the thought that eCOA will be more expensive in the long run. In truth, the use of eCOA requires the sponsor shift a larger portion of the data collection budget forward in the study, to the start of the study. But we believe, and we’ll show you today, that that up-front investment leads to improved timeliness and reduced cost.

So we come to the big question. And if I haven’t already hinted at the answer then I’m not doing my job. The big question is, is paper cheaper than eCOA. So to do that, to make that analysis, we’re going to run through a hypothetical mid-sized trial. We’ll walk through the costs and cost drivers that relate to this, and I’m going to pull out some salient points for the trial that we’re using here, so that we all move forward on the same basis. We created this trial, which draws upon real world examples, and the names were changed to protect the innocent. In our hypothetical trial here, we’re going to be conducting the trial in five countries. That will involve six languages. There will be 180 subjects who will be screened, and 132 of those will move on to enrolment. Each of those subjects will be completing two diaries each day for the 182 days of the trial. And there are 20 sites involved in the trial. When you look at the numbers here, you’ll see that with 132 subjects for 182 days and two diary entries a day, you’ll be generating about 50,000 diaries.

Oftentimes, people automatically assume that the paper diaries are free and don’t fully account for the cost budget of the paper. There are hidden costs in the overall budget that spread across several functional areas that are not clearly delineated from other costs. So let’s drill now into what it takes to run a trial with paper for those outcomes assessment collection. We’ve already built our hypothetic model we described before. And let’s walk through many of the typical components of a paper trial and how much they cost.


But first let’s take a moment to explore just more of the physical realities we talked about here, the scope of the data we’re dealing with. As I said before, we’ve got 132 subjects, 182 days, two pages per day, comes out to exactly 48,048 diary entries. Let’s call it 50,000 because it makes the math easy. What you see on the screen is ten cases of paper. That’s how much paper it’s going to take to conduct this trial. And remember, once used by the subject, each of the 50,000 pages that are turned in is source document, and it has to be handled, entered, tracked, stored, archived from then on. It’s a permanent record of this trial. It’s not unusual to find that that the number of entries recorded over time for a diary exceed the number of entries for the same subject’s CRO. So let’s go into what it takes to run a paper COA. We’ve already built our hypothetical, and now we’ll walk through some of the typical components of a paper diary and how much they cost.

First, out of the box, we look at the $10,000 expense, which comes from paper diary instrument site review. While there are many, we’re only going to focus on nine here, and this is the first of them. Reviewing paper diaries and site instruments raises huge time and therefore cost considerations and challenges for study coordinators. The cost of collating and cross checking huge volumes of data for each subject for the trial, not to mention decipher handwritten entries, obtaining missing information, and checking that all the required data elements are correct is a significant investment of time. We use the cost model here that predicts that the review for paper diaries to be just ten seconds per page. I honestly believe it would be longer, but for our purposes today we’ll speak of it as 10 seconds per page for the site to review as the data comes in. So again, looking at that 50,000 pages we referred to before, 49,000 pages, you would accumulate cost of approximately $10,000 for this element of the clinical trial alone. Spread across all the sites, when you add it up, that actually, three and a half weeks of effort piecemealed out across the 20 sites a few moments at a time, but it’s three and a half weeks of effort, manpower. And remember, as they collect the paper diary and review it, we spoke here about looking to make sure the essential data elements are there. Presumably, if the site is looking at it and they find something is not present, that adds additional time further to that.

Next, we’ll look at the use of a visual analogue scale. I hope everyone is familiar with them, there’s a sample there in the lower right-hand corner. The visual analogue scales on paper are at the mercy of the correct reading. The inaccuracies are likely to occur with paper logging will cause real challenges further down the line, and means of rectifying these issues can be costly. Interpretation is very subjective. Is the assessor using a proper ruler to measure it, and what about checkboxes or circling. We’ve had instances where, in reviewing these documents on paper, someone circled something, and in review on inspection, the question has come, where should the assessor be evaluating it—to the highest part of that circle, to the bottom part of the circle, to the centre, and how they determine the centre of the circle. These are challenges that are best avoided. If done in eCOA, the system automatically objectively interprets the values in real time rather than having a third party evaluate the values. And look at it only being $1000, but there’s the exposure that comes beyond that.


Diary data transcription. I spoke to this before as well, that collecting the paper diary, the diary is just the collection. It has to be entered into a database somewhere to become evaluable and assessable. Transcription errors are one of the biggest challenges in a paper-based diary. These kinds of inaccuracies not only put regulatory submissions at risk, but the force the study team, data management site personnel, to spend hours reviewing and keying in information and re-entering data and seeking follow-up for information. It is interesting that in the paper diary there is no access to the data until the subject attends the next site visit and brings with them the diary. That’s a significant effort that requires both—that creates a significant burden because you’re in the blind for a long period of time. If you have a subject who’s on a two-week visit, you don’t have any feedback that they’ve entered the data or not until you get it later. The study coordinator at the site, by the way, is now doing the data entry. If we use the number $52 an hour—your experience may be higher or lower—you’re winding up doing this at the site as a single data entry, which means the data has not been verified by a second operator. Looking at the number of pages that each patient received and the dollar value that we expected here for the site personnel, you’re spending about $3.50-4.00 per page to do data entry as a single data entry at the site. And again, as I said before, you’re taking time away from the site coordinator, the study nurse, to do this transcription instead of the more critical and essential items of patient safety and monitoring. 

Looking at source document verification. Because it’s generating data, we go back and forth sometimes calling it source data verification. But having started in the old school with paper I still call it source document verification. So what do we mean by source document verification in a paper diary solution? This means that someone has to go and compare the information received from the paper document completed by the respondent to the data that’s entered into the database. To ensure the validity and integrity of the clinical trials to 100% SDV of paper it is costly, time consuming, and again diverts attention and resources from more critical clinical trial activities. The raising of queries related to SDV adds further time challenges to the study team. Remembering that that source document, that SDV, is coming at some point after the data was entered, and the data was entered at some point after it was collected. So in an eSource, in an eCOA solution, this expense is avoided in large measure because the eCOA is the eSource, there is no need to go back and compare it with documents. It was completed by the subject, and that creates a direct data entry, there is no re-transcription. So that’s a $115,000 expense that can be avoided.

Additional monitoring visits, we’ve put a number here of $26,000. Your experience may vary, you may have fewer, you may have much greater. The greater incidence of errors and inaccuracies using paper means that additional monitoring visits often will have to occur compared to those that may have already been planned for the duration of the trial. So it’s an expense, it may or may not occur, it may be higher. We can’t tie that as directly quantifiable to the number and the number of incidences, number of points collected. It goes to the quality of the data collected, the responsiveness of the subjects, the timeliness. So we put this up as a placeholder of $26,000.

In our model, we talked about a screening period and we anticipated that 48 of the 180 subjects would not go on to enrollment, based upon the data collected during the screening diary period. Information gathered during those 14 days of screening must be assessed for all subjects prior to randomization. And so the analysis of paper diaries and questionnaires against inclusion criteria is not only difficult due to the fact that the teams are dealing with a large amount of paper, but there may be incomplete misleading information as well as falsified data entries, as we spoke of before, because the entries were entered not contemporaneously and after the fact. And all of this adds to the workload of the study teams. Paper also gives the subjects free reign with the information they provide. For example, they may write additional comments on the side of their questionnaire and that means there’s going to be additional time required, spent determining the relevance of this information. Deciding exactly what to do with this kind of data can be problematic for reviewers and require direction from the sponsor of what to do with that information. And it allows, finally, the on-site analysis of inclusion criteria allows for a certain amount of subjectivity where objectivity would certainly be favored. The dollar value we give here is just the time for analyzing. The exposure for unintended protocol violation and associated costs could be much higher. Again, in an eCOA solution, those inclusion criteria or the criteria you’re using based on the diary information provided is objective, timely, and it’s supported by the documentation.


You look here at the coordination and administration costs. Paper in and of itself adds a number of expenses. It has to be designed, you have to print it. Paper, ink, binding, packaging, shipping. All of these add up, from time and cost perspective, and not only increases with study size and multiple languages, but also adds a layer of complexity. Paper-based methodologies are not very scalable. That’s because there is an ever increasing amount of paper which gets exponentially more expensive as the trial progresses. And I’ll come back to it again in the future, but there’s also the consideration of, if there is a change in your diary, in your clinical outcomes assessments gathering tools during the course of the trial, you will have to destroy the documents that have not yet been used at the site, reprint, redistribute, and then you go on to further headaches, which we’ll talk about a little later.

And now we move to the big ticket item, the next one, which is the data management queries and data cleaning. Let’s be honest. The goal of any trial is to generate clean data. That’s the crux of the conduct of clinical trials. For the trial we’re talking about here, the subject has completed two diaries for 132 days. There are also some clinical outcomes assessments involved. It is almost entirely impractical to go back and make any inquiry or any investigation or any clarification against any single point in this trial, because each subject will have entered 264 entries. How would you go back and ask them to clearly differentiate one entry to get clarification?  In an eCOA solution, that doesn’t occur, because as a subject is entering data, first of all there may be edits in place if we constructed it with your guidance that limit what they can put in and keep the subject’s responses within the intelligence level. And the second thing is, you haven’t had to wait for that data to come in. That data comes in, in our instance, each and every night. If there is a lack of compliance, if there is some failure to update the database by the subject, the sponsor can alert the site, the site can actually be alerted automatically to go and contact the subject to make sure that they’re compliant moving forward. So we avoid the problems of some of the data management issues down the line, up front with the clients and with the subjects. And by the way, if you were going to go back and make some queries, it’s a multi-step process in a paper solution. You’re passing email messages back and forth, to and from site and data management. And everything that you’re doing in terms of getting the data collected and corrected has to be Part 11 compliant, you need the trail of why are the changes being made. It’s a monumental task. The number we show for the cost of $640,000 doesn’t account for the elapsed calendar time, which may then delay your time-to-market. Could you have made the filing sooner if the data were cleaner sooner? Quite possibly.

At the conclusion of the trial, all of the information must be archived. We’ve documented here archiving for 15 years. Different companies have different rules. Yours may be longer. Certainly, very few have shorter. Concerning the number of trials that a sponsor may be running, the requirement to file and store paper across thousands of patients and hundreds of investigator sites for this length of time, it’s a huge cost. In a single year, paper archiving at sites based on a storage of 49,000 paper diaries is predicted to cost $71,000, with paper archiving by the sponsor adding a further $36,000. And lest we forget to mention this, it’s a burden to the site, storing that paper. And also, let’s not forget why it’s archived. It’s archived to be able to be retrieved quickly and reliably in the event of an inspection. With eCOA, the information is available on a single CD or DVD provided both to the site and one to the sponsor, and the information is appropriately indexed to allow quick and accurate retrieval. With a paper diary solution, most organizations are going to ship the paper diary to an off-site storage, which is going to need to be retrieved. And even as retrieved, the condition of the paper and the sequence of the paper in the box retrieved, may present problems for quick and efficient retrieval.


So lets’ talk about a summary of what we are looking at here for the cost of paper. We’ve already established a baseline for doing the same trial that we’ve demonstrated here. This trial, as you add up the rounded—these are rounded costs. The rounded costs here that we’ve looked at for doing the trial on paper, comes to $1,163,000. And let’s compare and contrast that to doing the same trial with an electronic clinical outcomes assessment.

You remember these costs were spread over all over the place. SDV is being done by the monitors and team on site at the monitoring visits. The data management and queries and data cleaning are somewhere in your data management group. The diary data transcription is hidden in the increased cost by the sites. You need to transcribe the data or some double data entry group under your direction working somewhere else. These are all hidden costs, often not aggregated together and used in the evaluation. But it’s a requirement to do that to get the true total project cost. So now we have established the cost here, as we said, of $1,163,000. How does it compare to eCOA?

On the chart before you  can certainly see that the paper trial is a much more expensive proposition over the life of the trial. The eCOA, as we mentioned in white here, has up-front costs, the set-up costs are higher for eCOA, just as we discussed earlier. But there are substantial savings, as you can see by the much shorter blue bar for maintenance, during trial conduct. And lastly, database lock. We’re speaking here of cost. The database lock for a paper diary, moving it around and getting the data, various queries, settled, is much more expensive than doing an eCOA solution. But let us also remember that while it’s not only cheaper in an eCOA solution, it’s also much quicker.

Let’s look at that same data a different way. Look at it over time. Thinking back to the beginning of this presentation, one of the concerns or potential barriers to adopting eCOA is large up-front cost. The graph above, schematic, represents the cumulative cost incurred for eCOA versus paper over the same period of time. And as you can see, the eCOA, the yellow line, begins at a cost that is higher than paper. But you’ll see that with the lesser slope over the duration of the trial, the paper certainly begins to cross the yellow line and exceed it, reflecting the higher cost of the paper diary. And as you see, while the paper starts off at a relatively minimal expense, the steep slope quickly overtakes the eCOA as the trial progresses. And so the savings for the eCOA over paper is ultimately reflected at the far right hand side, as the gap between the two lines in the conclusion of the trial. And that’s the savings we’re speaking about here today.

Looking at those numbers in detail, you can see that that turns out to be about—in the trial that we’ve laid out today—$770,000. It’s a much more expensive proposition to conduct the trial on paper. Additionally because eCOA costs are typically up front, data volume doesn’t impact the cost as it does when using a paper diary solution. This same trial, had it continued instead of just for 132 days, for 150 days or 180 days, the cost would not have risen as fast as it would have risen for paper because of the front loading and cost of all the equipment up front and the design work done. The eCOA also provide cost efficiencies for scalability for allowing provisioned devices and reports and hardware to be re-used for several studies for maximum efficiency. Oftentimes when a single trial is done the follow-on trial or companion trials can reuse much of the same work, thereby providing reduced set-up costs, or elimination of certain set-up costs such as the activation of some equipment that can be re-used. Also keep in mind, depending on the requirements for subject diaries in this trial, it might be possible to deploy the same device to multiple subjects. Again, multiple subjects may be able to use the same device when enrolled later. You may have a short treatment period and a long study period, and a subject may return the device and it can be re-issued.


Look at the next, we’ll talk about hidden value. We focused up to this point strongly at looking at quantifiable hard currency savings. But let’s take a second to look at some of the other drivers that are non-monetary benefits of eCOA. There are some here, I’ll talk about a few of them. When we talk about improved monitoring with contemporaneous data, this has to do with, the data comes in, you know that it was collected because we have the date and time stamps. We know what time it was collected. The improved compliance using the ability to provide automated reminders and reinforcement training to sites where required. We have the capability of setting alarms, if a subject has not completely their diary entry within a set time window the device can alert them to that, to complete their diary information. Secondly, when I talk about reinforcement training for sites, because you see the data contemporaneously and automatically during the course of the trial not waiting for the subject visits or for the next monitoring visit. If you pick up something that’s going on at a site, that you don't think they’re well trained or they could be doing better, you have the opportunity to provide that reinforcement training. That all comes from the real-time access to data, which I speak to here, and also the use of advanced reporting tools that check and resolve issues sooner. We talk here as well about the ability to deploy mid-trial changes. You know, in trials, if the clinical outcomes assessment solution has changes in the midst of a trial, it may require the reprinting of thousands of paper pages, and has the additional burden at the site and for management of trying to manage which subjects should have been using which pages and when. In an eCOA solution, that is easily managed because the changes can be pushed out automatically even to the subject in the middle of a trial if appropriate. That will reduce times to database lock with expedited database cleaning. We talked about improved connectivity of the patients and the subjects. And finally, an eCOA solution must be Part 11 compliant. It has to have traceability back. Where did the data come from, when was it entered, who changed it, who touched it. That's all built into an eCOA solution. It’s an additional burden, because of your paper source, when you are going with a paper solution.

There are regulatory risks and potential delays in time-to-market when you’re using paper. First of all, the consistent use of paper, you can’t rely that the paper is going to be used consistently, and it’s hard to monitor that it is. That means that the data collected on the paper may not be reliable. Unreliable data is a risk. It’s a large risk. As we said, the objective of any trial is to get clean data. The gap between the collection of the data and the entry of it into the paper diary or paper system, there’s a potential for errors to exist for a long time that you’re not going to know about and you’re not going to be able to correct them. And obviously running a trial takes time, but even discounting that, taking time to close the database adds time to your trial. All the while, the patent clock is ticking, and in a competitive environment there may be somebody else trying to be first to market.

And so ultimately, having seen all of this, I would hope that the question in your mind is: is eCOA right for my trial. Together we walked through a model of a typical trial. Typical trial—your typical trial may be different than my typical trial. Each trial is unique in so many aspects. Is eCOA right? The smartest thing to do is to do the analysis on a trial or trials that you are considering. Each trial has its own constraints. This is something that we at CRF Health would be glad to work with you to review. If you’d like further information or to explore the applicability of eCOA to your next trial, we’re eager to provide that consultative service, to sit with you, to review your information, review your thoughts and design considerations, and help you select what is best for your trial. And maybe it’s not one trial. With proactive planning, a suite of trials using eCOA offers additional savings and leveraging of investment and training and process. I spoke to that before.


We’d be pleased to enter into these discussions with you, and we look forward to having the opportunity to talk with you.

Thank you for letting me speak to you about this today. And I’ll turn this back over to Naor.


Thank you. We already are starting to see some questions and we’ll start to answer them in just a second. I also would like to mention that if you would like what you’ve learned today, we have a gigantic resource library on our website. These are just some of the items you’ll find there, but it has over 60 pre-recorded webinars. Any articles that we’ve written in the past can all be found there and it’s quite easily searchable. So I encourage you to visit resources.crfhealth.com and have a look what else might be interesting to you.

[Q&A begins at 35:28]

Previous Video
3 Major Regulatory Risks You Can Reduce with Electronic Informed Consent
3 Major Regulatory Risks You Can Reduce with Electronic Informed Consent

Does your clinical trial’s informed consent process stand up under regulatory scrutiny? Learn how eConsent ...

Next Video
How to Visualize Data to Increase Trial Performance and Efficiencies
How to Visualize Data to Increase Trial Performance and Efficiencies

This live demo shows how new visualization tools in CRF Health’s TrialManager 6.0 enable monitoring of pati...