Watch on Demand

First Name
Last Name
Country (Citizenship)
Opt-In to Future Emails
Thank you!
Error - something went wrong!

Winning Strategies for Successful eCOA Implementation

March 11, 2016

An increasing number of clinical trials are using Electronic Clinical Outcome Assessments (eCOA) to capture more accurate patient data while creating a better trial experience for participants, site staff and study teams.

Ensuring a smooth implementation, encouraging widespread adoption and preparing your study for success depends on careful planning - and experienced guidance.

With over 20 years of clinical research and ePRO/eCOA implementation experience, industry expert Scottie Kern joins CRF Health to present his winning strategies for eCOA implementation.

Full Transcript



Well hello, and good morning and good afternoon, thank you all for joining us today. We have a really exciting live webinar planned. Today’s webinar is called Winning Strategies for Successful eCOA Implementation. Today’s guest speaker is Scottie Kern of Sacaja Consulting. Scottie is a good friend to CRF Health. He has, at this point, approaching 20 years of clinical research experience, throughout data management, project management roles and other roles in a variety of therapeutic areas and organization. Lately, Scottie has founded Sacaja Consulting in 2012 to address the need for genuine proven ePRO and eCOA implementation experience. And so we welcome him today. We’re really super excited to have him as our guest speaker.

We’d like to set the scene and get started with just a quick poll or two. So I’d like for you to take a moment and on your screen, let us know about your specific eCOA experience level. And you can do so by selecting. And I’ll give you a second to do that.

Great, so thank you for doing that. I’m going to keep this poll open for just a little bit longer, but as you can see, we have quite a variety of both sort of novice, a little bit expert, and everywhere in between. Which is great, because we’re going to be covering some things that I think are useful for all spectrum of implementation and experience levels.

The next poll is surrounding your biggest challenges with eCOA adoption. And here you can select more than one option, so I’d really like to just have a moment of your time, and let us know about your experience.

So as I look at this graph—and your results are still coming in—some of this is not necessarily new. I think obviously we’re always challenged with the perceived higher costs of eCOA, the lack of internal eCOA knowledge, by having to sell that value internally, the ROI is something that we always get asked, and lead time. So this is really interesting for us to know and this will help create some context around today’s presentation. So thank you for that.

And so, without further ado, I’m going to hand it over to Scottie, who will take us through the rest of the presentation. Scottie, please take it away.


Thank you very much indeed. And can I offer my own personal welcome to all who have kindly given over an hour of their time today to listen to my experiences, if you will, with eCOA, and hopefully provide a few interesting perspectives for you to take away.

I’ve had the opportunity actually to share, I guess, what I consider to be my best practices in a range of different forums and formats, whether it be coming to user group meetings to industry materials, magazines, newsletters, things of that nature, over the last 12 years or so. And what’s been interesting for me is that a few of them haven’t really changed since the first time I’ve done it. Obviously there have been some evolving factors around regulations and things like that. But I always come back to the same premise from the first time I did this, and that is: What are the ten things that I wish someone had told me in 2003, when I was starting my journey with eCOA, that subsequently would have made my life an awful lot easier, probably slightly less grey in the hair as well?

Now that sets the premise really for this presentation today. So it’s more generally aimed at early or new adopters to eCOA technology. But I think, as we’ve already discussed, there’s a few things that are certainly worthy of further discussion for those that are more seasoned, campaigned, battle-scarred eCOA veterans out there, who may well have some interesting perspectives on things that I will share. So I hope that there is something for everyone today.

And the first of these ten things that I wish someone had told me 13 years ago is around positioning of eCOA ownership.

Now, I feel very strongly, and I’ve learned through personal experience and through observation, that the location and the role and position of the eCOA owner within an organization must be held in an appropriate division of an organization. And further—as is normally the way with corporate-level implementations of something like a new technology—that we seek a project or system sponsor who is actually able to help you in your role as eCOA leader, elicit real change, and to help you navigate and deal with those challenges that will arise. Now this observation isn’t unique to eCOA by any means. But I have been in a position where I have been located somewhere that has helped me succeed as an eCOA leader and in other organizations where I’ve been positioned somewhere which has actively made it more difficult for me.


Now we all deal with different political structures within an organization. There’s consequences of acquisition and there’s globality to think of as well, where people are located and their head quarters are located, things of that nature which we can’t really do much about. But what I do know is that the individual who leads—or individuals who lead eCOA within an organization need to be in a place that they can interface as equals across all functions of clinical research, and even beyond the clinical research vertical itself, so moving into IT and other entities. One thing that I’ve found frustrating in the past, and I’m sure others have shared, is that people see eCOA as just technology. And it isn’t. It’s very important that we look at something like eCOA as being—as I’ve introduced here on the slide—as being a “business system.” And this is a concept that I worked with others to introduce in one of my former employees separates the technology, if you will, from really trying to understand and manage how eCOA implements in so many different things and how many various roles need to have input and contribute to management setup and oversight of eCOA. It’s not just a system. So system ownership is a very common thing for technologies and internal systems that we’re concerned about, everything from build, iterations, and various regulations, but we have to keep in mind that this is a business. It’s a business where eCOA needs to touch on so many different roles.

So I’ve found that—to give you an example, right now as an industry I think we’re starting to look more at thinking about clinical outcomes at the very earliest stages of clinical development for a particular therapy. And that wasn’t always the case. We started to do that a little later down the line. And nowadays because eCOA is becoming so prevalent, if we’re starting to consider that electronic solutions might be what best suits our clinical programs, those decisions need to be taken very early in a clinical development program, and start to prepare for them. Do we need to create new instruments? Do we need to look at novel methods of capturing our COA data? And if we’re not—basically, if people are positioned in parts of the business where they’re not getting access to those discussions, then that has already given us a handicap from the outset. So I feel very strongly that if we have the opportunity to influence where eCOA sits, it sits somewhere that allows them to work across divisions without restriction.

The Sponsor Status and Effectiveness thing is something that I’ve seen more than a few times, where there’s a project, we’d like to have someone high in the organization with genuine status, who is a champion for the cause and helps us publicize and get alignment to the mission of eCOA. But ultimately when things start to get a little more tricky and we start to maybe navigate into changing procedures or maybe—God forbid—we actually need resources to help us out, then things get difficult, and the individual high up there, high up the tree, up the ladder of the organization, isn’t actually able to help us deal with those problems. So whilst we’ve got the exposure, we don’t have the ways and means to get past the challenges that we—people in this audience today, I’m certain—have faced, as have I. So I do it’s very important that we get an effective sponsor, not necessarily just someone who is the big person that gets the most attention, but someone that can really help us get across those boundaries, those challenges of alignment.

The second point that I want to share with you. I think the poll that we’ve just seen has flagged this up as an issue, and the whole dialogue around cost is well established and it continues to be something that we have to work with and work against and manage with eCOA implementation. But one of the things that we’re often asked to do—and I’m asked to do, have been as an employee and have worked on more than a few times as a consultant, is trying to prove a positive return on investment. Because management require that to commit the dollars to make eCOA happen, because it’s a wait-and-see at a higher cost. That’s a whole separate webinar, a whole separate discussion, which we can have another time. But if management does require that you prove that positive return on investment, and that proof is done in advance of your commitment to adoption, then please be ready for that to be heavily challenged in retrospect.




 What do I mean by that? The fact is that it’s incredibly difficult to prove ROI. Anyone who has tried to do it will know this as well as I do. And there’s a number of factors that make it difficult, some of which I’ve listed here. The first of which are competing efficiency projects. Organizations seem to be in a constant state of turmoil around trying to improve and make more efficient how they do their business, which is necessary. But it does often complicate or even, at times, contradict what you’re trying to do with eCOA. I’ve been involved in both scenarios. Another type of challenge, which is just a consequence of where the business is, it’s all these partnership models that we have kind of trend of the last few years, where major CROs will partner and work very closely with a particular pharmaceutical company in a partner model, will take a certain percentage of executing maybe a number of trials for a program or perhaps the whole program. And now the establishment of those partnerships requires an awful lot of data, an awful lot of discussion, and a whole lot of things get discussed and agreed that don’t necessarily provide the information that we would need, as people championing eCOA, to get the decision and the right information on proving return on investment, positive or negative, in fact. It’s very hard to get to that granular level to know all the data that lets us find out how much it costs to process paper data as well as, indeed, electronic data.

Now, a quick point on that. Electronic eCOA implementation doesn’t just cost what CRF or another vendor charges you. There’s other costs associated with that, and everyone recognizes that, but it’s incredibly difficult to understand what those costs are. So, what is often the case is that what we might do is look to vendors like CRF and others, and ask, can you help us with this? How can we—what sort of models, what sort of algorithms do we have to help us prove that this makes sense from a financial perspective? And I worked very closely with Rachel Wyllie, the CEO of CRF Health, some years ago in her previous role, to try and work out what that looks like, to help internal customers of my employer at the time. And these are very useful tools. But that’s what they are, they’re useful. They don’t fully represent the absolute landscape that you operate in your organization. So it’s not a complete assessment of ROI. Interestingly, I’ve seen some companies commit to ePRO based on the assessments made on vendor provisioned tools. And that’s great. You’re having progress, but it isn’t necessarily absolute. So if we leave ourselves open to have this challenge, then we will do. It’s incredibly difficult to prove ROI.

Now what I will say without a shadow of a doubt is that I haven’t myself personally seen with positive return on investment unequivocally proven. I do think some of the calculations could have been challenged in work that I’ve done. We get as close as we can to giving an informed decision to those who make decisions, but I do feel as an industry, we can expect parity with cost. We can expect paper to cost the same as electronic. And indeed beyond that, we can start saving money through standardization, repeat learnings, generally adjusting our business to suit what eCOA can do for us.

A very big tale about the big CRO diary processing cost there. I’ve been, I guess, fortunate to see the pricing models provided by a major CRO to three different sponsor companies in my role as a consultant. And obviously that’s confidential information which I couldn’t possibly share. But there were two interesting things about that, the first of which being that the costs are different and significantly so in some cases. And secondly, this big CRO was actually costing the use of electronic clinical outcomes assessment data, based on a paper data model, i.e. per page. There aren’t really any pages in eCOA. And that shows you just how difficult it is when a major CRO is providing pricing on that basis. It’s a very big challenge. And essentially, I think we must always remember, it is really about quality.

Now our third point today is around qualification of your vendor partners, and obviously focusing specifically on eCOA. This is something that I do a reasonable amount of as a consultant, is help companies do this. So there’s one key piece of advice I give them, and that is to qualify your vendors in such a way that eCOA development for your trial, or trials, is not impeded. Because far too many times, it does impede on the trial. If we’re impeding on the eCOA development, we impede on the study itself, which is the very last thing we should do. My mantra, my own headline, if you will, of what drives me to do what I do, and what my objective is, if you will, mission statement for eCOA, is to have the minimum possible impact on clinical execution.


Now, this observation of mine is based on something I’ve seen, which I don’t think is specific to eCOA, but it’s something that does cause me a little concern. And that is the merging between the qualification of a provider and auditing a provider. And they’re two very different things, as we know. You qualify, probably to fix standard procedures, the appropriateness of a vendor to do its job, its financial status, its structure, a whole range of factors. And then, an audit is to assess that provider based on the work that they were asked to do, against regulations that govern their work.

But time and time again I see—and in the poll, earlier on in this presentation, we looked at a number of people saying that lead times are a problem. We kind of want to get eCOA in but we can’t, because we just don’t have enough time. And personally I think that’s a very wise decision. We cannot rush this. But at times we find ourselves starting to qualify a vendor and maybe things are observed, and they're put out there in the report, and we think okay we’ll manage those, we’ll deal with that. It’s not a high-level or serious observation, we’ll deal with that later. And then at audit, when we subsequently get up and running with the study—because auditors need to audit against a provided product, an eCOA solution—we find other things that either contradict what was put in the qualification or increase the level of seriousness or severity of the observation. And I see this actually a surprisingly large amount of times. And a thing that really concerns me most is that we’re getting in the way of the clinical trial. And the success of eCOA within your business, and in any business, is that it’s going to be a function of, really, how little noise it makes of a negative nature. It needs to be a positive experience. And I know studies that have had to delay start because of the eCOA solution, directly from failure to qualify the vendor appropriately.

This can be further complicated, I think—whilst I don’t want to give the impression that I’m being overly negative about CRO partnerships, but it is a layer of complexity that’s added to eCOA and general vendor management—is that if the CRO partner has qualified the vendor, does that qualification process the CRO followed mirror or at least be equivalent to your internal qualification process? So if Sponsor A went to a qualified vendor themselves, would the CRO’s qualification of that vendor be at least as thorough and to the same level of expectation? Because I have seen more than a few times that that isn’t the case. And the problem is again, it knocks on the study, and we simply cannot do that.

So I feel very strongly that if we have these observations, if things do come up—and every vendor will put up their hand and say, yeah, there’s things that different sponsors, different auditors, different inspectors, see things slightly differently—then deal with them, but let’s not impact on the trial. I very firmly believe, and have from very early in my eCOA career, that interface tied with partnership forums or vendor interaction forums, as I like to call them, which basically give us an opportunity to look at what we’re doing, what these challenges are, and can we manage them outside the boundaries of the clinical trial itself?

Now this whole point in general for me emphasizes one very key point, which I’m sure many people in this call have already experienced. And that is that, when we’re taking our first steps in eCOA, this truly emphasizes the value in selecting an appropriate pilot study, adopting a pilot approach for eCOA, and allowing very significant amounts of time, lead time, to get that study off the ground and consider all procedural impacts that you have, that eCOA introduces, because they are extensive. My most recent client, I suggested to them, you need to leave yourself at least nine months, which did not go down well, but I would say we have the very greatest chance of success and repeated success if we prepare appropriately.

Slide number four, the fourth recommendation here, is another one borne out of experience. What I want to clarify here is that, I mentioned alignment on regulations influencing eCOA. Just to clarify what we mean by regulations, that can include not just those guidances issued by our regulating bodies like the FDA, like the EMA, but also those industry-generated guidance documents, like those that we use regularly in this part of the business, generated by ISPOR, the excellent working practices and guidances that are generated by that group and others as well. And what I would say is that, we have these regulations, and there’s just enough room to interpret these regulations differently. Let’s multiply that by ten, fifteen people in a working group, trying to drive eCOA forward, or maybe oversee eCOA as you move through its use and scale up to larger and larger usage. I would have to say that it’s imperative that you ensure that there is full alignment in advance of using eCOA, on all those regulations that govern and influence eCOA use. And I said here, preferably document it, in fact please scrub that, please do document it. Have some way of having set in paper or electronic code at least, that the people who are stakeholders that make the key decisions over how you use eCOA are aligned on all regulations.


Now, there’s a few challenges to that, and the first of which is the sub-bullet there, please require that stakeholders actually read the documents, which is a tougher thing than it probably should be, but we’re all incredibly busy people. There’s lovely vendors out there who do a very good job of distilling guidance documents down to the key points that we need to digest. But that’s sometimes where the gaps can arise, because I’ve seen vendors interpret things in different ways. So require stakeholders to take the time, create the right working group, and make sure they read these documents, because they are pivotal to planning parts of your future strategy. And we always have to keep in mind, I think, that there can be divergence—not disagreement as such—but sometimes there’s gaps in the guidance documents in how they address certain points. And I think the classic example of that in recent times has been the draft eSource Guidance generated by the EMA and the FDA where there was actually—they went off in slightly different directions on a couple of aspects. And as versions of those documents are produced, maybe those will converge again. But as an organization, you need to have an agreed position on this, because these things end up coming up when the study is up and running and your eCOA is being used. For example, from an eSource perspective, is the investigator approving the data before it is released to the sponsor, which was a kind of mindset of the first draft of the eSource guidances, but that’s just simply not the case. We need to be aligned, or it creates challenges later down the line.

Another example of a different guidance document is that, for those of you that work in the Japanese market may be or may not be aware that Japanese pharmaceutical manufacturers association did actually issue an ePRO guidance—in fact, I think it was in 2012—which was an appendix to their EDC guidance. But it’s a public guidance generated by the Japanese pharmaceutical community. Are we aware of it? If you’re going to be working there, are there implications of that? We have to read and we have to align on this.

Now, one flag I’ll throw up is around the aspect of the data privacy, which obviously in Europe, we’re very keenly aware of. But I think the industry as a whole community are aware that data privacy is becoming more and more of consequence to our work. But I think because data privacy is an evolving fact, and there’s data privacy offices opening up, obviously, and have been opening up for some time obviously in some major organizations, that I think the implications in Europe can be very complex. A function of things like, where is the trial being run, where is your data being viewed, where can anyone see it, how is it being stored, all these aspects that should be a part of your qualification of the vendor have to be addressed within data privacy expectations certainly, specifically within Europe. And it can be a very complex pathway, weeks and weeks and weeks of discussion. This gets legal, this gets very complex. So please keep an eye on data privacy and engage with the data privacy office very early in your discussions around eCOA.

We’re talking regulation here, we’re talking about compliance. But just let me tell a little tale. And I’ll give you an opportunity to—we’re not going to do a poll here, but I’m just going to give you a few options on the next screen to choose. I’d like you to be able to tell me, try and guess, what a quality group representative said in a meeting that I attended at the beginning of last summer for a small biopharmaceutical company in mainland Europe, relating to eCOA usage. So if I can set the scene, an individual had been recruited as a consultant to consult on the quality aspects of their work at this particular small pharma. I attended the meeting, as I said, and the individual said one of these four things, one of these four statements. So I’m not going to poll on this, I’d just like you to consider them for a moment, which you think might have been what she said.


Was it: A full printout of all data must be generated at the site and signed off by the principle investigator.

Number 2: The patient must give separate permission for CRO staff to view their ePRO data.

Number 3: That the FDA PRO Guidance requires that a paper backup be employed.

Number 4: A data confidentiality assessment must be performed for each site for every trial that employs ePRO solutions.

So if anyone chose number three, I congratulate you. That was what was the actual statement that the lady in question said. And we had an interesting and absorbing discussion afterwards as to where on earth she got that from. But I will congratulate everyone who said number one, two, or four as well. Because the fact is that in my reasonably long career with eCOA, a quality group representative has said one of those four things. Now, many of us will know that there are challenges with those four statements which we need to navigate, but this more than anything for me emphasizes what we need full alignment across all stakeholders, and of pivotal importance for the quality function in contributing to a successful strategy.

Number five, prepare for your project, which might seem like the most idiotic statement ever, for which I apologize. But what I mean is that we need to help the project succeed by preparing for your project manager. And also to require that project manager to use tools that promote visibility to you in your role, and promote visibility on issue management.

Now what do I specifically mean by that?

One of the constant challenges, one of the things that—I have a lot of empathy for project management staff, program management staff in trying to help them succeed in my role. In my legacy roles, I want to make sure that I do my best to help the clinical trial team, the clinical project managers succeed. But in eCOA leadership, I want to see the eCOA project manager within the vendor be successful. Now, many of the challenges that I’ve observed the project managers find are in the early stages of the trial where they were making decisions about things that impact very heavily on eCOA, during or after the kickoff meeting of the project. We need to prepare better as organizations. And part of the work I end up doing a reasonable amount of is actually helping companies do this, to prepare, to get ready, to make sure eCOA is successful.

Now there’s a number of ways we can do that. I’ll speak to the point about promote visibility and issue management in a moment. But the first bullet there is around defining your eCOA policies. And that’s the term I use to describe what would historically have been called data handling criteria. What will you do with eCOA and what of the variables involved in eCOA do you need to make a decision on? To give you an example, I’ve worked in the past in organizations where an internal IT SOP procedure would govern the amount of time someone could enter a wrong access code. And by internal procedures, they would expect that SOP to govern the amount of times you can allow someone to try and enter a PIN code onto an eCOA device. So these sort of things need to be addressed in advance to prepare you for a better, successful, or higher chance of success with your eCOA implementation.

Now, you may ask, we don’t know those things yet, because we haven’t gone through eCOA implementation. And I think this is where—I know CRF have done a very good job in recent times to try to prepare their new clients for things you need to consider before we even come or even before CRF maybe even win the study, or other vendors might win the study, but we start to make decisions that will be possibly rate-limiting or get on the critical path of actually creating our eCOA solution for our study or studies.

Now, the FDA Guidance is very helpful on this point, because the FDA PRO Guidance actually requires us to have in detail, some reasonable level of detail within our protocols, how we’re going to be using the PRO solution. And so I thoroughly recommend, if you’re not doing it already, that your protocols have some pretty extensive content. Something that I do with the clients generally is that I help them create a supplement or an appendix to their standard protocol template. And then that covers an awful lot of the language, including things like data privacy, and other aspects like content and consent can help with with some other aspects as well.

But there’s much we need to think of before we even get to the kickoff meeting. And something that I’ve implemented in my past employers is a pre-kickoff meeting where we have a smaller group of people that look at those key aspects and try to make sure that we’re getting to decisions on them before we get the whole team in the room to kick off the project in general.


And there are a high number of common pain points which project managers and people on this call are very aware of, that we need to make decisions on early on, and not later down the line. Translations is always fun—one way to describe it—depending whether they’re being done internally or through a vendor’s partner. Logistics of where the device is going to go, who’s going to look after them, how are they going to be moved, whats the re-supply stretch. These things can be dealt with early on.

And one very important point for me is the reporting needs. Many people want to have access to the web portal, TrialManager in CRF Health’s case. Do they need access? What should they be able to see? What reports do they need to see? Different roles might want to see different things. Now, there’s a lot of flexibility in reporting across the vendor community. Generally speaking, vendors will be able to give you what you want. But you need to consider that very early in the process, because knowing what you want at the end is what we need to front-load and know from the beginning.

Another aspect I’d call up is around sponsor-created documents. A classic example here is around any sort of document or mechanism for which you assign user roles for people to access something like your EDC system that you used in a trial or a randomization of trial supply management, IVR type system. Those things need to go to the vendors as well to create access, and that can be a very very painful process. Now, if you’ve had the opportunity to work with some excellent clinical operations staff, clinical operations managers, who are on top of this sort of thing, then it’s—as I have in the recent past—then it’s great, but if we don’t prepare for this, it always causes problems down the line, and we could have dealt with this earlier.

Now I feel very strongly—and I make no apology for saying this time and time again—that our goal must be partnership. We must go into the first study under the impression and with the intent that this vendor, maybe CRF Health, will be someone we’ve worked with repeatedly in the past. So our early development process must start to look at what goes on in the future. So we learn as we go along, and we think about ways that we can get that information to the project manager in a timely way that enables them to be successful.

But just to come back to that point up in the blue box around “use tools that promote visibility and issue management.” Now Microsoft Project can be a painful beast, as I think we all know. But it’s an excellent tool for telling us where we’re at and where we should be, if used appropriately. It’s resource heavy to maintain, but it does a really good job of what we need it to, well, in the absence of anything better. There may be people out there on the call who’ve seen better tools than this, but I’ve kind of gotten used to Microsoft Project and it seems to be something that project managers don’t really want to update, they become like—oh we can update you in another way. But I really do feel that every sponsor should feel that they can instruct the vendor project manager to give me the information I need to show where we’re at, what are the dependencies, and where should we be.

User acceptance testing, user system testing, whatever we want to call it, is a very challenging aspect of eCOA. In fact, I think for general technology use, it can be a very onerous, challenge, and very resource-absorbing effort. Much of that is dependent on the strategies or I guess the philosophy of the sponsor organization, because one thing I’ve seen is that different companies often take very different approaches to something like UAT, as we’ll term it in this discussion. But what I do feel, and something that I think I learned from my very earliest adventures in eCOA is that the best way to test your eCOA solution is by living it. Make it work as if you were the patient. And be certain that your existing internal processes for clinical system validation are wholly applicable to eCOA.

Now I’ll come onto that second point in a moment. But testing your eCOA solution by living it. I genuinely do mean that if your device is to be used within a certain period of the day and a certain number of days for a certain period of time, whether it be a day or two to weeks, I do strongly feel, and I’ve seen, best successes when people actually go through it as if they were a patient. Now that can’t always necessarily be practical, and sometimes the tools can be manipulated to time travel, as we call it, and we can move backwards and forwards. But whilst it is onerous, and it can be very resource draining, I’ve seen the best success when we do it that way, and the lowest amount of potential issue. Because UAT then finds out the things that we wouldn’t have found out until it had gotten into the patient’s hands.


Now, UAT for me is an interesting thing because, essentially for me, UAT should be a testing process against the requirements that you as the sponsor, or CRO, what you document. Now whether that’s done in a collective manner with a vendor, with a requirements document, or whether you deliver something as a requirements framework to a vendor, that’s what you’re essentially test against. But there’s a gap there. There is a gap that I see time and time again, which is what I call the heuristic gap. And those are the things where experience tells you that things go a bit wrong. Or things you just didn’t know could possibly be a problem, but maybe are. An example there would be, we get into a live study and we have access to the web portal, and we see something we didn’t expect to see. A piece of information, a piece of data that we didn’t expect to see. Why am I seeing that? My requirements would not have said, make sure I don’t see anything I shouldn’t see, which is a ridiculous requirement. But it is, essentially, a kind of expectation. I shouldn’t see anything that you shouldn’t see. And sometimes it just—with experience, as many people in the audience today, they would have started to look at some things that they think oh we’ve got to keep an eye on that. The portal is a particular favourite of mine, I have to say. But it’s only when we’ve been through the whole process of being a study site coordinator, being an investigator in that sort of role, and really analyzing the solution in that way, can we actually learn these things.

Now, data transfer has to be part of your UAT from my perspective. There’s many company who deal with data transfer or maybe data integration if you’re lucky enough to be able to leverage that technology. They bleed into study starts, and I do feel very strongly that we need to know before the study goes live that we’re getting the data in the way we expected. So please build that into your time.

And also to be aware of the internal standards document, the beast that it is. Many companies have these internal standards documents which are things which maybe the sponsor or CRO can see, but are not available to be changed. It’s a core standards document, which makes sense. I learned early in my career that some companies are very protective of this document. In fact, they wouldn’t email it to me. So I had to go and visit them to see their internal standards document. I don’t quite know why that happened, but anyway, we moved on. Now, that constitutes part of your requirements as well. Their internal requirements for how they deliver or build their system as a vendor is essentially part of what you need from them. So you need to be aware of it. And there are some things that may well come up for discussion. So I would encourage anyone to make sure that you assess and discuss. I will confess, I have found faults in more than one internal standards document in my 12-13 years as an eCOA manager. And just to call out, for those that might not have much exposure to best practice or guidance or validation from an eCOA perspective, I’ve given a screenshot here of the ISPOR Guidance, which was generated last year or two years ago for the working group at ISPOR, which I was lucky enough to contribute to with some other more talented people. And this gives you some nice guidance of the types of things that you need to consider from an ePRO systems validation perspective. So that might be helpful for people who need to go off and consider that particular fact.

Number seven, and again this is one of my buzzwords really. We must invest in our CRAs. Now we rightly focus on the sites when it comes to technology. I think that maybe  a few people referenced in the poll here that site resistance was something that was an obstacle to eCOA implementation. I personally think we can always navigate that. I haven’t failed in trying to navigate with the site as yet. But we need to look at the relationship, we need to keep the sites comfortable.

But I do feel that the best way to do that is we need to have dialogue with those sites, of course, but we need to empower the CRA. The CRA owns that working relationship. The sponsor-to-site relationship is governed predominantly through that CRA. So we must empower them to be able to support and manage the site from an eCOA perspective. Obviously CRA has a million things to do. But if we under-train, if we under-equip our CRAs, it does cause problems. This isn’t a might cause problems, it does cause problems.


And so, in relation to that, I would say that please, as you’re assessing your sites, you’re building up your early study work, please be very wary of that site that says, we’re fine. They proclaim an almost absolute expertise with eCOA, for that particular eCOA solution. And also to be very very conscious of that site that has very little or no use that seems to think, oh it’s easy, this is no problem at all. Those two are the ones where I’ll spend most time. Because they are the ones where we’ll very often see the challenges once we go live. We must invest in the CRA’s knowledge.

An example of something that we try to do, which is sometimes a little difficult in terms of planning, but we try and equip CRAs as they’re going through site qualification selection processes with an actual eCOA device if we’re using something like a device-based solution, to allow the site to see it, give feedback and get an understanding of what’s involved. Some sites are very comfortable, some are not. They like that opportunity to see what’s happening. We give the CRAs something they’re comfortable with, they develop that relationship from the outset. And a good company will follow up on that. I’ve found myself doing quite a lot of work in that space for companies recently.

I do encourage you to really look at the CRA role, the CRA worksheet, the monitoring plan, look in depth of where eCOA touches them and then help educate them on what they need to do differently. Document it. Make sure they are fully trained. Give them the opportunity to attend a workshop where we bring the eCOA solution in. And every CRA has to leave that room happy. Many CRAs are obviously very accustomed to technology nowadays, and they’re having to work with it constantly. And it becomes fairly straightforward process for them, but for some it’s a little more difficult. If we don’t have the CRA comfortable, we can assume, many times, that the site is going to be fairly uncomfortable as well. And the facts are very clear. Under-preparedness of CRAs is a very significant and proven risk point.

I actually just— Just yesterday, I think, actually I read something in the Applied Clinical Trials magazine, that’s very interesting research around what concerns sites in terms of e-clinical technology usage. And reading that synergizes perfectly with this slide. Because there’s many statements about the CRA relationship with the site, and how that impacts on technology and site satisfaction with the use of technology. Yes, we must make the sites happy, but I do feel the way we do that is to make sure the sites are okay. One tip I will give in general is that if anyone is thinking otherwise, please have a hands-on session at your investigating meeting for site staff, and have it with the site staff that will actually be using the ePRO solution, which very often isn’t the investigator.

Number eight addresses the issue of non-compliant patients. Now this might be slightly controversial, but I think many of us think it. But maybe we don’t want to say it. But I feel we have to accept as a business that patients demonstrating non-compliance with the eCOA solution you are using in your study are also very likely to be non-compliant in other aspects. I feel at times, working with some—in my past as a colleague in some companies, but also as a consultant as well—it becomes very very easy to say the eCOA solution is the problem. And instantly we want to make sure that the patient isn’t being wrongly accused of being basically a less-than-perfect patient. And I think at times we just have to really step back and accept that many times that is the case.

It is very easy to lay the blame squarely at the eCOA solution. That attitude can be coming from the site, it can be coming from the CRO as well if we haven’t done a good job of preparing them. Or the site was, maybe someone who said they were experienced but ended up getting very upset when a patient doesn’t transmit their data after two days. Remember, that was a site that said they were comfortable. This happens so many times. And now, this also has a very close connection with the previous slide in terms of making sure that we support the CRA-site relationship.

But it is very important that we try and address and predict this might happen with our sites and try to prepare them for that. And one of the ways to do that is that how we make sure we ask—the manner in which we ask sites for access to something like their TrialManager or the portal, to view their data and perform their roles, which you should have detailed in the protocol, and how we oversee that. And the thing about eCOA is it gives—what eCOA essentially gives us an opportunity to do is to provide a very helpful mechanism for an investigator to maintain their source data. Maintain for us in this context means you go in and look at your data and assess the status of your patient. And that feeds into this CRA-site relationship thing because we’re seeing what’s going on at a site level, and so we can have discussions about these patients. And very often, these discussions lead us to conclude that maybe the patient wasn’t so good.


This is a benefit of eCOA. It’s not a bad thing to know that a patient is being non-compliant. There may—and there are, commonly—good reasons for it. Network problems, devices fail, can happen. Rare, but it can happen. But it is a very key benefit for me that I can see, in some ways, what’s going on out there in the patient’s home and their real lives. Some study project managers seem to think that this is a bad thing. I think it’s the complete opposite, and I think we need to have that mentality.

And to give you a little brief example, we have the tale with the SIM abusers. This is an unfortunate incident with a previous employer, where we were looking at the costs for data communications from a couple of patients in the trial. And we observed that, unfortunately, they were using an awful lot of data on the SIM cards that were inserted into the eCOA device. And it ran into a five-figure dollar amount each, when realistically this should be, if anything, cents, euros, pennies. Tiny amount of money. So fortunately, the vendor—it was CRF Health in this case—brought this to our attention and thought, this seems a little exceptional. And guess what. Those particular patients had removed the SIM card, which you could get away with very easily in those days, and put it into another device and were happily surfing the internet using our SIM cards at my employer’s expense. As a consequence from that, CRF very quickly and very capably put in a mechanism to deactivate SIM cards remotely—and I think most vendors do that now—to make sure that can’t happen again if a patient is observed to be non-compliant or stops using the device in the right way. But it gives us a nice example, given the fact that those particular two patients, I recall very clearly, were also very non-compliant patients who missed several visits, and when the SIM card was turned off, they never returned to the site.

Number nine is— I think as people who govern or lead an eCOA organization, they instantly and obviously want to get feedback from the success of it. That’s not unique to eCOA in any way. But one thing I do think is very important about eCOA. When we work in a business that is very much about trying to ensure that a patient fully understands what we’re asking of them, that our technology enables them to answer in a very accurate and contemporaneous way how they’re feeling, what’s their experience during the clinical trial. The feedback becomes even more important for me.

One of the reasons I love eCOA is because it’s in the patient’s hands. It’s not just about sites, it’s about people living with their condition or managing their reactions to something. It’s about their real life. And we can build in a whole layer of feedback loops that are interesting to review, but we need to act upon that feedback. I consider the ongoing validation of that eCOA strategy. What we thought we were doing, that helps us refine and improve.

Structure surveys, get them out there to patients as well as site staff. Go and meet sites, talk about their experience. And then show, as part of your relationship with sites, which I feel is of some importance as I’ve mentioned a few times, that we are going to make changes to how we do eCOA based on what you told us.

And we look at the overall process. What’s it like for your customers as clinical trial members, your data managers, your statisticians. Your auditors, as we mentioned earlier. What’s the experience? What more can we do? But for me, selfishly, those are important things. But I always think about what was it like for the patient, what’s the real opinion out there? And then sharing those learnings with relevant audiences.

Out of interest, one of the first surveys that I did with a population of elderly subjects in South Africa. Got some fantastic feedback, but several of them asked, can we see the results of this survey? Because they want to know what other people think as well. And we’re in a mode right now as an industry, but thinking more and more about the informed patient, we’re thinking more and more about how we share what’s going on, expose the real experience of what patients are—how they share amongst themselves. Patient communities, patient-driven study design, things like this. E-COA can tap into that. And that concept can help us improve how we administer and manage eCOA as well.

Something that, I think, sometimes causes a little resistance, but I do like the idea of every eCOA solution having at least one little survey within it. You can’t do it outside of the device, tag on four, five, six questions at the end of your eCOA solution, and then ask the subject or patient once, how was it for you? Was this easy or was this not? Very valuable data. And one thing I would say is that I presented at a DIA forum some years ago. The results from those first few surveys. and I will say that there’s three slides in that deck that were quite revolutionary in making people commit to eCOA, because their biggest concern was thinking that it would be a real burden to the patient. And when they saw that it wasn’t, we went over the cliff of eCOA adoption.




 Now, to my final slide.


 As I’ve just touched on very briefly from the last slide there is that I do think we need to look a little further beyond what we can do, the main capability of an eCOA solution. Now COA is essentially about reliably measuring change. But we must consider what else eCOA can allow us, what else it can let us know about the patient experience beyond just measuring that change. And this is about metadata, this—our tools right now, mobile health, bio-census, peripherals. What more can we do to get more and more data, which I think people are still struggling to work out what we do with that data, but what do we do with that data? What more can we learn, and what can it tell us about the experience of the patient?

And eCOA has been able to do that in some ways for years, but we just don’t use it. Maybe we don’t know the data is there. But it gives us a real opportunity to start profiling patients a little, thinking a bit more about how they interact with a device, how long they take, what things trouble them. We can learn a lot more from eCOA, so as we progress, I hope that as an industry we do a bit more to think about what’s available. And in fact, I also challenge the vendor community to come to the clients and say, yeah, can we have a bit more? This is what we can do. This is what we can show you. This is what we can learn. And then start to push the boundaries a bit more, the more we learn. Get more of that real-life experience. I do feel it is a nice adjunct to that dynamic of wearables right now, so we continue to feed that data-driven picture of what the real patient experience is.

And that comes to the end of my presentation. So I will hand back to the chairperson for the closer.

[Q&A section starts at 52:33]



Previous Video
eCOA Essentials: 1 Hour to Higher Quality Data
eCOA Essentials: 1 Hour to Higher Quality Data

This high-level introduction to electronic clinical outcome assessments will outline the many benefits of e...

Next Video
eCOA Design Like Nike: How a Great Interface Can Improve the Clinical Trial Experience
eCOA Design Like Nike: How a Great Interface Can Improve the Clinical Trial Experience

Paul Margerison demonstrates how principles of user experience design, used by companies like Nike, can be ...