Download Premium Content

First Name
Last Name
Country (Citizenship)
Opt-In to Future Emails
Thank you!
Error - something went wrong!

Approving Library Versions of Instruments

July 22, 2019


Every time I come here, I feel like they put me after a really great presentation and then you see my title and you’re like wah-wah. Because I’m always talking about something that’s kind of, how do we process things in a way that might be a bit better and make things more efficient, or something operational. I tend to take on those topics a lot. So I really want to thank TJ for his presentation this morning because it really brought back to mind for me, what is the main thing that got me involved. About seven or eight years ago, I said to other people at Optum, we ought to allow companies to have these libraries. They said, well why, and I said, well it takes time to go through these reviews and a lot of times they’re using the same programming and it just makes sense. And you know, you think about it from an efficiency standpoint, but the more important thing is the patient standpoint, because the longer it takes to get a trial into the field, the longer it takes to get data and get treatments that are of value out to patients. So thanks to him for that, and hopefully that tidbit will help keep people interested and say, this isn’t really just administrative, there are really very important outcomes to patients, based on how we go about doing things.

Two other things to start with. One is yes, I work for Optum, I work in the Patient Insights Division. That used to be QualityMetric, before we were bought. I work as a scientist, so my main interest is understanding patients and how they feel and function and what it’s like to live with diseases and their treatments. So while there is a licensing group, and I’m here talking a bit about how that group interacts, I’m actually coming from the science perspective on it a little bit. And the third thing is, just because my mind wanders a lot, I was thinking, how did I come up with this “library” word in the first place? Because really, everybody kind of keeps libraries or inventories of all sorts of things, right. Everybody has a file folder in their computer systems, and it’s not really anything relatively new, so you might be thinking, why do you have to have a process for this, we all keep and reuse things over and over again. And you know, what I hope you come away from this with is an understanding of why it’s important to have a sort of relationship about how you do it, with the instrument developer, whomever that is. As a scientist I work with lots of different instrument developers. And I chose library as opposed to inventory because inventory just sounds like one small person in this really big warehouse and eventually in a movie someone comes by with a forklift and chases them and there’s lots of blood and it’s a bad thing, whereas I grew up going to a library and you have a librarian and it’s very warm and somebody’s answering questions for you and there’s back and forth. That’s how I really want to see this, this is a relationship between eCOA vendors and study sponsors and instrument developers.

So for context here, some of these terms you’re going to know. Any instrument, any type of clinical outcome assessment may be available for licensed use in several different modes. You could get it on paper, and you might program it in a device to look exactly like it does on the paper version. You may use a single item form. In Optum we call that a SIF. And there’s two different versions that we have. One is for a large tablet, seven-inch or above, a larger diagonal or computer screen. And the other would be for a smartphone or a small screen.

Many instrument developers require form reviews of the eCOA programming prior to use, and that is to ensure proper migration. The positives of this: ensures the users program the correct version and mode correctly. And that can help ensure good implementation and interpretation of data. And I could tell you that there have been times when someone has forgotten to send form reviews and we’ve reviewed it and it’s been a significant problem. Like a repeat of a response choice, in which data was collected that had to then not be used. So it is an important thing to do.

There are some negatives to this process that include an impact on your study timeline, because it takes some time to get them reviewed. If not everything’s right, then you have to correct it and send it back again and it gets reviewed again. There’s a cost to the study sponsor to have this done, because it takes someone’s time to review it. And you know, there is this issue of sometimes having to repeat reviews for the same instrument, even for a same sponsor that’s for a different study.

So my goal today is really to describe and answer questions about our own Optum Patient Insight eCOA library program. I can’t speak on behalf of all similar programs. But we started doing this many years ago just for eCOA companies, and now we recognize the need for study sponsors to really understand what’s going on. So it’s a little bit different approach.


There are some definitions I’m going to use today. Some of these you may know. An eCOA, here for this presentation, is any clinical outcome assessment presented in any kind of electronic format. SIF, I’m going to say that, and that just refers to a single item form presentation of any instrument, as opposed to having many questions on a paper at once. Sponsor is any company funding a study that will use an eCOA. It could or could not be a pharmaceutical company, but in this audience I’m going to assume that it is a pharmaceutical company. I’m going to say eCOA vendor as opposed to eCOA company, because this presentation is a little more targeted for the study sponsor today. But it could be any company, whether it’s specifically an eCOA company or a larger company that just has a component that does that, or even a pharmaceutical company that has its own department and solution for that. Library, here we’re talking about a collection of approved eCOA programmed instruments. And they’re approved based on a form review of screenshots. And then a license is an agreement between an instrument developer or authorized distributor and sponsor, to use an instrument for a specific purpose.

I have to go through a little more boring stuff, sorry. I’m going to use the example of the SF-36v2. Traditionally, what a study sponsor does in order to use it is to get a license. And then the eCOA vendor fills out a separate form called the Acknowledgement by Agent, or ABA. And that allows them to use, receive, and program the SF-36v2 for just that study. Upon the execution of those, the study sponsor is the one who receives the correct forms, and they should be responsible for providing that to the eCOA vendor. And they receive three different forms. One is the instructions on how to correctly format it, and that only comes in English. Then, for each language and country, a Word document table that’s non-formatted that includes text strings that you could just copy and paste into programming, and then also a screenshot example. So in a non-library approach, you conduct form reviews for every single instrument in every single language and country for every study.

What is our form review process? After programming and the eCOA vendor’s own QC process, the eCOA vendor submits screenshots to this email address. Our science teams are the ones who conduct the review to identify errors and send back a review form that specifies any errors to be corrected. And the process is iterative, it goes until there’s no errors. The final form review has to be marked as “passed review.” And that’s approved for use for just that study.

Why would we not just let anyone reuse prior programming? Well our primary goal here is to ensure the highest quality standards for our COAs and eCOAs. And the incorrect reproduction of them can impact study findings and the resulting publications. And that can really be catastrophic for patients, healthcare providers, sponsors, and us. It’s really important, and we’ve gone back and looked at data before and found issues where items got reversed and then the scoring software worked incorrectly, and things have been missed. And that’s really important to not have it happen. We also know that eCOA companies or vendors also want the highest quality reproductions. They don’t want to have any problems with what they’re doing. And they want to be apprised also of any updates to instruments. We try to not update instruments as much as possible, but some countries change their entire language sometimes and there’s things that you have to do, or change a different grammatical aspect.

In the past, most devices were programmed specifically for every study and it was a little less feasible for eCOA vendors to maintain a library that’s more official. But now, different approaches make it more feasible.

The solution that we’re talking about today is this library partnership program, and it’s something that we’ve done for five years now, directly with vendors. But this year we started offering it to sponsors. I think, when we’ve worked with vendors, it gets a little difficult because they’re really trying to get every study underway, and this was a thing that just relied on them and us, and there was no sponsor involvement, so it’s really hard to get it going and have someone paying attention to it. So in this case what we’re doing is having a master licensing agreement for a library for the sponsor, which includes review of every instrument form until it’s passed to be able to enter the library. Once you have an MLA in place (the master license agreement), you have individual statements of work that just define usage for the study but you don’t have to go back through and do the form review again.


There is a pricing for it, but it varies depending on the number of forms and languages that you would want. And then the base package identifies only one eCOA vendor of choice, but you could expand it to work with multiple vendors, because we know that some companies work with more than one. It’s renewable annually, and the fee is reduced after year one. But I think it’s important to know that the fees do not include any fees that your eCOA vendor might charge to actually program these and maintain it.

I went through that kind of fast. So I said it was a partnership. So for the eCOA vendor side of it, the partner requirements that are in place is to submit a policy. It doesn’t have to be official SOP—because I know companies don’t like to give those out—but it at least has to be something that you put in writing that says, this is how we’re going to maintain this library, this is how we’re going to make sure that there’s a process for checking the code in and checking the code out, and so that there won’t be changes to it. Now that I’ve been doing this for quite some time, some people have probably also experienced this, that a programmer may say, oh I like this word better, and they change it and they put it back in and you check it, and you say, why is this changed, you guys did this many times before and it never was different. Oh, Jim over here in Ohio thought it would be a good idea and read better. And you’d be surprised at how often that happens. So you’d want to have these check-in and check-out procedures. You’d want to know who’s the point of contact for us. And we’d want to know any QC processes that would govern the library, including review for drift, so occasionally checking. You’d have to maintain an itemized list of the contents for each sponsor library, that specifies the approved instruments by name, form type, and language. And the reason we want the eCOA vendor to do this is because every so often we want to match that up with what we have and what we think you have in the library. And then participating in a recheck process, and I didn’t specify out too much what that was here, but generally it involves about every few months, hopefully if there’s a study upcoming that requires those languages but otherwise randomly selected, rechecking several forms. Meaning, we often do that and tell you yes, these are still okay, clearly your processes are working.

What do we need to think about? I just went through a lot of stuff kind of fast. For a sponsor, I think some of the questions—but I want to hear from you, I really want to hear what would make this work better for you. For a study sponsor, I think you’d need to think about: Which instruments does my company use? Which vendors does my company use? If I were to enter into a library program, how would I inform my colleagues or other internal stakeholders? Maybe you’re in an HEOR or PRO group within the company and you want to make sure that medical affairs knows, that different clinical departments know. Even education groups or whatever else, that they have this opportunity to use this on an ongoing basis. So how would you inform them, and how would you keep that relationship going internally. And then, who would communicate with Optum and the vendors to keep that list going of the updated library contents and understand what the new study needs might be if they change or something isn’t included.

For a vendor, I think the questions that you might have are: How will I maintain library content separately for different sponsors, or is this even required. So far, with different sponsors we’ve talked about, there has not been an interest in a combined library. But what we did before directly with vendors was not sponsor specific. Anyone could access it once it was in the library, and we’re not opposed to it at all. But we understand that study sponsors may want—they might have their own logos that they want to put on things, or other things like that they don’t want to share, or their own look and feel. So I think the vendors have to think about how they would be able to maintain those separately. And what is the best way to maintain the library contents and communicate with Optum?

And then, this one to me, this last one, is really important for vendors: How will I construct the library ahead of a need for a study. Because at least in my experience with eCOA companies is that you get a study contracted, you know what provisioned device or BYOD or whatever that you’re going to use is, and then you get the intellectual property and you start programming it and you go through QC processes. But in order to have a library, the idea is that you would program all of these ahead of time so that they would be ready when the sponsor needed them. There’s a couple different ways to do it. You could also build a library by saying once they’re in there, we keep it in there but we’re not going to build it in advance. But I think if the sponsor enters into this and says, hey we want to do this partnership and we want you to do that, you need to think about, well how am I going to do that, or what would I charge the sponsor to build this in advance so it would be ready for them.


So that’s something, and just to put it in context a little bit, the SF-36v2 standard, which is the four-week recall alone, we have over 180 translations, and each one of those is a form. And acute has just as many. Then you’ve got the SF-12 and many other forms. So we’re not talking a small amount of programming and list keeping and storage.

So those are some things that I think about when I think about how can we make this work, how can we build a partnership, and do this in a way that is going to help studies get into the field faster, while maintaining the high quality that we want.

I’m going to give a couple examples of different types of companies that might use this, and then I’m hoping that you guys can ask me some questions and give me feedback. So example one here would be a small pharma company that typically uses the same 10-20 languages, they don’t have that many programs underway and they tend to operate in the same countries, and they tend to use the SF-36v2 standard single-item form for smartphone, and they use the same eCOA vendor for all studies. That’s a pretty discrete example, because there are some companies that are very large and they may use tablets, they may use smartphones, they may use many other forms.

Example two is a large pharma company with many divisions that uses several vendors and wish to use all of the standard and acute forms for tablet and smartphone with two of their preferred vendors. So a company like that may have very different questions than the first one.

And example three is a large pharma company that uses five different clinical outcome assessments from us. And they aren’t sure how many languages are needed yet. And they want sort of a choose your own adventure library option where they can get up to a certain number of forms, and they’re not sure what they are yet.

So that is the idea. So I’d like to know your thoughts, what do you think? Crazy idea? Could we do it? Would it help people?

[Q&A – 18:10]


Thanks Michelle. There was a question here at the back.


From the perspective of translation development during these trials, I think this is a fantastic idea, just because it is such a large pain point for sponsors, in the middle of translation development to hit these review steps that are critically important but can be so time consuming so close to beginning the trial. So I don’t have any answers to any of your questions right now, but I think there really great questions.


Okay. Well thank you.


Michelle, I think just on the topic of translations, you hit on the question about whether pharma companies are willing to share, maybe investments they make in a library or whatever between companies. But we see that exact model in translations, right, because you own the translation, you own the IP for your instrument. So Pfizer comes along and says, I want it in this language that you haven’t got yet. Well, Pfizer paid for it and then you own the translation. And that’s kind of how it works with most instrument authors. And that’s acceptable, you know, companies accept that. So why aren’t we doing the same with library versions and some of the evidence that we generate around migration validity and all of that stuff? We ought to be able to share that as an industry, even if perhaps one sponsor pays for one thing and somebody pays for something else, or a vendor pays for a certain bit, does it really matter. I’d like to see us moving more towards that approach. And then, when it comes to a library version, we can just have the library version; we don’t need a Pfizer library version, and an AstraZeneca library version, and somebody else’s. It’s just the version. From those who are in the pharma companies, is there an issue with this? Right, okay.



By the way, thanks Bill. I appreciate those comments. And I think, like I said, I say we and our, but I work in the science group so it makes me a bit uncomfortable sometimes, you know, talking that way about “our,” because I also would like for everyone to be able to share it, and that’s a goal.


I’m in data management in a sponsor company, and I work across all programs, so not all function areas work across all programs. And so I am always budget conscious. So if one program is using SF-36 and we’ve already paid to have it translated in 15 different languages, why is another program going to turn around with the same vendor and pay for the same translations? And no one is keeping track of that right now. So we could be using the same vendor, two different programs, using the same questionnaire, same version, and we’re paying twice.


Well, I would argue that that shouldn’t be the way it works, because according to the developer guidelines, the only licensable versions are the ones that we developed. So there shouldn’t be other vendors out making lots of—and traditionally we have used a vendor to do it, but we have all those developed. So I don’t know who is making bunches—are you talking about the programmed versions, or—




Oh, I was confusing it with actually developing the translations themselves. Okay.


But even so, even if it’s programmed already at the vendor and you’ve already paid for it, why would you pay for it again? Because we already paid, we already own that translation, that work. So there’s a lot of inefficiencies, and it could really push along the timelines if everybody knew what the other person was doing and you had a library for each program. In the last company I was at before I left, we were trying to get there. We had done some extensive work on that. And it was close, we were close to finding a vendor to help us do that.


Yes, well we have done it before, worked with eCOA vendors where, no matter who was the funder of the study, once it went in the library it stayed in the library. And that still is completely something that we’re willing to do. So it is just a matter of eCOA companies wanting to do that also, without receiving funding from a study sponsor to do it, I guess.


This is about sharing knowledge in some ways. And it’s a bit of a pipe dream from the sponsor standpoint. So it’s not just eCOA, but also instrument development, scoring algorithms, the whole shebang. So what I have often found is that whereas the HEOR PRO people are very willing to share the instruments across, it’s the legal, the brand teams, the legal firms, they’re the ones who stop you because—when I was at Novartis, I think it was Abbott, I’m not so sure—we were talking to each other, and they were willing to give me the instrument, we were going to go forward, but then the legal people came in and said no, because in case if our drug development fails for some reason, we could blame Abbott Instruments. Because people who go into legal professions go for a certain reason, I don’t know what they are. Sorry if any of your partners are in the legal profession. But they look at things from a very different angle. They think of all the scenarios why things could go wrong. And this is one of the things, and they said, we don’t want to—we’d rather spend money developing our own than borrowing from someone else. And when it’s the other way around, when we want to share it with somebody else, why do you want to give a competitor an advantage of two or three years while we have spent—I don’t know—half a million dollars developing something. I mean so that’s the general thing.



Well I want to thank Ari for putting out there some of the things that—what we’ve heard as we’ve talked with different groups about this—because I think that really is laying it all on the line of what the issue is in terms of sharing. And you’re right. Those are very difficult issues, and I do understand the business cases. And I suffer from being a scientist, so I still want to argue that we have to find a way to make this work. Because it doesn’t—I mean I don’t mind, I like that we ensure that these things are correct, but certainly our scientific group wants to tackle very deep problems, and form reviews aren’t necessarily one of them. So we would love to have more that are in a library and able to be reused over and over again and have that as something that is helping benefit patients in that way. So anything I can do, you can reach out to me and say, you know what, that’s a good idea but I don’t think this will work because of this, this, and this. And I especially want to hear from eCOA companies also. I’ve had several conversations but I want to hear from them also because I sort of do see myself as a conduit between all these different licensing, legal, science, vendor, and it’s not an easy thing to try to tackle. But I’m convinced that as mundane and boring as it might be, that I’m going to get it done. And people will share one day.


Michelle, I have another question, over here to your right.


Oh Willie, hey!


You said you provide a Word file, which is text strings basically that can be copied, right? Have you considered providing that in another format that can be ingested directly versus having somebody do a menu copy and paste? So I’ve been looking at this for the last six or seven years to build a library with some of the different vendors in that space. I have actually very high hopes with the [unclear 27:20] ODM format and probably embed it in a blockchain with smart contracts, where you guys don’t have to do anything anymore, you just cash your checks, basically.


Well I want to hear more about that. But yes, actually the eCOA companies can keep the library in any sort of format you want. That’s what we send out because different groups use different formatting tools, but I would be very interested in having us expand our options of what we could offer into using a tool like that as an option instead. We do--80-90% of our licenses are actually for academic and free in student, which means that they’re probably working with SurveyMonkey or something, and so the Word document seems to work the best. But we’re more than willing to get those things together for other approaches. So I’d love to talk more about that.


Michelle, thank you very much, that was excellent. Thank you.

[END AT 28:23]



Previous Video
eCOA and BYOD - Recent Research and Implications
eCOA and BYOD - Recent Research and Implications

Recent research in eCOA & BYOD by Willie Muehlhausen, eCOA Forum 2018.

Next Video
Enhancing Data Quality and Rater Reliability with eClinROs
Enhancing Data Quality and Rater Reliability with eClinROs

Dan DeBonis discusses the value of rater reliability with eClinROs - eCOA Forum.