Download Premium Content

First Name
Last Name
Country (Citizenship)
Opt-In to Future Emails
Thank you!
Error - something went wrong!

PRO Development Looking Toward the Future

May 20, 2015

Sessions Include:

  1. PRO Development in a Decentralized, Connected World (Shimon Rura - PatientsLikeMe)
  2. (00:22) Beyond Paper: Designing Instruments for an Electronic Platform (Paul Margerison - CRF Health)
  3. (00:43) eCOA in the Age of Rare Disease Research Proliferation (Jason Cole, PPD)

Panel Discussion: Jason Cole, Paul Margerison, Shimon Rura

Full Transcript


So as we’ve learned through all the sessions so far, there are a lot of pressures on PRO development. Looking beyond just issues of equivalence or electronic issues in particular, we know that this world of PRO development calls for us to involve patients at every step. It’s something we’re hearing in industry guidance and, you know, everybody pretty much agrees that’s the best practice. But that can be difficult. So let’s say you’re setting up a clinical trial and you need to find or perhaps consider making the right measure for your trial. This can be tough. So many of the measures that are out there right now are too narrow or too broad in terms of their topical focus. They might be out of date because of changes in technology or changes in culture, changes in the treatments that are available to the types of patients that you’re looking to work with. And you know, these measures sometimes just don’t keep pace with the needs of researchers. A good example is a measure that I’m familiar with in the autism spectrum space, that asks about how you organize your CD collection. And I’m guessing some of the younger people in the room today even have never had a CD collection. So you know, that makes me feel a little old, but it’s an issue and these things do need to adapt.

Now if you can’t find something out there you want to use, you might consider building your own. And as we’ve also heard that’s a major research effort. Of course you need to generate the evidence that your measure works before you can use that measure as evidence in your trial. It’s not guaranteed that you’ll succeed in this endeavor. It can be expensive. And you know, often you’re doing work that’s pretty duplicate of existing measures. And I’m guessing some of you have stories about your own company developing measures for the same thing more than once, let alone considering across the landscape of everyone working here. You know, how much are we investing our effort in doing stuff that’s mostly the same as what’s been done before as opposed to really broadening the scope of health experience that can be understood through patient reporting.

On the other hand, if you want to take an existing measure and evolve it, you know, build upon it so that it’s better and more suited to what you need, you might have some real issues getting clarity on the legal or contract kind of permissioning that’s required to do that. Some measures flat out don’t allow it. Some measures flat out do allow it, but there’s a big grey area where you have to, you know, phone up the person who wrote it and figure out a time that you can talk with him and convince him that your project has merit and other things like that. It also is hard to establish which of the data that support the original instrument are applicable in your new context and what needs to be reevaluated.

So wouldn’t it be nice if, when we did develop a new measure, we could cycle that measure as it developed with patients as many times as needed, if we could really streamline the logistics of the fielding and analysis of responses, and if we could really focus our work on areas of unmet need instead of doing the same redundant stuff over and over again. Wouldn’t it be nice if the measures that were out there were easy to evaluate, if they had clear license terms dictating who could use them and how, and if they were actually set up to encourage building on each other’s work. And wouldn’t it be nice if the people who looked for PRO measures out there could easily find the ones that were appropriate to their job or their task and guarantee that they’d be able to field those in a way that is both good in terms of the data that it yields and the patient experience of participating.

Well, I want to introduce the concept of the patient-powered research network, because I think it has some applicability to those challenges. The patient-powered research network, or PPRN, is a network of patients that’s really enabled by the internet, right, it’s a fact of how patients are finding each other and connecting online. And these networks meet some really fundamental needs for patients. So they can let patients connect with others and get some emotional support. They can learn from each other about what works or what doesn’t work, you know, different things that they’ve tried, whether treatments or coping strategies or even strategies for handling the financial or family consequences, dealing with a complex illness.


And this is a growing field. Some examples of patient research networks are PCORnet, which is the Patient Centered Outcome Research Institute’s network of I think 20-ish, maybe 30-ish, patient powered research networks, usually focused on specific diseases. 23andMe is obviously very genomics focused but it’s also a patient-powered research network in that patients are coming together, they’re contributing some data, they’re learning from each other, facilitating research. And of course PatientsLikeMe where I work.

And really the defining characteristic of a patient-powered research network in my view is that the patients who are members of the network have an ongoing relationship with the network. It’s not just that they sort of have a transactional relationship where they participate in a study, they maybe get their compensation or incentive, and they’re done. There is something about membership in the network that keeps them involved with it on a longitudinal basis. And that could be that the network provides some sort of tools or connections that are valuable to them on an ongoing basis. Or it could simply be that a patient wants to contribute to research and sees membership in a network as a way to support that.

So at PatientsLikeMe, you know, this is really one of the major sides of our business, right. We have this patient-powered research network that patients can join, they can connect with each other socially to get some of the emotional and social support that’s needed in dealing with a complex condition. But they can also do a lot of data tracking. And then the other side of our business is all the research that we do with the data contributed by patients.

So just to give you some picture of what value patients get from this sort of experience. Patients have said, “It was like someone handed me a fortune to actually meet another person with this horrible disease. I guess it was the realization that I’m not alone in this. I knew this intellectually I guess, but to actually meet another patient like me helped me feel less isolated.” Another member said, “I think my favorite part of the site is keeping up the charts so I can see what’s going on but also have something to bring to the doc’s.” That speaks to the tracking value. And a third said, “I was diagnosed very recently and there’s no effective treatment for this disease. There is very little research being done, and I thought that this may be an effective tool to begin changing those factors.” That’s a patient who really identifies with the research mission of the network.

We crank out a lot of research publications from this data. I think we’ve done about 55 articles in peer-reviewed scientific journals to date. We’re working with a lot of different collaborators across pharma and academia. And the kind of data that gets tracked—so we have about 300,000 patients who have signed up as part of PatientsLikeMe. They’ve logged data on 2500 different conditions. A patient can log in and track any condition they wish. Some conditions we have, you know, very detailed tracking tools for, others we have more generic ones. Across these conditions they’ve contributed 25 million structured data points of treatments, symptoms, tolerability issues, etc. Three million plus free text posts in forums and individual journals. And they use a number of different PROs that are built into the system, targeted to specific conditions where they make sense.

And then I want to talk about some of the research output. Other artifacts that we maintain include a patient-generated taxonomy, so you know, if you want to understand a concept like dyspnea that has a certain ICD-10 code and all that stuff, that’s not necessarily the way that a patient would describe their symptoms. So we have a team that actually takes data as input by patients and organizes it and codes it so that it’s actually meaningful in kind of the standard research terms, as well as approachable by the patients who are entering that data.

I’ve had dozens of client engagements around this work. We have a great safety monitoring platform that helps our pharma partners ensure that they’re meeting all of their adverse event reporting requirements. And some of the measures we’ve developed have been used as endpoints in clinical trials.

The data we capture includes very basic stuff—age and sex, the diseases that someone’s dealing with, symptoms they’re facing, severity status, changes in that over time. Treatments, including what treatments they’re on, dosages, tolerability issues, side effects, reasons that they might stop or switch treatments. Their quality of life and behavioral status, so just kind of generically across the platform we track those sorts of changes and fluctuations. Depending on the disease we’ll have PRO measures of that particular disease. And then of course there’s the unstructured conversations that happen in forums and in journals.


So how does this look to someone on the network. So this could be your own profile or a profile of another patient, that you see on PatientsLikeMe. And it’s actually, this is a chart of a patient, so it’s looking over a certain period of time. And what you see at the top—this is a psoriasis patient—and the top is a  PRO chart for that patient. This is the dermatological life quality index. So I know it’s a little hard to see up here but that’s tracking how the patient has scored. They’re prompted by the site on a regular basis to fill out this PRO, and they’re seeing how their status has changed. And they’re seeing this on the same chart page and same time scale as variations they’ve logged in their symptoms or changes in their treatments. So if for instance a treatment really helps and reduces the severity of your disease, it’s something you might be able to see on this type of chart. And you could see this both for yourself and for any other patient who’s logged this data. When a patient joined PatientsLikeMe, they get access to what everybody else has logged, you know, both on an individual basis of finding a specific PatientLikeMe that I might learn from, and through aggregate reports about, say, how psoriasis patients are treating their condition.

So in a world where you have these patient-powered research networks, how would you approach building and testing PRO measures? So one really obvious thing you can do with patient-powered research networks is rapidly access very targeted cohorts of patients. So what does that mean? Well it means that compared to a clinical trial where you need to gradually enrol patients, where your studies have sites that maybe see—let’s say they see five or ten patients a day. It’s going to take time to collect responses and actually the development of your PRO measure is going to be dominated by the amount of time it takes to get patients in the door, field the measure to them, collect those results. With a PPRM where you have patients that can easily just dispatch an electronic study participation invitation to at any time, you can take something that might take you—you know, say it takes six months to get 500 responses in a clinical site—you could easily get that overnight on a patient powered research network with the right patients. And what that means is that you can kind of flip the process of development. So instead of being limited by the patient fielding time, you’re limited by the time it takes the research team to kind of absorb the input of patients and develop the next iteration of the questionnaire. And what that means is it might take away some of the pressure to get the instrument as perfect as possible, you know, without talking to patients before you kick off this big multi-month fielding process. So you can maybe do several rounds of cognitive debriefing. Or you can do maybe different targets where you explore how it performs in certain populations. And the hope is that, if you can do more of those iterations you can essentially mature your instrument further in a shorter period of time.

Another option is that you can develop some new research methods that let you really rapidly gather feedback at scale. So we talked a little bit about cognitive debriefing. Cognitive debriefing is a very expensive endeavour, right, you’re sitting down with a patient, have a trained interviewer. You’re going to go through a questionnaire, you’re going to probe on specific issues that come up in the response flow. That’s a very powerful technique. So is there something that’s kind of an analogue to that online? And our system actually does some of this, so you can actually field a PRO item and field some feedback questions about that item. So was it understandable, was it easy to choose a response option. This is clearly a very different thing from a typical cognitive interview. But it’s possible that it can get at some of the same issues. And if you compare those two methods, the in-person cognitive interview is something that you might be able to do with 10 or 15 patients, whereas an online version you might be able to do 100 or 500 patients in a similar amount of time. So really it lets you try some different approaches.

Another thing that this online world causes us to question is, can we apply different collaboration models to the world of PRO development. So this is where I think we can take a big lesson from open source software, where we can give people some great collaboration tools to work together online. So maybe we can do better than emailing around 3000 versions of an Excel document, which I’m sure many of you are familiar with. We can really open some of the licensing. So not only make it clear how an instrument can be used, but actually encourage people to build on each other’s work. And maybe we can include people who are not just the traditional measure development experts in the process a little more. So people who are subject matter experts in the condition, whether they’re patients or clinicians, can maybe have an increased role in the development of the measures.


So that’s what we tried to do with Open Research Exchange. And Open Research Exchange is a project we started at PatientsLikeMe in 2013 with support from the Robert Wood Johnson Foundation. And it’s really an attempt to offer a new way to develop and test PROs with patients. This is what it looks like, it’s at openresearchexchange.com. It focuses on tools to help you design your instrument, so you can actually create your questionnaire, you can create studies, including defining who’s eligible, and what different sort of pathways through the questionnaire might be available. When you have that study identified, you can actually run that study with the PatientsLikeMe user community. And that’s always an exciting part of the process for researchers, because all the study results come in live into the Open Research Exchange tool. So a researcher will usually be up at night at home hitting the refresh button on their web browser watching the response count count off and looking at the data as it comes in. And we encourage you to publish your instrument on the Open Research Exchange library, where anyone can come in and search for an instrument that’s been developed there, look at its history and if the licensing allows it, even pull that instrument into their project to create a new version of it.

Over the past couple of years we’ve done about a dozen pilot projects on this platform. I won’t go into the details of every one of them, but that has ranged from very generic instruments such as the burden of treatment for patients, understanding that, all the way down to specific things like hypertension self management. We’ve got a fantastic scientific advisory board as well. You all know Ari, he’s been helping us with that. And it’s really a group of people from across the world of PRO development who’ve really helped us ensure that our research output is top quality. We’ve also kind of expanded in the past year to include some people who can help us not only do goo research but help  make sure that the instruments we’re developing are aligned with the needs in the world of care and policy and impacting patients there.

So yeah, we started out with the initial build of this platform in 2013. Did three PROs on the platform at that point, and 2014 did ten more. One of those is actually patient led, where we put out a call for patients who wanted to measure something better, and got a great patient leader with MS who was interested in pain measurement. And we gave her a lot of technical and scientific support to actually execute that project. Done a lot of scientific publications that are starting to come out on those projects done on the platform. We developed some educational content about PROs and their applications. We started to look at uptake, so that’s where these PROs are used beyond just the world of research. We’ve made the platform kind of production ready and open. We’re doing a number of commercial projects on it as well. And last but not least, we started a great patient advisory board to advise us on how to best involve patients in the process.

So what’s next in PPRN-enabled PROs? So you know, this is me kind of looking into my crystal ball and I think we’ve talked a lot about new directions of taking PRO usage as well as development in this conference. One is just continuing new measure development with PPRNs, I think we’re going to increasingly studies leveraging these groups. And you know, this is both good for the scientists and good for the patients, because patients want to participant in improving the understanding of their disease. Even though in some cases they know it’s not going to benefit them directly, they see the value of this. Second, we definitely need to do some methodological comparisons. Certainly the strength of these online methods come from some of the ways that they challenge the normal ways of doing things. But we all know how beholden we are in this industry to kind of expectations of traditional standards. We just had a whole hour and a half discussion about paper and electronic formats. So you know, that’s going to be something that we need to figure out together.


Third and you know here I’m speculating a little more. I think we need to think about how we develop more holistic measurement models for health and item banks that support that. Because I think there’s a lot of duplicate effort in what we measure, and the more we can hone in on a core set of measures that, maybe they’re vital signs, maybe they’re disease specific, you know, I think the more we can standardize our measurement tools, the more data that we’re going to have behind those tools, and the more meaningful all of our measurements and data are going to be.

I also think we need to think about bridging different contexts of measurement. And this is kind of actually driven by the patient perspective. So for a patient who is living with a complex condition, you know, if they’re going to have their health tracked in a really meaningful way, in a clinical trial, and then that trial ends and they go back to having just their doctor check in with how are you doing every six months, that’s a real loss to a patient. If there’s a tracking tool that makes sense in a trial, can it make sense for self management, can it make sense in clinical care, can it actually make sense even at a population health level. Now of course there are going to be differences between these applications, and I don’t mean to say that the same instrument can be used in all of these contexts without any changes. But these things need to connect, and patients want them to connect.

And finally I think the other thing that PPRNs can really help us do is better align the research work that we’re doing to patient priorities. So we now have really powerful ways of identifying what’s important to patients, tracking that as it changes, doing it at scale, reaching big populations, and that should really help guide us in terms of where we invest our research effort.

Thank you.


So as Jill said, I’m what’s referred to as a user experience designer. Designer, generally. So I’d like to take a sort of design historical perspective on where we are with ePRO in the market, looking at what we refer to as ePRO, a sequence of questions and potential answers, as an object, as a thing which is designed for people to interact with and use. Where is it in its design evolution.

These are the things that I want to talk about. We’ve heard a bit about the gold standard—this phrase gold standard already in the earlier session. So the phrase seems to have some currency, if you pardon the pun. Where are we with the connection conceptually when we design a new study, design a PRO measure, where are we in that relationship. And going on to think about what might happen if that relationship just lost its relevance or faded away. That’s where I can fantasize a bit about where we might be going actually, do a whirlwind tour of the possibilities. But coming back then onto what our relationship might be, not just with paper methods but with the wider design industry, with consumer electronics and the design of screen-based content generally, because there’s some interesting parallels.

So this gold standard idea, the idea of—as we’ve been talking about today—the idea of when you come out to design a new instrument, do you need to think of it first of all as something that could be put onto paper, and if you do, then are you subjecting yourself to limitations that you needn’t be.


Just before going onto that I want to make a statement of what I feel about PROs. PROs, not e- or anything else, PROs generally. Because I’m quite new to the industry, I come from different commercial sectors, having worked in other commercial sectors. And I think the idea of a PRO is actually really quite an amazing thing. The idea of taking something so subjective as how do you feel, I feel fine. That I feel fine means something different to everybody in the world. And when we think what a blunt instrument language itself can be—and it’s interesting to see what Jason was saying about equivalence in translation—language is a very blunt instrument. You can struggle to say what’s really going on inside that head. And to be able to turn that into a measure that you can base something as important and potentially dangerous as a medical treatment, is quite an amazing thing. So it really doesn’t surprise me to hear that there can be a defensiveness amongst owners of PRO measures to change anything. You know, if something has trod that fine line of getting actually meaning out of such a subjective thing and turned it into a measurable item, it’s tempting to stick with it if it works, even if it works imperfectly.

Here are a couple of pie charts. We’ve heard some numbers about the potential growth of electronic media. These are COA rather than PRO. I think the two we can assume that their relationship is very close. Figures from a consulting company called LEK that suggest quite a strong growth. Last year, around 7% of all started trials used eCOA, and the expectation, the forecast, is that by 2018, 20%. So very strong growth. And I guess the first answer to the question where PRO is going, would be towards electronic, that’s a fairly uncontroversial statement. But secondly I’d pick out from those pie charts that even if the forecast is right, paper is still pretty strong, 80% in three years’ time. Somebody might argue that that’s still where the action is, and so should be loath to dismiss it so freely, and I think that there’s a general expectation in the room that electronic media work better, which is an opinion that I would share. But if you have a team which is used to working in a certain way and that has a track record of getting things right and it’s achieved a lot of label claims and so on through a method that it knows very well, then it’s not surprising that the shift can be quite slow. Also, from a design perspective it’s not surprising that we get a close link between the appearance and the feel of an electronic PRO measure and what came before, which was the paper version of the same thing.

This was a nice illustration of an engraving of Ransom Eli Olds and his first Oldsmobile. I wanted to illustrate what happens very frequently in the designing of new products. They look like the old ones, their appearance mimics what came before. And that’s just the way that things happen, the way that things get designed. If you put a horse onto the front of this thing, it looks complete. And without one it doesn’t. And I feel this is happening with PRO and ePRO as well. One illustration Paul O’Donohoe showed us a couple of measures earlier on, there was a paper version and its electronic variant, but if you Photoshopped away the tablet from the electronic variant, you wouldn’t really have been able to tell the difference very easily, which probably feels quite natural to us now.

Actually there are possibilities to break away from that model of presenting those very same questions and do it in such a way that’s starting to feel more natural to the way people consume their content today. But at the time, this was the natural way to begin, the natural way to start thinking about a new medium.


So I’ve got another metaphor from the early automobile industry. You probably know about this, from the 1860s onwards in the United Kingdom, and also in Vermont apparently. You had to drive your automobile preceded by a gentleman waving a red flag, to warn everybody else. And you know, it is comical, but it made a lot of sense at the time. Other road users did things in a certain way and they had behaviours that would be maybe taken by surprise by this monstrosity coming. So from that perspective, it was obvious that this early type of automobile that Olds and others were inventing, it’s obvious that that design was transitory now. But it’s not as obvious to our contemporary eyes that when we look at an ePRO measure, it’s not as obvious to us that that’s a transitory state. But it probably is. So it’s very interesting for us to think about what might well be coming next. And I think this is going to be a period of great invention for us.

This is where I’m going to rapidly—probably indecently rapidly—go through the possibilities because there are so many of them. And forgive my drawings. I wanted to show you also that I am actually very much in favour of low-tech solutions as well sometimes, when they’re necessary. Straightaway, moving to electronic, there are some benefits which you immediately get, like the ability to collect data in some quite challenging environments. Probably not in the bathtub. Jill pointed out to me that if you drop your tablet into the bath, it can be just as bad as paper, but still in the bathroom, a tablet or phone actually works better than a piece of paper which gets wet and you need a surface to write upon. So straightaway, things like that. Other challenging environments might be darkness. You can fill in a diary in the dark if you’re using an electronic device. The book with a padlock around it is just meant to emphasize the point that with an electronic diary you get assurance of privacy. So you might be able to design and write your instruments knowing that your patient is assured of privacy, and that might change the way that you write them.

But moving on a little bit, the more spectacular advances are, well, a PRO tends to be some questions and possible answers written down. But why not pose the questions in video or animation or through sound supplementing the words, or through drama or through music, or through high resolution imagery. Of course with high-res images you can ask different questions—does this look like the symptom that you’re experiencing.

The electronic media are also considered already to be assistive devices, that’s to say for certain conditions you may have, they’re actually easier to use than paper. If you see somebody entering data through their knuckles, similarly in the internet design world, you would naturally design something to be able to flip the contrast, so dark text on light or light text on dark for whichever is suitable to your vision, and of course to increase the font size if you want to. And those things do involve a slight difference in the way that you would actually write the measure in the first place.

PRO measures could well become personalized, know you personally without actually sending the personal information like the name Bob through to the study centre. It’s very easy to create a secure wall between the two. And in a similar way, involve a patient in the trial itself by replaying their own personal data back to them and allowing them to feel an active part of their own care, which the device even itself can be, and I think in certain circumstances it already is. you know, you can set thresholds above and below which the study centre is triggered to make contact with the patient. There are lots of possibilities that we could invent, we could explore, through the device itself being a part of the patient care. Or, as Shimon hinted, other patients being a part of the patient care. So this illustration is supposed to show one person offering emotional support to another person who is involved in the same trial, implicitly by means of social media or some other controlled social channel. You couldn’t do that with paper very easily.


Nor could you do this. You can’t have the branching logic, the fabulously complex branching logic that you can have with electronic devices, that allow you to write a questionnaire differently, without the patient even realizing that there’s this branching logic.

We have an electronics industry that’s already working for us to produce things that people naturally carry around with them, and—I’ve invented this quote: “I wouldn’t feel right leaving the house without my ePRO device.” Well it’s getting that way, and if people are carrying their ePRO device everywhere they go, then we’re getting closer to the clinical events if they should happen with the patient is out and about. Similarly with all of these wearables, there are many on the market. Some of them are consumer products and some of them are medical devices. All of them can be very easily made to communicate with a study centre. 

Moving out a little bit more widely, carrying devices around with you, and the cheapness of sensing devices these days, means that you can conceivably measure all sorts of environmental measures like temperature, sun, ambient noise, soon even pollen count, and correlate that with the patient reporting answers to your questions. And wider still, I won’t go into this very much because it really is probably next year’s theme, to blend PRO results with data that’s available outside in society everywhere, where you might be able to uncover relationships that are truly astonishing. This is a kind of comical example, it’s a crowd of people answering a question on their taste in cheese differently, depending on what the weather is like, some findings like the are happening in big data analysis.

That was very rapid, a very rapid run through of what might happen. But the point about all of that really was that you need to be working in the new medium in order to find out what the new medium can do. This you might recognize, it’s the very lovely Lillian Gish. She starred in a lot of films by DW Griffiths. And Griffiths was one of those filmmakers who had the brilliant idea of focusing in on a detail like a person’s face. So before Griffith and some of his contemporaries, the close-up was not known, so if you were filming some drama, you take a camera and put it in the front of the stage, the actors would walk in front of it and they’d walk off again, and effectively you had a reproduction of the previous format which was the theatre play. But Griffith had this brilliant idea, and we can’t imagine cinema without it anymore.

I want to finish up by suggesting that we do have a connection to the electronic publishing industry that might be a little bit uncomfortable. Electronic publishing doesn’t tend to be about such a weighty subject such as a subject with as much risk attached to it as PRO issues. But that relationship isn't necessarily an uncomfortable one. If we were talking about a year ago, we might be looking at the possibility that this, Google Glass, which is an ingenious way of putting data onto the real world that you’re looking at, recording the real world as you walk about in it, and it didn’t take off, it kind of bombed. And this is not a criticism of Google, because I’m very impressed that they had the goal with that. And it will find a different market anyway. But not ours, not yet anyway. So we should be quite relieved with the fact that we have an electronics industry that is doing this, trying to do the same things that we want, which is affordability, reliability, good user experience, things which create desirability of an object, and we like all of those, and we like to have those on the devices that carry our questionnaires around. So luckily we have an industry that’s several steps ahead of us in developing those things.

As I said I worked in other sectors designing information on screen and so when we talk about a screen-based publishing industry, if somebody went to sleep about five years ago, maybe eight years ago, and woke up today to ask what are the things that have been really hot in electronic publishing for about the last eight years or so, please because I’ve been asleep, these are probably they. So “mobile first” is a mantra that all designers repeat now before they go to bed. That anything you design, first of all design it so that you can view it very comfortably on a mobile device, on a phone. Separating content and platform, that’s to say your consumer might start reading your article on a desktop at work and then carry on on the phone and then continue on to some other device when they get home. So the content and the platform should be separate from each other.


Social content. You make the content social and new types of content that are truly useful and new for consumers. And the last of the bullet points is a good bit older than eight years but it’s still very important. User-centered designed, that’s putting the user right at the heart of the design process. And thereby changing the thing you design.

So that’s the wider electronic publishing industry. And then if I look at what seem to be the buzzwords today for us, abandoning the gold standard of paper seems to be a buzzword—not a buzzword but an important current theme, which resonates very strongly—it’s very similar to the idea of mobile first. It needs design for electronic first, if you an also show it on paper, so much the better, but really you need to design for electronic. That's the mantra of mobile first. Separating content and platform, that’s a restating of bring your own device, that’s all of the equivalence debates that we’ve been having, that’s the last five years of debate in electronic publishing too. Social media is set to be a really hot topic for us and I’m really excited about where we might go in this industry with social media. Could be an interesting time. And the last one. Who is tired of hearing the word patient centricity, it comes up all of the time. I’m not tired of it, because it does seem to be a restating of the very important principle of the way you design things, putting the end user of the thing into the design process.

So summing up, these are the three things I think we need to seriously understand our position regarding how we design a new measure—is it for paper, does paper have to have any say in it. Secondly, that any new medium brings new ways of using it that you don’t know about until you start working with them. And some of the new properties are fantastic. And thirdly, that you want to know where we’re going in terms of which platforms and which devices we’ll be using to put our PRO measures on, we can do worse than look at electronic publishing and learn the lessons—sometimes quite harsh and hard lessons that they’ve had.

Thank you very much.


What I want to talk to you about today—and I’m a walker too, hence the taking off the microphone—is not solely related to the e- part of the COA, but more broadly to COA, and what we’re going to be doing with rare diseases. Rare diseases are something that caught my eye about five years ago when I started working with Otsuka on ADPKD, and polycystic kidney disease broadly. And they had approval for rare disease, etc. Now this is a fun rare disease, because in all said, we published on some papers where we got to do over 250 patient interviews etc. so not your typical rare disease. Some of the rare diseases I’ve looked at were folks who have come to us and said, hey can you help us with the psychometrics, have had 23 patients enrolled in their pivotal trial. Well this completely throws out our standard process of patient-reported outcomes development and validation. Yet, a lot of these diseases are ones that are very amenable to having a patient-reported outcome. And in fact, in this new age of PRO specialists actually having to go, wait I think you have too many PROs in your study, I can’t believe I’m saying that, we’re finding folks are interested in including a patient-reported outcome in their study.

So I want to set the landscape a little bit first with a few of the slides that I have and then talk about some of the issues that are coming up.


So 2014 was one of the biggest years that the FDA had for new drug approvals. There were 41. And oncology, which tends to dominate, only had 19.5% of that. Rare diseases however had 17. Now some of those were oncology by the way. So there is a little bit of overlap there. But 41.5% were for rare diseases. That’s huge. And this reflects what I’ve seen as somebody on the CRO side and somebody on the consultant side, that we’re getting more and more requests to talk about rare diseases. Why? Big diseases are tackled. If a pharma company wants to make lots of money and go after a big disease, we’ve largely hit that. Now are there new things that we can do there? Of course. But the rare diseases offer a lot of niceties about them as well, and we’ll talk a little bit about that. But just briefly.

So a rare disease is something that affects 200,000 or fewer Americans. Now this was of course the FDA definition. But this is how you get the rare disease designation for the FDA. And the drugs that were released, those 17 are listed right here for you. I’ve been told by the way that the slide decks will be disseminated to all of the participants, so you’ll have this information here. And by the way that’s also very important when I get to the end of my slide deck and you see a big blank slide. We’ll go into that too.

Okay. So what were some of the key highlights? One of the things to note was the average number of subjects. And you can tell there are definitely two outliers here. Those stand up pretty ominously there of getting over 1000 subjects. And so hence our mean and median are quite different from one another. Our mean of 283 and our median of 173. The range, 26 subjects to 1334. I want to clarify that 26-subject, actually it was one drug that received three different indications, and only one of those indications had 26. But the next smallest was I think in the 30s. So again, very small. I know that there were a couple more in the 40 range. So you can see here that they vary out, and there’s a couple of really small ones on there.

Of these, I went through and looked at the label language, not looking at the details yet inside of clincialtrials.gov etc. But in the label language itself, I saw 2.1 of these studies listed a COA. Why .1? it was a six-minute walk test. People love to claim that that’s some kind of patient-reported outcome. And I heard somebody use a term here, like a performance outcome or something like that. Maybe. So I gave it the .1 designation. But two of them were actually very key. So 2.1, that’s a little small from what we might think. But if you think about some of these rare diseases, they’re very small subsets of cancer patients dealing with a specific molecular issue, patients don’t feel anything in these diseases like Ari talked about yesterday. So the studies themselves right now that are coming out in rare diseases, I think they’re the first onslaught of a big wave, and if the clients that I’ve spoken to over the last few years are an indication, I would expect the percent of studies that have patient-reported outcomes in their label to go up. Because they’re not just talking to us about hey, we want a PRO in for publication, which is a viable pursuit, but that they want it for a label claim. And sometimes we have to talk them off the ledge, but there’s a really interesting conversation with the FDA for these orphan drugs. And so I’ll get into that in a little bit here too.

How much of the rare disease space is left? A lot. So I think that this is one of the exciting pieces about rare disease. If we can place a lot of faith in this finding of only 5% of rare diseases have FDA-approved medicines, then we’re talking about a whole bunch of stuff that we can still go after here and help patients with. When we go into these rare disease areas, we know that there’s an appreciable patient burden typically with these diseases. They’re often genetic, and so you know, there can be very severe patient burden. Often, when they’re genetic too, they’re affecting children, which makes it a little more difficult but yet a little more involved for the patient advocacy groups to be involved and the caretaker advocacy groups to be involved.


And there’s an important issue here for observer-reported outcomes, usually from the parent or caregiver, but sometimes from a skilled nurse, physician, etc. So there may be an opportunity to develop a few different tools. When we were looking with Otsuka they noted that they needed an ADPKD instrument for adolescents as well as one for adults. And since they wanted to go younger we know that there was some cognitive difficulties there. My background is actually in intelligence testing, and we know that as the cognitive processes of a person develop, they start to differentiate concepts more readily so you get smaller big clusters in younger children, a few more differentiated clusters of domains in adolescence, and then by the time they get to adults they have more differentiation.

Okay. So what does this mean for development. Well first of all we know of course the standard approach won’t work. Why? The qualitative parts, we know they’re going to be difficult because we usually are not going to be able to go out and do four to six focus groups at a minimum or you know, eight to ten subjects in each of those. We’re going to find 100 subjects in a disease where the clinical trial has 23? I don’t think so. So you know, we’ve got to be able to think about what does that mean here. And adaptation is going to be the key. I’ve got some recommendations in the next section about how we might be able to do some of these things. The quantitative data collection, if you want to think about Cronbach’s alpha unto itself, in order to get a stable Cronbach’s alpha—and I’m sure all of you guys are reporting standard errors on your Cronbach’s alpha, right? Anybody? Ari said yes, excellent. So you need about 200 subjects. Obviously this is somewhat item dependent, but it’s item dependent on the reduction of your standard error, not item dependent on the fact that we know that the number of items you add influences your alpha level. So that’s the simple approach. But if we wanted to think about more sophisticated techniques that many of us are using now in establishing validity, we’re talking about structural equation modelling confirmatory factor analysis, and we’re talking about item response theory. A great thing about Rasch is, according to Ben Wright, we need like 20 subjects. Probably  a little more is necessary than 20 subjects but you know Ben was a happy guy, he liked to think positively. So you’re probably going to need 60. You know, 60 is a magic number in statistics, if you want to talk about the stability of large numbers, you tend to get it around 60. But that’s a total minimum. If you want a 2PL model like a lot of us really think is good, then what you want to think about is numbers that are probably in the 200 range. Confirmatory factor analysis has something called the NQ hypothesis. This is based on the number of free parameters in your model you want 10 subjects per. A simpler way of thinking about this is, you want 20 subject per item. Now you could have some complexity to you model that might change that a little bit, but you want 20 subjects per item is really what it comes down to. Until you hit about 300, then you’re probably good no matter what you do. So there’s all these complexities to how you get it. Any of those numbers sound reasonable for a very small scale study? Probably not. Okay.

So when I looked at the number of studies in the 2014 approvals, based on how many subjects that they had, you can see that the greatest number was in those less than 150, which basically blew all of the psychometric criteria away. We don’t have the numbers to get stable psychometrics for a lot of what we want to get at here. Even Cronbach’s alpha is probably going to be insufficient but it depends, you could find items that are just really happy together and you get that stability. But you’re probably not going to be able to do your 2PL model, you’re probably not going to be able to do your Rasch model, unless you’re talking about a five-item questionnaire, which may be completely fine. But if you’re looking at something more aggressive or the domain structure just has to have several domains with several items, then you’re going to be in trouble.


So how do we tailor this here? What do we do for rare diseases? Well one of the things that I noted earlier on was adaptation. And I think that adaptation is going to be a theme through all of this, including how you interact with the FDA and the EMA. I am not an FDA spokesperson, so take everything I say please with a grain of salt and don’t quote me. It’s a good thing this isn’t being recorded. Okay, so realistically the FDA and EMA are happy to talk to you about rare diseases and altering your path forward. We’ve seen the FDA be more dare I say liberal here than in almost anything else I’ve talked to them about. They tend to be very willing to accept that there are just no paths forward in using standard processes for PRO development when your total sample in the world is 500. And so you need to think about this differently. And if you can convince them that still getting the message from patients is really important, which they tend to be more than willing to accept, then you can have a good conversation with them. I believe more and more, unlike I did ten years ago, that the mantra of early and often is totally appropriate with the FDA anymore. And I think that that really applies here. Speak to them early, speak to them often about what you want to do and what your concerns are for the roadblocks, and what your plans are to overcome those to try to get them onboard.

Qualitative validation is still quintessential. We know that when we talk about the pendulum of evidence for the FDA and we have our qualitative here and our quantitative here, we’re over here. They are very much in favor of the qualitative data so you’ve got to get good rich data. But you really need to consider issues. Are you going to have to conduct exclusively IDIs. by the way I’m a fan of focus groups and IDIs, I think that both have their merits. But you may have to do 100% IDIs, you may have to do them in the patients’ home, in a hospital setting, etc. Your rare diseases are going to give you more obstacles, and the more you’re willing to adapt to say oh no I’ve got to do it in this facility and I’ve got to do it in this country, etc. There was one of the studies in 2014, one of the drugs, was approved on a study conducted exclusively in India. This was approved by the FDA. Exclusively in India, because that’s where there was a bolus of these rare patients. So we’ve got to understand where our patients are at and be wiling to go there to try to get that information from them as best as possible. Now of course if you can have geographic diversity that’s great. But understand that ahead of time what your limitations are going to be. 

Does psychometric validation processes, the key here is to establish what are the core elements that you can get at. Can you get at consistency among the items. There are lots of different ways to do this. Some of them are much more flexible. They need to use two or three different statistics in order to get at each of these elements. The reliability of the item to each other. The stability of your scores over time—test retest reliability. That’s one of the ones that’s actually more favourable. Even though you probably want at least about 60 subjects for this, you still, this is such an important statistic especially for the FDA, and yet it’s probably one of the easier ones to meet. Then what can you do about understanding the constructs themselves, you’re not going to be able to do confirmatory factor analysis in most of these. Don’t waste your time with exploratory factor analysis. Come talk to me over wine about that one sometime. But you can do things like correlating your scores with no measures and getting some appreciation of do you have the right constructs, are you measuring what you think you're measuring here. So come up with some flexibility in your plan. And just remember that non-perimetric statistics are your friend in this opportunity. I didn’t want to give a prescribed set here because there are just way too many options, and every time you can increase your data just a little bit, you really help yourself out.


You know, speaking of that by the way, there’s some statistical thresholds to think about—30 a big number; 60 a big number; 100 a big number; 150 a big number. So if somebody tells you, boy there’s no way we can get more than about 25, push them to get the extra five. I realize that if you think about the actual difference, it probably seems weird, but good things happen at those threshold numbers. So you know, if you can, go for it.

What about eCOA? So that’s what we’re all here for primarily. So the broad simple approach is, if you have a diary, you kind of have to—and it’s a core measure—you’re pretty much going to have to use eCOA anyways. To use an IVR which would also apply for the FDA regulations, I hate the auditory processing component of those. We’ve touched upon the IVR stuff today and why there are some differences. Auditory processing versus visual processing, huge difference. So I think that realistically you kind of get a green card here in that way. But if not, is eCOA going to be beneficial. Well there’s some simple sample size rules that say that well you know you can think about maybe if you had 200 subjects is a magic number. Probably not going to apply here because in most of our studies that we had in 2014, we didn't have 200 subjects. So you know, how can we get the right number? Well there’s this great paper out here that lists 12 different things up here that you can calculate. And I know that actually CRF will help you look at a return on investment calculation based on a bunch of different information that they have as well. And that’s part of my missing part of my slides right here, we’ll get into that. But basically you’re probably at a point where you’re going to have a hard time justifying this eCOA approach for all of your studies. So you may have to consider how many times you’re collecting it, how valuable those data to be accurate, can you live with paper, and this is not an easy decision, you know, you’ve got strong advocates on the eCOA side here, and I understand why. But you’re not always going to be able to do it in rare diseases.

So what was going to be on this slide were about three different tables where I applied some simple yes/no decisions based on the number of PROs, the number of subjects, and the number of data points for your study. There are more that go into it, but it’s a nice little way of looking at it and saying, am I in the ballpark here or not. And those slides will be included and will be disseminated to everybody, so I’m sorry we didn't have those ready for this, but I think that it’ll be a nice quick way to look up some information. But of course, if you really want to know if that ROI is there to justify it, work with somebody like CRF who can help you calculate it on all of your variables.

And that’s all I have. Thank you very much.

[END AT 01:03:00]

Previous Video
Panel Discussion: Emuella Flood, Ari Gnanasakthy, Michelle White
Panel Discussion: Emuella Flood, Ari Gnanasakthy, Michelle White

Next Video
PROs and Your Clinical Trial: Choosing, Administering, and Claiming
PROs and Your Clinical Trial: Choosing, Administering, and Claiming