Download Premium Content

First Name
Last Name
Country (Citizenship)
Opt-In to Future Emails
Thank you!
Error - something went wrong!

Beyond Paper PRO: Objective Data and Ecological Momentary Assessment

November 20, 2015

While electronic data capture is quickly establishing itself as the gold-standard method of collecting clinical outcome assessments in clinical trials, the actual questionnaires being deployed seem to be stuck firmly in the age of paper. How can eCOA take advantage of the exponential growth in technology with proactive data capture methods to drive insight into the true patient experience?

Full Transcript


Okay, so coming to the end of day two, we’ve had a lot of presentations sharing data, sharing findings from research studies. Giving a presentation myself yesterday. You’ve seen a lot of this in the workshops over the last two days. So I’ve decided to take this opportunity to kind of let my more whimsical scientific side come out, and present maybe a more kind of theoretical musing around what we might want to consider or an area we might want to explore moving forward with patient-reported outcomes. I want to take a look at kind of where we are now, a slightly slanted look at where we are now, where we might want to go, and potentially tie that into what the future of clinical trials could look like, or certainly an aspect of that future. And I want to do that by looking at three distinct but I think quite closely related—or at least I’m going to try and make them closely related—parts. That’s looking at maybe a different methodology for accessing the actual patient experience, which is really what we’ve been trying to do with patient-reported outcomes this whole time. And touch on something we’ve already heard quite a bit of—and Willie gave us a very good summary there—around sources of objective data and what that might be able to tell us about the patient experience, which is not something we've kind of put together in this field before. And also, kind of a more broad topic about bringing real-world, for want of a better word, data into the clinical investigation and what that might do, how that might change how we potentially view at least aspects of clinical trials. So let’s see how I get on with that.

Traditional patient-reported outcomes, kind of, in summary, they just tend to be pretty boring lists of questions. They’re not particularly exciting looking, they’re pretty straightforward, they just sit there, you respond to them. Occasionally you might include a picture, which can get quite exciting. Again, this body map comes up. Occasionally you might include loads of pictures, which can be quite engaging and colourful. When it comes to making electronic versions of those questionnaires, very often we find ourselves following the somewhat uninspiring source, so again we just tend to have representations of the paper version, which is another list of questions. That can look very similar to the original paper version. Sometimes when we get on to handheld implementations, things to one extent almost improve because things are simplified, you’re forced to only have one item per screen, the user experience is arguably maybe a bit better because they just focus on that one question. But it’s still just a question that’s kind of sitting there waiting for the patient to respond.

And I was amused to hear Willie bemoan this whole just-like-paper issue, which is not actually something we planned. But this is one of my least favourite phrases in this field, when you’re talking with sponsors or you’re talking with an instrument owner, who says just make it look just like the paper original version. And I can understand where this is coming from, I think it’s really driven by this fear of impacting the original properties of the questionnaire. Some of you heard me maybe having a mini rant yesterday about the fact that, you know, we’ve spent a lot of time as an industry getting to the point where the patient perspective, the patient experience, was key. We needed to understand the patient experience to get a proper understanding of the treatment and the treatment benefit. And so all this work went into developing patient-reported outcomes as a hard science. And it is a hard science, for many different definitions of hard. But it involves very rigorous process of making something reliable, making something valid, making something applicable. And so I understand why instrument owners and sponsors are reluctant to do anything that might impact that, and we already had a bit of a discussion yesterday around BYOD and the effect that might have on this issue.

So I understand why people push this idea of just like paper. But again, as you heard me reference yesterday and as Willie touched on, we’ve already done a lot of work exploring the paper-to-electronic comparability, and if you kind of look at the literature when you’re developing a brand new questionnaire, whether on paper or electronic platform, and you’re establishing its test-retest capability, so you’re just comparing how participants whose state haven’t changed, how they respond to the questionnaire at two different points, the assumption being if it's a good questionnaire they’ll respond the same, assuming that their state hasn’t changed. That’s the sign of a well-developed questionnaire. The kind of acceptable correlation for that at the moment is anything over .70, whereas in the paper to electronic comparisons that as we said we’ve done enough of them now that we have two meta-analyses out of them, typically we’re seeing high .80s, low .90s in the ICCs. So typically quite a bit higher than what we’re seeing in the test-retest of the development of original questionnaires.


So I don’t think it's too controversial to say that we’re not impacting the properties of the questionnaire when we move from the paper to electronic platform, even when we’re going to handheld, which is a big change from an A4 piece of paper, and even when we’re going to IVRS, which is just a whole different modality. So I don’t think we can make this argument anymore, and I’ve heard many people here express this frustration that you know it feels like this might be a settled issue, to an extent.

But it’s really more than the layout I kind of want us to start thinking about here. Because I have this feeling that we’re really stuck in what I term a paper frame of mind—and again, Willie touched on this in his presentation—that somewhere in our subconscious we treat paper, or at least the way paper behaves, almost as a gold standard. Because it has been a gold standard for years. And we almost treat electronic implementations of originally paper-based questionnaires as just kind of an expensive version of paper. It’s got more colours, it’s a bit interactive, but it’s just a flat surface on which we respond to that boring list of questions. So these questionnaires are very static and undynamic and unreactive. Even questionnaires that we’re developing on the electronic platform from scratch are still just questions that sit there and ask for participants to respond to them following a certain schedule, whether it be daily, whether it be weekly, whether it be monthly. And what I want to propose that we start thinking about is that there’s immense value in obviously this averaging of experiences, it’s how we get a kind of solid understanding of the impact of a treatment across a wide range of participants, so I’m not dismissing all of that. But I want us to starting thinking about the fact that maybe we’re missing an opportunity for kind of  amore fine-grained and a more nuanced insight into the patient experience.

This is a lovely event-driven diary, the Bristol stool scale, I’m sure some of you are familiar with. I think event-driven diaries are a step in the right direction. They’re a more dynamic way of collecting data. So it’s not following a fixed schedule. It’s participants responding when they have an event, so in this particular case, when they have a bowel movement. The participant then responds in that moment. So it’s a bit more dynamic than complete this every week, for example. And really, electronic data capture made event-driven diaries a feasible mode of data capture, of course the big one being time stamping, so you can ensure the participants weren’t just filling in their questionnaires before they came into the study visit. But also, reminders and ease of data entry has made event-driven diaries a pretty feasible approach to capturing quite insightful data for a clinical trial. But I still argue these are relatively static, they’re reliant on the patient to actually trigger the event, to acknowledge that an event has happened.

So I’d like us to think about something that comes more from the psychology and sociology field. Some of you may have heard of the concept of ecological momentary assessment, or EMA. And the basic definition of that is that you’re taking repeated measures of symptoms from a patient, close to a specific period of time, within the patient’s natural environment, and obtaining it many times over a specific course of a study. As I said, this is coming from the psychology and sociology background, so a lot of existing literature is focused on psycho-social factors. But kind of a nice way to think about an event-driven diary that’s not really tied to an event, but it’s tied to a specific moment in time. And it varies how you might define that moment in time—it could be a random point, or it could be based on other factors that I want to touch on in a bit  more detail.

Just to give you kind of some examples, lots of apps exist out there for capturing EMA data. It is used quite readily in the psychology and sociology field. So you see here on this app, time for a survey just before the beep. So a beep has gone off on this participant’s smartphone at a random time during the day, you can set the window for when it goes off. And the app is just, tell them how happy were you feeling. So how happy are you right in this moment in time. How stressed were you feeling. So it’s a report of how happy and how stressed the participant is in this very minute. Now think of that happening two or three times a day, seven days a week, across a number of weeks, a number of months. Think of the data you might start to capture there, the insight into the patient’s ebb and flow of experience. We’re not always happy. We’re not always stressed. When you average across time though, how happy were you in the last two weeks. I was very distressed this morning from the lack of sleep, but I’m feeling quite good now, because we’ve had some very good presentations. Across a two-week period, I don’t know if I can answer that accurately. 


Another nice example, it’s what were you doing right before the beep went off. Again, a randomly triggered beep. Reading computer, watching TV, movie, eating, drinking, physical activity, exercising, and other. So it’s getting insight into exactly what the participant is doing in this specific moment of time. And then you can dig into that in a bit more detail in just another version of that. So it’s really gaining insight into the patient’s natural environment as we said so the real world, for want of a better word, at discrete time points and repeated measures.

So another thread that I want to bring in here is this lovely objective data generation. So we’ve already had lots of talk about wearables in the workshop and Willie’s great presentation. The three I want to think most about are medical devices, so-called wellness devices, and sport and fitness devices. So these are all the wearable devices I think we’re most familiar with, whether it’s activity monitors, or those things you wear on your body that track various aspects of what a patient is doing. But what I really want us to think about is that lovely thing that all of us are carrying the vast majority of the time with us, and the data that that might be capturing from us. Our smartphones. And think about the sensors that that little block of metal and plastic has on it. And that are recording basically continuously. So you have a light sensor, so a monitoring of the ambient light conditions. Proximity sensor, typically two cameras, sometimes three. A number of microphones, you can get a sense of the ambient noise. Obviously the touchscreen, various things you can do with that. Position, whether it’s GPS but then also the actual physical position of the phone via accelerometers, etc. Magnetometer, gyroscopes, pressure sensor, temperature, humidity. Also obviously there’s various aspects of levels of communication. How many messages you sent to whom. How many phone calls you made, for how long. These are all things that our smartphone can collect, and we have to do absolutely nothing for them to collect it, they’re just doing it right now. Your smartphones know you’re in an Oxford right now. Your smartphones know what the noise is in the background of this room. Your smartphones know what the light levels in this room is, assuming they’re out on the table.

So what, you might ask. Well there’s some very fascinating stuff you can start to do with that passively captured data. So this is data you haven’t done anything to provide. This is some work from Madan from 2011, who were able to predict runny nose episodes, episodes of fever, and episodes of low mood in college students, self-reported low mood in college students, based entirely on the levels of communication as measured by their smartphones, so how actively engaged they were with their friends. How actively engaged they were at certain times of the day, whether in the morning or in the evening. It could also tell things by were they spending time in groups, by the number of connections their Bluetooth was making with other smartphones around them. As well as this kind of stuff they were exploring the dissemination of opinions though groups, based on again these smartphones making and breaking these network connections.

So to reiterate, this is all passively captured data, this is not something the participant is doing. They’re just generating this data, for want of a better word, we’re leaking data continuously through our smartphones. And yet we can make these links and from those links we can make these predictions on what’s going on in the participant’s life in any one point in time.

So another thread that I want to bring in, this whole real-world effectiveness of treatments. I don’t think it’s necessarily too controversial to say that medications are rarely as effective in the real world when you start giving them to patients day in day out as you might find there in a Phase III clinical trial. There’s some very obvious reasons for that. It’s a very specific population you’re looking at. You have quite strict inclusion-exclusion criteria, you’re making sure they don’t have any weird co-morbid diseases, you’re making sure they’re not taking any additional medication that might interfere with your drug. Basically, very different population to that that might in fact be taking the treatment once it gets out of the clinical trial environment.


And obviously a lot of attention is being paid to the patients in the clinical trial. And that’s a very reinforcing thing. That’s got to make you much more likely to be very compliant. And we know that compliance to your medication is one of the biggest predictors of how successful your treatment is going to be. Obviously in a clinical trial, the attention is very much on how you’re getting on with the medication, am I taking that medication as I promised I would. When you’re back home, the only person who’s really worrying about it is yourself, you’re understandably probably not going to be as compliant, so the treatment might not be as effective as it originally was in the clinical trial.

This kind of has got to such a concern that there is an ISPOR policy analysis written about it in I think 2007. And they basically just flagged up that randomized control trials, they’re obviously gold standard and you know we’re not going to do away with randomized controlled trials. But they operate very much in this idealized environment that I was talking about. And they don’t, really when you get down to it, necessarily indicate how effective a treatment will actually be. They can maybe indicate how effective a treatment is in that perfect situation, in that idealized environment, but unfortunately we don't live in an idealized environment. We live in a busy and hectic world where we forget to take our pill first thing in the morning. They also highlighted the fact that these so-called real-life studies can take advantage of data that randomized control trials mightn’t be able to take advantage of. Now we might see this developing, as Willie showed, clinical trials are starting to look at least at wearables and additional sources of data, and this is always something that’s going to be evolving over time as healthcare develops. But it’s also something that the regulators have been highlighting, that you know maybe we need to be looking at this real-life piece in a bit more detail.

So to summarize those different bits that I’m claiming that I’m going to bring together in a coherent argument, so there’s the subjective momentary rating, so it’s that input of how the patient is, closer to that traditional patient-reported outcome that we’re maybe more familiar with, but in that kind of more momentary trying to assess how someone is in a specific moment in time. Passive objective data, so whether it’s from wearables, whether it’s from a smartphone itself. This kind of real-world data, which is slightly different data to that that you might capture in a Phase III clinical trial. And then something I haven’t even touched on, but Anders will talk about it in a bit more detail, metadata, so data about the data that you’re capturing, which can tell you a huge amount as Anders will show us. And big data sources as well can tell us a massive amount about who the patient is and what the patient is doing. All of these can feed into, I would argue, a greater patient insight, or what I’m going to trademark as Health Outcomes 2.PRO, because I think it’s very clever.

So what am I really trying to say here? Well, I’d like us to think of maybe dynamic triggers for administering questionnaires based on objective or environmental conditions as rated by our smartphones, as rated by our wearables, as rated by sources of big data that know where we are, know what conditions are at that moment in time. And I think that would allow us to gain an understanding of the patient experience based on their current objective and environmental conditions and thus allow us to make those kind of predictions that we saw based on just that information around how many phone calls you were making and your communication levels, making those predictions around a low mood for example. And it will also ask questions based on the best time for the participant, so it will know when the participant is not halfway between the office and home, it will know when the participant is having some quiet time, that might be a good time to administer a traditional PRO, a longer PRO, and knows from experience that the participant always sits down in front of the television for half an hour at 5:45 every evening. Maybe that’s a good time to try to get them to complete a short questionnaire.

I think it ultimately reduces the burden of data reporting on the patients by driving information from the passive objective data, rather than having to seek large amounts of it directly from the patient. So we can make extrapolations from the objective data that they’re not doing anything to give us, they’re just being, they’re just existing. And yet we can get insight into their experience based on the work we’ve already done with some momentary assessment by asking how they are and tying that into the objective data they’re collecting at the same time.


And I think maybe a more palatable angle to take it, is that it potentially gives us better confidence in the traditional PRO data we’re collecting, because we can tie it to objective data. So we can tie the outcomes from traditional patient-reported outcomes to their activity levels, to other sources of data, sources of data that are not potentially impacted by the fact that the patient didn't’ have a good night’s sleep last night. So it helps, I argue, boost our confidence in the power of patient-reported outcomes in the first place by bringing back in this objective piece. We’ve almost worked quite hard to get rid of it because we focus so hard on making patient-reported outcomes this really hard science. But I think there’s so much value to bringing that objective piece back in and put them together to give us something new and something much more interesting. And we can fit it more seamlessly into the patient’s real life and capture some very interesting data there as well.

So obviously there’s always going to be a place for these traditional patient-reported outcomes, but I almost make this as a plea for instrument developers and for everyone who works in the space to get out of this paper frame of mind, where appropriate, to not limit ourselves to a list of questions that are administered following a thick schedule. We can do so much more with the technology. What we’re trying to measure are messy concepts, no matter how hard we work with psychometrics to really make it a very robust science, and it is a robust science, I’m not questioning that. It’s still pretty messy concepts we’re looking at. And I think asking these questions in different ways isn’t going to undermine the power of those questions. And I think we get caught up in the issues around variability, but I think the variability within the questionnaires is potentially higher than we’d like to admit, especially compared to the variability between the questionnaires, as we see with this issue of test-retest versus paper and electronic administrations. And obviously technology really gives us the approach to dig deep into these objective but also subjective pieces.

So basically, my summary is that there is room for—I argue anyway—a more nuanced insight into the patient experience, which can reduce the burden on the patient by kind of inducing from objective data sources the actual patient experience. But also by triggering questions on specific times to get a very nuanced and insightful experience of the actual patient experience, as well as triggering them at a more appropriate time, all in a real-life setting.

So like I said, this was a very whimsical presentation, but I would like us to think that maybe it gives us the possibility to think differently about what we’re doing with our data capture, what we’re doing with our patient-reported outcomes, and really start taking advantage of technology rather than, as I said, treating it as an expensive piece of paper.

So thank you very much. And like Willie said, we’ll leave questions until the end after Anders.

[END AT 23:09]

Previous Video
BYOD and Patients: Hype vs Reality
BYOD and Patients: Hype vs Reality

<p>Bring Your Own Device (BYOD) is the latest buzz-phrase in the eCOA industry, with the possibility of tak...

Next Video
eCOA and the Discipline of Accessible Interface Design
eCOA and the Discipline of Accessible Interface Design

<p>It has become the norm for designers in other commercial fields to try to make their interface designs '...