Watch on Demand

First Name
Last Name
Country (Citizenship)
Opt-In to Future Emails
Thank you!
Error - something went wrong!

What Metadata Can Teach Us About Actual Patient Reporting Behavior

February 5, 2016

By looking at real patient reported data, this webinar will explore how patients actually report data for event-driven and scheduled data capture, compared to expected behavior. 

Presented by Anders Mortin, Co-Founder at TriTiCon, who will discuss his research findings that reveal insights into long-term patient compliance with reporting data in clinical trials, daily reporting preferences, and event time versus actual the reporting time.  The metadata available should be considered when designing endpoints and patient reported outcomes.

Questions answered and discussed include:

  • Do patients actually report the data when we expect them to?
  • How well do patients keep up with long term reporting?
  • What are real patient reporting patterns and preferences?
  • How do those patterns and preferences differ in different subgroups?

Full Transcript



Joining us today on the call, and we appreciate your attendance. And today’s webinar is a particularly interesting topic. We’ve invited one of our good friends, Anders Mortin, to present. The topic is What Metadata Can Teach Us About Actual Patient Reporting Behavior.


 As I mentioned, our special guest speaker today is Anders Mortin who is the co-founder of TriTiCon. He is a good friend of CRF Health and brings with him over 20 years of industry expertise, having formerly worked at both Lundbeck and Ferring Pharmaceuticals, in quite a diverse variety of roles spanning clinical data management and project management and e-clinical program management.

I’ll let Anders continue in just a moment, but some housekeeping items. If you have any questions during the webinar you can feel free to type them into the chat box on your screen, and we will have some dedicated time for Q&A at the end. Also, there is no voiceover IP option, so make sure that you dial in using the number provided. And we will send a copy of the video recording for your follow-up tomorrow, so look out for that.

But without further ado, I’ll hand it over to Anders to continue with his introduction, to take you through the webinar. Thank you.



Thanks Naor. Yeah, this is Anders speaking. Thanks to everyone for joining. So my name is Anders Mortin, I’m talking to you actually from Copenhagen in Denmark right now, where I’m working for a small pharmaceutical consultant company. What we try to do with our experience is combine leadership, product management, and subject matter knowledge to provide business consulting, project management, data management, health economics in various combinations. And part of the things we do, like you will see in the next 20 minutes or so in my presentation.

So what I’ll try to do is look at actual trials, real clinical trial reported data, the data and metadata. We’ll try to learn from this metadata and figure out how we can use that to understand actual behavior from the patient. And we’ll try to relate that to what we expect and to the end point as well. So we’re looking to, do the patients actually report when we expect them to do—if we ask daily questions, morning questions, if you expect them to report things as they happen, do they really do that. How well do they keep up with long-term reporting.  You can say that’s kind of compliance questions, but from another angle there’s also more to it than that. Trying to understand do they keep up and if they don’t keep up what’s the effect of that. With patients, what do they prefer to do, and do they prefer to do things differently in different subgroups. So can we find any patterns to help us understand how different patients behave differently. What we’ll try to do is not to give you some brand new designs for your end-points or for your patient diaries or other instruments, but I’ll try to give some examples and some inspiration how metadata and another way of looking at data can help you understand these things and potentially going forward, help improve end points and technical design.

So the agenda I’ve set up to try to take through that is first slide, a short introduction about the trial. We’re going to look primarily at the metadata, primarily the time stamps and other things on the data. So, it doesn’t really matter that much, the details of the trial as such. But I’ll give you a brief introduction to it. And then we’ll jump directly into the patient reporting patterns. What do the patients actually do. As I mentioned before, we’ll do that looking at a whole lot of graphics, graphical displays. And I should say when you turn graphics and interactive graphics into PowerPoint format as we’ve done today, it becomes a little bit more flat and dead, if you like, than the more interactive version, but I’ll try to make it a little bit lively and moving around as well, to help you really understand what goes on with this data. As I said we’ll drill further down into the subgroups, do the patients behave differently, and depending on various parameters. And then I’ll have a small wrap-up and we’ll open up the Q&A, as Naor said, using the chat function. And of course you can even type in questions at any time to the chat, and we will pick them up when we come to the Q&A session at the end.


So introduction. As I said, one slide on the trial that we’re primarily going to look at. Indication, it’s a constipation trial, which means the patient's constipated, they don’t go to the bathroom as much as they would like and would need. So looking at bowel movements and their frequency, and the number of bowel movements. It’s a Phase III trial in 11 countries, in most of the world. And the primary end point is the number of bowel movements average a week. In addition to that there also are some secondary end points of various kinds. They are interesting because they are also based on some subjective observations, like pain and straining that the patients report. And then you also have some daily and weekly evaluations by the patient. So this means that it’s a mixed diary we’re looking at, done at home for six months by the patient every day. It’s got some scheduled activities—morning and evening questions—and it’s event-driven, which the key thing here, bowel movements, the  design and the idea is that the patient should report a bowel movement more or less as it happens. So that should be when it happens, if it happens, it should be reported at the same time. The frequency they have given is about 10-20 events per months. It’s not very high frequency to that, and what of course we want to achieve as a benefit from this is an increase in the number of events.

Just very quickly for you, what this trial we’re going to look at is all about. Again the key thing is, you’ve got the schedule, where you’ve got the time window every day we expect them to report in, and we’ve got another portion that’s event-driven, saying when you go to the bathroom, we want you to report that immediately after it happens, and give us that data.

So with that start-up introduction background, we’re jumping straight into actually looking at what do the patients do. And as a little bit of a warm-up thing, what we’re looking at is the long term stamina, or basically, you can see this is set up as a compliance graph. So what this is showing us is the compliance with the data question over study time. So what we have on the screen we’ve got a zero or one all the way to the left, and we move up to 180 days. So what we see when it comes to the daily question that the patients are expected to answer every day, we do see a small line here which is slight decline. Even if it’s not that much, we could ask ourselves the question, if this is slightly declining, does this translate into the event reporting. Because the event reporting has the challenging question too that we don’t know how much should be reported. Zero, no events, is also a fully valid acceptable answer. So of course the trick with event reporting is, we don’t know if there’s one, we don’t know if there should have been two. If it’s zero we don’t know if that’s correct or whether there should have been one or two or three. With the daily schedule questions of course we can do a compliance graph like this, we can see if we don’t have the right answer. So the reason for looking at this point is to ask yourself if this is also true for the event reporting. Because of course if they don’t report, we don’t see the effect. If they get more bowel movements that we want, but they don’t report them, that kills our effect signal. Of course you could say that would be seen in the placebo obviously because it did differentiate the placebo. Still, it would be very difficult to sell a drug where you want to increase something if you cannot show the increase. It’s difficult to say the placebo increased even less. So that was the reason for looking at this one from another angle and saying yes it tells us because overall we got 90% compliance, which is what you see in most trials, what you would expect. And you would normally be quite happy with that. Still, we got a 10% decline over the trial period, and we need to ask ourselves, how would that translate into the event reporting with a primary end point. I don’t have a clear answer. I’m just saying we need to know that as a background to look at the data.

The next angle of this we’re going to look at is also the daily reported questions. So this is where we had a time window, it’s between 4:00 in the afternoon and 11:00 or 12:00 in the evening. We have a window, that’s where you’re allowed to answer your daily questions. With that in mind, you could say bell-shaped pattern here, which will tell us that normally in most cases this would be okay. We are hitting the window pretty nicely. The patients prefer overall to report around 8:00 in the evening. But overall we seem to be hitting it pretty in the middle. So we would be pretty happy we see that the patients are pretty homogeneous in the way this is displayed. So the window seems right, but to get back to this graph when we looked at some other reporting patterns and some other subgroups, we can look at this one in a little bit more detail and see something where we see that on the surface this is quite good. But if you look more into detail, actually there are some strange things behind this one as well.


The next thing we’re going to look at is where we’ll spend the most of the time at this presentation, and this is now looking at the events that they’re reporting. So we can see at the top of the screen, we see the time of the actual event. So this is when the events happen, according to the patient. This is when they went to the bathroom and had a bowel movement. And this distribution here shows that. What we see beneath is the time of the reporting. And we can immediately tell, since these shapes are not the same, that they don’t report things as they happen. And that would be to say that they report the bowel movement that happened here would be reported down here at the same time, and these patterns would be the same. Here we see two quite different patterns. We see the tendency that most of it happens here in the morning, whereas most of the reporting goes on here in the evening. If we overlay these on top of the data and say let’s see when they report. So that’s the shape of the curve up here at the top right now. And then we color that with how long time ago was it when they report. So all the ones that were reported here in the evening, when did those events actually occur? So what’s the time delay, or the time window here, you could say. And we can see the blue ones are the ones that reported within an hour, that’s where we’d be pretty happy. Then you’ve got a few ones between one and two hours old, and then we’ve got the orange ones that are between three and six hours, and on you go. And that means that actually the ones that were reported here at 8:00 happened more than 12 hours ago. They actually happened here in the morning. So to give you an example of what really happened with this patient, actually from another trial with the same principle, we’ll say okay at the top here we’ve got the actual event. And down here we’ve got the reporting. So what happened is, the first ting that happened is that the patient reports something that happened the day before. Then you’ve got one, two, three, four events. They get reported three in a bunch and then one more a little bit later. Then we get a new event that’s getting more or less immediately reported. One more, reported. One more, reported, One more, reported. Then it stops. The reporting. Events continue to happen, and all these four here now get reported in the same bunch. So you could ask why it’s in a homogeneous pattern. And that is what’s resulting in what you see at the top, that a lot of these events get reported a lot later in various ways.

So is this good or bad? You could say 36% of the events were reported within one hour. It’s not too bad maybe. But actually more than 35% were reported after more than 6 hours. Six hours is quite long. If you say they left half of them out, you might say that six hours you would probably be pretty okay with remembering that. But how about the subjective data, the straining and the pain, how trustworthy is that after six, eight, ten, twelve hours. I don’t know, but at least I would say we need to ask ourselves that question, when we think that patients are reporting the pain and so forth when it happens. Because in a lot of the cases, more than a third of the cases, it’s more than six hours old, based on memory. Or, as in the next bullet, potentially on paper notes or something else. We also see from the pattern, maybe you can see behind the graph here, that the patient preferred to report during the morning, and primarily in the evening. So why do they want to do that. Of course I don’t really know why. We could ask if maybe bring your own device would help reporting during the day. This would carry an additional device like they did in this trial, would that help. Or if you feel uncomfortable and it’s difficult to bring a device really to the bathroom at work or an additional device that people start asking questions about. This was a small trial, straightforward, it was not a very sort of big thing, or thing that would stand out in any way. But still we see clear preference, patients do it in the morning preferably and in the evening.


So we need to know that and say whatever goes on, we have this clear tendency morning and evening, we don’t really like reporting during the day generally speaking. So now we know that, and from that the next question I had was, okay that’s good, but why is that. If there’s difference between the patients and why is it different. How does that look in different subgroups. Can we identify some subgroups and try to understand, A, if there are subgroups which subgroups are there. And also, lead us to why, because understanding would of course, enable us to do a lot more, potentially change some things, or know and understand better what goes on.

So what we did was we started to look at different subjects. And my first thought with this was, maybe we’ve got two groups of patients. We’ve got the patients that report as it happens, and we could call them direct reporters, and we’ve got the irregular reporters, the ones that doesn't do that, they just collect it in memory and report later on. So I thought there would be two groups. But there weren’t. What you see here is the percentage of events for each patient that were reported within one hour, you can see, there’s a pie chart here, to the right as well, and saying overall about a third of events reported was reported within one hour, two-thirds were reported up to more than one hour. But for the patients how is that distributed. And we can see that some over here that report everything, all their bowel movements are reported within one hour. And over to the right here, they are the patients that never report as it happens. But because we see that there are two groups, because there is a continuous distribution between the two, it’s very linear, you could say, from 100% to 0%. So there's no two distinct groups with a clear preference as doing it one way or the other. So there must be some other grouping behind this. So what we did here was we continued to look and see what other subgroups can we look at, and that we can find and see if we can find any other patterns.

So we looked at demographics, we said there are differences between genders, but practically not. So again what we’re seeing here, the blue ones are the ones who reported within one hour, the golden ones reported after more than one hour. Male or female, very similar. We had a questionnaire telling us employment status, so we compared employed to not employed. If you are at work, it might be more difficult to report during the day. If you are unemployed it might be easier, even if you have a lot of things to do to try to find a job and so forth. There actually are small differences and a little bit of a high degree of direct reporting if you’re unemployed. Not that much, though. Now we’ll look at the weekdays. And we’ll say okay, maybe it’s easier on the weekend to do it. But very little difference between the two, the weekdays and the workdays—the weekends or the weekdays. And we looked at age groups. And age of technology, looking into age groups, and there we definitely can see a difference. We can see the young people, quite a few in that subgroup actually this trial. But we look at the 20-29-year-olds actually are more on reporting directly. And the same goes for the older ones. To me that aligns perfectly with compliance numbers. These patients are typically more committed to following the instructions to doing what they should. Then we got mid-age, so that’s my germination. It turns out we are too busy during the day so we do it in the evening. So they are worse, if you like in reporting, to do tasks as they are expected to do when they should as they happen. So age definitely did have an influence.

We also then looked at countries. And it actually surprised me to see the big differences between countries. I didn’t expect this one to be that clear. You’ve got a country like Canada, they report a lot within one hour. Whereas you’ve got some like Slovakia and Poland, they do it very little. So clearly a difference here between different countries, much more than expected really. Canada high. Sweden as well. Great Britain quite high. And Belgium, whereas you’ve got Czech Republic, Germany, Slovakia, Poland, comparatively low. I don’t know at all why that is. But it’s pretty clear from this that there is a difference. And that’s also what we’ll come back to with the daily reporting in a little while.


Another aspect to look at was the more clinical aspect.  We said okay let’s look at the trial, the trial stage, in the beginning and in the end. Are they more willing to report at the beginning and then they go a little bit tired and they report less frequently or more in the evening and more retrospectively. And there from this quick look there’s very little difference between early in the trial and late in the trial. Now we’re going to look as well to compare this constipation trial and an OAB trial, an overactive bladder, which is much more high frequent indication with much more events per 24 hours than in the constipation. And there we can see a clear difference, that in the OAB, we’ve got a much more higher reporting. And you can say is there a difference, I can also tell you that there is a small—very similar diaries, but there are some small design differences in this way from the constipation trial. You also have more of a chance to report retrospectively due to the low frequency, whereas you could say with OAB we’re talking about between 5 and up to 15 or even more events in 24 hours. So it kind of is also required because you wouldn’t be able to remember. But still, it’s a clear difference that they report so much more directly with OAB trials compared to the constipation trial.

So in all, from this, looking at it, from this with do they report when the events happen or do they report at other time points then in retrospect. It’s clear gender, employment status, weekend/weekday didn’t influence reporting very much, but age group, country seem to have a clear influence. Reporting seems to be very persistent throughout the trial, but the difference is significant between indications/design. So that kind of tells us also that it’s quite dependent on the design or depending on the indication, actually patients that basically are doing the same thing actually behave and thereby report data quite differently.

So you can now go back to this one that we looked at earlier on in the beginning and said this was the time window where the patient reports the daily question within this window. And we saw that the peak was clear around 8:00 in the evening, but it’s nicely distributed over the window. But now that we know that there are some differences between subgroups of countries and age groups, let’s look at those aspects there as well. And actually, now you can see if you split this one out by country, where you can see that Poland peaks around 9:00 so they want to report much later. Or comparatively quite a bit later than for example the Czech Republic. So clear difference there. So we actually can see kind of a strange peak here, which is probably due to the sample size, but it’s also an indication of, actually in one of the slides, reporting is very very consistent around 6:00, which is kind of another story that will lead more into things like risk-based monitoring, abnormalities, is there something strange going on in this site, is there maybe too consistent, all the patients reporting their events in a smaller window. I can tell you there was nothing behind this, it was more clearly structured than that.

We see the same thing maybe not surprising but here we see it in graphic and colour as well, that all the people did it around 6:00 and maybe around having dinner, whereas 20-29-year-olds, up late, had a tendency of shifting here over to the later part of the time window, when they report or use their diaries.

The age group and country subgroups seem to have a difference here in when they preferred to report during the evening as well when they could choose within the window. We saw this artifact in the data which can actually show that for one site in the Czech Republic, more or less all the patients report very consistently around 6:00. And you can say it’s not very— what’s more with this step. One thing of course is part—one of these questions is referring to day, of course we also need to start out ourselves here about the day concept, if people report at, you know, 5:00 or at 11:00 and say “how was your day today.” Is that really the same definition of today between the two different groups, because for the ones reporting early of course they can still have a lot of things here in the evening, in the afternoon, that’s not captured in the answers reported at 5:00.


A final little piece we’ll look at is the technical design effects, I call it. This one shows the difference between the reported—the event times and the reporting times. A whole long time ago when you report that, how long time ago was the event. And what we’re seeing here is a very very clear spike. They said it was one hour ago, two hours ago, three hours, four ago, five. Exactly. And that this actually is in effect is what we call the spinner effect. So this is, to me, clearly an effect of the design of this diary, is one of the spinners where it starts with displaying the current time, and then you drill in sort of backwards to when they actually the event was. And what this tells us is that what the patients do, they guess it’s now 5:28, when was the bowel movement. It was 2 hours ago. And they will just scroll the hours back, two hours, and they will not change the minutes. This of course is a clear effect of this being done in retrospect. They’re saying it was 2 hours, 3 hours, 4 hours ago, without being there to do the details of whether it was 2 hours and 20 minutes, 2 hours and 15 minutes. I guess they don’t remember/don’t care. And of course these time points are critical. If that’s a really important thing in your study, you need to consider this. Because again it will basically skew data and put it together around this certain time ago. It also becomes really important at some point where you want to compare two time points, because now it will start depending on which one we report first. So report something, say it was 2 hours ago, and then while you’re doing that the clock ticks  a little bit on for two minutes and then you report the other one two hours ago. Which means you get a two-minute difference between these two time points, and depending if you have the option to do one first and then they other one, those two events would swap times in terms of which one came first. If that’s important for your trial, in what happens first and what happened after the other, of course this can have a really big impact on that and how that, for example how voiding is classified in an OAB trial if it was before or after, you go to bed or got up, which makes a big difference in those trials. 

So that was all the graphics and all the pictures I was going to show you today. Hopefully that was some examples and food for thought. We did see that the reporting preferences and behavior is different primarily for indication design in country and age groups. Beware of the spinners, or are there technical designs or implementations, that could actually affect data collection quite a lot. And to me, I think we need to consider these actual patient behaviors when we design our diaries and instruments. It’s easy to over-define. It’s easy to go in with the belief if we ask a patient a lot more impact in details, now we get all the exact time points and numbers and facts, and now we as a pharma and as a sponsor, can do all the calculations and count all the numbers and the times in between and the differences and all that correctly ourselves. But that’s under the clear assumption that the patients do report, for example, when it happens, as it happens, in a certain way. To me metadata, in this case time stamps and things like that, are very powerful to understand the patient behavior and to understand how the instrument really is working. I think it’s a very sensitive tool to show some indications what goes on at the different sites. Because these are really talking about, if you are involved with ePRO and some of the challenges around that, you can clearly see that if you’ve got something that’s not fully working at the site, it definitely gets translated out into the patient behavior. So seeing what the patients do, to me, is a very strong indicator of site performance and site quality. So really it's something we can look at in the context of risk based monitoring, sponsor over-design, and site performance as well, which is partly a different discussion, but it would be based on the same data and similar approach to viewing and understanding and looking at the data.

Before we jump into the open Q&A session, I can see questions are rolling in here, great, thank you for that, we’ll jump to those in a very short second. But first I’d like to just give an opportunity, if you have questions that you want to ask, you want to see something more this, and you want us to follow up directly with you, tick yes here, and we will get back to you by email or phone to follow up individually. And if you’ve got questions that you want to raise here, of course you can do that using the chat window. If you choose yes in this option here, we will contact you. Press no or just wait, and we will continue with questions and answers.


 [Q&A section starts at 30:45]



Previous Video
Translation of eCOA in Global Studies: Challenges and Solutions
Translation of eCOA in Global Studies: Challenges and Solutions

For global clinical trials, eCOA instruments must be translated and then validated. This webinar provides i...

Next Video
eCOA and Equivalence Testing: New Evidence from Meta-Analysis
eCOA and Equivalence Testing: New Evidence from Meta-Analysis

Is equivalence testing necessary to establish comparability between electronic and paper-based clinical tri...