Day 106 — User Feedback Report for App-Based Assessments Prototype

Breaking down the results of this week’s experiment on transition assessments

This week we tested a prototype meant to be an alternative way to deliver a transition readiness assessment to teens with type 1 diabetes.

Our design challenge for the week was this:

How might we create a fun and engaging data collection experience for teens?

The Testers

Demographics

• 3 female, 1 male (12, 17, 22,22)
• Also included 2 moms
• 1 in middle school, 1 in high school, 1 in college, 1 recent college grad

Favorite apps

• Snapchat, iMessage, Instagram
• Dexcom, Sugarmate, Youtube. Plays a lot of Roblox and even has own YouTube channel (mostly plays obstacle courses)

“How did you first learn about type 1?”

• “I first learned when I was diagnosed in the hospital. Didn’t know anything before then. From there, Mom showed me what to do.”
• “When I was 14, I met 16/17 year olds that were same gender (didn’t really meet any older boys) — helps seeing an adult when you’re little b/c shows you’re normal (‘oh my gosh I’m going to be weird but when you see an adult it’s like ohhh you’re chill, you’re not weird. But if you meet someone your age and like 1–2 years older you’re like, wow, you’re cool, too.’)”
• “I met a fellow in the ER who also had type 1. He came in and talked to me. It scared me b/c I saw tubing (‘omg this is my life’) — but he was so normal it was cool.”
• Mom — “when she was 1st diagnosed I didn’t know what type 1 was. Even said to the nurse, “Oh, they don’t eat a lot of sugar. It was completely shocking to us.”

Prototype Feedback

1 — Welcome

Welcome to Invincible

The goal of this section was to welcome users to the app and give them a sneak peak at what they can expect. I used VideoAsk to deliver interactive video content as a way to drive this engagement.

Feedback

• It’s easy to use…I like when you go to a normal app and it tells you how to navigate…really simple and straightforward.
• I think it gets the point across — it’s a good starter.
• I like it how you can side click it and click the button to see next video (referring to video interaction from VideoAsk)

Random Insights / Observations

• 2 people laughed at silly face in 1st video.
• Users were confused whether they could click play or should click continue.
• 2/4 users asked whether they would need to watch the welcome video every time (“Is this something every time you open the app you’ll go through vs. one-time?”) Not sure if this was a nice way to say they didn’t like the video or they just didn’t think they needed it more than once.

My Takeaways

• Video — The intro video is decent, but will be way better to include other kids/teens/young adults welcoming users to the app (e.g., other people talking and other funny faces to beginning of video). Shorten feedback part of video (and remove for prototypes like this). Might be nice to collect name, etc. to drive more engagement, too.
• UX — Improve UX for playing the video. It wasn’t clear to users what they should click and everyone hesitated on that screen.
• Overall — Intro is short and sweet and works reasonably well. I still think we could have a bigger hook at the beginning to get users more engaged and to quickly demonstrate what we’re all about.

2 — Tell Us Your Level

Letting users “test out” of intro lessons

This section was all about letting users opt-in to the assessment. My hypothesis was that giving users an opportunity to “test out” would be a welcomed activity — vs. forcing users to complete an “assessment” before they can access any of our content. I stole Duolingo’s approach for interactions but added the 3rd screen above since we’re asking users to honestly self-identify their skills and that can feel pretty scary.

Feedback

• Aww, I like that (“we’re not here to judge — that’s cute”). I like it b/c some people get hesitant: don’t want to feel like they’re doing it wrong/right — this makes people feel like I’m accepted.
• Good to make sure you’re going in the right direction. It prompts you — it was obvious this is what you have to do.

Random Insights / Observations

• Everyone clicked “I already know some diabetes”, though there seemed to be some hesitation over which option to select for a couple of the users.

My Takeaways

• Messaging — Could work on messaging a bit more to make it even clearer which path users should go down (newbie or advanced). Also, find ways to reassure users even more that there are no wrong answers. I found myself wanting to do this verbally but should take additional steps during assessment to drive this point home.
• Overall — I found this approach really effective to let users opt-in to the assessment.

3 — Diabetes Basics Section

Diabetes Basis Skills Self-Assessment

This section was meant to allow users to self-assess their diabetes behaviors. These questions correspond to some of the most important skills users will need to perform to manage the day-to-day with type 1 diabetes.

Feedback

• “OK, it’s like a game” — Yup, yup, — Take own medications: yup — “I didn’t know it was like a game, that’s really cool.”
• Thought it was a good to get to know somebody’s baseline — Thought it was straightforward / easy to use / good graphics — Some of things it was like (‘ohhh I do that’?).
• This was to the point where it’s not taking too much time…I know if it takes too much time to set something up I’m like “ehh nevermind”. With this, I could read the question and get through it.
• I liked the facts at the bottom — like that it’s encouraging.

Random Insights / Observations

• For 12-year old, was really valuable to have him complete this section with his mom. Was really fun to see them talk out the question and discuss who is responsible for each. Could be an effective tool to manage this shared responsibility.
• 1st question (do you test your blood sugar) — “I have a CGM” so don’t really “test”…sometimes I have to test my blood sugar but almost never: I only check w/ a meter when feels way off (don’t even calibrate it anymore).
• High blood sugar q — “I have a pump so my pump does it for me”
• Driving question should not be asked for users < 16, or enable an n/a option.

My Takeaways

• BG Questions — Account for CGM largely replacing BG Meters.
• Driving Question — Driving question should include n/a (or just not be asked to youngest users).
• Roles & Responsibilities — Consider ways to allow kids/parents to take this together and use as a tool to manage joint responsibility.
• Overall — This section was simple and straightforward for users to get through and gave a sense of what to expect for the app. Consider ways to emphasize that there are no wrong answers (and if user is clicking no a lot, provide add’l motivation).

4 —Space Balls Game (Diabetes Knowledge Assessment) Section

Space Balls Game

The goal of this section was to convert self-reported knowledge questions into a fun game where we could identify the user’s skill levels.

Feedback

• “OK, that was definitely fast” [note: made change to speed after this comment]
• That’s fun — definitely like some quick facts that not everyone realizes. Speed was good — on a phone it will be better (on computer w/ trackpad not as good).
• Some questions were too specific (e.g., ketone check at 300). Should be broad enough to include people but not limit others (e.g., ketone thresholds)
• I liked it, goes faster the farther you to get. Would be cool if you’re moving around vs. just up and down (e.g., moving the rocket in some way)

Random Insights / Observations

• UX on the ‘how to play’ screen was confusing for 1 of the users (clicked the buttons vs. start game button).
• Progressive speed was really effective to encourage users to continue.
• Move continue button on game over screen so users don’t accidentally press it while they think they’re still playing the game (to give them an opportunity to retry).

My Takeaways

• Game Speed — Progressive speed for questions is a great game dynamic that encourages ease of use to start but a challenge for even advanced users.
• Changes— Ensure questions we’re asking are tied to thresholds users actually follow (e.g., when to check ketones). Move the game over screen continue button so users don’t accidentally continue.
• Overall — This game is a winner for engagement and as a way to teach knowledge-based questions. Telling users they’ve “unlocked” a game presents a fun dynamic that may encourage more participation, especially if we add more games as they progress levels. As an assessment though, I’m not sure if this is the best way to truly “assess” skills, as users may miss questions just because it’s too fast. Definitely continue testing this.

5 —Adulting Section

The goal of this section was to understand adult behaviors that users will need to adopt to become fully “independent.” This section was very similar to the 1st except for I altered the yes/no thumbs up/thumbs down into an x and check just to mix it up and see which one worked.

Feedback

• Those are definitely adulting questions b/c little kids would not know anything about that.
• Adulting (shakes head, no thank you). I’m assuming at some point once you get through the setup….are there going to be resources as to “I know how to get in contact w/ someone”? (no big deal, we’ll help you get the resources you need)
• Mom — I like the words of encouragement and you can move at your own pace. This is probably better for an older audience, maybe like 16/17.

Random Insights / Observations

• The word “Adulting” received a positive response from young adults.
• Motivational feedback was really good in this one.
• Especially for 12-year old, almost all of these questions were a no. When asking questions, we should think hard about what’s actually age-appropriate and screen out questions that 12-year olds shouldn’t be doing (e.g., this entire section probably shouldn’t even be shown to a 12-year old).

My Takeaways

• Showing immediate Value — One user had some great ideas on how to tie resources to responses (e.g., if you say you don’t make your own appointments, we say that we’ll share some resources to help you do it). This could be huge, especially for demonstrating to users that their answers matter and that we’re here to help.
• Making it Age-Appropriate — This section generally isn’t useful for early teens. Do we remove it entirely or show these as a means of aspirational skills?
• Overall — For late teens and young adults, these questions make a lot of sense; for tweens and early teens…not so much. I think there is even more we can do to tie responses to relevant content so users can fix skill gaps.

6 — Stuff Section

The goal of this section is to better understand the supplies users carry and (eventually) to motivate these behaviors. In the original assessment, each item was it’s own individual question — my hope with this UX was that we could simplify the experience and make it really easy and quick to answer.

Random Insights / Observations

• Finds that glucagon is not something people realize they should carry it with them (her doc says that when getting new patients they don’t think to ask ‘are you carrying glucagon — you’re never prepared for an emergency) — always with her — in backpack if at school or in purse if not in room.
• “Glucose meter: do I have to carry if I still carry my CGM?”
• Snacks for exercise majority of time — majority of the time, not ALL

My Takeaways

• Overall — This was a really simple and effective way to get this information vs. asking 8 individual questions. Based on user responses, we should find ways to motivate the user behaviors reflected here.

7 —Support Section

The goal of this section was to evaluate some of the psychosocial components of care. I initially called it ‘psychosocial’, but I heard feedback from our first user that she doesn’t like that word and it carries its own stigma. So I switched it to “Support”. I went with a different UX format here so we could test a different style — in this case, multiple choice with a range.

Feedback

• Like that section: straightforward.
• Ohhhhh, this is a good question (weight). It’s such a huge question…so much pressure from society to be a certain weight — don’t think this is addressed enough in diabetes. I was 20 at the time and when my endo was like, hey by the way, but just so you know…this is what happens when you’re on insulin. It’s a question that I think needs to be more out there (better yet, a conversation that it’s ok to be whatever size as long as you’re healthy) — Everyone’s version of “healthy” is different. Encouraging of exercise (not that you need to be running a marathon every day but just doing little things that aren’t going to only benefit your health but also mental health, too)
• Liked how it asked how we feel.

Random Insights / Observations

• When I saw the feedback, thought that there was a right/wrong answer.

My Takeaways

• Weight Question — Weight is such a sensitive topic but one that absolutely needs to be addressed, especially for teens and young adult females. We need to take special care to message it effectively and show feedback that is understanding and motivating, regardless of the response.
• Overall — This was probably the best section for assessment effectiveness. It reflects the complexities of care and that there are few yes/no answers in diabetes. Definitely continue exploring this approach.

8 — Level Complete

Gamification

We’ve tested this in the last few prototypes and continued this as it seems to work reasonably well to motivate users.

Feedback

• “What’s 10 XP?”? — Assuming it’s points for something. Wasn’t sure if that was points or something you’re competing with other people for — Like the idea of lightning bolts since idea of app / program. What do use masks for? — It says level 1 — so that I’m working toward the new goal / moving up the ladder.
• I want to know what the cool person bonuses were — e.g., if I clicked yes for everything, does that mean I get a point for each question b/c I clicked yes or is that standard or…..(if I click no do I lose points…am I less cool b/c I don’t know what to do?)
• This app makes it bubbly and fun — having an app makes it normal (some people) — if there’s an app, it can’t be that rare of a disease — Really think app would helped after diagnosis.
• Mom: Like that you finish level and can complete. — Every 5–10 levels you can get a badge.

Random Insights / Observations

• Think the 1st video should explain the XP / masks / goal.

My Takeaways

• Overall — These continue to work reasonably well. The goal of these is to make the app more fun/engaging, so it’s hard to truly test this in such a short test. Consider adding a clearer explanation of the goal of the app to the intro video so users know what they’re working toward (OR: showing the goal early on tied to the points).

9—Results Section

Assessment Summary

My goal here was to show users the output of their assessment and test a profile concept where we visualize skill gaps to becoming independent.

Feedback

• I look at it and I say, “I’m 21 but I’m showing at the age of maybe 14 for psychosocial”
• I think the bars can be really discouraging if somebody would be 15 and barely hit the age marks. Would make me sad to see that I’m not even halfway.
• Like they can design a person / get a person and when they complete it get a superhero suit and this/that and another.
• Mom — liked how there’s different ranges, kinda shows where he’s at.

Random Insights / Observations

• 2 users asked whether they could customize the avatar. Wants to be able to add specific type of insulin pump / decorate character
• Fine line between competition and de-motivating. 2 users saw they were below the age mark and reacted negatively.

My Takeaways

• Overall — We’re on the right track here but it still needs some work. Continue testing to see whether this is motivating (e.g., a competition to complete the bubbles) or de-motivating (e.g., I’m below my peers for these skills so I’m going to give up).

10— Personalizing Content Section

The goal of this section was just to show the vision of personalizing content based on the results of the assessment.

Feedback

• Cool, so it’s like a progression to keep getting them to come back. I think I’m competitive where I’m like “oh, I’m going to go back b/c I want to see what the end result is
• Maybe like based on your results from the game, it might be a little personalized to you with how much you know.

Random Insights / Observations

• Feels like there will be tools/resources /suggestions based on what they answered and maybe a link of (hey, we think you would really enjoy these blogs / resources / accounts).
• Connects you to the community based on the results and how you know. Maybe you could pick if you’d like to talk to some older kids and how they’re dealing with it (might make me feel more comfortable if I was 11).

My Takeaways

• Overall — There wasn’t much to test here other than that we could show content based on answers. Next test consider delivering a smaller assessment and delivering content based on results.

Final Thoughts from Testers

1 — What is this for?

• Assuming to have a baseline of where kids are at and tools / resources they can use to propel them. — Going from baseline to adulting and help from transitional phase from child to adult.
• It’s a resource but it’s also kind of like a fun diabetes spot. I would go to it if I was looking for information after a quick google search and not finding what I was looking for.
• It’s a game. Kids get bored really easy. Game aspect is fun because it gets people engaged and they’re learning about their disease. It’s fun b/c it’s not very talked about. — Get to learn about their disease b/c it’s fun and simple.
• To see what range you’re in (e.g., the range for superpowers — see where you’re at with diabetes care)
• Mom — A good teaching point for kids to be aware of where they’re at, kind of like encouragement. Really good, especially for prepping to become more independent.

2 — How would you describe this to a friend?

• App that is able to help people with diabetes better understand how they can progress w/ their diabetes and manage it.
• App that is able to help you see where you’re at with your diabetes and how you manage it and different tricks tools to help you manage it. Maybe a different way if this way isn’t working to help you in the future (or possibly future resources). And test your knowledge to help you work on your own diabetes story.
• You just need to download it and do it…it’s an app of diabetic resources that’s like personalized towards you but it also has a game component. I felt like I was playing a game when I was doing it so that was fun.
• It’s a good app to learn more about diabetes.
• Mom: Good app to learn more and gain confidence in a fun way (interactive vs. a whole bunch of content).

3 —Who is this for?

• 10–15 age range (maybe a little bit younger). I think probably older people wouldn’t want the superhero theme but there’s a lot of customization
• 12–20: Used for anybody but can give it to a newly diagnosed person or if you’re trying to see where skill level is can use it to see.
• Mom — Can start as young as 8–9. When she thinks back, tech things always connected, so things in upper elementary age would have been helpful and enjoyed.

4 — If you had a magic wand to make this better for you, what would you use it for? Why?

• Add ability to customize background (e.g., change background that the play , Space themed background)
• Mom — Choice to customize their superhero a little bit
• Start by building avatar and then ask a few quick questions to get the baseline.

5—Favorite Random Insights

• “When I was 14, met 16/17 year olds that were same gender— helps seeing an adult when you’re little b/c shows you’re normal (‘oh my gosh I’m going to be weird but when you see an adult it’s like ohhh you’re chill, you’re not weird.’). Kids that are small look up to anyone who’s older.”
• “Lot of stigma for weight b/c you’re on hormones so weight is going to be higher. It’s a question that I think needs to be more out there (better yet, a conversation that it’s ok to be whatever size as long as you’re healthy).”
• “Having all the resources available up-front is really important. I’m looking at it from a mom’s point of view and needs a resource….if I went through it and didn’t get a resource, I’m wasting my time.”

Personally, I really enjoyed seeing the relationship of a mom and child and how they manage shared responsibilities. We need to factor parents into this process, especially for early teens who are taking over responsibilities from their parents.

Retrospective for User Testing

Overall, I’m super happy with where we’re at. This process works great — and turning around a prototype in a week with real user feedback is incredibly helpful and motivating. While we’re not there yet, I learned a lot and am confident this iterative process will drive us toward improvements that will make a meaningful difference over time.

What Went Well

• Game components are working.
• Talking to users is so insightful — never know what they’re going to say.
• Zoom interviews work well (as long as they have the right computer setup!).

Opportunities to Improve

• User Bias. Because testers know I created the prototype, there is definitely some hesitancy to share truly harsh feedback.
• Interview setup (phone and iPad interviews made it hard to test and see facial expressions). Inform users before interview that they need a computer to use these prototypes.
• Deliver consistent interview questions to ensure better results (1 interview went off a bit of a tangent).
• Find younger testers.
• 1:1 > 1:2 interviews: including moms is great but kids also seem hold back their responses / are talked over by moms.