Subscribe: Apple Podcasts | Spotify | Android
Building software involves understanding our users. This helps us build better software and consider new things. Sarah Hall, Lead User Researcher at Crunchyroll, explained how insights from users can drive product development. We talked about different types of user research, and various examples of how a product can improve from this. Sarah also talked about the role of a Lead User Researcher and important skills for the job.
Transcript
[00:00:00] ES: I’m Edaena Salinas, Software Engineer and host of The Women in Tech Show a podcast about what we work on, not what it feels like to be a woman in tech. For more information about the show go to wit.fm.
Building software involves understanding our users. This helps us build better software and consider new things. Sarah Hall, Lead User Researcher at Crunchyroll, explained how insights from users can drive product development. We talked about different types of user research, and various examples of how a product can improve from this. Sarah also talked about the role of a Lead User Researcher and important skills for the job.
Before we move on with the interview, I’m really excited to announce that Season 1 of the Five Minute Mentor Podcast is now available This is a podcast where you’ll get advice from prominent people in tech, authors, journalists, artists, and more. You can find more information about the show by going to mentors.fm. Thank you.
Sarah Hall, lead user researcher from Crunchyroll is joining us today. Welcome to The Women in Tech Show.
[00:01:16] SH: Thank you. I’m so excited to be here, it’s really good podcast.
[00:01:20] ES: Thank you. I’m excited to have you on the show too. Today we’re going to talk a lot about user researching, user experience. First, I want to begin talking a little bit about Crunchyroll. This is a leading company for distributing anime and manga. Can you explain in more detail what Crunchyroll is about? Sure.
[00:01:42] SH: Well, Crunchyroll is part of the bigger Warner media family. And it’s actually the world’s most popular anime brand. We have the biggest anime library, and we’re translated into eight languages across 200 countries. And we have something like 60 million registered users. And I like to think that we’re a bit more than just a streaming service. It’s more of like a global community. We offer our fans like a 360 experience that includes games. We make our own shows and movies from our studio. We have merchandise, we do live events, we have a huge convention every year. It’s really a fan community. But yeah, also a video streaming service dedicated to anime.
[00:02:25] ES: So it sounds like what you’re saying is very focused on specific kind of content, right? In the anime space.
[00:02:34] SH: It is. Yeah.
[00:02:35] ES: Okay. Before we get a little bit more into user research, can you just talk about some of the main functionality of the platform?
[00:02:45] SH: Sure. So yeah, we have over 1000 titles, 3000 episodes of anime, but then we also have a bunch of cool community stuff like we have forums and chats where you can go talk about the show and find other people that like the same stuff is you. We have a localized to different countries so that you can get really specific to your region. We have stores, like I said, we do things like anime awards, so you can vote each year on your favorite shows, there’s a bunch of event information, you can see original stuff, we actually have Crunchyroll games, some that we make ourselves some that we partner with others for. So it’s really a big kind of fan site in a way where you can get all of your favorite shows, but then you find everybody else who likes them too. And you can really connect with people all over the world.
[00:03:36] ES: So you’re highlighting this community aspect is very important. And there’s different ways people can interact, forums, chats, get recommendations vote for awards.
[00:03:47] SH: Yeah.
[00:03:48] ES: Now that we have that context, you specifically work as a lead user researcher. In your opinion, and from your experience, what does User Research consists of, what is it?
[00:04:02] SH: I think fundamentally, user research is understanding the motivations and behaviors of a user so that we can help them solve their problems and achieve their goals. And then my role specifically, I lead user research within a much broader and bigger data and Insights Team. And so we actually work incredibly close with the analytics half of our team to triangulate observational and self reported data. With these larger trends that we see in viewership data viewership data could just be like everyone watching a particular show, or, this show, it appeals mostly to girls, or, and so then in user research, we do stuff like interviews and focus groups and surveys. And we put all that data together. And this type of integration helps us capture a more holistic picture of the user.
[00:04:51] ES: So that sounds a little bit related to the behavior component because you mentioned two main components of user research are looking at motivations and behaviors. So I heard you list several behaviors. What would be an example of a motivation in the context of the product you work on, which is Crunchyroll.
[00:05:10] SH: The motivation is maybe like the desire to escape. Like, say you’re a college student, and I don’t know, you’re working on really complicated math problems all day, which might not be your favorite. And you just want like a brain break at the end of the day. And so you’re looking into maybe a particular anime genre like fantasy to let you sort of escape into another universe at the end of your day. And so that’s a motivation, but it can also be really baseline like, hey, my friend recommended this show. I’m looking for it on the site, and your motivation is just trying to find what you want.
[00:05:45] ES: Can you talk about the different areas of user research?
[00:05:50] SH: Sure. Well, I think it breaks down into the quantitative and the qualitative. Quantitative tells us what is happening. So, like I mentioned, if we see a big trend of like, oh, Dragonball Z is super popular, or oh, it seems like only people under 25 watch this particular show. And so you can get that kind of stuff from like viewership data or surveys or A/B testing or intercept surveys. But then the qualitative gives you the answer, it gives you the why. So we see these big trends, but, why are they happening? Why do only boys like the show are under 25? Or why is this show so popular? or Why does everyone seem to be having this particular stumble or problem on the website? And that’s where things like focus groups and interviewing and field research really come in handy to answer you can follow up and really round out those answers of like, why is this trend happening? And then there’s stuff like usability testing, and I feel like that is actually a nice mix of both. Because if you give someone a test, like say you’re showing them a new design and you ask them oh try and add this to your watch lists are your queue, and they stumble. Or you can count how many seconds it takes them to complete that task or how many, you know, incorrect places, they click on their path trying to achieve that task. But then afterwards, you can ask them why, like, why did you assume that you needed to click here to go there? And so you get that nice mix of both, which actually gives you the answer. I don’t think you can have one without the other.
[00:07:25] ES: What would be an example of a product decision that was driven by an insight that you got from user research or that the team as a whole got.
[00:7:35] SH: User research drives all kinds of product decisions, like how to bundle our subscription tiers, what different features are people willing to pay more or less for but I think a more interesting one that we’ve actually been working on this week revolves around our Hime, our mascot, so she’s a gingka with is a anthropomorphize version of inanimate object or concept. In our case, she’s the live version of our company. She’s a cute little fox spirit. And so we’ve been doing a ton of research these last few weeks just trying to figure out where people want to see her. Like, if they see an error message, are they happier to see her face next to it? Does that make that a more pleasurable experience for them? Do they want to see her introduce videos? Do they want to see her recommendations? And so I think it’s a particularly, maybe more fun than normal way that user research is driving some product decisions and design decisions, at least lately for us. And we really inviting users kind of steer the ship on this one and tell us exactly how much they want of her and where they want to her and how they want to see her and what she should be like. And so it’s really cool, because it’s a very user generated kind of organic process, I think.
[00:08:53] ES: For something like this, do you do both quantitative and qualitative types research?
[00:09:01] SH: Yeah, we did a surveys, we did an in-app survey. So everyone that came to our homepage was just asked quickly, if they knew her, and if they knew her what they thought about her. We did some survey research showing people different designs, and some that just had her in some that didn’t have her and we didn’t point her out explicitly. And then ask people which page was more compelling. We also in some focus groups brought her up and had people give their thoughts on her. And people actually have a lot of, really, passionate thoughts on her, and how she should be used. Also at conventions. we did some interviews where she came up as well. It’s the nice mix and that kind of painted us a clear picture of just how we should employ her in our designs.
[00:09:47] ES: How do you decide what things to investigate in the “why”, in the qualitative? Because I imagine not everything that is measured, gets to have a dedicated focus group and additional interviewing, right? Because that costs more money. Is there usually a process that you go through to the terming? Oh, this, we should have a focus group for this.
[00:10:12] SH: Absolutely. So we almost always start out with the quantitative. And it almost always starts out with just the data that we see on our own site. So I mentioned we’re global company. And so we’ve been seeing that our popularity and more so than just even Crunchyroll, anime’s popularity in Mexico and Brazil and Bhutan markets in general is just skyrocketing. Anime is getting so popular. And so we could see that in the quantitative detail of like, oh, Mexico City alone watches so much anime on our site. So maybe we should go down and talk to them. So for example, we went down to Mexico, Mexico City in February for some exploratory research to follow up on some of these big trends we’ve been seeing. We wanted to get a deeper understanding of our Mexican anime audience members, so we kind of saw it in the data that there’s this huge market we saw where they were quantitative. And then we did a big survey before we went to try and identify some trends like where do they buy anime? Where do they hang out? What do they want? How is it different than American markets? And then once we got down there, we followed up with a bunch of qualitative stuff we did ethnography is we did a focus group we even did some usability testing, but mostly we just spent a great deal of time in people’s homes. interviewing them. We walked around areas where anime fans hang out like anime theater showings comic bookstores, this place called Frikiplaza, which is this amazing six story building with has floors for video games. It has anime merch, it has anime inspired food vendors, tables for like card games like Magic the Gathering. It is definitely somewhere I would have hung out a lot when I was younger had it existed in America. It was really fun. But yeah, but by like immersing ourselves in their worlds, we could really explore these trends that we had seen first and the data on a deeper level. And I feel like it added so much depth to what we already had, like seen what they had said in all these surveys. So that’s a way that it pairs really nicely, I think with each other. But yeah, I don’t think we would know what to look for first, if we didn’t go with hard data to lead us.
[00:12:30] ES: I’m really excited about that. That your having an effort, especially because I’m from Mexico. So I appreciate companies, you know, making an effort in understanding other cultures and then improving the product based on that. I was watching anime at Crunchyroll I appreciated the Spanish subtitles.
[00:12:49] SH: Nice.
[00:12:50] ES: Yeah.
[00:12:50] SH: Oh, that’s exciting. Yeah, we’re actually our team is supposed to be in Brazil next month, but COVID has sort of shut down our plans. But yeah, we’re really lucky at our company, we have regional leads in each country. And so we really can go explore other cultures, because we have these people that are from there that are also, you know, huge anime fans and stuff and understand the business that can really like scout locations for us and translate for us and just, I don’t know, be our gateway into all of these different worlds. And I feel it brings us together so much more as a 360 company and a global community.
[00:13:26] ES: Exactly. Especially because yeah, you observe you are global community, so might as well make an effort to understand your global users.
[00:13:34] SH: Absolutely.
[00:13:35] ES: I want to talk a little bit now about the Lead User Research role, which is your current role. What would you say are some of the things that you do as a Lead User Researcher?
[00:13:48] SH: As a Lead User Researcher, I lead my team and I help them design the best ways to collect data. But then also one of the trickier parts is, how to pull out insights from that data and then make those insights actionable. Because it’s one thing to gain a little knowledge, it’s another to translate it into something that other teams and the rest of the company can actually move on. And then I think it’s also very important to disseminate that knowledge. Half the battle is just getting up in front of people and trying to be persuasive and being like, this is what we’ve seen. This is what our recommendation is, this is why this is important. This is the evidence to back that up and to really push and lead the team in those efforts. And then I think maybe the last part would be mentorship. I think it’s important, especially because user research is such a new field to really act as a mentor and help those that are just starting out and coming up to learn kind of best practices and methods and stuff like that. And that’s one reason why me and a colleague created the San Francisco User Research Group. I started more in academia. And when I switched over to user research, I worked for a series of small startups. And I was the only one, I was the only user researcher embedded on a design team or working under a PM or whatever. And I just, I felt I had to figure everything else out on my own. And that can be kind of tricky. And so I think it’s really important to get a group of professionals together, especially when a lot of us are siloed on design teams, or kind of by ourselves and smaller companies. So you can share best practices and methods and really help each other just establish really strong practices.
[00:15:43] ES: One of the things that you mentioned earlier on was the importance of being persuasive, and you’re talking a lot about using data and how that can help. Throughout your career, have there been other things that have helped you improve in how persuasive you can be
[00:16:00] SH: Yes, I think I had to get a lot better at public speaking, I feel like coming from a more medical and academic background, I was just like, okay, here’s all the hard data. Here’s a graph after graph after graph. But honestly, that’s really hard for people to absorb. I do think that part of being a good public speaker and a persuasive public speaker is being kind of funny is breaking stuff down into little, I don’t want to say sound bites, but little consumable pieces of like insider knowledge that they can walk away from and actually remember. So I think I had to work a lot on just my own presentation style and kind of lighten up and be more fun with it. And I think that I can still get across strong data messages. But when I do it in a lighter way, it resonates more deeply with my audience.
[00:16:56] ES: You mentioned that you came from a medical and academic background. I saw you have a background in psychology, which is the study of the mind and behavior. Have you found some similarities between your academic and medical background? And what you do now as a user researcher of a global platform?
[00:17:19] SH: Yeah, totally. So my background is in psychology, which I think is a really good underpinning for any user researcher. So if anyone is in that field and interested, I think it parlays really nicely to the user research field. But I was always research based, like it was never going to be a therapist. I always was very research specific. And I do think that psychology offers you lessons about people in general, and especially about their own mental models and the motivations behind their actions. So this is especially really helpful when you think about human bias, which we’re all subject to, we’re all very infallible. But there’s all kinds of different bias. The recency bias which is the tendency to hold on to the last thing you heard or the bandwagon effect, is you believe one thing if everyone else is doing it. And there’s all kinds of these little traps that people get sucked into. And so I think when you’re considering how to do a design that is the most usable, it’s really helpful to understand the way humans think, in general. And then I ended up weirdly working for UCSF and going into medical research just for a little bit, because after graduate school, I love San Francisco and I knew I wanted to live here and I was just looking at research jobs and stuff like that. And they were doing just some incredible cutting edge biomedical research. And so I just went to a few lectures, went to an interview and just thought I’d give it a try. And I was like, hey, it’s data, I can do data. I’ve been a researcher. I think that it was really good to train me and cut my teeth because medical research requires a very high level of precision. So really rigorous protocols. In a challenging recruitment, even analyzing qualitative data, we would use like complex software’s like Atlas to carefully, quantify qualitative data and code it. But it took years, it took so much time. I really had to then adapt those strategies back to user research when I joined tech because you innovate much faster. And you don’t have a year, five years to spend on the study, you have to be much quicker. But I do feel good that I built this like very solid foundation in research with some more traditional methods. And then ironically, UCSF actually paid for my path to user research because one of my doctors when I was in sports medicine, Dr. Anthony Luke offered me a job as a user researcher and his new startup called RaceSafe, which is a medical technology that allows race organizers and medical staff and stuff to talk to each other and be able to look up patients really quickly if they have an incident during a race and so That was my little bridge way into user research. Because honestly, these days, it just comes down to people and data and how people use their medical devices is not that different than how they use their streaming service platform. It’s all you know how people interact with technology.
[00:20:17] ES: Exactly. I feel this is very important to notice, especially if we have listeners, working in other fields, but they might think, Oh, well, I didn’t study computer science, or I’m doing medical research. Essentially, you’re gaining a set of skills that are applicable to other industries. So if you wanted to specially work more closely with the tech industry, the skills translate really well in a lot of cases. Right? And you’re explaining one of them.
[00:20:49] SH: 100% Yeah, we just hired a new data scientist and she comes from a health background, but she understands people in data and how people interact with their own data there on devices. And yeah, I don’t see any kind of barrier there. The fact that she maybe isn’t an anime expert or something, researchers research.
[00:21:09] ES: That too. I found that I’ve talked to people where they are interested in working for video game companies or products. And they’re like, Well, I’m not a gamer. And people tell them no, well, all we need is exactly what you’re describing people that can understand data problem solving. So there’s no real barriers there, I think.
[00:21:30] SH: Definitely. And you’ll learn and sometimes, not being really into the content that you’re studying actually gives you a kind of a more unbiased approach, because you don’t have strong feelings either way, you’re just looking at the data. And for someone who is like a data scientist that can be really good to like, be very unbiased.
[00:21:48] ES: Definitely. Well, Sarah, thank you for coming on the show. It’s been great talking to you about user research and Crunchyroll.
[00:21:56] SH: Oh, thank you. This is so nice, and I’m so glad you’re doing this. I can’t wait to listen to more powerful women more of your episodes. It’s been really fun.
Sponsors
The Women in Tech Show 