Artificial Intelligence
Consulting
Development ServicesMany clients that turn to Intersog for development of their custom software solutions underestimate or completely ignore the role of focus group research for general proof of concept (PoC) and usability testing. As we tend to engage each client in an interactive workshop to spec out the project together with our PMs and BAs prior to signing any work orders, we always stress on the importance of future-proofing application prototypes for user feedback and UX evaluation. We demonstrate on real-life examples how a proper target user research has helped our other clients save their development costs by avoiding creeping featurism, fixing issues early in product design, and better managing and minimizing risks.
As a result, many IT decision makers looking to build their solutions with Intersog buy into the idea of doing comprehensive UX research with focus groups.
In this article we'll shed light on how to determine the right target audience for usability testing and gain maximum value from it.
Who to involve in usability testing
The most important yet difficult thing is to determine who to involve in your focus group research. In any software development project, it's as important as writing a script in moviemaking. If you show your application to a software developer and your Mother, you'll most likely get two totally different opinions about it. Although we presume Mom will have more issues with usability, it doesn't necessarily mean that issues she'll point to are really those critical issues you need to fix in the first place! The reason is simple - your Mom is just not your app's target user!
That being said, don't focus much on user demographic. Oftentimes, when Intersog takes on a new project and discusses target users with the Product Owner, we get profiles that sound like: "60% of users are women aged 25-35, obsessed with fashion". Although this data is valuable, it's not sufficient to form an appropriate research sample. Let's say you want to test usability of a new custom built Gmail extension that allows for design of beautiful email templates. What would most companies do? They'd invite different Gmail users distributed by sex and age. Eventually, it turns out that some of these people rarely or never use their Gmail box, the others use their Gmail for subscriptions and spam only and the third don't use Gmail as their main mailbox. As such, the experience of most of these people will make no sense for objectivity of our usability testing results. It doesn't mean you should completely ignore demographic data, just don't use this criterion as the key one.
The key parameter for recruiting users for usability testing is experience with similar software products OR similar features. Based on this, we can distinguish three groups of target users:
- experienced users
- users with similar experience
- inexperienced users
Let's say we need to test an upgraded UX design of an online TV app. Who should we involve in research?
- Users of the current app version to make sure they navigate seamlessly and can perform well their habitual scenarios in your app after design upgrade
- Users of competitive online TV apps
- Old school guys who keep watching TV in a traditional way and don't use any online TV apps.
More than just product experience!
For successful recruitment of a focus group you need to clarify what you mean by "product user". The more detailed personas you create, the higher the value of your usability testing. First of all, think about the activity (page view depth, average visit duration, use of different features, etc) and product use peculiarities. For instance, in the above example with an online TV app, you may find it important to recruit people with different TV interests: soap opera lovers and sports show fans will undoubtedly have different requirements for the interface.
Beware of inexperienced users!
It's very important to track how inexperienced users interact with your software product, as it allows for simulation of a new product introduction to a target audience. As such, you need to be apprehensive when selecting inexperienced user audience. In the best case scenario, these should be not just people who don't use your product, but people who may potentially use it in the future.
The bottom line: if you invite people with no interest in features provided by your app being tested, they'll most likely test your app mechanically, just as a tick-off. It means their value will be low. Let's say you want to usability-test an app that cleans mailboxes from spam. You should focus on users who suffer from piles of unsorted emails they receive from their hundreds of subscriptions. If you have users test your app on their real mailboxes, you automatically increase the value of your focus group research!
Besides experience and demographic, you should consider:
- Mobile platform
Even if you have a hybrid app with the same interface for different platforms, you should still test it on each particular platform users. As the practice shows, issues that arise on iOS and Android are completely different and each platform users have different habits and UX requirements.
- Proficiency
Five years ago we recruited users for focus group research using terms such as "computer literacy" and "Internet experience". Today, the average Internet experience has more than doubled and there're many digital native millennials to take into consideration. So, just skip this criterion when usability-testing web services. However, when it comes to mobile services, proficiency matters a lot! Speaking conventionally, the average proficiency includes at least the ability to download and install an application without any external help.
And what about involving friends and colleagues to usability testing?
At Intersog, we had clients who said: "Our app is for absolutely everyone! Let's not waste money and have our colleagues test it!" While this idea seems cool as you can perform your focus group research for absolutely free, it's not that good in fact! Here're the pitfalls of this approach:
- Your professional network can be too advanced for providing a level of feedback you require. When some clients tell us, "Let's have your HR and marketing test our app", we reply that even non-technical personnel in our organization is quite advanced in terms of technology use, so they may provide feedback you wouldn't normally expect from inexperienced user group!
- If you're testing your new app among colleagues, do remember most of them can be your brand loyalists and think the following way: "I know the guys from that team, they did a great job. Well, this feature does't work, but we had the same issue on our project and I know how hard it is to resolve it." On the other hand, your colleagues can be too demanding and nagging, as their expectations are high. The same applies to friends. Some won't tell you the truth out of fear to offend you, while the others would want to help you so much that they'll scrutinize every single feature and find errors where there're no errors at all.
- Avoid involving subject matter experts in your focus group research. People can hardly abstract away from their own experience and knowledge and, thus, will not be able to assess your product as an average user. For instance, if you have a mobile developer test your mobile app's UX, they'll check your code quality, adherence to guidelines, app load speed and other technical KPIs. A marketer will scrutinize your in-app banner space, while game producers will evaluate how your game compares to competitors. As a result, you'll get an expert assessment instead of target user feedback! While such assessments are useful and valuable, they can significantly corrupt your focus group research results.
Despite the above, it's still better to at least test your product on friends and colleagues than not to test it at all.
What's the perfect sample size?
One of the most burning questions when it comes to usability testing is how many users to involve. Twenty years ago, usability guru Jacob Nielsen grass-rooted a "just a handful of users" concept and claimed five users were enough for efficient focus group research. This concept has run deeply in people's heads. Regardless of the research objectives, many product owners still believe a handful of users would suffice to evaluate their product. On the other hand, there appeared a lot of sceptics who refused to recognize results of usability testing performed on such a small sample.
Let's figure out first why a 5-person team is considered to be the perfect size. As the practice shows, the number of detected app issues decreases after the first five users have tested it. According to Nielsen, a 5-person sample allows detecting 85% of all usability issues. As such, involving more people may just not be rational.
Would you like to learn more
about talent solutions
In fact, those 85% Nielsen referred to relate to the most serious and frequent UX issues that 31% of users usually face. If we look at any interface, we'll most likely discover issues most of users are facing on a regular basis while using similar products (e.g., a broken contact form or a failed login to the game dashboard) as well as some app specific issues that are less frequent, but than can be very critical. As such, 5 people will discover only 50% of all issues faced by 15% of users! The larger your research sample, the higher the chance that truly critical issues will be found.
In 2012, Jeff Sauro, Principal at MeasuringU, polled over 100 blog readers aiming to figure out the perfect size of the usability test sample. His poll showed that over 81% of usability tests are based on 5+ users (median - 10).
What does focus group research sample size depend on?
5-person group - this size is perfect for prototype testing when you need to find the most critical issues, but don't have much functionality to manipulate with. However, you should remember that after prototype usability testing you'll have to conduct more usability tests to make sure all issues are on your radar.
Also, 3 to 5 person samples work well for iterative testing, i.e. when you test features, send app back to dev team for bug fixing and then test again. Apparently, it's the most effective way of usability testing if you have a dedicated team of product testers.
More than 5 people are needed in the following scenarios:
- User diversity
Are you really going to expect a diversity of experiences, interaction patterns and demographic within a 5-person team? Diversity that is required for objective results of your usability testing! In order to understand how many people you need on your test team, make a list of all of your requirements for the target audience and then recruit at least 3 different users per each requirement.
If you need to usability-test your hybrid mobile application, you'll need two groups of users (iOS and Android users) and two experience-based groups (newbies and users of similar apps). In this case you need at least 12 people to perform an effective usability testing.
- Testing scope
One test allows for running a limited number of scenarios. A proper usability test should last for an hour or so, otherwise users will get tired and bored. So, if our goal were to test the entire Facebook interface, we'd probably divide the project into several sub-tests to ensure adequate testing of feed functionality, images, video, groups and other interface elements. Make sure to include the most critical scenarios into the testing scope and have at least 3 users test each of them.
- You outsource your usability testing to a 3rd party provider like Intersog
If you're about to release a new app and outsource its UX testing to an external consultancy, don't skimp on the number of respondents to involve. While this will add to your budget, this will also improve the quality of your focus group research.
- Your goal is to better explore your target audiences and their behavior
Focus group research aims to not only discover bugs and errors in app's interface, but to collect more data and insights into user interaction patterns, app use cases and more. For this purpose it's highly recommended that you expand your research focus and its diversity.
More isn't always better!
While there're many arguments supporting the idea that the more people on the research team, the higher quality the results, don't go to extremes! Do remember that we're speaking about qualitative lab research. For such research projects you'll rarely need more than 24 testers. On average, 10 to 12 people would suffice for your focus group research!
Myth-busting
"It's not a serious research anyway, why pour money here?"
Although we know now that "just a handful of friends" isn't enough to perform usability testing effectively, UX remains primarily qualitative, which faces a lot of criticism among product owners. Most of them just don't want to accept the fact that they'll have to allot a separate budget for making strategic improvements to UX/UI design based on the feedback from a few random guys.
If you ask ten people what should be changed in this door design, you'll most likely get ten different answers. But if you ask them to enter a room through this door, you'll be able to see right away whether it should be pulled or pushed to open.
Remember once and for all: UX research isn't about people's opinions as much as it is about their behavior! If you need to learn someone's take on your subject matter, you need to ask as many people as possible. But if you want to find your app's interaction issues, even a small number of people will let you identify patterns and peculiarities typical for your target user base.
"What if your respondent is a moron?"
Would you like to learn more
about talent solutions
When one user can't handle a task or demonstrates illiteracy in a certain aspect, your dev team overseeing the test can think something like, "What if he's an idiot, but others aren't?" The answer is simple: each pattern in question should be tested on other people. The more people you involve, the better you'll understand that many users behave like idiots. We were really shocked to find out on one of our usability tests that some users don't even know how to copy an URL from a browser! To share a blog post, many copied the entire article, did screenshots or just copied the name of the article. When we saw the first user facing this issue, we thought it could be his personal lack of experience, but then we saw more and more users making the same mistakes and we realized the inevitable truth - some people are weird and spaced-out and you just have to face it!
