Website Logo. Upload to /source/logo.png ; disable in /source/_includes/logo.html

Chris Meier

Software Developer and Co-founder of FoodChain

Beta Testing Part 1: Organization

At FoodChain, we have recently finished our beta test and you can now download the app on the Google Play and Apple store. We learned quite a number of lessons during our testing phase and I would like to share them so that your beta tests can go smoothly.

Before we dive in, I want to highlight the importance of beta testing. When you near the final stages of development, it is only natural to want to share your app with the world. WHATEVER YOU DO, DON’T DO IT! I’ve chatted with other startup founders (usually non-technical) who immediately pushed their app to the stores and it did not work out well for them. They didn’t have a chance to work out any kinks and immediately plunged their users into a sea of bugs. Naturally, their apps were given poor reviews and they had trouble retaining users. And it’s because they didn’t beta test. When you launch your app you need to put your best foot forward and release a quality product, which beta testing helps with.

Cohorts

Your first instinct may be just to fire out the app to everyone and let them go at it. This is a bad idea since your app is still buggy (more than you think). We decided to split our testers into cohorts of increasing size. You can have as many cohorts as you want, but it is better to do many cohorts that grow by small increments than ones that increase in size by large numbers. The best part about cohorts is that you can better improve your app. Bugs will only affect a small number of users rather than a whole bunch. When your app starts to reach more beta testers it will look more polished and they will have a better impression, which will help in converting them into a regular user.

Depending how honest your friends and family are (ours were brutally honest), you should reach out to them as your first cohort. They hopefully know what you are working on and are probably pretty excited to get their hands on your app. They also know you. The biggest boon for us was that they weren’t afraid to contact us with a bug (and give us a good ribbing). This led to faster turn around on bug fixes. The most difficult part, since they were non-technical, was getting detailed bug reports from them. But talking to them for a bit got us all of the information we needed. We slowly expanded our cohorts to more friends and family until we ran out and entered into an open beta phase. This was our final cohort that continually grew without being limited by us.

Lesson: Incrementally increase the number of testers that will get your app so that bug fixing and bug management is easier.

Signups

For any beta test, you need users. We were in a bit of a stealth mode for our testing where anyone could sign up but you had to find us first. We have a form on the site where you can submit your name, email, and phone platform and we would send you the app. This obviously doesn’t scale well, but it fit into what we wanted to do.

As we neared our soft launch, we became a little more aggressive in our marketing. To get more beta testers we submitted our site to BetaList as well as posted on the startups and sideproject subreddits. This drastically increased our number of users, many of whom were very helpful. By this point your app should be pretty polished and be (almost) ready to be released into the wild. This part of the test is mostly to catch bugs at scale and to test performance on a wide range of phones.

Lesson: A stealth beta can help with controlling the trickle of testers, but get more aggressive in recruiting as you near the beta’s final stages.

Keeping in touch with users

When beta testing, it is easy to get lost in the seemingly endless stream of bug reports. However, it is very important that you keep up with your users. Keep them informed of what is going on internally and most importantly, let them know how thankful you are for their help. They will feel loved and know that you haven’t forgotten about them. The latter is important since they will know that their hard work is meaningful. One good rule to apply to communication is to keep it short. We didn’t do this initially and we found that (obviously) nobody is going to read a wall of text. Short and sweet with judicious use of white space makes effective messages.

Lesson: Be in contact with your users. You love them and they will love you.

In this first part we covered how to organize a beta test. Use cohorts of increasing size as your testing group. This will make managing the test easier since you won’t have to deal with a firehose of bugs and feature request. To tie in with cohorts, you should manage your signups. It’s important to have the number match the stage of your test so potential testers don’t forget about you and lose interest. In the next post, we will talk more in-depth on the logistics. It will cover app distribution and bug submission.