In this second part on running a beta test, we will focus more on infrastructure and logistics. The whole purpose of a beta test is to find bugs in your app and fix them. We will look at how to distribute the app to testers and handle their bug submissions. How you set these up will be key in setting the tone of your beta. If done correctly it will be a smooth and enjoyable experience. But if done incorrectly it can become a bit of a nightmare.
This is a huge aspect of your beta test. If testers can’t get your app, then it won’t be tested. For app distribution we used Crashlytics for Android and TestFlight for iOS. We initially used Crashlytics for both, but we ran into problems for iOS so we switched to TestFlight. A major benefit of TestFlight is that using will reduce the amount of time it takes for your app to be approved for the Apple Store.
One thing that we did was to automate the app distribution. This cuts down on errors when tracking who has the app. We set up a mailgun account that would automatically email a distribution link when someone signs up. They would click the link and they would receive instructions on downloading the app. This made installation incredibly simple. It also ensures that everyone who signs up gets the app. Doing a manual distribution is very prone to human error (we found this out the hard way) and you want to avoid this at all costs.
Crashes were easily handled with these frameworks. A fair warning, there will be a lot of them. We use Slack for team communication, so we naturally took advantage of Crashlytics’ Slack webhook. Anytime there is a crash, we are alerted with a full report of what happened. The reports will give you the activity, the error, as well as the line that the error occurred. This was a boon for us since we were able to greatly reduce the turnaround time on bug fixes. There was also an automation aspect. Emails are sent with these reports but it is only to the account holder. By using Slack webhooks, our development team was able to get the report immediately upon the app’s crash.
Lesson: Use a beta testing framework. This will make your life easier. You can easily get crash reports and automate app distribution.
Bug submissions were probably the hardest thing for us to manage. Mainly because we didn’t have a good framework in place. We did use Crashlytics and TestFlight to report crashes, but bugs that didn’t cause crashes were harder. Our friends and family would just text us, but it was harder for testers who didn’t know us personally. We had a bug submission form on our website that sent one of us an email with the submission. This was horrendous for tracking bugs. We did miss a few along the way that were fixed much later than they should have been. There was also the issue of the ease with which a user could submit. They had to go to our website and do it. It would have been much easier to put a form in the app itself to solicit feedback. These problems led me to create a tool to make this process easier. It is called Clover (blog post coming soon), which is a simple web app that connects to Github. A tester can submit a bug and it will automatically be opened as an issue in that code base’s repository. This way you can easily track bugs and feature requests.
Lesson: Come up with an efficient system to manage bug submissions. Minimize work for your testers as well as yourself. Once again, automation is key.
This wraps up the short series on beta testing. These issues were the ones that we at FoodChain encountered most prominently. You will definitely encounter more issues, but solving these ones ahead of time will allow you to focus on other problems. The biggest take away is to automate as much as you can. Just set and (not totally) forget. It will free you up to worry about the things that matter instead of the small stuff.