ONA Weekly #365: Ask The Right Questions For Your A/B Tests

By on June 23, 2021

The A’s and B’s of asking the right questions

A/B tests and surveys are extremely useful for collecting user feedback. Many newsrooms use A/B testing to gather data on headlines that work—for the Boston Globe, the removal of the word “bad” in one headline led to 12% more clicks. The Huffington Post has used A/B testing to determine where to place social share buttons and Vox used A/B testing on their StoryStream feature, which collects articles on a topic, to distinguish read and unread articles as a method of promoting engagement.

There’s plenty to test. But first, it’s important to make sure you’re asking the right questions and optimizing for the right thing. (In other words: it’s not always about click-through rate.)

Sophie Ho and Angela Wong, who both worked at the Washington Post, led an ONA17 session about how to make the most of A/B testing:

  • Determine the type of data you’re looking for: Are you looking for qualitative (comments, etc) or quantitative feedback (time spent, return visits)? What information is actually useful?
  • Be able to answer the question of who your ideal audience is, what the existing behavior is and what new behavior you’re trying to incentivize. 
  • Make sure you have a hypothesis and are testing that hypothesis, not just collecting information for the sake of it. Make a prediction and see how the data bears that out. 
  • Try the “five whys” method—asking “why” five times will help drill down to a problem’s root cause. For example, “Why would you want to use email?” and then keep asking “why?”
  • Use Optimizely’s sample size calculator to help you figure out how long you’ll need to run your test

Dig deeper: For more on the joys of testing, check out How To Grow Your Reader Revenue by Testing Faster and Better: Lessons from the Facebook Accelerator — one-hour recording from ONA20
Featured experts: Duc Luu, Sonya Quick
Takeaway: Case studies from the Facebook Accelerator’s Membership community show that small publishers can use nimble testing to achieve steady growth without expensive technology.

We’re always on the lookout for helpful resources and tips. If you have other examples to share, please reply directly to this email.


Take a Survey, Earn $10

How arlogo of Penn State College of Communicationse our intuitions linked to media values? Can our moral judgments shape better digital media policy? Spend 30 minutes to participate in our study here, and help us find out! Questions? Contact Dr. Patrick Lee Plaisance: plp22@psu.edu

ONA21 is under way!

ONA21 kicked off on Tuesday, but there’s still time to join us for three more days of inspiring sessions and networking opportunities. Registration is open. A conference pass will give you access to the remaining live sessions (see the full schedule here) plus on-demand recordings.

Here’s a peek at some sessions on the schedule today:

And don’t forget you can share your resume with the recruiters at the conference. See you on the ONA21 platform!

Stat of the week

According to the latest report from the Institute for Nonprofit News, the median revenue from individual giving to nonprofit publications increased to $118,000, a 41% gain from the prior year based on respondents with comparable 2019 and 2020 financial data.

Important dates

Note: All ONA events are currently happening online, and open to all regardless of where you’re located.

Career opportunities

ONA’s Career Center is an excellent resource for jobs, fellowships and internships in digital journalism. Recent postings include:

On our radar

The radar is now open for suggestions. Have you written or read a piece lately that would benefit others in journalism? We’re interested in sharing insights on trends, how-to guides, lessons from a project and other inspiration for innovation in digital journalism.
IMAGE CREDIT: Photo by HalGatewood.com on Unsplash
Join as an ONA member or make a donation to help sustain ONA’s mission to inspire and support innovation in digital journalism. If you’re looking to promote an event, product, job opening or other opportunity, explore options to reach the ONA community.

We also rely on philanthropic and corporate support. Contact Chief Strategic Partnerships Officer Jessica Strelitz at jessica@journalists.org to learn about the many ways to invest in our mission.