Bringing you Tips on Running....

A|B Testing | Optimization|Experimentation

  • Experimentation Expert

COVID-19 Takes over Experimentation in 5 steps

Article Level: Beginner

Article Type: Opinion (non factual)

A virus is a biological agent that reproduces inside the cells of living hosts. When infected by a virus, a host cell is forced to produce thousands of identical copies of the original virus at an extraordinary rate.

Much like a virus, experimentation took over the digital worlds of banking, retail, healthcare, technology and many other corporations within the last decade. No more remedial banner and CTA tests! We have taken social patterns and psychological impulses to another level by attempting to map data to them. But not just any kind of data, we are taking a very complex behavior from a species and are attempting to assign a few ancient business KPI's to complex actions. All with the hopes that we can create some sort of conclusion out of this chaos we call data.

It [experimentation] is not introducing the scientific method to business, but instead it is merely acknowledging it. I'm sure the data scientists are screaming "We've been telling them for years!" , as these oddly organized experimentation teams are stood up within their very cubicles. But I am not here to speak on those topics just yet. Today, we are living in a COVID-19 enflamed society. This pandemic has become a catalyst to rumored company layoffs, hiring freezes, store/restaurant/small business shutdowns, philanthropy initiatives, government finger pointing, racism, panic, fear, rage, and likely more in the impending aftermath.

As companies and families struggle to define the new normal, they are watching the world fight over who's data is real, who's data is the most reliable, up-to-date, and most of all PREDICTABLE! The just wan to stay alive. Some of the worst outbreaks and pandemics we've seen are what one would call the population source for our parameters. So when addressing predictive analyses for a thing like the corona virus strand we describe the data as "repeated history". The point is, regardless of how it is being measured and analyzed, projections are saying that we have a bell(gaussian) curve ahead of us. The question is, are we at the peak and on the downward slope?

Enough of the chatter and on to how you, the optimizer, experimenter, and leader of your AB testing team can take advantage of these times to bring some surprising or not so surprising insights to your business leaders. Out of all times for you to be the unforgiving unbiased experimenter, now is the time, so that your leaders can truly make data driven decisions.


What's our goal here?


To understand how our customer trends and predictions change in a health crisis. Your hypotheses for the observational tests ahead should be set to align to some foundational KPI to that goal. One experience should be business as usual while the other is directed towards current events.

Note: These will be "observational" because we are running them for the duration of this pandemic and depending on the type of business, you cannot afford to run this type of experiment at 50/50. Scale the test down to align with minimal impact per your business type and relevance to current customer needs.


How much traffic are we working with?

Ask your leaders if there is a percentage of your site that you can run these alterations on? What makes this a slightly elevated observational test, is that the content and products you serve will be shifted towards the emotions of the customers. If you are familiar with behavioral metrics and behavioral metric clusters, you will know that this will require that you track economic events on a timeline to address the deviations in the metrics for both experiences. Meaning, high impact announcements like "all gyms must close for 4 weeks" will serve as a milestone to observe drastic behavior changes in KPIs.

I would suggest a percentage of the site that will allow you to reach a comparable sample size for your business over the next 8 weeks . Pay close attention to the news and reliable scientific sources. Dedicate a small portion of your team obsessed with keeping up with current events. This information will need to be organized into a searchable timetable.This will serve as our comparison spectrum at the end of the tests.


What is your business?

Stick to what your business knows BEST

Let's choose three basic tiers: RETAIL < BANKING < HEALTHCARE

For the sake of this article, I will focus on a few mediums.


What is your KPI (Key Performance Indicator(s))

Retail Example Goal: Refrain from falling below xx% revenue during 8 week period (you will take a loss form this, except it)

Banking Example Goal: Decrease large withdrawal rates amongst loyal customers

Healthcare Example Goal: Provide a larger sample of data for post statistics, using the help of social media to then serve back to the public.


Decide your testing strategy?

Now this can be exciting. Remember these are non-factual opinions and suggestions that i have not researched in depth. Open dialogue and inspiration is the key.


Design your tests around emotional content, longterm product investments/insurance, and products of near/future necessity. You will be boosting your NPC [Non Product Content] to educate the public on the everyday things we are struggling with. Observe your new customer entry points. Center tests around boosting products for "household order", cheap DIY projects for the lockdown with product bundles, and blogs on how to ensure their projects go uninterrupted when/if life picks back up. Marketing should be focused on preparation tactics for the aftermath. Will customers be concerned with finishing their house, finishing their smart home security, buying new furniture, and switching up their wardrobe?

Example Fashion Company:

Test work from home cozy wardrobes, and encourage friends to meet up to work together to get through the long silent hours. You can still be glammed up for that virtual interview or conference. You never know who's watching!

Example Grocery:

Increase your visibility on social media for delivery. Post meal packages that support healthy recipes. Post real reviews from customers via their personal submitted videos. People like to know it's real. Or create super savor packages for items left on the shelves that when combined make for great small group meal gatherings. <insert cheesy millennial photos>

Example Hardware:

NPC on how to fix things around the house to make your home conducive to a work from home environment. With encouraged buy online "pick up in-store" or delivery to your vehicle, you can increase sales in easy DIY projects that eat up the time spent at home during these 8 weeks. "Redesign your home office you never had"


People are losing jobs, and the security of their funds for the far future are now becoming serious table talk for the near future. How do you comfort them on irrational financial decisions. Encouraging a small loan for someone with no clue on their job security could actually set them up for failure if they can't afford to pay it back. Unless you're a shark taking advantage of this weak time for people. This could be a causality for distrust in the near future and cause bank hopping.

Instead, focus on financial calculators, equity predictions and financial education. The idea is to discourage drastic withdrawals. For your plan, consider a targeted audience of faithful/longterm customers who may need that emotional reassurance. Crunch the numbers on how long it take a customer to trust your bank to understand the probability of the potential loss of customers who simply need to empty their savings and take high risks for their well being. STAY EMPATHETIC.


The education on COVID-19 is abundant, whether it's right or wrong. So you will be testing an experience dedicated to self help quizzes. The compliance that comes with healthcare data is at one of highest levels of security. So allow people to understand the probability of risks. The current age interval charts that show 0.2 < +8% mortality rates come with multiple layers of variance depending on the patients current health and family health history. Your results will not be immediate, but more of a post analysis for the public. Encourage the public to take this quiz via instagram, facebook, or your website. For example:

Hero Banner Quiz 1:

Take this 5 minute Health quiz! "You're World Depends on it"

(make sure your "inclusive" in your verbiage. Be sure to call out data that must be optional)

  1. Do you have healthcare?

  2. What age are you?

  3. What race are you?

  4. Do you have any of the [a:zz] diseases, ailments, etc. listed in this table (do not have them checkbox, this makes the quiz too long. Remember the point is to collect information from everyone to increase your sample data to not just be from the ill)

  5. How often do you consult a doctor for [a:zz] symptoms?

  6. What at home methods do you use when you are sick?

  7. How many people are in your household including you?

  8. Do you travel internationally? 3months etc.

  9. Have you travelled domestically in the last 3 months,6 months etc.

  10. Has anyone else in your household travelled in the last 3 months. 6 months etc

You get the point. Then make a Facebook filter that says "I took the health quiz!"

To toss around more ideas, subscribe now for exclusive publications!

Thank you and feedback is greatly appreciated.


Experimentation Expert

26 views0 comments