ANLY FPX 5510 Assessment 2

Executive Team Places Analytics Department on 90-Day Probation: Business Strategy at Stake

In the last few months, the executive team has been making crucial business decisions based on numbers presented by the analytics department. While some of the decisions have been successful, others have seen enormous sums of money lost. The analytics department has thus been placed on a 90-day probationary period. At the end of this time limit, the company leadership will decide whether the department is to be disbanded or continued. The head of marketing has been instructed to oversee the performance of the analytics team during this period, and all work will be thoroughly checked before being presented to the executives for final approval.

Business Problem: Evaluating Revenue through A/B Testing

To ascertain business performance, the company has chosen A/B testing as one of its metrics. This is achieved through comparing revenue gained per visitor from the same banner advertisement placed on two websites—Yahoo and CNN. The goal is to determine which website gains more revenue from the firm’s advertisements. This metric is targeted at streamlining advertising efforts and reducing the use of third-party service providers.

There were two hypotheses in this A/B test. The null hypothesis was that the average purchase price of Yahoo and CNN was equal, and the alternative hypothesis was that there would be a significant difference between the two sites in terms of purchase values. The test showed that there was a big difference between customers’ purchase actions, with Yahoo customers being highly receptive to the ad versus CNN customers.

Testing Methodology: Applying Hypothesis Testing to Business Strategy

The test method of choice is quite much in harmony with the business issue at hand. Just as A/B testing assesses ad performance between two websites, the same approach will be used to review the performance of the analytics team while under probation. The intention is to determine the value of the team based on how well they can create accurate, actionable insights.

Throughout the 90-day probation period, the analytics team will be put to the test using a formal A/B testing process. The marketing department head will formulate two hypotheses: the null hypothesis that the team will deliver acceptable results and the alternative hypothesis that the team will deliver below-average results. The outcome will determine whether the team will remain as is or will be dissolved.

ANLY FPX 5510 Assessment 2:A/B Testing: A Proven Marketing Experiment

A/B testing is an effective marketing tool to compare differences in campaigns to identify which version performs best. For example, one segment of the audience views version A of a marketing element, and another segment views version B. The goal is to determine what content produces the optimum response. To develop a successful A/B test, businesses must come up with two versions of the same content that differ in only one element and then observe how these two fare across two groups of audiences (Kolowich, 2019).

Requirements for Implementing A/B Testing in Business

A/B Testing Requirement

Research

Perform research on past performance and determine the variables.

Hypothesis Development

Develop hypotheses on future performance based on past data.

Test Implementation

Develop variations and implement the test over a sufficient time frame.

Data Analysis

Maintain proper representation and avoid biased sampling or length pollution.

Ensure proper representation and prevent biased sampling or length pollution.

Adequate A/B testing should be followed by adequate research and hypothesis creation. Research should be focused on the accuracy and validity of previous analytics reports to establish hypotheses on future performance. Having set the hypotheses, the next step is to create a version of the control version and perform the A/B test. Performing the A/B test within the 90-day probation period will allow the company to determine if the analytics team will be able to make a positive impact towards the company’s strategic goals.

However, certain potential problems such as biased sampling and length pollution are likely to occur. Biased sampling is when the testing sample does not represent the entire population, who may give inauthentic outcomes (Saleh, 2019). Length pollution is experienced when an A/B test stops too early with the outcome not being sufficient data. For top-traffic pages like Yahoo and CNN, it is advisable to test for a minimum of two weeks to ensure more authentic outcomes (Saleh, 2019).

Addressing Potential Issues in A/B Testing

Problem

Solution

Biased Sampling

Conduct tests regularly over a single or double business cycle.

Length Pollution

Decide on the test length to obtain enough data.

Rater Bias

Apply objective measures and provide bias-reduction training (Joseph, 2019).

In order to counter issues like biased sampling, businesses should test continuously across multiple business cycles to ensure proper representation. To avoid contamination by length, it is necessary to determine the correct length needed to get enough data before closing the test. On popular sites, imposing a conversion minimum (e.g., 1,000 conversions) will strengthen results. Objective performance measures can also deter rater bias, which can skew measurements of the analytics team’s performance (Joseph, 2019).

Conclusion: Deciding the Future of the Analytics Department

The next 90 days will be determining the future of the analytics department because their performance will be tested via an A/B testing paradigm. The executive group will decide to retain or destroy the department based on the results. The team’s work will be strictly checked by the marketing manager to ascertain whether it is up to the company’s quality and accuracy standards. The probationary period will provide feedback on the team’s potential to add to the company’s strategic goals.