Telenor Blue – Google Optimize A/B test case study

One of the main business focuses of Telenor is to increase the share of online channel among sales of devices and subscriptions. In order to reach this, they simultaneously work on developing and improving capabilities, traffic management, and conversion rate optimization.

INTREN is providing various digital services to Telenor, including Google Analytics and Google Tag Manager maintenance, management of PPC advertisements, reporting, and conversion rate optimization with various tools. This particular A/B-testing was focusing on the flow on online sales, which often start on tariff pages.

Telenor has been already using Google Analytics across their website. The current flagship product of Telenor is Blue tariff, which has its own product page, and it’s the second most used landing page, besides the main page of their website.

One of the insights gained from the page navigation reports was, that visitors preferred to buy devices along with tariffs, and therefore rarely used so-called SIM-only related CTA buttons, where no device is purchased. Also, the majority of visitors don’t select featured devices, but rather look for device related information in the main navigation, which leads to unnecessary navigation steps, or above-average exit rate on tariff pages.

“The hypothesis we’ve built from this, is that we need to make it easier to find a device to a selected tariff version, which will finally lead to more online sales transactions.”

We assumed, that if preferred options are easier to reach on the page, then number of sales could be increased, and share of users, who quit the page, could be decreased.

We have chosen Google Optimize to test our hypothesis. This tool supports changing relevant elements on the web page, without the need to alter the base code of the website, so testing can be started quickly and easily. On top of this, by integrating Google Optimize and Google Analytics, we are able to keep track of the whole sales journey, and to monitor actual sales transactions.

“During our test we were able to decide if our hypothesis had been correct. The test had run for 3 weeks, during which we were monitoring the visibility and performance of different variations. By the end of the test, we could compare the test variations, and Google Optimize could easily announce a winner.”

The test was running for 20 days, and reached 94% probability for the tested variant to be able to outperform the baseline, by 19%, considering the main target KPI, which was ecommerce transaction. We have also been able to verify in Analytics, using experiment and variant dimensions, that customers were more likely to progress through the sales journey step, like selecting device for the tariff, and filling the checkout form.

As formed in our hypothesis, tested variant helped customers to reach device lists and device product pages, compared to original version, where they relied on page navigation, resulting lower exit rate on the page involved in this testing. As a longterm effect, tariff and device recommendations are structured in a different way. Although both products are featured along the sales flow, device recommendations are now better differentiated from tariff selection, which earlier could have been perceived as a limitation of available bundles, and divert customers from reaching the overall device offering.