5 Optimization Discoveries from the SAP Website Test Lab Shawn - - PowerPoint PPT Presentation

▶

Apr 19, 2023 134 likes •527 views

5 Optimization Discoveries from the SAP Website Test Lab Shawn Burns Brian Carroll Global Vice President Executive Director, Digital Marketing MECLABS SAP Session Speaker Shawn Burns Global Vice President Digital Marketing, SAP Black

SLIDE 1

5 Optimization Discoveries from the SAP Website Test Lab

Shawn Burns Brian Carroll Global Vice President Executive Director, Digital Marketing MECLABS SAP

SLIDE 2

Session Speaker

Shawn Burns Global Vice President Digital Marketing, SAP Worldwide lead for SAP Digital Marketing, including

nline demand generation, search media

partnerships and our flagship website, SAP.com and its 72 country sites in 40 languages. Shawn is based in Paris, France with his wife and three sons.

Black & White Headshot

@shawnpburns

SLIDE 3

“Web Analytics is a table stake and it doesn’t get you very far”

SLIDE 4

“New People, new skills, and new attitudes are required”

SLIDE 5

“Testing without a process is driving without roads – there is no point to even having a car”

SLIDE 6

SAP Test Lab:

3 years in the making
140 tests launched

SLIDE 7

Some of the results…

SLIDE 8

5 Optimization Discoveries from the SAP Test Lab

Photographic imagery sharpens response Great results don’t have to be complex Make those CTAs easy to spot (and click)

2 3 4

Search is (even) more important than ever

1 Test surprises happen, which is why we test!

5

SLIDE 9

5 Optimization Discoveries from the SAP Test Lab

Search is (even) more important than ever

1

SLIDE 10

100% Multi-touch attribution vs. last touch approach uncovers true drivers

Last touch attribution: journey with 5 touch points – last one gets all the credit Multi-touch attribution: journey with 5 touch points – each getting partial, proper credit

20% 30% 10% 30% 20%

SLIDE 11

Experiment ID: SAPXXX Location: US Test Protocol: TPXXXX

Background: We tested a more detailed and balanced attribution approach versus the traditional last touch attribution model we’ve been using for years Objective: Understand what each media vehicle truly brings to the table, from start to finish of visitor journey Primary Research Question: How do media vehicles stack up when considering the end-to-end journey? Test Design: Deep-dive regression analysis for 5 months of historical data, spanning search, display and content syndication activity.

Research Notes:

Bonus Discovery: Multi-touch attribution vs. last touch approach uncovers true drivers

SLIDE 12

Showed the real contribution of each channel vs. last touch model

SEM gained 14% additional conversion credit, when moving from Last Touch to multi-touch attribution model

Vehicle Last Touch Multi-touch Difference Banner 4,634 4,248

SEO 3,732 3,839 3% SEM 1,792 2,035 14% Content Syndication 887 924 4%

Multi-touch attribution test: Results

What You Need to Understand: the content deluge = confusion for customers, pushing them to search more often as they educate themselves WELL BEFORE they have a selection set. This means search has moved well beyond lead gen into the heart of brand awareness.

SLIDE 13

5 Optimization Discoveries from the SAP Test Lab

Photographic imagery sharpens response

2 Search is (even) more important than ever

1

SLIDE 14

Discovery #2: Photographic imagery sharpens response

Experiment ID: SAPXXX Location: Globally Test Protocol: TPXXXX

Background: With different types of imagery in its arsenal the natural question of which is best surfaced the need to test Objective: Increase lead form submissions Primary Research Question: Which image type – vector iconography or photographic imagery – drives more response and conversion rates? Test Design: Test replaces vector style imagery across entire landing page with photographic imagery to gauge differences in response triggered by image type

Research Notes:

SLIDE 15

Photographic imagery sharpens response + 46%

Testing vector images VS photographic images . A story of customer response VS marketing production costs

SLIDE 16

Experiment 2: Results

46% increase in conversion

Sharper photographic imagery generates more visitor response and conversion rate (46% more)

What You Need to Understand: If you have different image types, it’s worth testing them against each other – results can be very disparate Page version Form Submits Control – hero, no CTA 93 Treatment – hero with CTA 134 Relative Difference (conversion rate) + 46%

SLIDE 17

5 Optimization Discoveries from the SAP Test Lab

Photographic imagery sharpens response Great results don’t have to be complex

2 3

Search is (even) more important than ever

1

SLIDE 18

Discovery #3: Great results don’t have to be complex

Experiment ID: SAPXXX Location: UK Test Protocol: TPXXXX

Background: Testing button copy on key landing pages is always part of our refinement and optimization strategy Objective: Increase lead form submissions Primary Research Question: What button copy works best at driving more conversions? (Get free premium content OR Download for free) Test Design: Simple AB test pitted 2 button texts against each other

Research Notes:

SLIDE 19

Great results don’t have to be complex

Copy test – small changes make a BIG impact

Get Free Premium Content › Download For Free ›

+ 47%

SLIDE 20

Experiment 3: Results

47% increase in conversion

Even with offers below hero, getting opportunity for action in prominent hero makes a substantial difference

What You Need to Understand: For business critical pages, testing CTA copy can drive non-trivial results! Page version Form Submits Control – hero, no CTA 127 Treatment – hero with CTA 160 Relative Difference (conversion rate) + 47%

SLIDE 21

5 Optimization Discoveries from the SAP Test Lab

Photographic imagery sharpens response Great results don’t have to be complex Make those CTAs easy to spot (and click)

2 3 4

Search is (even) more important than ever

1

SLIDE 22

Experiment ID: SAPXXX Location: US Test Protocol: TPXXXX

Background: SAP Test Lab wanted to see what the difference would be in click rates if we moved from text link to button in our hero areas of SAP.com Objective: Increase engagement rates with hero offers Primary Research Question: What CTA style garners more engagements (clicks) – text links or buttons? Test Design: Simple AB test pitted 2 CTA styles against each other

Research Notes:

Discovery #4: Make those CTAs easy to spot (and click)

SLIDE 23

Make those CTAs easy to spot (and click)

Key question – can CTAs be made easier to find?

Smaller text link Button design approach

+ 7%

SLIDE 24

Experiment 4: Results

7% increase in conversion

buttons garner more clicks – while lift was modest, when syndicating results globally, lift has strong impact on overall site engagement

What You Need to Understand: Make sure it is easy for visitors to spot those important CTAs Page version Hero Clicks Control – hero, text link CTA 212 Treatment – button CTA 227 Relative Difference (conversion rate) + 7%

SLIDE 25

5 Optimization Discoveries from the SAP Test Lab

Photographic imagery sharpens response Great results don’t have to be complex Make those CTAs easy to spot (and click)

2 3 4

Search is (even) more important than ever

1 Test surprises happen, which is why we test!

5

SLIDE 26

Experiment ID: SAPXXX Location: US Test Protocol: TPXXXX

Background: We set out to test different ways to distinguish premium (registration gated) and non-gated content to see which approach is optimal at driving registrations Objective: Increase lead form submissions Primary Research Question: What mix of icons and messaging that distinguishes gated and non-gated content drives the most registrations? Test Design: AB/n approach pitted 4 challengers against the control, which was no messaging at all

Research Notes:

Discovery #5: Test surprises happen, which is why we test!

SLIDE 27

Test surprises happen, which is why we test! + 17%

Testing assumptions about transparency with the reality of customer behavior

SLIDE 28

Experiment 5: Results

17% decrease increase in conversion

despite nice gesture of distinguishing gated and non-gated content, the visual cues suppressed registration, the thinking that visitors were more aware of where to find the “free’ content

What You Need to Understand: Giving visual cues as to what offers require registration can actually suppress overall conversion rates Page version Hero Clicks Control – no messaging 293 Treatment – “login required” 235 Relative Difference (conversion rate)

SLIDE 29

5 Optimization Discoveries from the SAP Test Lab

Putting content questions to the test Great results don’t have to be complex Make those CTAs easy to spot (and click)

2 3 4

Search is (even) more important than ever

1 Test surprises happen, which is why we test!

5

SLIDE 30

Holistic understanding where to focus efforts

Content and Offers Registration and Form Pages Design and Layout

*Potential Lift is based on the positive benefit seen in actual tests.

Up to 780% Up to 97% Up to 92%

SLIDE 31

A word about change management

It’s impossible to beat the “testing drum” too loud

What does SAP Test Lab do?

SAP Test Lab is an internal team at SAP dedicated to testing the SAP web presence. Using data from real customers, SAP Test Lab finds what works best to drive more leads from the traffic you already have, right now. That way, you can optimize your marketing strategy by uncovering new insights about your visitors and how to serve them best. 3

SLIDE 32

Thank You & Questions

Shawn Burns Global Vice President Digital Marketing SAP @shawnpburns

SLIDE 33

5 Optimization Discoveries from the SAP Test Lab

Turn heroes into action heroes

6

SLIDE 34

Discovery #1: Turn heroes into action heroes

Experiment ID: SAPXXX Location: North America Test Protocol: TPXXXX

Background: SAP leverages multivariate tests to fine tune and optimize existing page templates – in this case, an existing paid search landing page template Objective: Increase lead form submissions from landing page Primary Research Question: With actionable offers below hero, how important is it to have an actionable offer in the hero as well? Test Design: In addition to the offer versus no offer question in the hero area, we tested body with images versus no images and we tested a more human-oriented image

Research Notes:

SLIDE 35

Discovery #1: Turn heroes into action heroes

Control

SLIDE 36

Multivariate test across buttons in hero, body with images versus no images and more human-oriented imagery

Discovery #1: Turn heroes into action heroes + 19%

Treatment

SLIDE 37

Experiment 1: Results

19% increase in conversion

Even with offers below the hero, adding an opportunity for action in prominent hero makes a substantial difference

What You Need to Understand: Our big win in this multivariate test was getting an offer (CTA) into the hero and injecting some clear action in that key area. Page version Form Submits Control – hero, no CTA 179 Treatment – hero with CTA 226 Relative Difference (conversion rate) + 28%

SLIDE 38

THANKS and questions

Shawn Burns Brian Carroll Global Vice President Executive Director, Digital Marketing MECLABS SAP