Week 8: Measuring Effectiveness and Evaluating Campaigns

Week 8 closes the planning cycle: after objectives have been set, the creative and media strategy developed and the campaign launched, the marketer must find out whether it worked. Following Belch & Belch (2021), this chapter covers the arguments for and against measuring effectiveness, the decisions that can be tested (source, message, media and budget), when and where to test, the stages of the testing process from concept testing to finished-ad pretests and posttests, the PACT principles and the problems with current research. It then moves to the evaluation of the IMC program as a whole and to the metrics and analytics that dominate digital campaign evaluation: reach and cost measures, engagement and sentiment, attribution models, marketing mix modelling, A/B and holdout testing, ROI and ROAS, brand lift, and the balance between long-term brand building and short-term activation.

Why measure effectiveness, and why some firms do not

Measuring effectiveness is the evaluation and follow-up step of the IMC planning process. It links back to the objectives set earlier: communication objectives formulated along a hierarchy of effects (see DAGMAR in Week 3) only become useful when the results are actually measured against them, and the media plan's evaluation and follow-up phase (Week 6) asks the same question for media objectives.

Reasons to measure

• Avoiding costly mistakes. Media time and space are expensive. Testing tells the advertiser whether a campaign is likely to work before (or while) the money is spent, so poor executions can be dropped or changed.
• Evaluating alternative strategies. A firm often has several possible executions, appeals or media schedules. Research helps decide which one is most likely to achieve the objectives.
• Increasing the efficiency of advertising in general. Research builds knowledge about what works for the brand.
• Determining whether objectives are achieved. If objectives are specific and measurable, measurement shows whether they were met and supports accountability towards top management.

Reasons not to measure

• Cost. Good research is expensive and time consuming; some managers prefer to spend the money on more media.
• Research problems. It is difficult to isolate the effect of one ad or one promotional element on sales, because many other factors (price, distribution, competition, economic conditions) also influence results, and effects may be delayed.
• Disagreement on what to test. Sales, recall, attitude change and engagement are all possible criteria, and different stakeholders (client, agency, media department) may prefer different ones.
• Objections of creative staff. Creative people often argue that testing stifles creativity and favours safe, formula-driven ads, and that standard tests do not capture what makes an ad work.
• Time. Many managers feel they cannot wait for test results because the market will not wait, so the campaign is launched untested.

What to test

Belch & Belch link the decisions to be tested to the communication model: source, message, media and budget.

When and where to test

Pretests can be done at several points, from the first idea to the finished ad. They are diagnostic: they tell the advertiser what to change. Posttests are evaluative: they measure performance in the market, although their results also feed into future campaigns.

The trade-off is between control and realism. Laboratory tests give the researcher control over the exposure and the other variables, which makes it easier to attribute differences to the ad itself. Their weakness is testing bias: people who know they are being tested pay more attention than they would at home, and the setting lacks realism. Field tests are realistic (clutter, distraction, normal viewing) but offer less control, so other factors may influence the outcome.

The testing process: from concept to finished ad

Testing can take place at each stage of the development of an ad. Belch & Belch distinguish concept generation and testing, rough art, copy and commercial testing, finished-ad pretesting, and market testing (posttesting).

Concept testing

Concept tests usually rely on focus groups or individual interviews, increasingly online. They are fast and cheap, but small groups are not representative.

Rough art, copy and commercial testing

Because producing a finished TV commercial is expensive, many ads are tested in rough form: an animatic (a filmed sequence of drawings from the storyboard with a soundtrack), a photomatic (photographs instead of drawings) or a ripomatic (footage borrowed from existing commercials). Rough testing saves money, but people may react to the rough quality rather than to the idea. Two common approaches at this stage:

• Comprehension and reaction tests check whether the ad conveys the intended meaning and whether it provokes negative reactions or offends.
• Consumer juries ask a group of target consumers to rate or rank alternative ads (order of merit or paired comparison). The method is simple and inexpensive, but jurors may act as experts rather than consumers, a halo effect may occur (one aspect colours the whole judgement), and people tend to favour image ads in rankings.

Finished-ad pretesting

Finished ads are tested at the stage closest to the real execution. For print ads, methods include portfolio tests (test ads placed among other ads in a folder, followed by recall questions), readability tests and dummy vehicles (a test magazine containing the ads). For broadcast:

• Theater tests: participants are invited to view a pilot programme with commercials embedded in it. Brand preference is measured before and after exposure, and some systems let participants register their liking continuously with a dial.
• On-air tests: the commercial is run on a real channel in a test market and viewers are contacted the next day to measure recall and persuasion. This adds realism, but recall may be a weak indicator of effectiveness, especially for emotional ads.
• Physiological measures record involuntary responses during exposure (see below).

• Eye tracking follows where on the page or screen the eyes fixate and for how long. It shows which elements are seen, in which order, and which are skipped; it is widely used for print, packaging, websites and digital ads.
• Pupillometrics measure pupil dilation, which is related to arousal and interest. Dilation does not reveal whether the response is positive or negative.
• EEG (brain waves) records electrical activity in the brain, for example alpha activity as an indicator of attention, and hemispheric lateralization (left versus right brain processing).
• Galvanic skin response (electrodermal response) measures changes in skin conductance caused by sweating, as an indicator of arousal and emotional reaction.

These measures should be used with care. They show that a reaction occurred, not what the reaction means: arousal can be positive or negative, and a fixation can signal interest or confusion. Physiological data are therefore best combined with verbal measures such as comprehension, attitude and purchase intention.

Posttesting finished ads

Posttests measure how an ad or campaign performs in the market. The main methods are:

• Inquiry tests measure the number of inquiries or direct responses (coupons returned, calls, requests for information, website visits through a specific code) generated by an ad. They are useful when the objective is a direct response; a split-run (two versions in different copies of the same issue) allows comparison. Inquiries do not measure the broader communication effect.
• Recognition tests show respondents an ad and ask whether they remember having seen it. The best-known example is the Starch readership study for print, which reports the percentage of readers who noted the ad, who associated it with the brand, and who read most of the copy. Recognition tests are cheap and allow comparison with norms, but people may claim to have seen ads they did not see (false claiming), or forget ones they did.
• Recall tests ask what respondents remember without showing the ad (unaided) or with a cue such as the product category (aided). For TV, the classic method is day-after recall: viewers are called the day after a commercial aired and asked what they remember. Recall favours rational, information-rich ads and may underestimate emotional ads that work without being verbally recalled.
• Association measures test whether people link the message, slogan, character or visual to the right brand. A memorable ad that is attributed to a competitor has failed.
• Tracking studies measure awareness, recall, attitudes, image and purchase intention at regular intervals (often continuously) before, during and after a campaign. They show trends, can be related to spending levels and reveal wearout, but they are costly and do not by themselves prove causality.
• Single-source systems combine, for the same households, data on exposure to advertising with data on actual purchases (for example from scanner panels). They make it possible to relate exposure directly to buying behaviour, but they are expensive and panel members may not be fully representative.

The PACT principles

In 1982 a group of US advertising agencies published the PACT (Positioning Advertising Copy Testing) principles, which state what a good copy-testing system should do.

1. Provide measurements that are relevant to the objectives of the advertising.
2. Require agreement about how the results will be used in advance of each specific test.
3. Provide multiple measurements, because single measurements are generally inadequate to assess the performance of an ad.
4. Be based on a model of human response to communications: the reception of a stimulus, the comprehension of the stimulus and the response to the stimulus.
5. Allow for consideration of whether the advertising stimulus should be exposed more than once.
6. Recognize that the more finished a piece of copy is, the more soundly it can be evaluated, and require as a minimum that alternative executions be tested in the same degree of finish.
7. Provide controls to avoid the biasing effects of the exposure context.
8. Take into account basic considerations of sample definition.
9. Demonstrate reliability and validity.

Problems with current research and the essentials of effective testing

The main problem is often not whether firms test, but how. Many methods are used because they are cheap and familiar rather than because they are valid: recall is easy to measure but is not a good predictor of sales for every type of ad; tests of a single exposure ignore the effect of repetition; laboratory settings exaggerate attention. Belch & Belch summarise what effective testing requires:

• Establish communication objectives (it is very hard to evaluate an ad without knowing what it was supposed to achieve).
• Use a consumer response model (such as a hierarchy of effects) to decide which effects to measure.
• Use both pretests and posttests.
• Use multiple measures.
• Understand and implement proper research (adequate samples, controls, reliable and valid measures).

Measuring the effectiveness of the IMC program

An IMC program consists of more than advertising. Each element needs its own measures, and the program as a whole must be judged against the marketing and communication objectives.

• Sales promotions: redemption rates of coupons and offers, incremental sales during and after the promotion compared with a baseline, and whether the promotion merely pulled sales forward or attracted new buyers.
• Sponsorships and events: exposure (media coverage, logo visibility), awareness and image change among the target group, attendance, leads generated and sales during the period.
• Public relations: the volume and tone of media coverage, the share of coverage containing the key message, and changes in awareness or attitude, rather than only a count of clippings.

The challenge for IMC is that consumers are exposed to many touchpoints and the effects interact: a TV campaign raises searches, a social post drives a site visit, a promotion closes the sale. Evaluating each element in isolation therefore tends to give too much credit to the elements closest to the purchase and too little to those that built the brand. This is why attribution, marketing mix modelling and experiments are needed.

Digital and social media metrics

Digital media made many effects directly countable. The most common metrics are listed below. They are useful, but a metric that is easy to count is not automatically a metric that matters: a high click rate does not prove the campaign created sales that would not have happened anyway.

Definitions differ between platforms (for example whether engagement rate is divided by impressions or by followers), and view-through conversions easily overstate effect because many of those people would have bought anyway.

Attribution models and their limits

Example: a customer sees an Instagram ad, later clicks a YouTube ad, then clicks a Google search ad and buys for 120 euro. Last click gives all 120 euro to search, first click gives all of it to Instagram, and linear gives 40 euro to each.

All attribution models share important limits. They only see trackable digital touchpoints, so TV, outdoor, word of mouth and in-store experience are invisible to them. Tracking is increasingly incomplete because of privacy rules, cookie restrictions and people switching devices. Most importantly, attribution describes the paths of people who converted; it does not show whether the ad caused the conversion. A retargeting ad shown to someone who already had a full shopping cart will receive credit even if it changed nothing.

Marketing mix modelling, A/B testing and incrementality

MMM covers both offline and online media, does not depend on individual tracking and can estimate carry-over and diminishing returns. Its weaknesses are that it needs a long history of data with enough variation in spending, it works at an aggregated level (not per ad), it is slow and costly, and it still rests on correlations and modelling assumptions.

Because assignment is random, A/B and holdout tests show cause and effect, which attribution and MMM cannot do with the same certainty. They do, however, cost money (the holdout group receives no advertising), need sufficiently large samples, measure mostly short-term effects and are hard to run for national TV. Good practice combines the approaches: experiments to calibrate attribution and MMM, MMM for allocation across all media, attribution for daily digital optimisation.

ROI and ROAS

The example shows why ROAS alone is misleading: it ignores the cost of goods and it usually counts all attributed revenue rather than only the incremental part. The break-even ROAS at a 40% margin is 1 / 0.40 = 2.5, meaning every euro of advertising must generate 2.50 euro of incremental revenue just to cover itself. On the other hand, short-term ROI understates the value of advertising that builds the brand, because part of the return arrives later through higher loyalty, lower price sensitivity and future sales.

Brand lift and long-term brand building versus short-term activation

Brand lift studies apply the logic of the holdout test to communication effects rather than sales. They fit the hierarchy of effects: a campaign can be judged on whether it moved people from awareness to consideration, even if sales follow later. The results reflect short-term memory and attitude, not long-term brand equity.

The distinction between brand building and activation is central to the evaluation debate. Activation (search ads, promotions, retargeting) produces quick, measurable sales responses that look excellent in last-click attribution and short-term ROI. Brand building (emotional, broad-reach advertising) creates memory structures and preference that pay off over months and years and are hardly visible in short-term metrics. Binet and Field (The Long and the Short of It, IPA, 2013) analysed campaigns in the IPA effectiveness databank and argued that the strongest long-term results come from combining both, with the larger share of budget on brand building; their often-quoted rule of thumb is roughly 60% brand and 40% activation, a balance that varies by category and situation. The practical lesson for evaluation is that a firm that measures only short-term digital metrics will systematically shift money towards activation and may weaken its brand over time.

A balanced evaluation framework therefore combines short-term measures (reach, CPA, conversions, incremental sales) with long-term measures (tracking of awareness, consideration and brand associations, price premium, share of voice relative to market share, and the brand's customer-based equity as discussed with Keller's CBBE pyramid).

Summary

• Measurement avoids costly mistakes and shows whether objectives were met; firms skip it because of cost, research problems, disagreement on criteria, creative objections and time.
• Source, message, media and budget decisions can be pretested or posttested, in the laboratory (control) or the field (realism).
• Pretests run from concept tests via animatics to finished-ad tests; posttests include inquiry, recognition (Starch), recall, association measures, tracking studies and single-source systems.
• Digital metrics and attribution models are countable but not causal; MMM, A/B tests and holdout tests address incrementality, and ROI must rest on incremental profit rather than ROAS.
• Evaluation must balance short-term activation with long-term brand building.