How we develop and test messages
We follow a systematic, sequential process, starting by surveying current knowledge about what voters think and feel and what policy experts know; to developing messages with the potential to succeed; to assessing them with a large sample of voters.
Landscape analysis: We examine polling data to figure out where the public is, so we knew what feelings, values, and beliefs we could draw on or needed to change; study the relevant facts, scientific studies, and policy positions; and examine the messaging currently used by both sides, to know what we need to beat and to learn from our colleagues and avoid reinventing the wheel.
Message development: Our approach to message development is grounded not only in years of message testing with hundreds of thousands of voters but in a simple question: How would you talk with voters if you started with an accurate understanding of how our minds and brains work and evolved? It seems like a common-sense starting point, but it leads to a unique style of messaging. Everything from the syntax we use (e.g., avoiding third-person constructions such as “they” or “them” when trying to create a sense of empathy, inclusion, or commonality) to the values we invoke (e.g., family, community, and security), is grounded in an understanding of psychology, neuroscience, and evolution.
Scientific testing: : One of the distinguishing features of progressives is our commitment to science – except in messaging (e.g., naming major pieces of legislation without testing those names against potential alternatives). This guide reflects the assumption that the same scientific method we use to identify effective policies is the method we should use to identify effective messages: We should test hypotheses, derived from theory and prior research, to develop messages we believe will be compelling to voters and then test those hypotheses to see which are corroborated, disconfirmed, or require revision according to the data.
We test messages at multiple levels of language:
-
Winning Words: words and phrases to use or avoid.
​
-
Narratives: multiple ways of telling the story of an issue, condensed to roughly sixty seconds.
​
-
Soundbites: single-sentence messages designed to be emotionally resonant, resonant, and to tell a story in a sentence, which can be useful as taglines in ads, applause lines in speeches or debates, or language for social media.
​
-
Not Your Grandfather’s Talking Points: brief two- to three-sentences descriptions of problems or policies, written in emotionally compelling, values-driven, everyday language, unlike the drab talking points so widely disseminated by rarely used, which are also for interviews, debates, and brief enough for social media.
​
We typically use a stratified representative national sample of 1000 or more voters to test multiple messages at multiple levels, from words and phrase and single-sentence soundbites to sixty-second, paragraph-length narratives, drawing on a range of values, metaphors, and examples. We test them against each other and against the most disinforming, dog-whistling, race-baiting messages the opposition is actually using, so when we include a message in a table to be used, you know it is a winning message. We typically test six to ten words or phrases (aside from what we learn about them using other methods), 25 soundbites, 25 talking points, and six to nine paragraph-length narratives. We assess all responses, from words and phrase to narratives, for the extent to which voters find them compelling, using, at minimum, a 0-100 scale readily interpretable by decision makers.
Over many years, we have developed an iterative process to test and revise messages at multiple levels:
Stage 1: Message Development - We begin with a landscape analysis to identify facts that elicit emotions, scientific studies, and policy positions, and consult experts to ensure every message is accurate; polling on where the public stands; and messaging currently used by both sides. We then develop a range of messages, narratives, soundbites, and talking points to test, employing different language and metaphors, values, and examples of problems and policies. We only test our messages against the toughest, most hard-to-beat opposition
Stage 2: Quantitative Message Testing and Refinement - We then conduct as many rounds of testing is desirable and financially feasible, with large representative national samples of voters, including:
-
Online Dial-Tests measuring voters’ second-by-second responses to narratives, producing what look dial-tests on CNN during Presidential debates, except that ours reflect the responses of hundreds or thousands of voters, not thirty in the studio. Our data-collection platform produces separate lines by whatever demographics we select, so we can see if different groups are responding in different ways not only to the entire messages but to distinct words, phrases, or clauses; concepts and metaphors; and examples.
-
Voter ratings of messages at all levels of specificity, from words and phrases to narratives, so we can identify those that are the most effective and route out toxic terms widely used but that voters do not understand, find alienating, or both; for most tasks, we use a standard 0-100 scale that is readily understood, from strongly disapprove to strongly approve.
​
-
Open-ended language tests asking voters how they would define words progressives often use, to make sure voters understand what we are saying and to measure how they feel about them. We use AI to classify their responses as correct, partially correct, incorrect and “I don’t know”; and sentiment analysis to identify how they feel about them, classifying their responses as positive, ambivalent, hostile-negative, or empathic-negative (feeling for people who need help).
​
-
Associations: sometimes asking voters to give us their first associations to words, phrases, or the targets of messages they have just heard, using sentiment analysis and AI to classify their associations, which may be “words, images, feelings, or whatever comes to mind or pops into your head”
​
-
Experiments: presenting different voters with the same message but with variations in phrases, order of clauses, or concepts. We can improve voter ratings of a soundbite beginning with, “Our kids should learn the history of racism because…” by simply inserting two words: “Our kids should learn the history of race and racism because…” by over ten points.
​
-
Having identified and refined our top messages, we may repeat the process with a second sample allowing us to provide leaders quantitative data on how our messages fare and to generate automated, downloadable PDFs that array the top narratives, soundbites, and talking points in descending order of effectiveness in their neck of the woods, based on constituent demographics.
​
-
Unconscious emotional responses and associations: On issues on which voters’ conscious and unconscious attitudes can vary tremendously (e.g., race, ethnicity, gender), where possible, we also test voters’ unconscious emotional responses and unconscious associations to our top messages, using cutting-edge psychological technologies developed by Implicit Strategies (www.implicitstrategies.com), of which the principal on this project, Drew Westen, Ph.D., is co-founder with Dr. Joel Weinberger, the world’s leading expert on measurement of unconscious processes.
​
-
Messenger testing: Where useful, we presenting our top messages in identifiably distinct voices and/or images or AI-generated multimedia content that differ by race, ethnicity, and/or gender, or identifying the speaker as a member of a group (e.g., police chief, military leader, sports figure) and including appropriate still images or AI-generated content where realistic.
​
-
Video testing: On some issues, we will also create brief videos both to test the efficacy of our messages in different forms and to assess their utility for social media, typically using stories or taglines we identified as effective in earlier stages of the research now in multimedia format, which leaders can use, and we can use in experimental tests, with outcome metrics such as clicks-throughs, completion (watching the complete video), and sharing rates (e.g., re-Tweets).