Where Barna Studies Start: An FAQ

Barna gets a lot of questions about what we do, as well as how and why we do it. While we regularly use this space for releasing new data, we thought it might be interesting to go “behind the scenes” for those curious about the nitty-gritty details of conducting research or about Barna’s role in collecting and interpreting data.

In the upcoming Barna Trends 2018, readers will find an in-depth FAQ covering the details and milestones in a typical study. Here, our researchers and editors answer some key questions about where Barna studies start.

Q. How does Barna choose topics to study and report on?

A. While Barna’s researchers and writers sometimes explore questions in regular population surveys related to their own cultural or religious interests, the majority of the topics comes from partner organizations. Parachurch ministries, foundations and private companies come to Barna with particular questions about trends in culture, ministry, politics, education and so on, and we work with them to devise a research approach to effectively study that subject. Occasionally, these studies become online reports or printed monographs, magazine-style research reports published by the Barna team and client partners. These projects provide a deep dive into the data, incorporating different types of research with expert interviews and infographics to provide a holistic picture of the findings and help a particular audience implement them.

In the past year alone, we’ve released resources covering the wellbeing of pastors, the future of Christian higher education, the faith of youth in the Republic of Ireland, the identity of Jewish Millennials and generosity habits in the Church. Barna also conducted research that helped shape Andy Crouch’s recent book, The Tech-Wise Family.

Q. How do you find people to survey?

A. Due to an increasing number of households that do not have a landline telephone, at least half of a Barna phone survey includes adults interviewed on their cell phone. Telephone numbers are obtained through several reputable list brokers that offer consumer lists for purchase. Telephone samples include both listed and unlisted phone numbers. Interviewers place calls at various times of the day, various days of the week and multiple times if necessary in order to minimize bias in respondents.

An increasingly common way that our team conducts research is via online surveys conducted with an online research panel. Our panel partners are carefully vetted to ensure we are finding “real people” to respond to our surveys. Participants in the panel are recruited using a controlled, invitation-only method and are then put through multiple identity checks. For example, they must have a physical address and a working email address. All panelists are also required to double-opt in by confirming their interest through an additional email they receive. Digital fingerprinting collects data on their computer (operating system, browser version, etc.) that allows us to ascertain that they are a real person and that there are no duplicates in the file. Quality control checks are also included in the survey to make sure those who speed through the survey, give answers that don’t make sense, fail trick questions or appear unengaged are eliminated from our surveys, and, if multiple violations occur, from the research panel. Panelists are typically offered an incentive for their participation in the form of a gift certificate of some sort. Just as in telephone interviews, multiple reminders are sent to panelists to ensure that respondents are not just those who are the most eager to take the survey or who are more apt to check their email or answer the telephone often.

Using online and phone samples together allows the broadest distribution of individuals in the population an equal chance of participating in a survey, regardless of age, socioeconomics, geographical location or other factors.

Barna Trends 2018: What’s New and What’s Next at the Intersection of Faith and Culture

Q. What is the state of phone vs. online polling?

A. Almost all American adults (95%) own a cell phone, including more than three out of four (77%) who own a smartphone, according to Pew Research. A study conducted by the Centers for Disease Control reports that as of May 2017, half of households (51%) did not have a landline telephone at home. Additionally, more than one-third of households who own a cell phone and still have a landline (37%) say they answer all or most of their calls on their wireless phones. If combining cell phone–only households with the 15 percent of all households who have both a cell and landline phone but primarily or exclusively take their calls on their cell phone, nearly two-thirds of the adult population is considered unreachable by landline.

Barna found that the average adult cell phone user is 35 years old, compared to landline users, whose average age is 54. Cell users are also more likely to be male than female, more educated, a high earner and an ethnic minority (though many of these numbers are also unlisted).

The average adult cell phone user is 35 years old, compared to an average of 54 among landline users.

With declining response rates in telephone surveys (not as many people answer calls from an unknown caller or are willing to stay on the phone for a 15-minute survey) and rising costs of including cell phone interviews, researchers are finding the reliability of online surveys is increasing while the reliability of phone interviews is decreasing. Additionally, online surveys are less expensive to run, offer the researcher the opportunity to show graphics or videos and are easier for respondents to complete at their convenience when they are able to pay attention. The differences between data collected online and by telephone is relatively small, with a 3–5 percent difference in responses.

Of most importance, telephone surveys are more susceptible to “social desirability bias,” meaning that people are more likely to give answers that paint them in a positive light when they are speaking with a live interviewer. Subjects covering sensitive topics are best conducted online, allowing people to answer anonymously or express negative attitudes. Barna’s area of research—covering faith topics as well as some intimate subjects like financial giving or use of pornography—often lends itself to online surveys as a more appropriate channel.

Q. How do you go about interviewing pastors specifically?


A. Barna has developed a strong expertise in research with church leaders. Our participation rate in surveys with pastors is high; we’ve done over 14,000 interviews with pastors since 2006. Whether interviewed by phone or online, senior pastors are recruited from publicly available church listings covering 90 percent of U.S. churches that have a physical address and a listed telephone number or email address. Minimum and maximum ranges are placed on church size, denomination and region to ensure representation. Minimum weighting is also used to match church characteristics from the National Congregation Study (by Association of Statisticians of American Religious Bodies).

Barna has done over 14,000 interviews with pastors since 2006.

Unlike the general adult population, senior pastors are often best reached by telephone. However, Barna also has recruited a pastor panel from its phone surveys, and these pastors participate in frequent online surveys as well.

Learn more about Barna’s work at barna.com/services.

Comment on this research and follow our work:
Twitter: @davidkinnaman | @roxyleestone | @barnagroup
Facebook: Barna Group

Photo by rawpixel.com on Unsplash