A Barna Team Member Reflects on 25+ Years in Research

Barna Group, founded by George Barna in 1984, has long aided the Church in understanding cultural trends related to values, beliefs, attitudes and behaviors. Though much has changed in the field of social research since its beginning over three decades ago, Barna has kept up with advances in technology and analytics, ensuring reliable data that faith leaders can use to understand the times.

A big part of Barna’s success in this field is due to our dedicated research team, now led by Brooke Hempell. In an effort to understand more about Barna’s research process and how the company’s approach to collecting data has evolved over the years, we interviewed Barna’s senior research manager, Pam Jacob, who has been with the company for 26 years. As our State of the Church project begins to look back on decades’ worth of research, Pam reflects on her experiences collecting, analyzing and being impacted by that data.

Barna: When you first started at Barna Group, what was your role versus what is it now?

Pam: I was a part-time research assistant when I first started. I worked with the research manager and, believe it or not, it was just her and I in the research department, along with, of course, George Barna.

Barna: Looking back to your early years with the company, how was research conducted? Could you give me a brief overview from start to finish?

Pam: It’s really evolved at Barna since I started. Back then, George Barna would write the questionnaires and manage the clients, then pass it onto me at which point I would program the questionnaire, then work with the field center to interview people by telephone, run the data and send it back to George for the analysis. Then it would either be shared with the client, if it was a client-commissioned study, or analyzed for a Barna-released book, article or other medium.

It was like that from the time I started until probably 2010. Before that time, nearly all of the surveys were conducted by telephone. We also did what we called self-administered surveys where we would mail out a survey to the client, who would then distribute it to their audience, whether that be their donors, employees, youth groups or other audiences. We also created community surveys, where we’d actually mail hard copies of surveys to people within a certain geographic region for them to fill out and send back to us, at which point we would key-punch them all by hand before we could proceed with the analysis.

Primarily, though, the surveys were done via telephone—we would just call people up randomly and ask them to participate in a survey. We ran an in-house field center with as many as 35 callers for any one night who would interview people. Actually, that’s where David Kinnaman started. He was an intern at the time, going to school at Biola and then driving up to Glendale every night to supervise our field center.

Barna: Over the years, social research has seen drastic changes. What has it been like to keep up with those advances, especially moving into a digital age?

Pam: The biggest change I witnessed was when we started doing cell-phone interviewing by phone, and then online interviewing. This happened when people started switching over to cell phones and were becoming less available on their landlines.

Over the years, it’s just become harder and harder to conduct phone interviews. Conducting research over the phone is more expensive and it’s not as reliable as it used to be. When we first began interviewing by telephone, not only did cell phones not exist, caller-ID didn’t exist. Today the majority of households no longer have a landline telephone, and most phones, whether a cell phone or landline, have caller-ID. Because people are likely to reject a call from a number they are unfamiliar with, you’re only getting people who answer their phones, which means you are only interviewing a specific group of the population, which is not representative of the general population as a whole.

Barna: One thing that we will keep coming back to as we continue to look at the State of the Church 2020 project is Barna’s decades of tracking data. Can you tell us more about that?

Pam: In our general population studies and PastorPolls^SM, we ask the same questions about demographics, theolographics and certain attitudes in every study we conduct. Not only does this allow us to use the same criteria to segment the data during analysis, but we can also compare the data to results from other studies or to data from previous years. This allows us to see how the data has changed over the years.

That’s basically what tracking data is; it’s the same question asked multiple times in a similar manner for comparison purposes. Also, merging these same data points together gives us a larger sample size  so that we can do additional segmentation. We might use combined data for the same question to drill down in greater detail among a certain population segment in which we wouldn’t have enough data to analyze in our typical survey of 1,000 interviews. Combining datasets also allows us to decrease the overall error rate and improve the confidence we can have in the data when you’re looking at 5,000 interviews instead of just 1,000.

Barna: Share a fun fact about data.

Pam: I think some people question how we can survey a thousand people and call that representative of a population. Of course, we talk about the error rates, but if you do a study well, it is representative. In the example I gave earlier of 1,000 interviews, the error rate is plus or minus 3 percentage points. That means, if we repeated this same study, we can expect the results to be the same with 3 percentage points higher or lower than the original data.

It’s really affirming that there is validity to what we do because when we repeat it, we get the same numbers. So, that’s pretty fun and interesting.

Barna: In your time here, what has been a turning point for you as a researcher in terms of how you see the world or your faith?

Pam: Well, the faith one is easy: I wasn’t a Christian when I started here. I was going to Mothers of Preschoolers (MOPS) with Nancy, George’s wife, who had also invited me to her church which I attended a couple of times. Within the first month of working at Barna, I worked on one of our OmniPolls, in which we asked tracking questions like, “Do you believe the Bible is without error?” or “Do you believe that Jesus sinned?” I realized I didn’t know the answers to these questions.  There was also a section to test respondents’ Bible knowledge, and I didn’t know the answers to those either.

The director of research who I first worked under, Cindy, asked, “Do you have a Bible?” I responded that I didn’t. She said, “Well, I would be willing to get you one, and then we can meet before work and talk about some of these questions that you have.” I also started to go to church more with Nancy and then, three months later, I accepted the Lord. It was hard not to; I mean, I was surrounded by Christians doing Christian work.

Also, coming from a non-Christian upbringing, while I understand how the unchurched think, I have a passion for those who have fallen away from faith. More and more people are growing up in households like the one I grew up in, where we considered ourselves to be Christian but never attended church and never read the Bible.  It saddens me to see the number of people who consider themselves to be Christian but whose active faith is declining. It is exciting to be part of Barna and have the opportunity to inform the church of this decline and hopefully equip them with better tools to reach people for Christ.