Gathering and Analyzing Survey Responses Accurately is Critical to Compiling Useful Construction Data
People in fields such as economics, science, and politics often try to make predictions. Whether they’re about market shifts, consumer demand, manufacturing capacity, or growth in new areas, those who can predict change accurately have an advantage. This applies to the construction industry, where businesses need to make major decisions based on survey responses. But the reports are only as good as the quality of the information gathered and its analysis.
The validity of predictions is determined by how well the information is gathered and analyzed. Because good decisions are based on good information, collecting reliable, valid data is critically important.
Reliability Versus Validity
The terms “reliable” and “valid” are related but refer to two different concepts. “Reliable” data refers to its reproducibility. In other words, it is data that, when measured repeatedly, produces a consistent result.
“Valid” data, on the other hand, describes the result that was intended to be measured. The reliability of data and its validity are sometimes difficult to determine without other data to compare to, but there are some methods. Whether a researcher can make those comparisons or not, it is possible to collect good, useful data.
Collecting Good Data Using Samples
Collecting good data requires some idea of the total group or population intended to predict from and then capturing information from a smaller group or subset within it to get a “snapshot” of the whole. This is a representative sample that should be selected randomly. If a researcher wants to avoid bias (capturing a sample that is skewed toward or away from some demographic and thus not representative of the whole group or population), why not just get the whole population to respond? That would be ideal, but it is usually too difficult or expensive to accomplish.
Avoiding Bias and Comparing with Other Samples
There are some classic examples of incorrect predictions based on biased sampling that illustrate how easy it is to make a mistake. For example, in 1936, Literary Digest incorrectly predicted that Roosevelt would lose the presidential election by inadvertently introducing a bias toward affluent Americans by sending out surveys based on vehicle registrations, among other things. These past mistakes show that researchers must be aware of how a sample can include known or unknown bias, so they proceed with caution. Comparison to other studies or surveys with different sampling groups and methodologies can help validate a data set.
Avoiding Confirmation Bias in Surveys
One way to get a representative sample is through surveys. Survey data can be useful, but it must be kept in context. First, a survey will need to cast a wider net than the representative sample size because it is nearly impossible to get a 100% response rate. Second, those who do respond are often in groups that are either already favorable or have an axe to grind. Or perhaps all those who respond are interested, while those who are not interested don’t respond. Third, the questions chosen and how those questions are worded is crucial, as respondents can interpret questions differently.
Researchers can be tempted to ask questions that might confirm what they already think rather than provide the information that is actually desired. This is known as “confirmation bias.” For these reasons, it is helpful to have an experienced, impartial third-party review a survey and its questions to help weed out poorly worded questions and design the survey so information is gathered in an unbiased process. If a disinterested third party isn’t available to review a survey, researchers may instead use a few trusted sources and run a test survey before launching it on a large scale.
Strategies for Prompting a Good Survey Response Rate
How do researchers get a good response rate to a survey? This is challenging. They can certainly send out more surveys, but sometimes incentives help. Another tactic is to make the survey brief and easy to access and respond to. The downside is that the survey may not get the most useful information that way.
Designing a survey to reach a highly engaged group is another strategy to produce a quality survey with a good response rate, but that data isn’t necessarily reflective of an entire population. But then, that population may be those whose opinion matters the most to the researcher.
Determining Sample Size
What is a representative sample size? This varies and is never easy to define without knowledge of the population. That might sound like a bit of a circular explanation, and it is, but let’s start by defining what is not a good sample size. If there are 10,000 people in a population but only 10 are surveyed, that won’t produce good results. Surveying 500 is an improvement, but even more will lead to better data. A response rate of 5% to 30% is good, with more than 50% considered fantastic.
A good survey should try to get the largest sample possible at a reasonable cost. If a survey reveals what a whole population thinks, then the researcher can interpolate what a few random samples of different sizes are likely to say and compare them to the population. From there, the researcher can determine the probability that a sample of a particular size will respond the same way as the population.
When that probability is at a comfortable level — say 90% to 95% — then that’s the sample size to use. That is, of course, in hindsight. If there isn’t a history of population outcomes and sample responses, then it is best to just acquire the largest, most representative sample possible. That said, there are ways to check that the sample is at least consistent with itself.
For example, let’s say a survey was conducted with 1,000 responses received from a pool of 10,000 surveyed, and 500 random responses were analyzed to determine a specific response, such as agricultural market growth nationally and in the Midwest. This becomes the prediction or “model.”
The model can then be tested with random samplings of different sizes from the remaining 500 responses. This technique can be used to internally test the consistency of the data, which helps validate the results. A bonus is that the researcher can get a good idea of what a representative sample is if it produces responses with a 95% confidence level that are consistent with the model — from the 500 responses that were excluded from the model — at a particular sample size.
Analyzing the Data
Once data has been acquired, the researcher must take care to analyze it objectively. An interesting example of analysis without proper reflection (and perhaps incompletely done) is that of “New Coke.” in the mid-1980s, the Coca-Cola Company created a new formula and rebranding in response to external feedback from the heavily marketed Pepsi Challenge television ad campaign, as well as internal company taste tests. These blind taste tests suggested that people preferred a sweeter beverage more like Pepsi. New Coke was created in response, but it failed miserably, not because the taste testers were wrong, but because loyal Coca-Cola customers didn’t want a different product.
The Importance of Collecting Demographic Data
Survey researchers need to carefully determine what the data actually says — and equally important — what it doesn’t say. For example, let’s say a large group of contractors is surveyed and most responded that they were experiencing a significant increase in building contracts. This could mean that a lot of construction is taking place. But a closer examination might reveal that only smaller contractors were responding this way. This may indicate that the market conditions are very different for companies who primarily sell products to large commercial builders and contractors.
This is why a superficial analysis that reports only how most survey takers have responded can lead to overgeneralization and assumptions that produce inaccurate results. Complex and sometimes seemingly contradictory data are known as “confounding variables.” Because of their confusing interrelationships and interactions, it can be challenging to isolate them. Therefore, it is important to analyze data as thoroughly as possible.
A detailed analysis can be planned from the beginning of a survey by collecting basic demographic data about the respondents, such as contractor size or whether the contractor is residential or commercial. A simple way to do this is to look at survey responses in categories within the demographic data. For this example, it could involve analyzing the number of building contracts based on contractor size. If the demographic data isn’t available, then this factor would remain hidden, and the data could be misinterpreted. Analyzing data both as a whole and by demographic category can prove useful, especially when the story isn’t clear on first examination.
Collecting demographic data in a survey could also help determine if the sample obtained is at least capturing the population one is hoping to predict. If a survey turns up only responses from predominantly commercial construction, but the researchers were trying to capture residential, agricultural, and industrial as well, then it is clear that the sampling was biased. If instead, a researcher can compare the demographic data to known demographic information from another unrelated study or survey, and the demographic information is similar, then there is a good chance the data is not biased, at least for those demographics.
The Challenge of Conducting an Unbiased Survey
The final hurdle in researchers collecting good data is administering a survey well; that is, without bias. The challenge is in determining not only what is asked, but how it is asked, and drawing out, in a neutral manner, the intended information. This requires self-reflection and impartial communication.
Self-reflection is difficult for some researchers because they need to be willing to hear opinions or information they may not like. They must put aside their own biases and reject emotional responses. To do this, questions need to be carefully crafted to avoid leading respondents to answer in a certain way.
The Risk of Asking Emotion-Based Questions
A study of some political surveys can be a masterclass in how to ask questions poorly. They sometimes present a simplistic question that appeals to emotion rather than intellect to get a desired response. This is an attempt to change a survey respondent’s opinion, rather than to try to determine their current opinion. The order in which questions are asked can also have an impact on the results of a survey. That’s because previous questions can influence how a respondent answers later questions.
The Pros and Cons of Multimode Surveys
The method or mode in which a researcher conducts a survey matters. Multimode surveys have been effective where initial contacts are made through a particular survey mode, such as by mail. All those who respond by mail are then taken off the survey list and the remaining pool of non-respondents are sent an email. Some will respond to email and are then removed from the survey list. The remaining nonrespondents might receive a phone call or text.
Multimode response can be effective at driving up response rates, thus getting a better sample and better data but shouldn’t be used often, as it tends to make respondents feel pressured. A less aggressive version can begin with one mode and ask respondents to share their preferred response mode, which could be used for future surveys.
If multimode is used, a researcher would be wise to check if responses are comparable by survey mode among demographic groups, while also evaluating whether there are greater response rates to certain modes among various demographic groups. The urge to use the simplest, fastest survey method that gets the most responses should be weighed carefully to avoid a bias toward the most accessible respondents.
Conclusion
A well-crafted and properly administered survey with a response from an engaged group can produce good data. If the resulting data is analyzed carefully and impartially, it can produce accurate information that can be very helpful in making well-informed decisions. RB
