By Spencer Turer

When I get about halfway through a coffee article, I begin to personalize the topic and consider how I might apply the information to my own work. Specialty coffee professionals are curious people. Our industry likes to conduct research, and researchers often share their findings in writing, whether that’s through articles published in scientific or trade journals, or in blog posts or other self-published outlets. Scientific investigation, product research and quality exploration are noble pursuits worthy of our time and resources. Sharing what we learn is intellectual generosity. Believing everything we read is dangerous.

Even after 30 years in the coffee industry, I still find what we do exciting and personally rewarding. The low cost of entry for entrepreneurs seeking coffee careers can provide a false expectation for the sophistication and expertise required to be successful. Many of us started as entry-level workers in a roastery, baristas at a cafe, or working on a farm hoping for training, advancement and a rewarding career. Our on-the-job training does not always prepare us for conducting research, or how to consider the accuracy, transparency and practical application of what we read. Continuous learning advances our sophistication and expertise in the business and science of coffee, which can enhance our professional success.

Learning about research methods and scientific protocols will give you an advantage when it comes to assessing the practical applications of published experiments and studies. Photo courtesy of Coffee Enterprises.

SOURCES OF KNOWLEDGE IN COFFEE SCIENCE

This leads us to consider how we learn, where we get our information, and how we determine operational procedures, process controls and quality. Formal training programs are often company-specific. Trade association educational programs and conference workshops are easily accessible and provide peer-to-peer information. Coffee consultants share knowledge, though they should be vetted to ensure they have a history of success and practical experience that satisfies your needs. Trade publications and online outlets provide interesting and relevant articles, and a small number of universities around the world have begun offering coffee-specific courses and certificate programs. Considering that the coffee industry is without formal educational requirements or a clearly defined academic path, we have a tremendous amount of information available to us. However, this information requires context, thoughtful consideration and validation before we embrace it or apply it to our own businesses.

Seeking to understand coffee science and research findings is a shared responsibility of the researcher, the writer and the reader. Embracing generally accepted practices from the scientific community will greatly benefit the coffee industry and help us avoid obvious pitfalls in coffee research.

Ian Fretheim, director of sensory analysis at Cafe Imports, describes the need to validate sources, especially when online research reveals multiple sources simply parroting what their collective primary source published. “Once this occurred to me,” Fretheim says, “… it drove home pretty quickly the importance of thorough source validation, and also the idea that there was a lot of space in specialty coffee for improved practices in research, knowledge base, communication and protocol.”

It’s important to note the equipment used in coffee research—including model and manufacturer—so other coffee professionals and researchers can replicate and verify your results. Photo courtesy of Coffee Enterprises.

USING LOGICAL REASONING TO INTERPRET RESEARCH

Agatha Christie’s fictional detective Hercule Poirot famously touts the importance of order and method, meaning that the facts, taken methodologically and in their proper order, will lead to only one explanation. Logical reasoning helps us understand how facts can lead to a conclusion, and understanding the different types of logical reasoning can help us conduct and interpret research.

Deduction, or deductive reasoning, uses general ideas or premises to reach a logical conclusion. For example, healthy soil produces better coffee. Regenerative agriculture focuses on improving soil health. Thus, regenerative agriculture will produce better coffee. If the premises are true, the conclusion must also be true.

Inductive reasoning tries to establish a general conclusion based on patterns and observations. For example, all raw coffee beans we have seen are green, therefore, all raw coffee is probably green. While this is a logical conclusion, it is not necessarily correct. After all, there are raw coffee beans that are other colors because of age, defects or processing.

Finally, abductive reasoning is the process of reaching a likely explanation based on observation, often when information is incomplete. It is commonly used to explain an occurrence. For example, there is half a bag of roasted coffee on the counter. There is a full pot of coffee in the brewer. The most likely explanation is that the brewed coffee in the pot is from the bag on the counter.

When considering coffee research and the reasoning used to determine a conclusion, veteran coffee roaster and educator Jim Brady reminds us to recognize a common logical fallacy, “post hoc ergo propter hoc,” which translates from Latin to mean “after this, therefore because of this.” Mistaking correlation for causation is a common mistake and must be avoided when interpreting scientific research. Correlation describes the relationship between two variables; however, it does not automatically mean that one variable causes the other. For example, high speeds in packaging machines have been correlated with increased rates of staleness; however, the machine speed does not cause staling. Poor seals, seams that are not secure, and incomplete nitrogen flushing contribute to high oxygen inside the headspace of the packages, which causes coffee to stale.

When reading coffee research, we can learn a lot from Poirot’s detective skills by recognizing the different types of logical reasoning. Determining which type of reasoning was used to draw a conclusion will help us contextualize the usefulness of the information for our business purposes.

Recording observations and other data is part of Poirot’s detective process and is useful for specialty coffee professionals when interpreting research and the veracity of the presented conclusion. Fretheim says, “Each of these styles of reasoning will have natural proponents and flaws where the others have strengths.” For example, he explains, “Deduction is of course subject to mixing up correlation for causation, a sort of logical astigmatism. Inductive reasoning will be limited by one’s experience, a sort of logical hyperopia wherein the trees are missed for the forest. Abductive reasoning will tend to be bound by the immediate environs or evidence at hand, logical myopia, missing the forest for the trees.”

Shauna Seidenberg, category manager for dispensed beverage at EG America, adds, “Being rooted in science and logic is so important, and it’s also critical to consider the context in which the research is happening,”

Consider two businesses with different operational cultures. One is a flexible, entrepreneurial company with the ability to embrace new concepts easily and make instant changes. The other is a company characterized as paralysis by analysis, unable to make changes without extensive deliberation of all available information and options, with leaders who are afraid to make a wrong decision. Coffee research, when presented clearly, completely and transparently, can help both extremes—as well as companies in between—make useful decisions.

THE SCIENTIFIC METHOD IN COFFEE RESEARCH

A critical part of coffee research is understanding the scientific method. Since the Scientific Revolution in the 16th and 17th centuries, this process for acquiring knowledge has been fundamental for guiding observation, experimentation and analysis.

*Including a control group in double-blind testing will help prevent bias and increase confidence in research results. Photo courtesy of Coffee Enterprises.*

To briefly explain the process, first a question is posed for consideration, such as, “Does roast development impact consumer purchase intent?” Next, a hypothesis is proposed to provide a reasonable or logical supposition of the answer, for example, “Yes. Because consumers have taste preferences for light-, medium- or dark-roasted coffees, roasters indicate roast development level on their packaging to encourage purchases.” Testing is then performed using controlled experiments to collect information to prove or disprove the hypothesis.

In this example, the terms “consumer” and “roasted coffee” may be too broad, since they encompass all coffee and all consumers in the world. The question and hypothesis may need to be reconsidered in order to limit the scope of the research to focus on individual products or specific consumer groups. The question may be revised to, “Does roast development impact consumer purchase intent from specialty coffee roasters in Vermont selling directly to consumers?” Similarly, the hypothesis may be edited to, “No. Specialty coffee consumers purchase roasted coffee in Vermont based on flavor description and relationship with the roasting company, independent of roast development level.” Then testing will occur to prove or disprove this hypothesis. The results are evaluated for accuracy and reviewed for bias or influence. Finally, the researched conclusions are validated by independent review and the results are tested and replicated before the report is published.

Following the scientific method will help researchers. Understanding it will help writers present that research and readers contextualize the conclusions to make informed business decisions. There are real risks to our industry when researchers do not follow the scientific method, when coffee writers do not present the methods of analysis correctly and do not challenge the data collection methods or conclusions, and when readers blindly follow published coffee research without understanding how it was collected and how it should be used.

Before personalizing research to judge how I can apply it to business operations, I consider the source by reviewing the research team’s credentials and the transparency of analysis. I also try to identify any conflicts of interest, which are important to recognize because they can lead to unintended bias. A conflict might exist, often unintentionally, when research is conducted on one’s own inventory, supply chain, finished products, internal business operations or customers. Sponsorships are vital to support research financially, however, when there is the potential for outside influence in defining or leading the research, the integrity of the findings will be tainted.

After defining the research question and hypothesis, data collection begins. Sample size is the first question researchers are faced with, and the first question readers should consider. Regardless of whether the research is for product testing or consumer insight, the sample size for testing must be representative and relevant to the question.

*Ensuring research findings are applied correctly is a shared responsibility of the researcher, writer and reader.*

Dorothea Hescock, senior director of quality assurance at Massimo Zanetti Beverage USA, says, “The sample size should have enough data points to draw a statistically sound conclusion.” When the quantity of samples or number of respondents is not optimal for statistical analysis, she adds, “It may be necessary to utilize statistical charts to determine margins of error to help to determine the confidence level expected due to your potentially limited sample size.”

Small samples sizes are easy to manage but will reduce confidence in the analysis. Conclusions and business decisions should not be made from a single or limited sample. But sample sizes that are too large can make research impossible to manage. For this reason, the questions and hypotheses within academic research are typically narrowly focused on a specific product, process or market segment.

SAMPLE COLLECTION: THE FOUNDATION OF CREDIBLE DATA

Sample collection methods are critical to connecting one’s hypothesis to one’s conclusion. In our scientific method example, the sample collection for research data must include both quantitative and qualitative data. In our original example, quantitative data would have to include all qualities of coffee, the full spectrum of roast levels, both blends and single-origin products, and even soluble coffee products. Qualitative data would need to include consumer interviews in all geographic areas. Clearly, the original question is too broad as it would require extraordinary resources to execute with any confidence.

The practical application of sample size and sample collection can be illustrated by green coffee analysis, which requires a 350-gram sample size that is representative of an entire lot of coffee. This protocol is well known throughout the industry because it is used by the Coffee Quality Institute and the Specialty Coffee Association for the determination of specialty or standard quality. The Green Coffee Association’s grading for the Intercontinental Coffee Exchange goes further, requiring the sample collection to be drawn from at least 10 percent of the bags in a container or lot. Imagine how green coffee bags are stacked in a warehouse. Appropriate sample collection is not from the most convenient bags within reach at the front of the row; it is from a variety of bags throughout the lot, at the top, middle and bottom of the stack, and from the back, middle and front of the row to obtain a green coffee sample representing all sections of the lot.

Without the appropriate sample collection and the required sample size, there will be little confidence in the analysis results and conclusions.

Be precise about variables and measurements in your research. Instead of referring to “light-’” or “dark-roasted” coffees, provide the measurement and scale—Agtron gourmet or ColorTrack, for example—used for each coffee.

TESTING PROTOCOLS: REDUCING BIAS AND ENHANCING ACCURACY

As researchers, clearly defining the problem we are trying to solve or the question we are trying to answer “is important to maintain focus and accuracy,” says Sarah DiPasquale, research and development director at TreeHouse Foods. “Whether it is new product development to meet consumer needs, troubleshooting out-of-specification production, or managing raw material shifts for supply continuity, technical teams leverage proper experimental design to conduct research, draw conclusions and provide solutions.”

Readers should consider how the research was conducted when determining the credibility of the conclusion and the relevance to their businesses. As Hescock notes, “The type of test selected needs to take into consideration the purpose of the test.”

Researchers should follow methods of analysis published by an authority or regulatory body whenever possible. (See “Scientific Research Standards and Resources,” below.) These methods have been tested and validated by subject matter experts. When followed, the data is considered accurate and easily compared to other research using the same methods. Custom methods, often developed by creative researchers or businesses for internal purposes, may not be as reliable as these standardized methods. Simply stated, different tests produce different results, not because of variations in the coffee, but because of inconsistencies in the protocols and methods used for data collection. That’s not to say that custom methods of analysis cannot be useful and verifiable, but they must be developed and carried out in accordance with accepted scientific protocols.

Scientific Research Standards and Resources

Many associations and organizations publish analysis methods and product information developed by recognized subject matter experts to help reduce product failures through testing.

International Standards Organization (ISO): Subject matter experts detail the most efficient and accurate ways to measure and test. (iso.org/standards.html)
American Society for Testing and Materials (ASTM): Establishes testing standards for manufacturers to measure performance and foster consumer confidence. (astm.org)
Association of Official Analytical Chemists (AOAC): Provides scientifically rigorous analytical processes for industry and regulators to ensure agricultural products are safe. (aoac.org)
Green Coffee Association (GCA): Supports green coffee transactional stakeholders through resources developed by industry subject matter experts. (greencoffeeassociation.org)
Specialty Coffee Association (SCA): Publishes technical standards for coffee products, brewing equipment, sustainable practices and professional competencies for coffee professionals. (sca.coffee)
U.S. Food and Drug Administration (FDA): Publishes regulations and scientific methods of analysis to help ensure food safety. (fda.gov)
U.S. Department of Agriculture (USDA): Researches and presents data on global trade. (usda.gov)

*Research results should be evaluated for accuracy and reviewed for bias or influence. Photo by Juan José Sánchez Macías.*

Blind testing is important to prevent bias, especially when conducting qualitative or subjective evaluation by sensory observation, like cupping and tasting. When panelists have knowledge of the product’s provenance, supply stakeholders, format, package type, price or quality expectation, the credibility of analysis results is diminished because of the potential for conscious or unconscious bias and influence. Blind testing also includes the separation of physical analysis from sensory evaluation, so the cupper or taster is not influenced by visual appearance, which may create a flavor expectation that is challenging to ignore.

Academic research is often guided by double-blind studies, in which the respondent, assessor and researcher are unaware of the product details. Double-blind analysis substantially increases confidence in the data collection and research conclusions as it minimizes the potential for bias.

Consumer research should be conducted in a manner and with questions that do not imply or alert the respondent to the purpose of testing, or to the identity of the company or product being researched. These research projects can and should be double-blind to prevent unintentionally influencing panelists. To prevent bias and increase confidence in research results, a control group of respondents, or products that are confirmed to be of standard quality, should be used randomly in a double-blind process to enable accurate comparison. Subjective research such as cupping and tasting must include controls for variations in sensory panelists. Researchers should control for changes in results due to changes in the sensory panel and present explanations in the research conclusions.

*Blind testing separates physical analysis from sensory evaluation, so the sensory panel will not be influenced by visual appearance.*

As a researcher or a reader, be cautious of monadic testing, in which one product is evaluated at a time without a control sample for comparison. Monadic testing is useful and efficient when recording individual data, but it can be inaccurate when conducting comparative analysis. Analysis conclusions are more credible when a test sample is evaluated side-by-side with a control sample. Using a blind format for sample-to-standard testing, as is often used for product comparisons and timed stability testing, increases the credibility of the results.

Common coffee research tests such as triangle, duo-trio, tetrad and paired-comparison (see “Common Scientific Tests Used in Coffee,” below) are powerful tools when executed as prescribed to confirm samples are similar or dissimilar using statistical analysis. Before applying the research conclusions of these statistical discrimination tests, consider the protocols used by the researcher, which should be clearly described in written reporting.

Common Scientific Tests Used in Coffee

Triangle test (ISO 4120): A forced-choice test to determine if a perceptible sensory difference or similarity exists between two products. A single triad of three blind samples, with two that are alike and one that is different, is evaluated by an assessor. The assessor must select the one that is different. This method is also used for selecting, training and monitoring sensory panel members or assessors.
Duo-trio test (ISO 10399): A forced-choice test to determine if a sensory difference exists between two samples. An assessor is presented with two samples and a reference in a single triad. The assessor must select the sample they believe to be the most similar to the reference.
Tetrad test (ISO 6658/ASTM E3009): A forced-choice difference test involving two products. Two samples of each product are presented to the assessor who must group the similar pairs.
Paired comparison test (ISO 5495): A two-product directional difference test requiring the assessor to indicate which product presents the designated characteristic (for example, more berry sweetness or more nutty character).

For all of these tests, statistical tables are used to calculate the confidence in analysis results based on the prescribed number of assessors and the number of correct or incorrect responses.

TRANSPARENCY AND OBJECTIVITY IN REPORTING

Presenting the testing process and methods of analysis used is critical to the practical application of research. The level of specificity in research reporting will also aid readers in determining the relevance of the conclusion to their businesses and will enable them to replicate and confirm the analysis, as is recommended when practical. Equipment used for analysis should be cited in the report and should always be calibrated to manufacturer’s standards and operated using manufacturer’s directions. Readers should be skeptical and question conclusions when the details around data collection are not presented in scientific reports and articles. Details such as equipment manufacturers and models, ambient conditions, and the professional credentials of laboratory staff and panelists should be included.

Writers who cover coffee science and research can have great influence by adding context to consumer research. Knowing the methods used, the questions researchers asked, and specific consumer responses can help limit business risks. For example, asking about purchase intent—as in our scientific method example for what consumers will buy and why—is not the same as conducting preference tests or asking about likeability of different coffees. Consumer preference and likeability responses are often aspirational. Typically, there is a significant difference between quality scores, likability and what the consumer will purchase.

Selective reporting—focusing only on portions of the research and ignoring others—is also a serious concern for research and may cause conclusions to be invalidated because they are incomplete. For example, much can be learned from perceived failures, such as when the data does not support the hypothesis. Before discarding data collected during research, all aspects of the research should be verified for accurate execution according to the research plan. If everything has been executed and is functioning as expected, then the hypothesis should be challenged, not the data results. The credibility of the research conclusion relies on transparent data collection and reporting, thus any changes to the project must be recorded and presented in the final report for readers to consider.

Coffee writers must be diligent in separating opinions from facts and presenting each appropriately. They should also be cautious about reporting anecdotal observations. Personal accounts may not be indicative of typical experiences throughout the industry and should be presented accordingly.

Headlines, too, are a critical aspect of reporting and should align with the hypothesis and conclusions. Watch for headlines that are exaggerated or sensationalized to capture a reader’s attention and provoke interest. Misrepresenting the facts to attract readers may stifle progress by misleading industry stakeholders about the practical application of the conclusions. Rigorous research, accurate conclusions and detailed written reporting are all essential to help specialty coffee professionals make appropriate decisions.

*Always consider scientific methodology carefully before applying published research findings to your own business.*

FINAL CONSIDERATIONS FOR INTERPRETING AND APPLYING RESEARCH

Specialty coffee professionals often replicate research on our own to validate the data before applying the conclusions to our businesses. For this purpose, it is critical that all aspects of the research are accurately presented. If the results cannot be replicated independently, the conclusions should be questioned. Peer review by subject matter experts is a process used in academic research and often used in industry research to validate results. Peer review in research and writing can also identify when bias may have influenced a project.

Extrapolating conclusions by assuming presented research and data trends will continue may challenge the practical application to many businesses. In our scientific method example, extrapolating the research from specialty coffee roasters in Vermont and applying it to other companies and consumer markets may not be accurate.

As readers, we must evaluate the veracity of research when presented both formally in academic publications and informally through business-led experimentation. Before applying what we read to our own businesses, we must verify the scientific rigor used in the research and validate the conclusion as accurate and useful.

* * *

Spencer Turer is vice present of Coffee Enterprises in Hinesburg, Vermont. He is a founder of the Coffee Roasters Guild and the Coffee Writers Guild, a Q Arabica Grader, and a Specialty Coffee Association of America award winner. Turer is an active volunteer for the Specialty Coffee Association (SCA) and the National Coffee Association (NCA), and is an ambassador for the International Women’s Coffee Alliance (IWCA).