Sensostat Try the demo

Textbook · Module 09

Statistical Analysis of Sensory Data

Binomial tests and critical values, the Friedman test, ANOVA and mixed models, PCA/CA/MFA, power and panel size — with reproducible examples.

Module 09 of 11~35 min readinteractive + quiz

Statistics is how we tell signal from noise

This module brings the whole statistical toolkit of sensory science together in one place. The Sensostat rule: every numerical conclusion is accompanied by a test, a significance level and an interpretation in the client's language.

The binomial model of discrimination tests

If there is no difference, the assessor guesses with probability p₀ (triangle/tetrad — 1/3, duo-trio/paired — 1/2, two-out-of-five — 1/10). The number of correct answers X ~ Binomial(n, p₀). A difference is proven if X ≥ the critical value (the smallest x for which P(X ≥ x) ≤ α).

The binomial distribution live interactive

The orange zone is the critical region (difference proven). The vertical line is your observed result.

"Significant / not significant" is not the end: a confidence interval for p_d

The p-value says the difference is proven, but not how large it is. The proportion of discriminators is estimated as p̂_d = (p̂_c − p₀) / (1 − p₀), and its precision is given by the Clopper–Pearson confidence interval (the "exact" interval for a binomial proportion) rescaled to p_d:

CI(p_c) = [ B(α/2; x, n−x+1) ; B(1−α/2; x+1, n−x) ] → CI(p_d) = (CI(p_c) − p₀) / (1 − p₀)

Example: a triangle test, n = 48, x = 22 correct (the critical value is 22, i.e. p ≈ 0.048, a "borderline" difference). Then p̂_d = (22/48 − 1/3) / (2/3) = 0.188, and the 95 % Clopper–Pearson interval:

QuantityEstimate95 % CI
p_c (proportion correct)0.458[0.314 ; 0.608]
p_d (proportion of discriminators)0.188[0.000 ; 0.412]
Interpretation for the client: "The difference is proven (exact binomial test, p ≈ 0.048), but it can be anything from negligible to affecting up to 41 % of consumers — with this panel we cannot tell. If the decision is expensive (switching supplier), extend the panel to 90–100 and refine the estimate". An interval pressed against zero at the critical value is the norm, not a glitch: proving a difference and estimating its size are different tasks with different price tags.
# Clopper–Pearson interval and conversion to p_d (triangle test)
from scipy.stats import beta
n, x, p0 = 48, 22, 1/3
lo = beta.ppf(0.025, x, n - x + 1)      # 0.3137
hi = beta.ppf(0.975, x + 1, n - x)      # 0.6083
pd_hat = (x/n - p0) / (1 - p0)          # 0.188
pd_ci  = (max(0, (lo - p0)/(1 - p0)), (hi - p0)/(1 - p0))  # (0.000 ; 0.412)

Replicates: why "48 answers from 24 assessors" ≠ "48 answers"

In protocols with replicates (each assessor repeats the test r times) the observations within one person are dependent: one is consistently attentive, another is not. The binomial model treats all n·r answers as independent and underestimates the variance. The correct model is the beta-binomial one: each assessor has their own probability of a correct answer, beta-distributed across the panel; the parameter γ (overdispersion) measures the excess variability.

Var(X) = n · r · p(1−p) · (1 + (r−1)γ)

The factor (1 + (r−1)γ) is variance inflation. Example (triangle test, 24 assessors × 2 replicates = 48 answers, p₀ = 1/3):

ModelVar(X)SDComment
Binomial (γ = 0)10.673.27all answers independent
Beta-binomial, γ = 0.0511.203.35weak panel heterogeneity
Beta-binomial, γ = 0.1011.733.43typical practice
Beta-binomial, γ = 0.2012.803.58the panel is highly heterogeneous

At γ = 0.10–0.20 the critical value grows by 1–2 answers — so a protocol that "passed" by the binomial table may turn out non-significant. For the triangle test there is also a chance-corrected version of the model (chance-corrected beta-binomial): it estimates the proportion of discriminators p_d directly, accounting for heterogeneity. Implementation — the betabin functions of the sensR package; in Sensostat the model is applied automatically as soon as replicates are specified in the design.

Design rule: replicates increase power through panel precision, but they do not replace assessors: 24 × 2 is weaker than 40 × 1 at the same cost per answer. Replicates are justified when people are few and the sample is cheap (training, panel monitoring) — and they always require beta-binomial processing.

Ranking: the Friedman test

Ranking (ISO 8587:2006 / GOST R ISO 8587-2015) is analysed by the rank analogue of two-way ANOVA — the Friedman test: χ²F = 12/(n·k·(k+1)) · ΣRᵢ² − 3n(k+1), where Rᵢ are the sample rank sums.

Example. 12 assessors rank 4 coffees by bitterness (1 = least bitter). Rank sums: A=20, B=22, C=32, D=46.

χ²F = 12/(12·4·5)·(20²+22²+32²+46²) − 3·12·5 = 0.05·4024 − 180 = 21.2 > χ²0.05;3 = 7.815 (p < 0.001) → the samples differ. Post-hoc: a difference of rank sums is significant if it exceeds LSD ≈ t·√(n·k(k+1)/6) ≈ 12.7. Conclusion: D is significantly more bitter than A and B; A, B and C do not differ from each other.

ANOVA and mixed models for descriptive data

The basic panel model: Y = product + assessor + product×assessor (+ replicate). Assessor is a random factor; product significance is tested against the product×assessor interaction, not against the residual (see the analysis in module 05). Post-hoc — Tukey HSD.

A common mistake. Treating the assessor as a fixed factor inflates significance: F is computed against the residual with many degrees of freedom. The "all attributes significant" conclusion from such a model is an artefact.

Worked example: variance decomposition by hand

A QDA session, attribute "bitterness" (0–10 scale): 3 products × 4 assessors × 2 replicates. The total sum of squares decomposes into four components:

SS_total = SS_product + SS_assessor + SS_inter + SS_error
df: 23 = 2 + 3 + 6 + 12
SourceSSdfMSFp
Product29.08214.5412.03¹0.008
Assessor2.1330.711.55²0.254
Product × Assessor7.2561.212.64²0.072
Error (replicates)5.50120.46
Total43.9623

¹ against the interaction MS (mixed model, assessor random) · ² against the error MS.

See what the correct denominator does: the same product effect against the error MS would give F = 31.7 and p = 0.000016 — four orders of magnitude "stronger"! This is an illusion: the error MS measures only the replicate noise of one person, while we generalise the conclusion to the population of assessors. A significant interaction (here p = 0.072 — borderline) warns: the panel is not fully in agreement, assessors rank the products differently, and the product means should be read with caution.

Product means: A = 3.25; B = 5.75; C = 5.38. Tukey's post-hoc (honestly significant difference, HSD) based on the interaction:

HSD = q(0.05; k=3, df=6) · √(MS_inter / (J·r)) = 4.339 · √(1.208 / 8) = 1.69

Differences: B−A = 2.50 > 1.69 — significant; C−A = 2.13 > 1.69 — significant; B−C = 0.38 < 1.69 — no difference. Conclusion for the client: "The products differ in bitterness (mixed ANOVA, F(2,6) = 12.03; p = 0.008): recipe A is significantly milder than both competitors; B and C do not differ. Panel agreement is borderline (interaction p = 0.072) — we recommend a calibration session on the bitterness scale".

# The same calculation — reproducible (Python, statsmodels)
import pandas as pd, statsmodels.formula.api as smf
from statsmodels.stats.anova import anova_lm
df = pd.DataFrame({
 'y':[3,4,4,3,2,3,3,4, 6,7,4,5,7,6,5,6, 5,6,6,5,4,5,6,6],
 'prod': ['A']*8+['B']*8+['C']*8,
 'ass': ['J1']*2+['J2']*2+['J3']*2+['J4']*2})
m = anova_lm(smf.ols('y ~ C(prod)*C(ass)', df).fit(), typ=2)
F_prod = m.loc['C(prod)','MS'] / m.loc['C(prod):C(ass)','MS']   # 12.03

PCA: the product map

From the "panel means: products × attributes" matrix, a PCA (principal component analysis) is built with centring and scaling of the attributes. The biplot shows products (points) and attributes (vectors) simultaneously: the vector length is the attribute contribution, the direction — which products it is associated with.

PCA biplot: five chocolates, six attributes (module 05 data)

PC1 (84.0 % of the variance) is the "bitter cocoa ↔ sweet milky" axis; PC2 (14.6 %) singles out the acidity of "Ruby". Together the two components explain 98.6 % of the differences.

Related methods: MFA (multiple factor analysis) for napping and multi-block data; GPA (generalised Procrustes analysis) for free-choice profiling; the RV coefficient for comparing configurations (RV ≥ 0.8 — configurations are similar); AHC — product clustering.

The mathematics of PCA in three steps

1) Centre the columns: X ← X − mean → 2) S = XᵀX/(n−1) — covariances → 3) S v_j = λ_j v_j

The eigenvector v_j is a direction (a principal component), the eigenvalue λ_j is the variance along it. Product scores are the projections X·v_j, loadings are the attributes' links with the component. The share of explained variance: λ_j / Σλ. In the biplot above, the first two components explain the bulk of the differences between the five chocolates: PC1 separates the dark ones from the white and milk (cocoa, bitterness versus sweetness and vanilla), PC2 — ruby with its acidity. Practical rules: components with λ < 1 (the Kaiser rule) are usually noise; attributes whose arrows point the same way duplicate each other — candidates for trimming the lexicon.

The family is wider than PCA: CA (correspondence analysis) — for CATA frequencies; MFA — for data blocks from different respondents (napping, module 07); GPA (generalized Procrustes analysis) — when each assessor has their own coordinate system (free choice profiling); PLSR — when sensory data explain hedonic response (preference mapping, module 06).

Consumer statistics

  • Hedonic tests: means ± CI, mixed ANOVA (product fixed, respondent random), Tukey; preference segmentation — hierarchical clustering (Ward).
  • CATA: Cochran's Q test per attribute, post-hoc — pairwise McNemar with correction; the "products × attributes" map — correspondence analysis (CA).
  • JAR: penalty analysis — mean drop and t-test/bootstrap, the ≥ 20 % rule (module 06).
  • Preference mapping: external (PLSR: liking ~ sensory coordinates) and internal (MDPREF) — liking drivers.

Cochran's Q for CATA — a worked example

Four yogurts, 12 consumers, the attribute "creamy" (0/1 ticks). Product sums: A = 10, B = 4, C = 9, D = 3. The statistic:

Q = (k−1) · [ k·ΣCⱼ² − (ΣCⱼ)² ] / [ k·ΣRᵢ − ΣRᵢ² ] → χ²(k−1)

where Cⱼ are the product sums, Rᵢ the respondent sums, k = 4 products. Here Q = 10.57; χ²(3): p = 0.014 — the citation frequencies differ significantly. Post-hoc — pairwise McNemar on discordant pairs: A–D: 8/1, p = 0.039; C–D: 7/1, p = 0.070; A–B: 8/2, p = 0.109.

A multiple-comparison correction is mandatory. With k = 4 products there are 6 pairs; Bonferroni gives α′ = 0.0083 — and then even the A–D pair is formally non-significant. An honestly worded conclusion: "the 'creamy' attribute discriminates the products (Cochran's Q = 10.57, df = 3, p = 0.014); direction — products A and C are perceived as creamier than D; pairwise differences after the Bonferroni correction do not reach significance at n = 12 — to rank the pairs, increase the sample to 60+".

Power: how much data is enough

Panel size is chosen before the study, not after. For discrimination tests — from (α, β, p_d): an exact search is implemented in the module 04 calculator. Benchmarks for the triangle test at α=0.05, β=0.10: p_d=50 % → n≈20–25; p_d=30 % → n=53; p_d=20 % → n=117.

8–12trained DA panel (QDA), 2–3 replicates
24–30screening discrimination tests
100+consumer hedonic tests (150+ for segmentation)

For consumer tests, power is computed from the smallest meaningful difference of means (Δ liking) and the variance — using t-test formulas.

Further reading

Test yourself

module 09 quiz

Three questions on statistics

1. In the mixed panel model, the significance of the "product" factor is tested…

Assessor is a random factor, so the product F-test goes against MS(product×assessor). Against the residual — only if there is no interaction in the model.

2. In the Friedman example χ²_F = 21.2 against a critical value of 7.815. Conclusion?

21.2 > 7.815 → the null hypothesis of equality is rejected at the 0.05 level (in fact p < 0.001).

3. Two principal components in the chocolate example explain 98.6 % of the variance. This means…

High explained variance of two components is a sign that the biplot adequately reflects the data structure.

From theory to industries

Module 10: how sensory science works in the food industry, cosmetics, QC and marketing.

Module 10 →