Unit 26 · Statistics: From a Sample to a Claim

Eight students report how many hours they slept last night: 2, 4, 4, 4, 5, 5, 7, 9. The mean is the total divided by the count: 2 + 4 + 4 + 4 + 5 + 5 + 7 + 9 = 40, and 40 ÷ 8 = 5 hours. The median is the middle value when the numbers are in order. With eight values the middle is between the 4th and 5th, which are 4 and 5, so the median is 4.5 hours. Mean and median are both measures of center, but they answer slightly different questions: the mean balances the total, the median splits the group in half.

Center alone is not enough. Two classes can both average 5 hours while one class is tightly packed near 5 and the other ranges from 1 to 10. The standard deviation measures spread as a typical distance from the mean. For the sleep data: subtract the mean from each value to get −3, −1, −1, −1, 0, 0, 2, 4. Square each: 9, 1, 1, 1, 0, 0, 4, 16, which sum to 32. Divide by the count, 32 ÷ 8 = 4, and take the square root: √4 = 2. The standard deviation is 2 hours, so a typical student is about 2 hours from the mean. A calculator’s sample standard deviation divides by one less than the count and gives a slightly larger number; either way, bigger means more spread out.

Shape is the third piece. Draw a histogram, a bar for each range of values, and look at the pile. If the left and right halves are mirror images, the shape is symmetric, and the mean and median match. If one tail stretches far out on the right, as with household incomes where a few very large numbers pull the mean up, the shape is skewed right and the mean sits above the median. In the sleep data the 9 pulls the mean (5) above the median (4.5), a small right skew. A common mistake is to report the mean for skewed data as if it were typical; for skewed data the median is usually the more honest center.

Many measurements, such as heights of adults of the same sex, pile up in a smooth bell shape: highest at the mean, symmetric, tails thinning out on both sides. This bell is the normal curve, and it is completely described by two numbers, its mean and its standard deviation. Suppose a model for adult women’s heights uses a mean of 64 inches and a standard deviation of 2.5 inches. The peak of the bell is at 64. One standard deviation on each side reaches from 64 − 2.5 = 61.5 to 64 + 2.5 = 66.5 inches.

The empirical rule says what fraction of a normal pile sits in those bands. About 68% of values lie within 1 standard deviation of the mean, about 95% within 2, and about 99.7% within 3. For the height model: about 68% of women are between 61.5 and 66.5 inches; about 95% are between 64 − 5 = 59 and 64 + 5 = 69 inches; about 99.7% are between 56.5 and 71.5 inches. Because the curve is symmetric, each half of a band holds half its percent. So 34% are between 64 and 66.5, and 34% are between 61.5 and 64.

Now the ends. If 95% are within 2 standard deviations, the other 5% are outside, split evenly: 2.5% below 59 inches and 2.5% above 69 inches. If 68% are within 1 standard deviation, 32% are outside, so 16% are above 66.5 inches. A common error is to forget the split and say 32% are above 66.5. Another is to add standard deviations without multiplying: 2 standard deviations above 64 is 64 + 2 × 2.5 = 69, not 64 + 2 = 66. Write the band endpoints first, then read off the percents.

A history test has mean 70 and standard deviation 8. Jada scores 82. How unusual is that? Measure her distance from the mean in units of standard deviation: 82 − 70 = 12 points above the mean, and 12 ÷ 8 = 1.5 standard deviations. That number, 1.5, is her z-score. The formula is z = (value − mean) ÷ standard deviation. A score of 58 gives z = (58 − 70) ÷ 8 = −12 ÷ 8 = −1.5, the same distance but below the mean. The sign matters: positive means above, negative means below.

Z-scores let you compare things measured on different scales. Suppose Test A has mean 500 and standard deviation 100, and Test B has mean 20 and standard deviation 5. Luis scores 650 on Test A and 28 on Test B. Which is the stronger result? On A: z = (650 − 500) ÷ 100 = 150 ÷ 100 = 1.5. On B: z = (28 − 20) ÷ 5 = 8 ÷ 5 = 1.6. The 28 is slightly farther above its mean, so it is the stronger performance, even though 650 is a much bigger number.

You can also go backwards. If a value has z = 2 on the 100-and-15 scale, it is 2 × 15 = 30 points above 100, so the value is 130. In general, value = mean + z × standard deviation. Two mistakes are common: dividing the raw value by the standard deviation without subtracting the mean first (82 ÷ 8 ≈ 10.25, which means nothing), and dropping the negative sign on values below the mean. Always subtract, then divide, and keep the sign.

Back to Nia’s height chart. A percentile is the percent of the group that falls below a value. On a normal curve, each z-score matches one percentile, and a short table gives the match. A value exactly at the mean has z = 0 and is the 50th percentile: half the group is below. A value with z = 1 is at about the 84th percentile, because 50% are below the mean and another 34% are between the mean and 1 standard deviation above, and 50 + 34 = 84. A value with z = −1 is at about the 16th percentile, since 50 − 34 = 16.

The table fills in the values between. The z-score for the 90th percentile is about 1.28. If six-year-old girls’ heights are modeled with mean 45 inches and standard deviation 2 inches, then the 90th percentile height is 45 + 1.28 × 2 = 45 + 2.56 ≈ 47.6 inches. That is roughly where the nurse’s dot for Nia fell. Going the other way, a score of 82 on the test with mean 70 and standard deviation 8 has z = 1.5, and the table gives about 93% below, so 82 is at the 93rd percentile.

To find the percent between two values, find each percentile and subtract. Between z = −1 (about 15.9%) and z = 0.5 (about 69.1%): 69.1 − 15.9 = 53.2% of values. Two cautions. First, a percentile is a rank, not an amount: the 90th percentile is not twice the 45th. Second, the table only works if the data are actually normal, which is the subject of the next lesson. A student who plugs a skewed data set into a z-table will get a tidy percent that is simply wrong.

The normal curve is a model, and a model can be wrong. Before using z-scores or the empirical rule, look at a histogram. Heights of adult men, measurement errors, and averages of many small effects tend to be close to normal. Household income is not: most households are near the middle, but a few earn enormous amounts, and the tail on the right stretches for miles. Time waiting in line is not either: it cannot go below zero, but it can run very long. A normal model applied to either would predict percents that never happen.

Here is a quick test. Suppose a data set of wait times has mean 10 minutes and standard deviation 8 minutes. If the data were normal, the empirical rule would put 2.5% of waits below 10 − 2 × 8 = −6 minutes. A negative wait is impossible, so the model cannot fit. Any time the mean is less than about two standard deviations from a hard boundary such as zero, the normal curve is a poor choice. A second test is to count: if far fewer than 68% of the values are within 1 standard deviation of the mean, the shape is not a bell.

Bimodal data, which have two separate peaks, also break the model. Ages at a middle-school open house are bimodal: one hump near 13 for students and one near 42 for parents. The mean, somewhere around 28, describes almost nobody, and a z-score built from it is meaningless. When the model does not fit, use percentiles computed directly from the sorted data instead of from a z-table, and report the median and the middle 50% rather than the mean and standard deviation.

Flip a coin twice. The chance of heads on the first flip is 1/2, and the coin has no memory, so the chance of heads on the second flip is still 1/2 no matter what happened first. Two events are independent when one happening does not change the probability of the other. For independent events, the multiplication rule gives the chance that both happen: P(A and B) = P(A) × P(B). So P(heads, then heads) = 1/2 × 1/2 = 1/4. You can check by listing the four equally likely outcomes, HH, HT, TH, TT: exactly one of the four is HH.

A basketball player makes 80% of her free throws. If her shots are independent, the chance she makes two in a row is 0.8 × 0.8 = 0.64. The chance she misses at least one of the two is everything else: 1 − 0.64 = 0.36. That subtraction uses the complement: P(not A) = 1 − P(A). The chance of three makes in a row is 0.8 × 0.8 × 0.8 = 0.512, so even a good shooter misses at least one of three shots almost half the time.

Drawing without replacement is the classic case where independence fails. Pull one card from a standard 52-card deck: P(ace) = 4/52. Keep it out and pull a second: if the first was an ace, only 3 aces remain among 51 cards, so P(second ace) = 3/51. The chance of two aces is 4/52 × 3/51 = 12/2652 = 1/221. A common mistake is to multiply 4/52 × 4/52 as if the deck had been reset; that overstates the chance. Before multiplying, ask whether the first event changes the odds for the second.

A survey of 200 students asks two yes-or-no questions: do you play a school sport, and do you have a part-time job? The results fit in a two-way table. Of the 120 who play a sport, 40 have a job and 80 do not. Of the 80 who do not play a sport, 30 have a job and 50 do not. Column totals: 70 have a job, 130 do not. The overall chance that a randomly chosen student has a job is 70/200 = 0.35.

A conditional probability asks about a smaller group. Given that a student plays a sport, what is the chance they have a job? Now only the 120 sport players matter, and 40 of them have jobs: P(job | sport) = 40/120 = 1/3 ≈ 0.333. The vertical bar reads ’given.’ Flip the condition: P(sport | job) = 40/70 = 4/7 ≈ 0.571, because now the 70 job holders are the whole world and 40 of them play a sport. Notice these two are different numbers; mixing them up is the most common error in this topic.

Conditional probability gives a clean test for independence: two events are independent exactly when P(A | B) = P(A). Here P(job | sport) ≈ 0.333 but P(job) = 0.35, and P(job | not sport) = 30/80 = 0.375. The numbers are close but not equal, so job and sport are slightly linked: sport players are a little less likely to hold a job. When a table has raw counts, always divide by the total of the row or column you are conditioning on, never by the grand total.

Draw one card from a standard deck. What is the chance it is an ace or a heart? There are 4 aces and 13 hearts, but the ace of hearts is in both groups. Adding 4 + 13 = 17 counts that card twice. The addition rule fixes this: P(A or B) = P(A) + P(B) − P(A and B). So P(ace or heart) = 4/52 + 13/52 − 1/52 = 16/52 = 4/13. Check by counting: 4 aces plus the 12 hearts that are not aces gives 16 favorable cards out of 52.

Some events cannot happen together. Rolling a 1 and rolling a 2 on the same die are mutually exclusive: the overlap is empty and P(A and B) = 0. Then the rule simplifies to P(A or B) = P(A) + P(B) = 1/6 + 1/6 = 2/6 = 1/3. Now try ’even or greater than 4.’ Even is {2, 4, 6}, probability 3/6. Greater than 4 is {5, 6}, probability 2/6. Both at once is {6}, probability 1/6. So P = 3/6 + 2/6 − 1/6 = 4/6 = 2/3. Listing the winning faces, {2, 4, 5, 6}, confirms 4 out of 6.

Two mistakes come up again and again. The first is adding probabilities for events that overlap without subtracting: a student says P(even or greater than 4) = 5/6, which counts the 6 twice. The second is confusing ’or’ with ’and’: ’and’ asks for both, which is the overlap and is found by multiplying (when independent) or by reading the table cell, never by adding. Word the question to yourself first: is it ’both,’ or is it ’at least one’?

A club sells 100 raffle tickets at $2 each, and one ticket wins a $100 prize. If you buy one ticket, what do you get back on average? You win $100 with probability 1/100 and $0 with probability 99/100. The expected value of the prize is 100 × 1/100 + 0 × 99/100 = $1. You paid $2, so your expected net result is 1 − 2 = −$1 per ticket. That does not mean you lose exactly $1; you lose $2 or gain $98. It means that over many tickets your average result is a loss of $1 each, which is exactly how the club makes money.

The general recipe: list every outcome, multiply each by its probability, and add the products. A game pays the number of dollars showing on one roll of a fair die. The expected payout is 1 × 1/6 + 2 × 1/6 + 3 × 1/6 + 4 × 1/6 + 5 × 1/6 + 6 × 1/6 = 21/6 = $3.50. A fair price to play would be $3.50; at $4 the house wins 50 cents per game on average. A common error is to average the outcomes without their probabilities, which only works when all outcomes are equally likely, as they happen to be here.

Expected value guides decisions, but it is not the only consideration. A phone-repair plan costs $60 a year. Suppose a phone breaks with probability 0.1 in a year and the repair costs $300. Expected repair cost without the plan: 0.1 × 300 + 0.9 × 0 = $30. The plan costs $60, twice the expected loss, so on average it is a bad deal. But a person who could not possibly pay $300 at once might reasonably buy it anyway: expected value tells you the average, not how much a bad outcome would hurt. Report the expected value, then talk about the risk.

A free-throw shooter who makes half her shots takes 4 shots. How likely is exactly 2 makes? This is a binomial setting: a fixed number of trials (4), each with two outcomes (make or miss), the same probability of success on each (0.5), and independent trials. Write all 16 equally likely sequences of M and X. Exactly 2 makes happens in 6 of them: MMXX, MXMX, MXXM, XMMX, XMXM, XXMM. So P(exactly 2) = 6/16 = 0.375.

Counting sequences by hand gets slow, so use the pattern. The number of ways to choose which k of the n trials are successes is written C(n, k). For n = 4: C(4, 0) = 1, C(4, 1) = 4, C(4, 2) = 6, C(4, 3) = 4, C(4, 4) = 1. Each specific sequence with k successes has probability p^k × (1 − p)^(n − k). Multiply: P(k successes) = C(n, k) × p^k × (1 − p)^(n − k). For an 80% shooter taking 3 shots, P(exactly 2 makes) = C(3, 2) × 0.8^2 × 0.2^1 = 3 × 0.64 × 0.2 = 0.384.

Check the whole distribution for that shooter: P(0) = 1 × 0.2^3 = 0.008; P(1) = 3 × 0.8 × 0.04 = 0.096; P(2) = 0.384; P(3) = 0.8^3 = 0.512. The four add to 1.000, which is a good sign that nothing was dropped. Two things break the binomial setting: a probability that changes from trial to trial (drawing cards without replacement), and a number of trials that is not fixed in advance (shooting until the first miss). Check the four conditions before using the formula.

If a 70% free-throw shooter takes 100 shots, how many makes should we expect? Common sense says about 70, and the formula agrees: the mean of a binomial count is n × p = 100 × 0.7 = 70. The count varies from night to night, and its standard deviation is √(n × p × (1 − p)) = √(100 × 0.7 × 0.3) = √21 ≈ 4.6 makes. For a large n the binomial pile is close to a normal curve, so the empirical rule applies: on about 95% of 100-shot nights she makes between 70 − 2 × 4.6 ≈ 61 and 70 + 2 × 4.6 ≈ 79.

That spread lets you judge a claim. A student guesses randomly on a 20-question multiple-choice test with 4 choices each, so p = 0.25. Mean = 20 × 0.25 = 5 correct. Standard deviation = √(20 × 0.25 × 0.75) = √3.75 ≈ 1.94. Suppose the student gets 12 right. The z-score is (12 − 5) ÷ 1.94 ≈ 3.6, far beyond 3 standard deviations. Pure guessing almost never produces that, so the honest conclusion is that the student knew something (or copied). The formula turned a hunch into a number.

Flip a coin 100 times and get 65 heads. Is the coin fair? For a fair coin, mean = 50 and standard deviation = √(100 × 0.5 × 0.5) = √25 = 5. Then 65 heads is (65 − 50) ÷ 5 = 3 standard deviations above the mean, a result the empirical rule puts at about 1 in 1,000 for the upper tail alone. That is strong evidence against fairness, and it is the first step toward the hypothesis tests of the next chapter. A common slip is to take √(n × p) and forget the (1 − p) factor: √50 ≈ 7.1 would wrongly make 65 heads look ordinary.

Chicago Public Schools serves more than 300,000 students. A student newspaper wants to know what fraction of them ride a CTA bus or train to school. Asking every student is a census, which is slow and expensive. Instead the paper asks 500 students. The whole group of interest, every CPS student, is the population. The 500 who are asked are the sample. If 310 of the 500 say yes, the sample proportion is 310 ÷ 500 = 0.62, or 62%.

That 62% is a statistic: a number computed from the sample. The true fraction of all CPS students who ride the CTA, whatever it is, is a parameter: a fixed number about the population that we usually never learn exactly. Statistics estimate parameters. A second sample of 500 might give 295 ÷ 500 = 0.59. Neither is wrong; each is an estimate, and the rest of the chapter is about how close such estimates tend to be. Keep the vocabulary straight: parameter goes with population, statistic goes with sample.

The same words apply to averages. If the mean commute time in the sample of 500 is 34 minutes, that sample mean is a statistic estimating the parameter, the mean commute of all 300,000 students. Two common confusions: calling the sample the population (the 500 students are not ’everyone’), and treating the statistic as if it were exact (writing ’exactly 62% of CPS students ride the CTA’). A careful sentence says ’about 62% of the students surveyed,’ and then asks how far off that might be.

Three ways to collect data answer three different kinds of question. A survey asks people to report something: who they will vote for, how many hours they sleep, whether they ride the CTA. An observational study records what people already do and looks for patterns, such as comparing the grades of students who eat breakfast with the grades of those who skip it. An experiment actively assigns a treatment: the researcher decides which students get breakfast and which do not, then compares.

The difference matters because of confounding. Suppose the observational study finds that breakfast eaters average 8 points higher. Did breakfast cause the gain? Maybe. But students who eat breakfast may also sleep more, have more organized mornings, or come from homes with more support, and any of those could be the real reason. A hidden variable tangled up with the one you are studying is a confounder, and observation alone cannot untangle it. The study shows an association, not a cause.

An experiment breaks the tangle by random assignment. Flip a coin for each of 200 volunteers: heads eats a provided breakfast for a month, tails does not. Chance, not personal habits, decides who is in which group, so sleep, family support and everything else are spread roughly evenly across both groups. If the breakfast group then scores higher, breakfast is the most reasonable explanation. Whenever you read that something ’causes’ or ’leads to’ an outcome, ask: did the researchers assign it, or just observe it?

A simple random sample is chosen so that every possible group of the chosen size has the same chance of being picked. To draw 60 students at random from a school of 1,200, number the students 1 to 1,200, and let a random number generator pick 60 different numbers. No student can lobby to be included or excluded, and no teacher’s opinion of who is ’typical’ enters in. That is the point: chance does the choosing, so the sample’s tilt, if any, is only the luck of the draw, which the next lessons show how to measure.

Bias is a systematic tilt, a reason the sample tends to differ from the population in a predictable direction. The Literary Digest showed two kinds. Undercoverage means parts of the population never had a chance to be picked: no phone, no car, no subscription meant no ballot. Nonresponse bias arises when the people who answer differ from those who do not. A third kind, voluntary response, is worse still: a website poll that anyone can click answers only the question ’who felt strongly enough to click?’ A survey of sports fans who volunteer to rate the team’s coach will not describe the average fan.

Convenience samples, such as asking the first 50 people at the cafeteria door, share the same flaw: the people at the door at 11:15 are not a random slice of the school. Wording matters too. ’Do you support the plan to cut wasteful spending?’ and ’Do you support the plan to cut school programs?’ can describe the same plan and get opposite answers. When you judge a sample, ask three questions: who was left out, who did not answer, and what exactly was asked?

Imagine a jar of 10,000 beads, exactly half red and half white, so the true proportion of red is p = 0.5. Scoop out 100 beads at random and count the red ones. You might get 47. Put them back, stir, scoop again: 53. Again: 49, 55, 44. None of these is wrong. This bead-to-bead wobble is sampling variability, and it happens even when the sampling is done perfectly. The parameter never moved; only the sample did.

Now imagine doing 1,000 scoops of 100 and making a histogram of the 1,000 sample proportions. That histogram is the sampling distribution. For random samples it has a beautiful regularity: it is centered at the true value 0.5, it is roughly bell-shaped, and its spread depends on the sample size. Most scoops of 100 fall between 0.40 and 0.60. If instead each scoop held 400 beads, the histogram would be much narrower, with most scoops between 0.45 and 0.55.

Two lessons follow. First, a single sample’s proportion is a draw from this distribution, so it is ’off’ by a typical amount you can estimate. Second, bigger samples shrink that typical amount, but only for random samples. If the jar were stirred badly so that red beads sat on top, every scoop would run red, and scooping 400 instead of 100 would not help. Sampling variability is chance and can be reduced by size; bias is a tilt and cannot.

You do not need a jar of beads. To simulate sampling 100 people from a population where half say yes, flip a coin 100 times and count heads. Each run of 100 flips is one simulated sample. Do it 50 times and you have a rough sampling distribution. A computer can do 10,000 runs in a second, but the idea is the same: rebuild the chance process, run it many times, and look at what the results usually do.

The simulated histogram has a spread you can predict with the binomial formula from the last chapter. For a proportion, the standard error is √(p × (1 − p) ÷ n). With p = 0.5 and n = 100: √(0.5 × 0.5 ÷ 100) = √0.0025 = 0.05. So about 95% of simulated sample proportions land within 2 × 0.05 = 0.10 of 0.5, that is, between 0.40 and 0.60, matching the bead scoops. With n = 400: √(0.25 ÷ 400) = √0.000625 = 0.025, so 95% land between 0.45 and 0.55.

Notice the pattern: quadrupling n from 100 to 400 cut the spread in half, from 0.05 to 0.025. The spread shrinks with the square root of n, not with n itself. To cut it in half again you would need 1,600, and to reach a spread of 0.01 you would need 2,500. Doubling the sample only makes it about 1.4 times more precise. That is why polls of about 1,000 people are so common: they give a standard error near 0.016, a margin of about 3 points, and going to 10,000 people costs ten times as much just to bring the margin down to about 1 point.

A poll of 1,000 Illinois adults finds 54% in favor of a new transit plan. The true percent for all adults is unknown, but the sampling distribution tells us how far a sample of 1,000 typically strays. Standard error: √(0.54 × 0.46 ÷ 1,000) = √0.0002484 ≈ 0.0158. Two standard errors is about 0.032, or 3.2 percentage points. That is the margin of error. The confidence interval is 54% ± 3.2%, from 50.8% to 57.2%. A pollster would write: ’54%, with a margin of error of about 3 points.’

What does ’95% confidence’ mean? Not that there is a 95% chance the truth is in this one interval; the truth is a fixed number and is either in it or not. It means the method works 95% of the time: if you took many samples of 1,000 and built an interval from each, about 95 of every 100 intervals would capture the true value and about 5 would miss. You do not know whether yours is one of the 5, which is why a result at 50.8% to 57.2% supports ’more than half’ only cautiously.

A quick rule for a 95% margin of error when p is near 0.5 is 1 ÷ √n. For n = 1,000: 1 ÷ √1,000 ≈ 1 ÷ 31.6 ≈ 0.032, matching the careful calculation. For n = 400 it is 1 ÷ 20 = 0.05, or 5 points; for n = 2,500 it is 1 ÷ 50 = 0.02, or 2 points. Two cautions: the margin of error covers only random sampling variability, not bias, and a poll with 60% ± 5% does not mean ’60% is right and the rest is error.’ Every value from 55% to 65% is consistent with the data.

A friend hands you a coin and says it is fair. You flip it 100 times and get 60 heads. Fair coins usually give around 50, but 60 is not impossible. How do you decide? A hypothesis test works like a courtroom. The claim on trial, called the null hypothesis, is ’the coin is fair, p = 0.5.’ It gets the benefit of the doubt. The alternative hypothesis is what you would conclude if the evidence is strong enough: ’the coin favors heads.’ The data are the evidence, and the question is whether they are consistent with the null.

Assume the null is true and measure how unusual the result is. For a fair coin in 100 flips, the expected count is 50 with standard deviation √(100 × 0.5 × 0.5) = 5. Then 60 heads is (60 − 50) ÷ 5 = 2 standard deviations above expected. From the normal table, a fair coin produces 60 or more heads only about 2.3% of the time. That is rare. If we would see this in fewer than 5 of 100 tries, the usual convention says the result is statistically significant, and we reject the null: the evidence points to a coin that favors heads.

The logic has two sides that students often flip. Rejecting the null does not prove the alternative; it says the data would be surprising if the null were true, so the null is doubtful. And failing to reject does not prove the null; 54 heads is consistent with a fair coin, but also with a coin that has p = 0.52. A test can be wrong in two ways: rejecting a true null (convicting an innocent coin, which happens about 5% of the time at the 5% cutoff), or failing to reject a false null (letting a guilty coin go). Small samples make the second error common.

The number that measures surprise is the p-value: the probability, assuming the null hypothesis is true, of getting a result at least as extreme as the one observed. For the coin with 60 heads in 100 flips, the p-value is about 0.023, the fair-coin chance of 60 or more heads. A p-value is not the chance the null is true; it is the chance of data like yours if the null is true. Those are different sentences, and swapping them is the most common misreading of statistics in news stories.

You can find a p-value by simulation instead of a table. Take the cafeteria claim that 80% of students like the menu, with a sample of 50 that found only 32. Program a computer to draw 50 ’students’ at random from a population where 80% say yes, count the yeses, and repeat 10,000 times. Then count how many of the 10,000 runs gave 32 or fewer. If about 60 runs did, the p-value is roughly 60 ÷ 10,000 = 0.006. The claim would produce a sample this low only about 6 times in 1,000, so we reject it.

The usual cutoff, called the significance level, is 0.05: reject the null when p is below it. But the cutoff is a convention, not a law of nature. A p-value of 0.06 is barely different from 0.04, and a p-value of 0.30 means the data are ordinary under the null, not that the null is true. Try one: a candidate claims 50% support, a random sample of 200 shows 55%. Standard error √(0.5 × 0.5 ÷ 200) ≈ 0.0354, so 0.55 is about 1.41 standard errors above 0.5, and the upper-tail p-value is about 0.08. Above 0.05, so not significant, even though 55 sounds like more than 50.

A tutoring company tests a new app on 50,000 students, split at random into two groups. The app group averages 70.5 on a final exam and the control group averages 70.0. Because the samples are enormous, the standard error of the difference is tiny, about 0.2 points, so a 0.5-point gap is 2.5 standard errors and the p-value is below 0.05. The result is statistically significant. It is also nearly worthless: half a point on a 100-point exam changes nobody’s grade. The gap is real, but it is not important.

That is the difference between statistical significance and practical significance. Statistical significance answers: could this gap be a fluke of sampling? With 50,000 students, almost no gap is a fluke. Practical significance answers: is the gap big enough to matter to a student, a doctor or a city? Only judgment, not a formula, answers that. A report should always give the effect size, the actual difference or ratio, alongside the p-value, so a reader can see whether 0.5 points is worth an app subscription.

The mirror-image mistake happens with small samples. Suppose a new reading program is tried on 12 students and their scores rise 8 points on average, but with only 12 students the standard error is about 5 points, so the gain is 1.6 standard errors and the p-value is about 0.11, not significant. That does not mean the program failed; it means 12 students cannot tell an 8-point gain from noise. An honest summary says ’promising, but the study was too small to be sure.’ Sample size decides what a study can detect; it does not decide what matters.

’Daily coffee linked to 20% lower risk!’ Before sharing that, read it like a statistician. First, linked to is observational language: coffee drinkers may differ in exercise, income or sleep, so this is an association until an experiment says otherwise. Second, 20% lower than what? If the risk falls from 5 in 1,000 to 4 in 1,000, that is a 20% relative drop but only 1 in 1,000 in absolute terms. Relative percents make small changes sound enormous. Ask for the two actual rates every time.

Third, look for the sample. ’Nine out of ten dentists’ means little without knowing how many dentists were asked, how they were chosen, and who asked them. A survey of 20 dentists selected by a toothpaste company is not the same as a random sample of 2,000. Also look for the margin of error. If a poll says a candidate leads 48% to 46% with a margin of error of 3 points, the intervals overlap, and the honest headline is ’too close to call,’ not ’candidate takes the lead.’ A 2-point gap inside a 3-point margin is noise.

Fourth, ask who benefits. A study funded by the company whose product it praises is not automatically wrong, but it deserves a closer look, and so does a result that has never been repeated by anyone else. Finally, watch the axes on any graph: a bar chart whose vertical axis starts at 90 instead of 0 can make a 2% difference look like a cliff. None of these checks requires a formula. They require the habit of asking ’compared to what, measured how, and says who?’

To show that a treatment causes an outcome, an experiment needs four ingredients. First, a control group that does not get the treatment, so there is something to compare against. Second, random assignment of subjects to treatment or control, so the groups are alike except for the treatment. Third, when people’s expectations could affect the result, a placebo, a fake treatment that looks real, and blinding, so subjects (and ideally the people measuring them) do not know who got which. Fourth, enough subjects that a real effect is not lost in sampling variability.

The 1954 field trial of the Salk polio vaccine is the classic example. Hundreds of thousands of American children were randomly assigned to receive the vaccine or a placebo injection, and neither the children nor the doctors diagnosing polio knew which was which. Polio cases were much rarer in the vaccinated group, and because assignment was random and diagnosis was blind, that difference could not be explained by which families volunteered or by doctors expecting vaccinated children to be healthier. The vaccine was declared effective and released to the public.

Compare a weaker design: give the vaccine to children whose parents ask for it and compare them with everyone else. Parents who ask for a vaccine tend to differ in income, education and health habits, so any difference in polio rates would be tangled with those. The lesson for reading any claim of cause: find the control group, find the word random, and ask whether anyone could have known who got what. When an experiment is impossible or unethical, as with smoking, researchers rely on many observational studies that point the same way, plus a plausible mechanism, and they say so.