BIZOMAT MAKEOS · Quality
Process Change Trial (Sample Size, Welch t-Test, Improvement Decision)
How many parts should I run before and after a process change to detect the improvement I care about, and do my trial results prove the change made a real difference?
Planning and evaluation of a before-and-after trial of a process change: the sample size per group to detect your smallest worthwhile difference at the chosen significance and power (equivalent to the sample size for a mean tool with the margin set for a difference), and the Welch two-sample t-test on the results with p-value and confidence interval (as the two-sample t-test tool), then the decision: improvement confirmed, no proven difference, or made worse, with checks of sample size and practical size of the change.
Open the calculatorRuns in your browser. No account needed.
How this tool works
Inputs, units and where to find them
| Input | Unit | Typical source |
|---|---|---|
| Plan | ||
| Expected standard deviation | Standard deviation you expect, from process history. | |
| Smallest difference worth detecting | Smallest change in the mean that would matter to you. | |
| Significance level (two-sided) | choice | Chance you accept of claiming a difference that is not there. |
| Power | choice | Chance of detecting a real difference of the size you set. |
| Better is | choice | Whether a lower or a higher result is an improvement. |
| Before the change | ||
| Mean | Mean of the parts made before the change. | |
| Standard deviation | Standard deviation before the change. | |
| Sample size | items | Parts measured before the change. |
| After the change | ||
| Mean | Mean of the parts made after the change. | |
| Standard deviation | Standard deviation after the change. | |
| Sample size | items | Parts measured after the change. |
What each input means
Open an input to see what it is, its units, the worked example and the common mistakes, with a small diagram.
Plan
What this means: Expected standard deviation
- What it is
- The spread you expect in the measured result from part to part, as a standard deviation from process history, used to plan the sample size.
- Units and format
- Enter a plain number with no unit. The form accepts 0.000001 or more. Required: the tool does not calculate until every input is valid.
- Example
- The form opens with 2.2 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
- Effect on the result
- The sample size needed rises with the square of the standard deviation.
- Get it right
- Use the same units as the measurements and the difference worth detecting.
What this means: Smallest difference worth detecting
- What it is
- The smallest change in the process mean that would matter to you in practice, in the same units as the measurements.
- Units and format
- Enter a plain number with no unit. The form accepts 0.000001 or more. Required: the tool does not calculate until every input is valid.
- Example
- The form opens with 2 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
- Effect on the result
- Halving it multiplies the sample size needed by four.
- Get it right
- Set it from what matters to the product or cost, not from what the trial happens to show.
What this means: Significance level (two-sided)
- What it is
- The chance you accept of declaring a difference when the change really made none (a false alarm), tested in both directions.
- Units and format
- Choose one of 3 options from the list: 0.10; 0.05; 0.01.
- Example
- The form opens with “0.05” selected as a worked example. Choose the option that matches your case.
- Get it right
- A smaller level makes a difference harder to prove and needs more parts.
What this means: Power
- What it is
- The chance that the trial detects a real difference of the size you set as worth detecting.
- Units and format
- Choose one of 3 options from the list: 80%; 90%; 95%.
- Example
- The form opens with “80%” selected as a worked example. Choose the option that matches your case.
- Get it right
- Higher power needs more parts per group.
What this means: Better is
- What it is
- Whether a lower result (for example scrap, time or roughness) or a higher result (for example strength or yield) counts as an improvement.
- Units and format
- Choose one of 2 options from the list: Lower values (after below before); Higher values (after above before).
- Example
- The form opens with “Lower values (after below before)” selected as a worked example. Choose the option that matches your case.
- Get it right
- Choose before looking at the results.
Before the change
What this means: Mean
- What it is
- The average of the measurements on parts made before the change, under the old process.
- Units and format
- Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
- Example
- The form opens with 52.3 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
- Get it right
- Use parts made over the same kind of conditions as the trial parts after the change.
What this means: Standard deviation
- What it is
- The standard deviation of the measurements on parts made before the change, the sample value from those parts.
- Units and format
- Enter a plain number with no unit. The form accepts 0.000001 or more. Required: the tool does not calculate until every input is valid.
- Example
- The form opens with 2.1 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
- Get it right
- Use the sample standard deviation (n − 1), as a calculator or spreadsheet STDEV gives.
What this means: Sample size
- What it is
- The number of parts measured before the change, each giving one independent result.
- Units and format
- Enter a number in items: a count of items. The form accepts 2 items to 10000000 items. Use a whole number. Required: the tool does not calculate until every input is valid.
- Example
- The form opens with 15 items as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
- Get it right
- Repeated readings of the same part do not count as extra parts.
After the change
What this means: Mean
- What it is
- The average of the measurements on parts made after the change, under the new process.
- Units and format
- Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
- Example
- The form opens with 50.1 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
What this means: Standard deviation
- What it is
- The standard deviation of the measurements on parts made after the change, the sample value from those parts.
- Units and format
- Enter a plain number with no unit. The form accepts 0.000001 or more. Required: the tool does not calculate until every input is valid.
- Example
- The form opens with 2.4 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
- Get it right
- Use the sample standard deviation (n − 1), as a calculator or spreadsheet STDEV gives.
What this means: Sample size
- What it is
- The number of parts measured after the change, each giving one independent result.
- Units and format
- Enter a number in items: a count of items. The form accepts 2 items to 10000000 items. Use a whole number. Required: the tool does not calculate until every input is valid.
- Example
- The form opens with 12 items as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
- Get it right
- Repeated readings of the same part do not count as extra parts.
Method and formulas
- n per group = 2 [(z_α/2 + z_β) σ ÷ δ]², rounded up — the same as the sample size for a mean with margin δ ÷ [√2 (1 + z_β ÷ z_α/2)].
- Welch t = (x̄₁ − x̄₂) ÷ √(s₁²/n₁ + s₂²/n₂) with Welch–Satterthwaite degrees of freedom; two-sided p-value and confidence interval (as the two-sample t-test tool).
- Improvement confirmed if p < α and the difference is in the better direction.
- Example: plan 19 parts per group; the trial shows a difference of 2.2 with p = 0.020303.
Limitations
- Independent, roughly normal measurements; trends over time within a group invalidate the test.
- A significant result shows a difference in means only, not a change in spread.
- Example values are illustrations. Replace them with values for your actual task.
Typical uses
- Plan a trial of a new tool, material or setting
- Decide whether a change really improved the process
- Avoid acting on chance differences
Who it is for
- Quality engineers
- Process engineers
- Continuous improvement leads
What you get
- Sample size per group for the trial
- t statistic, p-value and confidence interval
- Decision and checks
Interpreting the result
- A difference can be statistically significant yet too small to matter: compare it with the difference you set as worthwhile.
References
- Machinery’s Handbook, 29th edition (Industrial Press, 2012), printed pp. 130–133. Topic reference; no source tables or recommended settings are reproduced.
Learn the Concepts Behind This Tool
These BIZOMAT Library readings explain the method this tool uses, with worked examples.
Method explained here · BIZOMAT original book
Proving a bonding process
In Adhesive Bonding by the Numbers
What it covers: Planning and running a fair trial of a process change under production conditions, setting up a bonding bay and its records, and counting the cost of bonding against what it replaced.
Method explained here · BIZOMAT original book
After the blender: bags, bins and discharge order
In Blending Bulk Solids by the Numbers
What it covers: How filling a container builds a segregated heap and discharging it delivers the blend in a changing sequence, how to measure that sequence, and how a packaging change was proved with a planned trial.
Method explained here · BIZOMAT original book
From map to cause and proof
What it covers: Turning map evidence into changes, planning a before-and-after trial, proving the defect rate fell, using a fresh map to credit each change correctly, and keeping the gain.
Method explained here · BIZOMAT original book
Measured performance: sample sizes for means and comparisons
In Prototypes and Simulation by the Numbers
What it covers: A 95% interval for the BPU-5's usable energy from 12 DVT units, sample sizes for a mean and for a supplier comparison, and a 3-pack test that nearly cost A$85,000 a year.
Method explained here · BIZOMAT original book
Substitution, reduction and proof
In Solvent Vapours by the Numbers
What it covers: Choosing a less volatile substitute that still cleans and spares plastic parts, measuring use per part, proving a reduction with a fair trial, and the solvent balance before and after.
BIZOMAT tools are screening and decision-support utilities. Verify important financial, engineering, safety, legal or operational decisions using appropriate professional standards and your organisation’s approved data.
Open the Process Change Trial (Sample Size, Welch t-Test, Improvement Decision) calculator