Conducting a hypothesis test is by nature an uncertain business. Just be cause you do not reject the null hypothesis, it does not mean that the null hypothesis is true. There is in fact insufficient evidence from the hypothesis test to mean that you can reject the null hypothesis. If for example, you conducted a 10% test, then even if the null hypothesis we true, 10% of the time using this test, you would conclude that the null hypothesis we false.

Conversely, if the null hypothesis is false, there is a non zero probability of accepting the null hypothesis when the null hypothesis is false.

A Type I Error is made if the null hypothesisis rejected when it is true.

The probability of making a Type I Error is equal to the significance level.

A Type II Error is made if the null hypothesisis accepted when it is false.

The power of a statistical test is the probability that the test will reject the null hypothesis when the null hypothesis is false (i.e. that it will not make a Type II error). More powerful tests are more useful, but working out the power of a test can be complicated.

We must minimise the probability of making a type I error and we can do this by reducing the significance level of the test since this is the probability of making a Type I Error, but then the probability of making a Type II Error is increased. In fact we cannot simultaneously reduce both the probability of a Type I Error and a Type II Error, and a balance between these two evils is always struck. A lot depends on which is the greater evil – on the consequences of falsely rejecting a null hypothesis, or falsely accepting a null hypothesis.

It may be useful to think of the null hypothesis as something like: The accused is always innocent. Then the alternative hypothesis as being: The accused is guilty. The prosecution must always prove the null hypothesis - guilt of the accused, but in fact the alternative hypothesis is never proved in a statistical hypothesis test. The probability that comes out of a hypothesis test, if the probability method is used, is the probability that the null hypothesis is true, so that the null hypothesis is only rejected if this probability is below the significance level of the test.