The ABC Study Guide, University education in plain English alphabetically indexed. Click here to go to the main index
Figure Home Page Research Science

ABC of Experiments

Words to help you think about Experiments and Statistical Experiments

Click coloured words to go where you want


Isaac Newton stressed the importance for science of combining mathematical theories ( models) of reality with experiments. He argued for an "experimental philosophy" of science. Science should not, as Descartes argued, be based on fundamental principles discovered by reason, but based on fundamental axioms shown to be true by experiments.

" although the arguing from experiments and observations by induction be no demonstration of general conclusions, yet it is the best way of arguing which the nature of things admits of." Newton, I. 1704 quoted Losee, J. 1972 p.81

In general terms, we use the word experiment in two ways. We can mean trying out new things, or testing by experience. A man wearing lipstick might be experimenting in both ways. He could be trying out something new for him. He might also be testing public reaction. His activities might be described as a social experiment, but they are not what we normally consider a scientific experiment.

A scientific experiment is a procedure constructed to discover, demonstrate or test significant truth about reality, under controlled conditions, in a way that can be repeated by other people with similar results. Like the man wearing lipstick, constructing the experiment requires imagination. Like the man wearing lipstick, experience is providing something that imagination on its own could not.

With a scientific experiment, however, the "truth" that we are discovering, demonstrating or testing will have to be carefully formulated, the conditions of the experiment will be carefully specified and the rational connection between the truth discovered, demonstrated or tested, and the experimental results, will be logically laid out. Newton formulated his truths mathematically and most scientific experiments are quantitative.

By saying that the truth should be significant, I mean that it should relate to a general body of theory that can be seen to rationally and usefully explain the world in a way acceptable to scientists. It may relate by just showing that an accepted scientific theory does not adequately explain reality - but it must relate in some way to scientific theory.

Statistical Experiments

experiments are common in many sciences, including biology, psychology, medicine and ecology. In Simple Statistics, Frances Clegg gives this "Summary of experimental procedure".

  1. Have an idea ( theory) about the effect of one variable upon another.

  2. Define the independent variables and the dependent variables

  3. Decide how the variables will be quantified. (What the units of measurement are)

  4. Express the idea formally as an experimental hypothesis .

  5. Decide what kind of statistical analysis will be appropriate.

  6. Specify a significance level and sample size.

  7. Select the sample to be used from the parent population which is under scrutiny.

  8. Divide the sample into two

  9. Apply the experimental treatment to one part of the sample, and treat the other as a control group.

  10. Collect the results. These will be two sets of scores, one for the experimental group and one for the control group, showing how the dependent variable altered as the independent variable was altered.

  11. Analyse the data

    • Establish the null hypothesis
    • Apply an appropriate statistical test or technique
    • Accept or reject the null hypothesis in the light of the last step

    • Draw a conclusion about whether the experimental hypothesis has been confirmed or not.

Randomised Controlled Trial

Definitions from the the National Institute for Health and Care Excellence (NICE) glossary:

Randomisation (random allocation) means assigning participants in a research study to different groups without taking any similarities or differences between them into account. For example, it could involve using a random numbers table or a computer-generated random sequence. It means that each individual (or each group in the case of cluster randomisation) has the same chance of receiving each intervention. A Randomised controlled trial is a study in which a number of similar people are randomly assigned to two (or more) groups to test a specific drug or treatment. One group (the experimental group) receives the treatment being tested, the other (the comparison or control group) receives an alternative treatment, a dummy treatment (placebo) or no treatment at all. The groups are followed up to see how effective the experimental treatment was. Outcomes are measured at specific times and any difference in response between the groups is assessed statistically. This method is also used to reduce bias.

Randomised controlled trial example

INTERACT: INvestigation of TExt message Reminders on Adherence to Cardiac Treatment - 1.4.2012 to 1.9.2014

Lowering blood pressure and cholesterol can reduce the risk of heart attacks and strokes, but about half of all patients prescribed medication discontinue it after about two years, leading to many thousands of avoidable deaths. Could some of these be avoided if people were sent text messages to remind them to take their medicine? To test this, anyone with a mobile phone prescribed cholesterol-lowering or blood pressure lowering treatment for the prevention of cardiovascular disease was invited to join a randomised controlled trial starting in April 2012. The aim was to recruit 300 participants who would be divided into a text-message group of 150 and a control group of 150.

Participants were randomly allocated to receiving a programme of text messages or to be part of a control group that did not. Text reminders were sent, first daily and then weekly, for a year to the first group. People who received texts were to text back to say if they had already taken the medication that day, had been reminded to take it, or had not yet taken it. People who had not yet taken it were telephoned later to see how they were getting on. "The intervention aims to remind, train and identify individuals who are not-adhering, so action can be taken to correct problems as they arise".

Both groups were assessed after six months ad after eighteen months. A questionnaire assessed adherence to the medication and measurements of blood pressure and blood cholesterol were taken.

The trial is expected to be completed by 1.9.2014, but results of a six- month pilot study with 64 participants have been published. The results were that

  • One patient out of 32 in the text message group discontinued their medication

  • Five out of 29 in the control group discontinued their medication and 3 were "lost to follow-up".

    Among those who continued their medication, the reported number of days medications were missed in the four weeks prior to follow-up were 0.7 in the text message group and 4.1 in the control arm; a difference of 3.4 days (1.7 to 5.1)

    Cholesterol and blood pressure were lower in the text message group.

    The report concluded that the pilot study suggested that text massaging "would be expected to reduce the risk of ischaemic heart disease by 48% and stroke by 47%" and that "These results now need to be confirmed in a larger trial."

    Sources include the current control trials register and an abstract on the pilot study

    Experimental Hypothesis

    A hypothesis is an idea or theory that predicts what might happen. An experimental hypothesis is a prediction, made to be tested, that one thing (
    variable) will affect another.

    The hypothesis will suggest that when one of two variables alters, the other will as well.

    Independent and Dependent Variables

    The first variable (the one we alter to affect the other) is called the independent variable. The variable we predict will be altered is called the dependent variable.

    Example "Drug x is good for making people with colds better" predicts that the independent variable "Drug x" will tend to make the dependent variable, "people with colds", better.

    Non-directional hypotheses A non-directional hypothesis does not predict which way the independent variable will affect the dependent variable.

    Example "Drug x will have an affect on people's colds"

    This does not predict if it will make the colds better or worse.

    Directional hypotheses A directional hypothesis predicts the way the independent variable will affect the dependent variable.


    "Drug x will make people's colds worse"
    "Drug x will make people's colds better"

    The slang term for a non-directional hypothesis is a two-tailed hypothesis. A directional hypothesis is called a one-tailed hypothesis. As with tossing a coin that has one head and one tail, only one out of two predictions can be correct. But the coin enalogy is misleading, because in the experiment there is the third possibility that the independent variable may have no effect. The drug may not influence the colds in any way. This possibility is called the null hypothesis

    Statistical Tests
    Tests of Significance

    Statistical tests are used to see if
    samples of numbers appear to have come from one or from two populations. The statistical test will also say what the likely margin of error is.

    Another way of saying this, is that the statistical test tests whether a difference observed between two samples is likely to reflect a real difference between two populations.

      For example:

      One sample of twenty people with colds was given drug x. A control sample of twenty people with colds was not given any treatment. 10 of the first sample had stopped sneezing two hours after taking drug x. At the same time, 8 of the control sample were found to have stopped sneezing.

      It would appear that drug x works. But is the difference between samples likely to reflect a real difference between the (hypothetical) population of all people with colds who might take drug x, and the (hypothetical) population of the (same) people with colds, but without taking drug x?

    Null Hypothesis

    Null Hypothesis is a hypothesis that, (despite the difference between the samples), there is no difference between the populations. This would mean that any difference we have observed between the samples is due to sampling error.

    The Null Hypothesis is contrasted with the

    Alternative Hypothesis

    The Alternative Hypothesis is the hypothesis we will accept if the Null Hypothesis is rejected.

    Tests of significance set up a model that is based on the assumption that there is no difference between the
    populations. This "null hypothesis" is only rejected if it is very difficult to fit the data to such a model. If this proves to be the case, the Alternative Hypothesis is accepted.

    Conclusions about the Experimental Hypothesis

    If it is found, by a statistical test of significance, that a difference between two samples reflects a real difference between two
    population, this is said in a way that shows how reliable the conclusion is.

    For Example:

    "The results of the statistical analysis were significant at the p is less than or equal to 0.05 level".

    This means that there is a 5% chance (p = probability) that, despite the difference observed between the two samples, there is no difference between the populations, and a 95% chance that the difference observed between the two samples reflects a real difference.

    Study Link
    Andrew Roberts' web Study Guide
    Top of Page Take a Break - Read a Poem
    Click coloured words to go where you want

    Andrew Roberts likes to hear from users:
    To contact him, please use the Communication Form

    © Andrew Roberts



    Social Experiments

    Laboratory Experiments

    Statistical Experiments

    Randomised Controlled Trial

    Experimental Hypothesis

    Statistical Test

    Null Hypothesis

    Alternative Hypothesis

    Conclusions about the Experimental Hypothesis