Top 10 Ways to Spot Bad Science

By Becca Smithers

Bad science is where people misuse science and scientific research, usually to get you to buy something, to change your mind, or to make something seem better than it actually is. People may deliberately misinterpret results of experiments to get the conclusions they want, or research may not go through the proper scientific peer review process where it is assessed by other scientists. It can be tricky to know who or what to believe!

This top 10 guide will help you to filter out the Bad Science (or the BS!) from the real science that you can trust.

CC-BY Michael J. Ermarth

CC-BY Michael J. Ermarth

1) Unrealistic claims

Face creams that make you look ten years younger; diets that make you lose 7lbs in 7 days, miracle cure pills – do these sound familiar? If you hear something like this and it doesn’t sound realistic or you don’t see how it could work, then question it. Usually these unrealistic “scientific” claims are there to grab your interest, but they are not usually true or don’t last very long.

If you want to check if a claim has any evidence towards it, you can look it up on Sense About Science’s Ask For Evidence page. You can look up claims that have already been looked into, or you can submit a claim you have seen and Sense About Science will look into it for you. They have confronted many companies about their use of bad science and showing just how important evidence is!

 

2) Small sample sizes

On the bottom of your television screen during an advert that refers to a statistic such as “7 out of 10 people said…”, you will see the sample size of the number of people included in the study. If the sample size is small and they are claiming something significant, then this is bad science. The larger the sample size means that you have more data to analyse and can get more significant results.

If a study interviewed 20 people asking them if they prefer tea or coffee and 18 people said tea, then you could conclude that 90% of people prefer tea to coffee. If 200 people were interviewed and 180 people preferred tea, then you can more confidently say that 90% of people prefer tea because you have more data to support you. Also, where did the data come from? Do the 20 people who were surveyed work in a tea room? If so it’s unsurprising that 18 of them prefer tea.

3) Correlation and causation confusion!

Correlation is when there is a pattern between two sets of data, and causation is where one thing causes another. Correlation and causation confusion is where people find correlations between two things and assume that one has caused the other. This happens a lot in bad science to make a point by misusing data. Spurious Correlations has some good examples of correlation causation confusion. Our favourite is the number of non-commercial space launches between 1997 and 2009 positively correlating with the number of sociology doctorates awarded in the US. This doesn’t mean that sociologists get a free ride into space with their PhD, it is just a coincidence!

CC-BY xkcd.com

CC-BY xkcd.com

4) Use of over-complicated (or even made up!) scientific language

This is very common in cosmetics adverts, particularly anti-ageing creams. You may hear words like “cellular rejuvenation”, “polypeptide” and “micro-exfoliating”, and no explanation as to what they mean! Some of the terms you see are actually proper scientific words, but are being used in the wrong context. For example, polypeptides (proteins) are often referred to in face creams but these proteins are too big to get into your skin cells so just sit on top of your skin, not doing very much! These words are used to make the products sound more technical and tested, and to make you trust them more, when really they are just using bad science.

5) Where does the research come from?

If research is being funded by a company or organisation then beware. This doesn’t automatically mean that it is bad science, but there could be a vested interest behind the study where the company hopes to gain something. This is where results could be leaned towards benefiting the company or organisation, or there could be a correlation causation confusion.

6) Non-committal language

If there is a claim that a product “may” do something, or can do something “up to” a certain level then the conclusions of the research are not certain and more tests probably need to be carried out. They will make it sound as positive as they can, but overall any non-committal language indicates that you can’t really trust the product to do what is being advertised as it probably won’t do it at all.

CC-BY xkcd.com

CC-BY xkcd.com

7) “Scientifically proven”

This is the thing about science: it doesn’t prove things. It gains evidence towards a hypothesis (an idea or a claim), this evidence is reviewed by other scientists to check that it is and if there is enough evidence towards a hypothesis it can become a theory. Gravity is a theory, any experiments testing gravity just gain evidence in favour of the theory but they do not prove it. If you hear someone claiming that something such as a skin care product has been scientifically proven to make you look 10 years younger, that is bad science.

There are lots of things that happen around us that have so much evidence that we do quite sensibly believe them to be proved, such as a sperm fertilising an egg makes a baby, and that if we jump up in the air gravity will always pull us back down again. Of course these things are “proven” because we haven’t seen anything to the contrary, but that doesn’t mean that we won’t find evidence against it. Science always gathers more evidence towards theories just to keep checking. Don’t go away and start doubting everything you know because it’s not proven, but just beware of bogus claims of products and services being “scientifically proven”. No scientist will claim that they have proved something, they will show evidence supporting a hypothesis or theory.

8) No control group

This one will take a bit of research on your part. If you want to explore some bad science, try and find the study being referred to. If it is difficult to find the study then there could be a reason why it is being hidden – it could be bad science! If you find the study, check to see if there is a control group. A control group helps to check the science. In a study of a drug, you would give one group of people the drug being tested, and another group of people a placebo (a fake pill), but the people won’t know which group they are in. You then test the effectiveness of the drug and compare that to results of the control group that show what happens when you don’t use the pill. All studies should have a control group show what happens when you don’t use what is being tested. If there is no control group then you can’t measure the effect of what is being tested. No control group = bad science.

9) Non-replicable results

CC-BY xkcd.com

CC-BY xkcd.com

Again, this requires a bit of research on your part. If you have managed to find the original study, look at what other papers and research is refers to and see if you can find these journals as well. A really important part of science is to make sure that the results of your experiment can be replicated to show that it wasn’t just a one-off set of data. Good science will refer to other researchers and their work in relation to the study being carried out. If the research doesn’t seem as if it can be replicated to give the same results, and there is little support from other scientists, then it’s likely you may have found some bad science!

10) Can you find the research it refers to?

If an advert or a campaign refers to some research or claims some statistics, they must have got them from somewhere. Sometimes these studies are published on websites or they can be found through a search engine, or even by you asking for them. But a lot of the time if these studies are bad science, they will be hidden so the public cannot read them, possibly because they don’t actually exist! Good science will openly have references and citations for you to be able to check their work, bad science will hide their research because they know it is flawed.

It is particularly important to be able to find research and results of clinical trials of drugs and healthcare initiatives so that everyone can really see the positives and negatives of new treatments. Unfortunately these are not always made available, and there are campaigns such as AllTrials, which aims to make sure all clinical trial results are registered and reported. This will help to stop bad science!


The number one rule to find bad science: question everything!


 

For more information about bad science, check out Dr. Ben Goldacre’s website, where he exposes the misuse of science. He has also written books called ‘Bad Science’, ‘Bad Pharma’ (focussing on medicine and pharmaceutical clinical trials), and ‘I think you’ll find it’s a bit more complicated than that’ (focussing on misuse of science by politicians and journalists).

 


Free water worksheets


 Sign up for our newsletter

Sign up to our newsletter for all the news on where we are performing, our new shows, the best of our blog, fun activities and our latest special offers.

We are giving away a series of 7 free water worksheets for KS2 and KS3 or simply to have a go at with the kids in your life, when you sign up to our newsletter.

These worksheets contain fun demonstrations, facts and tips on topics such as pressure, surface tension and water saving. Just pop your details in below and the sheets will be sent straight to your inbox.

 

_____________________________________________________________________________

sms group shot with props smallWe are science made simple, a social enterprise who perform science, maths and engineering shows to school, festival and public audiences.
You can find out more about what we do, check out our exciting range of shows, or sign up to our newsletter to keep updated on what we are up to!
Tagged with:
Posted in Exploring Science, Science Communication