Friday, November 16, 2012

Question: What constitutes evidence?

I like to think I run an evidence based blog here, and recently the idea of what constitutes evidence has been questioned. Let's take a look at what I mean.

Judge: Mr. Hutz. Do you have any evidence?
Lionel Hutz: I have conjecture and hearsay. Those are kinds of evidence. 

When I say evidence, I mean that reputable studies have been done - and, we hope, replicated - about that ingredient. For instance, I can say that sunflower oil has 61% to 73% linoleic acid, and that linoleic acid has been shown to help restore the barrier function and reduce scaling on your skin. Studies have been done on linoleic acid and these benefits have been demonstrated in other studies.

What constitutes a reputable study? A full accounting of the study with the names and locations of the researchers published in a peer reviewed journal. It is hoped it would be relatively current, but some things are so well established that no one is bothering to do studies any more. If the study isn't peer reviewed, I get a little wary about it. That's why I say I don't consider the press releases put out by EWG, Skin Deep, and the Suzuki Foundation to be evidence. They might be conducted by scientists - I'm not really sure as there isn't much information on the authors of these releases - and they might look like a study, but key bits of information aren't readily available, they aren't published in peer reviewed journals, there's no information on how the studies were conducted, and so on.

And no, I don't just trust everything scientists say. (Some people seem to think scientists are involved in some giant conspiracy theory to kill us all!) I can find a scientist to say just about anything for a sound bite. I can find scientists to deny climate change and evolution (click for Project Steve), and I'm sure I could find some who think the world is flat, giant lizards rule us all in secret, and the apocalypse is set for December 21st this year. (As a note, that's my birthday, so if you really believe the apocalypse is coming, then sell all your stuff and send me a great present!) The point isn't about the scientist - it's about the scientific method. This is why I trust peer reviewed material. It isn't about one scientist, it's about a group of them. And the hope is that when you get a group of people together to review material, someone somewhere will ask enough questions to ensure the study is valid.

I really encourage you to read Dr Ben Goldacre's Bad Science. He goes into great detail how we can be bamboozled by studies that aren't really studies, and it's an education and a half! Check out his blog here.  

When I say the plural of anecdote isn't data, I'm not putting down your personal experiences. I'm saying that your experiences don't constitute data. I love to hear about them, but your experience with an ingredient doesn't mean it works for everyone. For every opinion you can provide showing that using catnip worked as a conditioner for someone, I can provide you with the opposing opinion. We can't call using an ingredient and liking it evidence. It's your opinion and it is valid and wonderful and can be shared, but it call it evidence simply isn't right.

Someone said that I should run studies myself on things. For instance, when it came to washing one's hair with baking soda, someone suggested that the way I could gather evidence would be to try it myself. If I did that for a few days or a week or a month, it still wouldn't offer more than my opinion on how baking soda works in my hair. And I certainly couldn't extrapolate that to mean that it would or wouldn't work for your hair. I can't control for things I should control for in a study - things like the hardness of my water, the length of time I would wash my hair, my hair type, and so on - because I only have my hair and my house for the testing.

I hear a lot of "my experiences constitute data" arguments when it comes to making our own suncreen (please don't!), I've heard the argument that if we go into the sun and we don't burn, the product must work. Here's the problem - the information we gather still isn't evidence. Always consider the confounding factors! If I use my own sunscreen today and I don't burn, it could mean that I've made an awesome product that is a proper sunblocker, but it could also mean that I wasn't in the sun much today, that I put more than normal on, that I stopped taking those medications or using that product that made me sun sensitive, or a million other things that have nothing to do with the suncreen. It doesn't necessarily mean the product worked. And it doesn't mean that the product will work for me tomorrow or that it will work for you with a different skin type living in a different part of the world.

It's like the fabled tiger rock. This is my tiger rock. There are no tigers about. Therefore my rock must work to keep away tigers. Very faulty logic! 

If we consider cosmetic chemistry to be art plus science equals awesome, we need a balance between the two. We need to make creative choices based on our philosophy about ingredients, our preferences for skin feel, our budgets, and everything else we do to make our products unique! (I love the fact that we can all follow the same recipe but have different outcomes, even with the same ingredients!) But we can't do all of that without the evidence, the science part.

Our subjective experiences are vital when we are making products, but they aren't evidence. I really can't say this enough - our opinions are important and I love hearing what you think about what you're making or different ingredients, but saying that something worked or didn't work for you isn't evidence.

If you think I'm wrong and want to discuss it, click here for some ideas on how to share those thoughts with me! I'm always eager to hear more on a topic, and a good debate is always great fun!

Related posts:
Why we can't make claims about our products. 


catherine said...

For lotioning my shortcut for evidence based ingredients is, especially its ingredient dictionary. Paula begoun, its founder, wrote the book 'dont go to the cosmetics counter without me.'

I feel like this site does 'research on the research' and is really balanced. For example, it goes over 'natural' ingredients that are bad for skin and 'synthetic' ingredients that are good for skin.

Katie said...

I agree that peer reviewed is key. Without it, no one is checking other people's methods and data analysis, which allows for some pretty crazy ideas running amok over the internet!

Leslie said...

An excellent, excellent post! Thank you. :)

Nancy Liedel said...

Evidence is this and nothing more, a scientific test, that can be repeated, for efficacy using a blind study, with enough test subjects to be relevant and a constant. You cannot repeat it, no evidence. Anecdotal evidence will peek my curiosity, but honestly, it needs a base in science. If it really does work, a strong, well thought out test will show that. End of story. No bias going in, a hypothesis is nice, but the scientist needs an open mind if he finds out X works when he thought it would not, he being relevant to the gender of the tester, or does not. If not, why not? Do not discount anecdotes, but look for proof. Also note, the human mind is wide open to the placebo effect. Nothing wrong with that. If you believe a product is working, you are entitled to that belief, but it's going to work for some, not all. Also, some products do better on other skin types. Cosmetic science is bigger than anyone has heretofore thought and will continue to grow and minute changes in weather, climate and our skin's reaction to it, change.

You need a very open mind to be a good scientist and a good consumer. It's synergistic. Not right v. wrong. Which may make no sense. It's 6:30 am here and I have not had coffee. I am much more understandable on coffee. OK, no one ever gets me, but I it makes sense in this little adhd mind.

SarahF said...

Totally agree with Susan and Nancy.
As an aside, I'm a scientist (2 science degrees) and my birthday is also on 21st December - I'll believe the end of the world if and when I see it.

Susan Barclay-Nichols said...

Hi Sarah! I remember looking at the Book of Lists when I was 11 and thinking that I'd be dead thanks to a nuclear war before 2012 so what was there to worry about? (Yes, I was a barrel of fun as a teenager!) Now it's only a few weeks away - 33 days, if you want to be precise. Wow! (Mind you, thinking the apocalypse is coming because the Mayan calendar ran out is like thinking the world will end on December 31st every year!) Isn't December 21st the best day of the year? :-)

Hi Catherine. I like Paula Begoun, too!

Hi Katie! I know! Things can still slip through, but I think it's one of the best ways of checking that we have!

Hi Leslie! Thanks!

You've made me think about the idea of being balanced. What does it mean? Everyone thinks there are two sides to everything and that other side should be provided to be balanced. But there aren't two sides to everything - it depends upon the context, the people involved, the complexity of the issue, and so on - and I don't think I need to give the "breath of life"* to arguments that have no validity just to seem balanced.

*To quote Stephen J. Gould. Damn, I miss that man!

SarahF said...

Yup, Dec 21st is the best day of the year, but may not be this year!!

Check out wiki - the Mayan calendar finishes its 'Long Count' on Dec 21st and immediately starts a new one, so is this proof that we'll still be around on Dec 22nd?! Will the proof be when we wake up the next morning?!
What indeed constitutes proof?!