Friday, November 16, 2012
Question: What constitutes evidence?
Judge: Mr. Hutz. Do you have any evidence?
Lionel Hutz: I have conjecture and hearsay. Those are kinds of evidence.
When I say evidence, I mean that reputable studies have been done - and, we hope, replicated - about that ingredient. For instance, I can say that sunflower oil has 61% to 73% linoleic acid, and that linoleic acid has been shown to help restore the barrier function and reduce scaling on your skin. Studies have been done on linoleic acid and these benefits have been demonstrated in other studies.
What constitutes a reputable study? A full accounting of the study with the names and locations of the researchers published in a peer reviewed journal. It is hoped it would be relatively current, but some things are so well established that no one is bothering to do studies any more. If the study isn't peer reviewed, I get a little wary about it. That's why I say I don't consider the press releases put out by EWG, Skin Deep, and the Suzuki Foundation to be evidence. They might be conducted by scientists - I'm not really sure as there isn't much information on the authors of these releases - and they might look like a study, but key bits of information aren't readily available, they aren't published in peer reviewed journals, there's no information on how the studies were conducted, and so on.
click for Project Steve), and I'm sure I could find some who think the world is flat, giant lizards rule us all in secret, and the apocalypse is set for December 21st this year. (As a note, that's my birthday, so if you really believe the apocalypse is coming, then sell all your stuff and send me a great present!) The point isn't about the scientist - it's about the scientific method. This is why I trust peer reviewed material. It isn't about one scientist, it's about a group of them. And the hope is that when you get a group of people together to review material, someone somewhere will ask enough questions to ensure the study is valid.
I really encourage you to read Dr Ben Goldacre's Bad Science. He goes into great detail how we can be bamboozled by studies that aren't really studies, and it's an education and a half! Check out his blog here.
When I say the plural of anecdote isn't data, I'm not putting down your personal experiences. I'm saying that your experiences don't constitute data. I love to hear about them, but your experience with an ingredient doesn't mean it works for everyone. For every opinion you can provide showing that using catnip worked as a conditioner for someone, I can provide you with the opposing opinion. We can't call using an ingredient and liking it evidence. It's your opinion and it is valid and wonderful and can be shared, but it call it evidence simply isn't right.
Someone said that I should run studies myself on things. For instance, when it came to washing one's hair with baking soda, someone suggested that the way I could gather evidence would be to try it myself. If I did that for a few days or a week or a month, it still wouldn't offer more than my opinion on how baking soda works in my hair. And I certainly couldn't extrapolate that to mean that it would or wouldn't work for your hair. I can't control for things I should control for in a study - things like the hardness of my water, the length of time I would wash my hair, my hair type, and so on - because I only have my hair and my house for the testing.
I hear a lot of "my experiences constitute data" arguments when it comes to making our own suncreen (please don't!), I've heard the argument that if we go into the sun and we don't burn, the product must work. Here's the problem - the information we gather still isn't evidence. Always consider the confounding factors! If I use my own sunscreen today and I don't burn, it could mean that I've made an awesome product that is a proper sunblocker, but it could also mean that I wasn't in the sun much today, that I put more than normal on, that I stopped taking those medications or using that product that made me sun sensitive, or a million other things that have nothing to do with the suncreen. It doesn't necessarily mean the product worked. And it doesn't mean that the product will work for me tomorrow or that it will work for you with a different skin type living in a different part of the world.
If we consider cosmetic chemistry to be art plus science equals awesome, we need a balance between the two. We need to make creative choices based on our philosophy about ingredients, our preferences for skin feel, our budgets, and everything else we do to make our products unique! (I love the fact that we can all follow the same recipe but have different outcomes, even with the same ingredients!) But we can't do all of that without the evidence, the science part.
Our subjective experiences are vital when we are making products, but they aren't evidence. I really can't say this enough - our opinions are important and I love hearing what you think about what you're making or different ingredients, but saying that something worked or didn't work for you isn't evidence.
If you think I'm wrong and want to discuss it, click here for some ideas on how to share those thoughts with me! I'm always eager to hear more on a topic, and a good debate is always great fun!
Why we can't make claims about our products.