Studies/Research....How Much Do You Trust Them?

[dochielomn]

Asking this question to the crowd, if you read an article that says "we conducted a study and concluded......" or "research shows us....." how often do you say "that sounds correct, I trust thist" or "well, statistics can be manipulated to prove whatever the author is trying to prove"? I ask this because my wife and I got into an interesting discussion over this with me falling more towards the latter. Just curious if this is because my brain is more wired towards precision and being able to physically prove stuff from start to finish without too many outside or other variables needing to be factored in (such as with medical research where maybe someone says "well, you ate this food so you increased your odds for cancer" but maybe didn't factor in that this person has had like 100 x-rays or CT scans and such and that's why the person ended up with cancer).

[kevm14]

What you are getting at I tend to sum up with the phrase "correlation does not imply causation." Any scientific person ought to think about this next time data is interpreted.

[kevm14]

Here you go:
http://news.bbc.co.uk/2/hi/uk_news/magazine/7592579.stm

Here are a few examples, some new, some old, and all true except for the explanation. Work out the more likely chain of causation then think of some examples of your own.
- Whenever ice cream sales rise, so do shark attacks (eating ice cream makes you tastier?)
- As more economists are recruited to the Treasury, inflation rises (economists cause inflation?)
- In Scandinavia, storks appear more often on the rooftops of families with more babies (storks bring babies?)
- As vocabulary increases in infancy, so does appetite (words make you hungry?)

I picked up the shark and ice cream one from a video I just posted on another thread.

Bonus link: https://simple.wikipedia.org/wiki/Correlation

[dochielomn]

Well, the ice cream shark analogy is a tad extreme and not entirely accurate in terms of saying that a person tastes better if they eat icecream (then again, can't say I've ever been tempted to eat a person), but during my discussion with my wife, she mentioned "do seatbelt save lives" and when I think about it, yeah, I'd say that seatbelts probably save more lives but how can you really prove if wearing a seatbelt did save the person's life? So, by my own thinking, I'd have to say that perhaps seatbelts don't actually save lives because there's other variables that need to be accounted for first. I think ultimately, I have to think of things more along the lines of did the risk increase or not as oppose to definitive statements such as "if you do this, then that will happen". For example, can I say that smoking cigarettes will shorten someone's life span? If I think of it more in terms of "if you smoke cigarettes are you increasing the risk that something bad will happen to you (in this case, lung disease/failure)" then I can accept a study/research a tad more.

[kevm14]

Well, the ice cream shark analogy is a tad extreme and not entirely accurate in terms of saying that a person tastes better if they eat icecream

That went right over your head. Read it again. The parentheticals are facetious - of course it's not because ice cream makes you tastier. It's because warmer weather brings more swimmers, which means more shark attacks. Those same people also consume more ice cream. Perfectly correlatable yet ZERO causality.

[kevm14]

but during my discussion with my wife, she mentioned "do seatbelt save lives" and when I think about it, yeah, I'd say that seatbelts probably save more lives but how can you really prove if wearing a seatbelt did save the person's life? So, by my own thinking, I'd have to say that perhaps seatbelts don't actually save lives because there's other variables that need to be accounted for first. I think ultimately, I have to think of things more along the lines of did the risk increase or not as oppose to definitive statements such as "if you do this, then that will happen". For example, can I say that smoking cigarettes will shorten someone's life span? If I think of it more in terms of "if you smoke cigarettes are you increasing the risk that something bad will happen to you (in this case, lung disease/failure)" then I can accept a study/research a tad more.

Right. It is a statistics thing. There will always be freak cases where an airbag kills someone or someone who wasn't wearing a belt just happens to be miraculously ejected at the most optimum time and angle to save their life. But those are not the norm. Likewise, you are not guaranteed to get cancer if you smoke. But your risk goes up and we have largely quantified that kind of stuff. That is actuarial science, right? The basis for how insurance companies compute risk pools and premiums.

[dochielomn]

Well, the ice cream shark analogy is a tad extreme and not entirely accurate in terms of saying that a person tastes better if they eat icecream

That went right over your head. Read it again. The parentheticals are facetious - of course it's not because ice cream makes you tastier. It's because warmer weather brings more swimmers, which means more shark attacks. Those same people also consume more ice cream. Perfectly correlatable yet ZERO causality.

Eh, i'd still argue that there are far more factors to consider than just more swimmers = more attacks. But it goes back to my general problem with definitive statements. Think i'd have less of an issue with saying "warmer weather brings out more swimmers which can lead to increase shark attacks due to population in the water". But making a statement that if there are more swimmers, there will be more shark attacks isn't entirely accurate. Sort of like this- https://www.youtube.com/watch?v=fm2W0sq9ddU

[dochielomn]

but during my discussion with my wife, she mentioned "do seatbelt save lives" and when I think about it, yeah, I'd say that seatbelts probably save more lives but how can you really prove if wearing a seatbelt did save the person's life? So, by my own thinking, I'd have to say that perhaps seatbelts don't actually save lives because there's other variables that need to be accounted for first. I think ultimately, I have to think of things more along the lines of did the risk increase or not as oppose to definitive statements such as "if you do this, then that will happen". For example, can I say that smoking cigarettes will shorten someone's life span? If I think of it more in terms of "if you smoke cigarettes are you increasing the risk that something bad will happen to you (in this case, lung disease/failure)" then I can accept a study/research a tad more.

Right. It is a statistics thing. There will always be freak cases where an airbag kills someone or someone who wasn't wearing a belt just happens to be miraculously ejected at the most optimum time and angle to save their life. But those are not the norm. Likewise, you are not guaranteed to get cancer if you smoke. But your risk goes up and we have largely quantified that kind of stuff. That is actuarial science, right? The basis for how insurance companies compute risk pools and premiums.

Right, but to the core of my question, as an actuary, can I manipulate stats to prove what I want?

[kevm14]

Ok but you must agree that if you don't go in the ocean, you cannot be attacked by a shark. Let me rephrase that - if you do not go swimming where sharks are, you cannot be attacked by a shark (i.e. there is no land shark except in commercials). Therefore you have some quantifiable increased risk of shark attack by swimming in the ocean. It may be small, but it is bigger than if you didn't go into the ocean. This is NOT the same as saying "if you do not smoke, you will not get cancer." It is also not to say you should therefore not go swimming in the ocean. These are different things.

But you are missing the point. The point is not that swimmers are both eating ice cream while being attacked by a shark. The data shows that increased ice cream sales also correlate with shark attacks. When one goes up, the other is up. When one goes down, the other goes down. In general, on average (i.e. ice cream sales and shark attacks in Ohio probably have no correlation at all because there is no ocean). The point is that the two are completely non-causal, yet can be shown graphically to be related.

What are we arguing about again?

[kevm14]

Right, but to the core of my question, as an actuary, can I manipulate stats to prove what I want?

My answer is both yes and no.

The yes is true when the data is casually presented and the consumer is sort of forced to take the conclusion at face value from the presenter.

The no is true when the consumer has access to all data sources and the data itself that was used to create the presentation. It should be evident through investigation whether there was any foul play, but this investigation is tedious, time consuming and may even require subject matter expertise. Which is why the answer is probably "yes" a lot more often than "no," sadly.

So the trick is to identify when you are about to be manipulated and to critically ask for more information. If the response is "trust us" then you should really be skeptical.

By the way, this brings up another point. If an energy company tells me to relax and buy some oil, that they looked into it and it's just fine, should I be skeptical because of who is telling me? Of course I should. But science is supposed to be unbiased and an examination of data should reveal the true answer. Remember, everyone has an agenda.

And because everything is political:
http://historyhalf.com/liberal-morality ... the-means/

 “Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive.”  CS Lewis

In American politics, liberals often accuse conservatives of being not just wrong, but downright evil; and the accusation is true. It’s true, that is, if you accept the moral code that liberals use when they judge other people’s behavior.

There is no universally accepted code of Right and Wrong. An action can only be described as morally right or wrong when it is judged according to a set of moral standards, and we do not all use the same standards.

Of Ends and Means
One area where liberal morals differ from more traditional morals is on the question of whether “the end justifies the means.” Leftists tend to believe that any course of action is virtuous if the intention is to achieve some important end like banning guns, or promoting gay rights, or helping Democrats win political elections.

What this means is it is kind of a liberal thing to believe that the ends justify the means. What does this have to do with our discussion? Because they believe, in general, that even if you make up some facts or manipulate statistics, if you get that end result which is deemed (by their definition) to be morally correct, whatever happened in the middle is of no real consequence. I have a problem with this fundamentally and I think THAT is why all this AGW nonsense is so politicized (sorry to bring this over here but it is relevant). Example straw man I just made up:

Person A: "Humans are destroying the planet, we need to do something."
Person B: "Prove that."
Person A: "See this graph? Mean global temperature has been steady for 100 years and is now exponentially going to shoot up if we don't do something."
Person B: "Yeah, that actually was pseudo-science or even fraud, and that graph is not true."
Person A: "That doesn't matter - I KNOW humans are destroying the planet so why do I need to produce any facts at all? We just need to take action. My cause is noble and therefore facts don't matter."

Ok that was a bit of a tangent but again it is related to the other thread. This kind of stuff bothers me.