DUST Search - Anecdotal accounts are proof of nothing

Buster Friently wrote:

Skihids wrote:

The problem here is that certain people start with an opinion and then provide anecdotes as proof that they are correct in their opinion.

Science doesn't work that way.

You start with an idea, then design an experiment to determine if there is sufficient evidence to warrant that being an accurate model of the world or not.

The closest I've seen the argument come to science are the threads discussing damage per second, though it's tough to model every aspect.

Just to be pedantic. Science, ideally, works slightly differently. You come up with a theory, and then test it for failure. The scientific method never proves anything. It can only disprove things. Over time, as more and more incorrect theories are weeded out, we are left with a functional, if not "true", model of whatever is being studied.

Unfortunately, arguments over DPS leave so much out of the equation when it comes to FPSes, that there's very little data to be separated from the opinions. That's at least a starting point based on facts though.


Author	Thread Statistics \| Show CCP posts - 0 post(s)
Skihids Bullet Cluster 1834	Posted - 2013.08.08 17:46:00 - [1] - Quote I once killed a Madruger with a MD. You instinctively realize that says nothing about the relative merits of the MD vs. the HAV. One anecdote with no accounting for any other factor imparts zero information. For all you and I know the driver had to AFK to attend to a real life emergency. Yet these forums are replete with case after case of "proof" that one or another weapon is OP. The latest is where Xero attempts to prove that the MD is OP by showing us a single score of 32/3. A number of folks instinctively realized there were confounding factors such as running with a full proto-bear squad against randoms, but not being statisticians they could not fully articulate the the complete failure to provide evidence of his claim. CCP has several statisticians on its staff, so you are going to have to do better than that if you want to convince them. At the very least what you need to do is run multiple experiments with the two comparison weapons, the AR and MD in this case. You then measure the characteristics of the two populations of measurements and use a statistical test to determine if there is a significant difference between the two. The two major characteristics of a population is the mean (average) and standard deviation (the measure of how spread out the data is). You can then use a simple T-test to determine if the two populations of measurements are different or not. Now what usually happens is that you find that the variation in measurements (number of kills per match) obscures any difference. In that event you can attempt to remove the confounding factors (number and strength of players in your squad and team, length of match, presence of vehicles, strength and experience of the opposing force, type of match, etc.) to isolate the effect of the weapon. This is done with a randomized block design. I'm not expecting everyone here to be able to design and run a proper experiment. I would be satisfied if more people were simply aware of the requirements for a proper statistical proof that could be used to balance with and refrain from polluting the forums with anecdotal accounts and calling them proof..
Skihids Bullet Cluster 1835	Posted - 2013.08.08 17:55:00 - [2] - Quote Aisha Ctarl wrote: I think.....I think we all just got schooled with arithmetic guys. It's simple statistics and data evaluation, and it is actually as interesting as it is useful. I minored in applied statistics in my undergrad and it comes in handy even though I'm not a statistician by trade. Familiarity with just a few basic statistical concepts can help you evaluate claims such as one weapon being OP in relation to another, and helps you separate fact from opinion.
Skihids Bullet Cluster 1842	Posted - 2013.08.08 18:33:00 - [3] - Quote Skyhound Solbrave wrote: The sad part is that when someone gives a bullet proof logical argument, the other side tends to shut down and not answer it. As a result, it is forgotten and the idiotic argument on Xero's post continues. That's why I made this a separate topic. I didn't want it to be buried in the mud of bad arguments. Interestingly you can see the variation I am speaking of in his first score. He went 13 and was in the middle of the pack of his team. So we have two data points, 13 and 32 kills. That's a spread of 19 points.
Skihids Bullet Cluster 1844	Posted - 2013.08.08 21:06:00 - [4] - Quote The problem here is that certain people start with an opinion and then provide anecdotes as proof that they are correct in their opinion. Science doesn't work that way. You start with an idea, then design an experiment to determine if there is sufficient evidence to warrant that being an accurate model of the world or not. The closest I've seen the argument come to science are the threads discussing damage per second, though it's tough to model every aspect.
Skihids Bullet Cluster 1847	Posted - 2013.08.08 21:39:00 - [5] - Quote Buster Friently wrote: Skihids wrote: The problem here is that certain people start with an opinion and then provide anecdotes as proof that they are correct in their opinion. Science doesn't work that way. You start with an idea, then design an experiment to determine if there is sufficient evidence to warrant that being an accurate model of the world or not. The closest I've seen the argument come to science are the threads discussing damage per second, though it's tough to model every aspect. Just to be pedantic. Science, ideally, works slightly differently. You come up with a theory, and then test it for failure. The scientific method never proves anything. It can only disprove things. Over time, as more and more incorrect theories are weeded out, we are left with a functional, if not "true", model of whatever is being studied. Unfortunately, arguments over DPS leave so much out of the equation when it comes to FPSes, that there's very little data to be separated from the opinions. That's at least a starting point based on facts though. That's true, and real science is even messier than that, but the core notion s that you try to disprove your hypothesis, not cherry pick data that supports your idea. The deal is that individual anecdotes are worth nothing. You need hard data. People instinctively understand the need for comparison when they tell complainers to use the supposed OP weapon for a while to see how it affects their KDR. It's not a statistically rigorous test but it does use repeated measurements to obtain a new average for the weapon. The idea is that with enough tests (matches) the user will encounter enough conditions that the results will reflect the average reality. It's not perfect as KDR naturally fluctuates, but it's a good first test.
Skihids Bullet Cluster 1849	Posted - 2013.08.09 02:37:00 - [6] - Quote Interesting video. Subjective experience is a great starting point, but we need rigorous investigation after that. The lack of FF is definitely a factor, but I don't think it's a major one. The biggest factor is that it works differently than hit-scan weapons and there are some people who either can't or won't accept that.
Skihids Bullet Cluster 1851	Posted - 2013.08.09 14:23:00 - [7] - Quote BLKDG wrote: Thank you I was starting to think the lunies ran the asylum These forums... Man these forums. .. I put a lot of the blame on our educational system for neglecting basic reasoning and claim evaluation skills. People need a defense against the constant onslaught of Madison Avenue persuasion that as gotten quite scientific in its own right (creating price anchor points, giving false comparisons, etc.). A very brief introduction to statistical concepts goes a huge way to arming you against bad arguments and misuse of data, and it's actually quite interesting. The math is extremely basic for the most part and everyone uses a computer to crunch the numbers anyway.