The idea that some evidence categories are stronger than others. A possible, but not mutually-agreed-upon ordering is (strongest first): * '''Formal proof''': A proof, constructed from relevant axioms, often of mathematical or symbolic logic (assuming good givens). * '''Deterministic experiment''': Experiments which have been shown to be repeatable (by different observers), which are adequately controlled for the parameter being tested/measured, and which have otherwise sound methodology, and which consistently back up (with high correlation, preferably 1) the assertion. * '''Statistical experiment''': Experiments which have been shown to be repeatable (by different observers), which are adequately controlled for the parameter being tested/measured, and which have otherwise sound methodology, and which corroborate the assertion by a statistically significant factor. * '''Single-Sample Experiment''': Thorough example or live demonstration of results or performance, or a small set of such. * '''Simulation/Modelling''': Toy examples or scaled-down demonstration * '''Conjecture''': Philosophical reasoning. * '''Personal Anecdotal Observation:''' "It's always helped my projects run smoother." * '''CommonSense''': 'Normal' everyday reasoning * '''ArgumentFromPopularity''' * '''Emotions''': It "feels" right or wrong * '''FallaciousArgument''''''s''': AdVerecundiam arguments and anecdotes, other fallacies. Note that referring to a published scientific paper that contains the above (especially the first 3) is not AdVerecundiam. ---- '''Arguments about the order''' ''What is the reasoning for this ordering? I tend to disagree strongly with some of the placements, but would rather hear the rationale than just to change it for my preferences.'' It's the ordering of a Platonist. {I thought it this was obvious. I would be interested to see an alternative ranking.} I demand EmpiricalEvidence that this ranking is correct. ''I don't think it has to be correct more than agreed-upon between 2+ parties with issues to hash out. There is no use arguing about the existence of a chair if both parties agree that it is there.'' Why should there be a hierarchy? I would suggest that each of the above is a necessary component and the items form a self-reinforcing set. It is inappropriate to pull one item and declare it superior to the others with the implication that it can stand on its own. ''It's only a rough guide. You are welcome to present an alternative ranking or scoring system.'' ---- Perhaps AdVerecundiam should be split into individual authorities and collective statistical surveys (non-filtered). "Bob says that X is better" is weaker than "4 out of 5 software engineers out of 1000 surveyed say X is better". I would place the second above "philosophical reasoning" because it is possible for something to be better without knowing why. Of course, such surveys cannot filter out the BandWagonSyndrome and other peer-reinforcement effects. ``I would expect that the statistical reasoning of `4 out of 5...` fits closely into the `thorough example` end of thing anyway.`` {The quantity versus the type of evidence are generally orthogonal. We can have one anecdote or a thousand, but we could also have one actual experiment/simulation of a "representative" example, or hundreds of representative examples. However, our current list is one-dimensional such that it doesn't represent both axises (axi?) very well (type versus quantity).} ---- A Ladder metaphore might be more useful. I don't think a real totem pole implies rank. ''Man, stop kicking dirt into my real-life metaphores :-)'' ---- The above makes it sound like AdVerecundiam is "fallacious" (FallaciousArgument). It is weak evidence, not outright fallacious. ''In the context of rhetoric, it's fallacious. That doesn't mean that appeals to authority are wrong; just that it means -- like you said -- weak evidence.'' ------ Re: "referring to a published scientific paper that contains the above (especially the first 3) is not AdVerecundiam." ''I'm not that comfortable with "science==educated-votes". "Peer review" is merely a shortcut to more formal processes that are not otherwise practical.'' It is '''not''' AdVerecundiam unless the person referring to a paper is saying: "Look! This author is right because he's an authority!". I find it far more natural to assume the person is saying: "This paper already presents a reasonable argument and evidence towards this conclusion, so I have no reason to repeat it in full here." And if you read carefully you'll notice the 'that contains the above (especially the first 3)' IS indicating that the paper being referenced happens to contain a proper argument and evidence. I'm very much of the opinion that emotion cannot be placed into this list because emotion is often a very accurate guide - think of emotion as responses that are hard-coded in WetWare, responses that have been evolved over time. So long as the stimulus isn't too new, the instinctive response is quite likely to be extremely sensible. -- BenAveling ''Studies referenced in the book "Supercrunchers" would suggest otherwise. It appears that human subject experts tend to glom onto a few "pet factors" and over-magnify them when making summary judgments.'' What sort of decisions were they examining? ''Fields in which there were "experts" involved in making decisions. Typical examples include sports scouts, financial, wine futures, and law (such as predicting court outcomes). And they also did studies with regard to human versus machine prediction abilities on "lab games". The "few-factor-glom" pattern seems consistent. It appears an embedded feature of human nature. (Related: NarrowStaffSelectionFactors)'' Lawyers aren't "experts" at predicting court outcomes - they don't receive training in that field. Sports doesn't have measurable 'experts' (no tests to become a sports scout) so any claim of expertise is suspect. You certainly won't find human "experts" at ''lab'' games. And finance is one of those strange places where observations and making decisions strongly affects the outcome (making the domain highly subject to uncertainty principles) and where you can't point to any ['''Formal proof''', '''Deterministic experiment''', '''Statistical experiment''', '''Anecdotal Experiment''', '''Simulation/Modelling''', '''Conjecture''', or '''CommonSense'''] that does any better - the argument that "human subject experts tend to glom onto a few "pet factors" and over-magnify" seems like talking out of your ass '''unless''' you can point to the correct factors and the right magnification. I'd be more interested in a study regarding human subject experts in '''chess''' or '''go''' - i.e. people who are ''measurably'' "experts" - making 'summary judgements' based on their chosen 'pet factors' and comparing these results to intelligent but untrained persons attempting to track statistical properties, formal reasoning, and conjecture. I'd bet significant money that the 'experts' with their 'summary judgements' would make strong moves and win almost every time, thus supporting BenAveling's claim. ---- '''Arguments about the WikiName''' A totem pole is a carving that tells a story or family history. It may have other uses too. The animal at the bottom may or may not be more important then the animal on the top (as it is upon what the others stand. Without it, the totem is smaller.) Why not use EvidenceOrgChart. ''Too confusing IMO. People will think it about org charts. Maybe EvidenceRanking?'' ---- http://www.dictionary.com ''Main Entry:'' totem pole '''1 :''' a pole or pillar carved and painted with a series of totemic symbols representing family lineage and often mythical or historical incidents and erected before the houses of Indian tribes of the northwest coast of No. America '''2 :''' an order of rank : '''HIERARCHY''' ''Main Entry:'' crucifix ''':''' a representation of Christ on the cross ''Main Entry:'' cathedra ''':''' a bishop's official throne ---- ''Why a TotemPole? Does it have to be a religious structure? How would we feel about an EvidenceCathedral or EvidenceCrucifix?'' The assumption is that the totem pole represents a ranking or hierarchy of some sort with the most important beings on top. Crucifixes are not like this in design. '''OTOH''', there is plenty of cross-related imagery. WilliamJenningsBryan 's "cross of gold" comes to mind... ''I'm familiar with the usage ("low man on the totem pole"), but why use a religious reference at all when we're talking about evidence? Why not use ranking or hierarchy if that's what we mean?'' Framing the concept in secular terms might suggest we could build such a hierarchy objectively. {Further, I don't know of a decent secular visual/conceptual metaphor.} ''Ranking or rank. It doesn't suggest objectivity.'' Are you a Native American ("Indian") by chance? ''1/32nd Siksika. Are you?'' {So you are only 1/32 offended? That ain't bad.} I think we could dispense with the totem pole if we concentrated on a specific application. Given the lack of context, the totem pole seems, symbolically, quite appropriate. Does the "possible ordering" given below make any sense in my daily life? Can I apply this when I'm searching for a pair of socks in the morning. Or when I want to know what time it is? The fact is that many of us can go a whole day achieving personal goals and conquering small tasks without this ordering or something similar. ''I believe it originated in discussions about how to demonstrate that one paradigm is "better" than another.'' ---- '''Heated argument centered around human competence at making guesses''' ''Can't you disagree without being so rude? I'm tired of dealing with people who use such an aggressive style on this wiki. Go argue with yourself. My participation in this spot has ended.'' When you misrepresent evidence or studies as supporting a point it does not support, you deserve a kick in the teeth. Can't you present an argument without resorting to sophistry? ''I did not intentionally distort any info for the purpose of manipulation. If you have an issue with something, please state it without conspiratorial brushes. KeepCriticismNarrow.'' I consider making statements in ignorance a greater offense to reason than intentional conspiracy. Sorry for giving you the benefit of the doubt. ''Thanks for ruining what *could* have been in interesting discussion by using AspergersSyndrome-like communications "techniques".'' Hello, poT. I'm kettle. Maybe poT could have responded to content rather than throwing an anal-expulsive tantrum about the 'rudeness' of an appropriate and well-deserved jab against his lack of professional ethics when it comes to presenting arguments and putting forth the appropriate degree of effort to ensure his references support the claims he's making. Maybe, then, we'd have had an interesting conversation. But, instead, it seems poT wishes to focus on slights against him... perhaps poT finds throwing mud very 'interesting', because it's certainly what he spends most of his time focusing on. Does poT ruin what *could* have been interesting discussions intentionally (like an intelligent man), or through ignorance (like an emotional boar)? Who knows. But do be aware that, just as a person has no 'right' to incompetence the moment he presents or sells himself as a professional, a person has no 'right' to ignorance the moment he presents himself in an argument for purpose of making a point. ''I have no idea what you mean by "lack of professional ethics", but I don't want to know anymore. Your style so far sounds too similar to Lars who is offended by very hard-to-predict statements; and he's worn out his welcome with me. He lives in a funny little world with funny little morals. I can't stand having discussions with that......person. Have a nice day.'' * What exactly are "very hard-to-predict statements"? That term doesn't make any sense. Statements that people have trouble predicting what from? The meaning of the statement? * ''It perhaps should have said: "It's very hard to predict which statements will offend you."'' * On the other hand, couldn't it also be that the person writing those statements is not able to address their problem or acknowledge it? They don't even see it as a problem; they simply think they are polite, friendly, experienced, and better than others. I mean if someone is not self aware that they are acting like a total ass-wipe, and they are in denial that they are an ass-wipe, it's quite hard for that person to change. It is also quite hard for that person to even accept that they were being an ass-wipe. I'm not saying that this is the case, it could very well be that the person is not an ass-wipe. It's like the man who goes to the bar and gets punched in the face by several people. The punched man asks why in the world would someone treat me like that? Could there be a reason? Or is it just that some random ticking person is mean for no reason. Often there is a '''reason''' that people react to someone or something. Usually the something they are reacting to has a '''problem''' that is being addressed. Punching in the face is not the answer, and adding more gasoline to the flame doesn't always work, but unfortunately sometimes that is what happens; a fire is evidence that there is a problem that needs to be addressed. Unfortunately, it actually works out this way: treat those how you'd like to be treated. So if one cannot acknowledge that he is insulting and arrogant, then the cure for that person might be another of the same kind - an equally insulting person treats him with disrespect until finally and hopefully both treat each other with respect, once they learn that one should treat others how they wish to be treated. One person may know this all along and may be just playing the game, while the other person may be completely ignorant and living in a fantasy world where he thinks he is in fact a good person that doesn't insult or cause any trouble at all (it's all the other person's fault, and one hasn't looked in the mirror). * ''Perhaps the person in the bar keeps expressing political views and feel they have the right to express political views. Whether they are "morally" right to express such opinions may be a different issue than the question of what behaviors get one a fist-burger. That Danish cartooner would be hung if he walked into certain Middle East countries. That doesn't mean that his cartoons are wrong or evil in the absolute sense, only that certain behaviors in certain places set off certain other behaviors in others.'' Funny, your style so far sounds too similar to Top's, who persistently presents flawed and misleading arguments then smugly says it isn't 'intentional' on his part as though he's incapable of learning to do better. He's really worn out his welcome with me. He lives in an irrational little world where EverythingIsRelative, including truth and evidence, all so he can justify in his willful ignorance whichever unsubstantiated opinions strike his fancy at the time. He IS, however, utterly predictable in his tendency to favor whining about minor slights (be they deserved or not) over focusing on useful conversation, and in his tendency to go for the last word no matter how unprofitable the conversation. I can't stand having discussions with that......person. Have a nice day. [You two: Get a room.] ''Actually there'd be three people - probably two against one, which wouldn't be fair - so he'd have to forfeit. Top became the bottom.'' Maybe a kickboxing ring... '''Debate Boxing''', akin to http://en.wikipedia.org/wiki/Chess_boxing. I'd probably be able to stand TopMind's behavior if a judge were docking points for every fallacy and I could hit him between sessions. * ''I'd welcome finally getting REAL feedback, instead of insults packaged as "teaching".'' * You received a great deal of REAL feedback and one ''very'' minor bit of rudeness - that it seems you're "talking out of your ass" about "pet factors" and "over magnification" unless you have a way to identify the correct factors and magnification. But, being TopMind, you chose to whine about rudeness rather than welcome the feedback. If I didn't know you were completely and willfully ignorant of your own flaws, I'd say you were lying through your teeth about welcoming feedback - it seems to me you're instead looking for excuses to ignore it. * ''YOU started the direct rudeness when you said "talking out of your ass". YOU started it, not me. You are the guilty party. Face up to misdeeds, you weasel! If you wanted more details about the factors involved, then simply ask in a civilized way. Your "professional ethics" excuse for your "ass" comment was idiotic, Lars. Stop inventing Rube Goldberg excuses for direct insults.'' * See, you're doing it again: focusing on blame, whining, throwing a tantrum like a petulant child, hypocritically flinging mud and insults all the while demanding better of others, and blowing a minor bit of rudeness well out of proportion rather than rationally focusing upon and welcoming the significant feedback that accompanied this line you find so insulting. Your choice, or at least it would be a 'choice' if you made it consciously like a sapient animal. Anyhow, my opinion hasn't changed: I think people who misrepresent evidence deserve a kick in the teeth. A little rudeness is nothing compared to kicking you in the teeth, so if you're asking me to apologize for a little bit of rudeness you're unlikely to receive one - though if I could get you in a kickboxing ring, maybe I'd kick you in the teeth then apologize for the rudeness. And I'd hate to misrepresent Lars to you, so now that you've made explicit your assumption of my identity: I'm not him... though I wonder whether he'd be amused or offended at your assumptions. * ''Re: "I think people who misrepresent evidence deserve a kick in the teeth." - And thats a bigger "sin" than a personal insult? You are abnormal. I believe you deserve a kick in the teeth as well. How about we agree to meet in a dark park around 3am and have it out. Even if I lose the fight, getting in a few good punches would be quite satisfying. You are a really annoying and weird SOB.'' * You really think most people would prefer violence directed at them over insult? And I'm the abnormal one? * ''Where the flying fudge did I present them as a mutually-exclusive choice? Never mind, it's probably yet another rube-goldberg explanation. Hell, I bet your fighting would also be goldberg-style: a fist hooked up to a fish tank with a windmill and a bag of sand tied to an alarm clock.'' * And where did I imply you presented them as a mutually-exclusive choice? What you asked is "[You believe kicking someone in the teeth is] a bigger 'sin' than a personal insult?" And then you followed it up by calling me abnormal for believing violence a bigger 'sin' than personal insult ("A little rudeness is nothing compared to kicking you in the teeth"). Now, in my experience at least, most people would prefer being called names to sticks and stones, so I can't consider my own impression on this subject to be abnormal. Personally, when it comes to expressions of immense displeasure, I think I'm saner to prefer insults over violence. But if you find personal insults more harmful to you and your impervious arrogance-armored ego than a kick would be to your teeth, I'm cheerfully willing to apologize for my insults while kicking you in the teeth... just sign a waiver when you're ready. And if you don't favor mutual-exclusion, just let me know: I'll happily insult you while kicking you... at least if I don't get creeped out by the possibility of desiring insults and punishment being some sort of masochistic sexual thing. * ''This is growing more and more convoluted. Never mind, I don't wish to try to figure out your odd morality system. I am leaving that fractal cave system for now.'' [A couple of years ago, I designed a framework for hosting on-line Web-based structured debates. See InternetDebateProposal. I really should implement it.] ''Mutual hate, finally something we both agree on.'' -------- '''Utility of Experience''' Experience can be useful evidence even if one cannot name all the individual cases or scenarios that produced experience. Neural-network-based systems, such as the brain, often have '''lossy learning''' by storing the fact that a certain pattern is/was common, such as by reinforcing weightings or links between axons, but not necessarily the details of each experience leading up to it. For example, if you create a training data set for an artificial neural net, there's no way to extract the "images" of each individual training card or instance. Yet, the net has "learned" to recognize the pattern(s) of the training set. The more extreme experiences tend to be remembered, but not necessarily the mundane. For example, a bear may remember details of some fights it had with competitors, but not how it caught each and every fish it ever ate. Similarly, I still remember some of the specific mistakes I made in trigonometry because each red mark was a big mental "ouch!". However, I remember hardly anything specific about the problems that I got right. (It's possible the bear brain is very different from human brains, but bear with me for the illustration, pun intended.) Experience judgment in humans can be useful; however, there are at least two problems. First, if the training set is not documented, there's no way for the reader to verify. Second is bias. Humans tend to be biased toward certain patterns and tendencies. We don't absorb all lessons equally, or at least weighted consistently across individual people. For example, some may remember the effort to fix a bad problem more than how loud their boss yelled at them, and visa verse. - t ---- For consideration: Proof that visualizations, impressions, ideas and dreams involve all, or most, or some, or any of the sensations or capabilities we class as capabilities. To illustrate: I can not recall the color of, or in fact whether or not anyone I ever dreamed of as wearing their glasses, or for that matter, what color tee shirt, or sweater, or stocking cap they may have had on. What would prove I had? Would a photograph taken during that dream of the said object be evidence? In today's newspaper a member of law enforcement in Greece was shown on the ground, surrounded by and engulfed in the flames from a gasoline projectile. The immediate impression I had, before opening the paper to see a photograph below the fold, was that he most certainly would have died from what appeared to be happening. The photo below the fold and the description of the event showed him escaping to safety and surviving the flames. In this case, first impressions did not prove to be evidence of a fatality, for subsequent facts disproved what seemed apparent. Some evidence we amass is based on impressions of what we see, or imagine to be, in a moment, rather than what is in fact and evident, and demonstratable from what is or was. Did the magician really pull a silver dollar from behind another one's ear? Or did any of the things appearing to have happened really happen? Is seeing believing? I would suggest that much of what we come to believe or espouse can never be proven fully and with logic. If it could be disproven, I suggest that the magic and livelihood of living would disappear. ---- Should FormalProof be at the top of the pole? I'll grant that the two are rarely in conflict, but wouldn't broad-based experimentation trump formal proofs? If the formal proof says P and the experiments say not-P, then I'm convinced of not-P. Even formal proofs merely model reality. ''No, correct formal proof is absolute and incontrovertible, and can be verified by automation. Empirical experimentation is always subject to error, no matter how small. Unfortunately, the degree to which formal proof is applicable to the real world varies. In mathematics and computer science, the relationship is one of equivalence. In physics and branches of engineering, it should be the same, but the number of variables involved (some of which may be unknown) complicate this.'' SoftwareEngineering also has many variables. In fact, I'd say it's harder to objectively measure than engineering designs because engineering is heavily tied to physics, while SoftwareEngineering is largely tied to WetWare. It's only a little bit about computing machines and math. ''SoftwareEngineering is a branch of engineering.'' I'm not sure I agree. Somewhere there's a topic or two on that. Engineering is much more heavily guided by the laws of physics, while software engineering often revolves around WetWare, which is a very soft science so far. ''Mathematics constrains software engineering in pretty much the same ways that physics constrains, say, electrical engineering. There are also plenty of "WetWare" issues for the other engineering branches.'' Other than user interfaces, such as (real) control panels, what are common examples? And while I agree that math constrains software design, it's not a very narrow constrainer. It still leaves a hell of a lot wide open. * {A common example of mathematical constraint: Search time complexity of a binary tree is average O(log n) and worst case O(n). That is a mathematically-provable truth which has a direct impact on software performance and software design.} * Well, okay I agree on performance issues. Performance is one of the few areas I agree that theory has been heavily helpful. It's been great on the "machine" side, but iffy on the WetWare side. * {That's a concern for psychologists and sociologists. I'm a ComputerScientist. As such, performance, security, reliability, expressiveness, readability, and efficiency are -- among other things -- my concerns.} * ''There's also the limits imposed by RicesTheorem.'' ''The usual ones you point out. You still have to be able to communicate your designs to co-workers. You still have to have domain knowledge. You still have to find solutions to poorly specified problems.'' That's true, but those appear to be smaller issues in physical engineering. Writing software is primarily about communicating with humans. The machine is secondary. ''The only one where the issues are any smaller is in communicating designs. The methods used to communicate are largely set, so there's not much need to argue about which ones are better. The others are, if anything, larger issues. The cost of being wrong about them is larger for the other engineering disciplines.'' That's a big one. Making code grokkable and team-maintainable (often related issues) is the bulk of the effort. Well, understanding customer requirements is a big problem too in my experience, but that's typically not something we heavily debate about on this wiki. ---- See Also: EvidenceVersusProof, EvidenceEras, GoodMetricsProduceNumbers, OfficialCertifiedDoubleBlindPeerReviewedPublishedStudy, AnecdoteImpasse, AreWeaklyBackedOpinionsAcceptable, ProxyFactor ---- JulyZeroEight CategoryEvidence