Smarter Intelligence Checks

Robijeign, Arch-Diviner of Stull, gazed at the runes. Jagged hashes, rough, and deep, the strange markings had been found carved into the underside of the banquet table: ten of them, forming a circle, with an additional marking in the center. Eleven markings, total.

Mere moments earlier, the emissaries dining at that very same table had been at each other’s throats, literally. (Well, except for one diplomat who had managed to reach the carving knife. His work had been more … thorough.) Certainly, the factions at that table didn’t like each other. But the banquet had been held as a step toward peace, so the sudden bout of violence struck Robijeign as unfortunate–and odd. Suspecting some sort of arcane sabotage, Robijeign had alarmed everyone by flipping the table over, sending dishes, food, knives, and one dead body to the floor.

The runes, freshly carved, seemed to confirm his hypothesis, but he studied them anyway, tuning out the surgeons and aides scurrying around dealing with wounds and bloodshed. Who carved these runes? What spell, specifically, was this? What culture, tradition, or school might teach such magic? With all of these questions in mind, he considered what he knew…

DM: Roll an Intelligence (Arcana) check.

PC (Robijeign, rolling d20): Um, an 8 on dice. Plus 8 is a 16.

(DM consults his notes. The DC is 18.)

DM: You don’t know anything about those runes.

The most unrealistic moment in the narrative above isn’t the table curse. We’re witnessing a fantasy game; spells go with the territory. No, the most unrealistic moment is the DM’s interpretation of the skill check. He’s misunderstanding how knowledge works.

My colleague Wallace Cleaves has been sharing his expertise as a medievalist in a series of columns called “Get Medieval.”

I would like to dip into my own day-job for a moment and talk about Intelligence checks. When not running games, I teach teachers. Specifically, I train future teachers of writing how to teach something as difficult, complex, and elusive as writing. I also conduct research on ways recent discoveries in the science of learning, thinking, and knowing apply to what teachers do.

There are, it turns out, a lot of myths about the way the mind works. Normally, I keep my nitpicks to myself. However, in this case, that knowledge yields some suggestions that might improve the playing experiences of both DMs and players. Players often have to check to see what their characters know, apply knowledge to a puzzle and come up with a solution, or try to understand something. A better (if simplified) understanding of how those mental moves work can make for a richer game without any need for elaborate new game mechanics.

Consider Robijeign’s situation: He’s a diviner, a wizard, trained in Arcana (and probably History and Religion, too). How likely is it that he looks at a rune and comes up with … nothing? At all?

It helps sometimes to consider more familiar experiences. For instance, total amateurs with no previous alien encounters or expertise in geology look at photos of the Martian landscape. Instead of saying, with a shrug, “It all looks red to me,” they blog about how they see alien faces carved into the surface. The human mind abhors a vacuum. I remember being in a newsroom when the Oklahoma Federal Building was blown up in the 1990s. The reporters in that room were sure — positive — that it was the work of Islamic terrorists. They were wrong, of course. But they didn’t walk into the story from a vacuum of knowledge; they knew about Islamic terrorists, but not about the less active sorts closer to home. And when students walk into a classroom to learn something, the single greatest obstacle to their learning isn’t lack of knowledge–it’s all of the wrong stuff they think they already know. Very, very rarely will you look at a puzzle, mystery, object in a museum, whatever, and come up with nothing.

Instead, one of two things will likely happen:

1. You’ll be certain you know something, or certain  you have the right answer. (About this, you might be either right or wrong.)
2. You’ll be uncertain to some degree, thanks to a fuzzy mix of things you’ve heard or think you recall, some of which might be relevant, some of which might be irrelevant, some of which might be inaccurate, some of which might be accurate, and much of which may be incomplete. Maybe you’ll be undecided between just two theories. Maybe you’ll have too many ideas to sort through. But you have more than one idea to play with, at any rate.

To skip past the science and right to the punchline, click here. To cut straight to the table at the bottom of the article, click here. However, if you later decide to comment, please read the rationale first.

Inside an Expert’s Mind

Even with experts who have PhDs and who have mastered thousands of pages of material, the second case above is far more likely than the first. The smartest characters you see on TV or in movies (the Dana Scullys, Doctors, and Sherlocks) frequently go through a series of interesting and smart wrong answers before arriving at the right one, in a way that closely mirrors real expertise. As SF giant and chemist Isaac Asimov has argued in “The Relativity of Wrong,” not all wrong answers are equal. The best wrong answers tend to lead toward right ones–they give a smart mind a starting place for finding out the truth. The worst wrong answers, those of amateurs, are non-starters; if a scientist friend ever tells you your idea is “not even wrong,” that’s not a compliment.

Cognitive scientist Daniel T. Willingham likes to cite the TV show House when talking about critical thinking and the ways experts differ from novices. In the show, Gregory House, MD solves medical mysteries. He very rarely hits the right solution immediately. (If he did, each show would be five minutes long.) Moreover, the younger doctors who work for him often know the same facts he does–not just about the case, but about relevant medical conditions. However, House is better than his colleagues and subordinates at deciding what’s relevant or irrelevant, and when a new piece of information comes along, he’s quicker than they are to realize what makes it important. While everyone else is sidetracked by irrelevant fact A, he recognizes that it’s the seemingly trivial fact B that deserves more attention.

Expert knowledge works this way: It’s more about sifting through facts than having them. Anyone can use WebMD, and thus anyone with a phone today can “know” what it says. But you consult with an expert because the five symptoms you have might mean any of 20 things and, unless you’re medically trained, you’re unlikely to have the experience or insight required to eliminate all of the wrong answers.

I can elaborate better with an example from my own field. You have almost certainly heard of IQ. You have probably heard of multiple-intelligence or learning-styles theory, though possibly not by that name. (If you’re familiar with the idea that some people are verbal learners, some are musical learners, some are hands-on learners, some are visual learners, etcetera, you’re in the right ballpark.) Some educators who don’t do research in the field have heard a lot about learning-styles theory, believe it, and practice it. I frequently hear teachers talk about learning styles, and almost as frequently hear other, master’s-degree-level trainers of educators talk about them. The people who talk about learning styles can all point to books and articles and texts that support their belief that customizing learning experiences for students’ learning styles helps them learn better. But I and the PhD-level psychology researchers I know all think that idea is a damaging distraction. Why? There are several reasons, but the nail in the coffin is this article.

Ah, you say, but that’s just one article! Why should we believe that article over those that say learning style teaching works? Good question. Chiefly because experts differentiate among sources of information. You might already, yourself, differentiate between what’s reported by a credible news source and what’s being reported by friends on Twitter. You might also know that stories in The Onion are jokes. (Not everyone does.)

Just as you discriminate, so too do experts in a field. Different scholarly journals carry different weight. Different types of research or articles carry different weight, too. A theoretical argument isn’t as persuasive as an empirical test — an experiment — of that argument. Similarly, a single experiment isn’t as persuasive as a meta-analysis. A meta-analysis takes many previous studies, crunches their data together, and determines what they say as a whole. That’s an important move because any two individual experiments might seem to contradict each other. Randomness, uncontrolled variables, and other factors can hide the truth. Also, sometimes scholars don’t report results that seem to contradict expectations (or those results don’t get accepted for publication). However, a good meta-analysis can sift through the junk and identify what most of the experiments seem to be saying–or, in some cases, point out that important results probably aren’t getting published. It’s not for nothing that the top-ranked journal in education refuses to accept experimental reports, publishing only meta-analyses and similar types of advanced reviews.

Well, the learning-studies article I linked earlier to is a meta-analysis in a top-notch psychology journal. It crunched a bunch of studies on learning styles and came up with … nada. Teaching to learning styles doesn’t seem to accomplish anything. If you want to teach a kid to read, do it verbally, rather than musically. Teach math through math, not physical activities. Experts (people who do research in the field) can tell that the study in question is a good study, and it’s far more persuasive than any of the individual experiments it crunched. Someone else waving a different article with a different result from a single experiment in a lesser journal just isn’t going to be as persuasive.

In short, both the learning-styles advocate and the learning-styles critic may have read a bunch of the same information. They know the same facts. But they arrive at very different conclusions because each evaluates the evidence differently. Going back to the House example, the expert is often better able to tell which facts are important.

Applying the Concept to Role-Playing

The above phenomenon, which I realize may have seemed like a bit of a tangent, applies to occasions when PCs are supposed to be experts themselves. When two experts in the arcane study the same runes and only one of them succeeds at the skill check, it’s unlikely that one of them knows what the runes are and the other one doesn’t have a clue. What’s more likely is that they have competing theories, both of which sound kind of persuasive. Even more likely? Each one has several theories, maybe even the same theories, but they rank their likelihoods differently. Expert 1 thinks theory A is more compelling than theory B. Expert 2 thinks it’s the other way around.

And that’s cool. Why? Because it gives “intelligent” PCs intelligent things to do: they can argue, follow lines of inquiry, ask more questions, collect more facts, confirm hypothesis, disprove hypotheses. This activity can make them look and feel as smart as their INT scores say they are. Consider, Robijeign’s earlier Intelligence check. He’s rolled a 16. The DC was 18.

DM: You have some conflicting theories. You’re pretty sure they’re not Olem runes. But they might be Nar’tik or Gettic. Both of those Northern cultures use similar runes, as they are related. But you’re not sure which of the two Northern cultures they might represent. On the one hand, the symbols look more Gettic to you. On the other, a Nar’tik epic poem tells of a hero afflicted with a “strife rune” that had effects similar to these. You think whoever it was might have been trying to hide his or her nature by using a Gettic rune style in combination with a legendary Nar’tik effect. So…could be Gettic, could be Nar’tik, could be a third party who has studied such things. There are matters of tradition that might help you narrow the possibilities: A Gettic rune-graver would be tougher to spot than his Nar’tik cousin. Nar’tik runecasters are distinctive, wearing rune-style tattoos over their faces, and they are very protective of their magical lore, seldom teaching strangers. Gettic runemasters are less obvious and mingle more with other people, sometimes passing on their traditions to others.

PC (Robijeign): Okay. I tell the guards to look over the grounds for someone with distinctive facial tattoos. I suspect that’s a dead-end, though. Pretty sure someone like that would have already drawn attention. What about motives?

An Intelligence (History) check ensues.

DM: The Gettic folk were one faction at the table. Not all of them felt like compromising in this conflict, and one of Gettic assembly had used a carving knife on another diplomat. On the other hand, the Nar’tik have border tensions with the Gettic. They might benefit from ensuring that the Gettic people are in conflict on another front.

PC (Robijeign): This is very Agatha Christie. Everyone’s a suspect! Okay, let’s pursue that third option a little bit. There’s a university in this city. Do any of the sages there study Nar’tik rune magic?

DM: As it happens, you’ve read a treatise on them by Aragam Aurostaff, a dean at that university.

PC (Robijeign): I rally the rest of the party. I think we’ve got a starting place.

I prefer to think of the Intelligence check not as a “knowledge” check, but as a Hypothesis Check or a Processing Check. My bell curve for Hypothesis/Processing Checks looks something like this:

Check Result	Nutshell Description	Notes
DC +4 or higher	Certainty (Correct)	Character has great certainty in a single answer.
DC +1 to +3	Confidence (Success)	Character has confidence in idea A (which is correct) but has idea B as a back-up.
DC-1 to DC	Confidence (Near Success)	Character has confidence in idea A (incorrect) but has idea B (correct) as a back-up.
DC -2	Uncertainty (Hypothesizing)	Character entertains three or more ideas, one of which (not necessarily first or last) is right.
DC -3	Lost	The character either has too many possibilities to sort through or else not enough information to go on (depending on which interpretation makes the most sense). On a good note, the character recognizes his or her limits.
DC -4	Uncertainty (Fishing)	The character entertains three or more ideas, none of which is correct, though one or more might lead to more clues.
DC -5	Confidence (Failure)	The character is confident in one idea (which is wrong), with another idea (also incorrect) as backup.
DC -6 or more, or Natural “1”	Certainty	The character is absolutely certain of something that is superficially persuasive but wrong. Character is likely suffering from the Dunning-Kruger effect. (That is, her lack of relevant knowledge prevents her from realizing how wrong she is.)

The above table isn’t a rule, of course; it’s just a rule of thumb, a sketch of what the bell curve looks like. At either end of the scale, people are very confident. In the middle, people juggle multiple competing ideas, some of which might be right, but they have to test them out or collect more facts to confirm them. The middle results are normal, with high results leading to faster deductions than low results. The low results might still eventually lead PCs the right direction, but, like Gregory House, they may hit a few blind alleys on the way there. The same principle can apply to Wisdom (Survival) checks when tracking the raiders who sacked a small town, and other sorts of thinking checks besides.

I don’t advocate using the above approach for every check made during a game. Notice that the DM didn’t have Robijeign roll for that last question; it was a simple, straightforward fact, so the DM just gave it to him. However, the approach I just described will tend to improve a game when used in cases with the following characteristics:

1. You know in advance the PCs will make the check as a planned part of the story.
2. The check will call for processing, theorizing, or evaluating of information — not just simple factual recall.

When I know such situations are coming, I like sketch out a handful of likely-sounding theories, one of which is correct, two of which are wrong but likely to lead to more evidence, and one of which might lead somewhere both wrong and dangerous. Depending on the result of the check, then, I tell the players which theories came to mind. Because group checks tend to ensure someone will always roll well, I might also follow DM David’s advice by giving trained experts in a subject somewhat better possibilities than I give people who simply rolled well. Used for every single check, such an approach will lead to paralysis. But for key moments, like Robijeign’s, it can be a door to adventure.

Before wrapping up, I’ll quickly address some questions I suspect readers might have after looking at the above chart:

• Why is a clear-cut, single, correct answer listed at 4 above the DC? Shouldn’t that be set at DC and above? You could set it at DC+, of course. But if I did that, I would be sorely tempted to make all of the thinking (INT, WIS) 4 to 5 points more difficult. Why? Because experts very rarely arrive at perfect, confident, single answers. It’s just weird. It can happen if the check is easy or the person is really, really, really brilliant. But most of the time, if you go to a doctor with a bucket of symptoms, the doctor’s going to go through several hypotheses. Not just one. So for me, success (DC and higher) equals having a set of good hypotheses, one of which is the right one.
• Why does a character who rolls a 1 receive just one bad idea, while someone who rolls better gets multiple bad ideas? Uncertainty is a good thing. It’s far more dangerous to be certain and wrong than to be uncertain and sifting through several bad ideas. The uncertain character knows to be cautious and might take reasonable precautions testing any of his or her wrong ideas, and in so doing, is likely to come across other clues or evidence and get another shot at a better insight. It’s the person who is absolutely certain and absolutely wrong who will accidentally set a demon loose on the countryside. If you’re not sure about this, try it in play, with actual players before discarding the idea. Watch how the playing style changes depending on whether the character is certain about a single wrong answer or uncertain about several possibilities. I will bet that it becomes clear that the latter is the better, less-dangerous result.

Your mileage may vary, of course. If you have other ideas for how to approach the messy ways that brains work, sound off in the comments below!

♦ Graham Robert Scott writes regularly for Ludus Ludorum when not teaching or writing scholarly stuff. Graham has also written a Dungeon Magazine city adventure titled “Thirds of Purloined Vellum” and a fantasy novella titled Godfathom. Like the Ludus on Facebook to get a heads-up when we publish new content. 

 

Update: Kamishiro_Rin, a reader of this article and GM on the Myth-Weavers site, has given me permission to link to a page in which his game started using the approach described above, so you can see it in action with a real game instead of a hypothetical. (It’s pretty cool.) Thanks, Rin!