...where distraction is the main attraction.

Sunday, January 5, 2014

The new grading system

After 18-20 hours of editing posts and spreadsheets, I think it's time for me to roll out the new grades.

When I first started the Single Malt Reports almost 2 1/2 years ago, I didn't have my own history of rating whiskies to reference.  Either it was excellent or good or drinkable or dumpable.  Though that's still sort of the basis of grading whisky anyway, I didn't have much of a system in place.

I'd spent the previous 15 years of my life privately grading films.  That system started with a five-star structure, but then expanded out into a number of spreadsheets, partially because I love numbers and partially because the sheer quantity of films allowed me to compare and contrast and thus produce a more honed 100-point system.

A week before I posted that first Single Malt Report, I figured out how to create stars and 1/2 symbols via HTML.  That was fun, so I used those.  After a little while, I had the brilliant stupid idea to actually rank the stuff I was drinking.  That's fine with 50 whiskies or so.  At 200 it got to be too much of a pain in the ass.  Plus that at level of exactitude, 200+ separate "scores", it stopped making sense.  After whisky #200, I knew it was time to start transitioning to a 100-point scale.

Michael Jackson had a 100-point scale.  Ralfy has a 100-point scale.  Robert Parker has a 100-point scale.  Jim Murray has a 100-point scale.  Whisky Advocate has a 100-point scale.  Serge has a 100-point scale.  But they're all different 100-point scales.  Parker assigns 100 points to wines rather regularly, to the extent that a 90 point Cab sounds like it's crap.  Murray's faves score at 97.5 each year.  Jackson didn't go above 90 often and neither does Ralfy.  While Serge does go over 90, he had only 3 out of 1000+ whiskies scored at 94 points or more in 2013.

There's also been a bit of discussion about whether reviewers really utilize the entire 100-point scale or if they just park themselves in the 80-90 range, using 10% of the possible grades.  I can understand that gripe.  Whisky Advocate rarely grades any whisky below 80.  Serge often drops into the 70s and a few times a year descends into the 50s and 60s when he finds real stinkers.  Ralfy seems to limit his video reviews to 80 or better.  So, yes, it can appear as if many of the major reviewers are neglecting a big part of their grading range.  Smaller reviewers, including me, dip below 70 rarely as well.  But that has to do with selection bias.  We don't have access to thousands of samples.  And if we do drink something, we're more motivated to go with a potentially enjoyable choice rather than something Cutty Sark.  For me, whisky is the best thing I've ever drunk, so I'm still going to find positives in the mediocre products.  On the other hand, most of the vodkas and tequilas I've tried would never make it north of 70 points.

All of my grades from this site are available on The Big Whisk(e)y List page (which has replaced the Whisky Rankings page).  And to be fully open with my approach, I'm also including the mean grade, along with the standard deviation, and a comparable A-F grade list utilizing the STDEVs (located at the bottom of the list).  When I first did these calculations, I happily discovered that my bias wasn't too high grading-wise.  Maybe a point of bloat overall, but that will shift and adjust as my future scores roll in.  I hope to enjoy many lovely drinks this year, so perhaps the mean may rise.

You can think of my system in terms of school test graves.  There's about a point of a curve up right now, so a 90 is still a B+.  Also, I want to make clear that when I score something in the low 80s, the whisky is not bad.  82 is middle range for me, think Glenfiddich 12 or Strathisla 12.  When the score drops into the low 70s, then we're talking about a D grade; the whisky isn't a total failure but it's not easily recommended.  When the score heads into the 60s or lower, I'm concluding something went terribly wrong.  And a score of 80 still means I'd pay for it at a bar, if I can afford it.

Which brings me to the $$$ situation.  The grades are generated without taking price into consideration.  There are 90+ point whiskies which are priced totally out of whack, while there are 70+ point whiskies that don't cost much.  I'm not taking price into consideration in my scoring because I'm simply (or not so simply) rating aesthetics.  I don't feel comfortable listing a price-quality ratio/grade because the damn drink keeps getting more expensive every week.  But don't worry, I'll bitch about discuss pricing issues in almost every post anyway, usually right above where I list the score.

So there are two ways to view the 100-point system.

There are comparable letter grades:

98 to 100 - ???
96 to 97 - A+ : The stuff that dreams are made of.
93 to 95 - A : Truly thrilling.  Can make one ponder his or her existence.
91 to 92 - A- : If the style suits your palate, I do not hesitate to recommend this lovely drink.
88 to 90 - B+ : Grand stuff, if I don't already own it then it's because the money tree croaked
85 to 87 - B : Great whisky. It makes me smile when I see it on the shelf at a retailer or bar.
83 to 84 - B- : Solid, reliable whisky. Not without its flaws, but the positives out weigh 'em.
80 to 82 - C+ : Average. Pending the price, something I'd still order at a bar.
78 to 79 - C : Drinkable, but mildly disappointing as the bad starts to outweigh the good.
75 to 77 - C- : Not terrible, may be just for specific palates.
72 to 74 - D+ : Difficult to recommended. Either it's too sleepy or it's among the least of its type.
70 to 71 - D : Not recommended.
67 to 69 - D- : Considerably broken whisky.  Should avoid.
00 to 66 - F : Failure from top to bottom.

Or a more simplified...

90 - 100 - Excellent.  A lovely drink and something I would hope to possess for myself.
80 - 89 - Good.  Always reliable and I would be happy to barter some cash for the liquid.
70 - 79 - Drinkable.  I wouldn't turn it down, but I personally wouldn't pay for it.
60 - 69 - Dumpable.  Mediocre or worse.  Truly not recommendable to anyone.
00 - 59 - Disgusting.  Need I say more?

Please know that I recognize how goofy it can be to assign measurements to something as gorgeous as whisky, whether with 5-star or 100-point scales.  I hope these changes add to my reviews rather than subtract, because they're here to stay.  Cheers!