...where distraction is the main attraction.

Sunday, January 5, 2014

The new grading system

After 18-20 hours of editing posts and spreadsheets, I think it's time for me to roll out the new grades.

When I first started the Single Malt Reports almost 2 1/2 years ago, I didn't have my own history of rating whiskies to reference.  Either it was excellent or good or drinkable or dumpable.  Though that's still sort of the basis of grading whisky anyway, I didn't have much of a system in place.

I'd spent the previous 15 years of my life privately grading films.  That system started with a five-star structure, but then expanded out into a number of spreadsheets, partially because I love numbers and partially because the sheer quantity of films allowed me to compare and contrast and thus produce a more honed 100-point system.

A week before I posted that first Single Malt Report, I figured out how to create stars and 1/2 symbols via HTML.  That was fun, so I used those.  After a little while, I had the brilliant stupid idea to actually rank the stuff I was drinking.  That's fine with 50 whiskies or so.  At 200 it got to be too much of a pain in the ass.  Plus that at level of exactitude, 200+ separate "scores", it stopped making sense.  After whisky #200, I knew it was time to start transitioning to a 100-point scale.

Michael Jackson had a 100-point scale.  Ralfy has a 100-point scale.  Robert Parker has a 100-point scale.  Jim Murray has a 100-point scale.  Whisky Advocate has a 100-point scale.  Serge has a 100-point scale.  But they're all different 100-point scales.  Parker assigns 100 points to wines rather regularly, to the extent that a 90 point Cab sounds like it's crap.  Murray's faves score at 97.5 each year.  Jackson didn't go above 90 often and neither does Ralfy.  While Serge does go over 90, he had only 3 out of 1000+ whiskies scored at 94 points or more in 2013.

There's also been a bit of discussion about whether reviewers really utilize the entire 100-point scale or if they just park themselves in the 80-90 range, using 10% of the possible grades.  I can understand that gripe.  Whisky Advocate rarely grades any whisky below 80.  Serge often drops into the 70s and a few times a year descends into the 50s and 60s when he finds real stinkers.  Ralfy seems to limit his video reviews to 80 or better.  So, yes, it can appear as if many of the major reviewers are neglecting a big part of their grading range.  Smaller reviewers, including me, dip below 70 rarely as well.  But that has to do with selection bias.  We don't have access to thousands of samples.  And if we do drink something, we're more motivated to go with a potentially enjoyable choice rather than something Cutty Sark.  For me, whisky is the best thing I've ever drunk, so I'm still going to find positives in the mediocre products.  On the other hand, most of the vodkas and tequilas I've tried would never make it north of 70 points.

All of my grades from this site are available on The Big Whisk(e)y List page (which has replaced the Whisky Rankings page).  And to be fully open with my approach, I'm also including the mean grade, along with the standard deviation, and a comparable A-F grade list utilizing the STDEVs (located at the bottom of the list).  When I first did these calculations, I happily discovered that my bias wasn't too high grading-wise.  Maybe a point of bloat overall, but that will shift and adjust as my future scores roll in.  I hope to enjoy many lovely drinks this year, so perhaps the mean may rise.

You can think of my system in terms of school test graves.  There's about a point of a curve up right now, so a 90 is still a B+.  Also, I want to make clear that when I score something in the low 80s, the whisky is not bad.  82 is middle range for me, think Glenfiddich 12 or Strathisla 12.  When the score drops into the low 70s, then we're talking about a D grade; the whisky isn't a total failure but it's not easily recommended.  When the score heads into the 60s or lower, I'm concluding something went terribly wrong.  And a score of 80 still means I'd pay for it at a bar, if I can afford it.

Which brings me to the $$$ situation.  The grades are generated without taking price into consideration.  There are 90+ point whiskies which are priced totally out of whack, while there are 70+ point whiskies that don't cost much.  I'm not taking price into consideration in my scoring because I'm simply (or not so simply) rating aesthetics.  I don't feel comfortable listing a price-quality ratio/grade because the damn drink keeps getting more expensive every week.  But don't worry, I'll bitch about discuss pricing issues in almost every post anyway, usually right above where I list the score.

So there are two ways to view the 100-point system.

There are comparable letter grades:

98 to 100 - ???
96 to 97 - A+ : The stuff that dreams are made of.
93 to 95 - A : Truly thrilling.  Can make one ponder his or her existence.
91 to 92 - A- : If the style suits your palate, I do not hesitate to recommend this lovely drink.
88 to 90 - B+ : Grand stuff, if I don't already own it then it's because the money tree croaked
85 to 87 - B : Great whisky. It makes me smile when I see it on the shelf at a retailer or bar.
83 to 84 - B- : Solid, reliable whisky. Not without its flaws, but the positives out weigh 'em.
80 to 82 - C+ : Average. Pending the price, something I'd still order at a bar.
78 to 79 - C : Drinkable, but mildly disappointing as the bad starts to outweigh the good.
75 to 77 - C- : Not terrible, may be just for specific palates.
72 to 74 - D+ : Difficult to recommended. Either it's too sleepy or it's among the least of its type.
70 to 71 - D : Not recommended.
67 to 69 - D- : Considerably broken whisky.  Should avoid.
00 to 66 - F : Failure from top to bottom.

Or a more simplified...

90 - 100 - Excellent.  A lovely drink and something I would hope to possess for myself.
80 - 89 - Good.  Always reliable and I would be happy to barter some cash for the liquid.
70 - 79 - Drinkable.  I wouldn't turn it down, but I personally wouldn't pay for it.
60 - 69 - Dumpable.  Mediocre or worse.  Truly not recommendable to anyone.
00 - 59 - Disgusting.  Need I say more?

Please know that I recognize how goofy it can be to assign measurements to something as gorgeous as whisky, whether with 5-star or 100-point scales.  I hope these changes add to my reviews rather than subtract, because they're here to stay.  Cheers!

2 comments:

  1. I must say, I really liked the 5* system. Why? It's simple and direct. The well-known flaws of the 100pt (or in your case the 97pt) system are that in the 1-79 range the points don't mean a great deal, but then in the 80-89 range they all crowd up. So a difference between 75 and 80 is not the same as a difference between 85 and 90. I get it that you want to be able to rank an 87 vs. 88 - that's why you do this, right? But I'm not sure why that's important. Are you sure that on a different day your 87 is not an 88 and vice-versa? I started with the 100pt scale but inspired by you and others I switched to the 5* scale. This feels much more comfortable to me.

    Also, looking at your letter grades, they seem to be biased downwards. If I give a student a B I can't tell them that they did "great", nor a B+ that s/he is "grand".

    Finally, how do your stars translate into points and grades?

    This is what I had used in my notes for scale translation, you'll see it differs from yours. The descriptions are inspired/stolen from LAWS:
    5* A+ perfection 98-100
    4.5* A superstar 95-97
    4* A- excellent 90-94
    3.5* B+ great 85-89
    3* B very good, might want 80-84
    2.5* B- good, worth trying 75-79
    2* C fine to drink 60-74
    1* D unpleasant 50-59
    0* F horrific 0-49

    ReplyDelete
    Replies
    1. Sorry for the delay in reply. There was a kink in the travel plans. If you've heard about the hundreds or thousands of flights cancelled due to the Midwest storms, well, five of those cancellations were ours. We just arrived home six days late. Now that I'm sitting in the comfort of our dining room with a cocktail of my own devising in hand, I'm going to attempt to address your valued comments.

      Part of the change in the grading system is due to an obsessive need for more granularity, yet not something so micro that I'm delving into decimals or ranking every single whisky against each other. That was a selfish emotional need. The more fact-based catalyst -- which was influenced by how I wanted to communicate my conclusions to the reader -- is that as whiskies began to crowd the three star range there was considerable variety of quality began to form within that group. For instance the 3-star Old Pulteney 12 was much closer to the quality of 3.5-star BenRiach 12 than the 3-star Bowmore 15. Or to take a recent example, the K&L Bowmore received a 3 stars, while the infamous Ledaig Blackadder received 2.5 stars. But the K&L Bowmore was much closer quality-wise to the Ledaig and other 2.5ers than many of its 3star brethren. Meanwhile, I gave BenRiach 12 3.5-stars, the same score I gave BenRiach 21 Authenticas, but I wanted to recommend the Authenticas so much higher. Thus I desired a way to chop things up finer, and not just for my own OCD needs, but also to communicate that some whiskies within the same star rating were better than others.

      As I'm sure you can relate, I'm not kidding about my feelings about the quality and potential of single malt whisky when compared to other distilled (aged or non-aged) spirits. This liquid frequents a profound sensory territory rarely tread by other spirits. So it's true there's a bit of a grouping in the upper 80s. If I had the means, then the mean score would probably find its way north of 85. And that is related to the selection bias I referenced (and also mentioned in some of the comments on MAO's site). If I actually dug into all of the blends and cheap(ish) bourbons, I believe there would be considerable grouping in the 75-80 range (and that's me being optimistic). That may give my numbers some more stability -- because I do believe there is a difference in quality between Kilbeggan and Jameson. But because I've fallen liver over heels for Scotch single malts, my choices have been leaning towards towards the drinks that lead me to lyricism.

      My star system actually only ran from 0.5 to 4.5 due to optimism. I believe the best of the best and the worst of the worst are still out there for my finding. Otherwise my scale ran similar to yours in the upper regions; it's in the lower half that we differ. I'll definitely give your scale a thought, but I can't say that something that scores under 70 is something I want tell folks that it's fine to drink. Perhaps my detailed letter grade descriptions are problematic. Or because I believe there are so many excellent whiskies out there, something great can still score only 86. That, of course, leads back to your comment about the difference between 87 and 88, which in turn leads me back to my comment that there can be granularity at the lower levels as well. Again, I'll think about my detailed letter grade descriptions. But I'm much more secure with my "simplified" descriptions.

      This is a difficult place where objectivity, subjectivity, emotion, and statistics meet. I knew this would get me into hot water with real numbers people.

      Delete