Wine Adventures

Wine Ratings -- Pros vs. Amateurs

Are you a wine critic?  Probably.

The wine world is a big place, and it churns with dozens of issues that drive people crazy.  Of all the topics that are grounds for spirited (and sometimes bitter) debate, the issue of wine ratings is right at the top of the list.  Do they matter?  Do those point scores really steer less-knowledgeable consumers toward “better” wines?  Are they fair?  Accurate?  Who does the ratings?  Are they qualified?

Major national wine consumer magazines, such as Wine Spectator, Decanter, and Wine Enthusiast, have panels of editors who taste wines blind and bestow scores on them that will either propel the bottles to greatness (and elevated prices) or doom them to the bargain basket.  There are independent critics, as well, such as Robert Parker, James Suckling, and Steven Tanzer, to name only a few.  Consumers have turned to people like Parker and others for many years, hoping for guidance in their wine selections.

How are ratings determined?  And what difference does it make if a Wine Spectator critic gives a wine 85 or 95 points?  How legitimate are their critiques?  To answer that, the news website Vox.com recently conducted a fascinating study that compared wine ratings by a wide group of critics to ratings of the same wines by normal ordinary people.

Who were those ordinary people?  Vox.com researchers accessed thousands of reviews posted on a website called CellarTracker.  Wine lovers and collectors subscribe to it so they can upload their purchases and track their bottle inventory.  The site, established by former Microsoft group program manager Eric LeVine, allows members to post evaluations and reviews of the wines they drink, and, of course, assign point scores.

CellarTracker is important to this divisive issue because the research was no casual, anecdotal study.  The Vox people developed complex algorithms, created detailed graphs and charts, and plotted the compared scores of over 10,000 wines given by professional wine writers against those of ordinary normal CellarTracker people (like me) who have a few hundred bottles in their houses.

Wine.  It’s one of the largest food/beverage categories on the planet.  And unlike breakfast cereal, for example, the range of prices is bewildering, if not staggering.  You can pick up a bottle of Oregon Pinot Noir for around $9, or a bottle of Pinot Noir from Burgundy for $300.  And sometimes, the point scores assigned by critics will be scarily similar across price points.  The fact is, many times there is little correlation between a wine’s price and the points it receives from influential critics. 

An interesting point about when the write-ups and ratings are issued.  A critic rates a wine from a barrel tasting, or when it’s first released.  So if I look at a rating of a 2005 wine, the information is 10-12 years old.  Somebody on CellarTracker may have tasted that same wine this year, so they’ll write about how it’s drinking today…not ten years ago.

It’s only recently, with the advent of the Internet, that casual wine drinkers and collectors have been able to voice – and publish – their opinions and reach a sizable audience.  This has become extremely important in the restaurant industry – especially at restaurants that are known for their wine selection.

 “The information is out there,” says Peter Hyzak, wine manager at Ruth’s Chris Steakhouse in Florida.  “Guests in the restaurant will scan our wine list, take out their cell phones, and do some research before they order.”  He notes that guests scan online information for three reasons.  “First is name recognition, which plays the biggest role in restaurant wine sales.  Caymus is Caymus, and everybody knows that wine.  Second is the vintage.  People research the ratings of the vintage of a wine we’re offering.  Having the best-rated vintage of any wine is absolutely imperative.  And third is price.  The retail prices of wines are easy to find on the Internet.”

Predictably enough, professional critics have consistently knocked ratings done by amateurs.  They have no formal training, they maintain.  They lack expertise.

Turns out not to be true.  The correlation between wine ratings from the influential (and highly respected) Wine Advocate group of critics with the average scores given by CellarTracker members was surprisingly close.  Some were even identical to the professional score.

However, the study revealed an even more startling fact.  There was a much larger difference in the scores among the professional critics.  In rating the same wine, prestigious wine writers were often three to four points apart in their evaluations.

So what can we learn from this?  First, the study indicates that when enthusiasts like you and me rate wines, we agree with the experts a lot more than they agree with each other. 

Just one more thing.  There is virtually no correlation between the price of a wine and the ratings assigned by critics.  Every month, this is proven by all the major consumer wine journals.  The wines are tasted blind, so evaluators don’t know who makes the wine or what it sells for.  Case in point:  a recent issue of Wine Spectator listed tasting notes for a wide range of Spanish wines.  One Ribera del Duero received 95 points and sold for $505 a bottle.  Another received 94 points and sold for $30.  Which would you buy?

Of course, the professional wine ratings do have a place.  Wine shops always post ratings (when they’re good) for the bottles on the shelf.  They do serve as a guide for wine buyers who are not all that knowledgeable

So, if you’re looking at a wine that you’re not familiar with, a rating by…someone…will give you at least a bit of guidance.  Buy a bottle, take it home and try it.  If you like it, go back and get some more.