He’s baaaaaaaaaaaack.
Robin Goldstein, who shook the wine world’s foundations in 2008 when he won Wine Spectator’s restaurant Award of Excellence after creating a fictitious restaurant whose wine list included some of their lowest-scoring Italian wines in the past two decades (triggering one of the most heated public debates of the year in the wine world), is back.
With a vengeance.
Not that Robin’s disappeared since my last interview with him (which long-time 1WD readers will recall generated some very compelling debate – some of which, you will come to learn, influenced his latest project): he blogs regularly at BlindTaste.com, helped follow up the 2010 edition of The Wine Trials with The Beer Trials (a similar take on blind tasting ratings, applied to commercial beers), and has co-authored the new release The Wine Trials 2011.
Once again, I greedily devoured the results in my review copy of The Wine Trials, and just as in the 2010 versions, I found the them nothing short of compelling.
For starters, the consumers’ choices (for the most part) are very good bargain wines: take Dona Paula, Aveleda, Hugel, Nobilo, and Sebeka for examples.
Additionally, the blind tasting regimen for the trials (which once again pitted inexpensive wines against similar but much pricier brands) was enhanced with a bit more of the science behind them explained, and the results were similar to those in 2010: non-experts prefer less expensive wines, by a significant statistical margin.
Finally, Robin and his co-authors seem to take an even harder line in The 2011 Wine Trials against the use of point scores by leading wine publications, including taking Wine Spectator to task for how they handled the Award of Excellence kerfuffle in 2008. Whether or not you agree with their stance and their findings, the Wine Trials team at Fearless Critic Media are clearly not interested in backing down anytime soon.
Robin (once again) kindly agreed to talk to me about his controversial new release, and (once again) he has a lot to say about Wine Spectator, the 100 point wine scoring system, and how wine consumers can enhance their own perceptions (and use their own preferences to rally against snobbery in the wine world). Oh, yeah, and he talks RUSH!
Enjoy!…
1WineDude: Our interview back in January created quite a stir and some great discussion both in support and in criticism of the 2010 Wine Trials. Did the discussions and splash coming from the previous releases influence the approach to the 2011 edition?
Robin Goldstein: Well, for starters, the title of your post was certainly provocative. It’s definitely the first time I’ve been referred to as “Satan” (or “Savior”, for that matter) in a headline. Both in terms of the number of comments and the content, it was one of the most spirited debates I’ve seen since the flurry that followed release of our first edition in spring 2008. I love listening to (and participating in) debates about the purpose, value, and importance of blind tasting, as one of the central purposes of the first half of The Wine Trials was to encourage that kind of discussion.
One thing that surprised me was how some people seemed to think that the importance of context in our experience was an argument *against* blind tasting as a way of evaluating wine. If you taste blind, some people argued, you’re missing a crucial element of wine’s pleasure. (This point had also been made in the past by Eric Asimov–who almost always tastes blind these days, by the way–and others.) In fact, that’s precisely my point: context matters a lot, and you’ll get *much* more pleasure–not just a little bit–when you’re given positive information about the wine. That’s why we need to taste blind when we’re rating: so that context doesn’t bias critical quality assessments that have real impact in the marketplace.
The fact that people’s experiences of wine are so individual and subjective isn’t an argument *against* the blind tasting method; it’s an argument *for* it. Consumers are already subject to the forces of bias that the label, appellation, etc. create. The critic’s rating is supposed to act as a foil for those biases by providing an independent variable that evaluates the wine’s physical properties — an additional piece of information that we’re to use when making a buying decision. If that data point is subject to the same forces of bias that consumers are, then it’s no longer an independent variable. It loses its value in our buying equation. We should evaluate wine blind and drink it non-blind.
That discussion was part of what encouraged me to incorporate more specific scientific evidence for this point into the 2011 version of the “blind tasting manifesto” in The Wine Trials. It also helped me to better articulate my views on Robert Parker’s guide to wines under $25 in the book review that I published in the latest Journal of Wine Economics. All of this said, I think that among my friends who read the article, more people asked me about my comments on RUSH than any of my long-winded answers about perceptual bias and sensory experience. That’s either another testament to the fact that you are much better than I am at knowing what will attract readers’ immediate attention, or it’s evidence that, contrary to your argument, it’s actually you that’s Satan, and the members of RUSH once sold you their souls in exchange for the opportunity, while on Earth, to be superior to all other entities in the known universe, whether physical or intellectual. That’s also possible.
[ Editor’s note: at the advice of counsel, I decline to comment! ]
“The critic’s rating is supposed to act as a foil for those biases by providing an independent variable that evaluates the wine’s physical properties — an additional piece of information that we’re to use when making a buying decision. If that data point is subject to the same forces of bias that consumers are, then it’s no longer an independent variable. It loses its value in our buying equation. We should evaluate wine blind and drink it non-blind.”
1WD: What’s different between the 2011 and 2010 book releases? Which I guess is another way of saying “sell me on buying the new edition” :-)
RG: Wine gets a new vintage every year, so a wine guide needs one, too. In a way, it’s misleading for us just to call it a “new edition”–as far as the wine reviews go, it’s an entirely new book. The 175 full-page reviews in The Wine Trials 2011 (up from 150 in 2010) are based entirely on a blind tasting process that starts over every year. Although certain facts about the winery or bottle sometimes remain constant, every single review in this book is based on the tasting panel’s assessment of a new, different vintage of each wine in consideration. Wines vary vastly from one vintage to the next, and it’s a myth that inexpensive wines are immune to this phenomenon–on the contrary, they vary considerably, sometimes for reasons that are less related to climate than to a producer’s decision to change the style of winemaking in response to consumer demand trends.
As such, and also because of stiffer competition in the ever-more-crowded under-$15 market, less than half of the winners of the 2010 Wine Trials re-qualified for the 2011 edition, and even for those that did re-qualify, the 2010 book doesn’t describe the current release. It might help you do better than chance at the wine store now, but only The Wine Trials 2011 actually reviews the wines that are now available in the marketplace.
1WD: Have you received any death threats written using cut-out letters from magazine and newspaper articles but inadvertently photocopied onto
Wine Spectator letterhead?
RG: No death threats…but I still get a steady stream of thank yous for the role played by the Osteria L’Intrepido experiment in exposing the Wine Spectator Award of Excellence program’s misrepresentation of an advertising scheme as an exercise of expert judgment. I find it fascinating that two years later, Wine Spectator has still never apologized to its readers for this fraudulent awards program. They have never explained why the magazine would define as “excellent” a wine program whose high-priced “reserve” list was a catalog of the very worst Italian wines, as measured by Wine Spectator’s own ratings.
As I wrote in my response to Thomas Matthews’ post in the comments on your Wine Trials 2010 article: “Every wine on my reserve list was priced above US$100, every one of them was rated by *Wine Spectator*, and the average score of the wines on that list, in your magazine’s own judgment, was 70.6 points. Of the 35,498 Italian wines rated in WS’s database, only 239 of them–less than 1%–scored 71 or below. That would put the average wine on Osteria L’Intrepido’s “reserve list” safely in the bottom 1% of all Italian wines rated by *Wine Spectator*. Is that your magazine’s definition of ‘excellence’?”
Perhaps unsurprisingly, Matthews did not respond to that comment. Neither he nor anyone else at Wine Spectator has ever announced any changes to the judgment process, and the program seems to be continuing just as before. I think that, in retrospect, the editorial staff’s decision to try to smear me personally instead of apologizing to its readers and fixing the awards program was a poor one. At the time Wine Spectator people calling me a “mugger” guilty of “malicious duplicity,” restaurateurs and wineries everywhere were thanking me for exposing what they viewed as an extortion scheme. It was clear from the blogosphere and other coverage of the exposé that a large swath of wine writers, industry people, and academics found my results to be salient and the Awards of Excellence shown to be fraudulent, which made these charges of “malicious duplicity” seem a bit ironic. I think that many readers might have responded better to an apology by the editors and a pledge to change the program.
All of this said, I harbor no personal grudge against Thomas Matthews or anyone else at Wine Spectator. I know that Matthews was a former academic, and, from all accounts I’ve heard, an intelligent, mild-mannered guy. And it’s clear from his writing that he’s thoughtful and articulate. But he works for an institution whose journalistic ethics are seriously suspect. When large media organizations become too concerned with profitability, the people who work there can tend to forget the financial consequences that their actions have on the “little guys”–real financial consequences. Every time Wine Spectator rewards a big, well-capitalized restaurant for its ability to afford the $250 entry fee, or agrees to review a wine that’s been submitted by an advertiser when another similar wine from a non-advertiser goes unnoticed, the little guy loses just because he’s little–or because he’s principled and doesn’t believe in this form of media racketeering. I don’t have any reason to believe that the Wine Spectator are bad people, but it seems evident that there’s a certain peripheral blindness to the magazine’s editorial vision. Through the haze of their particular corporate culture, they seem not to see what is so obvious to so many: when a well-known critic accepts money from the entities he or she is purporting to criticize, it does a disservice to readers, consumers, and businesses.
“[Wine Spectator] seem not to see what is so obvious to so many: when a well-known critic accepts money from the entities he or she is purporting to criticize, it does a disservice to readers, consumers, and businesses.”
1WD: The new release’s Appendix (with details on the statistics behind The Wine Trials) isn’t easy reading. If you had to dumb it down to one or two sentences, what does the science behind The Wine Trails tell us?
RG: The appendix describes how we conducted, and analyzed the results of, our 2008 blind-tasting experiment that tested the connection between the price of wine and how much people like the wine. We used two different methods for the analysis (known as “ordered probit” and “OLS”), both of which revealed the same small but significant effect: on the whole, amongst the population at large, the more expensive the wine, the less people liked it.
This claim was counter-intuitive enough that we felt it would be helpful for those econ or stats geeks out there who were skeptical to look at our math a bit more closely. But we’re not under the delusion that most people read the appendix. Anyone who doesn’t like looking at regression models with Greek letters should definitely feel free to skip it; the methodology of the experiment is explained in plain English (in more detail than here) in the text of the book’s first couple of chapters.
1WD: Regarding luxury wine brands, The Wine Trials 2011 states that “The Central problem is that wine pricing is almost completely arbitrary… To accept the premises of this book and pay real attention to every glass you drink is to shift the burden back onto luxury producers to make a produce that actually differentiates itself.” Are there any wines that you feel are too damn expensive but that you drink often anyway because you just dig them?
RG: Yes, and I’d do it even more if I could afford to: old red Burgundy. There’s something magical about it. Its cost of production is vastly higher than most wine, but that factor can explain only a small fraction of the retail price. What can I say? It’s overpriced. I save it for special occasions, and I drink it less than old Italian wine, for one (very lucky) reason: in the late 1990s, when I was living in Italy as a teacher and travel writer, I bought and cellared a lot of wine from the late-’80s-to-mid-’90s that it would be impossible or prohibitively expensive for me to buy now. Every time I came back to the US to visit friends or family, I’d bring 15 or even 20 bottles back with me. (These were the days of carry-on liquids and 2,400 lire–84 euro cents–to the dollar.)
I was less confident in my wine preferences at the time than I am now, and I made some (in retrospect) poor buying decisions based more on “tre bicchieri” ratings from the Gambero Rosso wine guide than on a real understanding of my palate. But even the over-concentrated New-World-ish wines, made in a style that I’ve come to dislike in the years since, have softened enough over time that they now have some of those same beautiful, mulchy, flowery, tea-like aromatics that I love in old Burgundy, yet still retain a bit more power at the back end. Some of them are also totally oxidized, or cooked, or washed out with acetic acid, or just DOA. I haven’t been going through them fast enough, partly because I travel too much, and partly because I am possessed by a certain sense of loss every time I open one, like a reminder that I’m growing old.
But the guessing game of old wine is the fun of it, too. I think I have more of a taste for past-its-prime wine than most people I know. I have a lot of old stuff from Sicily that probably peaked 10 years ago, but I still like it — it’s dusty, barnyardy, soft, weird, and kind of awesome. I bought a lot of Sangiovese di Romagna in those days, because Emilia-Romagna was one of my Fodor’s regions for many years, and most of it is aging really well. Remembering where and when you bought such-and-such a bottle 15 years ago can be an exquisite pleasure–a temporary bout with the insane metaphysics of escapist wine geekdom. Imagine drinking blind the bottle of 1988 Gaja Barbaresco that my roommate Nikia bought for me as a going-away present! What a monumental waste that would be.
1WD: The Wine Trials doesn’t treat extravagant Bordeaux prices, or Robert Parker’s influence over Bordeaux wine prices, very fondly. In my interview with Parker, he seemed to agree at least on the Bordeaux prices (“With respect to Bordeaux, future prices have certainly gotten out of control. There was a real incentive to buy futures in a top vintage when I started 32 years ago because the prices were low, and if you bought the right wines, they obviously appreciated in value… The Bordelais have increasingly raised prices for their most prestigious estates. Except for the top wines of the great vintages, they really don’t appreciate much in price, so there is no point in buying them as futures and tying up your money two years prior to delivery”).
Why do you think Parker and WS have such price influence in the first place? Is it due to lazy marketing based on points, a lack of wine publications to challenge widespread use of their point reviews? Both? Something else entirely?
RG: It’s something else entirely–and this is one of the central issues that I tackle in the Parker book review that I mentioned above. (It’s a book review in the academic-paper sense only–that is, using a review of someone else’s book as a platform for making much larger points yourself.) I think the above quote reflects well on Parker, to whom I think I’m very fair in the review. The quote reinforces one of the central points that I make in that review–that it’s unfair to suggest that Parker has any designs on driving prices up. Here’s a clip:
“Many of the people within the wine world that have become increasingly disgusted with so-called ‘Parkerization’—the tinkering with a style of winemaking to bring out more fruit, more oak, and more alcohol in hopes of improving a Parker score—would paint the celebrated critic as a power-hungry dictator with designs on reshaping the wine world just to please his palate and fortify his wealth. But to adopt that view is to misunderstand the fundamental human mechanics of Parker’s vast appeal. Winemakers may feel obliged to please him, but consumers are under no obligation to follow him. If you want to understand Parker, look in the mirror.
Robert Parker is no dictator. He is a storyteller. The magnetism of his prose is that of J.K. Rowling’s, too: you’re first presented with a set of familiar facts and situations, and then, slowly, you’re seduced into suspending reason and believing in the perfectly impossible. Escape into a Parker review, and for a few sentences, there you are, back in junior high, the great critic’s palate—and yours, too—cured of its nagging mortality. In this counter-factual place, there is no perceptual bias, just perception. There is no confidence interval, just confidence. Parker’s 100-point wine is Gatsby’s green light, the orgiastic ghost of taste’s future, the tongue a sudden lattice of infinite resolution, the nose a sudden instrument of preternatural whiff.”
“Parker’s 100-point wine is Gatsby’s green light, the orgiastic ghost of taste’s future, the tongue a sudden lattice of infinite resolution, the nose a sudden instrument of preternatural whiff.”
1WD: Still no mention of RUSH in the Wine Trials? Honestly, what the hell are you waiting for already? I mean, they get passed up for the Rock & Roll Hall of Fame again and now *this*?!??
RG: While I will acknowledge that the Rock & Roll Hall of Fame snub is evidence against the Joe Roberts as Satan theory, I don’t think it’s *conclusive* evidence. It’s possible that LL Cool J and/or Neil Diamond were also working side deals with you.
[ Editor’s note: to the best of my knowledge, no evidence exists linking me to LL or Neil D… ]
1WD: Do you think The Wine Trials shows that wine scores are a dying breed, “Q.E.D.” style?
RG: Tough question. I think that one change that’s already happened is that wine scores are becoming more and more associated with nouveau-riche conspicuous consumption, and that segment is becoming more and more marginalized by people who are more traditionally sophisticated about wine–MWs, sommeliers, wine writers, wine bloggers, and such. In those circles, the rejection of 100-point-scale scores has become fashionable, and new status symbols, like knowledge about obscure regions and appellations, have emerged. (Going back to Eric Asimov, I think he’s representative of this new fashion, and I don’t mean that in a bad way–it’s a much more interesting fashion!)
At the same time, neuroscience research points ever more clearly toward the problems with sensory inter-subjectivity. I’ve spent the last three days at the fascinating Society of Sensory Professionals conference in Napa, where I’ve spent some time with Tim Hanni, whose work pushes us away from the outright dismissal of people who don’t like, say, wine with very high acidity and toward the segmentation of tasters according to their palate “phenotypes.” Tim’s work should be marginalized less and appreciated more as our understanding of interpersonal neurophysiological diversity advances, particularly as concerns sensory pathways that work with ion channels, like olfactory receptors.
[Editor’s note: read more about Tim’s work in his recent 1WD interview ].
All sounds great, right? The problem is that at the same time, wine collectors and drinkers in emerging markets, especially China, are loving Parker and WS scores more than ever–they’ve become mainstream conspicuous consumption icons. In that sense, the wine world is becoming more fragmented–between the new American-style “experts” who understand our sensory systems better than ever before, and new conspicuous-consumption markets popping up everywhere. Unfortunately, it may turn out that wine producers find that catering to the latter is far more lucrative than catering to the former, and thus will still continue to care about scores.
“Wine collectors and drinkers in emerging markets, especially China, are loving Parker and WS scores more than ever–they’ve become mainstream conspicuous consumption icons. In that sense, the wine world is becoming more fragmented–between the new American-style “experts” who understand our sensory systems better than ever before, and new conspicuous-consumption markets popping up everywhere.”
1WD: The Wine Trials has a great central message, I think, which at its heart is providing measurable data to support what many people know instinctively: they should be confident in asserting their own preferences and buying wines to suit, and not be afraid to try new things and expand their wine horizons inexpensively. However, it cautions that wine is not somehow above any other luxury good in terms of how it’s marketed, and that a lot of wine media may be a bit out of touch with the average wine consumer’s preferences. How would you recommend that those new to wine get started, and what resources they should use to get their wine knowledge?
RG: I couldn’t agree more. I’m a big advocate of blind tasting as a way of learning about wine — it trains your palate to discriminate between different styles of wine, and it makes your own preferences much clearer. Brown-bag six bottles of red, and test yourself on whether you can identify which wine is which. Throw two identical wines into the mix, and see if you can separate them out. Once you’re used to *identifying* wine, you’ll be naturally led to a clearer picture of your preferences, and you’ll start to feel less insecure about what you like and don’t like. Beyond The Wine Trials, I think Mark Oldman’s new book, Brave New World of Wine, is a good resource for beginners. I also like Jancis Robinson’s entry-level books. Beginners should also consider amateur wine courses. Tim Hanni does seminars and educational programs, and Karen MacNeil (author of “The Wine Bible,” another Workman title) is a great wine educator at the CIA Greystone in Napa. WSET courses (I’d skip the overly elementary Level 1 and start with Level 3), although they don’t focus enough on blind tasting, are another option. But there’s no substitute for tasting — a lot — at home.
[ Editor’s note: for more on Mark Oldman’s new book, see my recent interview with Mark; for more info. on the WSET and other wine certs, see the Learning Wine category ]
1WD: What’s next for Fearless Media and the Wine / Beer Trials?
RG: I’m working on an as-yet-untitled book on taste perception and the tricks our brains play on us when we eat and drink, and Fearless Critic has just released its Seattle restaurant guide, which will be followed by a San Antonio guide in November and Dallas in the spring. The Beer Trials will get an update next year, and next summer we’ll release a Liquor Trials book.
Cheers!
(images: amazon.com, blindtaste.com)
Joe, have your readers take a look at my book review on this: vinofictions.blgospot.com
Met Robin last December. It may not surprise many that I am almost in agreement with both the thesis and the results, with one or two reservations about presentation.
Thanks, Thom – with the interview we did on TWT 2010, there were quite a few readers who thought the tasting process itself had flaws, so will be interesting to see if they have changed their view with TWT 2011 or not. Cheers!
Fantastic interview. I think I love Robin – I found myself agreeing wholeheartedly with darn near everything he said. It's so nice to read an opinion that balances the constant parade of propaganda that's perpetuated by all wine magazines. Sure, those mags can give you some tips as to what to look for and what to avoid, but for the most part I think people would be better off investing the money they spend on buying wine magazines on actual bottles of wine, that they can taste and judge for themselves. Down with points scores! Wine is so much more than a number, and shame on anyone who chases bottles purely because someone says it's worth 90 points.
I also think that blind tasting is extremely worthwhile. It removes all of one's prejudices, conscious or otherwise, and it lets you really focus on actually tasting what's in your glass. I find it an extremely useful exercise.
Mel – "for the most part I think people would be better off investing the money they spend on buying wine magazines on actual bottles of wine, that they can taste and judge for themselves"
I'd never actually thought of it that way… it's a compelling statement! Sort of like it could be good for us to take breaks from those publications, focus on learning more about our own palates and preferences, and then armed with that knowledge, go back to the sources of wine media and chose those that align best with our outlook on wine.
Cheers!
Joe – I couldn't have said it better myself! Wine mags are very dangerous things for the novice wine drinker. It's too easy to be swayed by their supposed "authority" into subscribing to whatever brands happens to be advertised at that time. Much better for them to just start (blind) tasting an assortment of inexpensive bottles from a good local wine shop, and keep notes on their impressions of everything they taste. This is the advice I give to anyone looking to get into wine.
Thanks, Mel – I'd wish that I'd done it your way; I think reading the wine mags first set me back many months in terms of learning more about wine from a personal perspective. Nothing against them, they just were not the right tools for me to be using at the time!
I think the bigger problem is that with Parker, et al, is that you are getting one person's opinion, and that opinion is rooted in their preferred style. If you don't like their style, you're going to look at their ratings, taste the wines, and decide "this person doesn't know what the hell they are talking about." This is why panels of tasters are a better arbiter: you get more varied opinion, and whatever comes out on top should, at the very least, be a consensus.
As to the issue of price vs. quality, every month, we have a blind tasting of about 15-20 wines, most are in the $50-$75 price point. Since I do not have the discretionary income of most of the people involved, my bottle tends to be in the $30-$40 range. The wine I pick tends to do very well because it's a good wine, regardless of price. The only problem is when everyone finds out what I brought, they go out the next day and buy cases of it, leaving nothing for me. :(
Thanks, Richard! Two things:
1) The topic of ratings vs. preference is timely (for this blog, anyway) as it forms the basis of my first "Going Pro" article (publishing tomorrow).
2) Your bargain wine getting bought up and no longer being a bargain wine scenario is one that I am CONSTANTLY dealing with myself! :)
Cheers!
3. A detailed methodology of how the wines in the 2011 edition were selected and rated was once again omitted. The book does mention the tastings started with more than 500 wines and that was narrowed down to 175. How many consumers tasted these wines? How were the tastings conducted? How many wines did each consumer taste? What were the ages, genders, and place of residence of those consumers? Where were the tastings conducted? There are plenty of other unanswered questions that strike directly at the validity of selections in the book. The books presents the science surrounding the theory behind the book, but fails to go into specifics concerning the tasting methodology that led to the wines in the 2011 edition. The omission of that information calls into question the results.
a. As a related aside, the 2011 edition thanks the same 11 restaurants (as they did in the 2010 edition) where it conducted tastings. Those restaurants are in MA, CT, NY and TX. Are those the only places where these tastings were conducted?
Joe,
Do you agreement with the statement from Wine Trials 2011 that those who accept PR junkets have an obvious conflict of interest, and have an interest in "rating the products they're promotong higher so as to be invited on the next junket;.."?
Hey Richard – thanks, I was waiting (hoping & anticipating) that you'd chime back in on this!
Before I answer on the junkets, I'd like to comment on your comments (and it's *my* blog so I will! :-) :
The wine-with-food argument is a good one, I think. I have found myself giving much higher ratings to some wines that don't jump out at tastings, precisely because they are understated and have good acidity and I thought they would pair well with food. I think you're also assuming (not without excellent reasons!) that the blind tastings are marathon-like, palate-besieging events (that's been my experience with blind tastings, anyway). I was invited by Fearless Critic to be a member on one of the expert panels for TWT 2011 blind tastings, but was unable to make it due to my schedule. So, alas, I didn't get an opportunity to look at the tasting method and ask about the selection processes while they were happening. We'll need Robin to chime in on those aspects (and the others that you raised).
Regarding the junkets: I found some (I thought rather minor) contradiction in how TWT 2011 addresses the topics of advertising and junkets, in that they seem to reserve this more for print media than for blogs, though one could certainly argue that for now the stakes and dollars involved in those things are much, MUCH higher for print than on-line media. But that is probably going to change in the future, or at least that's the impression I get.
The media world has changed and the belts have gotten tighter. We now have situations where bloggers AND journalists writing primarily for print cannot afford to travel to wine destinations, events, etc. In those cases, the coverage would then only be limited to those who could afford it and local coverage, neither of which may be totally in line with the needs of some wine markets who are looking to gain notice in markets not in their locality/country/etc.
So, we have a growing amount of examples where no junket = no coverage, and that's bad for wine regions, journalists, and readers.
So I don't agree with TWT 2011 assessment as you've presented it above because it assumes that 1) the stakes are high for all those accepting junkets (I'd argue this is only the case for the largest of wine publications), and 2) journalists are "addicted" to the junkets.
Look, I've biked along the Mosel river in amazing weather and didn't pay for it, and had countless other amazing experiences like that, thanks to junkets. I've also panned some of the trips and some of the wines – and in those cases, I wasn't worried about whether or not I'd get blacklisted for junkets, because FOR ME (cannot speak for all, of course) it matters more what value the readers of 1WD are getting,and it's less about how much travel I get to do.
I view my approach as pretty even-handed, and I've actually been INVITED BACK to the places that I've panned, and have had to turn down dozens of invitations for junkets after those articles. So from my perspective there is no fear about whether or not I will or won't get the next junket (I also request in writing that those offering the junkets confirm their understanding that I may not write about the trip and if I do it may not be positive – which one would think would cause the less scrupulous offers to be rescinded anyway).
I've also had rather vocal arguments with PR folks who after agreeing to my terms have called me demanding to know where coverage of brand so-and-so is, to which my response is almost always "you just killed it, you moron!" :)
All of this is a very long-winded way of saying that you cannot ignore the potential of conflict of interest in those situations, but you CAN be transparent and seek as much as possible to minimize it. Ultimately, the readers can decide if I got the balance right or not.
Cheers!
Joe,
Your last paragraph sums up the issue well, and coincides with my own opinions concerning junkets and potential bias. I do agree with you that they are needed to expand coverage, but we just have to be careful how we handle it.
Richard – yeah, *very* careful. I've lost count of the number of times I've felt uncomfortable on junkets due the effusive behavior of my travel-mates. In almost all of those cases, the others on the junkets weren't promising any coverage, either, but I felt their behavior might have been setting unfair expectations of that to the PR folks and the wine producers. I mean, when someone says the wine is amazing and that they could write five stories about the winemakers' property alone… what would YOU take away from that as a producer? You'd think you had a new best friend for positive coverage! :)
And one last comment (I hate the error msgs that my comments are too long on this system).
Do not the authors of the Wine Trials have a conflict of interest? They have a financial interest in the publication of this book, so doesn't that potentially taint the tastings they do for this book? If the higher cost wines actually beat out the lower cost wines, they would not be able to sell their book. Might they skew the results just so they can sell more books?
Sorry Richard – not sure how to fix that (and it seems to be happening only with people commenting from behind firewalls) but am looking into it! :-(
Richard,
I believe that you're confusing two different things here: first, there's our 2008 "Do More Expensive Wines Taste Better?" experiment, published in the Journal of Wine Economics. The experiment was conducted two years ago, so obviously, the results of that experiment (and the appendix in which the experiment is explained in greater detail), and its implications as set out in the "blind tasting manifesto" that forms the introductory portion of the book, don't change from one year to the next! I do modify the manifesto when new information or results come out that require mention, explanation, or modification; those changes are integrated into the manifesto text and also summarized in the "New Experimental Results" section, which I do precisely so that people (like you) who have read previous editions and don't feel like reading through all of the arguments again can read only that chapter and get a quick summary of what's new in the world of behavioral wine experiments. It's the second two-thirds of the book — the "Wine Trials" — that is completely rewritten every year based on the new blind tastings. That's the section that reveals the top wines under $15 from our blind tastings of the newly released vintages.
On travel junkets: we don't accept them. With all due respect to both of you, I don't agree that they're necessary to review wines.
On "skewing the results" to include more wines: The number of wines to be included in the book isn't determined until after the tastings are completed. If fewer wines qualified, we would simply include fewer wines in the book. In the past three editions, we've actually had the opposite problem: we're limited by space constraints to including a certain number of wines. Either way, there's no conflict of interest, and there's no evidence of any direct connection between the number of wines recommended and the number of books we sell. On the contrary, we've gotten many comments from readers that they appreciate the book's conciseness and the fact that we are rigorous about limiting how many wines we recommend.
Robin Goldstein
Robin – thanks as always for chiming in!
I appreciate the clarification on the number of wines included. Regarding junkets, I see that as a separate though related topic: are the necessary to review wines generally for the public? Probably not. Are the necessary for coverage about wine and where it comes from, how it's made, etc., to help connect readers closer to those things? I would argue the answer is "absolutely."
Cheers!
Robin:
You have not addressed what I consider the most important parts of my previous comments. And I am not confusing the 08 experiment and the rest of the book.
–My #1 comment was more factual, simply clarifying what was new and what was not in the book. Your first paragraph addresses my point too.
–You failed to comment on my #2 point, concerning blind tastings and food.
–You failed to comment on my #3 point, concerning the missing methodology for how the wines were selected and rated, as well as all of the questions I asked in that point.
–You failed to comment on my point #3a, concerning the restaurants where tastings occurred.
–On your second paragraph, the junket issue, that was more directed at Joe. And though maybe essential, they can provide some invaluable information and experiences.
–On my final point, concerning a potential conflict of interest, I thnk you misunderstood my point. It has nothing to do with including more wines, so your comments don't address the issue. My point has to do with a potential financial motivaton to skew the results of your wine trials. It is a fact that you have a financial interest in the sales of this book. Now, if the results of the trials did not go the way you wished, if the more expensive wines beat out the lower priced wines, that would defeat the entire premise of your book. So, you would probably sell less books, if you even published the book, and make less money. So, there is a potential incentive for you to skew the trial results to ensure they conform to your premise. I am certainly not alleging that is what has happened. But, it seems to me to be very similar to the conflict of interest issues you raise in the book, such as for junkets and advertising.
One point I'd like to make, about tasting wine with food: most critics who taste wine and don't do it blind, also don't taste the wine with food for evaluation. So what's the difference?
The difference is that tasting blind removes clues and bias concerning the identity of the wine and that releases tasters to focus on their palate and not on what to expect from the wine.
While wine is a food item–it is to me, and that's how I use it–but wine is also an entity on its own, and trained people can and do evaluate it on that basis. If you want to taste wine with food for evaluation, nothing stopping either critics or Goldstein from doing it, but that isn't the particular agenda either of the Trials or of critics, as I understand it.
In short: bringing up the food with wine conversation is a separate issue and it seems like a straw man at the same time.
Thanks, Thom – I'm not sure I'd totally separate that discussion, as there's no question the acidity level, balance, and other related factors play a huge part in reviewing wines for me. But having said that, we do need to acknowledge that the logistics for that kind of evaluation are often, out of necessity, totally different than big blind tastings.
Hi Thomas:
The issue is that average consumers, not "trained people," were the ones evaluating the 175 wines in the book, assessing them as better than more expensive wines. In their tastings, these average consumers would not be extrapolating how a wine might pair with with food.
Can Robin elaborate on his idea that Wine Spectator might review a wine submitted by an advertiser, and decline to review a wine submitted by a non-advertiser? Is he aware that wines from Michigan, and the Niagara Escarpment, and Arizona, and other ostensible backwaters get reviewed all the time by WS? And that the only magic way to get a review is to… submit the wine? I'm serious. I've spoken to small producers in tiny appellations who will say, "I had no clue that all we had to do was submit the wine to get it reviewed. We thought they never would." But maybe Mr. Goldstein has more information on this and will share it.
Great questions (as always), Evan – hopefully we can get Robin to chime in on those.
Sorry for taking so long to respond to this, but thanks for your comments. I've also spoken to small producers who have had a lot of trouble getting their wine reviewed. Since Wine Spectator won't be transparent about "what it takes," it's hard for me to speculate on that.
Evan,
You might also ask Robin if any New York or wines from states other than the West Coast, were included in the blind tasting. It appears not, but I don't know for sure.
Joe,
I agree that food and wine evaluations are valuable Still,I can't imagine why it is a must that they be conducted in order to evaluate strictly a wine preference, unless the evaluators have no wine training. Wine is an entity and it can be evaluated as such.
My overarching point, however, is that the major critics and magazines don't include food in their wine evaluations, so why bring it up as an issue for the Wine Trials?
Understood, Thom! :)
Thomas:
As I mentioned above, these Wine Trials did involve average consumers, without wine training. The initial winnowing of the wine entries may have been done by trained professionals, but the second stage was average consumers.
Robin Goldstein was and still is spot on by clowning the fools who offer such silly "awards" and the jokers who partake in such an archaic and antiquated system that comprises the 100 point rating system. I knew I was in bazarro world the moment I received a huge score for one of my zins and then a sales pitch and ad "opportunity" very shortly thereafter from the corporate wine peddler who gave this crappy award.
Why don't reviewers who like a wine USE THEIR WORDS instead of simply throwing out a number and a few adjectives.
Wineries should place the responsibility of marketing on an inside person rather than WS, WE or RP. These pub's are located back east… What the ____ do they know about grape growing or wine making? Exactly… they don't. They have no real connection to Wine Country.
Other than Steve H., who does at least make himself available on the blogs to defend his corporate handlers, why aren't the guys in Q out there on the blogs engaging us?
This whole is crazy. If I were the bank loaning the number monkeys $ for the next season, I'd tell them to go and get me some real business plans, one's that involve more than, "we'll we hit some great scores this year and plan to make the same style next year so the raters should love it… now please extend my LOC."
Randy – thanks for chiming in, I often find myself asking (when visiting wineries / wine regions) what winemakers think about scores. The answers vary quite dramatically, and your comment here, while probably falling pretty far on one side of the continuum, is a good example of the answer that many have given to me privately or probably wish they could say publicly.
Having said that, and while I'm not a fan of them myself, scores must provide some value to consumers (even if the system isn't totally transparent about all of the potential negatives) – they're still around, after all, and no one has come up with a consumer-friendly alternative (yet).
My view is that scores make things too easy, to the point that they kind of dumb-down the consumer and therefore the industry as a whole who ultimately need to give consumers things that they want and will buy. Wine *can* be made accessible without dumbing it down too much, but it's a slower-moving process to educate than it is to abuse the score concept (which was established to give a then-much-needed shot in the arm to the average wine quality/price ratio) – which is what is happening now I think in marketing and POS.
I think that TWT, while not perfect, is a step in the right direction away from dumbing down the situation for consumers, and acknowledging that people who buy wine are smart enough to establish their own opinions and preferences.
Cheers!
Richard,
I don't understand your position at all. If professional reviews are generally done without food, and average consumers accept those reviews, then it seems the standard of without food is consistent even when average consumers review for preference. The complaint should be that the pros don't review with food.
As I've said, wine is an entity–it can be evaluated as such, and in fact unless the evaluation is specifically attached to food, wine should be evaluated as an entity.
Richard,
To make it clear, which obviously I haven't done yet: What I mean by "trained" are the pros, the ones that don't generally evaluate wine with food. They do it all the time, and consumers use that information to buy wine. So why complain about Goldstein's system that has consumers evaluating without food–the same evaluation that they would get from the pros.
Incidentally, I do believe that a vast patch of wine consumers don't pair wine with food every day.
Scores reflect the palate of the scorer — as a consumer, we need the reviewers to be consistent and to accurately articulate the wines characteristics. We as consumers then need to taste such scored wines to see if we agree and to callibrate our palates (I like when James Laube tastes "cassis" and do not like when James Molesworth tastes "garrigue" regardless of the scores). Then we can use the scores and tasting notes as one of the factors to guide purchasing decisions. The benefit of this for me is that they taste enormous numbers of wines that I cannot ever hope to try–thus opening whole new worlds of wines to me (for example Fingers Lakes Rieslings etc, etc and from both small and large producers). Assuming that WS does what it says it does in its tasting, then there cannot be a possibility of commerical bias. The real story would be to determine if they really do not follow protocol — that would be news. Otherwise, I think it is time to shut up about this…scores are what they are and they are not going anywhere because they work (as noted above)–if you do not like Laubes or Parker palate, then do not buy the wines they reccommend
Andy,
True–but Goldstein's thesis isn't about the scoring system itself, but about how that system represents an aspirational rather than a preference-based marketing concept.