About ten days after the Court of Arbitration for Sport announced their decision in the Floyd Landis case, I had the chance to speak for almost an hour with Mike Straubel, the head of the Sports Law Clinic at the Valparaiso University Law School. Valpo’s Sports Law clinic is, to the best of my knowledge, the only clinic of its type in the United States, dedicated to representing athletes who need legal assistance but who don’t have the financial resources to hire an attorney. Among other things, their clients may be embroiled in defending themselves against doping allegations. Professor Straubel and his team of law students have represented a number of athletes, most notably LaTasha Jenkins. They are currently the only lawyers who have actually successfully defended an athlete (Jenkins) in a doping arbitration case prosecuted by the United States Anti-Doping Agency.
In this first part of the interview, Professor Straubel talks about the Landis case. In the the second part, to be published in the near future, Professor Straubel answers questions about other cases and about the how the current anti-doping adjudication system works, and how he would change the system. Given the timing of the interview — just days after this year’s eventful Tour began and all that’s come up before and during the Beijing Olympics — it’s taken me more time than I would have liked to post the interview. And so, we begin…
DR: When I first read the Landis decision, I thought, “Boy, this is a really strongly worded opinion.” It really seems like [the CAS panel are] taking Landis’ legal team to task.
MS: They certainly believed all of USADA’s/WADA’s experts and none of Landis’ experts. And it sounds as if not only the legal strategy and the way in which Landis’ lawyers made their arguments, but the way in which the experts … were aggressive [in their testimony and statements for Landis] backfired, too. I think there was one section of the opinion where the panel took the experts to task for being a little too partisan and a bit too much advocates rather than experts.
DR: It did seem that way. That didn’t seem like that was a typical response, so it left me wondering what, exactly, had gone on in the hearing room. Especially given that during the original hearings, the AAA panel didn’t take them to task for that. Or, at least, not as harshly as this panel did.
MS: No, in fact the first panel in a few key situations sided with Landis, and found some violations of international standards, characterized the lab’s work as sloppy, and this panel didn’t do any of that.
DR: That’s kind of what surprised me. I know that the reputation of Landis’ legal team is that they’re supposed to be some pretty good lawyers. So I would expect that, at the very least, they might land one or two punches. Maybe not significant blows, but I was stunned that really the only thing that they seemed to connect on was pushing back on USADA’s attempt to have the suspension start in September, rather than in January of last year.
MS: Right. It’s hard to tell, but one possibility is that these are American lawyers who sometimes are taught or find that being aggressive works. The arbitration panel has some Europeans, some non-Americans and they may not have taken so kindly to the more aggressive style of American lawyers.
DR: If I recall, Jan Paulsson is from France/Scandanavia and David Williams is from New Zealand.
MS: Right. I think David Rivkin was the other one.
DR: He was the American on the panel.
MS: Yes. I’ve done one arbitration hearing with him. And he was very, very careful, precise and cordial and gentlemanly in the way he conducted the hearing as the arbitrator. So, he may not have taken kindly to more aggressive tactics, also.
DR: So, one thing I’ve been thinking about since the verdict came down is that that Team Landis’ strategy didn’t connect well with the arbitrators. And the way the panel’s decision reads suggests that Landis’ lawyers may have made some rather large strategic errors in how they approached this case. Do you think that’s a fair characterization?
MS: Generally, yes. I guess it goes back to overall being aggressive and particularly asserting that there was some, outright asserting that there was fraud, conspiracy and attacking some of the witnesses – either directly or indirectly in that way. With non-American arbitrators you need to be a little more indirect in saying that, rather than fraud was committed, that, you know, some accidents, missteps took place that resulted in errors in the results. These kinds of simple ways of characterizing might have been more effective.
DR: So, they might have been better advised to have been a bit more diplomatic is what you’re telling me.
MS: Yes.
DR: And, it seems like that’s a fair point, because, unless you have actual proof of fraud, that’s a pretty serious charge to make.
MS: The panel said a couple of times that there was absolutely no evidence of wrongful doing or fraud or anything that is akin to intentionally doing that.
DR: Unless somebody came forward and said, “We fabricated this document, and we did it because USADA told us they needed a document that said ‘X’.” I don’t know how you’re actually going to prove fraud.
MS: No, you’re not. You know, the discovery and rules of evidence in these proceedings are pretty minimal. It’s not like the discovery that’s permitted in a US court.
DR: It seems like this is rather far removed from what us laypeople think of as justice.
MS: [laughs] Yeah, … the rules are softer, and there’s a lot more leeway in the way things can be done. But the arbitrators have a lot more control over the process than perhaps judges do in US courts.
DR: It seems to me, in reading how the CAS accepted the arguments from USADA, that they’ve really shut down the types of ISL violations that could be shown, on the athlete’s side, to overturn or to contradict an adverse analytical finding. In the case of LaTasha Jenkins, if I understood that case correctly, it was the same kind of thing as in Landaluce, where a person running some of the B sample tests had been involved in the A sample.
MS: Right.
DR: And it seems like the only things an athlete is going to be able to use to refute a doping charge is something that obvious. They’re not going to be able to go in and say, “Look, you know, the ISL says you should run this particular test a certain way” because the sense that I got from the Landis decision is that the technical documents and the ISLs are a little squishy in that regard. They said something about, in terms of chain of custody, as long as long as they maintained or followed the concepts, rather than the actual written material, then that was OK.
MS: Yes. They said that the concepts in the technical document by WADA – if they follow the concepts and not the precise wording of it. That does allow for a fair amount of play. … That was troubling, because they said that as long as you followed the principles of chain of custody, that’s OK. And then they placed; they specifically placed the burden on the athlete, Landis in this situation, to show that the lab tech had actually tampered with the sample during any potential breach in the chain of custody.
Well, the purpose of the chain of custody rule is to sort of create an ironclad proof of integrity. And [problems with the chain of custody] – that’s [information that is] almost impossible for an athlete to obtain. You’re not going to find a lab tech who’s going to step forward and say, “Yeah, I tampered with it.”
The World Anti-Doping Code and the International Standards limit the amount of discovery and information that the athlete can obtain. We had a big battle in the Jenkins getting the standard operating procedures of the lab.
DR: From my reading of parts of the code, it appears that they aren’t required to give you that.
MS: No, they aren’t required to. We actually had to really make a strong, forceful argument in order to get the standard operating procedures in the Jenkins case. To their credit, the panel decided, even though the International Standards say they were not supposed to do this, they said they were going to [require the lab to produce their SOPs].
DR: That is to their credit, because if the panel goes the other way and says, “Look, the lab is not required to hand over the SOPs” that really shuts down a lot of avenues that the defense can go down to look for any possible breaches of conduct that might have caused an AAF.
MS: I can understand needing to make a decent showing of need for these documents, some grounds that they could be relevant. But to have an absolute ban on getting hold of the documents, limiting information that you can obtain on discovery, just makes it seem like a black box that you can’t look into.
DR: In looking at the Landis decision, one thing that struck me was that $100,000 penalty that they slapped him with. That could send a chilling message to potential litigants in the future, saying, “Look, you know, if you test positive and you challenge it, not only will we uphold the AAF but we may fine you, as well.” If I were an athlete looking at trying to defend myself in the future, I would seriously question whether I would want to put not only my legal fees at risk, but fees beyond that. Do you think that would have a chilling effect on people defending themselves?
MS: A little bit. I’m not as worried as I originally was, because I look at some of the types of arguments that were made here, and as we talked about earlier, the lack of, perhaps, diplomacy in making some of the arguments. I guess I’m willing to say that it’s a little bit of a unique situation. I would have to advise a client about this possibility and say a really aggressive defense could backfire.
DR: I think that’s kind of the message. You can defend yourself, but there’s a fine line between defending yourself and crossing over into being too aggressive about it.
MS: Right. And I’d have to advise a client, that’s what we’re talking about, how far to go with certain arguments. I think that in most of the cases it’s not going to be a problem. I think it’s going to be a problem in the higher profile cases, rather than just the standard cases.
DR: I saw one analysis of part of the Landis result on a blog called WADAWatch, and the author of that blog has some experience in drafting international treaties, and has a strong interest in the Landis case. One of the observations he made was that with this particular decision, it seems like the Quigley rule has been if not dismantled, kind of close to being dismantled by this panel of the CAS. Do CAS decisions carry the kind of precedent that say a Supreme Court precedent might carry, and do you think that the result in Landis could actually undo the Quigley rule?
MS: First off, as to precedent, if you talk to Matthieu Reeb, the [general secretary] of the CAS, he will deny and take the standard European approach and say, “No, we really don’t have precedents here.” But if you look at the practice, particularly in the US and I think growing out of the European decisions, they really are valuing precedent. So, whether you’re going to have a self-correcting precedent like you have in the Supreme Court, where their decision is going to be superior to or replace a previous decision, it’s going to take a little while for that absolutely to develop, but I think the system will recognize yes, precedent is important, precedent and the change and the evolution of precedent is going to happen in the system. I guess that’s the long answer to saying, yeah, I think precedent is at work in the system. I guess I have to ask for a little clarification from you what you mean about changing the Quigley rule.
DR: If I understand the Quigley rule correctly, I think that was the ruling about 14 years ago that basically said that the rules that athletes are subjected to have to be predictable, that these rules shouldn’t be made on the fly by judicial review. It seems like in USADA/WADA cases, a lot of rules are being made on the fly and interpreted by the arbitration panels. And that leaves the athletes in a situation where they don’t really know what the standards and what the rules are.
MS: OK. Now I understand what you mean by the Quigley rule, because the other rule that Quigley is cited for quite often is the basic foundation of the strict liability doctrine.
The idea of predictability and transparency… is what you’re talking about. A good precedent system does actually enhance predictability, because you can look at the cases and know how the panels are expected to interpret and apply the rules. So when it breaks down is when you first of all don’t have all the decisions available. And that’s the problem with the CAS system. Not all the decisions are available. For example, we recently won a CAS appeal and it’s not been published anywhere yet on CAS’ website or USADA’s website.
DR: I was going to ask you about that. I thought I hadn’t really seen the full-fledged decision on Jenkins yet.
MS: This is a different case. Eric Thompson, a track athlete. The decision hasn’t been published anywhere. We’ve got a copy of it, USADA’s got a copy, WADA’s got a copy, but the public doesn’t have a copy. [Note: About two weeks after this interview was conducted, the Thompson decision was posted on USADA’s web site.]
DR: That’s not much for transparency there.
MS: No. No, and when you have a system that I’ll call flat, where there’s all these arbitration panels rendering decisions, and none of them sits above all the rest and will correct any inconsistencies, then there is a problem, in fact, with predictability. The panel decisions that come out of CAS are all considered to be equal. There isn’t one Supreme Court like there is in the United States that’s going to decide any inconsistencies between them. And that creates a problem with consistency and predictability. So I think there needs to be a bit more of a pyramid structure to this. So, for example, if Landis lawyers looked at his case and then found that there’s rulings from another case that’s exactly to the opposite conclusion on the same exact issue, then you would have another level of court that would reconcile that difference.
DR: It seems like in these kinds of proceedings, once you get to the CAS level, if you don’t like their results, then according to the way I understand it, the next step is to go into the Swiss courts. And that’s only worked in one or two occasions. I think it was the Guillermo Cañas case, where it got sent back to the CAS panel, and they said, “Yeah, OK, so we’re going to decide the same way again.”
MS: The way arbitration is set up is that the government courts, the courts of law, will only review the process. If the process was done according to the basic principles of fairness, they won’t review the outcome. So, if Landis’ people want to challenge the decision in court, they’d have to show some violation of procedural rules or inability to make an argument or bias, extreme bias, by the arbitrators. – as opposed to the decision was wrong.
DR: It sounds like they would have a very high hurdle to go over in order to reverse this ruling.
MS: Correct.
DR: I’ve seen some talk that Landis’ team might go into the US courts, but it seems like Gatlin did that and he basically got turned away. Do you think that would happen to Landis if he went into the US courts?
MS: Yeah, I think so. He would have to find jurisdiction under the Federal Arbitration act and then argue that there was something just like we talked about, the arguments that would have to be made in the Swiss court – extreme bias of the arbitrators, or procedural irregularities that denied them the opportunity to make their arguments.
DR: Having reviewed the ruling, do you think that the arbitrators came to a fair ruling? Do you think it was the right ruling, or do you think that part of the reason for this ruling might have been their annoyance with the way that Landis’ lawyers had pursued the case?
MS: I guess I’d have to say, based on the rules, based on the way WADA and the Olympic Movement has drafted the World Anti-Doping Code, and the way the UCI set up its rules, that the decision was appropriate. The thing that I could never do is go in there and decide which of the experts is more credible. That, I think, was fundamental or pivotal to the decision. As long as their judgment on the experts was appropriate, then I think the decision was correct according to the law. I do have some concerns about the rules and the law that it’s based on. If [USADA’s] experts were credible, then the decision, based on the way the ISL and the World Anti-Doping Code are written, [then the panel’s decision] is appropriate. But I do, as I’ve said, have problems with the inability to attack the ISL and some of the burdens created by [the CAS panel’s] interpretations of the technical documents.
To be continued …
I have wondered just what it was that set the CAS panel so off against Landis and his team. I didn’t have the time to wade through all the transcripts, nor the expertise to understand and perhaps read between the lines. Now I have a better idea, although I still think that CAS stepped over the line in their criticism. After all, aren’t lawyers supposed to fight vigorously for their client? Shouldn’t an expert who finds the evidence flawed argue strongly? And when someone does step over the line, shouldn’t that be dealt with (by objections from opposing counsel) there and then. I imagine that quite often in the US courts lawyers try to push the limits; I don’t think you would get this sort of criticism in the decision, but rather at the time.
I still wonder, however, if there was more to it.
William,
It seems to me, that the criticism is mostly over style. I haven’t read the full transcripts in detail, but the panel apparently had a big issue with the style of defense and how Maurice Suh, et. al. presented it. Apparently, these proceedings are usually more genteel and gentlemanly, or at least that’s what the non-American arbitrators prefer. So a more subtle approach might have served them better. But, that said, the decision should have been about substance. And, despite the style employed, my impression is the panel gave short shrift to the substance of Team Landis’ arguments. The problem is that they’ve couched their decision in such a way that I suspect going into either the Swiss (too late for that now) or US courts will be difficult, and difficult to win.
Hello Rant,
From der Kurier an Austrian newspaper.
William, I’ve been struggling with the same question, as well as the question of how the Landis case (and recently, the Mayo case) was handled.
It is easy to argue that the CAS simply sides with the ADAs, but that doesn’t explain the relatively small number of cases where the athlete is able to win.
I am playing with the following idea, not sure I believe it yet: perhaps the CAS sees itself as part of the process of sport itself. Perhaps it is the ultimate goal of CAS to do what is best for sports, and that it is guided not by principles of justice and equity, but by the values associated with athletic competition and fair play. Per these values, the individual rights of the athlete boil down to the right to participate in a fair competition, which will most often subordinated to what’s in the best interests of the sport in question (and sports in general). But sometimes, it’s “fair play” for the athlete to win. (It can’t be “fair play” for one team to always win and the other team to always lose.)
Consider what happens in a baseball game when the umpire blows a call, costing your favorite team a critical game. We expect that the players and team manager will be plenty upset about the blown call, but that they will express their upset within certain limits. So, it’s OK to utter the time-honored objections, the “we wuz robbed” sort of complaints, but it is bad form to cross the line and question the overall competency or honesty of the umpires, or worse, that the umpires may be corrupt and making fraudulent calls.
It may be that the attorneys for athletes must also respect a similar line, drawn between questioning the results of a particular doping test and accusing the entire system of being rigged and biased. Perhaps Suh would have gotten further if he’d limited his criticism to the French lab, while making appropriate noises in support of the anti-doping system.
If you read the Jenkins decision, you may get the impression (as I did) that Prof. Straubel took a much gentler and more respectful tone with the arbitrators than did Mr. Suh. Of course, Prof. Straubel also had the advantage of arguing a clear-cut and well established violation of the WADA rules.
The impression I have of the Jenkins case is that Prof. Straubel got the CAS to view the actions of the labs as somehow “unsporting” or “unsportsmanlike”. The labs had not followed their own rules, and had argued that following these rules was unimportant in the face of the doping offense at issue. How could it be good for sport, consistent with fair competition, if the labs do not follow the rules? Isn’t “following the rules” the very essence of fair play? If the athlete has to “play fair”, then so do the labs and the ADAs.
Apparently, Prof. Straubel was able to accuse the labs of having cheated, without at the same time accusing anyone of dishonesty, or fraud, or corruption, or even incompetence. Any such accusation would be bad for sport, and if the CAS sees itself as the protector of sport, the CAS is not going to side with an athlete at the cost of damaging the sport in which the athlete participates. How could the CAS be said to be upholding the principle of fair competition if it diminished the value or importance of the competition? The CAS was able to side in Jenkins favor in a quiet way, without having to draw much attention to itself, and certainly without causing any serious harm to sports or even to the anti-doping system. Of course, the CAS could not have done the same thing in the Landis case
The other problem with the Landis case is that Landis had so much abuse heaped on him, long before his legal team was in place and the legal battle had truly begun. You have to question whether it would have been possible for the Landis team to act the part of “good sports”. If the athlete’s character is put into question, if the goal of the process is to condemn doping and anyone successfully accused of doping, if the process is intended to reach past the competition in question and to impose punitive sanctions, then it’s reasonable to expect the athlete to fight back with everything that he (or she) has.
Ah well. At the end of the day, I am a practicing lawyer, and questions of justice and equity take second place to what kind of practical help I can provide to a client. The lesson to be learned from the Jenkins and Landis cases may be more a matter of style than substance. I bet if we could ask Prof. Straubel, HE’D say that he fought like hell for Ms. Jenkins. But perhaps if you fight the fight with a nod towards propriety, with a gentle and respectful tone, with a stated goal of fair competition, in an effort to bring light and clarity to just a few pertinent matters in the interest of fairness and in the spirit of cooperation, and of course with the greatest respect to the ADAs and the sporting authorities…
… hindsight being 20-20 … maybe you can get your story heard. Even by the CAS.
Honestly? I doubt that anything could have been done differently to help Landis. His case was too high-profile. As much as it rankles to say this, I think that the powers that be decided that Landis’ conviction was in the best interests of cycling in particular and sports in general … and this certainly percolated its way to the very top of the system, to the CAS, whose true goal may be to do what is best for sport.
Rant, I read the first 25 pages of the decision and then paged to the back to see the reasoning on the different column on the GC/IRMS. I got a similar impression as Straubel – that Landis’ Attorney and experts just completely angered the panel to the point that they weren’t going to give him the time of day. I view this as really unfortunate, because I thought there were plenty of scientific issues significant enough that he might have actually won. The positive outcome of him winning might have been a systematic review of LNDD QC and procedures (at least) or (better)losing their WADA accredidation.
Larry:
Your analogy with an umpire is one that strikes home for me, as I officiated several high school sports for a number of years. There are, however, a number of differences. In the first place, there are officially only a very limited number of situations where any sort of official protest can be lodged, ignoring the use of instant replay today at upper levels of sports (which by the way was banned in HS sports). So the large part of coaches arguing with officials is doomed from the start: no official is going to reverse his ruling on any particular play. There are times when one official may confer with others to determine what happened before making a final ruling, but if an official believes that he saw all that was necessary, he will make the ruling on his own. And officials are supposed to leave their own personal feelings about any particular team/coach/player out of the equation, indeed, if it is known that there are problems between any team and official, that official may not be scheduled to officiate that team. So, to sum up, whether or not a coach argues with an official, and how he does so should not have any direct effect on the game outcome, unless the protest escalates to such a level that a forfeit is in order.
Perhaps if I read the transcripts in detail, I would get a different picture, but it rather seems to me that Suh and company was allowed to present their case in this manner, and then after everything was said and done, nailed for it. If USADA found Suh’s manner inappropriate, shouldn’t they have objected? Or should the panel have warned Suh and or his witnesses that they were crossing some line? I am assuming here that there is no manual of procedure which lays things out such that Suh should have know in advance.
Of course, there is always the possibility that Suh and Landis know in advance their chances of winning were slim to none, no matter how they presented the case, and the presentation was more for the wider audience. As you mention, Landis’ character has come under wide-spread attack, and he may have been making some kind of statement here. To me, some of the things LNDD did, producing “documentation” long after the fact to cover up what appeared to be errors very strongly smacks of something that was cooked up which is a kind of fraud, IMO.
William, I understand that the analogy to an umpire’s decision is not a perfect one. Sure, I know that there’s not much point in arguing an umpire’s decision, as they’re almost always final. My analogy was intended to point out that there’s a limit on how hard an athlete is allowed to argue if the argument is to be seen as “part of the game”. I think you know some of the unwritten rules on arguing with umpires: you can scream, but profanity is limited. You can get very close, face to face, but you can’t touch. Spitting is WAY over the line. You can kick dirt but not toss bases. Etc, etc.
Obviously, we’re not dealing with a bright line here, and of course it gets harder to define where to draw the line in a legal proceeding. Actually, even if we’re talking about a normal legal proceeding with judges and courts, there are lines you don’t normally cross. For example, a defense attorney may be quite critical of the work performed by the police, but may hesitate before criticizing the entire police department. There are no hard and fast rules here, and the court is unlikely to tell the defense attorney that he’s acting like an ass and alienating the judge and jury. (There are different rules, better defined, for “contempt of court”, where the judge WILL warn the attorney that he’s stepping out of line. But short of contempt of court, an attorney has the right to choose an offensive and ineffective line of argument, and it’s not up to the prosecution or the judge to issue any warnings.)
It appears that Suh’s tactics worked better in the initial AAA arbitration than they did on appeal to the CAS. Prof. Straubel may be right that the CAS is more European and expects more genteel tactics. Or maybe Suh stepped up the level of aggression and contention over time.
You bring up the possibility that Suh knew exactly what he was doing, and that he was intentionally provocative. I think that this is exactly what happened. Suh is an excellent lawyer, by my observation, and I think he knew he was arguing very aggressively and risked offending the very people he needed to persuade. I had the impression that Suh had already decided he could not win before the CAS, and his arguments were being made to set up a future case in state or federal court. However, I think that Suh was caught by surprise by the CAS’s complete rejection of all of the Landis arguments as well as by the monetary sanctions.
In any event … a lot of this is hindsight. I am trying to understand why Jenkins won and Landis lost. It’s pretty easy to say today that an athlete should argue like Jenkins and not like Landis. But if Landis HAD argued like Jenkins, he probably STILL would have lost, and then we’d probably question why he didn’t fight harder.
d-Bob, I agree with your analysis, but I think you’re skipped a critical factor in how the CAS dealt with the scientific issues. The CAS decided scientific issues based on the credentials of the experts, and not by considering the issues themselves. While that probably sounds like a harsh criticism, I don’t mean to sound critical. In these forums, people like me take shots at scientists who have devoted their lives to an understanding of their fields of expertise. This is a luxury I have, sitting as I do on the sidelines in relative anonymity. As a judge or a member of a jury, I’d be loathe to try and “out-expert the experts” and decide a case based on my meager understanding of which expert was right and which was wrong (an understand that would probably be 100% based on the testimony of the experts themselves).
In cases like the Landis case that turn on a “battle of the experts”, the judge or jury typically does one of two things. First, the judge/jury may simply ignore the expert evidence and decide the case on whatever other evidence is available. This would have been a disaster for Landis, whose case was based on expert evidence that his AAF was the result of bad lab procedures. Second, the judge may choose to believe whatever expert seems most credible from a non-expert point of view. Which expert went to the best school, holds the most prestigious job, has the most relevant professional experience? This is pretty much what happened at the CAS: the panel sided with the USADA experts on the basis of their better experience with doping testing in athletics.
Interestingly, the original AAA panel took a third approach, stating that the approach was necessary in order to avoid the classic problems with a “battle of the experts”. The AAA panel hired their own scientific advisor. This decision was widely criticized, here and elsewhere by those of us sympathetic to Landis, as an abandonment by the AAA panel of its obligation to itself evaluate the evidence. Also, we weren’t exactly happy with the panel’s choice of expert. However, with 20-20 hindsight, we can appreciate that there WAS no ideal way for the AAA panel to solve the “battle of the experts” problem, and that the AAA panel probably would have reached a decision like the CAS panel if not for the presence of the scientific expert.
Larry, First, I’m not taking what you say as criticism, because I know you’ve spent a lot of time thinking about this.
You’re right, they didn’t consider the scientific issues, and they sided with the ADA experts. It seemed to me there was more to it than the ADA experts had better credentials. It looked to me that they decided to side with them because 1.) Suh and Landis’ experts angered them ,or 2.) they decided to ahead of time, possibly for political purposes (but I’m NOT cynical ;-)). I think they should have tried to understand the scientific issues because, many times, that is what these cases turn on. If they are incapable of that, then I think it’s a huge disservice to fairness, and (as I’ve stated multiple times before) I think the panel should be comprised of (or at least include) Analytical Chemists. Said another way, if these panels are not capable of discerning between two experts arguing whether a scientific issue should cause a burden flip, what good are they?
d-bob, IMHO the problem of the “battle of the experts” is not solvable. You can take a few different imperfect approaches to address the problem, but the problem will remain.
When I was young, I was appointed as a consumer representative to a national task force that wrote a safety standard for miniature Christmas tree lights. I can assure you, I knew nothing about electrical safety. We knew that we were supposed to hear testimony from all sorts of people: domestic manufacturers, importers of lights from overseas, fire safety experts, and so forth. We wanted to come up with a simple standard, one that would not favor any existing manufacturer, one that would not add any more cost to the lights than necessary, one that would be flexible and easy to enforce and would not require 50 pages of new regulation.
What our panel ended up doing was hiring an independent “expert” to guide us. Our expert had worked with Underwriter’s Laboratory, and he definitely had a U.L. kind of bias for heavy wire and domestic manufacture. We actually pushed him pretty hard to consider other points of view, and I think at the end of the day he wasn’t crazy for the standard we came up with, but at the same time the standard had U.L.’s fingerprints all over it. I think we did a pretty good job, all things considered, and I certainly tried to learn all I could about the relevant scientific issues, but I was not going to put 100 million American households at risk of their Christmas trees bursting into flame on the basis of what I could learn about electrical engineering. I relied on the experts. What choice did I have?
Of course, the government could have kicked all of us consumer representatives off the panel, and replaced us with electrical engineers, which, in the end, was more or less what they did. I’m not sure that the engineers did a better job than we had done. Every “expert” has an expertise based on his or her own experience. An electrical engineer who designs the wiring for a high rise building will bring a different attitude to a project like this than one who designs wiring for a toaster oven. Every engineer has his or her own internal calculus for a safety cost versus benefit analysis, his own approach to how to build safety into a product in a way that is reliable and testable.
And at the end of the day, an electrical engineer is going to defer to the “experts”, same as anyone else. Would a guy who designed the toaster oven wiring really place his own expertise ahead of the guy from U.L., whose JOB it is to verify the safety of home electrical appliances?
d-bob, you could put analytical chemists on an arbitration panel, but I don’t think those chemists are going to decide that they know more about gas chromatography than Shackleton, Brenna and the other experts called by the parties. Heck, at the CAS USADA called as a witness a guy who claimed to have INVENTED GC-IRMS (and based on my research, he probably WAS a co-inventor). What is your panel of analytical chemists going to do when they’re faced with expert witnesses of this quality? Would they really be more likely to see through the testimony of these eminent experts than, say, a panel of lawyers? And would they have the courage to tell the experts that the experts are full of … er … beans?
My experience as a lawyer familiar with government processes is that your panel of analytical chemists would be extremely conservative and highly unlikely to question the status quo testimony of the ADA experts. We already have many government agencies manned by “experts” in the fields they are supposed to regulate — communications “experts” at the F.C.C., pharma “experts” at the F.D.A., and we see that these “experts” tend to strongly favor the interests of the industries they are supposed to regulate. These experts are very good at enforcing status quo standards, and very bad at thinking outside of the box when such thinking is necessary. This is not solely due to corruption or the so-called “revolving door”, but is also a problem that “expertise” is itself conservative. “Expertise” is often just another way of describing a familiarity with how things are done.
This may shed some light. From ‘Global Arbitration Review’ – a website for lawyers and others who work in international arbitration:
HEADLINE: Landis panel criticises legal team.
2 july 2008
A new Court of Arbitration for Sport decision shows how to rub arbitrators the wrong way.
A high profile panel hearing an appeal against US cyclist Floyd Landis’s doping ban issued a 56-page decision this week. The panel – David AR Williams QC, David W Rivkin and Jan Paulsson – rejected all of Landis’s 17 points.
The decision also shows the panel were unimpressed by the Landis legal team’s conduct of the case – particularly towards the end of the hearing.
In fact the award repeats an exchange between Jan Paulsson and Landis’s lawyer, Maurice Suh of Gibson Dunn & Crutcher LLP, in which Paulsson criticises Suh’s reliance on the “rhetoric of persuasion” rather than the evidence.
In his closing submissions Suh had asserted: “Mr Landis’ search for the truth in this case has been obstructed – often with devastating results – by the presence of bias, inconsistent and false statements and fraudulent documents.”
Paulsson said Suh needed not rhetoric but firm evidence to support serious allegations of bias, fraud and cover-up levelled at the LNDD, the French anti-doping laboratory that analysed Landis’s samples. “I think this is the time…not so much for prose, but for references,” Paulsson said, adding that the panel required not a “lot of adjectives” but clear notations of “the evidence of those propositions, in objective form.”
Instead Suh had used “the language of indicia of falsity.”
In the award, the tribunal’s president David AR Williams QC reiterated the tribunal’s surprise at the style of the appellant’s closing brief.
“The panel has found no evidence at all to sustain any of these serious allegations,” he said. “Moreover the panel is surprised that such serious allegations should be pursued in the closing brief when it must have been clear at the end of the hearing that there was no evidential basis from expert testimony or otherwise to support them.”
Before a previous challenge to his ban in front of an American Arbitration Association tribunal Landis said he intended to “take down the French lab in an embarrassing way.” Williams said the cyclist’s gameplan was “perhaps understandable” when his livelihood and reputation were at stake. However, “when it emerged at the end of this hearing that the evidence would not support the strategy it should not have been pursued further,” he said.
The panel awarded the respondent, the US Anti Doping Authority, US$100,000 in legal costs to reflect the fact that nine witnesses summoned to the hearing in New York at Landis’s insistence, were never called.
It also criticised Suh’s consistent failure to put the fraud allegations to LNDD witnesses – “a fundamental aspect of fairness toward witnesses.”
Landis’s lawyer wasn’t the only person who left the panel underwhelmed. Several expert witnesses also faced criticism for “acting as advocates for Landis’s cause” rather than “scientists assisting the objective search for truth.”
Some had failed to respect ” the clear distinction between administrative deficiencies, bad laboratory practice, procedural error, or other honest inadequacy on one hand, and dishonesty or bad faith on the other,” the panel said.
The award cites an exchange between Paulsson and an English expert witness who had alleged in a report that the French laboratory covered-up internal errors “for the purpose of establishing an anti-doping violation…when the scientific evidence does not support it.”
Paulsson noted that the expert’s report was more strident than his oral evidence and questioned whether this was “English understatement”.
“Should this laboratory be shut down?” Paulsson asked, adding, “It’s hard for me to think of anything worse that a scientist would do than to cover up and to act with a purpose of establishing a violation when the scientific data isn’t there.”
The witness replied that the laboratory had “tried to make the analysis look better than it was” but did not “deliberately” seek to mislead – a contradiction of his previous positon, the panel noted.
Landis won the 2006 Tour de France with a stunning 120 kilometre solo breakaway in the race’s 17th stage. However, organisers stripped him of his yellow jersey after he failed a drug test. Urine samples suggested use of the performance enhancing drug testosterone.
Last year a AAA arbitration panel found Landis guilty of doping but issued a split decision containing criticism of laboratory procedures. Landis opted to re-argue every point before the CAS panel, which said it would hear the case afresh rather than simply reviewing the last panel’s findings.
The CAS panel said the laboratory had not departed from international standards in its sample analysis. “All that Landis has established after a wide ranging attack on LNDD is that there were some minor procedural imperfections,” it said. A more focused challenge would have been preferable, the panel stated.
The CAS panel upheld Landis’s two year disqualification and the loss of his 2006 Tour de France title.
The title has passed to the second-place cyclist, Spain’s Óscar Pereiro Sio.
AR
The Battle of the Experts issue is one that has much wider impact than Landis and WADA, as it occurs in both criminal as well as civil cases in court systems everywhere. I personally am not happy with any of Larry’s 3 scenarios. #1, ignore the experts and decide on other issues is perhaps the least worst of the lot, but it assumes there is other good evidence to base one’s decision on. I think we are seeing some of this when someone argues that Landis’ performance was too good to be clean, or alternative, that his performance was well within his published training figures. Scenario #2, side with the “house” experts is troubling, what if there are legitimate issues raised by the challengers? #3, bring in your own expert is something I question, especially when the “adviser” has quite clear ties to one side, as Botre did, and when the advice is given in camera. We have no idea of what his advice was, whether or not it was good advice, or simply advice on how to word the final opinion so as to seem to have given proper consideration to both sides. Now don’t get me wrong here, I am not accusing Botre of anything, the issue is we don’t know, Landis’ lawyers did not have any chance to cross-examine or rebut him, etc.
I am going to present another possible scenario: the judge or jury considers the experts based on how they present themselves in court. Does an expert seem to be knowledgeable, yet able to present his/her testimony in terms a lay person can understand, does the expert seem creditable, does he/she hold up well under cross-examination, even the expert’s personal appearance can be used to judge. This can also be troubling, as an expert well-versed in public speaking could glibly present false or mis-leading testimony and appear to be credible while another expert could be presenting the correct information, yet come across poorly.
I served on a jury about 30 years ago on a case involving an expert witness. Briefly, it was a wrongful death civil case, and the defendants had already stipulated to being liable, so the only issue was the amount of the award. A young man had been killed in a car wreck and the parents were suing. The expert was a professor of economics, brought in by the plaintiffs in attempt to establish the victims “worth”. In the end, we the jury felt that an underlying assumption, namely that the parents would have been the life-long recipients of the victim’s money, was false, and cut the award from a suggested $750,000 to $75,000. I would like to think that we made a reasonable evaluation of the testimony.
William, your scenario #4 is one of the reasons why some people like juries. As non-experts, juries are not always impressed by experts. You saw in the O.J. case that the O.J. jury was completely willing to ignore the DNA evidence. Whether you see hope in that case that juries are a good alternative for evaluating scientific evidence is up to you.
I had a jury experience similar to yours, with a woman injured in a traffic accident. The woman’s lawyer introduced an “expert”, a doctor who proposed a novel and expensive form of treatment for the woman’s injuries. As in your case, my jury also awarded the woman less than she had asked for, in part (I think) to keep her as far away from the expert doctor’s treatment as possible!
I’ll still stand on my statement that there’s no solution to the problem of the “battle of the experts”, but I will acknowledge that the jury system is another imperfect solution.
Jean C,
Interesting tidbit there. Do you know of any articles that have greater detail?
David,
Welcome. Hadn’t seen that article before. Thanks for providing it. It does shed some light on how the arbitrators viewed the Landis case.
Larry, William, dailbob,
Good discussion you’ve got going there. From where I see it, most doping cases are likely to be battles of the experts. How else can an athlete defend him/herself? The problem is how to resolve such battles. And, I’m afraid, it’s a problem that each panel will have to determine for themselves. Different panels will find different ways of striking that balance. I’m not sure what the answer is, and I suspect there is no perfect solution, but I think that part of the answer is that the arbitrators — or whoever is deciding the cases — ought to balance out the substance of the arguments, rather than the presentation style.
Unfortunately, sometimes appearances matter. And it appears this panel preferred a certain style of presentation. Rather than telling Maurice Suh that he needed to back down a bit, it seems that the CAS panel willingly gave him all the rope he needed to hang himself and his client. I suspect that in a court case, an attorney going too far off the rails would get a warning from the judge before the case was over. Too bad that didn’t happen for Floyd’s legal team. But arbitration is different.
I don’t know what might happen next. I’m sure that there’s a plan of some sort (even if only to lay low until the suspension is over), but I haven’t heard what it is, yet. My understanding is that to file a case in the US courts, there’s a 90-day time limit. That would mean that any attempt to overturn the ban in the US courts would have to be filed before the end of this month. We’ll have to wait and see what happens.
There are 3 possible things Landis could seek if he choose to pursue the case further: overturn the suspension, re-instatement as the 2006 TdF winner, or seek damages. The suspension is pretty much moot as by the time any court action was taken, the ban would probably be over anyway. No such limit would apply to reclaiming the victory, but the US court system lacks jurisdiction. An appeal to the Swiss court system would at best result in the case being sent back to CAS for re-hearing, and ultimately, the same results in all likelihood.
As far as seeking damages, I doubt he could bring such a suit against CAS or the panel members in the US court. He might be able to bring USADA and/or its lawyers to the US courts, but I would think he would have to prove that they deliberately railroaded him, or at least, proceeded with wanton disregard for the truth. I am not sure that USADA it guilty of such.
Another possibility, hinted at here and at TBV, is that by getting some case into the US court system, different rules regarding things like discovery might force into the open further documentation and reveal additional evidence of LNDD’s flawed procedures. Even if this did not result in any address of the 3 things I mentioned above, it could possibly provide some vindication in the proverbial court of public opinion.
Of course, Landis might simply decide enough is enough, lay low until the suspension is over and then attempt a comeback. He might ultimately attempt to go directly to the Court of Public Opinion, a sequel to Positively False could be in the works.
Re: Larry’s UL comment… hit a pet peeve of mine, so here’s a personal rant.
I have had pretty extensive dealings with UL. They actually have a neat museum-ish display for tourists if you visit their IL campus, discussing how they started as a safety after the Great Chicago Fire.
Having said that, UL is a bureaucracy themselves, and subject to the same pitfalls as everything else. I have documented evidence that they mismeasured test samples, and couldn’t even calculate a simple average correctly, yet they charged us for taking up their time when we had a meeting to demonsrate their incorrect data so they wouldn’t pull our UL certification. They consistently overcharge for laboratory work, and are in this nice little tax-exempt status where they don’t have to disclose their financial data because they are a special kind of non-profit and have Congress in their pocket.
Plus they are now outsourcing laboratory work overseas, and I don’t trust the overseas labs licensing much either. Ironically UL has a division that audits other people to ISO standards yet they themselves do not have licensed ISO labs. Talk about a double standard. (I wanted to pull their ISO audits and complain to ISO but found I couldn’t since they didn’t have an actual ISO laboratory certification)
It is embedded in the public consciousness that UL *should* mean something, even though a “UL” mark may not apply to everything you think it might. For example, a hairdryer could have “UL” on the plug, and it may just apply to the outlet plug, not to the entire hairdryer. They are probably a necessary evil, but as so-called “experts” I don’t trust them any farther than I can throw them either.
Ironically was my experience with UL, that had me initially skeptical on the Landis case way back when. They all start with a good cause and the best of intentions, and then fall prey to crappy lab work, and bureaucratic ass-covering somewhere along the way.
banana OJ, I found your post VERY interesting. I have no experience with UL in the last 30 years (dating myself …). But what you’re describing is what I’ve usually found in dealing with experts. You hope that with experts, you’re getting the best thinking available, but often you’re getting some kind of standardized, homogenized, bureaucratic and politically safe line of thinking.
I’ve always loved the story of Richard Feynman’s testimony before Congress on the Space Shuttle Challenger accident. Feynman was one of the experts on the commission that investigated the accident. He famously demonstrated how the shuttle O-rings would become less resilient at ice-cold temperatures by immersing a sample of the material in a glass of ice water. Now, there’s the kind of expert we’d all want to serve on a WADA arbitration panel. What did the other experts on the commission think of Feynman? Reportedly, they thought that he was a pain in the butt.
This from the Wikipedia article on Feynman:
Feynman has been called the “Great Explainer”. He gained a reputation for taking great care when giving explanations to his students and for assigning himself a moral duty to make the topic accessible. His guiding principle was that if a topic could not be explained in a freshman lecture it was not yet fully understood.
Sounds like a good quality for an expert witness.
William,
Feynman was, indeed, the great explainer. His Feynman Lectures on Physics are considered by many to be the classic introduction to modern physics. I had the pleasure of meeting him on several occasions, and he was every bit the great explainer and a “curious character” (as one of his book titles called him).
I’m going to babble on about UL a bit more. Compare and contrast to the doping/cycling alphabet soup as you wish. This post contains its own alphabet soup. I apologize in advance if it is unnecesarily confusing. All information below is my opinion, I’m not guaranteeing 100% accuracy. Ken from the Environmental Chemistry Site probably knows more about this than I do.
Something that has happened recently (probably the last 10-sh years) in the “Saftey Assurance” market with companies such as UL, is that competition between various certification agencies has greatly increased. This is both good and bad. Some certification companies are buying out other “quality cert” companies, to increase their range of customers.
NSF (the National Sanitary Foundation), appears to be setting up to become the biggest rival to UL. They previously didn’t really infringe much on each other’s “testing turf” but the line has become much more blurred lately. NSF has also bought out some biomedical type certification companies and branched into athletic areas lately, such as certifing that supplements don’t contain any prohibited additives. This is beginning to infringe on what has traditionally been USP (United States Pharmacopiea) turf. (The FDA doesn’t count as far as I’m concerned because while their standards are useful guidelines they are toothless in general for enforcement.)
UL is expanding in other directions, and appears to be making more overseas inroads, particularly in Asian markets, that want to sell to the U.S.
All of these companies are going through internal growing pains. UL seems to be more mired in buracracy and “fuddy duddy” even as it tries to respond to market demands, and I have doubts about the quality of their testing data. NSF is growing so rapidly the right hand doesn’t know what the left hand is doing, and their intercompany comunication (particularly auditing with relation to testing) absolutely sucks. Having said that, NSF has more transparency in their testing procedures and results than UL, and I trust their raw data more.
These “expert certifications” seem to have exponentially increased in just the last 5 years, partially because of EU regulation standards. There are still variations in standards between European countries which make things even trickier. I think the tightening and unifying of standards is a good thing in general. However, like some other laboratories discussed here, there is very little real accountability or liability of the certifying bodies. If your company’s product fails, you can use the certifying paperwork to back up your due dilligence type claims, but trying to hold the certifying agency responsible for incomplete or inaccurate testing of your product appears to be as difficult as getting USADA to overturn a doping violation.
Larry,
While I definitely respect your opinion, I’m still in a different place (probably due to the fact that, having worked in a lab environment my whole career, I’m still disgusted/stunned that data obtained with such poor practice was actually used to destroy a person’s reputation). You said:
“What is your panel of analytical chemists going to do when they’re faced with expert witnesses of this quality? Would they really be more likely to see through the testimony of these eminent experts than, say, a panel of lawyers? And would they have the courage to tell the experts that the experts are full of “¦ er “¦ beans?”
To use a picture analogy, while I don’t expect the panel members (whether Lawyers or Analytical Chemists) to be able to discern different species of fern seed on the forest floor, I do expect them to be able to see the elephant crashing through the trees three feet away. So while an Analytical Chemist might not have the in-depth expertise on on the particular diagnostic ions of a particular doping metabolite, they would know that a fundamental principle of GC identification is that what you’re trying to identify MUST be tied to a known standard. When Brenna replies that this is impossible due to the MS and IRMS being different machines, the Chemist would ask why, because Agilent’s retention locking software should make this fairly easy, or why can’t you run them in the mix-cal acetate?. When the documentation says that a different column was used on the IRMS, the Analytical Chemist wouldn’t except the lame story about the LNDD Tecnicians forgetting to write down that they changed it back, because they know that everyone trained to work in this environment is trained that your documentation defines what happened (this is “lab science 101” which a Chemist would be extremely sensitive to). There are other things, but you get my point. A lot of this is just me still venting, because I’m still so blown away, and I’m thinking that something has to change so that no lab could submit stuff this crappy for something so important ever again. For me the one small consolation from this is that LNDD has been made a laughing stock among the scientists that have looked at this case, but that is small consolation.
Rant,
Sorry, about Austrian Mayer I don’t have more details.
Dailbob,
About the different column used on the IRMS, sometimes equipment can be used successuful out of their specification. Of course it can be only done for some particuliar cases. Probably that was one or that was the good column because few people have complained about it.
But that wan’t LNDD’s story, they claimed that it was switched back to the other column and they “just forgot to document it.”
And results were enough coherent, isn’t it?
Wasn’t it coherent with the testing of other samples?
dbob –
You say that LNDD has been made a laughingstock among the scientists that have looked at this case? I’m not seeing that. I’ve seen that the Landis witnesses criticized LNDD. There are a few scientists in the blogosphere (like you and Ken at Environmental Chemistry) who have been publicly critical. I remember seeing one article critical of LNDD in a forensics science online site, and I remember hearing about a conference where a speaker planned to use the Landis case as an example of badly performed scientific work. Finally, we have the Berry article in Nature, which is critical of the methodology used to validate WADA lab testing but didn’t look carefully at LNDD’s lab practices. You add all this up, and I don’t think it exactly totals to a universal denunciation of the LNDD’s work. In fact, to the extent that there is a general public perception that LNDD did sloppy lab work, I’d argue that this perception is based almost entirely on the opinion issued by the AAA arbitrators — precisely the group you’re arguing should be replaced with scientists.
dbob, I’m simply not seeing anything out there to make me conclude that LNDD’s work is atypical. Has LNDD received any kind of sanction? We’ve discussed in the past how there is no official chief body of lab technicians (roughly corresponding to the American Bar Association as the body for lawyers) that could step forward and condemn the LNDD lab practices. But there IS lab accreditation. LNDD still has its WADA accreditation and its ISO accreditation. They are still being permitted to operate exactly as they operated in the Landis case.
The saddest thing about this situation is that, for many of us, this is the closest we’ve ever come to seeing and understanding how a testing lab performs its job. People used to say, if your life was at stake, would you want to send your lab work to the LNDD? I’ve since concluded that the LNDD is probably a pretty good real world example of how my lab work IS being handled. To be certain, if this IS the way my lab work is being handled, I can conclude from the LNDD’s example that the lab will be permitted to continue to botch my lab work without consequences.
So … either the scientific community is not all that critical of the LNDD’s performance in the Landis case, or else we cannot count on the scientific community to effectively watch over the conduct of testing labs. Or both.
Anyhoo … I’m interested in two points you’ve raised about the lab testing. First is the idea that “your documentation defines what happened”. I think this is the right reading of the WADA rules, but I’m interested in learning more about how this works in lab science 101. There must be SOME way to correct mistakes in lab documentation. If you wouldn’t mind, I’d like to hear you discuss this.
Second point is tying GC identification to a known standard. Are you talking here about IRMS peak identification? I’ve devoted a lot of time to that issue, and I’ve tried to read all I could find in the scientific literature about that issue, and this non-scientist has come to the conclusion that the problem of IRMS peak identification is not an issue recognized by the scientists. I agree that the LNDD should have run a mix through the GC/MS and GC/IRMS containing the 6 metabolites at issue, and identified the IRMS peaks in the Landis samples by comparing retention times between the Landis samples and the mix. However, I’ve seen nothing in the scientific literature to indicate that this is what scientists do in practice, or even that the scientists do more to identify IRMS peaks than what Brenna described in his testimony. (I agree that GC/MS peak identification is important and that it should be done by examining the complete mass spectra for the peak. I also agree that there is a logical gap between good GC/MS peak identification and how to use this identification to figure out which IRMS peak is which, but outside of the Landis case I’ve never seen this matter discussed. I also agree that the WADA rules required LNDD to have standards in place for IRMS peak identification, and that no such standards WERE in place.)
Jean C, the documentation problem being discussed here — that the LNDD’s documentation indicates that it used different types of gas chromatography (GC) columns for its GC/MS and GC/IRMS testing — is a problem for all of the tests run on the Landis samples, and not just the tests run on the stage 17 samples.
In order to correctly perform the testing in the Landis case, it is critical to set up the right GC conditions. The selection of the GC column is the most important of these conditions. The test performed for testosterone doping is in two parts: a GC/MS test to identify the substances in the athlete’s sample, and a GC/IRMS test to measure the substances for a property (dC13) that WADA believes is associated with testosterone doping. The GC conditions need to be kept consistent between the GC/MS test and the GC/IRMS test, or else the labs cannot identify the substances measured in the GC/IRMS testing.
If the LNDD’s lab documentation is correct, then LNDD did NOT keep its GC conditions consistent as required, and the validity of the LNDD testing would be placed under question. USADA understood the serious nature of this challenge to the LNDD testing. USADA never tried to argue that the LNDD documentation was correct, and that the documented procedures were valid procedures. Instead, USADA argued that LNDD’s documentation was wrong, and that in reality the same GC column was used for both the GC/MS and GC/IRMS tests.
This evidence was critical for USADA’s case — without this evidence, USADA would probably have lost the case.
Obviously, this was not LNDD’s finest hour. In order to win the case, USADA had to argue that LNDD’s documentation was incorrect on this critical scientific point. As dbob points out here, labs are judged on the quality of their documentation, so it reflects badly on the LNDD if its documentation was in error. Moreover, if LNDD’s documentation was in error, then it was in error not only for the testing of Landis’ stage 17 sample, but also for the testing of the other Landis samples (which took place 10 months later). This means that the LNDD’s documentation of ALL of its testosterone doping tests during this period — for all athletes — contained this same error. In fact, it is likely that LNDD did not catch this error until the Landis team pointed it out to them, something that did not happen until after the case had been appealed to CAS. So if USADA is to be believed, LNDD had this same error in its documentation of something like 15 months worth of testosterone doping tests. LNDD does something like 150 of these tests a year. All of them would have contained the same error.
USADA made two arguments before the CAS to prove that LNDD’s documentation was in error. First, USADA presented testimony from the ADA scientists. The scientists argued that if the documentation was correct and different GC columns HAD been used for the two parts of this test, then the results would have been so badly screwed up that the lab technicians would have known that something was wrong. We can discuss this point in some detail — I don’t think the column switch would have produced results that would have been as obviously wrong as argued by USADA. However, this USADA evidence indicates the importance of the selection of the right GC column — USADA effectively admitted that if LNDD had used the wrong column, the test results would not have been reliable.
The second piece of evidence offered by USADA was the testimony of a man who serviced the GC machines for LNDD. According to this evidence, the service man would begin the servicing of the GC machine by replacing the column in the machine with a column the man carried with him. The man claimed that the substitute column was of the same type reported in the documentation for the Landis tests. At the end of the machine servicing, the man would have removed the substitute column and replaced it with the original column. The service man indicated that when he began his servicing of the GC machine, he must have changed the settings on the GC machine to indicate the switch to the substitute column, but when he finished his work, he forgot to change the settings back to the original column. The service man indicated in his testimony that he did not specifically remember taking any of these steps, only that he would have noticed it if he’d left the substitute column in the GC machine by mistake.
There are some obvious weaknesses in the service man’s testimony. First, the service man could not remember if he’d changed the GC settings and forgot to change them back. Second and more important, the service man did not indicate what column type was originally in the GC machine before the servicing began. It could be the case that the machine originally contained the right column type, or that it originally contained the column type indicated in the LNDD documentation, or that it originally contained a third column type. In other words, the MOST that can be said for the service man’s testimony is that the service man might have forgotten to change the GC settings when he finished performing his service work, causing 15 months worth of LNDD documentation error. But the testimony does NOT indicate how this error should be corrected. The testimony does NOT indicate that LNDD actually used the right GC column for the Landis test.
Note that USADA could have offered additional evidence to support its claim that the LNDD used the right column for the Landis test. For example, USADA could have introduced evidence to the effect that PRIOR to the servicing of this GC machine, the documentation of the LNDD testosterone testing showed the presence of the correct GC column. No such evidence was presented.
Jean C, I have personally concluded from the Landis case and other cases that regardless of whatever is provided for in the WADA rules, it is impossible to challenge the scientific validity of the results produced by a WADA lab. But I think that dbob is making a slightly different point. dbob is saying that there should be strict limits on the ability of a WADA lab to correct its documentation after the fact, to eliminate a problem in the documentation that would tend to disprove a case against an athlete. Remember that the lab enjoys a presumption under the WADA rules that it has done everything in accordance with the WADA standards. This means that the lab’s paperwork is presumed to be correct. If the paperwork is proven to be incorrect by the athlete, it is unfair for the lab to then turn and say that the paperwork does not accurately represent what really happened during the lab testing.
This is a good question that dbob has raised “How should a lab be allowed to correct it mistakes, and even, when it should not be allowed to.” In my opinion, 15 months after the fact, when the putative error is pointed out by the other side in a legal proceeding, is far too late. The LNDD/USADA story may indeed be to correct case, but it looks to much like a story cobbled up well after the case to CYA, which comes quite close to the fraud that Suh et al alleged.
Suppose instead, that someone at LNDD noticed the error shortly after analyzing Landis’ sample back in the summer of 2006, and that he or she pointed out to others there at the lab “hey guys, it says we used column X, but we really were using column Y” and others there verify that they indeed used column Y and that the error in the setting had no effect on the operation of the equipment. Then LNDD could have amended their documentation, not changing it, but putting in an addition explaining what happened, when they noticed the problem, etc.
Note: I am assuming that the putatively erroneous setting did not effect either the operation of the equipment nor the analysis of the data produced.
Larry, I did not mean to imply that there is a large, organized condemnation of LNDD by the scientific community – mainly because, the number of scientists that have looked at this case is neither large nor organized (in fact, it unfortunately may only add up to Ken, Duckstrap, OMJ, myself and the guys in my analytical lab!)! I only meant to say, that to a person, they’ve all been critical (RH may be the exception). I’ve found some satisfaction in this. And, (you’re going to think I’ve completely lost my mind when I say this) I really believe that the other WADA Lab Directors were shaking their heads behind the scenes (e.g., Caitlin’s “C” grade of LNDD’s chromatography). I say this, because I know they know better, despite their testimony on LNDDs behalf.
With the caveat/disclaimer that my scope is limited to my experience where I work (broadened a little bit by people that have come to us from other companies), I feel safe in saying that the slop at LNDD IS atypical in analytical labs in the US that operate within FDA/USDA/EPA regulated industries. This is probably due to the fact that we have more to lose than LNDD. If there is ANY question whether you’re endangering the public health, the consequences for us are potentially severe (product seizure, being forced to operate under a consent decree, even being shut down). SO, if my life was at stake, I wouldn’t want my test run by LNDD, because I believe with all my being that the norm is significantly better.
You mentioned that we cannot count on the scientific community to effectively watch over the conduct of testing labs. After some soul searching, I have to unfortunately agree with you. Accredidation outfits (ISO in this case) have a conflict of interest (you pay them to get accredited), so they’re ineffective as a policing outfit. In my world, the policeman is the FDA, and he has a really big stick. However, I see no analogous policeman for the WADA testing labs. You would think it would/should be WADA using the threat of pulling their accredidation. However, their mission is to get the dopers, so they are on the lab’s side, not the athelete’s. In my world, this would be like the FDA’s mission being changed from protecting the public health (i.e., being on the side of the consumer) to something else where, say, they are investigating the consumer, using my results. In this scenario, they’ve gone from making sure I have all my ducks in a row to assuming I’m right, because now I’m their tool. The sorry result of this is exactly what you’ve stated, there is probably no end in sight to the current situation.
We can expand on this more, but in regard to correcting errors in lab documentation, the standard in the U.S. regulated industries is a single line through the error (so the error can still be plainly read – note that this completely precludes the use of white-out, which would obliterate the original error), plainly writing the corrected result, with the initials of the person making the correction, the date, and a brief explanation. I believe this, or something very similar, is also used worldwide. Our SOP requires that we correct our errors “immediately.” However, if you don’t notice it until the next day, I’m not going to tell you corrections don’t happen then. I’m aware of no time limit in the FDA guidelines, but I can tell you if the FDA had us in court, had contacted us requesting information, or unexpectedly shown up at the front door to conduct an audit, it’s too late. I’m traveling with our Director of Quality Assurance next week. I’ll ask him about this.
Yes, I am talking about peak identification on the IRMS, but I’ll come back to this tomorrow. I’m getting up at 5 AM for a 55 mile ride tomorrow, so I’m getting some shut-eye!
dbob, quick thoughts as I wait for you to complete your ride.
First, I think that Catlin is a rare bird in WADA world — I may not always like what he has to say, but I think he’s honest and does not always follow the WADA line. I’ve seen no evidence to indicate that any other lab director was critical of LNDD.
I think that the WADA rules require labs to correct mistakes in their documentation — it’s not just a matter of avoiding white-out, but proactively correcting matters that need to be changed.
As for the other things you said: noted and agreed.
Here’s some more thoughts of mine about the GC column issue. The service guy says he switched the column back to the one that was on when he came. OK, does LNDD use this particular machine for tests others than the types used in the Landis case, and if so, does it use different columns? Or is it only used for tests that only require the one column?
William, my undertanding is that the GC machine at issue was used only in conjunction with the IRMS machine that did the Landis testosterone testing, and that this particular combination of machines did the vast majority of the LNDD testosterone testing. So, I think that LNDD probably used only a single type of column in this particular machine.
dbob can help on this, but it’s my understanding that GC columns must be periodically replaced. So during the time period in question (between when the service guy did his servicing and when the LNDD finally learned that its documentation showed the “wrong” GC column), LNDD probably replaced the original column multiple times. But as it never changed the type of column in the machine, the lab arguably had no reason to consider that it might have to review and change the GC settings that were responsible for reporting the column type in the lab’s documentation.
You haven’t asked the question … but (this is my understanding) when a lab first sets up a GC machine for a particular job, the lab chooses a number of settings for the GC machine, and this includes an indication of the type of column that has been installed in the GC machine. The GC machine then reports this column type as the column in use for all tests run on the GC machine until this GC setting is changed. It is my understanding that the GC machine cannot automatically detect the column type, so the column type reported by the machine is not necessarily the column type actually in place in the GC machine. Moreover, the GC machine CAN operate properly even if the GC machine settings fail to correctly indicate the column type installed in the machine. The GC machine might not work properly if the GC settings fail to correctly specify the column length and diameter, but again, I’m not aware of any problem with the LNDD GC settings that would have affected the operation of the GC machine if the “right” column had actually been installed in the GC machine.
So … if USADA has given us the truth and the correct column was installed in the GC machine, then the GC machine was probably capable of producing good test results (albeit with incorrect documentation). (I think that dbob will take me to task for saying that you can have “good” test results with inaccurate documentation.)
Of course, dbob should review and confirm all this, if possible.
Larry,
In the abstract, I’d agree that it’s possible to get good results even if the wrong column might be listed on the documentation. But that hinges on whether the actual column used is appropriate for the testing being performed. If it’s not (wrong polarity, for example), then the data may well be meaningless.
Problem is, after the fact, all we have to go on is the documentation. Maybe LNDD really did use the right column. But how are we to know? The documentation doesn’t tell us so. And memory is a tricky thing. There are things I swear happened when I was in high school and college. And yet, when I talk with my friends who were there, they have entirely different memories of the same events. (I never knew that Genny Cream Ale had such effects on memory … that is, if it really was Genny Cream that I was drinking … I can’t quite remember.)
Good documentation should allow a knowledgeable expert who wasn’t present during the testing to evaluate the data and determine if he/she agrees with the original interpretation. Relying on memory to salvage a poorly documented test is, in my opinion, not scientifically sound.
Rant, I probably wasn’t being clear. Yes, it’s essential to use the correct column type. The question I was trying to address is: if the correct column type (say, type “A”) is actually installed in the GC machine, but the GC machine settings indicate that column type “B” is installed in the machine, will the machine function properly with column type “A” installed? Or will column type “A” fail to function the way that a column type “A” is supposed to function, if the GC machine setting indicates (mistakenly) that column type “B” is installed?
As an analogy, think about an old-fashioned camera, the kind that uses film. You can put many different speeds of film in the camera, but depending on the camera, you may have to change the settings of the camera to adjust for the different film speeds. If the settings on the camera don’t match the speed of the film you’ve selected, your photographs may not come out correctly.
This is the question I was trying to address: does it matter for the functioning of the GC machine that the GC setting for column type ACTUALLY match the column installed in the machine? The answer is, I think not.
Larry, I wanted to expand a bit more on “documentation defines what happens,” because you expressed interest in this and, as a Lawyer, appropriately so.
Disclaimer: I’m projecting what is required of us, and how our Regulating Authority evaluates us, onto the relation ship between WADA/ISL and LNDD. I think I’m correct in doing this, because the WADA/ISL/LNDD relationship/process I’ve seen you, Mike, and TBV describe sounds exactly like what I live everyday. I will provide an example below that will provide insight into why I made this statement. I’d like your viewpoint on whether my projection is legitimate, because you’ve studied the ISL way more than I have.
As you are aware, I work in an FDA regulated environment. Some of the products we make contain “active ingredients,” which the FDA considers a drug. We are required to be able to prove that the “strength, purity and efficacy” are what we say it is, and have anything that can affect the product in this regard always under control,and documentation to be able to prove that for every batch. FDA has issued guidelines on their expectations on how to do this, i.e., “current good manufacturing practice.” How industry complies with this is through the writing of SOPs (where you say what your going to do) and keeping documentation that proves you did what you said you were going to do.
As you can imagine, there are a number of things that would affect “strength, purity, efficacy.” One, as an example, would be the formula itself. If you were to accidently put too much of a particular inactive ingredient into the batch, it would decrease the percentage of your active (decreased strength). So, we have an SOP that requires the person weighing each ingredient to write down what he weighed out and initial it. The SOP also requires a witness that agrees that’s what the scale said, and he also initials it, so two people have agreed on the amount weighed out (double sign-off). There is a similar SOP for when the material actually gets added to the batch.
Now, let’s say my product causes a rash on a number of people, and the FDA gets wind of it. They’ll want to see my SOPs (to see what I was doing to maintain control of my product formula) AND the documentation proving that I did everything that my SOP requires. This is the only proof FDA is interested in, and will accept. If they find a problem/error in my documentation that indicates the formula is not right (undel control), it’s game over. I’m either voluntarily recalling the product, or they will sieze it. This is why I said ‘documentation defines what happened” (because that is what my Regulating Authority is saying to me). This is the way the process works, I understand that (i.e., the rules are clear), so I act accordingly.
From reading the things you, Mike, and TBV have discussed about the ISL over at TBV, it’s apparent that it requires procedures (=SOPs) and that there be documention indicating what was performed. Also, since the objective of the defense is to show an ISL violation, and all the defense has to work with is the documentation (results), this process seems very analogous to what I described above. Consequently, when I heard after Landis’ first hearing that the documentation said the wrong column was used on the IRMS, I thought game over, Landis wins, because “documentation defines what happened,” and this seems like a clear ISL violation. Instead, other evidencce was allowed explaining it away (like this was a civil case). I found this completely dumbfounding, because it’s like the CAS panel changed the evidence rules to achieve the outcome they wanted (appreciate your thoughts on this), and it’s so foriegn to the process on how laboratries prove they followed their process to get their results.
Larry, I’m traveling most of next week, and I’m not sure if I’m taking a computer with me, so it may be a while before I get back. I still want to discuss your comments on the lack of information out there on IRMS procedure, and I can try and address the column questions then.
Larry,
I’d expect that the answer to the question you pose is correct. It’s not crucial to the functioning of the machine. It will still do whatever it’s supposed to do, just as a camera set to the wrong film speed will still work.
Now, whether or not the resulting product — pictures in your analogy, or data from the GC/IRMS machine in the other — will mean anything or be recognizable is another matter. In the camera example if, by chance, you made a compensating error in exposure, you could conceivably come up with good pictures despite the wrong setting on the film speed.
On the GC/IRMS question, it’s a bit more complicated. Since the documentation states that a particular column was used, one needs to evaluate the data in light of the equipment used, and what that data determined by the documented column would mean. At a much later date, absent any fairly immediate corrections to the documentation, how can we know what was really used? How can we be sure that the results mean what the lab says?
To have someone like a repairman come in and testify, almost two years later, that he must have replaced the column with the correct one, but didn’t change the machine’s settings asks us to believe his memory, as compared to trusting the documentation. And that opens up a whole slew of questions, such as: If we can’t trust the documentation on what equipment was used, how can we then trust the data reported or its interpretation?
At the very least, it was sloppiness on the part of someone (repairman, lab tech, other person unknown) to not properly document the equipment used. If they can’t get the equipment documented properly, I have my doubts about anything else reported.
Rant –
No, I don’t think I’ve made myself understood.
Let’s go back to the analogy of the camera. Imagine that your camera has two settings: (1) the ASA setting for the film speed, and (2) a setting for the correct date and time, that stamps the date and time for each picture you take onto the film in such a way that the date and time appears on the negative underneath the picture.
First, imagine that your setting for the film speed is correct, but that your setting for the date and time is wrong: the date and time is set to some time in 1997. You shoot a roll of film and take it to your local shop for development. Your pictures will come out fine and accurate, as the setting does not affect the ability of the camera to take a good picture. You may not even notice the problem with the date and time unless you carefully look at the negatives (and who looks at the negatives?).
Next, imagine that the date and time setting is correct, but the ASA is set for a much faster film speed. If you are not aware that the setting was wrong, then when you get your pictures back from the shop, you’ll see that they are all badly underexposed — in this sense, the pictures are NOT accurate. Unlike the date-time setting, the ASA setting actually affects the ability of the camera to take a good picture. (Yes, as you suggested, if you KNOW that the ASA settings are incorrect, you could adjust other camera settings to compensate for the incorrect ASA setting, but I’m assuming here that you did NOT know that your settings are incorrect.)
OK, now back to the problem with LNDD and its GC machine. Assume for the moment that the GC machine was set up with column type 1, and that this column type is the correct column type for performance of the testosterone doping test. But also assume that the GC settings indicate that column type 2 is installed in the GC machine. What kind of problem does this cause? Is the problem like the one we saw with the camera and the date-time stamp, where the GC machine will run just fine, except that it will create documents indicating that column type 2 is in use on the machine? Or is the problem like we saw with the camera and the ASA setting, where the incorrect setting will affect the ability of the GC machine to take an accurate picture?
(You know, it’s HARD to write clear descriptions of these kinds of questions!)
The problem with the GC column was brought to our attention late last year over at TBV, and a bunch of us actually read through the operators manual for the GC machine to try and figure out the effect of failing to specify the correct column in the settings for the GC machine. My recollection is that this setting probably does not affect the ability of the GC machine to take an accurate picture. This setting DOES of course result in an error in the documentation produced by the machine.
I’m not trying to minimize the importance of getting the documentation right, as you have pointed out and as dbob has emphasized. This is the subject of our current discussion here, at least for the moment. I was only trying to anticipate a question that I thought William would answer, which is whether this GC setting would affect something more than just the documentation.
Actually, Larry, I did get that. Even if my response didn’t quite mesh. By the way, with the underexposure portion of your example, that’s a technique used for shooting indoor sports, called “pushing” the film speed. Expose as if it’s a higher film speed (ISO 1600 vs. ISO 400 for Tri-X, for example) and then extend development times to compensate. The flip side is “pulling” the film speed, where you expose as if the film is a lower-numbered ISO rating (100 instead of 400, for example), and then reducing the development time accordingly. But more than once, your illustration of inadvertent underexposure has happened, and more than one photographer has gotten back dim, awful results (or grossly overexposed results).
The error in documentation for the GC column, is a bit of a different kettle of fish, much like the date analogy you used. Coming back at a later time, unless you know something about the pictures, how could another observer know whether the dates were correct? (Sometimes manufacturers put a date of manufacture in the edge of the film, so the error you postulate can sometimes be detected.)
Back to the columns. If the columns are of different types, and that difference affects the order and placement of the peaks, then that has a real effect on the interpretation of the data. That’s a bit harder to reconcile. The machine functioned correctly, as it was set up. And the picture, may be accurate, given the setup. But the interpretation of what the picture says can be different depending on the type of column used. Therein lies the rub, from my perspective. Are we looking at a picture that’s the equivalent of something written English, Danish, Sanskrit or Mandarin? And, once we know what language, what does it say?
The error in the description of the column should have been caught and corrected much sooner, if that was really the case. That it perhaps wasn’t caught until more than a year later makes me question just how accurate the rest of the documentation and interpretation of the results is. At the very least, it’s sloppiness on the part of the lab.
Rant –
OK, now we’re both on the same page, or at least very close.
You’re raising the question, what is the correct column to install in the GC machine for the GC-IRMS test? Well, the answer is two-fold. First, it should be the same column type used in the GC-MS test. Second, it should be the column type that does the best job of separating the peaks in the athlete’s sample.
If the lab goofs and does not install the correct column in the GC machine, then you have a serious problem with the “picture” taken by the GC machine. I’m not an expert, but I would doubt that any such problem could be corrected.
It’s been a while since I looked … but my recollection is that the LNDD’s SOPs DO provide for the installation of the correct column type in the GC machine. This is the column type that USADA claims to have been installed in the GC machine for the Landis tests. Whether this column type was ACTUALLY installed in the GC machine is the other question.
Larry,
I share your skepticism that there is no way to correct an incorrect “picture” vis-a-vis the GC machine.
I think you’re right about the SOP requiring the right column. One way for Team Landis to argue an ISL violation would be the lab’s failure to follow their own SOP. If they prove that, it’s in the lab’s/ADA’s court to rebut, given the whole burden-flip thing. We can never know whether or not the correct column was actually installed. We can only look at the evidence to try and figure out what happened. To sort it out, there’s the documentation (which suggests the correct column wasn’t installed), and there’s testimony from the repairman and others (which suggests it was).
The problem for the CAS panel was: Which to believe? They chose the testimony of the repairman and others, which goes by memory of events that occurred about two years prior (a decidedly tricky thing), vs. the contemporaneous, uncorrected lab documentation package. Why they chose to go that way puzzles me, but I suspect it has to do with their frustration with the style of Maurice Suh’s defense of Floyd.
dbob, I am working on a reply to your post … which was terrific and thought-provoking. It requires a careful response.
If LNDD used the particular machine in question only for the one type of test used in the Landis case, or at least, a set of tests closely related enough that only one particular type of column is ever used, it could explain (but not necessarily excuse) why the error in the machine settings went undetected. If all you ever use in your camera is Tri-X and you don’t go is for pushing, then you probably don’t bother to check what you have set your camera too. The serviceman says he services the machine with his own column, and since he took it away with him, he must have put the LNDD column back on. If LNDD only uses one type of column, this is credible.
But if LNDD has occasion to use other types of columns, then there is a lack of creditability. Could they also use the type of column used by the serviceman for their own maintenance?
I came to the conclusion that the settings on the machine don’t have any effect on the data produced (like the date setting on a camera); otherwise I think that this would have been a big issue, which would have been very hard, if not impossible, to explain away.
Of course, all this is rather moot, unless Landis does decide to pursue the case further in court.
Larry,
Your posting on September 5th at 12:44 pm is one of the best things I’ve ever read about this case. I hadn’t seen it expressly stated anywhere else that USADA had to argue that LNDD screwed up in order to win this case.
Again, the answer seems to boil down to “trust us, we did it right”. It really makes me wonder how Straubel won his case. Couldn’t they just argue that “trust us, we did it right”? Even if they clearly violated the ISL, why couldn’t they rule that, because they did it right anyway they still stayed within the spirit of the ISL?
On the issue of evidence, a few years ago I sat on a jury of a civil case for six days, a medical malpractice suit brought by someone with incurable cancer of the salivary gland. There was quite a bit of expert testimony, and what I remember is the judge was VERY careful about what the witnesses were allowed to say, and weren’t allowed to say. Foundation had to be established answer-by-answer, and there was a big conflict about whether or not a witness was allowed to testify about precise growth rates of the cancer. Ultimately the judge decided there was not sufficient basis for the expert to offer this testimony, and it was supressed (I learned the details after the case was over).
We ruled that the doctor didn’t do anything wrong, and I dont think this point that they spent half a day arguing would have affected the outcome. But the judge was definitely very careful in dealing with the experts.
And I have jury duty again next month…
tom