Red Flags - the CME Course!

6 Comments »

1. Comment by mlang52

   I do not understand why the 771 PAIN PATIENTS tested positive only 24% for the list of drugs. Were these patients on them by prescription, were they illegal, were they self-medicating? What about the 76% that did not test positive for the drugs. Were they not using their medications correctly? Or were they a result of the poor quality (76% negative) of the “drug screens” being used in the study?! It seems to make it completely useless, study -wise. But, I may not have all of the information I need to understand this (24%+) statistic that was thrown out.

2. Comment by James Stacks

   (the previous post was an error [and was deleted ..alex...])

   This paper advances as its “primary result” (see the first sentence in the Discussion) a sample based population proportion estimate of .038 for opioid abuse disorder by DSM-IV criteria. The authors provide no statistical test of significance for that result, but state that this proportion is “four times” higher than a “general population” proportion of 0.009. Clearly, by elementary methods, the CI_95 for the sample proportion estimate does not include 0.009, so the result is significant. The conclusion is therefore statistically correct, but there are serious methodological problems with attempting to take any valid conclusion from the statistical result. For instance, if the result is meant to imply that chronic pain patients undergoing opioid therapy have a higher rate of opioid abuse disorder than chronic pain patients not getting opioids, that conclusion is probably invalid. Even more conservative ideas to the effect that there are more opioid abusers among chronic pain patients than “in general” is probably not supportable by this result because of the problems discussed below.

   I can find no information in the manuscript about the source of the “general population” estimate of 0.009. In medical research, that may not be an issue (as everyone might know what they mean by “general population” — there may be some official standard). In behavioral research, it is poor methodology. To base the main finding of a paper on a comparison between a population estimate and a sample estimate, without demonstrating explicitly the similarities between the methods by which the population estimate was obtained and the methods actually used in the study, provides no useful information. At such low base rates, any small difference in sampling methods and inclusion criteria, and most importantly, any difference in measurement, could account for the findings altogether. For instance, were the people on which the 0.009 “general population” estimate is based paid $50 for participation? Were they referred by “varied” physicians in Wisconsin? Was the diagnosis by SDSS? That is only the beginning of the possible differences. The authors cite “Substance Abuse and Mental health Services Administration (SAMHSA) Office of Applied Studies: Results from the 2005 National Survey on Drug Use and Health: National Findings. http://tinyurl.com/2tnfc5. Pages 71-75″ for some population statements, but I cannot find any specific source for the 0.9%.

   In regard to sampling, there are several other concerns. The response rate was 78%, which is fairly high. The authors note that it is “possible that the nonrespondents had higher rates of substance use disorders.” Obviously, it is equally possible that they had lower rates. The assumed desire to hide drug abuse is not the most parsimonious explanation for refusing participation. Most likely, they just don’t care, don’t feel like it, don’t have the time, and they don’t need the $50. If there was widespread conspiracy to hide drug abuse from doctors, then this study would not have detected as many as it did!

   Furthermore, the sample is entirely from 8 counties deemed “throughout” one state (Wisconsin), and only vague reference is made to the fact that proportions of participants “varied” between the relatively few physicians who referred potential participants. Any estimates obtained from such a sample are unlikely to be population estimates that could be meaningfully compared to a “general population” estimate of any kind, especially on differences in the range of 2-3 percent. It would be very important to understand where the 0.009 proportion came from.

   I am not sure what the implications for selection of the 801 daily users from the 1009 original sample is, but it is worrisome to me. Also, an enormous proportion of this sample is on income disability. I have no reference on what the authors consider to be the “general population” (did anyone see a specific reference?), but I suspect the rate of income disability in this sample is much greater than the “general population”. These folks don’t look like they are even in the military, much less “generals”!

   The authors entered several variables into two logistic regression models, one with a dichotomous criterion variable of “substance use disorder” and one with “opioid use disorder” (What is a “use” disorder? My DSM-IV doesn’t have one.) These were evidently simultaneous or one-step logistic regressions. The criterion variables were based on the SDSS instrument, which is a structured interview that was administered by trained research assistants. DSM-IV diagnosis was apparently not made by a physician or psychologist. The predictor variables included common demographics, lifetime history of drug and alcohol dependence, psychiatric history, severity of pain, drug test results for cocaine and marijuana, and the four “aberrant behaviors”. Apparently, for the data reported, all variables were entered simultaneously, and the four aberrant behaviors were the only uniquely significant predictor of opioid use disorder (although less than high-school education came close to traditional criteria). The more general “substance use disorder” was predicted significantly by age (negatively correlated, no surprise), positive drug tests for marijuana and cocaine, psychiatric history, and the four “aberrant behaviors”.

   For opioid therapy advocates, there are several sources of good news in the results. First, it is unlikely that the proportion of participants meeting study criteria for opioid use disorder can be compared to a “general” population estimate of opioid use disorder (of unknown origin). The proportion of opioid use disorder in the sample is low, by any standard. The four “aberrant behaviors” are likely to be correlates of the criterion variable to begin with, as the SDSS interview (although I am not familiar with it) probably contains self-reports of behavior that are very similar to the “aberrant” predictor behaviors themselves. That is, is the “prediction” actually a tautology?

   This may not be entirely true as the attempted distinction between dependence and addiction may be important, but I will leave that for those with more expertise and more mainstream opinions, as I see no demonstrable distinction between euphoria and analgesia (but that is for another day — in the words of Bush, we can fight them there, or we can fight them here).

   Also, for those that would like to have physicians “weeding out” pain patients based on a history of substance abuse treatment, the authors note that “a lifetime history of substance abuse was not statistically associated with current substance use disorders.” I am assuming they meant no raw initial correlation, rather than not having unique predictive ability (which is all the logistic regression shows).

   I also liked the table showing percentages of the different groups that had the individual “behaviors”. Obviously, these are candidates for (potentially very revealing) positive predictive analyses. These behaviors don’t appear to be worth much at all in those terms. If a false positive means suffering for an “innocent” person, then I say let the “guilty” go (but then I would say to let them go anyway).

   Overall, the authors do not appear to endorse the use of these results for opioid fear-mongering. The paper almost has the quality of having been written for fear-mongering use, but then edited by more knowledgeable people (perhaps reviewers? perhaps written by students and edited by the authors?). Their conclusions are that the results should not increase fear of prescribing adequate pain meds.

   It looks like the corporatist fear-mongers are launching a major literature offensive. I saw this FDA warning based on a case study from Lancet today: “Pharmacogenetics of morphine poisoning in a breastfed neonate of a codeine-prescribed mother” Gideon Koren, James Cairns, David Chitayat, Andrea Gaedigk, Steven J Leeder, The Lancet - Vol. 368, Issue 9536, 19 August 2006, Page 704

   This quickly “metabolized” into: “Breastfeeding moms taking codeine could kill their babies” ( http://tinyurl.com/yogen8 ).

   Killers! Killers! Stop the killers!

3. Comment by doctordeluca

   mlang52 wrote: “I do not understand why the 771 PAIN PATIENTS tested positive only 24% for the list of drugs. Were these patients on them by prescription, were they illegal, were they self-medicating?…”

   Hi mlang52, well, this is the sort of confusion that arises when one rushes too much to get a CME course on the market.

   You are confused, because the CME course text is wrong in this bullet-point (I should have numbered them):

   “Of 771 patients who submitted postinterview urine specimens, 24% tested positive for opioids, methadone, propoxyphene, benzodiazepines, cocaine metabolites, amphetamines, phencyclidine, barbiturates, or cannabinoids.”

   OK, working from the full text of the paper, let me review the tox proceedures and results.

   – one tox specimen was requested, collected at end of approx 2hr interview. 11 did not contribute “because of time restraints or med problem”; another 19 specimens had (unspecified) “laboratory process issues”. 801-11-19 = 771 which is where that number came from

   – Here is relevant sentence from text: “Twenty-four percent of the sample (n = 185) tested positive for cannabinoids, cocaine, and other illicit drugs.” (pg 579) Looking at Table 7 on the same page, percentage positive and number positive are given for women, men, and total, for the following drug classes: “Cannabinoids” “Cocaine” “Phencyclidine”[aka PCP aka angel dust] and “Any illicit substance”. I take the last category means any of the first three specified drugs or classes, but you can’t tell just from adding the columns in the table, because a person could be positive for more than one drug.

   – So, mlang52, I think the CME text is wrong in suggesting that opioids were tested for - at least that is not reported in the text. Nor is there any mention of urine tox for: methadone, propoxyphene, benzodiazepines, amphetamines, or barbiturates in the full text. So, they are trying to present urine tox results only for sustances that shouldn’t be present. —– [aside] - (Though a patient might be on prescribed Marinol (THC) which would give + for cannabinoids; and almost certainly many of the cannabis users were using medicinally as cannabinoids are known to potentiate the analgesic effect of opioids, as well as treating other common symptoms in pain patients, in some people - cannabis effects vary ENORMOUSLY and the dose range varies ENOROMOUSLY as well).

   – A few observations from Table 7 and text on Tox results: —– 84% of the ‘Any illicit substance’ accounted for by cannabinoids —– tox detected cannabis in 156 compared to 100 self-report —– tox detected cocaine in 60 vs. 26 by self report

   – Finally, from the text discussion of DSM-IV diagnosis proceedures (pg 577) we get this paragraph, which I think bears on your question:

   “The majority of patients who reported alcohol and drug use did not meet DSM-IV criteria for abuse or dependence. Alcohol use was reported by 35.7% (n = 286) of the sample, marijuana by 13.2% (n = 106) subjects, and cocaine by 3.2% (n = 26). All 15 subjects who reported amphetamine use (1.9%, n = 15) were taking prescription amphetamines for a mental health disorder. Forty percent of subjects reported sedative use, primarily prescription benzodiazepines, with less than 1% meeting 30-day DSM-IV criteria for sedative abuse or dependence.”

   —– I would then ask, ‘how many of the cannabis users were using medically, for pain or other medical symptom control. Similarly, how many of the alcohol users were using alcohol for insomnia or anxiety or depression - all very common in chronic pain patients.

   Thanks for a good Comment, mlang - my apologies for taking so long to respond.

   ..alex…

4. Comment by doctordeluca

   James wrote: “The four “aberrant behaviors” are likely to be correlates of the criterion variable to begin with, as the SDSS interview (although I am not familiar with it) probably contains self-reports of behavior that are very similar to the “aberrant” predictor behaviors themselves. That is, is the “prediction” actually a tautology?”

   Hello James, From the text (576-577): “We chose to use 4 of the 12 aberrant behaviors selected for the study for inclusion in logistic regression models. These four questions included a) purposely oversedating oneself, b) using opioids for nonpain reasons, c) increasing opioid dose without authorization, and d) felt intoxicated. These 4 behaviors were chosen on the basis of an analysis reported in another study focused on aberrant behaviors (Fleming, Brown, and Passik, in review, Pain). For the model, we used the total scores for the 4 behaviors (0 to 16). The final models are presented in Tables 1 and 2. Odds ratios and confidence intervals were used to assess the statistical significance of these factors. for the study for inclusion in logistic regression models.”

   I’m not sure what relevant distinction is between a) and d) - ‘purposefully oversedating’ and ‘felt intoxicated.’ Further, c) = increasing dose without authorization would be prerequisite for a) and/or d). Finally I don’t seen how c ==leads-to==> a +/or d without b) = ‘using opioids for non-pain reasons’. (in psuedomath: b+c = a AOR d) (in English: using opioids for non-pain reasons in a higher dose than scheduled leads to oversedation or other symptoms of intoxication)

   James - I’m saying that the 4 ADRBs they choose are essentially all the same thing, or rather, one implies the others. And this thing is what they plugged into their logistic regression models. And, surprise, in both ‘prediction of sub use disorders’ (BTW - ‘Use Disorders’ is correct DSM-IV terminlogy, encompassing ‘abuse’ and ‘dependence’) in table 1 and prediction of opioid use disorder in table 2, ADRBs are far and away the strongest predictors.

   This is the tautology, yes? If you have a chance and feel like it, James, could you talk a little more about the implications of this for the conclusions of the study? Is constructing a variable in this way for logistic regression a major no-no? Any ideas on how the study design might have been different so as not to have this problem arise?

   [ BTW - have made validity study of SDSS available: http://tinyurl.com/2mwrkf ]

   ..alex…

5. Comment by James Stacks

   I regret it has taken so long to get back. I traveled, slept, and had several emergencies (I have to make a living, although this is a preferred avocation!). I also struggle with RA, and I avoid opiates (I guess I am so interested in this because I realize I will someday really need them).

   I don’t think there is anything inherently wrong with using the sum of the four behavior ratings as a logistic regression predictor. In fact, the idea that they are all closely related is a good thing, rather than being a liability. That is the essence of internal consistency reliability. The sum could be a reliable measure of a very narrowly defined construct (e.g. use of prescription meds contrary to instructions to achieve psychoactive effects). It doesn’t take a rocket scientist to see why that would be expected to correlate with an opioid abuse measure, which probably contains a measure of the exact same construct! In fact, it would be expected beforehand to correlate much higher with an opioid abuse measure than with the smattering of demographic variables and other loosely related constructs analyzed. It is sort of like fatness predicting obesity much better than any one of a number of related variables such as age, education, weight, waist size, weight of clothing, caloric intake, activity, etc.

   If I am interpreting this correctly, the behavior ratings and the outcome can be reduced to a point-biserial correlation (Pearson r with one variable being dichotomous) of about .70 (or 50% shared “variance”). That is indeed large, if not very large, in the normal scheme of what is expected in this type of measurement. The point is, it is “too” large. It is most likely due the tautological relationship between the ratings and SDSS. Same predicts same. In fact, this looks almost like a high validity coefficient or a low test-retest coefficient!

   I have not read the article which justifies concentrating on these four behaviors, but I do question what value they would have for a physician. None of the behaviors could very easily be observed by the physician if the patient wanted to hide them. The behaviors are the essence of what is accepted widely as substance abuse (which is why I bet my sandwich that they will be found almost verbatim in the SDSS). Pardon my use of the terms “drug abuse” and “substance abuse”, as the authors remind us that they are diagnostically criterionless terms. I guess “aberrant behaviors” are much more objective and rigorously operationalized! You see, as a psychologist, I have a few problems with these “behaviors”. The problem is, they are not “behaviors” at all. Behaviors can be measured by some physical energy they impart on the environment. “Feeling intoxicated” is not a behavior, although reporting it may be. There is not much in the psychological theory of behaviorism for ideas like “for the purpose of” and “tried”, or, for that matter, “pain”. This is odd, because the term “behavior” as it is being used in this field has obviously been chosen for the purpose of sounding scientific!

   What is still vaguely problematic for me in the logistic regressions is the absence of overall model statistics, the low base rate problem (30 cases in the disease condition), and no positive predictive analysis for the “behaviors”. That simply leaves us wondering about the extent to which prediction really is “behaviors uber alles”, although these highly pampered and selected “behaviors” probably do uniquely correlate with the criterion.

   I also see an issue with the statement about “variables dropping out”. Regarding the other variables besides the behavior ratings, the authors state, “these variables dropped out of Table 2 because of extremely strong relations between these behaviors and opioid use disorders.” If that is what they really meant, then it does not support the “uber alles” idea. If the behavior ratings caused the other variables to loose their significance, then that means the other variables also predicted well (as a set), but the predictive ability that they shared with the behaviors got assigned to the behaviors in the final model. I don’t suspect that is what they really meant, though. That is, the statement is probably not based on a change in the coefficients upon adding the behavior ratings to the model.

   There are some other issues here. The 30 people in the opioid disorder condition are being stretched rather thin across a lot of categories. That means there may be some cells where expected frequencies are not up to snuff. If that is the case, power could be reduced substantially in the tests on the other variables. One question I would feel obliged to address in such an analysis is what happens when you remove everything except the behaviors? The other variables may be adding significantly as a set, and the overall model statistics would tell us about that. Those statistics are missing. Furthermore, interactions are not being considered, and interactions could very well be good predictors among these variables.

   More and more, I see a problem with the hypothetico-deductive protocol here. Just what was the hypothesis, and how was it reasoned and constructed? They didn’t state that the purpose of the study was the “uber alles” hypothesis. They didn’t state what the hypotheses were at all! They did say in the first sentence of the Methods section that “A study was conducted to assess the frequency of opioid use and substance use disorders in a sample of primary care patients receiving opioids for chronic pain.” Apparently, that is exactly what they did! They assessed a frequency in a sample of convenience based on a specific operationalization of a vague and elusive construct. They didn’t do much else! There is apparently no acknowledged basis for comparison of the frequency to any other number. The sample looks very different from a “general population” in many ways other than substance abuse. Little is said about the logistic regressions other than they were “exploratory”, and the variables chosen by the researcher “included a number of variables based on a priori hypotheses from the literature and the clinical experience of the PI.” No citations were given.

   I’ll work more on this later. Maybe I should try to look at those CME questions and see if they can be matched in any way with the methods or results of this study.

6. Comment by James Stacks

   I looked at the CME test questions. The first question avoids the issue of “which population?” As for the second question, with the data reported, I assume the answer they are looking for is “dose increase”. I don’t think that is technically supported by the results. The “dose increase” behavior is only one of four of the items combined for the predictor variable in the equation. The “dose increase” behavior could have zero predictive ability alone and the other three behaviors could supply all the predictive ability reported in the result. This addresses one issue with the combination of the four items which Alex asked about earlier. There is no way to single out “dose increase” as a predictor once it has been embedded with the other three “behaviors”. The predictor variable is now a composite, and I cannot attribute the predictive ability of the composite to any one behavior which made up the composite, nor can I say anything about how predictive ability is distributed among the four behaviors which were summed. Unless they run “dose increase” separately, we can’t answer question 2.

RSS feed for comments on this post. TrackBack URI

Leave a comment

If you want to leave a feedback to this post or to some other user´s comment, simply fill out the form below.