The Institute for Safe Medication Practices (ISMP) issued a report in their QuarterWatch bulletin in late January 2015 entitled “A Critique of a Key Drug Safety Reporting System”.
The report was picked up fairly widely in the media. See the NY Times.
This is a very important report and it will be covered here in detail over two postings.
The ISMP is a nonprofit organization devoted to medication error prevention and safe medication use founded over 35 years ago. See their website. Do not confuse the ISMP with the Institute of Medicine (IOM) which is part of the National Academy of Sciences and is also a non-profit, non-governmental organization. Both are superb organizations devoted to the public good.
For everyone who does drug safety and PV for a living, this report is worth reading and, perhaps more importantly, those who DO NOT do drug safety for a living should also read it. It runs about 22 pages and includes a short executive summary.
What is most interesting and, particularly of interest to those of us in the industry, is that this report is something of a bench-marking study of how manufacturers differ in handling individual case safety reports (ICSRs).
As I noted in my last posting, there is remarkable ignorance in the country about what the FDA is, who works for them, how drugs are approved, how safety and side effects are tracked and what “safe and effective” really means.
One interesting aspect of the ISMP report is that it names names. Here is a brief summary:
- The ISMP looked at the FDA FAERS system covering safety reports from March 2013 to March 2014. They examined industry submissions of ICSRs and their contents. They note that the reporting by consumers and health care professionals (HCPs) is voluntary.
- 847,039 reports were received by FDA of which 615,124 were from the US during this period. There were about 48,000 deaths – half from the US and half ex-US. Some 97% of all reports were from the manufacturers; only 3% (29,314) went directly to FDA. Most cases are for brand name products. Note that in this document I am rounding percentages to whole numbers where appropriate.
The report puts the onus for safety reporting squarely on the manufacturers: “the quality and value of the safety surveillance depends primarily on how well the manufacturers collect, code, and follow up on adverse drug events of which they become aware.” Further the ISMP notes: “the quality issues outlined in this report are a shared responsibility between the FDA, which establishes and enforces reporting requirements, and the industry, which must fulfill them.”
Methodology
- The ISMP described its methodology in a separate document. They excluded foreign reports, clinical trial reports, lawyer reports as well as a few other unusual cases. They looked at product quality and medication error reports. They used primarily the last/latest version of the case in FAERS if there were follow up reports. They note the complexity involved when looking at FDA’s quarterly FAERs data as these files often contain older reports with follow ups, as well as data entered from aggregate reports (e.g. PADERs, PSURs) that are submitted either quarterly or annually.
- There are also problems presented by naming issues and some products are grouped together for analysis by the ISMP in a “more or less arbitrary solution”. They use MedDRA and SMQs (Standardized MedDRA Queries) to find and group cases. This is also complex because of the way MedDRA is set up and the way SMQs work. They also calculate Proportional Reporting Ratios. See this document for more details.
- In the report itself, the ISMP describes their definition of “completeness”:
- “Reasonably complete” means the report had information on the patient age, gender and event date.
- “Minimally complete” means at least age and gender were complete.
Completeness
|
Reasonably Complete |
Minimally Complete |
| Direct to FDA all cases |
81% |
88% |
| Direct to FDA serious cases |
85% |
90% |
| Manufacturer cases all |
46% |
62% |
| Manufacturer cases serious* |
Not supplied |
Not supplied |
* See below for Serious, unexpected cases
Comment: This is an arbitrary definition of completeness and really does not reflect what a “medically complete” case should have. The age and sex are important. The event date is arguably less important for the medical/signal review but is important to see if the company is in compliance in terms of submitting 15 day expedited reports to FDA. Obviously, one doesn’t want a company to sit on or bury a report for months or years but in terms of signal evaluation and safety analysis this is, perhaps, less critical. One might argue that co-medications, medical history and a cogent narrative are more important for completeness and might have been included in the definition. It’s also noteworthy that companies often prioritize their resources (as does FDA) doing greater and deeper follow up and analysis on serious unexpected cases and less on non-serious or even serious expected cases.
Nonetheless, these definitions of completeness are useful.
Serious, Unexpected AEs
The ISMP correctly focused on post-marketed expedited reports: serious, unexpected AEs. Overall 49% of the expedited cases were “reasonably complete” compared to 44% of the non-expedited cases from manufacturers versus 85% reasonably complete from direct to FDA expedited cases.
Age and gender were present in 66% of the expedited cases and 58% of the non-expedited cases. The report refers to cases with “dire outcomes” (which they did not define) and noted that an even smaller percentage of cases were reasonably complete (e.g. 25% for birth defect cases).
Updated reports from manufacturers were better: 62% reasonably complete compared to 40% on initial reports.
Comment: The numbers are a bit confusing here. Some of the data is presented as a box chart without the exact values. It is also not clear how to compare the serious, unexpected data with the follow up data as the numbers do not seem to match totally. However, the take-home message here is that the FDA did pretty well (85%) and the companies did markedly less well.
What can we make of this? Well, several things. First, when an HCP gets a call from a company they may or may not jump to answer and supply follow up information. When an HCP gets a call from the US federal government, they usually do jump! The FDA receives far fewer cases than the manufacturers. In any case, the FDA does have a better “batting average” than the companies.
“Prompt” Reports (Expedited Reports)
The ISMP notes that the on-time reporting of expedited reports was 89% from manufacturers. They correctly note that “Although manufacturer reports reach the FDA quickly, this does not mean that the reports are rapidly translated into new restrictions and warnings.” They also correctly note that “Once a signal is clearly identified, a label change can often take a year or longer and involve extensive interactions with the manufacturer.”
Comment: This is a bit surprising since most companies, in my experience, take great pains to avoid late expedited reports. The numbers of on-time reports that I have heard anecdotally and that I see in most audits I do run around 96-98% unless the audit revolves around an FDA 483 citation for late cases.
Do Manufacturers Differ?
This is a clear YES. Using the reasonably complete percentage for direct-to-FDA reports of 85%, the manufacturers fell short. None attained 85%. The best numbers were reported for cases from Ariad, GE Healthcare, Vertex, United Therapeutics and Biogen. The first four companies reported between 776 and 4497 cases. Biogen reported 10,413 cases. Minimally complete cases ranged from 75% to 92%.
The high volume submitters (big pharma not surprisingly) were Pfizer, Novartis, Roche, Janssen and Amgen which all submitted between 24,000 and 36,000 cases. “Reasonably complete” cases were between 44% and 63%. Minimally complete cases ranged from 58% to 85%.
The “weakest” performers ranged form 0.9% to 16.2% and minimally complete from 11% to 83%. The ISMP report names the names but I will refrain here. If you want to see them, look at the report.
Performance over Time
The ISMP compared the 2013/2014 data with 2001 data. They noted a 6-fold increase in case volume. However, the completeness dropped: Reasonably complete manufacturers reports dropped from 59% to 48% and minimally complete from 70 to 65%.
Causality & Sources
Next the ISMP has a long discussion on the concept of causality and the inference that all spontaneous cases are possibly related. They also discuss how cases are obtained (spontaneous reports, patient contacts for marketing purposes, restricted distribution systems, patient support systems and others). This is primarily an educational section for the reader who is unfamiliar with the way such cases are handled.
Questionable Death Reports
This is a very interesting and way too brief section in the report. They note that many cases reported to FDA only have the outcome of “death” which they call an AE but is actually an outcome. If there is no other information or AE, then the death is reported, in effect, as an AE. They found that about 28% (24,939) cases had no useful information beyond the death. Overall 67% of death reports were of “limited value” with “the absence of one or more of the following: age, gender, or event date.”
Comment: This is striking and a bit surprising. Tens of thousands of death reports with no useful information. We have all seen cases where there is a death, perhaps a sudden death, and no further information is available for multiple reasons. However, the high number is a surprise. It would be good to see a further discussion of this.
Reports of Minimal Value
The ISMP noted many non-serious reports with a single minor AE such as injection site pain, “injury not otherwise specified”, “AE not otherwise specified”, nasopharyngitis, sinusitis and “no adverse event”.
Comment: The implication here is that perhaps these reports are not worth the effort to collect and process.
Event was the Underlying Disease
The ISMP noted several thousand cases of reports of the indication for treatment as the AE such as Crohn’s disease, psoriasis, and rheumatoid arthritis. It is not clear, they note, that the disease worsened but that it was simply present and recorded as an AE. They note that drugs can aggravate an underlying condition but it is not clear that this was the situation with these reports.
Comment: In my many years of auditing I have noted this and am continually surprised. I’ve not gotten rational explanations beyond those that note that “the reporter said this and we have to process it as an AE”, “It’s in our SOPs”, “we did a lot of work on this case with multiple follow up attempts and this is the conclusion we came to and we will report it to show we work well and hard”, “we are covering ourselves.” I agree it is not at all clear that this adds to the safety profile of the drug and whether it is worth devoting resources to these cases.
We are going to break here. The conclusion of this posting will appear in the next posting.
