Research Articles

Jan 17, 2018

Positive and Negative Syndrome Scale (PANSS) Training: Challenges, Solutions, and Future Directions

Authors:
Mark G. A. Opler, PhD, Christian Yavorsky, PhD, David G. Daniel, PhD

Rater training and the maintenance of the consistency of ratings are critical to ensuring reliability of study measures and sensitivity to changes in the course of a clinical trial. The Positive and Negative Syndrome Scale (PANSS) has been widely used in clinical trials of schizophrenia and other disorders and is considered the “gold standard” for assessment of antipsychotic treatment efficacy. The various features associated with training and calibration of this scale are complex, reflecting the intricacy and heterogeneity of the disorders that the PANSS is used to evaluate. In this article, the authors review the methods for ensuring reliability of the PANSS as well as a proposed trajectory for its use in the future. An overview of the current principles, implementation, technologies, and strategies for the best use of the PANSS; tips for how to achieve consistency among raters; and optimal training practices of this instrument are presented.

Oct 01, 2016

The Depression Inventory Development Workgroup: A Collaborative, Empirically Driven Initiative to Develop a New Assessment Tool for Major Depressive Disorder

Authors:
Anthony L. Vaccarino, PhD, Kenneth R. Evans, PhD, Amir H. Kalali, MD, Sidney H. Kennedy, MD, FRCPC, Nina Engelhardt, PhD, Benico N. Frey, MD, MSc, PhD, John H. Greist, MD, Kenneth A. Kobak, PhD, Raymond W. Lam, MD, FRCPC, Glenda MacQueen, MD, PhD, FRCPC, Roumen Milev, MD, PhD, FRCPC, FRCPsych, Franca M. Placenza, PhD, Arun V. Ravindran, MB, MSc, PhD, FRCPC, FRCPsych, David V. Sheehan, MD, MBA, Terrence Sills, PhD, and Janet B. Williams, PhD

The Depression Inventory Development project is an initiative of the International Society for CNS Drug Development whose goal is to develop a comprehensive and psychometrically sound measurement tool to be utilized as a primary endpoint in clinical trials for major depressive disorder. Using an iterative process between field testing and psychometric analysis and drawing upon expertise of international researchers in depression, the Depression Inventory Development team has established an empirically driven and collaborative protocol for the creation of items to assess symptoms in major depressive disorder. Depression-relevant symptom clusters were identified based on expert clinical and patient input. In addition, as an aid for symptom identification and item construction, the psychometric properties of existing clinical scales (assessing depression and related indications) were evaluated using blinded datasets from pharmaceutical antidepressant drug trials. A series of field tests in patients with major depressive disorder provided the team with data to inform the iterative process of scale development. We report here an overview of the Depression Inventory Development initiative, including results of the third iteration of items assessing symptoms related to anhedonia, cognition, fatigue, general malaise, motivation, anxiety, negative thinking, pain and appetite. The strategies adopted from the Depression Inventory Development program, as an empirically driven and collaborative process for scale development, have provided the foundation to develop and validate measurement tools in other therapeutic areas as well.

Feb 01, 2016

Data Quality Monitoring in Clinical Trials: Has It Been Worth It? An Evaluation and Prediction of the Future by All Stakeholders

Authors:
David Daniel, MD, Amir Kalali, MD, Mark West, David Walling, PhD, Dana Hilt, MD, Nina Engelhardt, PhD, Larry Alphs, MD, PhD, Antony Loebel, MD, Kim Vanover, PhD, Sarah Atkinson, MD, Mark Opler, PhD, MPH, Gary Sachs, MD, Kari Nations, PhD, and Chris Brady, PsyD

This paper summarizes the results of the CNS Summit Data Quality Monitoring Workgroup analysis of current data quality monitoring techniques used in central nervous system (CNS) clinical trials. Based on audience polls conducted at the CNS Summit 2014, the panel determined that current techniques used to monitor data and quality in clinical trials are broad, uncontrolled, and lack independent verification. The majority of those polled endorse the value of monitoring data. Case examples of current data quality methodology are presented and discussed. Perspectives of pharmaceutical companies and trial sites regarding data quality monitoring are presented. Potential future developments in CNS data quality monitoring are described. Increased utilization of biomarkers as objective outcomes and for patient selection is considered to be the most impactful development in data quality monitoring over the next 10 years. Additional future outcome measures and patient selection approaches are discussed.

Jan 01, 2016

Norms for Healthy Adults Aged 18–87 Years for the Cognitive Drug Research System: An Automated Set of Tests of Attention, Information Processing and Memory for Use in Clinical Trials

Authors:
Keith A Wesnes, Cynthia McNamara, Peter Annas

The Cognitive Drug Research (CDR) System is a set of nine computerized tests of attention, information processing, working memory, executive control and episodic memory which was designed for repeated assessments in research projects. The CDR System has been used extensively in clinical trials involving healthy volunteers for over 30 years, and a database of 7751 individuals aged 18–87 years has been accumulated for pre-treatment data from these studies. This database has been analysed, and the relationships between the various scores with factors, including age, gender and years of full-time education, have been identified. These analyses are reported in this paper, along with tables of norms for the various key measures from the core tasks stratified by age and gender. These norms can be used for a variety of purposes, including the determination of eligibility for participation in clinical trials and the everyday relevance of research findings from the system. In addition, these norms provide valuable information on gender differences and the effects of normal ageing on major aspects of human cognitive function.

Source:
Wesnes, K, McNamara, C, Annas, P. Norms for Healthy Adults Aged 18 to 87 Years for the CDR System: An Automated Set of Tests of Attention, Information Processing and Memory for Use in Clinical Trials. J Psychopharmacol. 2016;30(3): 263-272.
Dec 01, 2015

Establishing Equivalence of Electronic Clinician-Reported Outcome Measures

Authors:
Rebecca L. M. Fuller, PhD, Cynthia W. McNamara, PhD, William R. Lenderking, PhD, Chris Edgar, PhD, Angela Rylands, PhD, Todd Feaster, PsyD, Donald Sabatino, MA, and David S. Miller, MD

Background: Electronic administration of clinician-reported outcomes (eClinROs) has advantages over paper-based methods, but the mode of administration change has the potential to affect the validity of the scale. The literature on migration of patient reported outcomes (PROs) suggests that there are different levels of modification, which necessitate different approaches to demonstrating mode equivalence. However, little has been written on the migration of ClinROs to electronic administration. Methods: We propose a method of comparing paper and electronic versions of scales that includes a comparison based on content and a comparison based on format. The determination of whether the eClinRO has undergone minor, moderate, or substantial modification will drive the necessary studies required for validation. The unique characteristics of ClinROs suggest 2 additional types of modifications, including functionality adaptation and adaptation of instructions. Conclusions: In many respects, the migration of a ClinRO to electronic administration is similar to that of a PRO. This article has explored the ways in which there might be special considerations for ClinROs that have not been elaborated for PROs

Dec 01, 2013

Reliability of the Alzheimer’s Disease Assessment Scale (ADAS-Cog) in Longitudinal Studies

Authors:
Anzalee Khan, Christian Yavorsky, Guillermo Di Clemente, Mark Opler, Stacy Liechti, Brian Rothman and Sofija Jovic

Background: Considering the scarcity of longitudinal assessments of reliability, there is need for a more precise understanding of cognitive decline in Alzheimer’s Disease (AD). The primary goal was to assess longitudinal changes in inter-rater reliability, test retest reliability and internal consistency of scores of the ADAS-Cog. Methods: 2,618 AD subjects were enrolled in seven randomized, double-blind, placebo-controlled, multicenter-trials from 1986 to 2009. Reliability, internal-consistency and cross-sectional analysis of ADAS-Cog and MMSE across seven visits were examined. Results: Intra-class correlation (ICC) for ADAS-Cog was moderate to high supporting their reliability. Absolute Agreement ICCs 0.392 (Visit-7) to 0.806 (Visit-2) showed a progressive decrease in correlations across time. Item analysis revealed a decrease in item correlations, with the lowest correlations for Visit 7 for Commands (ICC=0.148), Comprehension (ICC=0.092), Spoken Language (ICC=0.044). Discussion: Suitable assessment of AD treatments is maintained through accurate measurement of clinically significant outcomes. Targeted rater education of ADAS-Cog items over-time can improve ability to administer and score the scale.

Source:
Khan, A, Yavorsky, C, Di Clemente, G, Opler, M, Liechti, S, Rothman, B, et al. Reliability of the Alzheimer’s Disease Assessment Scale (ADAS-Cog) in Longitudinal Studies. Current Alzheimer Research. 2013;10(9):952-63.
Jul 01, 2013

A New Integrated Negative Symptom Structure of the Positive and Negative Syndrome Scale (PANSS) in Schizophrenia Using Item Response Analysis

Authors:
Anzalee Khan, Jean-Pierre Lindenmayer, Mark Opler, Christian Yavorsky, Brian Rothman, Luka Lucic

Background: Debate persists with regard to how best to categorize the syndromal dimension of negative symptoms in schizophrenia. The aim was to first review published Principle Components Analysis (PCA) of the PANSS, and extract items most frequently included in the negative domain, and secondly, to examine the quality of items using Item Response Theory (IRT) to select items that best represent a measurable dimension (or dimensions) of negative symptoms. Methods: First, 22 factor analyses and PCA met were included. Second, using a large dataset (n = 7187) of participants in clinical trials with chronic schizophrenia, we extracted items loading on one or more PCA. Third, items not loading with a value of ≥0.5, or loading on more than one component with values of ≥0.5 were discarded. Fourth, resulting items were included in a non-parametric IRT and retained based on Option Characteristic Curves (OCCs) and Item Characteristic Curves (ICCs). Results: 15 items loaded on a negative domain in at least one study, with Emotional Withdrawal loading on all studies. Non-parametric IRT retained nine items as an Integrated Negative Factor: Emotional Withdrawal, Blunted Affect, Passive/Apathetic Social Withdrawal, Poor Rapport, Lack of Spontaneity/Conversation Flow, Active Social Avoidance, Disturbance of Volition, Stereotyped Thinking and Difficulty in Abstract Thinking. Conclusions: This is the first study to use a psychometric IRT process to arrive at a set of negative symptom items. Future steps will include further examination of these nine items in terms of their stability, sensitivity to change, and correlations with functional and cognitive outcomes.

Source:
Khan, A, Lindenmayer J-P, Opler, M, Yavorsky, C, Rothman, B, Lucic, L. A New Integrated Negative Symptom Structure of the Positive and Negative Syndrome Scale (PANSS) in Schizophrenia Using Item Response Analysis. Schizophrenia Research. 2013;150(1):185-96.
Feb 01, 2013

Assessing the Sources of Unreliability (Rater, Subject, Time-Point) in a Failed Clinical Trial Using Items of the Positive and Negative Syndrome Scale (PANSS)

Authors:
Anzalee Khan, PhD, William Christian Yavorsky, PhD, Stacy Liechti, PhD, Guillermo Di Clemente, PhD, Brian Rothman, PhD, Mark Opler, PhD, MPH, Ashleigh DeFries, MA, and Sofija Jovic, PhD

Background: Considering the increasing attention to the study of failed clinical trials, the goal of this study was to identify the sources of unreliability in a failed clinical trial by assessing scores on the Positive and Negative Syndrome Scale (PANSS). Methods: This study is a substudy from a failed phase 2 double-blind, placebo-controlled trial of schizophrenia. Using the generalizability theory, this substudy assesses reliability on 3 conditions: raters, time points (PANSS evaluations, 1 week apart), subjects for 3 groups (placebo responders, placebo nonresponders, and treatment group). Results: The placebo response rate was 40.07% (32/71). For all PANSS positive symptom items, the most variability was for raters (range, 33%Y72%) for the placebo responders, 31% to 68% for the placebo nonresponders, and 29% to 60% for the treatment group. The variability of the interaction of rater and time point was the second source of unreliability, with an average of 12.28% compared to 12.00% for the placebo nonresponders and 10.00% for the treatment group. All items of the negative symptom subscale showed the most percent variability for raters, for all groups. For general psychopathology items (except preoccupation), raters accounted for the most variability in the scores for placebo responders with an average of 51.00% across items. A similar pattern was observed for the placebo nonresponders and for the treatment group; for the treatment group, the interaction between rater and time point accounted for the most variability for somatic concern and anxiety. Conclusions: Results confirm the efficacy of applying the generalizability theory to the estimation of reliability to identify a source of unreliability and provide evidence for the relationship between low reliability and failed trials. Findings can be used to guide data monitoring, rater training, and identification of PANSS items, which may require supplementary training.

Source:
Khan, A, Yavorsky, C, Liechti, S, Di Clemente, G, Rothman, B, Opler, M, et al. Assessing the Sources of Unreliability (Rater, Subject, Time-Point) in a Failed Clinical Trial Using Items of the Positive and Negative Syndrome Scale (PANSS). Journal of Clinical Psychopharmacology. 2013;33(1):109-17.

Scientific Posters

Feb 26, 2018

The Rater Applied Performance Scale: Evaluating Clinical Interview Skill via Audio Recordings of MADRS Assessments in a Clinical Drug Trial

Authors:
Engelhardt N, Yavorsky C, McNamara C, Wolanski K, Burger F, Di Clemente G
Source:
International Society for CNS Clinical Trials and Methodology (ISCTM) 14th Annual Scientific Meeting, Washington, DC, USA (February 2018)
Nov 30, 2017

Can the MADRS Measure Rapid Changes in Depressive Symptoms in Response to Esketamine Treatment?

Authors:
Yavorsky C, Singh J, Engelhardt N, McNamara C, Di Clemente G
Source:
American College of Neuropsychopharmacology (ACNP) 56th Annual Meeting, Palm Springs, CA, USA (December 2017)
Sep 04, 2017

Psychometric Data Monitoring in Clinical Trials: Can Disparities Between Patient and Clinician Reported Outcomes Predict Measurement Error?

Authors:
Yavorsky C, Engelhardt N, McNamara C, Di Clemente G
Source:
International Society For CNS Clinical Trials And Methodology (ISCTM) 13th Annual Scientific Meeting, Washington, DC, USA (February 2017)
May 30, 2017

Identification of Elevated and Irritable Subtypes in Bipolar I Disorder in a Treatment Sample: What are the Differences in Treatment Response?

Authors:
Yavorsky C, Cox K, Burger F, Ivanova E, McNamara C
Source:
International Society for Bipolar Disorders (ISBD) 19th Annual Conference, Washington, DC, USA (May 2017)
Oct 31, 2016

Are there Reporting Differences at Screening on the Montgomery-Åsberg Depression Rating Scale (MADRS) Between Older and Younger Adults when Using Remote Assessments?

Authors:
Yavorsky C, Director M, Goldensohn I, Engelhardt N, McNamara C
Source:
CNS Summit (Collaborating for Novel Solutions), Boca Raton, FL, USA (October 2016)
Apr 06, 2016

Effectivity of a Digital PANSS-Training

Authors:
Klaasen N, van Asperen L, Kos C, Mulder PJ, Knegtering H, Wolanski K, Yavorski C, Aleman A, Bruggeman R
Source:
Schizophrenia International Research Society (SIRS) Biennial Meeting, Florence, Italy (April 2016)
Feb 18, 2016

Precision of Estimating the Sample Size Needed to Power Clinical Trials: What is the Effect of Risk-based Monitoring?

Authors:
Yavorsky C, Meares K, McNamara C, Di Clemente G, Burger F
Source:
International Society For CNS Clinical Trials And Methodology (ISCTM) 12th Annual Scientific Meeting, Washington, DC, USA (February 2016) and American College of Neuropsychopharmacology (ACNP) 55th Annual Meeting, Hollywood, FL, USA (December 2016)
Feb 18, 2016

Assessing Clinician’s Subjective Experience with Psychometric Tools for Suicide Assessment

Authors:
Dallabrida S., Yavorsky C.
Source:
International Society For CNS Clinical Trials And Methodology (ISCTM) 12th Annual Scientific Meeting, Washington, DC, USA (February 2016)
Oct 30, 2015

What can Data Monitoring Tell us About Where to Focus Efforts to Remediate Problematic Scoring of the Positive and Negative Syndrome Scale?

Authors:
Engelhardt N, Masotti M, Wolanski K, Burger F, Yavorsky C, Di Clemente G
Source:
American Society for Clinical Pathology (ASCP) Annual Meeting, Long Beach, CA, USA (October 2015)
Mar 30, 2015

Testing the Results of Risk-based Data-monitoring Algorithms for the PANSS: What is the Predictive Ability of this Approach Given Empirical Results in a Pooled Sample from Eight Phase III Schizophrenia Trials?

Authors:
Yavorsky C, Tran L, Di Clemente G, Geiger M, Meares K, Engelhardt N
Source:
International Congress on Schizophrenia Research, Colorado Springs, CO, USA (March 2015)
Feb 19, 2015

What is the Empirical Evidence for the Sensitivity of Data Monitoring Algorithms: Results from Pooled Data across Eight Schizophrenia Trials.

Authors:
Yavorsky C, Tran L, Engelhardt N
Source:
International Society For CNS Clinical Trials And Methodology (ISCTM) 11th Annual Scientific Meeting, Washington, DC, USA (February 2015)
Feb 20, 2014

High Fidelity: What is the Real Impact of a Data Monitoring Program on Data Quality?

Authors:
Yavorsky C, Di Clemente G, Wolanski K, Burger F
Source:
International Society For CNS Clinical Trials And Methodology (ISCTM) 10th Annual Scientific Meeting, Washington, DC, USA (February 2014)