Advertisement

Replacing performance status with a simple patient-reported outcome in palliative radiotherapy prognostic modelling

Open AccessPublished:October 01, 2022DOI:https://doi.org/10.1016/j.ctro.2022.09.008

      Highlights

      • The inclusion of clinician-assessed performance status in prognostic tools restricts their use to specialist settings.
      • We investigate the use of patient-reported outcome measures in replacing clinician-assessed performance status in prognostic models.
      • Patient-reported outcome measures are likely to be at least as useful in prognostic modelling as clinician-assessed performance status.
      • Larger, more contemporaneous datasets, with broader patient populations are needed to confirm these results and the acceptability of this approach to patients and clinicians.

      Abstract

      Background and purpose

      Prognostication is key to determining care in advanced incurable cancer. Although performance status (PS) has been shown to be a strong prognostic predictor, inter-rater reliability is limited, restricting models to specialist settings. This study assessed the extent to which a simple patient-reported outcome measure (PROM), the EQ-5D, may replace PS for prognosis of patients with bone metastases.

      Materials and methods

      Data from 1,011 patients in the Dutch Bone Metastasis Study were used. Cox proportional hazards models were developed to investigate the prognostic value of models incorporating PS alone, the EQ-5D SC dimension alone, all EQ-5D dimensions and EQ–VAS, and finally all dimensions and PS. Three prognostic groups were identified and performance assessed using the Harrell’s C-index and Altman-Royston index of separation.

      Results

      Replacing performance status (PS) with the self-care (SC) dimension of the EQ-5D provides similar model performance. In our SC-based model, three groups are identified with median survival of 86 days (95 % CI 76–101), 174 days (95 % CI 145–213), and 483 days (95 % CI 431–539). Whilst not statistically significantly different, the C-index was 0.706 for the PS-only model, 0.718 for SC-only and 0.717 in our full model, suggesting patient-report outcome models perform as well as that based on PS.

      Conclusion

      Prognostic performance was similar across all models. The SC model provides prognostic value similar to that of PS, particularly where a prognosis of<6 months is considered. Larger, more contemporaneous studies are needed to assess the extent to which PROMs may be of prognostic value, particularly where specialist assessment is less feasible.

      Introduction

      Estimates of expected prognosis play a key role in clinical decision-making in advanced incurable cancer. It is, however, well documented that oncologists’ predictions of prognosis are often inaccurate and specifically tend to be overly optimistic. [

      Glare P. A systematic review of physicians’ survival predictions in terminally ill cancer patients. BMJ. 2003 Jul 24;327(7408):195–0.

      ,
      • Chow E.
      • Davis L.
      • Panzarella T.
      • Hayter C.
      • Szumacher E.
      • Loblaw A.
      • et al.
      Accuracy of survival prediction by palliative radiation oncologists.
      ] It has been suggested that optimistic prognostic estimates may play a role in overly aggressive care near the end of life. [
      • Gripp S.
      • Mjartan S.
      • Boelke E.
      • Willers R.
      Palliative radiotherapy tailored to life expectancy in end-stage cancer patients: reality or myth?.
      ,
      • Tseng Y.D.
      • Krishnan M.S.
      • Sullivan A.J.
      • Jones J.A.
      • Chow E.
      • Balboni T.A.
      How radiation oncologists evaluate and incorporate life expectancy estimates into the treatment of palliative cancer patients: a survey-based study.
      ,
      • Weeks J.C.
      • Cook E.F.
      • O’Day S.J.
      • Peterson L.M.
      • Wenger N.
      • Reding D.
      • et al.
      Relationship Between Cancer Patients’ Predictions of Prognosis and Their Treatment Preferences.
      ] As such, a number of prognostic models have been developed in order to support improved prognostication and thus better informed treatment decisions. [
      • Chow E.
      • Abdolell M.
      • Panzarella T.
      • Harris K.
      • Bezjak A.
      • Warde P.
      • et al.
      Predictive model for survival in patients with advanced cancer.
      ,
      • Dosani M.
      • Tyldesley S.
      • Bakos B.
      • Hamm J.
      • Kong T.
      • Lucas S.
      • et al.
      The TEACHH model to predict life expectancy in patients presenting for palliative spine radiotherapy: external validation and comparison with alternate models.
      ,
      • Westhoff P.G.
      • de Graeff A.
      • Monninkhof E.M.
      • Bollen L.
      • Dijkstra S.P.
      • van der Steen-Banasik E.M.
      • et al.
      An Easy Tool to Predict Survival in Patients Receiving Radiation Therapy for Painful Bone Metastases.
      ,
      • Bollen L.
      • van der Linden Y.M.
      • Pondaag W.
      • Fiocco M.
      • Pattynama B.P.M.
      • Marijnen C.A.M.
      • et al.
      Prognostic factors associated with survival in patients with symptomatic spinal bone metastases: a retrospective cohort study of 1 043 patients.
      ] Multiple models have been developed in palliative radiotherapy where the decision to treat and to fractionate may vary with the expected prognosis of the patient.
      Specialised radiotherapy treatment centres are, however, usually located in large cities. As a consequence, many patients will first be cared for by local non-specialist healthcare providers, often without specific experience in the assessment of prognosis in advanced incurable cancer. Travel to a treating centre, and the disruption associated with this, may therefore be undertaken before such an assessment takes place. Alternative prognostic models might help to avoid this in patients very close to the end of life. Whilst previously developed models vary, they broadly rely upon a number of key variables with primary cancer diagnosis and performance status being particularly notable for their presence in a wide range of models. [
      • Bollen L.
      • van der Linden Y.M.
      • Pondaag W.
      • Fiocco M.
      • Pattynama B.P.M.
      • Marijnen C.A.M.
      • et al.
      Prognostic factors associated with survival in patients with symptomatic spinal bone metastases: a retrospective cohort study of 1 043 patients.
      ].
      Performance status is a measure of a patient’s ability to carry out their activities of daily living. It has been demonstrated to be a strong predictor of prognosis, however, inter-rater reliability of performance status has been found to be very mixed. [
      • Chow R.
      • Chiu N.
      • Bruera E.
      • Krishnan M.
      • Chiu L.
      • Lam H.
      • et al.
      Inter-rater reliability in performance status assessment among health care professionals: a systematic review. Ann.
      ] Notably, concordance appears to be particularly reduced in comparisons between specialisations e.g. general or palliative care physician versus radiation oncologist. [
      • Chow R.
      • Chiu N.
      • Bruera E.
      • Krishnan M.
      • Chiu L.
      • Lam H.
      • et al.
      Inter-rater reliability in performance status assessment among health care professionals: a systematic review. Ann.
      ] This may in part reflect the routine use of performance status by oncologists distinct from its limited use in other medical and surgical specialities. It is notable that models incorporating performance status in the assessment of prognosis following palliative radiotherapy have relied upon performance status as determined by the treating oncologist. As such, within these models, the challenges of inter-rater reliability are likely to be reduced. [
      • May C.H.
      • Lester J.F.
      • Lee S.
      PS score discordance and why it matters.
      ] The impact of incorporating performance status, as assessed by a wider clinical team, upon prognostic model performance is not clear. If an increase in measurement error (due to inter-rater reliability) is seen, model discrimination may deteriorate, reducing validity outside of the oncology clinic. [
      • Pajouheshnia R.
      • van Smeden M.
      • Peelen L.M.
      • Groenwold R.H.H.
      How variation in predictor measurement affects the discriminative ability and transportability of a prediction model.
      ] As such, developing prognostic models which are not reliant upon performance status will be valuable. Particularly in a setting where patients with very limited prognosis might experience significant disruption to undergo assessment, and potentially a subsequent intervention with limited benefit, very close to the end of life.
      Patient reported outcomes have been demonstrated previously to have prognostic value in a range of diagnoses, including advanced incurable cancer. [
      • Park S.M.
      • Park M.H.
      • Won J.H.
      • Lee K.O.
      • Choe W.S.
      • Heo D.S.
      • et al.
      EuroQol and survival prediction in terminal cancer patients: a multicenter prospective study in hospice-palliative care units.
      ,
      • Diouf M.
      • Chibaudel B.
      • Filleron T.
      • Tournigand C.
      • Hug de Larauze M.
      • Garcia-Larnicol M.-L.
      • et al.
      Could baseline health-related quality of life (QoL) predict overall survival in metastatic colorectal cancer? The results of the GERCOR OPTIMOX 1 study.
      ,
      • Clarke P.M.
      • Hayes A.J.
      • Glasziou P.G.
      • Scott R.
      • Simes J.
      • Keech A.C.
      Using the EQ-5D Index Score as a Predictor of Outcomes in Patients With Type 2 Diabetes.
      ,
      • Mathews W.C.
      • May S.
      EuroQol (EQ-5D) measure of quality of life predicts mortality, emergency department utilization, and hospital discharge rates in HIV-infected adults under care.
      ] There is also good reason to think that these outcomes might align with performance status; the measurement of performance status relies upon an assessment of an individual’s ability to carry out daily activities, mobilise and perform self-care tasks. [

      ECOG Performance Status [Internet]. ECOG-ACRIN. [cited 2020 Aug 10]. Available from: https://ecog-acrin.org/resources/ecog-performance-status.

      ] These dimensions align closely with a range of patient reported outcome questionnaires and specifically are key to the EuroQoL group’s EQ-5D questionnaire. [

      EuroQol Group. EQ-5D-3L [Internet]. [cited 2018 Oct 4]. Available from: https://euroqol.org/eq-5d-instruments/eq-5d-3l-about/.

      ].
      Given the potential limitations to the use of performance status where prognostic models are used beyond the oncology clinic this study uses data collected within the Dutch Bone Metastasis Study to assess the extent to which patient-reported EQ-5D questionnaire results may be able to replace clinician assessed performance status in a patient-assessed prognostic model. [
      • Steenland E.
      • Leer J.
      • van Houwelingen H.
      • Post W.J.
      • van den Hout W.B.
      • Kievit J.
      • et al.
      The effect of a single fraction compared to multiple fractions on painful bone metastases: a global analysis of the Dutch Bone Metastasis Study.
      ,
      • van den Hout W.B.
      • van der Linden Y.M.
      • Steenland E.
      • Wiggenraad R.G.J.
      • Kievit J.
      • de Haes H.
      • et al.
      Single- versus multiple-fraction radiotherapy in patients with painful bone metastases: cost-utility analysis based on a randomized trial.
      ].

      Materials and methods

      The DBMS dataset contains information on 1,157 patients treated within a randomised study assessing dose fractionation of palliative radiotherapy for uncomplicated painful bone metastases between 1996 and 1998. 146 patients with missing data within the baseline EQ-5D questionnaire were excluded, leaving 1,011 patients (complete case analysis). No significant differences were observed (at the 10 % level) in the survival time or baseline characteristics of included and excluded individuals. 264, or 26.1 % of, the patients were censored for survival (median follow-up time of censored patients 425 days). The median survival in the full sample was 188 days, compared to 190 days after exclusion of observations exhibiting missingness on the EQ-5D.
      The EQ-5D provides a simple measure including five dimension questions assessing an individual’s mobility, self-care, usual activities, pain/discomfort, and anxiety/depression. Each dimension of the EQ-5D in our dataset includes three levels of response (no, some, or severe problems). In addition, a 0–100 visual analogue scale (EQ-VAS) allows patients to report their self-rated overall health.
      Multivariable Cox regression was used to estimate models incorporating Karnofsky performances status, the three-level EQ-5D (based on mailed patient-completed questionnaires [
      • van den Hout W.B.
      • van der Linden Y.M.
      • Steenland E.
      • Wiggenraad R.G.J.
      • Kievit J.
      • de Haes H.
      • et al.
      Single- versus multiple-fraction radiotherapy in patients with painful bone metastases: cost-utility analysis based on a randomized trial.
      ,
      • van der Linden Y.M.
      • Lok J.J.
      • Steenland E.
      • Martijn H.
      • van Houwelingen H.
      • Marijnen C.A.M.
      • et al.
      Single fraction radiotherapy is efficacious: a further analysis of the Dutch Bone Metastasis Study controlling for the influence of retreatment.
      ], cancer diagnosis and disease extent. The palliative radiotherapy regimen used for bone metastases has been shown not to influence survival and was therefore not included within these models. [
      • Westhoff P.G.
      • de Graeff A.
      • Monninkhof E.M.
      • Bollen L.
      • Dijkstra S.P.
      • van der Steen-Banasik E.M.
      • et al.
      An Easy Tool to Predict Survival in Patients Receiving Radiation Therapy for Painful Bone Metastases.
      ,
      • Steenland E.
      • Leer J.
      • van Houwelingen H.
      • Post W.J.
      • van den Hout W.B.
      • Kievit J.
      • et al.
      The effect of a single fraction compared to multiple fractions on painful bone metastases: a global analysis of the Dutch Bone Metastasis Study.
      ] Models based on combinations of these variables are used to predict median survival times, with individuals grouped into terciles according to these survival times, resulting in three prognostic groups for each model [
      • Chow E.
      • Abdolell M.
      • Panzarella T.
      • Harris K.
      • Bezjak A.
      • Warde P.
      • et al.
      Predictive model for survival in patients with advanced cancer.
      ].
      A baseline model incorporates only the variables used by Chow et al [
      • Chow E.
      • Abdolell M.
      • Panzarella T.
      • Harris K.
      • Bezjak A.
      • Warde P.
      • et al.
      Predictive model for survival in patients with advanced cancer.
      ]: a dichotomised indicator of Karnovsky performance score (<=60 vs > 60), the site of primary cancer diagnosis (breast, prostate, lung, other), and the presence of visceral metastases.
      Further models incorporate these latter two variables as well as varying combinations of performance status, the EQ-5D dimensions and EQ-VAS [

      EuroQol Group. EQ-5D-3L [Internet]. [cited 2018 Oct 4]. Available from: https://euroqol.org/eq-5d-instruments/eq-5d-3l-about/.

      ].
      The first alternative model aimed to use the EQ-5D dimensions to develop a maximally parsimonious model [
      • Chow E.
      • Abdolell M.
      • Panzarella T.
      • Harris K.
      • Bezjak A.
      • Warde P.
      • et al.
      Predictive model for survival in patients with advanced cancer.
      ]. In doing so, we initially estimated the relationship between all dimensions, retaining a single dimension with the greatest significance in multivariable Cox proportional hazards regression. This resulted in a model which used only the self-care EQ-5D dimension.
      Further alternative EQ-5D based models incorporated only the EQ-VAS and separately all EQ-5D dimensions (as categorical variables) and the EQ-VAS in order to examine the additional benefits available from these more complex models. The incremental benefit of including performance status over and above the EQ-5D questionnaire information was assessed for each EQ-5D based model, and a final model includes all information included in any of our models. Table 1 summarises variable inclusion.
      Table 1Variables incorporated into alternative model specifications.
      ModelPrimary site of diagnosisNon-bone metastasesKarnovsky performance scoreEQ-5D self-care onlyEQ-5D all dimensions

      EQ-VAS
      ps_only
      sc_only
      sc_ps
      alleq5d_vas
      vas_only
      vas_ps
      full
      We present two diagnostic measures of model performance for all models: the Harrell’s C-index and the Altman-Royston index of separation. [
      • Royston P.
      • Sauerbrei W.
      A new measure of prognostic separation in survival data.
      ,
      • Altman D.G.
      • Royston P.
      What do we mean by validating a prognostic model?.
      ] The Harrell’s C-index quantifies the extent to which the model accurately allocates pairs of individuals to appropriate groups (discrimination). The Altman-Royston index of separation gives an indication of the meaningfulness of the three groups produced by each model, defined by the mean survival probability to a particular point in time in the group with the highest survival probability minus that of the group with the lowest survival probability. We calculate this value at intervals of 50 days in order to assess whether the most preferred model may vary depending on the period of prognosis required.
      Rather than present a single split of the sample into training and validation, we carry out bootstrap resampling with replacement of the observed data. This involves creating 10,000 datasets of 1,011 sampled observations. We then split each bootstrap sample into training and validation (assigning as close to 50 % to each as is possible given the odd number of observations in our sample), and present diagnostic statistics on these in order to characterise uncertainty. This guards against overfitting, and allows internal validation of our results [
      • Riley R.D.
      • van der Windt D.
      • Croft P.
      • Prognosis M.K.
      Research in Healthcare: Concepts, Methods, and Impact.
      ].

      Results

      Table 2 presents baseline characteristics of the 1,011 included patients. Median survival time for these 1,011 individuals was 190 days (95 % CI 176–209). Based on the baseline performance status model, the observed survival times for the three groups in this population were 95 days (95 % CI 79–106), 210 days (95 % CI 181–245) and 515 days (95 % CI 458–572) for groups 1, 2 and 3 respectively (model C-statistic 0.706).
      Table 2Baseline characteristics of the study population. N = 1,011.
      VariableMean / nsd / %
      Age (years)64.7811.27
      Primary site of diagnosis
      Breast39138.67 %
      Prostate23923.64 %
      Lung24624.33 %
      Other13513.35 %
      Metastatic sites
      Bone metastases only73772.90 %
      Non-bone metastases27427.10 %
      Performance status
      KPS ≤ 6029429.08 %
      KPS > 6071770.92 %
      EQ-5D mobility
      153853.21 %
      236335.91 %
      311010.88 %
      EQ-5D self-care
      151250.64 %
      239639.17 %
      310310.19 %
      EQ-5D usual activities
      113513.35 %
      242942.43 %
      344744.21 %
      EQ-5D pain and discomfort
      1232.27 %
      263462.71 %
      335435.01 %
      EQ-5D anxiety and depression
      145044.51 %
      247947.38 %
      3828.11 %
      EQ-VAS
      <10454.45 %
      10–19949.30 %
      20–2915515.33 %
      30–3912212.07 %
      40–4917217.01 %
      50–5914113.95 %
      60–6911711.57 %
      70–7910410.29 %
      80+616.03 %
      When replacing PS in the multi-variable Cox model, the EQ-5D self-care dimensions is found to be a significant predictor of survival probability (p < 0.001). Based on this model the median survival time for group 1 was 86 days (95 % CI 76–101), for group 2 174 days (95 % CI 145–213), and for group 3 483 days (95 % CI 431–539) (C-statistic 0.718). For the baseline (PS only) and SC-only models we present predicted and observed survival curves by group in Fig. 1. Bootstrapped uncertainty around our survival curves is presented in Fig. 2. Compared to those reporting a level of 1 in the EQ-5D self-care dimension, in our SC-only model, individuals reporting levels 2 and 3 have an elevation in their hazard of death of around 47 % and 95 % respectively (p < 0.001).
      Figure thumbnail gr1
      Fig. 1Kaplan-Meier survival curves in observed data and model predicted survival in a) PS-only and b) SC-only models.
      Figure thumbnail gr2
      Fig. 2Estimated out of sample survival for a) PS-only and b) SC-only models (solid lines, point estimates; dashed lines, 95% confidence intervals).
      Table 3 presents the model estimates for all considered models and Harrel’s C-statistic of each. For all variables in our EQ-5D + EQ-VAS model with significance of p < 0.05, we observe relationships with the hazard of death in expected directions – i.e. with worse self-reported health implying a higher hazard of survival. In this model, only one level of the mobility dimension is found to be significant at the 5 % level, with self-care continuing to offer the strongest such relevance.
      Table 3Cox proportional hazards model estimated coefficients from all 1,011 observations in our estimation sample.
      ps_onlysc_onlysc_psvas_onlyvas_psalleq5d_vasfull
      HR / 95 % CI / pHR / 95 % CI / pHR / 95 % CI / pHR / 95 % CI / pHR / 95 % CI / pHR / 95 % CI / pHR / 95 % CI / p
      Primary diagnosis (breast omitted)
      Prostate2.1922.1052.2282.0692.2352.1952.28
      [1.779,2.700][1.712,2.587][1.809,2.744][1.681,2.547][1.812,2.758][1.775,2.715][1.842,2.823]
      (<0.001)(<0.001)(<0.001)(<0.001)(<0.001)(<0.001)(<0.001)
      Lung4.6844.8074.8444.6214.7455.0415.012
      [3.853,5.694][3.954,5.844][3.981,5.895][3.803,5.615][3.899,5.774][4.126,6.160][4.101,6.125]
      (<0.001)(<0.001)(<0.001)(<0.001)(<0.001)(<0.001)(<0.001)
      Other3.5333.8673.8333.3563.4994.0443.92
      [2.805,4.449][3.062,4.884][3.035,4.841][2.662,4.232][2.776,4.412][3.182,5.140][3.085,4.980]
      (<0.001)(<0.001)(<0.001)(<0.001)(<0.001)(<0.001)(<0.001)
      Site of metastases (bone only omitted)
      Bone & non-bone1.5271.4971.511.6361.6041.571.579
      [1.291,1.806][1.264,1.772][1.276,1.788][1.379,1.942][1.352,1.904][1.321,1.866][1.328,1.877]
      (<0.001)(<0.001)(<0.001)(<0.001)(<0.001)(<0.001)(<0.001)
      Performance status (>60 omitted)
      KPS <=602.1761.7682.021.673
      [1.864,2.542][1.481,2.111][1.718,2.376][1.392,2.011]
      (<0.001)(<0.001)(<0.001)(<0.001)
      EQ-5D self-care (1omitted)
      21.7141.4981.3461.25
      [1.468,2.003][1.273,1.763][1.124,1.612][1.041,1.501]
      (<0.001)(<0.001)(0.001)(0.017)
      32.5831.731.7121.307
      [2.043,3.266][1.325,2.257][1.274,2.300][0.959,1.781]
      (<0.001)(<0.001)(<0.001)(0.090)
      EQ-5D mobility (1 omitted)
      21.3011.196
      [1.093,1.548][1.002,1.426]
      −0.003−0.047
      31.3191.287
      [0.994,1.749][0.972,1.703]
      (0.055)(0.078)
      EQ-5D usual activities (1 omitted)
      21.4941.445
      [1.131,1.972][1.096,1.905]
      (0.005)(0.009)
      31.5921.488
      [1.172,2.162][1.096,2.019]
      (0.003)(0.011)
      EQ-5D pain and discomfort (1 omitted)
      20.8120.855
      [0.483,1.366][0.510,1.434]
      (0.433)(0.553)
      30.9951.052
      [0.580,1.708][0.615,1.798]
      (0.987)(0.854)
      EQ-5D anxiety and depression (1 omitted)
      21.0001.042
      [0.857,1.168][0.893,1.218]
      (0.996)(0.599)
      30.9430.893
      [0.696,1.275][0.659,1.211]
      (0.701)(0.468)
      EQ-VAS (0–10 omitted)(.)(.)(.)
      10–191.1011.1481.191.162
      [0.742,1.633][0.774,1.703][0.799,1.771][0.780,1.731]
      (0.632)(0.493)(0.392)(0.460)
      20–290.8930.9951.0271.035
      [0.614,1.296][0.684,1.446][0.701,1.503][0.706,1.517]
      (0.550)(0.978)(0.893)(0.859)
      30–390.6920.8150.9140.924
      [0.469,1.021][0.551,1.206][0.604,1.383][0.611,1.398]
      (0.063)(0.306)(0.671)(0.709)
      40–490.7100.8451.0401.035
      [0.489,1.030][0.581,1.229][0.697,1.552][0.694,1.543]
      (0.071)(0.378)(0.847)(0.865)
      50–590.7320.9681.0391.129
      [0.501,1.069][0.659,1.421][0.694,1.556][0.753,1.693]
      (0.106)(0.867)(0.851)(0.556)
      60–690.5850.7350.950.965
      [0.394,0.868][0.493,1.095][0.618,1.458][0.629,1.481]
      (0.008)(0.130)(0.813)(0.871)
      70–790.5080.6590.9920.988
      [0.340,0.758][0.439,0.988][0.638,1.544][0.636,1.534]
      (0.001)(0.044)(0.972)(0.956)
      80+0.4280.5640.8460.861
      [0.265,0.690][0.348,0.914][0.507,1.413][0.515,1.437]
      (<0.001)(0.020)(0.523)(0.566)
      C-statistic0.7060.7180.7280.7020.7110.7170.718
      Exponentiated coefficients. 95% confidence intervals in square brackets; p-values in parentheses.
      Whilst no statistical difference is observed between models, the estimated discrimination of the EQ-5D dimensions models (self-care only (C-index 0.718) and EQ-5D dimensions plus EQ-VAS (0.717)) appears superior to that of the baseline performance status model (0.706), whilst the EQ-VAS alone performs worse (0.702). Conversely, the incorporation of performance status into these patient-assessed models provides only a marginal improvement in the estimated model performance.
      Notably, the Altman-Royston separation index demonstrates limited separation at early time points: the average difference in the probability of dying between groups is inevitably small at this point due to a low number of deaths. With longer follow-up, differences between indices emerge (Fig. 3, also Table A1), although no statistically significant difference was observed. In the first six months, dimension-based models (including the maximally parsimonious self-care only model) appear to perform best whilst in longer follow-up those incorporating the EQ-VAS deliver improved separation with performance status providing a further marginal improvement in some cases.
      Figure thumbnail gr3
      Fig. 3Mean Altman-Royston separation indices for each model at varying time-points. Based on bootstrapped internal validation.

      Discussion

      We demonstrate that as a quick and easy to complete patient questionnaire the EQ-5D offers valuable prognostic information which may be able to replace the role of clinician-assessed performance status in a simple prognostic model. Whilst we do confirm the prognostic value of clinician-assessed performance status by itself, we find that its additional value is reduced in the presence of the EQ-5D, its inclusion providing a non-significant marginal benefit in terms of model performance.
      The model including performance status alone performs poorly in this sample, both in terms of discrimination and separation. In all EQ-5D based models, the addition of performance status appears to deliver a non-significant marginal improvement in discrimination. Similarly, model separation appears minimally impacted by the addition of performance status. Indeed, during the first 5 months of follow-up, the model incorporating the EQ-5D self-care dimension alone delivered the best separation. The simplicity of the EQ-5D questionnaire and independence of the predictors from clinician judgement offer a key advantage over existing models, potentially ensuring that such a model is accessible to non-specialist clinicians. The extent to which the extra discrimination delivered by performance status justifies its inclusion, as compared to a model which is independent of specialist clinician judgement, will be dependent upon the proposed use of the model.
      By introducing the use of bootstrap resampling methods, we are able to demonstrate robust evidence of separation between groups. Our results suggest that depending upon the relevant time-period for prediction, the optimal variables for inclusion in the model may vary. For example, any benefit of including the EQ-VAS as a predictor variable only becomes apparent for predicted survival beyond 150 days. Conversely, compared to more complex models the most parsimonious model appears to provide similar separation between prognostic groups where very short prognosis is considered. It must be noted, however, that these differences are not found to be significant at conventional levels, and our results point to the need for further research in this area. Future studies should carefully define the question to be answered by the developed model and proposed environment for implementation. For example, defining the probability of survival at a very short interval may be valuable in considering the role of radiotherapy. In this case independence from a specialist clinician assessment may be desirable to avoid unnecessary travel for specialist review near the end of life. Conversely, determining if fractionation is appropriate would require a longer time horizon and thus incorporation of other variables. For these later end-points, relevant to the specialist clinician, inclusion of a clinician-assessed performance status may remain appropriate. Model usage and thus priorities should be identified a priori and variables incorporated to deliver these.
      Whilst performance status has inherent limitations due to inter-rater reliability it should be recognised that patient-reported outcomes in this context are also not without their challenges. The test–retest reliability of the EQ-5D dimensions has shown inconsistent results ranging from moderate to excellent. [
      • Buchholz I.
      • Janssen M.F.
      • Kohlmann T.
      • Feng Y.S.
      A Systematic Review of Studies Comparing the Measurement Properties of the Three-Level and Five-Level Versions of the EQ-5D.
      ] It is generally accepted that the burden of questionnaires should be minimised for patients who are near the end of life. [
      • Groenvold M.
      • Petersen M.A.
      • Aaronson N.K.
      • Arraras J.I.
      • Blazeby J.M.
      • Bottomley A.
      • et al.
      The development of the EORTC QLQ-C15-PAL: A shortened questionnaire for cancer patients in palliative care.
      ] Indeed, in clinical trials missing data is recognised to increase with proximity to death. [
      • Hussain J.A.
      • White I.R.
      • Langan D.
      • Johnson M.J.
      • Currow D.C.
      • Torgerson D.J.
      • et al.
      Missing data in randomized controlled trials testing palliative interventions pose a significant risk of bias and loss of power: a systematic review and meta-analyses.
      ] A number of studies have reported reduced questionnaire returns and lower completion rates with longer questionnaires, although this is not a consistent finding and has not been assessed specifically in a patient population near the end of life with advanced incurable cancer. [
      • Rolstad S.
      • Adler J.
      • Rydén A.
      Response Burden and Questionnaire Length: Is Shorter Better?.
      ].
      These challenges raise an important potential limitation for the use of patient-reported outcomes in prognostic modelling, and highlight the probable existence of a trade-off between complexity of data collection and the prognostic value of what is obtained. We would argue, however, that the EQ-5D questionnaire used here offers the advantage of including only six questions even in its full form. Furthermore, the burden of questionnaire completion could be reduced, for instance, by the omission of the relatively complex EQ-VAS, leaving only five questions with a three-level response, and potentially making the questionnaire easier to complete. Longer questionnaires, such as the EORTC QLQ-C30 summary score have been shown to be prognostic: however, these may be considered too burdensome near the end of life. [

      Husson O, de Rooij BH, Kieffer JM, Oerlemans S, Mols F, Aaronson NK, et al. Independent prognostic value of the EORTC QLQ-C30 summary score on all-cause mortality: Results from the population-based PROFILES registry. J Clin Oncol. 2018 May 20;36(15_suppl):10070–10070.

      ] The EQ-5D was designed to span the full range of quality of life, but in generic terms rather than, for instance, with particular regard to aspects of quality of life that may be specific to cancer. If it is deemed that greater data collection would be justified by additional prognostic value, additional items, such as fatigue, appetite and shortness of breath, or indeed patient-assessed performance status, may offer further value. [
      • Chow E.
      • Abdolell M.
      • Panzarella T.
      • Harris K.
      • Bezjak A.
      • Warde P.
      • et al.
      Predictive model for survival in patients with advanced cancer.
      ] The incorporation of the remaining four EQ-5D dimensions (mobility, usual activities, pain/discomfort, and anxiety/depression) and EQ-VAS did not significantly improve model performance, although greater separation was observed at late time-points. Coefficient point-estimates for these dimensions were in expected directions (with worse health being associated with worse survival) and as such, we are unable to draw strong conclusions regarding their inclusion due to limited power.
      Future studies, with larger patient cohorts should consider the inclusion of extra items or indeed, the five-level EQ-5D-5L whose additional levels may provide valuable prognostic information with limited impact upon questionnaire burden. The appropriateness of the inclusion of additional (or less) data collection is likely to be highly context-specific. This does not detract from the principal conclusion here, however, that a simple patient-reported outcome measure (PROM) is able to replace performance status in predicting prognosis for patients with advanced incurable cancer. Indeed, the ability to replace performance status based only upon a short and generic questionnaire is a clear strength in a population near the end of life. There is a need for simplicity in prognostic models such that these are suitable for use in the clinical environment. The inclusion of greater numbers of PROM domains must be clearly justified by the model improvement delivered.
      Conversely, the demonstration that the EQ-5D does not offer superior prognostication to that offered by specialist assessment of performance status, supports the continued use of the latter in models for use in specialist environments. Where models are developed for use outside this environment, the role of PROMs in place of clinician-assessed measures should be considered. The acceptability of this approach for patients and clinicians requires assessment: are the questions used acceptable?; are patients and clinicians willing to use the results to guide their decision-making?; is proxy PROM completion acceptable across relevant domains when necessary? [
      • Hounsome N.
      • Orrell M.
      • Edwards R.T.
      EQ-5D as a quality of life measure in people with dementia and their carers: evidence and key issues.
      ,
      • Pickard A.S.
      • Lin H.W.
      • Knight S.J.
      • Sharifi R.
      • Wu Z.
      • Hung S.Y.
      • et al.
      Proxy Assessment of Health-Related Quality of Life in African American and White Respondents With Prostate Cancer.
      ]; when PROMs are used to inform wider routine care (e.g. through symptom control) are they robust to risks of well-meaning manipulation?
      Beyond the limitations of power outlined above our study has other limitations worth considering. Our data are relatively old and given the transformational change that has been witnessed in, for example, NSCLC with the arrival of immunotherapies this may significantly impact upon the model, particularly given the extent to which primary diagnosis is a significant predictor of outcome. External validation within a separate, larger, more contemporaneous dataset is now required. Additionally, unlike Chow et al. our dataset is limited to a patient population with known painful bone metastases. Future studies should consider a broader population of patients with advanced incurable cancer where the domains not included here might well offer further valuable information. In addition, the potential for the EQ-5D to perform differently in the routine clinical setting, particularly if it were specifically stated to be used to support clinical decision-making, requires future consideration. It is notable that the results presented here reflect a comparison between patient-reported outcomes and specialist-assessed performance status, rather than non-specialist professional-assessed performance status. This latter comparison could not be made but would reasonably be expected to be detrimental to predictions based on specialist-assessed performance status and would, therefore, not be expected to change the conclusions of this study. Patient-reported performance status has, however, been shown to be a strong predictor of prognosis. How this performs as a prognostic factor in routine care should be considered. Finally, given that our study appears to be underpowered to detect any difference in performance between the different models under consideration, future studies should be designed in order to detect a difference deemed to meaningfully impact upon model use by both clinicians and patients.
      Given the challenges of inter-rater reliability in performance status and the nature of the treatment referral pathway, including both specialist and non-specialist clinicians, flexibility is needed. These results support the development of prognostic models which incorporate patient-reported outcomes to circumvent these issues where necessary. In the specialist environment, performance status maybe the simplest predictor to incorporate, however, in non-specialist settings consideration should be given to alternative approaches incorporating PROMs. Larger, more contemporaneous datasets are now required to further optimise this approach and provide external validation.
      Funding: This study was funded by the EuroQoL group. The funding organisation had no role in the design, conduct or reporting of this study.

      Declaration of Competing Interest

      The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
      This work was undertaken on ARC3, part of the High Performance Computing facilities at the University of Leeds, UK.

      Appendix A. Supplementary material

      The following are the Supplementary data to this article:

      References

      1. Glare P. A systematic review of physicians’ survival predictions in terminally ill cancer patients. BMJ. 2003 Jul 24;327(7408):195–0.

        • Chow E.
        • Davis L.
        • Panzarella T.
        • Hayter C.
        • Szumacher E.
        • Loblaw A.
        • et al.
        Accuracy of survival prediction by palliative radiation oncologists.
        Int J Radiat Oncol Biol Phys. 2005; 61: 870-873
        • Gripp S.
        • Mjartan S.
        • Boelke E.
        • Willers R.
        Palliative radiotherapy tailored to life expectancy in end-stage cancer patients: reality or myth?.
        Cancer. 2010 Jul 1; 116: 3251-3256
        • Tseng Y.D.
        • Krishnan M.S.
        • Sullivan A.J.
        • Jones J.A.
        • Chow E.
        • Balboni T.A.
        How radiation oncologists evaluate and incorporate life expectancy estimates into the treatment of palliative cancer patients: a survey-based study.
        Int J Radiat Oncol Biol Phys. 2013 Nov 1; 87: 471-478
        • Weeks J.C.
        • Cook E.F.
        • O’Day S.J.
        • Peterson L.M.
        • Wenger N.
        • Reding D.
        • et al.
        Relationship Between Cancer Patients’ Predictions of Prognosis and Their Treatment Preferences.
        JAMA. 1998 Jun 3; 279: 1709-1714
        • Chow E.
        • Abdolell M.
        • Panzarella T.
        • Harris K.
        • Bezjak A.
        • Warde P.
        • et al.
        Predictive model for survival in patients with advanced cancer.
        J Clin Oncol. 2008; 26: 5863-5869
        • Dosani M.
        • Tyldesley S.
        • Bakos B.
        • Hamm J.
        • Kong T.
        • Lucas S.
        • et al.
        The TEACHH model to predict life expectancy in patients presenting for palliative spine radiotherapy: external validation and comparison with alternate models.
        Support Care Cancer. 2018; 26: 2217-2227
        • Westhoff P.G.
        • de Graeff A.
        • Monninkhof E.M.
        • Bollen L.
        • Dijkstra S.P.
        • van der Steen-Banasik E.M.
        • et al.
        An Easy Tool to Predict Survival in Patients Receiving Radiation Therapy for Painful Bone Metastases.
        Int J Radiat Oncol. 2014; 90: 739-747
        • Bollen L.
        • van der Linden Y.M.
        • Pondaag W.
        • Fiocco M.
        • Pattynama B.P.M.
        • Marijnen C.A.M.
        • et al.
        Prognostic factors associated with survival in patients with symptomatic spinal bone metastases: a retrospective cohort study of 1 043 patients.
        Neuro-Oncol. 2014; 16: 991-998
        • Chow R.
        • Chiu N.
        • Bruera E.
        • Krishnan M.
        • Chiu L.
        • Lam H.
        • et al.
        Inter-rater reliability in performance status assessment among health care professionals: a systematic review. Ann.
        Palliat Med. 2016; 5: 83-92
        • May C.H.
        • Lester J.F.
        • Lee S.
        PS score discordance and why it matters.
        Lung Cancer. 2012 Jan; 75: S18
        • Pajouheshnia R.
        • van Smeden M.
        • Peelen L.M.
        • Groenwold R.H.H.
        How variation in predictor measurement affects the discriminative ability and transportability of a prediction model.
        J Clin Epidemiol. 2019 Jan; 1: 136-141
        • Park S.M.
        • Park M.H.
        • Won J.H.
        • Lee K.O.
        • Choe W.S.
        • Heo D.S.
        • et al.
        EuroQol and survival prediction in terminal cancer patients: a multicenter prospective study in hospice-palliative care units.
        Support Care Cancer. 2006; 14: 329-333
        • Diouf M.
        • Chibaudel B.
        • Filleron T.
        • Tournigand C.
        • Hug de Larauze M.
        • Garcia-Larnicol M.-L.
        • et al.
        Could baseline health-related quality of life (QoL) predict overall survival in metastatic colorectal cancer? The results of the GERCOR OPTIMOX 1 study.
        Health Qual Life Outcomes. 2014; 12: 69
        • Clarke P.M.
        • Hayes A.J.
        • Glasziou P.G.
        • Scott R.
        • Simes J.
        • Keech A.C.
        Using the EQ-5D Index Score as a Predictor of Outcomes in Patients With Type 2 Diabetes.
        Med Care. 2009 Jan; 47: 61-68
        • Mathews W.C.
        • May S.
        EuroQol (EQ-5D) measure of quality of life predicts mortality, emergency department utilization, and hospital discharge rates in HIV-infected adults under care.
        Health Qual Life Outcomes. 2007 Jan 25; 5: 5
      2. ECOG Performance Status [Internet]. ECOG-ACRIN. [cited 2020 Aug 10]. Available from: https://ecog-acrin.org/resources/ecog-performance-status.

      3. EuroQol Group. EQ-5D-3L [Internet]. [cited 2018 Oct 4]. Available from: https://euroqol.org/eq-5d-instruments/eq-5d-3l-about/.

        • Steenland E.
        • Leer J.
        • van Houwelingen H.
        • Post W.J.
        • van den Hout W.B.
        • Kievit J.
        • et al.
        The effect of a single fraction compared to multiple fractions on painful bone metastases: a global analysis of the Dutch Bone Metastasis Study.
        Radiother Oncol. 1999; 52: 101-109
        • van den Hout W.B.
        • van der Linden Y.M.
        • Steenland E.
        • Wiggenraad R.G.J.
        • Kievit J.
        • de Haes H.
        • et al.
        Single- versus multiple-fraction radiotherapy in patients with painful bone metastases: cost-utility analysis based on a randomized trial.
        J Natl Cancer Inst. 2003; 95: 222-229
        • van der Linden Y.M.
        • Lok J.J.
        • Steenland E.
        • Martijn H.
        • van Houwelingen H.
        • Marijnen C.A.M.
        • et al.
        Single fraction radiotherapy is efficacious: a further analysis of the Dutch Bone Metastasis Study controlling for the influence of retreatment.
        Int J Radiat Oncol. 2004; 59: 528-537
        • Royston P.
        • Sauerbrei W.
        A new measure of prognostic separation in survival data.
        Stat Med. 2004 Mar 15; 23: 723-748
        • Altman D.G.
        • Royston P.
        What do we mean by validating a prognostic model?.
        Stat Med. 2000 Feb 29; 19: 453-473
        • Riley R.D.
        • van der Windt D.
        • Croft P.
        • Prognosis M.K.
        Research in Healthcare: Concepts, Methods, and Impact.
        Oxford University Press, 2019: 373 p.
        • Buchholz I.
        • Janssen M.F.
        • Kohlmann T.
        • Feng Y.S.
        A Systematic Review of Studies Comparing the Measurement Properties of the Three-Level and Five-Level Versions of the EQ-5D.
        PharmacoEconomics. 2018 Jun 1; 36: 645-661
        • Groenvold M.
        • Petersen M.A.
        • Aaronson N.K.
        • Arraras J.I.
        • Blazeby J.M.
        • Bottomley A.
        • et al.
        The development of the EORTC QLQ-C15-PAL: A shortened questionnaire for cancer patients in palliative care.
        Eur J Cancer. 2006; 42: 55-64
        • Hussain J.A.
        • White I.R.
        • Langan D.
        • Johnson M.J.
        • Currow D.C.
        • Torgerson D.J.
        • et al.
        Missing data in randomized controlled trials testing palliative interventions pose a significant risk of bias and loss of power: a systematic review and meta-analyses.
        J Clin Epidemiol. 2016; 74: 57-65
        • Rolstad S.
        • Adler J.
        • Rydén A.
        Response Burden and Questionnaire Length: Is Shorter Better?.
        A Review and Meta-analysis Value Health. 2011 Dec 1; 14: 1101-1108
      4. Husson O, de Rooij BH, Kieffer JM, Oerlemans S, Mols F, Aaronson NK, et al. Independent prognostic value of the EORTC QLQ-C30 summary score on all-cause mortality: Results from the population-based PROFILES registry. J Clin Oncol. 2018 May 20;36(15_suppl):10070–10070.

        • Hounsome N.
        • Orrell M.
        • Edwards R.T.
        EQ-5D as a quality of life measure in people with dementia and their carers: evidence and key issues.
        Value Health J Int Soc Pharmacoeconomics Outcomes Res. 2011 Apr; 14: 390-399
        • Pickard A.S.
        • Lin H.W.
        • Knight S.J.
        • Sharifi R.
        • Wu Z.
        • Hung S.Y.
        • et al.
        Proxy Assessment of Health-Related Quality of Life in African American and White Respondents With Prostate Cancer.
        Med Care. 2009 Feb; 47: 176-183