|Year : 2021 | Volume
| Issue : 1 | Page : 2-7
Toward a validated diagnostic test with machine learning algorithm for interstitial cystitis
Michael B Chancellor, Laura E Lamb
Department of Urology, Beaumont Health System, Royal Oak, MI, USA
|Date of Submission||19-Nov-2020|
|Date of Decision||14-Dec-2020|
|Date of Acceptance||24-Dec-2020|
|Date of Web Publication||27-Mar-2021|
Michael B Chancellor
Department of Urology, Beaumont Health System, 3811 West 13 Mile Road, Suite 504, Royal Oak 48073, MI
Source of Support: None, Conflict of Interest: None
Diagnosing interstitial cystitis/bladder pain syndrome (IC/BPS) is difficult as there is no definitive test for IC/BPS. Instead, the diagnosis is based on urinary symptoms and cystoscopy may be recommended. However, cystoscopic diagnosis is associated with potentially exacerbating painful side effects and is highly subjective among physicians. Furthermore, IC/PBS symptoms overlap with symptoms of bladder cancer, urinary tract infection, or overactive bladder. As a result, many patients may go years without a correct diagnosis and proper disease management. The goal of our current IC/BPS research is to develop a simple diagnostic test based on several urine proteins called the IC-risk score (IC-RS). A machine learning (ML) algorithm uses this information to determine if a person has IC/BPS or not; if they have IC/BPS, whether their IC/BPS is characterized by Hunner's lesions. We are currently in the middle of a grant to collect urine samples from 1000 patients with IC/BPS and 1,000 normal controls from across the United States. We are using social media such as Twitter and Facebook and working with patient advocacy organizations to collect urine samples from across the country. We hope to validate the IC-RS and apply for regulatory approval. Having a validated diagnostic test for IC/BPS would be a major advancement to help urology patients. In addition, drug companies developing new drugs and therapies for IC/BPS would have a better way to determine who to include in their clinical trials, and possibly another way to measure if their drug or therapy is effective. We will hereby review the steps that have led us in urine biomarker discovery research from urine protein assessment to use crowdsourcing stakeholders participation to ML algorithm IC-RS score development.
Keywords: Bladder, cytokine, interstitial cystitis, machine learning, urine
|How to cite this article:|
Chancellor MB, Lamb LE. Toward a validated diagnostic test with machine learning algorithm for interstitial cystitis. Urol Sci 2021;32:2-7
| Introduction|| |
Interstitial cystitis/bladder pain syndrome (IC/BPS) is a debilitating symptom complex including urinary urgency, frequency, and pain., The estimated prevalence of IC/BPS is between 3 and 8 million women and 1–4 million men,, in the US, with similar incidence across developed countries of the world. Symptoms can result in fatigue, sexual dysfunction, depression, and suicidal ideation. There is a lack of consensus regarding diagnostic criteria, etiology, and even appropriate terminology for labeling this complex and chronic symptom-based syndrome. Most experts use cystoscopic findings to categorize IC/BPS patients into two general subtypes: With Hunner's lesions (HIC) and without Hunner's lesions (NHIC). 80%–90% of IC/BPS patients do not have Hunner's lesion (NHIC) while 10%–20% have the ulcerative subtype based on the presence of one or more distinct, recurring inflammatory lesions in the bladder (HIC).,, Patients with identifiable ulcers tend to be older, have lower bladder capacities, greater urinary frequency, and exhibit other differences that implicate inflammatory processes.,,, HIC exhibits mast cell proliferation which causes inflammation and Hunner's lesion. This inflammation results in bladder pain with sensitization of peripheral and central sensory nociceptive nerve endings. Diagnosis is made after excluding other conditions with similar characteristics. NHIC has been shown to be associated with fibromyalgia and irritable bowel syndrome, and organ cross-talk and systemic upregulation of pain nerve fibers have been implicated in the IC/BPS pathophysiology.,,,,
| Diagnostic Challenges|| |
IC/BPS is currently diagnosed by urinary symptoms and pain levels, as well as physical examination. There is currently no commercial biomarker test available. Pain is the most bothersome symptom to most IC/BPS patients., The O'Leary-Sant Problem and Symptom Index (OPSI) is validated patient-reported symptom-based survey instruments that were developed to identify IC/BPS patients for inclusion into clinical trials. While the sensitivity has been demonstrated by several groups and the OPSI has been used as a screening tool, the specificity is very low; other confusable disorders with overlapping frequency and urgency symptoms are not distinguished by the OPSI.,, A receiver operating characteristic (ROC) curve is used to evaluate discrimination between two groups by plotting sensitivity as a function of specificity. The greater the area under the curve (AUC), the more discriminative the score; an AUC of 1.00 is a score that correctly discriminates 100% of cases. The AUC for OPSI has been reported to range from 0.492 to 0.891 depending on the study., As such, OPSI has not been used by itself for diagnosis. HIC is diagnosed by cystoscopy with hydrodistention to demonstrate distinct inflammatory lesions typically on the dome or lateral sidewalls of the bladder.
The diagnostic tool we are developing is the IC-risk score (IC-RS), which may aid clinicians and researchers in classifying IC/BPS patients, track disease progression, and possibly determine response to therapy. Biomarker discovery in IC/BPS has been challenging with considerable clinical effort and expense, in part due to the need to process and freeze urine samples in a timely manner. We recognized that many excellent papers have been published on evaluation of urine-based inflammatory biomarkers, including important series from Taiwan. However, biomarker discovery in IC/BPS has been challenging, with considerable clinical effort and expense, and usually done only at centers of excellence because of need to process and freeze the urine samples in a timely manner. Our goal is to develop a urine-based assay to identify IC/BPS while avoiding these restraints. We identified several challenges to develop a useful biomarker: (1) It must be developed and validated in a large number of samples; (2) It must be validated nationally; and (3) It needs to overcome the cold-supply chain constraints.
| The IP4IC Study|| |
We addressed these challenges through our innovative IP4IC study, which took advantage of crowdsourcing samples in collaboration with social media (Facebook, Twitter, E-mail blasts, and YouTube) and an IC/BPS patient advocacy group, the IC Association, to drive recruitment and engagement of both IC/BPS patients and asymptomatic controls [Figure 1]. We circumvented the need for a cold-supply chain for urine sample collection using a commercially available urine preservative that allowed shipment and storage of urine samples at ambient temperature (Norgen Biotek). Participants were sent a urine specimen collection kit that allowed for urine collection, shipment, and storage at ambient room temperature through the use of a urine preservative directly manufactured directly into the urine collection cup. We validated that the urinary proteins of interest were stable at temperatures up to 30°C for up to 1 year, so mailing kits also included an irreversible temperature label that indicated if and for how long a package had been exposed to this temperature. The IP4IC Study was immensely successful; enrollment goals were attained within 2 weeks of study launch and we collected urine samples from 448 participants representing 46 US States in 3 months [Figure 2]., The IP4IC/BPS study samples consisted of 153 IC/BPS patients (147 females, 6 males), of which 54 were HIC/BPS patients (50 females, 4 males), and 159 females and 136 males age-matched controls. Corresponding demographics, IC/BPS IC symptom index (ICSI) and IC pain index, and bladder voiding diaries were also collected. Results were validated through an independent study (termed the P3 study) that was collected in the clinic with physician associated IC/BPS diagnosis. Crowdsourcing for sample collection proved to be fast, reliable, cost-effective, and efficient at collecting large numbers of both IC/BPS and control participants nationally.
|Figure 1: The use of crowdsourcing for clinical sample collection. With permission, Reference 1. JU 2018. Crowdsourcing to solve problems in biomarker discovery. Partnership and workflow for IP4IC/bladder pain syndrome Study (1) that utilized social media and a patient advocacy organization, the interstitial cystitis association, to crowd source hundreds of urine samples with correlation patient data for biomarker research|
Click here to view
|Figure 2: Results of crowdsourcing sample collection using social media. IP4IC/bladder pain syndrome sample collection by week. 448 urine samples were collected from 46 US states within 3 months|
Click here to view
| Machine Learning in Biomarker Development|| |
Machine learning (ML) is increasingly utilized to deal with complex biological data. Random forest classification (RFC) is an efficient ML method for classification of samples based on genetic or proteomic expression. The expression of three urinary cytokines were quantified using Luminex Assays [Figure 3]a,[Figure 3]b,[Figure 3]c. Although there was a significant difference in the means between HIC compared to controls or NHIC, there was considerable overlap between the potential biomarkers if used independently [Figure 3]a,[Figure 3]b,[Figure 3]c. We utilized RFC to combine the three biomarkers from the IP4IC training set into a RS, which identifies patients with HIC versus NHIC versus asymptomatic controls [Figure 3]d. We then did external validation in a separate and clinically collected validation set, termed P3 [Figure 3]d. A ROC analysis resulted in an impressive AUC for the IP4IC training dataset of 0.971 and the P3 validation dataset of 0.919 [Figure 3]e.
|Figure 3: Interstitial cystitis risk score development. Bladder pain-risk score development. (a-c) Although statistically different, individual cytokine expression alone is not sufficient to classify controls and NHIC/bladder pain syndrome patients from HIC/bladder pain syndrome patients due to overlap between groups. (d) The bladder pain-risk score, a combination of urinary biomarkers, can successfully distinguish HIC/bladder pain syndrome patients for both the IP4IC training set and P3 validation set. (e) Receiver operating characteristic curve. The strong area under the curve values for IP4IC/bladder pain syndrome training|
Click here to view
Overall, 88.7% of the entire P3 validation set was correctly predicted. Three cytokines (GRO/CXCL1, interleukin [IL]-6, and IL-8) were all required for the RFC to classify patients but they were not equally important (e.g., GRO/CXCL1 contributed more to the final classifier than IL-6). Gender did not impact the results with respect to the RFC. The IP4IC/BPS training set and P3 validation set had a false positive rate of 2.2% and 4.7%, respectively. There was a negative predictive value of 96.0% and 93.2%, respectively, and a positive predictive value of 84.9% and 75.0%, respectively. The methodology, IC-RS score, and the crowdsourcing have been published.,
Our current research aims to build a new ML RS to distinguish all IC/BPS participants (both NHIC and HIC) from controls, the IC-RS. We have generated strong pilot data that laid the foundation for this grant proposal. In addition to using the three cytokines described above, another noninvasive parameter was included as a feature in the RFC. This was the corresponding ICSI/PI scores which incorporates measurements for pain and urinary symptoms. Proof-of-concept for the IC-RS tool was generated in follow-up analysis of the initial IC-RS study. This study focused on the P3 cohort as samples from the IP4IC cohort were depleted. For our pilot study, the IC-RS correctly identified 100% (n = 23/23) IC/BPS and 96.2% (n = 25/26) control participants in the P3 cohort [Figure 4]. The AUC of the ROC curve for the pilot set was 0.994. Given the heterogeneous population of IC/BPS, the performance of the IC-RS was both surprising and exciting.
|Figure 4: Interstitial cystitis-risk score area under curve value. Interstitial cystitis-risk score for IC/bladder pain syndrome diagnosis of P3 clinical cohort (a) by group and (b) area under curve value.|
Click here to view
Our current work supports development of the IC-RS as a new diagnostic multivariate score in a large national cohort of IC/BPS, control, and other confusable pelvic pain disorder participants to evaluate specificity and determine if it can serve as a new clinical outcome assessment. We also seek to develop patient engagement tools and resources effective for crowdsourcing samples. This includes new methodologies for biomarker development to support mass testing of urine samples from across the United States, including online support tools for patient engagement, urine sample shipping kits, and ML for translating urinary protein levels into meaningful disease diagnostic criteria. Samples will be collected from civilians and military personnel by crowdsourcing, which we have successfully done in the past. We will also collect urine samples in clinic from three academic urology departments to compare patient collected samples with samples from physician diagnosed IC/BPS. We will test urine from patients with overlapping symptoms including documented urinary tract infection (UTI), overactive bladder (OAB) with incontinence (OAB-wet), and bladder cancer to assess test specificity. Our goal is to develop a clinical tool for diagnosis and individualized guided treatment, in addition to seeking partnerships with commercial development partners to bring the tool to market.
| Crowdsourcing in Medical Research|| |
We have engaged with IC/BPS patients from around the United States in focus groups prior to crowdsourcing urine collection to engage active study design input from patients' perspectives. Patient and patient advocacy organization engagement and involvement are a central theme of our research that helps make the results clinically meaningful. The translation of ideas from bench-to-bedside is challenging.
We have developed, manufactured, and verified that the urine preservative is effective at stabilizing expression of our proteins of interest for up to 1 year at room temperature [Figure 5]. We send our research participants a participant package includes instructions, a urine collection cup, a questionnaire regarding symptom changes or UTI since completed online survey, a bladder diary, and a prepaid return shipping package. Online instructions and YouTube videos are also provided on the website. All return shipping packages also have nonreversible temperature recording labels indicating temperatures exceeding 30°C for over 30 min. At temperatures exceeding 30°C, there is a decrease in protein stability that may affect downstream testing. In IP4IC, this only impacted 2 out of 448 samples so we do not anticipate this to be an issue. Return shipment packages were prepaid and meet all federal regulations for urine transport.
|Figure 5: Ambient temperature urine collection kit for mailing. Collection kit for preserved urine|
Click here to view
Returned urine samples were aliquoted and stored at room temperature and ten 0.5 mL aliquots stored at-80°C. Samples were randomized and assigned an experimental ID number to run and analyze samples in a blinded fashion. Creatinine was be measured by high-performance liquid chromatography to normalize across patients. The expression of a panel of cytokines were be determined using the MILLIPLEX MAP Human Cytokine/Chemokine Multiplex Immunoassay (Millipore) following manufacturer's protocol and detailed in Lamb et al., 2017. A supervised ML method described in,, was then used to calculate the IC-RS. The IC-RS will classify a patient as having IC/BPS or as a control. If positive for IC/BPS, the IC-RS will also classify the patient as having HIC or NHIC.
| Development of Interstitial Cystitis-Risk Score through Machine Learning|| |
The IC-RS ML classifier was trained using data obtained in our previous urine collection. This RFC binary classifier will categorize subjects as either having or not having IC/BPS based on the levels of measured cytokines and OPSI score. RFC is well suited to datasets of this size and has outperformed other algorithms in our experience with the IC-RS. Each subject will be given a label of 0 (lacks IC/BPS) or 1 (has IC/BPS). Patients that are classified as having IC/BPS will then undergo a second analysis to determine if that patient has HIC (label of 1 in second analysis) or NHIC (label of 0 in second analysis). The RFC algorithm uses measured cytokine levels and OPSI scores of a random subset of subjects to build a single decision tree. The subjects not sampled for constructing the decision tree are used to test the predictive performance of that tree-this is analogs to cross-validation that would be used with other statistical methods. Multiple decision trees are built in this manner. The combination of all these decision trees is the classifier, where subjects are classified based on majority vote of the component decision trees.
Because resampling-based validation is used in training the RFC algorithm, it is necessary to also evaluate the resulting classifier using external validation of another dataset. This approach will aid in the optimization of parameters used to build the IC-RS classifier; various parameters required for the RFC are, for example, the number of decision trees to build which criterion to measure the quality of the splits in the decision tree, and how deep/branched the tree should be. These parameters will be optimized through software that was developed by our group by testing the trained classifiers against the smaller validation dataset. The highly parallelized software can test millions of unique parameter combinations on a standard desktop computer in less than a week in order to obtain a set of parameters that provide the greatest predictive performance and minimize overfitting in the final RFC.
New, unlabeled (blinded) cytokines and OPSI data that is obtained from our proposed study will be tested with the optimal RFC and tested against each decision tree. Each decision tree will provide a probability score for each label (0 or 1). The average of all of these probabilities from all of the trees will determine which prediction has been made for a given subject; based on our preliminary data. If the mean is <0.5, the subject will be classified as lacking IC/BPS; alternatively, if the mean is ≥0.5, the subject will be classified as having IC/BPS. The second analysis for IC/BPS classified patients for having HIC will be determined in a similar manner. After all newly obtained data have been classified, the self-reported diagnosis of each patient will be unblinded so that the performance of the optimal IC-RS classifier can be assessed.
| Translational Research|| |
The result of this research is hopefully a diagnostic product that will be the first regulatory approved clinical laboratory test for IC/BPS. As there are no regulatory approved or objective diagnostic tests for IC/BPS, there remains a great unmet medical need and a value proposition for IC/BPS. IC-RS can aid as companion diagnostic for new IC/BPS treatment. Our current research is the final step for validation of a test that can enter the regulatory approval laboratory certification pathway. Translational research, moving an idea from research laboratory's bench top to a commercial product requires a product that is patent protected. In parallel with our research, we have applied for and received issued patent that covers the technologies descripted in this review.
| Conclusion|| |
We are excited about the potential to apply ML based medical diagnostic score for IC/BPS. Novel features of our project that may serve the broader urology community include:
- We have optimized a urine preservative that allows for shipping and assay of urine samples without preparation. No centrifugation or refrigeration with our method of room temperature urine sample preparation and use of standard mail shipping
- Social engagement of stakeholders in research through social media with regulatory compliant human research protocol
- Crowdsourcing of samples instead of hospital and clinic-based samples collection
- A diagnostic score based on ML of several urine proteins that may diagnose IC/BPS when individual protein levels lack adequate sensitivity or specificity. This is in line with the U. S. Food Drug Administration's proposed regulatory framework for artificial intelligence/ML based software as a medical device (SaMD).
The result of this research is hopefully a regulatory approved diagnostic product that can help physicians diagnose IC/BPS. We believe a great part of the future of health care is computer science based and SaMD can play an important role. We do not know what the future will hold or if our dream can be achieve. We do know the journal has been an adventure and joy and we want to thank all the people, health-care providers, scientists and patients who have worked with us and helped us.
Financial support and sponsorship
This study was supported by the Office of the Assistant Secretary of Defense for Health Affairs through the Technology/Therapeutic Development Research Program under Award No. W81XWH-19-1-0288. The opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the Department of Defense.
Conflicts of interest
Prof. Michael B. Chancellor, an editorial board member at Urological Science, had no role in the peer review process of or decision to publish this article. The other author declared that she has no conflict of interest.
| References|| |
Chancellor MB, Bartolone SN, Veerecke A, Lamb LE. Crowdsourcing disease biomarker discovery research; The IP4IC/BPS study. J Urol 2017;199:1344-50.
Clemens JQ, Mullins C, Kusek JW, Kirkali Z, Mayer EA, Rodríguez LV, et al
. The MAPP research network: A novel study of urologic chronic pelvic pain syndromes. BMC Urol 2014;14:57.
Berry SH, Elliott MN, Suttorp M, Bogart LM, Stoto MA, Eggers P, et al
. Prevalence of symptoms of bladder pain syndrome/interstitial cystitis among adult females in the United States. J Urol 2011;186:540-4.
Daniels NA, Link CL, Barry MJ, McKinlay JB. Association between past urinary tract infections and current symptoms suggestive of chronic prostatitis/chronic pelvic pain syndrome. J Natl Med Assoc 2007;99:509-16.
Suskind AM, Berry SH, Ewing BA, Elliott MN, Suttorp MJ, Clemens JQ. The prevalence and overlap of interstitial cystitis/bladder pain syndrome and chronic prostatitis/chronic pelvic pain syndrome in men: Results of the RAND Interstitial Cystitis Epidemiology male study. J Urol 2013;189:141-5.
Hepner KA, Watkins KE, Elliott MN, Clemens JQ, Hilton LG, Berry SH. Suicidal ideation among patients with bladder pain syndrome/interstitial cystitis. Urology 2012;80:280-5.
Grégoire M, Liandier F, Naud A, Lacombe L, Fradet Y. Does the potassium stimulation test predict cystometric, cystoscopic outcome in interstitial cystitis? J Urol 2002;168:556-7.
Koziol JA, Adams HP, Frutos A. Discrimination between the ulcerous and the nonulcerous forms of interstitial cystitis by noninvasive findings. J Urol 1996;155:87-90.
Simon LJ, Landis JR, Erickson DR, Nyberg LM. The interstitial cystitis data base study: Concepts and preliminary baseline descriptive statistics. Urology 1997;49:64-75.
Braunstein R, Shapiro E, Kaye J, Moldwin R. The role of cystoscopy in the diagnosis of Hunner's ulcer disease. J Urol 2008;180:1383-6.
Tomaszewski JE, Landis JR, Russack V, Williams TM, Wang LP, Hardy C, et al
. Biopsy features are associated with primary symptoms in interstitial cystitis: Results from the interstitial cystitis database study. Urology 2001;57:67-81.
Peeker R, Fall M. Toward a precise definition of interstitial cystitis: Further evidence of differences in classic and nonulcer disease. J Urol 2002;167:2470-2.
Moutzouris DA, Falagas ME. Interstitial cystitis: An unsolved enigma. Clin J Am Soc Nephrol 2009;4:1844-57.
Lai HH, Gardner V, Ness TJ, Gereau RW 4th
. Segmental hyperalgesia to mechanical stimulus in interstitial cystitis/bladder pain syndrome: Evidence of central sensitization. J Urol 2014;191:1294-9.
Nickel JC, Tripp DA, Pontari M, Moldwin R, Mayer R, Carr LK, et al
. Interstitial cystitis/painful bladder syndrome and associated medical conditions with an emphasis on irritable bowel syndrome, fibromyalgia and chronic fatigue syndrome. J Urol 2010;184:1358-63.
National Guideline Clearinghouse (NGC). Guideline Summary: Diagnosis and Treatment of Interstitial Cystitis/Bladder Pain Syndrome [web site]. Rockville MD. Agency for Healthcare Research and Quality (AHRQ); 2014. Available from: https://www.guideline.gov/summaries/summary/48745
. [Last accessed on 2018 Jun 07].
Mathias SD, Kuppermann M, Liberman RF, Lipschutz RC, Steege JF. Chronic pelvic pain: Prevalence, health-related quality of life, and economic correlates. Obstet Gynecol 1996;87:321-7.
Lutgendorf SK, Kreder KJ, Rothrock NE, Ratliff TL, Zimmerman B. Stress and symptomatology in patients with interstitial cystitis: A laboratory stress model. J Urol 2000;164:1265-9.
Hanno PM, Erickson D, Moldwin R, Faraday MM, American Urological Association. Diagnosis and treatment of interstitial cystitis/bladder pain syndrome: AUA guideline amendment. J Urol 2015;193:1545-53.
Interstitial Cystitis Association. IC/BPS/BPS Symptom Burden: Results from an Interstitial Cystitis Association (IC/BPSA) Survey. Presented at United States FDA Meeting of the Bone, Reproductive and UrologIC/BPS Drugs Advisory Committee; 12/07/17; Silver Spring, Maryland. Available from: https://www.IC/BPShelp.org/wp-content/uploads/2017/12/FDAslides.pdf2017
. [Last accessed on 2020 Nov 01].
O'Leary MP, Sant GR, Fowler FJ Jr, Whitmore KE, Spolarich-Kroll J. The interstitial cystitis symptom index and problem index. Urology 1997;49(5A Suppl):58-63.
Xu L, Zhang P, Zhang N, Yang Y, Wu ZJ, Zhang CH, et al
. Efficiency of O'Leary-Sant symptom index and problem index in the diagnosis of interstitial cystitis. Zhonghua Yi Xue Za Zhi 2013;93:3347-50.
Kushner L, Moldwin RM. Efficiency of questionnaires used to screen for interstitial cystitis. J Urol 2006;176:587-92.
Sirinian E, Azevedo K, Payne CK. Correlation between 2 interstitial cystitis symptom instruments. J Urol 2005;173:835-40.
Humphrey L, Arbuckle R, Moldwin R, Nordling J, van de Merwe JP, Meunier J, et al
. The bladder pain/interstitial cystitis symptom score: Development, validation, and identification of a cut score. Eur Urol 2012;61:271-9.
Lamb LE, Janicki JJ, Bartolone SN, Peters KM, Chancellor MB. Development of an interstitial cystitis risk score for bladder permeability. PLoS One 2017;12:e0185686.
Qi Y. Random forest for BioinformatIC/BPSs. In: Zhang C, Ma Y, editors. Ensemble Machine Learning: Methods and ApplIC/BPSations. Boston, MA: Springer US; 2012. p. 307-23.
Mambatta AK, Jayarajan J, Rashme VL, Harini S, Menon S, Kuppusamy J. Reliability of dipstick assay in predicting urinary tract infection. J Family Med Prim Care 2015;4:265-8.
] [Full text]
Memişoğullari R, Yüksel H, Ak Yıldırım H, Yavuz O. Performance characteristIC/BPSs of dipstIC/BPSk and mIC/BPSroscopIC/BPS urinalysis for diagnosis of urinary tract infection. Eur J Gen Med 2010;7:174-8.
[Figure 1], [Figure 2], [Figure 3], [Figure 4], [Figure 5]