Urological Science

REVIEW ARTICLE
Year
: 2021  |  Volume : 32  |  Issue : 1  |  Page : 2--7

Toward a validated diagnostic test with machine learning algorithm for interstitial cystitis


Michael B Chancellor, Laura E Lamb 
 Department of Urology, Beaumont Health System, Royal Oak, MI, USA

Correspondence Address:
Michael B Chancellor
Department of Urology, Beaumont Health System, 3811 West 13 Mile Road, Suite 504, Royal Oak 48073, MI
USA

Abstract

Diagnosing interstitial cystitis/bladder pain syndrome (IC/BPS) is difficult as there is no definitive test for IC/BPS. Instead, the diagnosis is based on urinary symptoms and cystoscopy may be recommended. However, cystoscopic diagnosis is associated with potentially exacerbating painful side effects and is highly subjective among physicians. Furthermore, IC/PBS symptoms overlap with symptoms of bladder cancer, urinary tract infection, or overactive bladder. As a result, many patients may go years without a correct diagnosis and proper disease management. The goal of our current IC/BPS research is to develop a simple diagnostic test based on several urine proteins called the IC-risk score (IC-RS). A machine learning (ML) algorithm uses this information to determine if a person has IC/BPS or not; if they have IC/BPS, whether their IC/BPS is characterized by Hunner's lesions. We are currently in the middle of a grant to collect urine samples from 1000 patients with IC/BPS and 1,000 normal controls from across the United States. We are using social media such as Twitter and Facebook and working with patient advocacy organizations to collect urine samples from across the country. We hope to validate the IC-RS and apply for regulatory approval. Having a validated diagnostic test for IC/BPS would be a major advancement to help urology patients. In addition, drug companies developing new drugs and therapies for IC/BPS would have a better way to determine who to include in their clinical trials, and possibly another way to measure if their drug or therapy is effective. We will hereby review the steps that have led us in urine biomarker discovery research from urine protein assessment to use crowdsourcing stakeholders participation to ML algorithm IC-RS score development.



How to cite this article:
Chancellor MB, Lamb LE. Toward a validated diagnostic test with machine learning algorithm for interstitial cystitis.Urol Sci 2021;32:2-7


How to cite this URL:
Chancellor MB, Lamb LE. Toward a validated diagnostic test with machine learning algorithm for interstitial cystitis. Urol Sci [serial online] 2021 [cited 2021 Apr 19 ];32:2-7
Available from: https://www.e-urol-sci.com/text.asp?2021/32/1/2/312431


Full Text



 Introduction



Interstitial cystitis/bladder pain syndrome (IC/BPS) is a debilitating symptom complex including urinary urgency, frequency, and pain.[1],[2] The estimated prevalence of IC/BPS is between 3 and 8 million women and 1–4 million men[3],[4],[5] in the US, with similar incidence across developed countries of the world. Symptoms can result in fatigue, sexual dysfunction, depression, and suicidal ideation.[6] There is a lack of consensus regarding diagnostic criteria, etiology, and even appropriate terminology for labeling this complex and chronic symptom-based syndrome. Most experts use cystoscopic findings to categorize IC/BPS patients into two general subtypes: With Hunner's lesions (HIC) and without Hunner's lesions (NHIC). 80%–90% of IC/BPS patients do not have Hunner's lesion (NHIC) while 10%–20% have the ulcerative subtype based on the presence of one or more distinct, recurring inflammatory lesions in the bladder (HIC).[7],[8],[9] Patients with identifiable ulcers tend to be older, have lower bladder capacities, greater urinary frequency, and exhibit other differences that implicate inflammatory processes.[8],[10],[11],[12] HIC exhibits mast cell proliferation which causes inflammation and Hunner's lesion. This inflammation results in bladder pain with sensitization of peripheral and central sensory nociceptive nerve endings.[13] Diagnosis is made after excluding other conditions with similar characteristics. NHIC has been shown to be associated with fibromyalgia and irritable bowel syndrome, and organ cross-talk and systemic upregulation of pain nerve fibers have been implicated in the IC/BPS pathophysiology.[14],[15],[16],[17],[18]

 Diagnostic Challenges



IC/BPS is currently diagnosed by urinary symptoms and pain levels, as well as physical examination. There is currently no commercial biomarker test available.[19] Pain is the most bothersome symptom to most IC/BPS patients.[20],[21] The O'Leary-Sant Problem and Symptom Index (OPSI) is validated patient-reported symptom-based survey instruments that were developed to identify IC/BPS patients for inclusion into clinical trials.[22] While the sensitivity has been demonstrated by several groups and the OPSI has been used as a screening tool, the specificity is very low; other confusable disorders with overlapping frequency and urgency symptoms are not distinguished by the OPSI.[23],[24],[25] A receiver operating characteristic (ROC) curve is used to evaluate discrimination between two groups by plotting sensitivity as a function of specificity. The greater the area under the curve (AUC), the more discriminative the score; an AUC of 1.00 is a score that correctly discriminates 100% of cases. The AUC for OPSI has been reported to range from 0.492 to 0.891 depending on the study.[23],[26] As such, OPSI has not been used by itself for diagnosis. HIC is diagnosed by cystoscopy with hydrodistention to demonstrate distinct inflammatory lesions typically on the dome or lateral sidewalls of the bladder.[19]

The diagnostic tool we are developing is the IC-risk score (IC-RS), which may aid clinicians and researchers in classifying IC/BPS patients, track disease progression, and possibly determine response to therapy. Biomarker discovery in IC/BPS has been challenging with considerable clinical effort and expense, in part due to the need to process and freeze urine samples in a timely manner. We recognized that many excellent papers have been published on evaluation of urine-based inflammatory biomarkers, including important series from Taiwan. However, biomarker discovery in IC/BPS has been challenging, with considerable clinical effort and expense, and usually done only at centers of excellence because of need to process and freeze the urine samples in a timely manner. Our goal is to develop a urine-based assay to identify IC/BPS while avoiding these restraints. We identified several challenges to develop a useful biomarker: (1) It must be developed and validated in a large number of samples; (2) It must be validated nationally; and (3) It needs to overcome the cold-supply chain constraints.

 The IP4IC Study



We addressed these challenges through our innovative IP4IC study, which took advantage of crowdsourcing samples in collaboration with social media (Facebook, Twitter, E-mail blasts, and YouTube) and an IC/BPS patient advocacy group, the IC Association, to drive recruitment and engagement of both IC/BPS patients and asymptomatic controls [Figure 1]. We circumvented the need for a cold-supply chain for urine sample collection using a commercially available urine preservative that allowed shipment and storage of urine samples at ambient temperature (Norgen Biotek). Participants were sent a urine specimen collection kit that allowed for urine collection, shipment, and storage at ambient room temperature through the use of a urine preservative directly manufactured directly into the urine collection cup. We validated that the urinary proteins of interest were stable at temperatures up to 30°C for up to 1 year, so mailing kits also included an irreversible temperature label that indicated if and for how long a package had been exposed to this temperature. The IP4IC Study was immensely successful; enrollment goals were attained within 2 weeks of study launch and we collected urine samples from 448 participants representing 46 US States in 3 months [Figure 2].[1],[27] The IP4IC/BPS study samples consisted of 153 IC/BPS patients (147 females, 6 males), of which 54 were HIC/BPS patients (50 females, 4 males), and 159 females and 136 males age-matched controls. Corresponding demographics, IC/BPS IC symptom index (ICSI) and IC pain index, and bladder voiding diaries were also collected. Results were validated through an independent study (termed the P3 study) that was collected in the clinic with physician associated IC/BPS diagnosis. Crowdsourcing for sample collection proved to be fast, reliable, cost-effective, and efficient at collecting large numbers of both IC/BPS and control participants nationally.{Figure 1}{Figure 2}

 Machine Learning in Biomarker Development



Machine learning (ML) is increasingly utilized to deal with complex biological data. Random forest classification (RFC) is an efficient ML method for classification of samples based on genetic or proteomic expression.[28] The expression of three urinary cytokines were quantified using Luminex Assays [Figure 3]a,[Figure 3]b,[Figure 3]c. Although there was a significant difference in the means between HIC compared to controls or NHIC, there was considerable overlap between the potential biomarkers if used independently [Figure 3]a,[Figure 3]b,[Figure 3]c. We utilized RFC to combine the three biomarkers from the IP4IC training set into a RS, which identifies patients with HIC versus NHIC versus asymptomatic controls [Figure 3]d. We then did external validation in a separate and clinically collected validation set, termed P3 [Figure 3]d. A ROC analysis resulted in an impressive AUC for the IP4IC training dataset of 0.971 and the P3 validation dataset of 0.919 [Figure 3]e.{Figure 3}

Overall, 88.7% of the entire P3 validation set was correctly predicted. Three cytokines (GRO/CXCL1, interleukin [IL]-6, and IL-8) were all required for the RFC to classify patients but they were not equally important (e.g., GRO/CXCL1 contributed more to the final classifier than IL-6). Gender did not impact the results with respect to the RFC. The IP4IC/BPS training set and P3 validation set had a false positive rate of 2.2% and 4.7%, respectively. There was a negative predictive value of 96.0% and 93.2%, respectively, and a positive predictive value of 84.9% and 75.0%, respectively. The methodology, IC-RS score, and the crowdsourcing have been published.[1],[27]

Our current research aims to build a new ML RS to distinguish all IC/BPS participants (both NHIC and HIC) from controls, the IC-RS. We have generated strong pilot data that laid the foundation for this grant proposal. In addition to using the three cytokines described above, another noninvasive parameter was included as a feature in the RFC. This was the corresponding ICSI/PI scores which incorporates measurements for pain and urinary symptoms. Proof-of-concept for the IC-RS tool was generated in follow-up analysis of the initial IC-RS study. This study focused on the P3 cohort as samples from the IP4IC cohort were depleted. For our pilot study, the IC-RS correctly identified 100% (n = 23/23) IC/BPS and 96.2% (n = 25/26) control participants in the P3 cohort [Figure 4]. The AUC of the ROC curve for the pilot set was 0.994. Given the heterogeneous population of IC/BPS, the performance of the IC-RS was both surprising and exciting.{Figure 4}

Our current work supports development of the IC-RS as a new diagnostic multivariate score in a large national cohort of IC/BPS, control, and other confusable pelvic pain disorder participants to evaluate specificity and determine if it can serve as a new clinical outcome assessment. We also seek to develop patient engagement tools and resources effective for crowdsourcing samples. This includes new methodologies for biomarker development to support mass testing of urine samples from across the United States, including online support tools for patient engagement, urine sample shipping kits, and ML for translating urinary protein levels into meaningful disease diagnostic criteria. Samples will be collected from civilians and military personnel by crowdsourcing, which we have successfully done in the past. We will also collect urine samples in clinic from three academic urology departments to compare patient collected samples with samples from physician diagnosed IC/BPS. We will test urine from patients with overlapping symptoms including documented urinary tract infection (UTI), overactive bladder (OAB) with incontinence (OAB-wet), and bladder cancer to assess test specificity. Our goal is to develop a clinical tool for diagnosis and individualized guided treatment, in addition to seeking partnerships with commercial development partners to bring the tool to market.

 Crowdsourcing in Medical Research



We have engaged with IC/BPS patients from around the United States in focus groups prior to crowdsourcing urine collection to engage active study design input from patients' perspectives. Patient and patient advocacy organization engagement and involvement are a central theme of our research that helps make the results clinically meaningful. The translation of ideas from bench-to-bedside is challenging.

We have developed, manufactured, and verified that the urine preservative is effective at stabilizing expression of our proteins of interest for up to 1 year at room temperature [Figure 5]. We send our research participants a participant package includes instructions, a urine collection cup, a questionnaire regarding symptom changes or UTI since completed online survey, a bladder diary, and a prepaid return shipping package. Online instructions and YouTube videos are also provided on the website. All return shipping packages also have nonreversible temperature recording labels indicating temperatures exceeding 30°C for over 30 min. At temperatures exceeding 30°C, there is a decrease in protein stability that may affect downstream testing. In IP4IC, this only impacted 2 out of 448 samples so we do not anticipate this to be an issue. Return shipment packages were prepaid and meet all federal regulations for urine transport.{Figure 5}

Returned urine samples were aliquoted and stored at room temperature and ten 0.5 mL aliquots stored at-80°C. Samples were randomized and assigned an experimental ID number to run and analyze samples in a blinded fashion. Creatinine was be measured by high-performance liquid chromatography to normalize across patients. The expression of a panel of cytokines were be determined using the MILLIPLEX MAP Human Cytokine/Chemokine Multiplex Immunoassay (Millipore) following manufacturer's protocol and detailed in Lamb et al., 2017.[27] A supervised ML method described in[27],[29],[30] was then used to calculate the IC-RS. The IC-RS will classify a patient as having IC/BPS or as a control. If positive for IC/BPS, the IC-RS will also classify the patient as having HIC or NHIC.

 Development of Interstitial Cystitis-Risk Score through Machine Learning



The IC-RS ML classifier was trained using data obtained in our previous urine collection. This RFC binary classifier will categorize subjects as either having or not having IC/BPS based on the levels of measured cytokines and OPSI score. RFC is well suited to datasets of this size and has outperformed other algorithms in our experience with the IC-RS. Each subject will be given a label of 0 (lacks IC/BPS) or 1 (has IC/BPS). Patients that are classified as having IC/BPS will then undergo a second analysis to determine if that patient has HIC (label of 1 in second analysis) or NHIC (label of 0 in second analysis). The RFC algorithm uses measured cytokine levels and OPSI scores of a random subset of subjects to build a single decision tree. The subjects not sampled for constructing the decision tree are used to test the predictive performance of that tree-this is analogs to cross-validation that would be used with other statistical methods. Multiple decision trees are built in this manner. The combination of all these decision trees is the classifier, where subjects are classified based on majority vote of the component decision trees.

Because resampling-based validation is used in training the RFC algorithm, it is necessary to also evaluate the resulting classifier using external validation of another dataset. This approach will aid in the optimization of parameters used to build the IC-RS classifier; various parameters required for the RFC are, for example, the number of decision trees to build which criterion to measure the quality of the splits in the decision tree, and how deep/branched the tree should be. These parameters will be optimized through software that was developed by our group by testing the trained classifiers against the smaller validation dataset. The highly parallelized software can test millions of unique parameter combinations on a standard desktop computer in less than a week in order to obtain a set of parameters that provide the greatest predictive performance and minimize overfitting in the final RFC.

New, unlabeled (blinded) cytokines and OPSI data that is obtained from our proposed study will be tested with the optimal RFC and tested against each decision tree. Each decision tree will provide a probability score for each label (0 or 1). The average of all of these probabilities from all of the trees will determine which prediction has been made for a given subject; based on our preliminary data. If the mean is <0.5, the subject will be classified as lacking IC/BPS; alternatively, if the mean is ≥0.5, the subject will be classified as having IC/BPS. The second analysis for IC/BPS classified patients for having HIC will be determined in a similar manner. After all newly obtained data have been classified, the self-reported diagnosis of each patient will be unblinded so that the performance of the optimal IC-RS classifier can be assessed.

 Translational Research



The result of this research is hopefully a diagnostic product that will be the first regulatory approved clinical laboratory test for IC/BPS. As there are no regulatory approved or objective diagnostic tests for IC/BPS, there remains a great unmet medical need and a value proposition for IC/BPS. IC-RS can aid as companion diagnostic for new IC/BPS treatment. Our current research is the final step for validation of a test that can enter the regulatory approval laboratory certification pathway. Translational research, moving an idea from research laboratory's bench top to a commercial product requires a product that is patent protected. In parallel with our research, we have applied for and received issued patent that covers the technologies descripted in this review.

 Conclusion



We are excited about the potential to apply ML based medical diagnostic score for IC/BPS. Novel features of our project that may serve the broader urology community include:

We have optimized a urine preservative that allows for shipping and assay of urine samples without preparation. No centrifugation or refrigeration with our method of room temperature urine sample preparation and use of standard mail shippingSocial engagement of stakeholders in research through social media with regulatory compliant human research protocolCrowdsourcing of samples instead of hospital and clinic-based samples collectionA diagnostic score based on ML of several urine proteins that may diagnose IC/BPS when individual protein levels lack adequate sensitivity or specificity. This is in line with the U. S. Food Drug Administration's proposed regulatory framework for artificial intelligence/ML based software as a medical device (SaMD).

The result of this research is hopefully a regulatory approved diagnostic product that can help physicians diagnose IC/BPS. We believe a great part of the future of health care is computer science based and SaMD can play an important role. We do not know what the future will hold or if our dream can be achieve. We do know the journal has been an adventure and joy and we want to thank all the people, health-care providers, scientists and patients who have worked with us and helped us.

Financial support and sponsorship

This study was supported by the Office of the Assistant Secretary of Defense for Health Affairs through the Technology/Therapeutic Development Research Program under Award No. W81XWH-19-1-0288. The opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the Department of Defense.

Conflicts of interest

Prof. Michael B. Chancellor, an editorial board member at Urological Science, had no role in the peer review process of or decision to publish this article. The other author declared that she has no conflict of interest.

References

1Chancellor MB, Bartolone SN, Veerecke A, Lamb LE. Crowdsourcing disease biomarker discovery research; The IP4IC/BPS study. J Urol 2017;199:1344-50.
2Clemens JQ, Mullins C, Kusek JW, Kirkali Z, Mayer EA, Rodríguez LV, et al. The MAPP research network: A novel study of urologic chronic pelvic pain syndromes. BMC Urol 2014;14:57.
3Berry SH, Elliott MN, Suttorp M, Bogart LM, Stoto MA, Eggers P, et al. Prevalence of symptoms of bladder pain syndrome/interstitial cystitis among adult females in the United States. J Urol 2011;186:540-4.
4Daniels NA, Link CL, Barry MJ, McKinlay JB. Association between past urinary tract infections and current symptoms suggestive of chronic prostatitis/chronic pelvic pain syndrome. J Natl Med Assoc 2007;99:509-16.
5Suskind AM, Berry SH, Ewing BA, Elliott MN, Suttorp MJ, Clemens JQ. The prevalence and overlap of interstitial cystitis/bladder pain syndrome and chronic prostatitis/chronic pelvic pain syndrome in men: Results of the RAND Interstitial Cystitis Epidemiology male study. J Urol 2013;189:141-5.
6Hepner KA, Watkins KE, Elliott MN, Clemens JQ, Hilton LG, Berry SH. Suicidal ideation among patients with bladder pain syndrome/interstitial cystitis. Urology 2012;80:280-5.
7Grégoire M, Liandier F, Naud A, Lacombe L, Fradet Y. Does the potassium stimulation test predict cystometric, cystoscopic outcome in interstitial cystitis? J Urol 2002;168:556-7.
8Koziol JA, Adams HP, Frutos A. Discrimination between the ulcerous and the nonulcerous forms of interstitial cystitis by noninvasive findings. J Urol 1996;155:87-90.
9Simon LJ, Landis JR, Erickson DR, Nyberg LM. The interstitial cystitis data base study: Concepts and preliminary baseline descriptive statistics. Urology 1997;49:64-75.
10Braunstein R, Shapiro E, Kaye J, Moldwin R. The role of cystoscopy in the diagnosis of Hunner's ulcer disease. J Urol 2008;180:1383-6.
11Tomaszewski JE, Landis JR, Russack V, Williams TM, Wang LP, Hardy C, et al. Biopsy features are associated with primary symptoms in interstitial cystitis: Results from the interstitial cystitis database study. Urology 2001;57:67-81.
12Peeker R, Fall M. Toward a precise definition of interstitial cystitis: Further evidence of differences in classic and nonulcer disease. J Urol 2002;167:2470-2.
13Moutzouris DA, Falagas ME. Interstitial cystitis: An unsolved enigma. Clin J Am Soc Nephrol 2009;4:1844-57.
14Lai HH, Gardner V, Ness TJ, Gereau RW 4th. Segmental hyperalgesia to mechanical stimulus in interstitial cystitis/bladder pain syndrome: Evidence of central sensitization. J Urol 2014;191:1294-9.
15Nickel JC, Tripp DA, Pontari M, Moldwin R, Mayer R, Carr LK, et al. Interstitial cystitis/painful bladder syndrome and associated medical conditions with an emphasis on irritable bowel syndrome, fibromyalgia and chronic fatigue syndrome. J Urol 2010;184:1358-63.
16National Guideline Clearinghouse (NGC). Guideline Summary: Diagnosis and Treatment of Interstitial Cystitis/Bladder Pain Syndrome [web site]. Rockville MD. Agency for Healthcare Research and Quality (AHRQ); 2014. Available from: https://www.guideline.gov/summaries/summary/48745. [Last accessed on 2018 Jun 07].
17Mathias SD, Kuppermann M, Liberman RF, Lipschutz RC, Steege JF. Chronic pelvic pain: Prevalence, health-related quality of life, and economic correlates. Obstet Gynecol 1996;87:321-7.
18Lutgendorf SK, Kreder KJ, Rothrock NE, Ratliff TL, Zimmerman B. Stress and symptomatology in patients with interstitial cystitis: A laboratory stress model. J Urol 2000;164:1265-9.
19Hanno PM, Erickson D, Moldwin R, Faraday MM, American Urological Association. Diagnosis and treatment of interstitial cystitis/bladder pain syndrome: AUA guideline amendment. J Urol 2015;193:1545-53.
20Interstitial Cystitis Association. IC/BPS/BPS Symptom Burden: Results from an Interstitial Cystitis Association (IC/BPSA) Survey. Presented at United States FDA Meeting of the Bone, Reproductive and UrologIC/BPS Drugs Advisory Committee; 12/07/17; Silver Spring, Maryland. Available from: https://www.IC/BPShelp.org/wp-content/uploads/2017/12/FDAslides.pdf2017. [Last accessed on 2020 Nov 01].
21Aquinox PharmaceutIC/BPSals Inc. Aquinox Announces Topline Results of Phase 3 LEADERSHIP 301 ClinIC/BPSal Trial in Interstitial Cystitis/Bladder Pain Syndrome Vancouver, British Columbia: Globe Newswire; 2018. 15 August, 2018. Press Release. Available from: http://investor.aqxpharma.com/news-releases/news-release-details/aquinox-announces-topline-results-phase-3-leadership-301. [Last accessed on 2020 Nov 01].
22O'Leary MP, Sant GR, Fowler FJ Jr, Whitmore KE, Spolarich-Kroll J. The interstitial cystitis symptom index and problem index. Urology 1997;49(5A Suppl):58-63.
23Xu L, Zhang P, Zhang N, Yang Y, Wu ZJ, Zhang CH, et al. Efficiency of O'Leary-Sant symptom index and problem index in the diagnosis of interstitial cystitis. Zhonghua Yi Xue Za Zhi 2013;93:3347-50.
24Kushner L, Moldwin RM. Efficiency of questionnaires used to screen for interstitial cystitis. J Urol 2006;176:587-92.
25Sirinian E, Azevedo K, Payne CK. Correlation between 2 interstitial cystitis symptom instruments. J Urol 2005;173:835-40.
26Humphrey L, Arbuckle R, Moldwin R, Nordling J, van de Merwe JP, Meunier J, et al. The bladder pain/interstitial cystitis symptom score: Development, validation, and identification of a cut score. Eur Urol 2012;61:271-9.
27Lamb LE, Janicki JJ, Bartolone SN, Peters KM, Chancellor MB. Development of an interstitial cystitis risk score for bladder permeability. PLoS One 2017;12:e0185686.
28Qi Y. Random forest for BioinformatIC/BPSs. In: Zhang C, Ma Y, editors. Ensemble Machine Learning: Methods and ApplIC/BPSations. Boston, MA: Springer US; 2012. p. 307-23.
29Mambatta AK, Jayarajan J, Rashme VL, Harini S, Menon S, Kuppusamy J. Reliability of dipstick assay in predicting urinary tract infection. J Family Med Prim Care 2015;4:265-8.
30Memişoğullari R, Yüksel H, Ak Yıldırım H, Yavuz O. Performance characteristIC/BPSs of dipstIC/BPSk and mIC/BPSroscopIC/BPS urinalysis for diagnosis of urinary tract infection. Eur J Gen Med 2010;7:174-8.