International Journal of Computer
Trends and Technology

Research Article | Open Access | Download PDF

Volume 72 | Issue 7 | Year 2024 | Article Id. IJCTT-V72I7P114 | DOI : https://doi.org/10.14445/22312803/IJCTT-V72I7P114

Empirical Assessment of Nonconformity Score Functions for Classifiers in Conformal Prediction

Bhargava Kumar, Tejaswini Kumar, Swapna Nadakuditi

Received Revised Accepted Published
20 May 2024 29 Jun 2024 19 Jul 2024 31 Jul 2024

Citation :

Bhargava Kumar, Tejaswini Kumar, Swapna Nadakuditi, "Empirical Assessment of Nonconformity Score Functions for Classifiers in Conformal Prediction," International Journal of Computer Trends and Technology (IJCTT), vol. 72, no. 7, pp. 108-112, 2024. Crossref, https://doi.org/10.14445/22312803/ IJCTT-V72I7P114

Abstract

Conformal prediction provides statistically guaranteed confidence measures for any machine learning model. This study investigates the effectiveness of three non-conformity score functions, namely Adaptive Prediction Sets (APS), Regularized Adaptive Prediction Sets (RAPS), and Sorted Adaptive Prediction Sets (SAPS), for sentiment analysis tasks. Expanding on past research that demonstrated the superiority of SAPS in classification tasks for image data, this study assesses whether this superiority extends to other domains, such as sentiment analysis. The study aims to evaluate these non-conformity score functions based on coverage and set sizes. The researchers conducted extensive experiments on a sentiment classification task using the GoEmotions dataset to gain insights into the versatility of SAPS and compared its performance with APS and RAPS. By examining the effectiveness of these non-conformity score functions, this study contributes to the understanding of the practicality of conformal prediction methods in real-world machine learning tasks beyond image classification.

Keywords

Adaptive Prediction Sets (APS), Conformal prediction, Non-Conformity Score Functions, Regularized Adaptive Prediction Sets (RAPS), Sorted Adaptive Prediction Sets (SAPS).

References

[1] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer, Algorithmic Learning in a Random World, Springer, pp. 1-476, 2022.
[Google Scholar] [Publisher Link]
[2] Jonathan Alvarsson et al., “Predicting with Confidence: Using Conformal Prediction in Drug Discovery,” Journal of Pharmaceutical Sciences, vol. 110, no. 1, pp. 42–49, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[3] Margaux Zaffran et al., “Adaptive Conformal Predictions for Time Series,” Proceedings of the 39th International Conference on Machine Learning, pp. 25834-25866, 2022.
[Google Scholar] [Publisher Link]
[4] Xianghao Zhan et al., “An Electronic Nose-based Assistive Diagnostic Prototype for Lung Cancer Detection with Conformal Prediction,” Measurement, vol. 158, p. 107588, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[5] Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès, “Classification with Valid and Adaptive Coverage,” arXiv, 2020.
[CrossRef] [Publisher Link]
[6] Anastasios Angelopoulos et al., “Uncertainty Sets for Image Classifiers using Conformal Prediction,” arXiv, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[7] Jianguo Huang et al., “Conformal Prediction for Deep Classifier via Label Ranking,” arXiv, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[8] Dorottya Demszky et al., “GoEmotions: A Dataset of Fine-Grained Emotions,” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4040-4054, 2020.
[CrossRef] [Publisher Link]
[9] Jesse C. Cresswell et al., “Conformal Prediction Sets Improve Human Decision Making,” arXiv, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[10] Yinhan Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” arXiv, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[11] Hugging Face, SamLowe/roberta-base-go_emotions, 2023. [Online]. Available: https://huggingface.co/SamLowe/roberta-base-go_emotions
[12] Hongxin Wei, and Jianguo Huang, “TorchCP: A Library for Conformal Prediction based on PyTorch,” arXiv, 2024.
[CrossRef] [Google Scholar] [Publisher Link]