{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,9]],"date-time":"2026-04-09T19:59:23Z","timestamp":1775764763606,"version":"3.50.1"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1012639","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2025,1,17]],"date-time":"2025-01-17T00:00:00Z","timestamp":1737072000000}}],"reference-count":44,"publisher":"Public Library of Science (PLoS)","issue":"1","license":[{"start":{"date-parts":[[2025,1,7]],"date-time":"2025-01-07T00:00:00Z","timestamp":1736208000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/linproxy.fan.workers.dev:443\/http\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1745302"],"award-info":[{"award-number":["1745302"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Machine learning sequence-function models for proteins could enable significant advances in protein engineering, especially when paired with state-of-the-art methods to select new sequences for property optimization and\/or model improvement. Such methods (Bayesian optimization and active learning) require calibrated estimations of model uncertainty. While studies have benchmarked a variety of deep learning uncertainty quantification (UQ) methods on standard and molecular machine-learning datasets, it is not clear if these results extend to protein datasets. In this work, we implemented a panel of deep learning UQ methods on regression tasks from the Fitness Landscape Inference for Proteins (FLIP) benchmark. We compared results across different degrees of distributional shift using metrics that assess each UQ method\u2019s accuracy, calibration, coverage, width, and rank correlation. Additionally, we compared these metrics using one-hot encoding and pretrained language model representations, and we tested the UQ methods in retrospective active learning and Bayesian optimization settings. Our results indicate that there is no single best UQ method across all datasets, splits, and metrics, and that uncertainty-based sampling is often unable to outperform greedy sampling in Bayesian optimization. These benchmarks enable us to provide recommendations for more effective design of biological sequences using machine learning.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1012639","type":"journal-article","created":{"date-parts":[[2025,1,7]],"date-time":"2025-01-07T13:45:57Z","timestamp":1736257557000},"page":"e1012639","update-policy":"https:\/\/linproxy.fan.workers.dev:443\/https\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":12,"title":["Benchmarking uncertainty quantification for protein engineering"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/linproxy.fan.workers.dev:443\/https\/orcid.org\/0000-0002-6466-1401","authenticated-orcid":true,"given":"Kevin P.","family":"Greenman","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/linproxy.fan.workers.dev:443\/https\/orcid.org\/0000-0002-8601-6040","authenticated-orcid":true,"given":"Ava P.","family":"Amini","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/linproxy.fan.workers.dev:443\/https\/orcid.org\/0000-0001-9045-6826","authenticated-orcid":true,"given":"Kevin K.","family":"Yang","sequence":"additional","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2025,1,7]]},"reference":[{"issue":"8","key":"pcbi.1012639.ref001","doi-asserted-by":"crossref","first-page":"687","DOI":"10.1038\/s41592-019-0496-6","article-title":"Machine-learning-guided directed evolution for protein engineering","volume":"16","author":"KK Yang","year":"2019","journal-title":"Nature Methods"},{"key":"pcbi.1012639.ref002","unstructured":"Kendall A, Gal Y. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, et al., editors. Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc.; 2017. Available from: https:\/\/linproxy.fan.workers.dev:443\/https\/proceedings.neurips.cc\/paper_files\/paper\/2017\/file\/2650d6089a6d640c5e85b2b88265dc2b-Paper.pdf."},{"key":"pcbi.1012639.ref003","doi-asserted-by":"crossref","unstructured":"Dallago C, Mou J, Johnston KE, Wittmann BJ, Bhattacharya N, Goldman S, et al.. FLIP: Benchmark tasks in fitness landscape inference for proteins; BioRxiv [Preprint]. 2021. Available from: https:\/\/linproxy.fan.workers.dev:443\/https\/www.biorxiv.org\/content\/10.1101\/2021.11.09.467890v2.","DOI":"10.1101\/2021.11.09.467890"},{"issue":"6","key":"pcbi.1012639.ref004","doi-asserted-by":"crossref","first-page":"2697","DOI":"10.1021\/acs.jcim.9b00975","article-title":"Evaluating scalable uncertainty estimation methods for deep learning-based molecular property prediction","volume":"60","author":"G Scalia","year":"2020","journal-title":"Journal of Chemical Information and Modeling"},{"issue":"2","key":"pcbi.1012639.ref005","first-page":"025006","article-title":"Methods for comparing uncertainty quantifications for material property predictions","volume":"1","author":"K Tran","year":"2020","journal-title":"Machine Learning: Science and Technology"},{"issue":"8","key":"pcbi.1012639.ref006","doi-asserted-by":"crossref","first-page":"3770","DOI":"10.1021\/acs.jcim.0c00502","article-title":"Uncertainty quantification using neural networks for molecular property prediction","volume":"60","author":"L Hirschfeld","year":"2020","journal-title":"Journal of Chemical Information and Modeling"},{"issue":"9","key":"pcbi.1012639.ref007","doi-asserted-by":"crossref","first-page":"1009","DOI":"10.1080\/17460441.2021.1925247","article-title":"Assigning confidence to molecular property prediction","volume":"16","author":"A Nigam","year":"2021","journal-title":"Expert Opinion on Drug Discovery"},{"issue":"8","key":"pcbi.1012639.ref008","doi-asserted-by":"crossref","first-page":"1356","DOI":"10.1021\/acscentsci.1c00546","article-title":"Evidential deep learning for guided molecular property prediction and discovery","volume":"7","author":"AP Soleimany","year":"2021","journal-title":"ACS Central Science"},{"issue":"2","key":"pcbi.1012639.ref009","first-page":"025019","article-title":"Clarifying trust of materials property predictions using neural networks with distribution-specific uncertainty quantification","volume":"4","author":"CJ Gruich","year":"2023","journal-title":"Machine Learning: Science and Technology"},{"key":"pcbi.1012639.ref010","unstructured":"Mariet Z, Jerfel G, Wang Z, Angerm\u00fcller C, Belanger D, Vora S, et al. Deep Uncertainty and the Search for Proteins. In: NeurIPS Workshop: Machine Learning for Molecules; 2020. Available from: https:\/\/linproxy.fan.workers.dev:443\/https\/ml4molecules.github.io\/papers2020\/ML4Molecules_2020_paper_23.pdf."},{"issue":"5","key":"pcbi.1012639.ref011","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1016\/j.cels.2020.09.007","article-title":"Leveraging Uncertainty in Machine Learning Accelerates Biological Discovery and Design","volume":"11","author":"B Hie","year":"2020","journal-title":"Cell Systems"},{"issue":"15","key":"pcbi.1012639.ref012","doi-asserted-by":"crossref","first-page":"4589","DOI":"10.1021\/acs.jcim.3c00601","article-title":"Linear-Scaling kernels for protein sequences and small molecules outperform deep learning while providing uncertainty quantitation and improved interpretability","volume":"63","author":"J Parkinson","year":"2023","journal-title":"Journal of Chemical Information and Modeling"},{"key":"pcbi.1012639.ref013","unstructured":"Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding; arXiv:1810.04805 [Preprint]. 2019. Available from: https:\/\/linproxy.fan.workers.dev:443\/https\/arxiv.org\/abs\/1810.04805."},{"key":"pcbi.1012639.ref014","unstructured":"Gruver N, Stanton S, Kirichenko P, Finzi M, Maffettone P, Myers V, et al. Effective surrogate models for protein design with bayesian optimization. In: ICML Workshop on Computational Biology; 2021. Available from: https:\/\/linproxy.fan.workers.dev:443\/https\/icml-compbio.github.io\/2021\/papers\/WCBICML2021_paper_61.pdf."},{"issue":"15","key":"pcbi.1012639.ref015","doi-asserted-by":"crossref","first-page":"e2016239118","DOI":"10.1073\/pnas.2016239118","article-title":"Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences","volume":"118","author":"A Rives","year":"2021","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"3","key":"pcbi.1012639.ref016","doi-asserted-by":"crossref","first-page":"415","DOI":"10.1162\/neco.1992.4.3.415","article-title":"Bayesian interpolation","volume":"4","author":"DJ MacKay","year":"1992","journal-title":"Neural Computation"},{"issue":"Jun","key":"pcbi.1012639.ref017","first-page":"211","article-title":"Sparse Bayesian learning and the relevance vector machine","volume":"1","author":"ME Tipping","year":"2001","journal-title":"Journal of Machine Learning Research"},{"key":"pcbi.1012639.ref018","doi-asserted-by":"crossref","unstructured":"Williams CK, Rasmussen CE. Gaussian Processes for Machine Learning. vol. 2. MIT Press Cambridge, MA; 2006.","DOI":"10.7551\/mitpress\/3206.001.0001"},{"key":"pcbi.1012639.ref019","unstructured":"Gal Y, Ghahramani Z. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In: Balcan MF, Weinberger KQ, editors. Proceedings of The 33rd International Conference on Machine Learning. vol. 48 of Proceedings of Machine Learning Research. New York, New York, USA: PMLR; 2016. p. 1050\u20131059. Available from: https:\/\/linproxy.fan.workers.dev:443\/https\/proceedings.mlr.press\/v48\/gal16.html."},{"key":"pcbi.1012639.ref020","unstructured":"Lakshminarayanan B, Pritzel A, Blundell C. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, et al., editors. Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc.; 2017. Available from: https:\/\/linproxy.fan.workers.dev:443\/https\/proceedings.neurips.cc\/paper\/2017\/file\/9ef2ed4b7fd2c810847ffa5fa85bce38-Paper.pdf."},{"key":"pcbi.1012639.ref021","unstructured":"Amini A, Schwarting W, Soleimany A, Rus D. Deep Evidential Regression. In: Larochelle H, Ranzato M, Hadsell R, Balcan MF, Lin H, editors. Advances in Neural Information Processing Systems. vol. 33. Curran Associates, Inc.; 2020. p. 14927\u201314937. Available from: https:\/\/linproxy.fan.workers.dev:443\/https\/proceedings.neurips.cc\/paper\/2020\/file\/aab085461de182608ee9f607f3f7d18f-Paper.pdf."},{"key":"pcbi.1012639.ref022","doi-asserted-by":"crossref","unstructured":"Nix DA, Weigend AS. Estimating the mean and variance of the target probability distribution. In: Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN\u201994). vol. 1. IEEE; 1994. p. 55\u201360.","DOI":"10.1109\/ICNN.1994.374138"},{"key":"pcbi.1012639.ref023","article-title":"Stochastic variational inference","author":"MD Hoffman","year":"2013","journal-title":"Journal of Machine Learning Research"},{"issue":"3","key":"pcbi.1012639.ref024","doi-asserted-by":"crossref","first-page":"E193","DOI":"10.1073\/pnas.1215251110","article-title":"Navigating the protein fitness landscape with Gaussian processes","volume":"110","author":"PA Romero","year":"2013","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"10","key":"pcbi.1012639.ref025","doi-asserted-by":"crossref","first-page":"e1005786","DOI":"10.1371\/journal.pcbi.1005786","article-title":"Machine learning to design integral membrane channelrhodopsins for efficient eukaryotic expression and plasma membrane localization","volume":"13","author":"CN Bedbrook","year":"2017","journal-title":"PLOS Computational Biology"},{"issue":"11","key":"pcbi.1012639.ref026","doi-asserted-by":"crossref","first-page":"1176","DOI":"10.1038\/s41592-019-0583-8","article-title":"Machine learning-guided channelrhodopsin engineering enables minimally invasive optogenetics","volume":"16","author":"CN Bedbrook","year":"2019","journal-title":"Nature Methods"},{"issue":"1","key":"pcbi.1012639.ref027","doi-asserted-by":"crossref","first-page":"5825","DOI":"10.1038\/s41467-021-25831-w","article-title":"Machine learning-guided acyl-ACP reductase engineering for improved in vivo fatty alcohol production","volume":"12","author":"JC Greenhalgh","year":"2021","journal-title":"Nature Communications"},{"key":"pcbi.1012639.ref028","volume-title":"Bayesian learning for neural networks","author":"RM Neal","year":"2012"},{"issue":"6","key":"pcbi.1012639.ref029","doi-asserted-by":"crossref","first-page":"1596","DOI":"10.1021\/ci5001168","article-title":"Introducing conformal prediction in predictive modeling. A transparent and flexible alternative to applicability domain determination","volume":"54","author":"U Norinder","year":"2014","journal-title":"Journal of Chemical Information and Modeling"},{"issue":"43","key":"pcbi.1012639.ref030","doi-asserted-by":"crossref","first-page":"e2204569119","DOI":"10.1073\/pnas.2204569119","article-title":"Conformal prediction under feedback covariate shift for biomolecular design","volume":"119","author":"C Fannjiang","year":"2022","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"15","key":"pcbi.1012639.ref031","doi-asserted-by":"crossref","first-page":"5540","DOI":"10.3390\/s22155540","article-title":"Evaluating and calibrating uncertainty prediction in regression tasks","volume":"22","author":"D Levi","year":"2022","journal-title":"Sensors"},{"issue":"477","key":"pcbi.1012639.ref032","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1198\/016214506000001437","article-title":"Strictly proper scoring rules, prediction, and estimation","volume":"102","author":"T Gneiting","year":"2007","journal-title":"Journal of the American Statistical Association"},{"issue":"6637","key":"pcbi.1012639.ref033","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1126\/science.ade2574","article-title":"Evolutionary-scale prediction of atomic-level protein structure with a language model","volume":"379","author":"Z Lin","year":"2023","journal-title":"Science"},{"key":"pcbi.1012639.ref034","unstructured":"Zelikman E, Healy C, Zhou S, Avati A. CRUDE: Calibrating Regression Uncertainty Distributions Empirically; arXiv:2005.12496 [Preprint]. 2021. Available from: https:\/\/linproxy.fan.workers.dev:443\/https\/arxiv.org\/abs\/2005.12496."},{"key":"pcbi.1012639.ref035","unstructured":"Kirsch A, van Amersfoort J, Gal Y. BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning. In: Wallach H, Larochelle H, Beygelzimer A, d'Alch\u00e9-Buc F, Fox E, Garnett R, editors. Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc.; 2019. Available from: https:\/\/linproxy.fan.workers.dev:443\/https\/proceedings.neurips.cc\/paper_files\/paper\/2019\/file\/95323660ed2124450caaac2c46b5ed90-Paper.pdf."},{"key":"pcbi.1012639.ref036","unstructured":"Shanehsazzadeh A, Belanger D, Dohan D. Is Transfer Learning Necessary for Protein Landscape Prediction?; arXiv:2011.03443 [Preprint]. 2020. Available from: https:\/\/linproxy.fan.workers.dev:443\/https\/arxiv.org\/abs\/2011.03443."},{"key":"pcbi.1012639.ref037","unstructured":"Kingma DP, Ba J. Adam: A Method for Stochastic Optimization; arXiv:1412.6980 [Preprint]. 2017. Available from: https:\/\/linproxy.fan.workers.dev:443\/https\/arxiv.org\/abs\/1412.6980."},{"key":"pcbi.1012639.ref038","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"F Pedregosa","year":"2011","journal-title":"Journal of Machine Learning Research"},{"key":"pcbi.1012639.ref039","unstructured":"Gardner J, Pleiss G, Weinberger KQ, Bindel D, Wilson AG. GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration. In: Bengio S, Wallach H, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R, editors. Advances in Neural Information Processing Systems. vol. 31. Curran Associates, Inc.; 2018. Available from: https:\/\/linproxy.fan.workers.dev:443\/https\/proceedings.neurips.cc\/paper_files\/paper\/2018\/file\/27e8e17134dd7083b050476733207ea1-Paper.pdf."},{"key":"pcbi.1012639.ref040","doi-asserted-by":"crossref","unstructured":"Gustafsson FK, Danelljan M, Schon TB. Evaluating scalable bayesian deep learning methods for robust computer vision. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition workshops; 2020. p. 318\u2013319.","DOI":"10.1109\/CVPRW50498.2020.00167"},{"issue":"12","key":"pcbi.1012639.ref041","doi-asserted-by":"crossref","DOI":"10.3390\/e23121608","article-title":"Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures","volume":"23","author":"B Kompa","year":"2021","journal-title":"Entropy"},{"issue":"5","key":"pcbi.1012639.ref042","doi-asserted-by":"crossref","first-page":"3250","DOI":"10.1109\/TIT.2011.2182033","article-title":"Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting","volume":"58","author":"N Srinivas","year":"2012","journal-title":"IEEE Transactions on Information Theory"},{"key":"pcbi.1012639.ref043","unstructured":"Chapelle O, Li L. An Empirical Evaluation of Thompson Sampling. In: Shawe-Taylor J, Zemel R, Bartlett P, Pereira F, Weinberger KQ, editors. Advances in Neural Information Processing Systems. vol. 24. Curran Associates, Inc.; 2011. Available from: https:\/\/linproxy.fan.workers.dev:443\/https\/proceedings.neurips.cc\/paper_files\/paper\/2011\/file\/e53a0a2978c28872a4505bdb51db06dc-Paper.pdf."},{"key":"pcbi.1012639.ref044","doi-asserted-by":"crossref","unstructured":"Reuther A, Kepner J, Byun C, Samsi S, Arcand W, Bestor D, et al. Interactive supercomputing on 40,000 cores for machine learning and data analysis. In: 2018 IEEE High Performance extreme Computing Conference (HPEC). IEEE; 2018. p. 1\u20136.","DOI":"10.1109\/HPEC.2018.8547629"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1012639","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2025,1,17]],"date-time":"2025-01-17T00:00:00Z","timestamp":1737072000000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/dx.plos.org\/10.1371\/journal.pcbi.1012639","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,17]],"date-time":"2025-01-17T15:15:47Z","timestamp":1737126947000},"score":1,"resource":{"primary":{"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/dx.plos.org\/10.1371\/journal.pcbi.1012639"}},"subtitle":[],"editor":[{"given":"Rachel","family":"Kolodny","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2025,1,7]]},"references-count":44,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,1,7]]}},"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/doi.org\/10.1371\/journal.pcbi.1012639","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.04.17.536962","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,7]]}}}