Needle-in-a-Haystack problems exist across a wide range of applications including rare disease prediction,ecological resource management,fraud detection,and material property optimization.A Needle-in-a-Haystack proble...Needle-in-a-Haystack problems exist across a wide range of applications including rare disease prediction,ecological resource management,fraud detection,and material property optimization.A Needle-in-a-Haystack problem arises when there is an extreme imbalance of optimum conditions relative to the size of the dataset.However,current state-of-the-art optimization algorithms are not designed with the capabilities to find solutions to these challenging multidimensional Needle-in-a-Haystack problems,resulting in slow convergence or pigeonholing into a local minimum.In this paper,we present a Zooming Memory-Based Initialization algorithm,entitled ZoMBI,that builds on conventional Bayesian optimization principles to quickly and efficiently optimize Needle-in-a-Haystack problems in both less time and fewer experiments.The ZoMBI algorithm demonstrates compute time speed-ups of 400×compared to traditional Bayesian optimization as well as efficiently discovering optima in under 100 experiments that are up to 3×more highly optimized than those discovered by similar methods.展开更多
基金acknowledges support from the National Research Foundation,Singapore(project No.NRF-NRFF13-2021-0005)the Ministry of Education,Singapore,under its Research Centre of Excellence award to I-FIM(project No.EDUNC-33-18-279-V12).
文摘Needle-in-a-Haystack problems exist across a wide range of applications including rare disease prediction,ecological resource management,fraud detection,and material property optimization.A Needle-in-a-Haystack problem arises when there is an extreme imbalance of optimum conditions relative to the size of the dataset.However,current state-of-the-art optimization algorithms are not designed with the capabilities to find solutions to these challenging multidimensional Needle-in-a-Haystack problems,resulting in slow convergence or pigeonholing into a local minimum.In this paper,we present a Zooming Memory-Based Initialization algorithm,entitled ZoMBI,that builds on conventional Bayesian optimization principles to quickly and efficiently optimize Needle-in-a-Haystack problems in both less time and fewer experiments.The ZoMBI algorithm demonstrates compute time speed-ups of 400×compared to traditional Bayesian optimization as well as efficiently discovering optima in under 100 experiments that are up to 3×more highly optimized than those discovered by similar methods.
