Fast Bayesian optimization of Needle-in-a-Haystack problems using zooming memory-based initialization (ZoMBI)

Needle-in-a-Haystack problems exist across a wide range of applications including rare disease prediction,ecological resource management,fraud detection,and material property optimization.A Needle-in-a-Haystack problem arises when there is an extreme imbalance of optimum conditions relative to the size of the dataset.However,current state-of-the-art optimization algorithms are not designed with the capabilities to find solutions to these challenging multidimensional Needle-in-a-Haystack problems,resulting in slow convergence or pigeonholing into a local minimum.In this paper,we present a Zooming Memory-Based Initialization algorithm,entitled ZoMBI,that builds on conventional Bayesian optimization principles to quickly and efficiently optimize Needle-in-a-Haystack problems in both less time and fewer experiments.The ZoMBI algorithm demonstrates compute time speed-ups of 400×compared to traditional Bayesian optimization as well as efficiently discovering optima in under 100 experiments that are up to 3×more highly optimized than those discovered by similar methods.