Adaptive Hyperheuristic Framework for Hyperparameter Tuning: A Q-Learning-Based Heuristic Selection Approach with Simulated Annealing Acceptance Criteria

Abstract

Hyperparameter tuning is a crucial step in optimizing machine learning models, directly impacting their performance and generalization capabilities. Traditional approaches, such as grid search, random search, and Bayesian optimization, often suffer from inefficiencies, especially in high-dimensional hyperparameter spaces. To address these challenges, this paper proposes an adaptive hyperheuristic framework for hyperparameter tuning, integrating Q-learning-based heuristic selection and simulated annealing acceptance criteria. The proposed  model is referred to as\textit{AHPQA framework} in this article. The framework employs a two-layered approach: a high-level heuristic selection strategy driven by Q-learning, and a set of low-level heuristics categorized into constructive, improvement, and perturbation heuristics. The Q-learning model dynamically selects the most effective heuristic based on historical performance, ensuring an adaptive exploration-exploitation balance. Additionally, the acceptance of new hyperparameter configurations follows a simulated annealing-based probabilistic function, allowing the search process to escape local optima. The proposed method is evaluated on benchmark machine learning models, including deep learning architectures and ensemble classifiers, using publicly available datasets. Comparative analysis against conventional tuning approaches demonstrates superior convergence speed, computational efficiency, and model performance. The results indicate that the adaptive hyperheuristic approach significantly reduces the computational overhead while achieving competitive or improved model accuracy. This study contributes a novel hyperheuristic-based optimization framework for hyperparameter tuning, providing a scalable, adaptable, and efficient solution applicable across various machine learning domains. Future research directions include extending the framework to reinforcement learning environments and integrating explainable AI techniques for improved interpretability.