Optimization – Best Package for Hyperparameter Optimization with Categorical Values

Question

Context

I'm trying to solve a black-box optimization problem, and I can "reformulate" parts of the problem is different ways that may lead to lower or higher costs, and which can interact between themselves.

Currently I have 8 parameters I can control, 7 of which have 3 different approaches and 1 which has 2 different approaches. Thus, there are $2 \times 3^7 = 4374$ different formulations of the same problem. Once I fix one of the formulations, I apply simulated annealing to the problem itself to obtain the lowest cost possible. To obtain results that are usable to me, I need to run SA for dozens of minutes to a couple of hours on the chosen formulation.

I see many parallels between this and the idea of hyperparameter optimization, so I decided to apply the idea to find what is the best among the 4,374 formulations of my problem, without exhaustive search (since each evaluation, running SA for dozen of minutes to a couple of hours, is quite expensive).

Attempts at a solution

I found this ranking of different Python optimization packages. I tried to go for the top ranked package, but it doesn't directly support categorical variables. I tried to follow the suggestion given on its documentation, which consists of rounding continuous values to integers, and then mapping to categories. For instance, if I have 3 possible categories, I choose a bound of $[0,3]$, and map the first category to $[0,1)$, the second to $[1,2)$, and the third to $[2,3]$. Unfortunately, I'm not getting very good results — there is no pattern of finding better formulations as time goes by, and moreover, I see that often the same formulations are tested, given that they actually represent different points of the continuous 8-dimensional space. Here is a plot of the obtained costs after running for 200 iterations:

It appears to be that there is no improvement over time, which is not what I would expect, given this is a Bayesian optimization package. Note also that there are some discussions and requests for implementing integer/categorical variables in this package, but there is nothing concrete yet.

I also tried a different package, also based on Bayesian optimization, which despite a much lower position in the ranking, does have native support for categorical variables. The following plot shows the progress over time, which unlike the previous case, does appear to be converging to better solutions over time:

The questions

I have two questions:

1. Are there high-performing packages (using Bayesian optimization or otherwise) for hyperparameter optimization which natively support categorical variables?

2. Failing that, is there a better alternative to mapping continuous values to categorical data that achieves decent performance?

Answer 1

While I haven't personally used it, I believe SMAC3 should do what you're looking for.