I'm quite new to Bayesian models. I'm trying to implement Bayesian Optimization where the sum of parameters should be less than a constant.
Say, the parameters are x1, x2, x3;
then, (x1 + x2 + x3) < k where k is a constant and x1>0, x2>0, X3>0.
Ultimately I'm trying to find the values of x1, x2, and x3 such that y is minimum where y = f(x1, x2, x3)
The issue I have is, what is the best method to obey the constraint (or the relationship) on the parameters during this optimization.
Highly appreciate if someone can point me in the direction, or provide an example of a similar problem.
Best Answer
One simple solution is to search an unconstrained space (i.e. the d-dimensional real numbers for some dimension $d$) and transform to the constraint space after.
In your case, you can search on $\bf{x}' \in \mathcal{R}^3$. Then, when you use the parameter values, you introduce an artificial $x_4' := -\sum_j^3 x_j'$. The, you take the softmax and multiply by $k$: i.e. $x_i = k \times \frac{e^{x_i'}}{\sum_j^4 e^{x_j'}}$.
The resulting $x_1$, $x_2$ and $x_3$ satisfy your constraints and you don't use $x_4$ any further.
Why do we introduce $x_4'$ in the way we do? We could search $\bf{x}' \in \mathcal{R}^4$ in the first place, but then the problem would be overparameterized and we'd definitely have an infinite number of equivalent solutions in 4-D that (not helpful, just bound to cause problems). Thus, we do it this way (it's a trick I've seen used for how to map a unconstrained space to a simplex for Bayesian modeling, e.g. for the parameter vector of a categorical distribution). When $x_4'$ is a lot smaller than the other values (i.e. when $x_4$ is really close to zero), the constraint will be nearly exactly reached, otherwise the sum will be smaller than $k$.