A modified version of sigmoid activation function

exponential functionfunctions

I am training a neural network to output agent actions (reinforcement learning). The value range for actions is [0,1], and I want the init value to be 0.05. I am thinking of using some modified version of sigmoid function, same output range, but a bit skewed, the output for 0 is 0.05 (instead of the default 0.5).

As shown in the following drawing, the curve on the right changes sharper (with bigger gradient) compared with the left side.

Any suggestions on how to construct such a function?

enter image description here

EDIT:
Two possible solutions to this question,

  1. piecewise exponential functions, as suggested by @Minus One-Twelfth,
  2. piecewise (max) sigmoid functions.

Here are some plots (www.wolframalpha.com) for the four functions,

enter image description here

Best Answer

Sketch of a suggestion: One possibility for a piecewise function would be to try something like $y = f(x)$ where $$f(x) = \begin{cases} 0.05e^{ax}& \text{if } x \le 0 \\ 1 - 0.95e^{-bx} & \text{if } x > 0 \end{cases}. $$ (Essentially using exponentials.)

I have left some constants $a$ and $b$ unspecified; you can experiment with different values for them to see what you want. Note that increasing $a$ will make the curve steeper at $x = 0$. Also, if you want the curve to have equal slope either side of $x =0$, you should choose $a$ and $b$ such that $0.05a = 0.95b$. Best of luck!

Related Question