How to use sympy library to create a leakey Relu function [For implementation]

Question

This question appears to be off-topic because it focuses on programming, debugging, or performing routine operations, or it asks about obtaining datasets. You could try the support links we maintain or the Open Data site instead.

---

If the question is actually a statistical topic disguised as a coding question, then OP should edit the question to clarify this. After the statistical content has been clarified, the question is eligible for reopening.

Closed 2 years ago.

Improve this question

the main problem is that how to use sympy library to create a Leakey Relu function.
Now I can use NumPy to construct the Leakey Relu function. I found three methods to do it as the following codes shown

import numpy as np                                                 

x = np.random.normal(size=[1, 5])

# first approach                           
leaky_way1 = np.where(x > 0, x, x * 0.01)                          

# second approach                                                                   
y1 = ((x > 0) * x)                                                 
y2 = ((x <= 0) * x * 0.01)                                         
leaky_way2 = y1 + y2

# Third approach   
def leaky_relu(arr):
    alpha = 0.1
 
    return np.maximum(alpha*arr, arr)

But I need to calculate the intergral of variables's leakey relu, so I use the sympy library to construct the leakey relu first.

but I have no idea in compute the judgment used to determine whether an element is greater than zero such as np.where(x > 0),(x > 0),and np.maximum()

Thanks a lot!

Answer 1

The leaky ReLu is given as $f(x)= \begin{cases} \alpha x & ,x\leq 0 \\x & ,otherwise \end{cases} \qquad ,with \quad\alpha = 0.01$

So you can determine the antiderivative, to calculate integrals as: $F(x)= \begin{cases} \frac{\alpha}{2} x^2 & ,x\leq 0 \\\frac{1}{2} x^2 & ,otherwise \end{cases}$

You can implement the antiderivative with numpys where() function as well:

leaky_relu = np.where(x > 0, x, x * 0.01)  
leaky_relu_integral = np.where(x > 0, x * x / 2, x * x * 0.01 / 2)  

For sympy (V1.8) you can implement leaky ReLu using the piecewise function:

from sympy import *  
x = Symbol('x')
leaky_relu = Piecewise((0.01*x,x<0),(x, x>=0))

# test leaky relu
leaky_relu.subs(x, 5)  # gives 5
leaky_relu.subs(x, -5) # gives -0.05

# integrate leaky relu
integrate(leaky_relu, x)