Solved – Definition of Regressor

machine learning

Hi i have been learning Machine Learning for over six months, and I started to see the term Regressor and can 't really google it out for an exact definetion.

I fully understand logistic regression and linear regression..but just can't figure out this term,..(and sometimes so many terms coming out from ML but they more or less refers to similar things..)

I just got this answer from wiki:

The independent variables, also known in a statistical context as regressors, represent inputs or causes, i.e., potential reasons for variation or, in the experimental setting, the variable controlled by the experimenter.

Is this correct – the learning of ML, to me, really is so much slightly varying terminologies… (which can be completely absent in one course or framework, and suddenly they appear elsewhere..)

Best Answer

Yes, your understanding is correct. Feature, independent variable, explanatory variable, regressor, covariate, or predictor are all names of the variables that are used to predict the target, outcome, dependent variable, regressand, or response. The terminology is ambiguous as it comes from different fields: statistics, econometrics, and machine learning.

Related Solutions

Machine Learning – Difference Between Supervised Machine Learning and Design of Experiments

Your question is difficult to answer because there is no "supervised ML algorithm". There are a large number of different ML algorithms that can be optimized in a supervised fashion, each with their strengths and weaknesses.

On a very abstract level, you can define Machine Learning (ML) as a search through some space $P$ for a parameterization ($\theta$) of a given model $M$ such that $M(x;\theta)$ gives a minimal value (though not always globally) for the cost function ($\mathcal{C}$) and input ($x$). More formally:

$$\arg\min_{\theta\in P} \mathcal{C}(M(\theta))$$

For supervised learning, one form the cost function can take is (given $y$ as ground truth):

$$\mathcal{C}(M(x; \theta)) = ||M(x;\theta) - y||_2$$

Any search through $P$ that minimizes the cost function can fit in this framework, and thus you can claim DOE as a ML algorithm if you want. Specifically, an ML algorithm is defined by: the optimization technique employed, the model used, and your cost function. If you fill those in for DOE, you can start to compare it against other ML algorithms.

Solved – feature scaling giving reduced output (linear regression using gradient descent)

My guess is that you have accidentally transformed y_train (somewhere hidden in the code you have not posted). This because this reproducible snippets works

import numpy as np
import pandas as pd
import math
from sklearn import preprocessing

dat = pd.read_csv("/home/steffen/workspaces/airfoil/airfoil_self_noise.dat",sep="\t",low_memory=False,header=None)

apply_scaler = True

# split into train 2/3 and test 1/3
rng = np.random.RandomState(42)

n_rows = dat.shape[0]
n_train = math.floor(0.66*n_rows)

permutated_indices = rng.permutation(n_rows)

train_dat = dat.loc[permutated_indices[:n_train],:]
test_dat =  dat.loc[permutated_indices[n_train:],:]

# separate the response variable (last column) from the predictor variables
x_train = train_dat.iloc[:,1:-1]
y_train = (train_dat.iloc[:,-1])[:, np.newaxis]

x_test = test_dat.iloc[:,1:-1]
y_test = (test_dat.iloc[:,-1])[:, np.newaxis]

# train
# fit the scaler to predictor variables and apply it afterwards
scaler = preprocessing.StandardScaler().fit(x_train)

if apply_scaler:
    x_train = pd.DataFrame(scaler.transform(x_train))

# add constant one for the intercept parameter
x_train = pd.concat([pd.DataFrame(np.ones(shape=(x_train.shape[0],1)),index=x_train.index),x_train],axis=1)

# fit parameters of linear regression using batch gradient descent
# Hands-On Machine Learning with Scikit-Learn & Tensorflow, page 115
eta = 0.1 # learning rate
n_iterations = 1000
m = x_train.shape[0]
theta = rng.randn(x_train.shape[1],1)

for iteration in range(n_iterations):
    gradients = (2 / m) * x_train.T.dot(x_train.dot(theta) - y_train)
    theta = theta - eta * gradients

# to apply the fitted parameters, first we have to transform the test-data in the same way
# apply scaler
if apply_scaler:
    x_test = pd.DataFrame(scaler.transform(x_test))

# add constant one for the intercept parameter
x_test = pd.concat([pd.DataFrame(np.ones(shape=(x_test.shape[0],1)),index=x_test.index),x_test],axis=1)

# apply fitted parameters
y_predict =x_test.dot(theta)

# compare output
out=np.column_stack((y_test, y_predict))
print(pd.DataFrame(out).head())
# root mean squared error
print("error %f"% np.sqrt(np.power(y_test-y_predict,2).mean()))

This leads to this output

         0           1
0  120.573  127.108268
1  127.220  123.492931
2  113.045  122.393120
3  119.606  122.570836
4  131.971  127.270743
error 6.175637

which is fine.

It is interesting to see that for learning rate 0.1 this simple batch gradient descent implementation fails to converge if no normalization is performed (apply_scaler=False, eta=0.1), while the Linear Regression implementation of scikit learn still finds a solution. Reducing the learning rate dramatically (eta=0.0001) leads to convergence again.

This is one example where the Gradient Descent is limited, as discussed here: Do we need gradient descent to find the coefficients of a linear regression model.

Best Answer

Related Solutions

Machine Learning – Difference Between Supervised Machine Learning and Design of Experiments

Solved – feature scaling giving reduced output (linear regression using gradient descent)

Related Question