Linear gradient regression in Java
This is a bit of a long shot, but I am wondering if anyone can take a look at this. Am I doing batch gradient descent for linear regression correctly? It gives the expected answers for one independent and interception, but not for multiple independent variables.
/**
* (using Colt Matrix library)
* @param alpha Learning Rate
* @param thetas Current Thetas
* @param independent
* @param dependent
* @return new Thetas
*/
public DoubleMatrix1D descent(double alpha,
DoubleMatrix1D thetas,
DoubleMatrix2D independent,
DoubleMatrix1D dependent ) {
Algebra algebra = new Algebra();
// ALPHA*(1/M) in one.
double modifier = alpha / (double)independent.rows();
//I think this can just skip the transpose of theta.
//This is the result of every Xi run through the theta (hypothesis fn)
//So each Xj feature is multiplied by its Theata, to get the results of the hypothesis
DoubleMatrix1D hypothesies = algebra.mult( independent, thetas );
//hypothesis - Y
//Now we have for each Xi, the difference between predictect by the hypothesis and the actual Yi
hypothesies.assign(dependent, Functions.minus);
//Transpose Examples(MxN) to NxM so we can matrix multiply by hypothesis Nx1
DoubleMatrix2D transposed = algebra.transpose(independent);
DoubleMatrix1D deltas = algebra.mult(transposed, hypothesies );
// Scale the deltas by 1/m and learning rate alhpa. (alpha/m)
deltas.assign(Functions.mult(modifier));
//Theta = Theta - Deltas
thetas.assign( deltas, Functions.minus );
return( thetas );
}
source to share
There is nothing wrong with your implementation and based on your comment, the problem is in collinearity
what you are calling on creation x2
. This is problematic in regression estimation.
To test your algorithm, you can create two independent columns of random numbers. Select a value w0
, w1
and w2
i.e. coefficients for intercept
, x1
and, x2
respectively. Calculate the dependent value y
.
Then see if your stochastic / batch gradient can reconstruct the values w0
, w1
andw2
source to share