Multivariable Regression: does “no exact multicolinearity between variables†mean the same as saying that the X's are independent of one another
Clash Royale CLAN TAG#URR8PPP
up vote
0
down vote
favorite
I'm looking at some notes on multivariable regression. It states that one of the assumptions for Least Squares to give good estimates is that,
"There is no exact linear relationship between any of the variables (no exact multicolinearity)".
Would this be the same as saying that the explanatory variables are all independent of one another.
On a related note, does the word "exact" in the statement above refer to perfect correlation between two variables. What if the correlation was strong but not perfect, would OLS still give good estimates?
statistics regression
asked Aug 26 at 21:36
NumberCruncher
767
add a comment |Â
up vote
0
down vote
favorite
I'm looking at some notes on multivariable regression. It states that one of the assumptions for Least Squares to give good estimates is that,
"There is no exact linear relationship between any of the variables (no exact multicolinearity)".
Would this be the same as saying that the explanatory variables are all independent of one another.
On a related note, does the word "exact" in the statement above refer to perfect correlation between two variables. What if the correlation was strong but not perfect, would OLS still give good estimates?
statistics regression
asked Aug 26 at 21:36
NumberCruncher
767
No: Let X be uniform [-1,1] Let Y be two valued with P(Y=-1)=P(Y=1)= 1/2. Let Z=XY. X and Z are uncorrelated, but they are certainly dependent.
– herb steinberg
Aug 26 at 21:46
add a comment |Â
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I'm looking at some notes on multivariable regression. It states that one of the assumptions for Least Squares to give good estimates is that,
"There is no exact linear relationship between any of the variables (no exact multicolinearity)".
Would this be the same as saying that the explanatory variables are all independent of one another.
On a related note, does the word "exact" in the statement above refer to perfect correlation between two variables. What if the correlation was strong but not perfect, would OLS still give good estimates?
statistics regression
asked Aug 26 at 21:36
NumberCruncher
767
I'm looking at some notes on multivariable regression. It states that one of the assumptions for Least Squares to give good estimates is that,
"There is no exact linear relationship between any of the variables (no exact multicolinearity)".
Would this be the same as saying that the explanatory variables are all independent of one another.
On a related note, does the word "exact" in the statement above refer to perfect correlation between two variables. What if the correlation was strong but not perfect, would OLS still give good estimates?
statistics regression
asked Aug 26 at 21:36
NumberCruncher
767
asked Aug 26 at 21:36
NumberCruncher
767
asked Aug 26 at 21:36
NumberCruncher
767
asked Aug 26 at 21:36
NumberCruncher
767
767
No: Let X be uniform [-1,1] Let Y be two valued with P(Y=-1)=P(Y=1)= 1/2. Let Z=XY. X and Z are uncorrelated, but they are certainly dependent.
– herb steinberg
Aug 26 at 21:46
add a comment |Â
No: Let X be uniform [-1,1] Let Y be two valued with P(Y=-1)=P(Y=1)= 1/2. Let Z=XY. X and Z are uncorrelated, but they are certainly dependent.
– herb steinberg
Aug 26 at 21:46
No: Let X be uniform [-1,1] Let Y be two valued with P(Y=-1)=P(Y=1)= 1/2. Let Z=XY. X and Z are uncorrelated, but they are certainly dependent.
– herb steinberg
Aug 26 at 21:46
No: Let X be uniform [-1,1] Let Y be two valued with P(Y=-1)=P(Y=1)= 1/2. Let Z=XY. X and Z are uncorrelated, but they are certainly dependent.
– herb steinberg
Aug 26 at 21:46
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
1
down vote
"No multicollinearity" means that no variable is a linear combination of the other variables, i.e. $x_1 = 2 x_2 - 4 x_3 + x_4$ is not allowed. This is equivalent to saying that the design matrix $X$ (where each row corresponds to a data point and each column corresponds to a variable) has linearly independent columns.
Note that there is no notion of randomness in this definition. You do not need a probabilistic model to talk about multicollinearity; it is a statement about the actual data that you have (the design matrix $X$).
answered Aug 26 at 21:51
angryavian
35.1k12976
Ok, thanks. I think I understand what you mean. So in the example you've cited would the correlation coefficient between $x_1$ and $2x_2-4x_3+x_4$ be equal to 1?
– NumberCruncher
Aug 26 at 21:59
One final question, if some variables are colinear does that mean it would not be possible to calculate all/some of the regression coefficients?
– NumberCruncher
Aug 26 at 22:19
@NumberCruncher Yes that correlation would be $1$. Collinearity would mean there is no unique least squares coefficient (actually there will be infinitely many settings of coefficients that minimize the least squares criterion). In terms of the $hatbeta = (X^top X)^-1 X^top y$ formula for computing the least squares coefficient, the fact that $X$ has linearly dependent columns makes $X^top X$ not invertible.
– angryavian
Aug 26 at 22:23
That helps a lot, thank you very much for the answers.
– NumberCruncher
Aug 26 at 22:29
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
"No multicollinearity" means that no variable is a linear combination of the other variables, i.e. $x_1 = 2 x_2 - 4 x_3 + x_4$ is not allowed. This is equivalent to saying that the design matrix $X$ (where each row corresponds to a data point and each column corresponds to a variable) has linearly independent columns.
Note that there is no notion of randomness in this definition. You do not need a probabilistic model to talk about multicollinearity; it is a statement about the actual data that you have (the design matrix $X$).
answered Aug 26 at 21:51
angryavian
35.1k12976
Ok, thanks. I think I understand what you mean. So in the example you've cited would the correlation coefficient between $x_1$ and $2x_2-4x_3+x_4$ be equal to 1?
– NumberCruncher
Aug 26 at 21:59
One final question, if some variables are colinear does that mean it would not be possible to calculate all/some of the regression coefficients?
– NumberCruncher
Aug 26 at 22:19
@NumberCruncher Yes that correlation would be $1$. Collinearity would mean there is no unique least squares coefficient (actually there will be infinitely many settings of coefficients that minimize the least squares criterion). In terms of the $hatbeta = (X^top X)^-1 X^top y$ formula for computing the least squares coefficient, the fact that $X$ has linearly dependent columns makes $X^top X$ not invertible.
– angryavian
Aug 26 at 22:23
That helps a lot, thank you very much for the answers.
– NumberCruncher
Aug 26 at 22:29
add a comment |Â
up vote
1
down vote
"No multicollinearity" means that no variable is a linear combination of the other variables, i.e. $x_1 = 2 x_2 - 4 x_3 + x_4$ is not allowed. This is equivalent to saying that the design matrix $X$ (where each row corresponds to a data point and each column corresponds to a variable) has linearly independent columns.
Note that there is no notion of randomness in this definition. You do not need a probabilistic model to talk about multicollinearity; it is a statement about the actual data that you have (the design matrix $X$).
answered Aug 26 at 21:51
angryavian
35.1k12976
Ok, thanks. I think I understand what you mean. So in the example you've cited would the correlation coefficient between $x_1$ and $2x_2-4x_3+x_4$ be equal to 1?
– NumberCruncher
Aug 26 at 21:59
One final question, if some variables are colinear does that mean it would not be possible to calculate all/some of the regression coefficients?
– NumberCruncher
Aug 26 at 22:19
@NumberCruncher Yes that correlation would be $1$. Collinearity would mean there is no unique least squares coefficient (actually there will be infinitely many settings of coefficients that minimize the least squares criterion). In terms of the $hatbeta = (X^top X)^-1 X^top y$ formula for computing the least squares coefficient, the fact that $X$ has linearly dependent columns makes $X^top X$ not invertible.
– angryavian
Aug 26 at 22:23
That helps a lot, thank you very much for the answers.
– NumberCruncher
Aug 26 at 22:29
add a comment |Â
up vote
1
down vote
up vote
1
down vote
"No multicollinearity" means that no variable is a linear combination of the other variables, i.e. $x_1 = 2 x_2 - 4 x_3 + x_4$ is not allowed. This is equivalent to saying that the design matrix $X$ (where each row corresponds to a data point and each column corresponds to a variable) has linearly independent columns.
Note that there is no notion of randomness in this definition. You do not need a probabilistic model to talk about multicollinearity; it is a statement about the actual data that you have (the design matrix $X$).
answered Aug 26 at 21:51
angryavian
35.1k12976
"No multicollinearity" means that no variable is a linear combination of the other variables, i.e. $x_1 = 2 x_2 - 4 x_3 + x_4$ is not allowed. This is equivalent to saying that the design matrix $X$ (where each row corresponds to a data point and each column corresponds to a variable) has linearly independent columns.
Note that there is no notion of randomness in this definition. You do not need a probabilistic model to talk about multicollinearity; it is a statement about the actual data that you have (the design matrix $X$).
answered Aug 26 at 21:51
angryavian
35.1k12976
answered Aug 26 at 21:51
angryavian
35.1k12976
answered Aug 26 at 21:51
angryavian
35.1k12976
answered Aug 26 at 21:51
angryavian
35.1k12976
35.1k12976
Ok, thanks. I think I understand what you mean. So in the example you've cited would the correlation coefficient between $x_1$ and $2x_2-4x_3+x_4$ be equal to 1?
– NumberCruncher
Aug 26 at 21:59
One final question, if some variables are colinear does that mean it would not be possible to calculate all/some of the regression coefficients?
– NumberCruncher
Aug 26 at 22:19
@NumberCruncher Yes that correlation would be $1$. Collinearity would mean there is no unique least squares coefficient (actually there will be infinitely many settings of coefficients that minimize the least squares criterion). In terms of the $hatbeta = (X^top X)^-1 X^top y$ formula for computing the least squares coefficient, the fact that $X$ has linearly dependent columns makes $X^top X$ not invertible.
– angryavian
Aug 26 at 22:23
That helps a lot, thank you very much for the answers.
– NumberCruncher
Aug 26 at 22:29
add a comment |Â
Ok, thanks. I think I understand what you mean. So in the example you've cited would the correlation coefficient between $x_1$ and $2x_2-4x_3+x_4$ be equal to 1?
– NumberCruncher
Aug 26 at 21:59
One final question, if some variables are colinear does that mean it would not be possible to calculate all/some of the regression coefficients?
– NumberCruncher
Aug 26 at 22:19
@NumberCruncher Yes that correlation would be $1$. Collinearity would mean there is no unique least squares coefficient (actually there will be infinitely many settings of coefficients that minimize the least squares criterion). In terms of the $hatbeta = (X^top X)^-1 X^top y$ formula for computing the least squares coefficient, the fact that $X$ has linearly dependent columns makes $X^top X$ not invertible.
– angryavian
Aug 26 at 22:23
That helps a lot, thank you very much for the answers.
– NumberCruncher
Aug 26 at 22:29
Ok, thanks. I think I understand what you mean. So in the example you've cited would the correlation coefficient between $x_1$ and $2x_2-4x_3+x_4$ be equal to 1?
– NumberCruncher
Aug 26 at 21:59
Ok, thanks. I think I understand what you mean. So in the example you've cited would the correlation coefficient between $x_1$ and $2x_2-4x_3+x_4$ be equal to 1?
– NumberCruncher
Aug 26 at 21:59
One final question, if some variables are colinear does that mean it would not be possible to calculate all/some of the regression coefficients?
– NumberCruncher
Aug 26 at 22:19
One final question, if some variables are colinear does that mean it would not be possible to calculate all/some of the regression coefficients?
– NumberCruncher
Aug 26 at 22:19
@NumberCruncher Yes that correlation would be $1$. Collinearity would mean there is no unique least squares coefficient (actually there will be infinitely many settings of coefficients that minimize the least squares criterion). In terms of the $hatbeta = (X^top X)^-1 X^top y$ formula for computing the least squares coefficient, the fact that $X$ has linearly dependent columns makes $X^top X$ not invertible.
– angryavian
Aug 26 at 22:23
@NumberCruncher Yes that correlation would be $1$. Collinearity would mean there is no unique least squares coefficient (actually there will be infinitely many settings of coefficients that minimize the least squares criterion). In terms of the $hatbeta = (X^top X)^-1 X^top y$ formula for computing the least squares coefficient, the fact that $X$ has linearly dependent columns makes $X^top X$ not invertible.
– angryavian
Aug 26 at 22:23
That helps a lot, thank you very much for the answers.
– NumberCruncher
Aug 26 at 22:29
That helps a lot, thank you very much for the answers.
– NumberCruncher
Aug 26 at 22:29
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2895551%2fmultivariable-regression-does-no-exact-multicolinearity-between-variables-mea%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
No: Let X be uniform [-1,1] Let Y be two valued with P(Y=-1)=P(Y=1)= 1/2. Let Z=XY. X and Z are uncorrelated, but they are certainly dependent.
– herb steinberg
Aug 26 at 21:46