Department of Building and Real Estate
First Year Statistics

LINEAR REGRESSION

Queries can be sent to me

1. Introduction
   • very often when 2 (or more) variables are observed, relationship between them can be visualized
   • predictions are always required in economics or physical science from existing and historical data
   • regression analysis is used to help formulate these predictions and relationships
   • linear regression is a special kind of regression analysis in which 2 variables are studied and a straight-line relationship is assumed
   • linear regression is important because
     1. there exist many relationships that are of this form
     2. it provides close approximations to complicated relationships which would otherwise be difficult to describe
   • the 2 variables are divided into (i) independent variable and (ii) dependent variable
   • Dependent Variable is the variable that we want to forecast
   • Independent Variable is the variable that we use to make the forecast
   • e.g. Time vs. GNP (time is independent, GNP is dependent)
   • scatter diagrams are used to graphically presenting the relationship between the 2 variables
   • usually the independent variable is drawn on the horizontal axis (X) and the dependent variable on vertical axis (Y)
   • the regression line is also called the regression line of Y on X

2. Assumptions
   • there is a linear relationship as determined (observed) from the scatter diagram
   • the dependent values (Y) are independent of each other, i.e. if we obtain a large value of Y on the first observation, the result of the second and subsequent observations will not necessarily provide a large value. In simple term, there should not be auto-correlation
   • for each value of X the corresponding Y values are normally distributed
   • the standard deviations of the Y values for each value of X are the same, i.e. homoscedasticity

3. Process
   • observe and note what is happening in a systematic way
   • form some kind of theory about the observed facts
   • draw a scatter diagram to visualize relationship
   • generate the relationship by mathematical formula
   • make use of the mathematical formula to predict

4. Method of Least Squares
   • from a scatter diagram, there is virtually no limit as to the number of lines that can be drawn to make a linear relationship between the 2 variables
   • the objective is to create a BEST FIT line to the data concerned
   • the criterion is the called the method of least squares
   • i.e. the sum of squares of the vertical deviations from the points to the line be a minimum (based on the fact that the dependent variable is drawn on the vertical axis)
   • the linear relationship between the dependent variable (Y) and the independent variable can be written as Y = a + bX , where a and b are parameters describing the vertical intercept and the slope of the regression line respectively

5. Calculating a and b
   

   

   • where X and Y are the raw values of the 2 variables
   • and are means of the 2 variables

6. Correlation
   • when the value of one variable is related to the value of another, they are said to be correlated
   • there are 3 types of correlation: (i) perfectly correlated; (ii) partially correlated; (iii) uncorrelated
   • Coefficient of Correlation (r) measures such a relationship

     

   • the value of r ranges from -1 (perfectly correlated in the negative direction) to +1 (perfectly correlated in the positive direction)
   • when r = 0, the 2 variables are not correlated

7. Coefficient of Determination
   • this calculates the proportion of the variation in the actual values which can be predicted by changes in the values of the independent variable
   • denoted by , the square of the coefficient of correlation

     

   • ranges from 0 to 1 (r ranges from -1 to +1)
   • expressed as a percentage, it represents the proportion that can be predicted by the regression line
   • the value 1 - is therefore the proportion contributed by other factors

8. Standard Error of Estimate (SEE)
   • a measure of the variability of the regression line, i.e. the dispersion around the regression line
   • it tells how much variation there is in the dependent variable between the raw value and the expected value in the regression

     

   • this SEE allows us to generate the confidence interval on the regression line as we did in the estimation of means

9. Confidence interval for the regression line (estimating the expected value)
   • estimating the mean value of for a given value of X is a very important practical problem
   • e.g. if a corporation's profit Y is linearly related to its advertising expenditures X, the corporation may want to estimate the mean profit for a given expenditure X
   • this is given by the formula

     

   • at n-2 degrees of freedom for the t-distribution

10. Confidence interval for individual prediction
    • for technical reason, the above formula must be amended and is given by

      

An Example

There is a linear relation between the results of Accounting and Statistics as shown from the scatter diagram in Figure 1. A linear regression analysis was done using the least-square method. The resultant regression line is represented by in which X represents the results of Accounting and Y that of Statistics. Figure 2 shows the regression line. In this example, the choice of dependent and indeendent variables is arbitrary. It can be said that the results of Statistics are correlated to that of Accounting or vice versa.

The Coefficient of Determination is 0.9194. This shows that the two variables are correlated. Nearly 92% of the variation in Y is explained by the regression line.

The Coefficient of Correlation (r) has a value of 0.8453. This indicates that the two variables are positively correlated (Y increases as X increases).

Queries can be sent to me

SM 18Oct95