Diamonds Regression Lab 202 - Revision history

Carver: /* Step 4: Data Cleaning: Outlier Elimination */

2020-01-31T22:10:08Z

‎Step 4: Data Cleaning: Outlier Elimination

Carver: /* Step 9: Publication */

2020-01-31T14:09:37Z

‎Step 9: Publication

Carver: /* Step 9: Publication */

2020-01-31T14:08:42Z

‎Step 9: Publication

Carver: /* Step 7: Your tool in production */

2020-01-29T20:51:33Z

‎Step 7: Your tool in production

Carver: /* Step 9: Publication */

2020-01-29T20:40:48Z

‎Step 9: Publication

Carver: /* Step 9: Publication */

2020-01-29T20:39:06Z

‎Step 9: Publication

Carver: /* Step 7: Your tool in production */

2020-01-29T20:37:03Z

‎Step 7: Your tool in production

Carver: /* Step 9: Publication */

2020-01-29T20:36:05Z

‎Step 9: Publication

Carver: /* Do assumptions of regression affect the story told by the intercept? */

2020-01-29T20:34:53Z

‎Do assumptions of regression affect the story told by the intercept?

Carver: /* Step 9: Publication */

2020-01-29T20:33:56Z

‎Step 9: Publication

@@ Line 56: / Line 56: @@
 How can you tell if the dimensions are wrong?  In StatCrunch, click on the most extreme outlier.  It will turn pink on the scatter plot and other graphs created.  Now plot three scatter plots showing (x, carat), (y, carat), (z, carat) and locate the pink outlier on the graphs.  Could you change a single measurement of the four (i.e. x, y, z and carat) without affecting at least one of the others?  What does this tell you about the validity measurements?  It would be too tedious to check all of the other outliers (but if you were working for a client, that's what you would have to do) so we are going to assume that all other outliers tell the same story.  Looks like the jeweler could have used your formula!
-Now we want to eliminate all outliers at once.  Redo the regression of the chosen model and save residuals (that's an option in the Regression dialog box).  Now redo the regression again and specify "Where abs(Residuals) < 'Threshold'".  Here 'Threshold' should be some number that you get to choose, but choose appropriately, to eliminate outliers with large (positive or negative) residuals that stand apart from the main part of the graph.  Look at the residual plot to see where to draw the line.  Compare the R^2 value with and without outliers.  Compare your slopes and intercepts with and without outliers.
+Now we want to eliminate all outliers at once.  Redo the regression of the chosen model and save residuals (that's an option in the Regression dialog box).  Now redo the regression again and specify "Where abs(Residuals) < 'Threshold'".  Here 'Threshold' should be some number that you get to choose, but choose appropriately, to eliminate outliers with large (positive or negative) residuals that stand apart from the main part of the graph.  To help you decide where to put the threshold, look at three plots:  First, look at the residual versus x*y*z plot (here x*y*z is what the StatCrunch calls "x").  Now, each of these residuals has a value which you can save in a column, putting each residual in the row corresponding to the data point of said residual.  In the Regression dialog, look under "Save" and choose "Residuals" then press "Compute".  Once you have the column of residuals, you can look at the second plot: the histogram of the residuals, and the third plot: the box plot of the residuals.  The is no one correct Threshold, it is correct as long as you can justify it.
 === Step 5: Model interpretation: Intercept ===

@@ Line 112: / Line 112: @@
 * Show the scatter plot of carat versus x+y+z for the whole data set.
 * Give the justification for your selection of model in step 3.
-* Show the scatter plot of carat versus x*y*z with the outliers removed (with your choice of threshold).
+* Show the scatter plot of carat versus x*y*z with the outliers removed.  Justify your choice of threshold for removing outliers.
 * Values for b0, b1 and R^2 for model Carat = b0 + b1 * x*y*z '''with outliers removed'''.
 * Show Residual versus x*y*z plot with outliers removed.

@@ Line 115: / Line 115: @@
 * Values for b0, b1 and R^2 for model Carat = b0 + b1 * x*y*z '''with outliers removed'''.
 * Show Residual versus x*y*z plot with outliers removed.
-* Report final model dropping b0.
+* Report final model dropping b0.  Report your value of "a" in step 6.
 * Report chosen threshold for flagging diamond as improperly measured (all without b0).
 * Give the inequalities for diamonds that are flagged and not flagged as improperly measured.
 * Show an image from StatCrunch of count in data set of flagged diamonds.

@@ Line 95: / Line 95: @@
   Carat = b1*x*y*z
-We are going to measure x, y, z, and carat for the diamond, then compute a residual.  If the residual is too high or too low we are going to flag the diamond.  You choose what residuals are too high and too low based on your understanding of the data.  It may be reasonable to choose the same threshold as chosen in step 3. Based on this choice, write a formula indicating which diamonds are flagged involving x, y, z, b1, and carat, and inequalities.
+We are going to measure x, y, z, and carat for the diamond, then compute a residual.  If the residual is too high or too low we are going to flag the diamond.  You choose what residuals are too high and too low based on your understanding of the data.  It may be reasonable to choose the same threshold as chosen in step 4. Based on this choice, write a formula indicating which diamonds are flagged involving x, y, z, b1, and carat, and inequalities.
 === Step 8: Testing your tool ===

@@ Line 115: / Line 115: @@
 * Values for b0, b1 and R^2 for model Carat = b0 + b1 * x*y*z '''with outliers removed'''.
 * Show Residual versus x*y*z plot with outliers removed.
-* Final model dropping b0.
+* Report final model dropping b0.
-* Report chosen threshold for flagging diamond as improperly measured.
+* Report chosen threshold for flagging diamond as improperly measured (all without b0).
 * Give the inequalities for diamonds that are flagged and not flagged as improperly measured.
 * Show an image from StatCrunch of count in data set of flagged diamonds.

@@ Line 109: / Line 109: @@
 Please include in your report:
-* A scatter plot of carat versus x*y*z for the whole data set.
+* Show the scatter plot of carat versus x*y*z for the whole data set.
-* A scatter plot of carat versus x+y+z for the whole data set.
+* Show the scatter plot of carat versus x+y+z for the whole data set.
-* A justification for your selection of model in step 3.
+* Give the justification for your selection of model in step 3.
-* A scatter plot of carat versus x*y*z with the outliers removed (with your choice of threshold).
+* Show the scatter plot of carat versus x*y*z with the outliers removed (with your choice of threshold).
 * Values for b0, b1 and R^2 for model Carat = b0 + b1 * x*y*z '''with outliers removed'''.
-* Residual versus x*y*z plot with outliers removed.
+* Show Residual versus x*y*z plot with outliers removed.
 * Final model dropping b0.
 * Chosen threshold for flagging diamond as improperly measured.
 * Image from StatCrunch of count in data set of flagged diamonds.

@@ Line 70: / Line 70: @@
 ====Do assumptions of regression affect the story told by the intercept? ====
-Consider the residual plot after outliers are removed.  What does that plot tell you about the model?  Does this plot satisfy the "Does the plot thicken?" condition?  Could this failed condition tell us that we cannot interpret any conclusion drawn from regression (as explained above) to say that the zero-volume diamonds have non-zero weight?
+Consider the residual plot (Residual versus x*y*z) after outliers are removed.  What does that plot tell you about the model?  Does this plot satisfy the "Does the plot thicken?" condition?  Could this failed condition tell us that we cannot interpret any conclusion drawn from regression (as explained above) to say that the zero-volume diamonds have non-zero weight?
 ==== How can we fix our model so that the intercept is as we expect? ====