1 / 33

VLSI Systems Design—Experiments Necessary steps: Explore the problem space Design experiment(s)

VLSI Systems Design—Experiments Necessary steps: Explore the problem space Design experiment(s) Carry out experiment(s) Analyze results software packages: R, Matlab, … Report results. Example: design a “better” transistor What do we mean by “better”?

crescent
Télécharger la présentation

VLSI Systems Design—Experiments Necessary steps: Explore the problem space Design experiment(s)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. VLSI Systems Design—Experiments • Necessary steps: • Explore the problem space • Design experiment(s) • Carry out experiment(s) • Analyze results • software packages: R, Matlab, … • Report results

  2. Example: design a “better” transistor What do we mean by “better”? What FACTORS influence design? --fabrication --design --environmental For which of these is there random variation?

  3. Which “factors” do we want to investigate?

  4. SUMMARY—15 IMPORTANT POINTS FOR EXPERIMENTERS: 1. Even careful experimentation and observation may miss important facts; new experiments may cause old conclusions to be discarded; EXPERIMENTS ARE NOT PROOFS. 2. It is just as important to report NEGATIVE results as to report POSITIVE results. The experimenter must always accurately record and thoroughly report ALL results. 3. IGNORING IMPORTANT FACTORS CAN LEAD TO ERRONEOUS CONCLUSIONS, SOMETIMES WITH TRAGIC RESULTS. 4. YOUR RESULTS ARE ONLY VALID FOR THE PART OF THE DATA-TREATMENT SPACE YOU HAVE EXPLORED; YOU CANNOT CLAIM KNOWLEDGE OF WHAT YOU HAVE NOT EXPLORED 5. An experiment is worthless unless it can be REPEATED by other researchers using the same experimental setup; experimenters have a duty to the research community to report enough about their experiment and data so that other researchers can verify their claims 6. YOU ONLY GET ANSWERS TO THE QUESTIONS YOU ASK 7. if your are going to use a (pseudo-)RANDOM NUMBER GENERATOR, make sure the output behaves enough like a sequence of TRUE RANDOM NUMBERS 8. An experiment must be repeated a SUFFICIENT NUMBER OF TIMES for the results to be attributed to more than random error 9. Choosing the CORRECT MEASURE for the question you are asking is an important part of the experimental design 10. Reporting CORRECT results, PROPERLY DISPLAYED, is an integral part of a well-done experiment 11. MISUSE OF GRAPH LABELING can lead to MISLEADING RESULTS AND INCORRECT CONCLUSIONS 12. INTERPOLATING your results to regions you have not explored can lead to INCORRECT CONCLUSIONS 13. IGNORING the “NULL HYPOTHESIS” when reporting your results can be very misleading 14. Don’t mistake CORRELATION for DEPENDENCE 15. Justify your choice of CURVE using VALID STATISTICS, not “appearance”

  5. Topics • Analyzing and Displaying Data • Simple Statistical Analysis • Comparing Results • Curve Fitting • Designing Experiments: Factorial Designs • 2K Designs Including Replications • Full Factorial Designs • Ensuring Data Meets Analysis Criteria • Presenting Your Results; Drawing Conclusions

  6. Example: A System Factors (Experimental Conditions) System (“Black Box”) System Inputs System Outputs Responses (Experimental Results)

  7. Experimental Research • Define system outputs first • Then define system inputs • Finally, define behavior (i.e., transfer function) Define System Identify Factors and Levels • Identify system parameters that vary (many) • Reduce parameters to important factors (few) • Identify values (i.e., levels) for each factor Identify Response(s) • Identify time, space, etc. effects of interest Design Experiments • Identify factor-level experiments

  8. Create and Execute System; Analyze Data • Workload can be a factor (but often isn't) • Workloads are inputs that are applied to system Define Workload • Create system so it can be executed • Real prototype • Simulation model • Empirical equations Create System Execute System • Execute system for each factor-level binding • Collect and archive response data • Analyze data according to experiment design • Evaluate raw and analyzed data for errors • Display raw and analyzed data to draw conclusions Analyze & Display Data

  9. Epitaxial growth New method using non-linear temp profile What is the system? Responses Total time Quality of layer Total energy required Maximum layer thickness Factors Temperature profile Oxygen density Initial temperature Ambient temperature Some Examples Analog Simulation • Which of three solvers is best? • What is the system? • Responses • Fastest simulation time • Most accurate result • Most robust to types of circuits being simulated • Factors • Solver • Type of circuit model • Matrix data structure

  10. Basic Descriptive Statistics for a Random Sample X • Mean • Median • Mode • Variance / standard deviation • Z scores: Z = (X – mean)/ (standard deviation) • Quartiles, box plots • Q-Q plot Note: these can be deceptive. For example, if P (X = 0) = P(X = 100) = 0.5 and P (Y = 50 ) = 1, Then X and Y have the same mean (and nastier examples can be constructed) home.oise.utoronto.ca/~thollenstein/Exploratory%20Data%20Analysis.ppt

  11. SIMPLE MODELS OF DATA Example: Evaluation of a new wireless network protocol System: wireless network with new protocol • Workload: • 10 messages applied at single source • Each message identical configuration • Experiment output: • Roundtrip latency per message (ms) Data file “latency.dat” Ms. # Latency 1 22 2 23 3 19 4 18 5 15 6 20 7 26 8 17 9 19 10 17 Mean: 19.6 ms Variance: 10.71 ms2 Std Dev: 3.27 ms Hypothesis: Distribution is N(m,s2)

  12. Verify Model Preconditions Check normal distribution Use quantile-quantile (Q-Q) plot Pattern adheres consistently along ideal quantile-quantile line http://itl.nist.gov/div898/software/dataplot/refman1/ch2/quantile.pdf Check randomness Use plot of residuals around mean Residuals “appear” random

  13. Confidence Intervals Sample mean vs Population mean If many samples are collected, about 1 - a will contain the “true mean” CI: > 30 samples CI: < 30 samples For the latency data, m = 10, a = 0.05: (17.26, 21.94) Raj Jain, “The Art of Computer Systems Performance Analysis,” Wiley, 1991.

  14. Depth Resistance 1 1.689015 2 4.486722 3 7.915209 4 6.362388 5 11.830739 6 12.329104 7 14.011396 8 17.600094 9 19.022146 10 21.513802 Scatter and Line Plots Resistance profile of doped silicon epitaxial layer • Expect linear resistance increase as depth increases

  15. Linear Regression Statistics(hypothesis: resistance = b0 + b1*depth + error) model = lm(Resistance ~ Depth) summary(model) Residuals: Min 1Q Median 3Q Max -2.11330 -0.40679 0.05759 0.51211 1.57310 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -0.05863 0.76366 -0.077 0.94 Depth 2.13358 0.12308 17.336 1.25e-07 *** --- Signif. codes: 0 `***' 0.001 `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1 Residual standard error: 1.118 on 8 degrees of freedom “variance of error: (1.118)2” Multiple R-Squared: 0.9741, Adjusted R-squared: 0.9708 F-statistic: 300.5 on 1 and 8 DF, p-value: 1.249e-07 “evidence this estimate valid” “reject hypotheses b0 = 0, b1 = 0” (Using R system; based on http://www.stat.umn.edu/geyer/5102/examp/reg.html

  16. Validating Residuals Errors are marginally normally distributed due to “tails”

  17. Comparing Two Sets of Data Example: Consider two different wireless access points. Which one is faster? • Inputs: same set of 10 messages communicated through both access points. • Response (usecs): Latency1 Latency2 22 19 23 20 19 24 18 20 15 14 20 18 26 21 17 17 19 17 17 18 Approach: • Take difference of data and determine CI of difference. • If CI straddles zero, cannot tell which access point is faster. CI95% = (-1.27, 2.87) usecs Confidence interval straddles zero. Thus, cannot determine which is faster with 95% confidence

  18. Plots with error bars Execution time of SuperLU linear system solution on parallel computer • Ax = b • For each p, ran problem multiple times with same matrix size but different values • Determined mean and CI for each p to obtain curve and error intervals Matrix density p

  19. Curve Fitting > model = lm(t ~ poly(p,4)) > summary(model) Call: lm(formula = t ~ poly(p, 4)) Residuals: 1 2 3 4 5 6 7 8 9 -0.4072 0.7790 0.5840 -1.3090 -0.9755 0.8501 2.6749 -3.1528 0.9564 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 236.9444 0.7908 299.636 7.44e-10 *** poly(p, 4)1 679.5924 2.3723 286.467 8.91e-10 *** poly(p, 4)2 268.3677 2.3723 113.124 3.66e-08 *** poly(p, 4)3 42.8772 2.3723 18.074 5.51e-05 *** poly(p, 4)4 2.4249 2.3723 1.022 0.364 --- Signif. codes: 0 `***' 0.001 `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1 Residual standard error: 2.372 on 4 degrees of freedom Multiple R-Squared: 1, Adjusted R-squared: 0.9999 F-statistic: 2.38e+04 on 4 and 4 DF, p-value: 5.297e-09

  20. Model Validation: y’ = ax + bR2 – Coefficient of Determination“How well does the data fit your model?” What proportion of the “variability” is accounted for by the statistical model? (what is ratio of explained variation to total variation?) Suppose we have measurements y1, y2, …, ynwith mean m And predicted values y1’, y2’, …, yn’ (yi’ = axi + b = yi + ei) SSE = sum of squared errors = ∑(yi – yi’)2= ∑ei2 SST = total sum of squares =∑ (yi – m)2 SSR = SST – SSE = residual sum of squares = ∑(m – yi’)2 R2 = SSR/SST = (SST-SSE)/SST R2 is a measure of how good the model is. The closer R2 is to 1 the better. Example: Let SST = 1499 and SSE = 97. Then R2 = 93.5% http://www-stat.stanford.edu/~jtaylo/courses/stats191/notes/simple_diagnostics.pdf

  21. Using the t-test Consider the following data (“sleep.R”) extra group 1 0.7 1 2 -1.6 1 3 -0.2 1 4 -1.2 1 5 -0.1 1 6 3.4 1 7 3.7 1 8 0.8 1 9 0.0 1 10 2.0 1 11 1.9 2 12 0.8 2 13 1.1 2 14 0.1 2 15 -0.1 2 16 4.4 2 17 5.5 2 18 1.6 2 19 4.6 2 20 3.4 2 From “Introduction to R”, http://www.R-project.org

  22. T.test result > t.test(extra ~ group, data = sleep) Welch Two Sample t-test data: extra by group t = -1.8608, df = 17.776, p-value = 0.0794 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -3.3654832 0.2054832 sample estimates: mean of x mean of y 0.75 2.33 p-value is smallest 1- confidence where null hypothesis. not true. p-value = 0.0794 means difference not 0 above 92%

  23. Factorial Design What “factors” need to be taken into account? How do we design an efficient experiment to test all these factors? How much do the factors and the interactions among the factors contribute to the variation in results? Example: 3 factors a,b,c, each with 2 values: 8 combinations But what if we want random order of experiments? What if each of a,b,c has 3 values? Do we need to run all experiments? http://www.itl.nist.gov/div898/handbook/pri/section3/pri3332.htm

  24. Standard Procedure-Full Factorial Design (Example) Variables A,B,C: each with 3 values, Low, Medium, High (coded as -1,0,1) “Signs Table”: • Run the experiments in the table (“2 level, full factorial design”) • Repeat the experiments in this order n times by using rows 1,…,8,1,…,8, … (“replication”) • Use step 2, but choose the rows randomly (“randomization”) • Use step 4, but add some “center point runs”, for example, run the case 0,0,0, then use 8 rows, then run 0,0,0, …finish with a 0,0,0 case • In general, for 5 or more factors, use a “fractional factorial design”

  25. 2k Factorial Design Example: k = 2, factors are A,B, and X’s are computed from the signs table: y = q0 + qAxA + qBxB + qABxAB SST = total variation around the mean = ∑ (yi – mean)2 = SSA+SSB+SSAB where SSA = 22qA2 (variation allocated to A), and SSB, SSAB are defined similarly Note: var(y) = SST/( 2k – 1) Fraction of variation explained by A = SSA/SST

  26. Example: 2k Design Factor Levels Line Length (L) 32, 512 words No. Sections (K) 4, 16 sections Control Method (C) multiplexed, linear www.stat.nuk.edu.tw/Ray-Bing /ex-design/ex-design/ExChapter6.ppt Cache Experiment Design L K C Misses 32 4 mux 512 4 mux 32 16 mux 512 16 mux 32 4 lin 512 4 lin 32 16 lin 512 16 lin Misses Address Trace Are all factors needed? If a factor has little effect on the variability of the output, why study it further? Method? a. Evaluate variation for each factor using only two levels each b. Must consider interactions as well Encoded Experiment Design L K C Misses -1 -1 -1 1 -1 -1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 1 -1 1 1 1 1 1 Interaction: effect of a factor dependent on the levels of another

  27. Example: 2k Design (continued) http://www.cs.wustl.edu/~jain/cse567-06/ftp/k_172kd/sld001.htm Analyze Results (Sign Table) Obtain Reponses I L K C LK LC KC LKC Miss.Rate (yj) 1 -1 -1 -1 1 1 1 -1 14 1 1 -1 -1 -1 -1 1 1 22 1 -1 1 -1 -1 1 -1 1 10 1 1 1 -1 1 -1 -1 -1 34 1 -1 -1 1 1 -1 -1 1 46 1 1 -1 1 -1 1 -1 -1 58 1 -1 1 1 -1 -1 1 -1 50 1 1 1 1 1 1 1 1 86 L K C Misses -1 -1 -1 14 1 -1 -1 22 -1 1 -1 10 1 1 -1 34 -1 -1 1 46 1 -1 1 58 -1 1 1 50 1 1 1 86 Ex: y1 = 14 = q0 – qL –qK –qC + qLK + qLC + qKC – qLKC Solve for q’s qi: 40 10 5 20 5 2 3 1 = 1/∑(signi*Responsei) SSL = 23q2L = 800 SST = SSL+SSK+SSC+SSLK+SSLC+SSKC+SSLKC = 800+200+3200+200+32+72+8 = 4512 %variation(L) = SSL/SST = 800/4512 = 17.7%

  28. Full Factorial Design Model: yij = m+ai + bj + eij Effects computed such that ∑ai = 0 and ∑bj = 0 m = mean(y..) aj = mean(y.j) – m bi = mean(yi.) – m Experimental Errors SSE = ei2j SS0 = abm2 SSA= b∑a2 SSB= a∑b2 SST = SS0+SSA+SSB+SSE

  29. Full-Factorial Design Example Determination of the speed of light Morley Experiments Factors: Experiment No. (Expt) Run No. (Run) Levels: Expt – 5 experiments Run – 20 repeated runs Expt Run Speed 001 1 1 850 002 1 2 740 003 1 3 900 004 1 4 1070 <more data> 019 1 19 960 020 1 20 960 021 2 1 960 022 2 2 940 023 2 3 960 <more data> 096 5 16 940 097 5 17 950 098 5 18 800 099 5 19 810 100 5 20 870

  30. Box Plots of Factors

  31. Two-Factor Full Factorial > fm <- aov(Speed~Run+Expt, data=mm) # Determine ANOVA > summary(fm) # Display ANOVA of factors Df Sum Sq Mean Sq F value Pr(>F) Run 19 113344 5965 1.1053 0.363209 Expt 4 94514 23629 4.3781 0.003071 ** Residuals 76 410166 5397 --- Signif. codes: 0 `***' 0.001 `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1 Conclusion: Data across experiments has acceptably small variation, but variation within runs is significant

  32. Visualizing Results: Tufte’s Principles • Have a properly chosen format and design • Use words, numbers, and drawing together • Reflect a balance, a proportion, a sense of relevant scale • Display an accessible complexity of detail • Have a story to tell about the data • Draw in a professional manner • Avoid content-free decoration, including “chart junk”

  33. Back to the transistor: • What factors are there? • Which ones do we want to investigate? • How should we define our experiments? • What role will randomness play? (simulation/actual) • How should we report the results?

More Related