Observational Surveys: Implementation and Analysis

1. Observational Surveys: Implementation and Analysis

2. Observational Surveys - Speakers • William W. Stenzel, D.Sc. • Associate Director, Center for Public Safety (Management Consulting) • Roy E. Lucke • Director, Research and Development for the Center for Public Safety

3. Notebook Materials • A hardcopy of all of the PowerPoint slides for this session (“Observational Surveys”) can be found in the seminar notebook.

4. Why Use Observational Surveys? • For the Illinois Traffic Stop Statistics Study, “disparities” are going to be calculated by comparing: • the racial composition of traffic stops • the racial composition of the driver population. • The racial composition of the driver population is going to estimated by using “adjusted census data” at the city and county level.

5. Why Use Observational Surveys? • What options does an agency have if there is concern that the adjusted census data will not provide accurate information about the racial composition of the driver population in its jurisdiction? • One option is to obtain a better estimate of the racial composition of the driver population with the use of observational surveys.

6. Center for Public Safety Experience with Observational Surveys • The Center for Public Safety has conducted three observational surveys for agencies in Illinois: • Highland Park (Sept-Oct 2001 – Stenzel) • Hinsdale (May 2004 – Lucke) • Schaumburg (August 2004 – Lucke)

7. Observational Survey Topics • Observational Surveys: • Topic 1: Data Collection • the nuts and bolts of how to conduct a survey • Topic 2: Data Summarization • putting the survey data into a format suitable for review and analysis • Topic 3: Data Analysis • comparing and assessing the survey and traffic stop data

8. Observational Surveys - Topic 1 Topic 1: Data Collection The nuts and bolts of how to conduct a survey

9. Conducting Observational Studies Once the decision is made to do an observation study, there are three major tasks: • Determining what data to collect • Identifying data collection sites • Recruiting and training observers

10. Conducting Observational Studies In addition to the three major tasks, other steps include: • Scheduling data collection • Equipping the data collectors • Developing forms, data entry and data analysis procedures

11. Preliminary Information • It is only possible to obtain good driver demographic information from stopped vehicles • Observations can only be made at intersections with traffic signals or stop signs • Efforts to observe drivers on controlled-access roadways were not successful

12. Site Selection • Primary Criteria • Conduct observations at or near intersections that are among the agency’s high traffic stop locations. • Agency should try to identify locations for all stops, not just where citations are issued • Also identify times of day and days of week for stops so observation times can be matched as well as possible

13. Site Selection, Continued • Site must provide a good view of stopped vehicles • Steep shoulders may raise observers too high • “Sweeping” right turn lanes might keep observers too far from lanes to see • No limit on number of lanes – observers need only check lanes they can clearly see • Site must be safe for observers • There must be a shoulder or sidewalk – curbs are desirable • Observers must be free from potential harassment

14. Survey Sessions • Session is a 2 or 3 hour observation period • Sessions should be distributed across all days of the week, according to traffic stop information • Sessions can be done • Mornings • Afternoons • Early evenings • Again, dependent on stop information and available daylight • Surveys should be done in both directions of travel

15. The Survey Team • Three individuals are needed for each session • Observer • Recorder • Counter

16. The Survey Team, Continued • Team members can be recruited from a number of possible sources. • Agency volunteers or auxiliaries • College students (e.g., criminal justice students) • Crossing guards • Temporary labor pools • Etc.

17. The Survey Team, Continued • Training must be provided to survey team members • Classroom instruction covering the nature of the project and what they will be expected to do • Practice sessions under guidance of project leaders • Team members must be scheduled in groups of threes at dates and times identified for surveys. • Have substitutes available • Project leaders should oversee all observation sessions

18. Sample Agenda for Observer Training Agenda • Review Agenda • Complete Forms • Driver Survey • Vehicle Counter • Field Work • Schedule/Signups

19. Survey Team Equipment • Safety vests • Traffic counting devices (or digital cameras) • Clipboards and pencils • Rain gear (ponchos, umbrellas, “writing pouches”)

20. Data Items to be Recorded • Each agency must decide what data items they believe are important to capture. Candidates items include: • Driver race/ethnicity • Driver gender • Driver age • Number of passengers in vehicle • Driver residency • Type of vehicle

21. Data Collection • Paper “check mark” form • Scantron form • Palm or other hand-held device • Tablet-type personal computer • Each session should be stored as a separate file, either in a physical packet or data file

22. Sample Data Collection Form

23. Number of Observations • The number of data collection sessions can be affected by: • Traffic volume • Roadway configuration (number of lanes) • Stop signs or traffic signals • Number of data items to be collected • General observations: • Higher capture rates (75% - 100% of drivers) at stop signs, but usually lower traffic volumes • Lower capture rates (20% - 75%) at signalized intersection depending on volume, number of lanes, and signal timing

24. Observation Limitations • Can be done only during daylight hours • Glare from windows (or window tinting) can affect observation • Weather (rain, snow, excessive heat) • Subjective decisions be observers • Cost of doing surveys (labor intensive activity)

25. Topic 2: Data Summarization • Data Summarization: • Recordkeeping • Data Entry • Data Base Software

26. Topic 2: Data Summarization • Recordkeeping - additional information that should/can be added to each observation (record): • Location (should) • Day of the week (should) • Time of day (should) • Direction of traffic (optional) • Data collectors (optional)

27. Topic 2: Data Summarization • Data Entry - getting the data into an electronic format • “The old fashion way” – keying the data in • Use machine-readable data collection forms (e.g., Scantron) • Download from a file created at the time of data collection (e.g., from a Palm Pilot or a PC tablet)

28. Topic 2: Data Summarization • Data Base Software - a computer program that can be used to: • Manipulate the data (i.e., sort and filter) • Display the data (i.e., print summary tables and charts) • Describe the data (i.e., compute various descriptive attributes): • number of observations • Average value • Minimum and maximum values • Examples: (Access, EXCEL)

29. Topic 2: Data Summarization • Example of a printout

30. Observational Surveys - Topic 3 Topic 3: Data Analysis Comparing and assessing the survey and traffic stop data

31. Topic 3: Data Analysis • Data analysis consists of comparing two sets of data: • Traffic stop data • Driver survey data • And addressing the question: Are differences between the two sets of data important?

32. Data Analysis: A Sample Comparison 5 Two data sets • Question: Are the differences in the percentages between the traffic stop data and the driver survey data in the each racial category important? • Are differences due only to natural variation, or • Are differences due to the some outside influence on the officer’s decision about whom to stop (e.g., race)?

33. Statistical Benchmarking • Statistical benchmarking consists of: • Comparing two sets of data: • Encounter data: the racial composition of drivers in traffic stops, and • Survey data: the racial composition of drivers who are potential participants in a traffic stop • A procedure for assessing the significance of differences in the percentages between the two data sets and

34. Statistical Benchmarking • Highland Park • Statistical benchmarks were used to assess the importance of the differences in the percentages in the driver survey and traffic stop data. • The benchmarks were determined using a statistical procedure called “confidence intervals.”

35. Confidence Interval Example • Example: Is the difference between the two percentages for Hispanics (i.e., 24.0% and 18.6%) important? • One way to address this is to determine a range of values (i.e., a confidence interval) for the expected number of traffic stops involving Hispanic drivers.

36. Confidence Interval Example • Example: The confidence interval for the number of Hispanics in the traffic stop data shown above is: [64, 96]. • This interval can be interpreted as follows: • If the decision about who to stop is not influenced by race, then the expected number of Hispanics stopped, due to normal variation, should fall between 64 and 96.

37. Confidence Intervals Example • The upper and lower limits for the confidence interval can be interpreted as statistical benchmarks for the number of Hispanics stopped. • The limits are determined based on: • Total number of traffic stops (408) • Estimated number of Hispanics in the driver population • Selected confidence level

38. Confidence Interval Example 1 • Example: The statistical benchmarks for the expected number of Hispanics, [64, 96], is based on a confidence level of 95%. • The 95% confidence level means the margin of error is 5%. • A 5% margin of error means that there is 5% chance that even with normal statistical variation the number of Hispanics stopped could fall below 64 or above 96.

39. Statistical Benchmark Example 2 • The upper and lower benchmarks for each racial category are shown at the bottom of the table. • These benchmarks are compared with the actual number of encounters in each racial category. • Except for Hispanics, the actual number of stops within each category falls within the benchmark limits.

40. Statistical Benchmark Example • The number of Hispanics stopped in this example, 98, is outside the statistical benchmarks of [64, 96]. • THIS DOES NOT PROVE RACIAL PROFILING. • It indicates that further investigation is needed to determine what special circumstances might be present that are influencing the number of Hispanics that are stopped.

41. Why Use 95%? • Use of 95% for the confidence interval is a conservative approach that assumes that racially motivated policing is not occurring unless there is significant evidence to the contrary. • Justification for a conservative approach is appropriate in view of the many uncertainties associated with the data: • Difficulty in identifying race • Different driver behaviors by race • Different driver behaviors by gender and age • Unknown mix of drivers by gender and age by race

42. How Can I Use Statistical Benchmarks? • The benchmarking procedure described is based on statistical procedure called the “two-sample test for proportions.” • Its use requires a basic understanding of applied statistics. (Note: “Statistical Benchmarks for Police Traffic Stops” in seminar notebook.) • To help departments that may want to use statistical benchmarking based on this procedure, the Center for Public Program has put an easy-to-use spreadsheet on its website that can be used to find statistical benchmarks.

43. Statistical Benchmarking Spreadsheet

44. Statistical Benchmark Spreadsheet • The statistical benchmarking spreadsheet can be found on the website for the Center for Public Safety: www.northwestern.edu/nucps Select “Links” Select “Racial Profiling” At bottom of page under “Recent Articles” find: “Benchmarking Spreadsheet”

45. Contact Information • William Stenzel • 847/491-8995 • wstenzel@northwestern.edu • Roy Lucke • 847/491-3469 • rlucke@northwestern.edu