1 / 6

VVSG 1.1 Reliability

VVSG 1.1 Reliability. David Flater, Ph.D. Computer Scientist, Software and Systems Division, ITL http://vote.nist.gov. Previous Public Review Draft.

kylar
Télécharger la présentation

VVSG 1.1 Reliability

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. VVSG 1.1Reliability David Flater, Ph.D. Computer Scientist, Software and Systems Division, ITL http://vote.nist.gov

  2. Previous Public Review Draft The reliability benchmarks were made more stringent and traceable to a use case provided by former TGDC member Paul Miller working with other election officials The test method was changed from a standalone Probability Ratio Sequential Test to classical hypothesis testing using all available data: A demonstration of non-conformity can easily occur Conclusive results are never guaranteed and are impossible without at least X volume of testing. The plan was to give a pass to any system that did not demonstrate non-conformity

  3. Response from EAC To pass without demonstrating conformity is unacceptable Testing long enough to demonstrate conformity is not doable and would be of limited validity anyway. (Reliability can't be tested in; it must be built in) Move to best practices for quality assurance, reliability engineering and analysis Volume and stress testing is a validation of that work, not a demonstration of reliability in and of itself Specific methods of reliability analysis should not be prescribed

  4. Impact on VVSG 1.1 The reliability benchmarks will be expressed in terms of the probabilities of critical and non-critical failures Manufacturers will be required to deliver credible reliability analyses for their systems (e.g., FMEA). The specific methods to be used will not be prescribed Hypothesis testing will still be used for accuracy and misfeed rate, but demonstration of conformity will be required Incidentally, the maintainability and availability sections will go away

  5. Limitations In a reliability analysis, the probability of a software (logic) failure "cannot be determined;"* at best it can be extrapolated from the observed rate of failure or fault correction using a statistical model The previous reliability tests were strictly hardware-oriented, so this is actually an improvement Conformity assessment will require the "expert judgment" of a reliability engineer * Clifton A. Ericson II, Hazard Analysis Techniques for System Safety, 2005, Table 13.1 (Hardware/Software FMEA Characteristics)

  6. Discussion/Questions Page 6

More Related