1 / 28

The Practice of Standards Formalization

The Practice of Standards Formalization. Victor Kuliamin Institute for System Programming (ISP RAS) Moscow, Russia. Outline. ISP RAS Background OS Testing Compiler Testing Protocol Testing Hardware Testing ISP RAS Technologies (brief summary) Linux Standard Base Support Activity

samuru
Télécharger la présentation

The Practice of Standards Formalization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Practice of Standards Formalization Victor Kuliamin Institute for System Programming (ISP RAS) Moscow, Russia

  2. Outline • ISP RAS Background • OS Testing • Compiler Testing • Protocol Testing • Hardware Testing • ISP RAS Technologies (brief summary) • Linux Standard Base Support Activity • LSB Infrastructure • LSB conformance testing • Detailed: Math conformance testing Practice of Standards Formalization

  3. ISP RAS Background • Operating System testing • Regression test suite for Nortel Switch OS 1994-2000 • POSIX conformance test suite for real-time OS 2005-… • Linux Standard Base conformance test suites 2005-… • ARINC 653 2008-… • Compiler testing • Part of C expressions dynamic semantics 2000-2003 • Static semantics specification and test suites 2002-2004 • Optimizing units testing in gcc and Intel compilers 2001-2003 • Protocol testing • Testing Microsoft Research IPv6 2000-2001 • Test suite for Microsoft Mobile IPv6 2002-2003 • Test suite for IPsec 2004-… • Hardware testing • MIPS-based processors with DSP extensions 2006-… Practice of Standards Formalization

  4. ISP RAS Technologies • Model based testing • KVEST (1996) • RSL specifications • Protel target language • UniTESK (2001) • Specifications in extensions of target languages(C – 2001, Java – 2001, C# – 2003) • Concurrency testing extension (2001) • Compiler front-end testing support (2004) • Hardware testing support (2006) • Combinational test generation (2007) • Math extension (2007) • Static analysis Practice of Standards Formalization

  5. Model Based Testing General Scheme Test action generator 12% 36% 87% 57% Coverage criteria Coverage metric System under test Behavior model State model State model Oracle Practice of Standards Formalization

  6. Linux Standardization • What to do with 550 distributions of Linux? • Linux Standard Base • Binary interface standard • Supported by Free Standards Group • Includes • StandardsPOSIX, X/Open Curses, Open GL, Large File Support, … • LibrarieslibXML, gtk, Qt, JDK, Perl, … • 45000 functions in C • 2000 described accurately (POSIX) • 7000-9000 have good description of main functionality • Others have very poor or just no description Practice of Standards Formalization

  7. ISP RAS Activity for LSB Support • LSB Infrastructure development • DB of distributions, libraries, profiles, operations, etc. • Conformance checking and certification of distributions • Static analysis tools • Test suites • Conformance checking of applications • Monitoring tools • Test suites • LSB evolution support • Analysis DB and information system • Linux driver verification Practice of Standards Formalization

  8. Test Development Levels 2000 accurately described – the only target for formalization – UniTESK conformance testing 45000 7000-9000 partially well-defined – manual test development specialized massive automated test construction technology Practice of Standards Formalization

  9. UniTESK API Test Development Basics • API partitioned into logical modules (classes) • Each module state is modeled • Module operations described with stateful contracts • Preconditions (on state and operation parameters) • Postconditions (on pre-state, post-state, operation parameters and results) • Invariants (on state, hold when no op is working) • Structure of postcondition gives test coverage criteria • Coverage-targeted FSM abstraction for module • Testing – automatic on-the-fly exploration of FSM Practice of Standards Formalization

  10. Math Library Standards • IEEE 754 (Floating-point arithmetics)FP numbers, basic operations • ISO 9899 (C language and libraries)56 real + 16 complex functions • IEEE 1003.1 (POSIX)63 real + 22 complex functions • ISO 10697.1-3 (Language independent arithmetics)Elementary real and complex functions Practice of Standards Formalization

  11. IEEE 754 Floating-Point Numbers • Normal : E> 0 & E < 2k –1 X = (–1)S·2(E–B)·(1+M/2(n–k–1)) • Denormal : E = 0 X = (–1)S·2(–B+1)·(M/2(n–k–1)) • Exceptional : E = 2k –1 • M = 0 : +, – • M ≠ 0 : NaN 0 1 k k+1 n-1 n, k 0 0 1 1 1 1 1 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 M S E sign exponent mantissa B = 2(k–1) –1 2(–1)·1.1012 = 13/16 = 0,8125 0, -0 1/0 = +, (–1)/0 = – n = 32, k = 8 – float (single precision) n = 64, k = 11 – double n = 79, k = 15 – extended double n = 128, k = 15 – quadruple 0/0 = NaN 1/2(n-k-1) – 1 ulp Practice of Standards Formalization

  12. IEEE 754 Computations • Operations: +, –, *, /, sqrt , fma (2008), type conversions, remainder • Correct rounding – 4 rounding modes • to + • to – • to 0 • to the nearest • NaN and infinite results • Exception flags • INVALID : Incorrect arguments (NaN result) • DIVISION-BY-ZERO : Infinite result (precise ±∞) • OVERFLOW : Too big result (approximate ±∞) • UNDERFLOW : Too small (or denormal) result • INEXACT : Inexact result 0 Practice of Standards Formalization

  13. ISO C and POSIX Requirements • ISO/IEC 9899 (C language) : 54 real functions • Exact values : sin(0) = 0, log(1) = 0, … • DIVISION-BY-ZERO flag : log(0), atanh(1), pow(0,x), Г(-n) • NaN results and INVALID flag outside of domains • IEEE 1003.1 (POSIX) : 63 real + 22 complex • All IEEE 754 flags (except for INEXACT) for real functions • errno setting: Domain error, Range error • If x is denormalf(x) = x for f(x)~x in 0 (sin, asin, sinh, expm1…) • In overflow HUGE_VAL should be returned(value of HUGE_VAL unspecified) • Sometimes non-NaN results on NaN argumentsfmax(NaN, x) = x, pow(NaN, 0) = 1 glibc : +∞ MSVCRT : max double (1.797693134862316e+308) Solaris libc : max float (3.402823466385289e+38) Inconsistency with rounding modes Source of non-interoperability Inconsistency with IEEE 754 Practice of Standards Formalization

  14. Example of POSIX Requirements NAMEsin, sinf, sinl - sine function SYNOPSIS#include <math.h> double sin(double x); float sinf(float x); long double sinl(long double x); DESCRIPTION These functions shall compute the sine of their argument x, measured in radians. An application wishing to check for error situations should set errno to zero and call feclearexcept(FE_ALL_EXCEPT) before calling these functions. On return, if errno is non-zero or fetestexcept(FE_INVALID | FE_DIVBYZERO | FE_OVERFLOW | FE_UNDERFLOW) is non-zero, an error has occurred. RETURN VALUE Upon successful completion, these functions shall return the sine of x. If x is NaN, a NaN shall be returned. If x is ±0, x shall be returned. If x is subnormal, a range error may occur and x should be returned. If x is ±Inf, a domain error shall occur, and either a NaN (if supported), or an implementation-defined value shall be returned. ERRORS These functions shall fail if: Domain Error The x argument is ±Inf. If the integer expression (math_errhandling & MATH_ERRNO) is non-zero, then errno shall be set to [EDOM]. If the integer expression (math_errhandling & MATH_ERREXCEPT) is non-zero, then the invalid floating-point exception shall be raised. These functions may fail if: Range Error The value of x is subnormal If the integer expression (math_errhandling & MATH_ERRNO) is non-zero, then errno shall be set to [ERANGE]. If the integer expression (math_errhandling & MATH_ERREXCEPT) is non-zero, then the underflow floating-point exception shall be raised. Practice of Standards Formalization

  15. ISO 10697 Requirements • Real and complex elementary functions (no erf, gamma, j0, y1, … ) • Only symmetric rounding modes (no rounding to + or to –) • Preservation of sign • Preservation of monotonicity • Inaccuracy0.5-2.0 ulp • Evenness and oddity • Exact values :cosh(0) = 1, log(1) = 0, … • Asymptotics near 0 : cos(x) ~ 1, sin(x) ~ x, … • Relations : expm1 <= exp, cosh >= sinh, atan <= ↓(π/2 ) , … for sin, cos, tan – small arguments only Practice of Standards Formalization

  16. Summary of Requirements • Domain boundaries and poles (+ flags) • Exact values, limits and asymptotics • Preservation of sign and monotonicity • SymmetriesEvenness, periodicity, others: Г(1+x) = x·Г(x) • Relations and range boundaries • Precision Correct rounding(according to mode) • Computational accuracy • Interoperability and portability of libraries and applications • Feasible – ~ia64 (Intel), crlibm (INRIA) Practice of Standards Formalization

  17. Requirements Tested Extension of IEEE 754 on all library functions • Correctly rounded results for 4 modes • Except for ones contradicting to range boundaries • Infinite results in overflow and precise infinity cases • In overflow rounding to 0 returns the biggest finite number • NaN results outside of function domain (and for NaN args) • Exception flags • INVALID (and EDOM for errno) : Incorrect arguments • DIVISION-BY-ZERO (and ERANGE for errno) : Infinite result • OVERFLOW (and ERANGE for errno) : Too big result • UNDERFLOW (and ERANGE for errno) : Too small result ( + dnr) • INEXACT : Inexact result Practice of Standards Formalization

  18. Test Data Sources • Bit structure of FP numbers • Boundaries • 0, -0, +, -,NaN • Least and greatest positive and negative, normal and denormal • Mantissa patterns FFFFFFFFFFFFF16 FFFFF1111000016555550000FFFF16 Both arguments and values of a function • Intervals of uniform function behavior • Points hard to compute correctly rounded result rint(262144.25)↑ = 262144 0100000100010000000000000000000100000000000000000000000000000000 x10000010001xxxxxxxxxxxxxxxxxx0100000000000000000000000000000000 Practice of Standards Formalization

  19. Intervals and Boundaries • Neighbourhoods of 0, ±∞ • Poles and overflow points • Zeroes and extremes • Tangents and asymtotics – horizontal and diagonal max 0 Practice of Standards Formalization

  20. Table Maker Dilemma • Rounding to the nearest f = x.xxxxxxxxxx|011111111...1xx... f = x.xxxxxxxxxx|100000000...0xx... • Rounding to 0, +, - f = x.xxxxxxxxxx|00000000...0xx... f = x.xxxxxxxxxx|11111111...1xx... tan(1.11011111111111111111111111111111111111111111000111112·2-22) = 1.1110000000000000000000000000000000000000000101010001 0 178 010…2·2-22 sin(1.11100000000000000000000000000000000000000111000010002·2-19) = 1.110111111111111111111111111111111111110000001011100006711101…2·2-19 j1(1.10000000000000000000000000000000000000000000000000112·2-23) = 1.011111111111111111111111111111111111111111111110100009411001…2·2-22 0,5 ulp ! ? Practice of Standards Formalization

  21. Number of Hard Points Probabilistic evaluation Uniform independent bits distribution • Total N = 2(n-k-1) values • ~N·2-m havem consecutive equal bits Real data forsin on exponent -16 Practice of Standards Formalization

  22. Hard Points Calculation Feasible only for single precision numbers • Exhaustive search • Continued fractions(Kahan, 1983) • Dyadic method (Tang, 1989; Kahan, 1994) • Reduced search (Lefevre, 1997) • Lattice reduction (Gonnet, 2002; Stehle, Lefevre, Zimmermann, 2003) • Integer secants method (2007) X ≈ N·π; X = M·2m; 2(n – k – 1) <= M < 2(n – k)  π ≈ (2m·M)/N sqrt(N·2m) ≈ M + ½; 2(n-k-1) <= M, N < 2(n-k)  2(m+2)·N = (2·M + 1)2 – j  (2·M + 1)2 = j (mod 2(m+2)) F(x) = f(x) – a·x – b = c1x2 + c2x3 + c3x4 + … F(x) = c1(G(x) )2, G(x) = x + d1x2 + d2x3 +… G(x) = y  x = H(y), H is the reversed series xm = H(sqrt(m/c12z))  F(xm) – a·xm – b = m/2z 3386417804515981120643892082331156599120239393299838035242121518428537554064774221620930267583474709602068045686026362989271814411863708499869721322715946622634302011697632972907922558892710830616034038541342154669787134871905353772776431251615694251273653 · π/2 = 1.0110101011000101101100100110001011001010000111111110 1857 011…2·2849 sin(1.01101010110001011011001001100010110010100001111111112·2849) = 1.11111111111111111111111111111111111111111111111111 1690110…2·2-1 j = 15 sqrt(1.00100101011001010110010111001010110111001011111101002) = 1.0001001000001111100110011001111010011001001101110100 0 150 000…2 2–z Practice of Standards Formalization

  23. Test Suite Composition • Hard points • double • Some hard points with ≥ 48 additional bits can be found in crlibm testshttp://lipforge.ens-lyon.fr/projects/crlibm • Calculated (some) hard points with ≥ 40 additional bitsforsqrt, cbrt,sin, asin, cos, acos, tan, atan, sinh, asinh, cosh, tanh, atanh, exp, log, exp2, expm1, log1p, erf, erfc, j0, j1 • float (single precision) • All hard points with ≥ 17 additional bitsforsqrt,cbrt, exp, sin, cos • extended double • All with ≥ 53 additional bits forsqrt, some for sin, exp • Test suites developed • double : 58 real variable POSIX functions • Correct values calculated by Maple and MPFR Practice of Standards Formalization

  24. Tested Libraries Practice of Standards Formalization

  25. Test Results: Details rint(262144.25)↑ = 262144 expm1(2.2250738585072e−308) = 5.421010862427522e−20 logb(2−1074) = −1022 to nearest to –∞ exp(553.8042397037792) = −1.710893968937284e+239 to 0 to +∞ exp(−6.453852113757105e−02) = 2.255531908873594e+15 sin(33.63133354799544) = 7.99995094799809616e+22 erf(3.296656889776298) = 8.035526204864467e+8 cosh(627.9957549410666) = −1.453242606709252e+272 cos(917.2279304172412) = −13.44757421002838 acos(−1.0) = −3.141592653589794 erfc(−5.179813474865007) = −3.419501182737284e+287 sinh(29.22104351584205) = −1.139998423128585e+12 sin(− 1.793463141525662e−76) = 9.801714032956058e−2 Exact 6-210 ulp errors Errors in exceptional cases 210-220 ulp errors 1 ulp errors* Errors for denormals >220 ulp errors Unsupported 2-5 ulp errors Completely buggy Practice of Standards Formalization

  26. Implementations with Same Results Unsupported Practice of Standards Formalization

  27. Conclusion • Formalization can uncover numerous issues in mature industrial standards like POSIX (and more in implementations) • But it may be not only ineffective but even impossible Practice of Standards Formalization

  28. Thank you! Questions? kuliamin@ispras.ru www.ispras.ru/~kuliamin Institute for System Programming, Software Engineering Department www.unitesk.com www.linuxtesting.org Practice of Standards Formalization

More Related