1 / 25

TEST SUITE DEVELOPMENT FOR CONFORMANCE TESTING OF EMAIL PROTOCOLS

TEST SUITE DEVELOPMENT FOR CONFORMANCE TESTING OF EMAIL PROTOCOLS. Anastasia Tugaenko Scientific Adviser : Nikolay Pakulin, PhD Institute for System Programming RAS, Moscow. SYRCoSE 2010. 1-2 June, 2010 . Nizhny Novgorod, Russia. Conformance Testing.

ross-norris
Télécharger la présentation

TEST SUITE DEVELOPMENT FOR CONFORMANCE TESTING OF EMAIL PROTOCOLS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. TEST SUITE DEVELOPMENT FOR CONFORMANCE TESTING OF EMAIL PROTOCOLS Anastasia Tugaenko Scientific Adviser: Nikolay Pakulin, PhD Institute for System Programming RAS, Moscow SYRCoSE 2010. 1-2 June, 2010. Nizhny Novgorod, Russia

  2. Conformance Testing • Many implementations of protocols from different developers are functioning in the contemporary Internet • The reliability of data transfer substantially depends on implementations compatibility • The basic method of attesting implementations compatibility is conformance testing (testing to conform to the standard)

  3. Simple Mail Transfer Protocol User agent (UA) Relay mail transfer agent (Relay MTA) Local mail transfer agent (Local MTA) Internet Local mail transfer agent (Local MTA) Relay mail transfer agent (Relay MTA) User agent (UA)

  4. Simple Mail Transfer Protocol S: 220 foo.com Simple Mail Transfer Service Ready C: EHLO bar.com S: 250-foo.com greets bar.com S: 250-8BITMIME S: 250 HELP C: MAIL FROM:<Smith@bar.com> S: 250 OK C: RCPT TO:<Jones@foo.com> S: 250 OK C: RCPT TO:<Green@foo.com> S: 550 No such user here C: RCPT TO:<Brown@foo.com> S: 250 OK C: DATA S: 354 Start mail input; end with <CRLF>.<CRLF> C: Blah blah blah... C: ...etc. etc. etc. C: . S: 250 OK C: QUIT S: 221 foo.com Service closing transmission channel

  5. Post Office Protocol (version 3) User Agent POP server commands User Mailbox replies and data

  6. Post Office Protocol (version 3) S: +OK POP3 server ready C: USER mrose S: +OK mrose is a real hoopy frood C: PASS secret S: +OK mrose's maildrop has 2 messages (320 octets) C: LIST S: +OK 2 messages (320 octets) S: 1 120 S: 2 200 S: . C: RETR 1 S: +OK 120 octets S: <the POP3 server sends message 1> S: . C: DELE 3 S: -ERRno such message C: QUIT S: +OK dewey POP3 server signing off (maildrop empty)

  7. Mail Protocols Features • Mail protocols are underspecified • Mail protocols are nondeterministic • Mail protocols requirements differ in the level of obligations (MUST, SHOULD, MAY, …) • Protocol architecture is extensible

  8. Requirements for Test Suite • Requirements traceability • Availability of option defining the set of requirements supported by IUT • Completeness in terms of requirements coverage

  9. Traditional Methodology of Conformance Testing Test suite consists of formal given tests which are not connected to the implementation Connections between requirements and test purposes and between test purposes and their tests are informal Test purposes describe situations to be tested. Test purpose is realized in one or several tests Implementation is considered to conform to the standard if all test purposes have been passed

  10. Testing withMail Protocol Tester (James MPT) Script: S: 220.* -- expectedresponse (pattern) C: HELO example.org -- test stimuli S: 250.* C: QUIT S: 221.* Result: Connecting to localhost:25 <-220 petr SMTP Server (JAMES SMTP Server 2.3.1) ready +<date> -> HELO example.org <-250 petr Hello example.org (localhost [127.0.0.1]) -> QUIT <-221 2.0.0 petr Service closing transmission channel closing

  11. UniTESKTechnology • Specifications formalize requirements as pre- and postconditions • From specifications test oracles are generated • Test sequence is generated from test state machinetraversal • Observable behavior is automatically verified by test oracles

  12. Methodfor Mail Protocols Conformance Testing Developing of elementary tests Creation of requirements catalogue Designing of lite protocol model Building test state machine with dedicated states Requirements formalization Enhancement of scenario and specification Execution of test suites and analyzing the results

  13. Methodfor Mail Protocols Conformance Testing Developing of elementary tests Creation of requirements catalogue Designing of lite protocol model Building test state machine with dedicated states Requirements formalization Enhancement of scenario and specification Execution of test suites and analyzing the results

  14. Methodfor Mail Protocols Conformance Testing Developing of elementary tests Creation of requirements catalogue Designing of lite protocol model Building test state machine with dedicated states Requirements formalization Enhancement of scenario and specification Execution of test suites and analyzing the results

  15. Methodfor Mail Protocols Conformance Testing Developing of elementary tests Creation of requirements catalogue Designing of lite protocol model Building test state machine with dedicated states Requirements formalization Enhancement of scenario and specification Execution of test suites and analyzing the results

  16. Methodfor Mail Protocols Conformance Testing Developing of elementary tests Creation of requirements catalogue Designing of lite protocol model Building test state machine with dedicated states Requirements formalization Enhancement of scenario and specification Execution of test suites and analyzing the results

  17. Methodfor Mail Protocols Conformance Testing Developing of elementary tests Creation of requirements catalogue Designing of lite protocol model Building test state machine with dedicated states Requirements formalization Enhancement of scenario and specification Execution of test suites and analyzing the results

  18. Methodfor Mail Protocols Conformance Testing Developing of elementary tests Creation of requirements catalogue Designing of lite protocol model Building test state machine with dedicated states Requirements formalization Enhancement of scenario and specification Execution of test suites and analyzing the results

  19. Example: Generated Automaton for SMTP Protocol

  20. Example: Generated Report

  21. Results Number of main requirements: SMTP RFC 5321, 51 requirements POP3 RFC 1939, 58 requirements Size of the test suite: Specifications: SMTP 1400 lines, POP3 1600 lines Test scenarios: 7 scenarios, 2500 lines total Mediators: 3000 lines Implementations tested: James: 5 defects revealed HMail: 2 defects revealed

  22. Method Analysis: the Lows The absence of quick test suites updating ability (but after all preparations have done one got not a single test but a set of tests responsible for certain requirements class) Formal specification description is very laborious process

  23. Method Analysis: the Advantages High order of reusing test suite components Test oracles Test data iterators Automated generation of test sequences Automated calculation of reached test coverage Automated verdict pronouncement

  24. Conclusion The method for test sequence generation for email protocols implementation testing is presented Test suite for conformance testing of email protocols is constructed The following disagreements between email implementations and standards were detected: Absence of required commands supporting Protocol rules violation Wrong reply codes to the protocol commands Cycling while redirecting mail

  25. Thank You!Questions? Anastasia Tugaenko tugaenko@ispras.ru

More Related