1 / 39

Automated Testing Using Titan

Automated Testing Using Titan. Cristopherson Martinez Gustavo Diaz Prado Open Source Software Team July 15 th , 2010. Agenda. Overview Our testing problem What is Titan? How can I get Titan? Requirements How it works? Preparing to test Choosing a device under test WebServerSdp

wenda
Télécharger la présentation

Automated Testing Using Titan

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Automated Testing Using Titan Cristopherson Martinez Gustavo Diaz Prado Open Source Software Team July 15th, 2010

  2. Agenda • Overview • Our testing problem • What is Titan? • How can I get Titan? • Requirements • How it works? • Preparing to test • Choosing a device under test • WebServerSdp • Phidget • Telnet • What is a test descriptor? • Choosing a test provider to get the test descriptors • Running a test in Titan • Checking the auto-generated results • Differences between using Titan with LDP and an SDP

  3. Agenda • Non-Interactive mode • Limitations. • What is NOT Titan? • Titan applications • Parallel testing (Experimental) only available for TelnetConnection DeviceUnderTest • Continuous Testing Support

  4. 1. Overview • a) Our testing problem • The Linux Baseport (LBP) Testing team developed many test cases for several drivers which are executed on a development/release basis for the Linux kernel projects. • A test case is executed manually by entering to the board’s terminal using the serial port or by telnet using the ethernet port. • When a test case is executed the tester hopes it will begin and finishes cleanly without hanging the board so more test cases can be executed without power cycling the board. • However when a test case hangs the board a manual power cycle must be performed by the tester, otherwise the testing cannot continue. • After the board is reset the tester will need to setup the board again then continue with the testing. This can be time consuming. • Other tests includes resetting the board after each run which can be time consuming and annoying too.

  5. 1. Overview • b) What is Titan? • Titan (Texas Instruments Test Automation) is a testing framework programmed with the Java language born in the Linux Baseport team. • The Linux Baseport (LBP) Team created Titan to automate the testing of a great number of the drivers on the kernel code freeze and release phases without human interaction. • Titan can know if the board is frozen so it can actually power cycle the board without human interaction and continue with the testing. This approach has been greatly useful for nightly testing in the Test team. • Titan has been created with the “user friendly” idea in mind, it is actually easy to use.

  6. 1. Overview • c) How can I get Titan? • Binaries are available in Titan GForge page: • https://gforge.ti.com/gf/project/titan/frs/ • Titan Tool code can be downloaded from : • http://dev.omapzoom.org/?p=cristopherson/titan-development.git;a=summary • Titan Test Suites : • http://dev.omapzoom.org/?p=richo/device_driver_test.git;a=tree;f=automation;h=dce463979139c4f0d12ebad878deafa426b1b8ef;hb=f75ee6c4d2a39059f084e88020fbf31fdf032e26

  7. 1. Overview • d) Requirements • Titan uses the Java Standard Edition Runtime Environment 6 (JRE) technology which can be downloaded from Sun Microsystems here: http://java.com/en/download/index.jsp (Useful for windows users) http://java.sun.com/javase/downloads/index.jsp • Windows or Linux with a graphical environment (X Server) • Titan uses some third party libraries. • RXTX, this libraries are already included in the binary package in Titan GForge page. • Linux: librxtxParallel.so, librxtxSerial.so • Windows: rxtxParallel.dll, rxtxSerial.dll • Phidget, for Linux OS the library is already included but for Windows System it should be downloaded from Phidgets Home page: http://www.phidgets.com/downloads/libraries/Phidget21-windevel_2.1.6.20090317.zip • Click on Phidget 21 libraries uncompress the zip file and look for phidget21.dll inside of the architecture folder, for this example x86 folder. • Linux: libphidget21.so • Windows: phidget21.dll

  8. 2. How it works • Titan takes the board control and sends data through serial port or telnet connections. The data is a set of commands which are executed on the target device one by one, this data is read from a “test descriptor” file that contains the commands to execute during a test. • Titan stores the output from each command for analysis. Using this data Titan can know when a test has passed or has failed. • When all the tests finish Titan interprets this information and store it in a detailed results report.

  9. 3. Preparing to test • a) Choosing a device under test • WebServerSdp • We need to tell Titan which device under test implementation we want to use. It means which board we wants to use for the testing. • Go to the menu Tools -> Settings -> Device under test and select WebServerSdpBoard. Click OK. Note: WebServerSdpBoard only works for SDP boards

  10. 3. Preparing to test • In the next window you need to configure your device under test properties. When you are done click Save. This is the typical configuration that is being used in your minicom or teraterm terminal to communicate with the board trough serial port.

  11. 3. Preparing to test This configuration can be taken from the board u-boot environment.

  12. 3. Preparing to test The webserver makes reference to an IP Address that is assigned to the board in order to turn it on or off from a remote server. It is provided by the display located in the left-bottom side of the board (SDP3430 and SDP3630)

  13. 3. Preparing to test Select the board type and FS that are going to be booted. After setting all the environment click save

  14. 3. Preparing to test • Phidget • Also there is the option to select a Phiget DUT but in order to use this option, there is an extra hardware that is required. • The inputs are the same as the used ones for WebServerSdpBoard. • Go to the menu Tools -> Settings -> Device under test and select PhigetBoard. Click OK. Note: This can work for both SDP and LDP boards

  15. 3. Preparing to test NC3CNO NC2CNO NC1CNO NC0CNO The phidget needs some setup before it can be used. You will need some cable, a soldering gun, a female and male connectors similar to those you find in the SDP power supply and the SDP board. They can be easily found in an Electronics shop. To get the phidget working wire the relay labeled as 0 (zero) in the common terminal and the normally Opened terminal. You should end with phidget wired like the diagram. Note: The Phidget we are using is the PhidgetInterfaceKit 0/0/4: http://www.phidgets.com/products.php?product_id=1014

  16. 3. Preparing to test • We will need an OMAP Zoom LDP, a Phidget and a battery emulator connected like the diagram below. Phidget connected by USB to your PC LDP Common SDP Power Supply Battery emulator Serial port cable connected to your PC NOTE: Why use a Phidget? The phidget allows Titan to control the energy flow from the battery emulator to the LDP. If a test needs to reboot the LDP Titan will ask the Phidget to cut the current and restore it making the LDP to start in a fresh environment. Titan is smart enough to know when the LDP has hung, when this happens it will do a power cycle and continue testing.

  17. 3. Preparing to test • Telnet • TelnetConnection establish a communication with the board using ethernet. • The board should be already booted to use this option • Telnet server should be running in the target machine. • Go to the menu Tools -> Settings -> Device under test and select TelnetConnectionBoard. Click OK.

  18. 3. Preparing to test To use telnet the user should provide the following inputs: • TelnetServer: The target board IP address. • Telnet Port: The port used to establish the connection, by default is 23 • Login Required: Mark this option if login is required. • Login User: The user to use to access the target board. Only enabled if login required option is selected. Note: This option does not support reboot command.

  19. 3. Preparing to test • b) What is a test descriptor? • A test descriptor is an XML file that tells Titan how to run a test. • It contains important information like: • The unique test id • An overview of what the test does and the authors. • How Titan will execute the test case. • How Titan will know when a test failed. • How Titan will know when a test passed.

  20. 3. Preparing to test • Test descriptor example <titan> <test id="L_DD_AUDIO-ALSA_0001"> <information> <autor> Ricardo Perez </autor> <description> AUDIO-ALSA Test Scenario 0001 </description> <dateofcreation> 03/26/2008 </dateofcreation> </information> <run> <action> <deviceundertest>reboot</deviceundertest> </action> <action> <command>cd /12x/audio-alsa/scripts/</command> </action> <action> <command>./test_runner.sh L_DD_AUDIO-ALSA_0001</command> </action> </run> <pass> <output> <value>stat=0</value> </output> </pass> <fail> <timeout time="240"/> <output> <regularexpression>stat=[^0]\d*</regularexpression> </output> </fail> </test> </titan>

  21. 3. Preparing to test • c) Choosing a test provider to get the test descriptors • We need to feed Titan with the test descriptor files specifying a test provider. • Go to the menu Tools -> Settings -> Test provider and select localtestrepository. Click OK.

  22. 3. Preparing to test • The local test repository test provider implementation reads the XML files present in a folder in your hard disk. • In the next window you need to select in which folder are the test descriptor files. Titan comes preconfigured to look in the test_descriptors folder in the Titan installation files so it is safe to click Save without changes. • You can put new test descriptors in that folder at will.

  23. 3. Preparing to test • After clicking Save the main window will refresh showing the test descriptors found by the selected test provider implementation.

  24. 4. Running a test in Titan • To select the tests you want to run just click in the checkbox of the test in the tree. If you want to select an entire branch click on the parent node instead of clicking each test. • You can see a brief summary of the test you have already selected in the bottom panel of the main window. • To run the tests press the Run tests button in the main window (below the test description).

  25. 4. Running a test in Titan • You will see in a new window to see how the tests are executing within Titan. When the testing is done a PDF report will be generated with a detailed information of what happened during one test execution. The path where you can get the report is shown in the Titan log.

  26. 5. Checking the auto-generated results • The automated report is a PDF file which contains a list of the executed tests and their results. By clicking on each test you can see a detailed summary.

  27. 5. Checking the auto-generated results • A test result is determined in the following way: • Green means the test passed • If the pass condition happens • Red means the test did not pass • If the fail condition happens. It doesn’t care if the pass condition happens or not. • Blue means the test had a timeout • If the test time reaches the specified timeout and the run actions have not finished yet. • Orange means Titan couldn’t determine a result so it marks it as unknown • If the pass and fail conditions didn’t happen and the timeout was not reached

  28. 6. Differences between using Titan with LDP and an SDP • A SDP board can do a power cycle using a Webserver and a Phidget device. • A LDP (Zoom2 or Zoom3) board just can use a Phidget device in order to do that. • Why use a Phidget? • The phidget allows Titan to current flow from the battery emulator to the LDP. If a test needs to reboot the LDP, Titan will ask the Phidget to cut the current and restore it making the LDP to start in a fresh environment. Titan is smart enough to know when the LDP has hung, when this happens it will do a power cycle and continue testing. • For more information about how to setup the LDP with a Phidget got to the LBP Titan Sharepoint site: Linux Baseport > Shared Documents > Automation > TITAN (TI Test AutomatioN) > Slides  > Titan setup-LDP

  29. 7. Non-interactive Mode Titan can be executed from the command line and run testcases without human interaction. This is available in the tarball Titan version and it has been tested on Ubuntu8.04 Linux Machine. To Run Titan in non-interactive move just type the following command: ./start.sh cmd dma • cmd paramater specifies that Titan is going to run in non-interactive mode. • dma parameter is the name of the TestCases Group that is going to be executed.

  30. 7. Non-interactive Mode TestCases groups are defined in settings/titan-selectedtests.xml This is an example to provide three different test groups. <titan> <configuration name="dma"> <scenario>L_DD_DMA_0001</scenario> <scenario>L_DD_DMA_0002</scenario> <scenario>L_DD_DMA_0003</scenario> <scenario>L_DD_DMA_0004</scenario> <scenario>L_DD_DMA_0005</scenario> </configuration> <configuration name=“gpio"> <scenario>L_DD_GPIO_0001_0001</scenario> <scenario>L_DD_GPIO_0001_0002</scenario> <scenario>L_DD_GPIO_0001_0003</scenario> </configuration> <configuration name=“ethernet"> <scenario>L_DD_ETHERNET_0001</scenario> <scenario>L_DD_ETHERNET_0002</scenario> </configuration> </titan> TestCases groups can be added or removed just modifying this file

  31. 8. Limitations • a) What is NOT Titan? • It’s not a development tool. • It’s not a debugger. • It’s not a bug fixer tool. • Titan cannot determine the visual or audio quality of the tests. Surely it can help to automate the testing process but in this cases a human should validate the test results. • The devices under test implementations currently support Linux only (no Windows Mobile support).

  32. 9. Titan Applications • Parallel testing (Experimental) only available for TelnetConnection DeviceUnderTest

  33. 9. Titan Applications Continuous Testing Support (Experimental) • This application provides the capability to execute a set of test cases several times continuously until user stops the execution or the stop testing condition is satisfied. • There are two ways to determine how long titan should perform this testing: • Iterations: How many times titan should execute the set of test cases • Time: How much time titan should execute the set of test cases, the time is jus an approx, since even if the time condition has been satisfied Titan will not stop its execution until all the test cases have been executed.

  34. 9. Titan Applications Continuous Testing Support (Experimental) To select this functionality you just need to do click in the Tools Menu and then in the Continuous Testing SubMenu

  35. 9. Titan Applications Continuous Testing Support (Experimental) The Continuous Testing Dialog will be displayed, by default this functionality is disabled, but you can enable it just marking the check box

  36. 9. Titan Applications Continuous Testing Support (Experimental) After that you can select either Iterations or time radio buttons.

  37. 9. Titan Applications Continuous Testing Support (Experimental) Iterations input should be the number of times that the set of test cases will be executed Time input should be the duration of the test cases execution and also provide the Unit Time

  38. 9. Titan Applications Continuous Testing Support (Experimental) After configuring the Continuous Testing Parameters, Titan can be run as normally. In this case when the TestMonitor Dialog is open you can track the current status of your continuous testing doing click in the Run menu and then in the continuous testing submenu

  39. Troubleshooting • The board boots but it gets stuck in the kernel prompt • Check if the bootcmd variable is not set, if so, remove it: setenv bootcmd • Boards is not booting, nothing is seen in the Titan log. • Check that there is not an application like minicon or teraterm already open, if so, it is needed to close it. • Phidget Timeout dialog is displayed. • Phidget device is not attached in check that the phidget device or cables are working OK. • Non-interactive mode is getting stuck and there is not activity from Titan process. • Before using non-interactive mode Titan settings should be specified first. This procedure is the same used in the step 3. • Also there should be a valid TestCase group in the second parameter. To see the available TestCase groups, open the settings/titan-selectedtests.xml file located in Titan folder.

More Related