Jens Schwarzbach Center for Mind/Brain Sciences University of Trento, Italy

1. Introduction toA Simple Framework (ASF)for behavioral and neuroimaging experiments based on the Psychophysics Toolbox for Matlab Jens Schwarzbach Center for Mind/Brain Sciences University of Trento, Italy

2. ASF • ASF stands for A Simple Framework • It is based on the Psychophysics Toolbox for Matlab • It is an attempt to take care of those tasks that are inherent in every computer controlled experiment: stimulus presentation and response collection • Its emphasis is on accurate timing, easy interfacing with different hardware and clear documented interface of what goes in and what comes out

3. ASF a simple framework • When things are really simple • ASF’s standard setup is to show prefabricated pictures and prefabricated sounds based on information that the user supplies in text-files • When things are not so simple • Sometimes there are so many stimuli to be prepared in advance (e.g. moving stimuli, or parametric variation of size and position) that it becomes a burden or impossible to prepare or load all stimuli into graphics memory ahead of the experiment. Another example is presenting text: It is very inefficient, inflexible, and ugly to render text on bitmaps for slide shows • In this case, you can revert to online rendering. The function that shows bitmaps in standard mode gets replaced with a function that creates the stimulus on the fly. In ASF such a function is called a user-supplied trial function or just a plugin. You can program your own. The ASF package provides you with a template .\ASimpleFramework\code\ASF_showTrial_plugins \ASF_showTrialSample.m, which you can use as a starting point for your own developments

4. A Simple Framework • Divides experiments into different modules: • Design module • Randomization • Calculation of Timing • Stimulus Rendering • Runtime module • Stimulus Presentation • Response Collection • Analysis module • Data Analysis

5. The Architecture of ASF DESIGN (USER) RUN (ASF) ANALYZE (USER) Stimulus Definition File (STD) contains names of bitmaps • Trial Definition File (TRD) contains info about: • Which condition does trial belong to (code)? • What bitmap shall be shown when? • When shall response collection start)? • What counts as a correct response? • Output Filecontains variable ExpInfo that holds data for: • What was shown when? • What did participant respond when? • What hard- and software were used? Runtime Engine presents pictures and sounds as determined by STD, TRD, and Cfg • Configuration variable (Cfg) contains info about: • Hardware • User-defined settings

6. A s(till) Simple Framework • Divide into different modules: • Design module • Randomization • Calculation of Timing • Runtime module • Stimulus Rendering • Stimulus Presentation • Response Collection • Analysis module • Data Analysis • Putting stimulus rendering into the runtime module is more flexible, but it requires some effort in realtime-programming. ASF deals with this using plugins.

7. The Architecture of ASF with plugins DESIGN (YOU) RUN (ASF) ANALYZE (YOU) Stimulus Definition File (STD) contains names of bitmaps • Trial Definition File (TRD) contains info about: • Which condition does trial belong to (code)? • Info that gets interpreted by your ShowTrial plugin • When shall response collection start)? • What counts as a correct response? • Output Filecontains variable ExpInfo that holds data for: • What was shown when? • What did participant respond when? • What hard- and software were used? Runtime Engine renders stimuli as determined by STD, TRD, Cfg and whatever you have programmed in your ShowTrial plugin • Configuration variable (Cfg) contains info about: • Hardware • User-defined settings

8. Files that belong to the ASF packageThese are functions you can call in your own plugins, but which must not be edited. • For running experiments • ASF.m the core program, which you call to run an experiment • KillSerial.mwhen something in an MR experiment crashed and you want to start again, this makes sure that the communication with the scanner can be reestablished • Useful for creating trd-files • ASF_encode.mencode factorial information to single number code • Useful for your plugins • ASF_showTrialSample.mcopy and make your own experiment with online rendering, adaptive testing, control a space mission • ASF_xFlip.mflip display synchronized with vertical retrace (and much more). One of the most important functions of ASF. • ASF_playSound.mwrapper function for different methods to play sound • ASF_sendMessageToEyelink.mfor logging stimulus or other events in the stream of eyetracking data • ASF_checkforuserabort.mchecks whether somebody has pressed the q key to quit program • ASF_decode.mdecoding codes to factorial information • ASF_setTrigger.msend signals to parallel port or an Arduino board (useful for TMS, EEG, MEG) • ASF_arduinoTrigger.m (helper function for triggering using an Arduino board) • ASF_waitForMousePressBenign.mjust waits for a mouse press • ASF_waitForResponse.mwaits for a response. Can handle many different response devices. • ASF_waitForScannerSynch.mpauses until a (MR-)scanner trigger arrives • Useful for data analysis • ASF_timingDiagnosis.mto check whether everything happened when is was supposed to • ASF_readExpInfo.mreads log-files created by an ASF experiment • ASF_getTrialOnsetTimes .mreturns a vector for trial onset times • Not really needed for your programs (ASF needs it though) • ASF_initEyelinkConnection.m establishes initialization of an EyeLink eyetracker • ASF_shutdownEyelink.mcloses connection with EyeLinkeyetracker • ASF_initParallelPortInput.m initializes the parallel port using the data acquisition toolbox • ASF_initResponseDevice.m initializes response device (mouse, voicekey, Lumina-parallel, Lumina-serial, keyboard) • ASF_readTrialDefs.m reads trd-files into a structure • ASF_readStimulus.mreads bitmaps and avi files • ASF_makeTexture.minternal function for mapping bitmaps onto textures • ASF_PTBExit.mgenerate a graceful exit (save logs, shut down hardware, remove stuff from memory) • Experimental or not-really-ready stuff • ASF_checkTrials.mundocumented feature • ASF_onlineFeedback.mon dual screen setups, this shows the current cumulative info on errors and reaction time (even for factorial experiments) • ASF_pulseTrainNI.mcan create pulse trains on a national instruments card for rTMS • ASF_pulseTrainNI2.m can create pulse trains on a national instruments card for rTMS

9. Why not edit ASF? • ASF is designed such that it is sufficient for the purpose of your experiment… • or that you can make it so by programming a plugin. • Should that be not the case, OK, edit it but get back to jens.schwarzbach@unitn.it ASAP such that I can see how to enhance ASF’s capabilities. Otherwise you would be stuck with your customized ASF and you cannot profit from updates.

10. A remark on how I think about ASF • ASF is a deluxe slide show program. Some defined slides are shown in a defined order for defined times. • So, if you want to present text or sound, write a plugin that shows empty slides at those times when you want to present text/sound and make your plugin not only present the slide, but also the other stimulus class. • Why not just presenting text or sound at some time and not think about presenting slides? Because when showing slides the graphics card controls the timing. And graphics cards have the best timers available. They also produce events (vertical retrace) to which you can hook up your program. At the moment of a retrace your computer is in a defined state, and that helps in making experiments stable. • This approach forces you to see time to be quantized (frame 1 … n) instead of something continuous. For computers this closer how they deal with time.

11. Important: Please Note • When using ASF and related code on different computers, you have to keep in mind the following: • Use the same refresh rate (configurable) otherwise the timing of experiments will differ. • Different display sizes or distances between participant and display require adjusting stimulus-size in order to keep visual angle constant. • You may want to use a calibration device to keep average display luminance constant. • You may want to gamma-correct your display.

12. Problems I see often • Potential accumulation of timing errors • Supose you want to show on each trial a different picture for two seconds. Then you have two options for defining this in the trd file Code 0 picnum 120 startRTonPageendRTonPagecorrectResponse Code 0 picnum 120 startRTonPageendRTonPagecorrectResponse Code 0 picnum 120 startRTonPageendRTonPagecorrectResponse … • But you can achieve the same with (make sure that Cfg.useTrialOnsetTimes = 1) Code 0 picnum 1 startRTonPageendRTonPagecorrectResponse Code 2 picnum 1 startRTonPageendRTonPagecorrectResponse Code 4 picnum 1 startRTonPageendRTonPagecorrectResponse … • The first method can be disadvantageous, because it can happen that you are accumulating timing errors (suppose it took your computer some time to start the presentation of picture 1, e.g. the start happened at t = 0.1s instead of t = 0.0 s; every subsequent picture would start 0.1s too late).In the latter solution the program can catch up. There may be delays in a given trial, but they would never accumulate. This is in particular important for fMRI experiments. For MEG/EEG experiments you should send a trigger at any given stimulus presentation and record that for later correct data analysis.The latter solution is also better for plugins in which some stimulus rendering has to be done. In this particular example the upper solution would leave less than 1 frame (<16.67ms) time for such an operation, the lower solution would leave almost 2 seconds time (120-1 = 119 frames (<1983.3 ms)). That is a huge difference! • Also for implementing jittering of onset times, this is the way to go. Make the last picture of any given trial as short as possible (I do this by adding a blank or fixation picture with duration of 1 frame to the end of any stimulus sequence) and put the absolute trial onset time for the next trial into column 2 of the next trial definition in the trd file.

13. What can I do if I don’t want to collect a response in a trial? • assign startRTonPage a number higher than the highest page number in you trial definition