CS 352: Computer Graphics

1. CS 352: Computer Graphics Input andInteraction

2. What good is Computer Graphics? • JS1k canvas examples: 12345 • Games, visual demos…of what value? Is there a Christian perspective? • Communication of information to the user Data visualization, simulation, GUI Even a word processor is an "interactive graphics program" • Communication between users "Collaborative environments" are hot (multi-player games?) Social networking is transforming the world… • Interaction is an essential component

3. Interaction • Much of the fun and utility of graphics is in interaction:accepting user input that affects the display • Paint programs, modeling, games, word processors, spreadsheets, powerpoint … • User initiates input events such as clicking on a menu or drawing a circle • Computer response by changing graphical display

5. Projects 3 • Projects 3 will be a paint program • we'll learn 2-D graphics, interaction, event-loop programming, double-buffering simple animation, basic image processing • Features: • Pen, line, and rectangle tools • Color, pen size selectors • Image processing (sharpen, blur, detect edges) • Open, save images • Toolbar and menu for controlling application

6. How to set up an application UI? • How to make a menu? • How to make a color picker? • How to make a toolbar? • How to make toolbar buttons pop in and out?

7. How to paint? • How do I make a colored line follow the mouse or fingertip? • Interactive prog: • Input devices • Event handlers • Event loop

8. Input devices • Interaction requires handling input devices • Physical:mouse, fingertip, keyboard, joystick, digitizer, accelerometer, head tracker… • Logical: • Text • Locator • Valuator (slider) • Stroke • Color picker • How to read?

9. Input devices • Sample mode • Program reads the current state of input device (frequently) • Event mode • Each click or motion generates an event that goes on a queue • Program can process events from queue • HTML?

10. Event-loop programming • Events: button press, slider value change, menu selection, mouse move, keypress • Event loop: • Poll (or wait) for events • Process events, update state • Update display • (ideally: wait for rest of frame time to elapse)

11. State-driven • Typically, the event loop is driven by a big state table • e.g. depressing the paintbrush tool releases other tools, puts me in "paint mode" • Good libraries will handle some of the bookkeeping for you • You still typically have to handle some state

12. State diagram

13. Event loop in HTML/JS • HTML/JS provides event queue, support for many basic events (mousedown, mouseup mouseover, mousemove, keypress, key release, value change, button click, etc.) • You are on your own for higher-level events, e.g. clicking on a toolbar tool • It's also possible to set a function to run every 15 ms, sample input devices

14. Events for painting? • mousedown: • go into paint mode • store mouse position • draw initial dot • mouseup: • exit paint mode • mousemove: • if in paint mode • draw a line from old mouse position to current • set old mouse position to current position

15. Event handling with jQuery • Binding events to functions $(cpaint.canvas).bind('mousedown', cpaint.drawStart) • Processing events cpaint.drawStart = function(ev) { var x, y; x = ev.pageX - $(cpaint.canvas).offset().left; y = ev.pageY - $(cpaint.canvas).offset().top; ev.preventDefault(); cpaint.paintmode = true; cpaint.oldX = x; cpaint.oldY = y;

16. Looking neat and spiffy • How to avoid crinkles in your paint strokes?

17. Looking neat and spiffy • How to avoid crinkles in your paint strokes? • Draw connected paths • Or just use line caps

18. Menus • There are many jQuery menu plug-ins… <ul id="mainmenu"> <li>File <ul> <li id="menuNew">New</li> <li id="menuOpen">Open</li> <li id="menuSave">Save</li> ------------------------------------- $('#menuNew').bind('click', cpaint.clear); $('#menuOpen').bind('click',cpaint.open); $('#menuSave').bind('click',cpaint.save);

19. Toolbar • How to make a toolbar? • How should buttons behave? • State diagram?

20. Buttons • Widgets may have several states • State should be evident in visual feedback • E.g. how exactly does a button work? States? • State transition diagram • Most buttons: six states, with six different appearances • neutral • neutral-highlighted • neutral-depressed • selected • selected-highlighted • selected-depressed • Events: mousedown, mouseup, enter, exit • Transitions: what happens in each state under each event?

21. Button state diagram (buttons selectable, not unselectable) N S NH SH N: neutral H: highlighted (usually mouse over) S: selected D: mouse down NHD SHD Press Move

22. Other button considerations • Could also consider • Tooltips • Whether button merely clicks and releases or can be selected • Whether button can be unselected (e.g. B/I/U vs. Left, Center, Right) • Want consistent appearance, behavior over whole program – or whole computer • Really need a library implementation and a strict set of UI guidelines…

23. 4-State toolbar buttons: CSS .toolbarCell { background-color:#ddd; width:20pt; height:20pt; border: solid #eee 2px; -webkit-box-shadow: 2px 2px 2px #666; } #markerButton { background: url(img/paintbrush.png) no-repeat center center; } .selected { -webkit-box-shadow: inset 2px 2px 2px #666; } .toolbarCell:hover { border:solid #555 2px; }

24. 4-State toolbar buttons: JS $('#markerButton').bind('click', {tool:"marker"}, cpaint.selectTool); ------------------- cpaint.selectTool = function(ev) { cpaint.tool = ev.data.tool; // get tool name $('.toolbarCell').each(function(index) { // unselect $(this).removeClass('selected'); // others }); var tool = '#' + cpaint.tool + 'Button'; // get ID $(tool).addClass('selected'); // select }

25. Paintbrush size selector • How?

26. Color picker • Google "jQuery color picker"…

27. How draw a line? • What kind of feedback is normal? • Rubber Banding • Events and actions?

28. Save & restore • Create your own off-screen canvas • Copy it back on each mouse movement • Events and actions?

29. Save & Restore event handling • Mousedown • enter line mode • remember mouse position as startx, starty • save screen • draw initial dot • Mousemove • paste saved screen • draw line from startx, starty to current mouse pos • Mouseup • exit line mode

30. Analysis • Drawbacks? • slow – eats memory bandwidth for breakfast • copy smallest possible rectangle? Only points from line? • possible flickering • use double buffering?

31. Double buffering • Have two frame buffers, "front" and "back" • Draw into back buffer • At vertical retrace time, swap buffers • This is a fundamental graphics technique… • …not built into canvas… • …though, some drawing aggregation seems to happen automatically, behind the scenes; • …not usually necessary in canvas, but browser dependent

32. Fake Double Buffering in Canvas • Create off-screen canvas canvasBuffer = document.createElement('canvas'); canvasBuffer.width = canvas.width; canvasBuffer.height= canvas.height; canvasBufferContext= canvasBuffer.getContext('2d'); • Draw into off-screen canvas canvasBufferContext.[drawSomeStuff…] • Copy onto display canvas context.drawImage(canvasBuffer, 0, 0);

33. How to erase? • Draw background color…

34. Save, Load? • Save • Copy pixels to an image so users can right-click, save-as • Load • What are the security risks? • Can only load files from the same server • Can use a file chooser if it's a local app • Security policies are not entirely in place

35. Resizing • How would you allow the user to shrink, expand image? • What would happen to image resolution?

36. Store drawing commands • Realistically, to redraw effectively, you have to save all drawing commands pen color red fill color blue line width 5 circle 5 10 12 textfont Times 12 text 10 10 "hello world" • Replay commands to redraw scene • Could store commands in a file (e.g. Mac PICT file) • For some kinds of drawings, files are smaller and more accurate than bitmaps

37. Other toolkits • HTML offers basic UI capabilities in a cross-platform context • Writing your own UI extensions is tedious • jQuery plugins extend capabilities • Ideally, all UI elements ought to be built in • In the real world, they come in platform-specific 'toolkits' or 'windowing libraries' • E.g. MFC, QT, OpenLook, Cocoa …

38. Image processing • Image processing examples: • blur, sharpen • detect edges • enhance contrast • noise reduction • posterize • fisheye • redeye reduction • find faces • …

39. How to fade an image? • Could average pixel colors with white • Or just decrease alpha…

40. How to blur an image? • Blurring is an averaging process • Convolution: apply a convolution kernel, e.g. • Center on pixel of interest • Multiply each value by color val underneath • Add results • Gaussian smoothing

41. Gaussian smoothing

43. How to sharpen an image? • It's a differentiation process • What's Unsharp Masking?

44. How to sharpen an image? • It's a differentiation process • What's Unsharp Masking? • Subtract blurred version from original • Add back to original

45. Sharpening • Subtract neighbors • Like subtracting a blurred version of the image • Unsharp Masking • E.g. convolve with a kernel like one of these:

46. Edge detection? Wikipedia

47. Sobel Edge Operators • Look for horizontal and vertical variation • Could do this at different sales or resolutions • Note: results maybe positive or negative numbers; must normalize

48. Image Compression • What if you created say six different lower-resolution images and only stored the difference at each resolution? • Note: most of the data is in the highest-frequency component • An early image compression technique

49. The alg that changed imagery… • Discrete cosinetransform (DCT) • (Wikipedia)

50. JPEG image compression • DCT to convert to frequency domain • Perceptual modeling • Coefficient quantization • Entropy coding