ACM SIGCSE 2004: Multimedia Construction Projects

1. ACM SIGCSE 2004: Multimedia Construction Projects Mark Guzdial College of Computing Georgia Institute of Technology guzdial@cc.gatech.edu http://www.cc.gatech.edu/~mark.guzdial Funding to support this work came from the National Science Foundation, Georgia Tech's College of Computing, GVU Center, Al West Fund, and President's Undergraduate Research Award.

2. Plan • 7-7:10: Introductions • Why are we all here? Tune workshop to needs • 7:10-7:30 (23 slides) • What’s on your CD • Why media computation • What we’re doing in the course at Georgia Tech • 7:30-8:00: Picture manipulations in Python (65 slides) • 8:00-8:15: Sound manipulations in Python (40 slides) • Video processing (probably won’t get to) (21 slides) • 8:15-8:30: Break • 8:30-8:45: Mapping to Java (Squeak, if desired) • 8:45-9:45: You Play • 9:45-10: Wrap-up

3. Introductions • Your name,Where you’re from,What do you teach where you’d like to use media projects,What languages you teach,and What do you want to get out of this workshop?

4. What’s on your CD • Materials from our Introduction to Media Computation course • Pre-release PDF of Media Computation book in Jython • Course slides • MediaTools: Squeak-based media exploration tools • Python (Jython) • Jython Environment for Students (JES) • Instructor Resources (how to tune for different places, grading tool) • Java • Media classes • DrJava • First three chapters of Media Computation book in Java • Squeak • Latest Squeak • Squeak computer music essays • Material for this workshop • Workshop, Java, and Squeak API slides

5. A Computer Science Undergraduate Degree is Facing Challenging times • We’re losing students, at an increasing rate. • Women and minority percentage of enrollment dropping • High failure rates in CS1 (35-50% or more) • Fewer applications into CS • “All programming jobs going overseas” • Research results: “Tedious,” “boring,” “lacking creativity,” “asocial” • All of this at a time when we recognize the critical role of IT in our economy, in all jobs

6. Strategy: Ubiquitous Computing Education • Everyone needs computing, andwe should be able to teaching computing that everyone cares about. • Make computing relevant and accessible to students. • Minors, certificates, joint degrees, alternative paths to the major. • At Georgia Tech, we do require every student to take an introductory computing course. • Used to be just one, based on TeachScheme approach (HTDP) • Then came the “Recent Unpleasantness”…

7. Computer science is more important than Calculus • In 1961, Alan Perlis argued that computer science is more important in a liberal education than calculus • Explicitly, he argued that all students should learn to program. • Calculus is about rates, and that’s important to many. • Computer science is about process, which is important to everyone

8. CS1315 Introduction to Media Computation • Focus: Learning programming and CS concepts within the context of media manipulation and creation • Converting images to grayscale and negatives, splicing and reversing sounds, writing programs to generate HTML, creating movies out of Web-accessed content. • Computing for communications, not calculation • Required course at Georgia Tech for Architecture, Management, and Liberal Arts students; Optional for Biology • 121 students in Spring 2003,303 in Fall ’03, and 395 for Spring ‘04 • 2/3 female in Spring 2003 MediaComp

9. Course Objectives • Students will be able to read, understand, and modify programs that achieve useful communication tasks • Not programming from a blank piece of paper • Students will learn what computer science is about, especially data representations, algorithms, encodings, forms of programming. • Students will learn useful computing skills, including graphing and database concepts

10. Python as the programming language • Huge issue • Use in commercial contexts authenticates the choice • IL&M, Google, Nextel, etc. • Minimal syntax • Looks like other programming languages • Potential for transfer

11. Rough overview of Syllabus • Defining and executing functions • Pictures • Psychophysics, data structures, defining functions, for loops, if conditionals • Bitmap vs. vector notations • Sounds • Psychophysics, data structures, defining functions, for loops, if conditionals • Sampled sounds vs. synthesized, MP3 vs. MIDI • Text • Converting between media, generating HTML, database, and networking • A little trees (directories) and hash tables (database) • Movies • Then, Computer Science topics (last 1/3 class)

12. Some Computer Science Topics inter-mixed • We talk about algorithms across media • Sampling a picture (to scale it) is the samealgorithm as sampling a sound (to shift frequency) • Blending two pictures (fading one into the other) and two sounds is the same algorithm. • We talk about representations and mappings (Goedel) • From samples to numbers (and into Excel), through a mapping to pixel colors • We talk about design and debugging • But they mostly don’t hear us

13. Computer Science Topicsas solutions to their problems • “Why is PhotoShop so much faster?” • Compiling vs. interpreting • Machine language and how the computer works • “Writing programs is hard! Are there ways to make it easier? Or at least shorter?” • Object-oriented programming • Functional programming and recursion • “Movie-manipulating programs take a long time to execute. Why? How fast/slow can programs be?” • Algorithmic complexity

14. Does this motivate and engage students? • Homework assignments suggest they were. • Shared on-line in collaborative web space (CoWeb) • Some students reported writing programs outside of class for fun.

15. Does this motivate and engage students?

16. Does this motivate and engage students? Soup Stephen Hawking

17. Latest: Spring 2004 “Well, I looked at last years’ collages, and I certainly can’t be beat.”

18. Relevance through Data-first Computing • Real users come to a user with data that they care about, then they (unwillingly) learn the computer to manipulate their data as they need. • “Media Computation” works like that. • We use pictures of students in class demonstrations. • Students do use their own pictures as starting points for manipulations. • Some students reversed sounds looking for hidden messages. • They started doing this in the second week • How often do students use their second week of CS1 on their own data? • How does that change the students’ relationship to the material?

19. Does the class work to address failure/drop rates? • In Spring 2003, 121 students (2/3 female), 3 drops • Fall 2003, 303 students, 8 drops • Spring 2004, 395 students, 19 drops • 60% of students surveyed at end of Sp2003 course say that they want a second course. • These are non-majors, who have already fulfilled their requirement • We are getting transfers into the CS major.

20. Were Students Motivated and Engaged? Survey responses (Sp03) suggest that students responded well to the context of media manipulation and creation.

21. How did Women Respond to the Course? • Did we make it: • Relevant? • Creative? • Social?

22. How did Women Respond to the Course? • Did we make it: • Relevant? “I dreaded CS, but ALL of the topics thus far have been applicable to my future career (& personal) plans—there isn't anything I don't like about this class!!!” • Creative? • Social?

23. How did Women Respond to the Course? • Did we make it: • Relevant? • Creative? “I just wish I had more time to play around with that and make neat effects. But JES will be on my computer forever, so… that’s the nice thing about this class is that you could go as deep into the homework as you wanted. So, I’d turn it in and then me and my roommate would do more after to see what we could do with it.” • Social?

24. How did Women Respond to the Course? 20% of Spring 2003 students said “Collaboration” was best part of CS1315 • Did we make it: • Relevant? • Creative? • Social? “Actually, I think [collaboration] is one of the best things about this class. My roommate and I abided by all the rules... but we took full advantage of the collaboration. It was more just the ideas bouncing off each other. I don’t think this class would have been as much fun if I wasn’t able to collaborate.”On CoWeb use: “Yes, it’s not just about the class… people talk about anything, it’s a little bit more friendly than just here’s your assignment.”

25. Next steps…A second course and an alternative path • CS1316 Representing structure and behavior to be offered in Spring 2005 • Essentially, data structures in a media context • Lead into how professional animators and computer musicians do their programming • The two courses (CS1315 and CS1316) will be sufficient to take the rest of our traditional CS courses • Defining a CS minor now • Creating BS in Computational Media • Joint with School of Literature, Communication, and Culture

26. Next steps…Moving beyond GT • Versions of Media Computation appearing at other institutions • Gainesville College (2 year in Ga.) has offered the course twice • DePauw, Brandeis (in Scheme), Georgia Perimeter College (in Java), U. California Santa Cruz, and U. Maryland at College Park (in Java) teaching their own versions using our materials.

27. Next steps…Moving beyond Undergrad • Teaching Georgia’s HS teachers how to teach programming starting Summer 2004 • Using a MediaComp approach • After all, teachers are typically non-CS majors… • Helping the State certify CS teachers (for No Child Left Behind Act), and lead to more CS Advanced Placement teachers • Developing two workshops • From no-programming to teaching-programming in 2 weeks • From teaching-programming to teaching-Java-AP in 1 week • 75 teachers this summer (45 from-scratch, 30 to-AP)

28. Image Processing • We’re going to use Python as a kind of pseudo-code. • Goals: • Give you the basic understanding of image processing, including psychophysics of sight, • Identify some interesting examples to use, • Especially those that have the potential for going deep into CS content.

29. We perceive light different from how it actually is • Color is continuous • Visible light is in the wavelengths between 370 and 730 nanometers • That’s 0.00000037 and 0.00000073 meters • But we perceive light with color sensors that peak around 425 nm (blue), 550 nm (green), and 560 nm (red). • Our brain figures out which color is which by figuring out how much of each kind of sensor is responding • One implication: We perceive two kinds of “orange” — one that’s spectral and one that’s red+yellow (hits our color sensors just right) • Dogs and other simpler animals have only two kinds of sensors • They do see color. Just less color.

30. Luminance vs. Color • We perceive borders of things, motion, depth via luminance • Luminance is not the amount of light, but our perception of the amount of light. • We see blue as “darker” than red, even if same amount of light. • Much of our luminance perception is based on comparison to backgrounds, not raw values. Luminance is actually color blind. Completely different part of the brain.

31. Digitizing pictures as bunches of little dots • We digitize pictures into lots of little dots • Enough dots and it looks like a continuous whole to our eye • Our eye has limited resolution • Our background/depth acuity is particulary low • Each picture element is referred to as a pixel • Pixels are picture elements • Each pixel object knows its color • It also knows where it is in its picture

32. Encoding color • Each pixel encodes color at that position in the picture • Lots of encodings for color • Printers use CMYK: Cyan, Magenta, Yellow, and blacK. • Others use HSB for Hue, Saturation, and Brightness (also called HSV for Hue, Saturation, and Brightness • We’ll use the most common for computers • RGB: Red, Green, Blue

33. RGB • In RGB, each color has three component colors: • Amount of redness • Amount of greenness • Amount of blueness • Each does appear as a separate dot on most devices, but our eye blends them. • In most computer-based models of RGB, a single byte (8 bits) is used for each • So a complete RGB color is 24 bits, 8 bits of each

34. Encoding RGB • Each component color (red, green, and blue) is encoded as a single byte • Colors go from (0,0,0) to (255,255,255) • If all three components are the same, the color is in greyscale • (50,50,50) at (2,2) • (0,0,0) (at position (1,2) in example) is black • (255,255,255) is white

35. Is that enough? • We’re representing color in 24 (3 * 8) bits. • That’s 16,777,216 (224) possible colors • We can see more colors than that • But the real limitation is the physical devices: • We can get 16 million colors out of a monitor • But that doesn’t cover all of the colors we can see • Some graphics systems support 32 bits per pixel • May be more pixels for color, or an additional 8 bits to represent 256 levels of translucence

36. Basic Picture Functions • makePicture(filename) creates and returns a picture object, from the JPEG file at the filename • show(picture) displays a picture in a window • We’ll learn functions for manipulating pictures later, like getColor, setColor, and repaint

37. Writing a recipe: Making our own functions • To make a function, use the command def • Then, the name of the function, and the names of the input values between parentheses (“(input1)”) • End the line with a colon (“:”) • The body of the recipe is indented (Hint: Use two spaces) • Your function does NOT exist for JES until you load it

38. Use a loop!Our first picture recipe def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5) Used like this: >>> file="/Users/guzdial/mediasources/barbara.jpg" >>> picture=makePicture(file) >>> show(picture) >>> decreaseRed(picture) >>> repaint(picture)

39. def clearRed(picture): for pixel in getPixels(picture): setRed(pixel,0) def greyscale(picture): for p in getPixels(picture): redness=getRed(p) greenness=getGreen(p) blueness=getBlue(p) luminance=(redness+blueness+greenness)/3 setColor(p, makeColor(luminance,luminance,luminance)) def negative(picture): for px in getPixels(picture): red=getRed(px) green=getGreen(px) blue=getBlue(px) negColor=makeColor(255-red,255-green,255-blue) setColor(px,negColor)

40. Increasing Red What happened here?!? Remember that the limit for redness is 255. If you go beyond 255, all kinds of weird things can happen: Wrap around def increaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*1.2)

41. Clearing Blue def clearBlue(picture): for p in getPixels(picture): setBlue(p,0)

42. Combining into a sunset function • How do we turn this beach scene into a sunset? • What happens at sunset? • At first, I tried increasing the red, but that made things like red specks in the sand REALLY prominent. • That can’t be how it really works • New Theory: As the sun sets, less blue and green is visible, which makes things look more red.

43. A Sunset-generation Function def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.7) value=getGreen(p) setGreen(p,value*0.7)

44. Creating a negative • Let’s think it through • R,G,B go from 0 to 255 • Let’s say Red is 10. That’s very light red. • What’s the opposite? LOTS of Red! • The negative of that would be 245: 255-10 • So, for each pixel, if we negate each color component in creating a new color, we negate the whole picture.

45. Recipe for creating a negative def negative(picture): for px in getPixels(picture): red=getRed(px) green=getGreen(px) blue=getBlue(px) negColor=makeColor( 255-red, 255-green, 255-blue) setColor(px,negColor)

46. Original, negative, negative-negative Introducing information-preserving transformation (as opposed to grayscale, which loses information)

47. Converting to greyscale • We know that if red=green=blue, we get grey • But what value do we set all three to? • What we need is a value representing the darkness of the color, the luminance • There are lots of ways of getting it, but one way that works reasonably well is dirt simple—simply take the average:

48. Converting to grayscale def grayScale(picture): for p in getPixels(picture): intensity = (getRed(p)+getGreen(p)+getBlue(p))/3 setColor(p,makeColor(intensity,intensity,intensity))

49. But that’s not really the best grayscale • In reality, we don’t perceive red, green, and blue as equal in their amount of luminance: How bright (or non-bright) something is. • We tend to see blue as “darker” and red as “brighter” • Even if, physically, the same amount of light is coming off of each • Photoshop’s grayscale is very nice: Very similar to the way that our eye sees it • B&W TV’s are also pretty good

50. Building a better grayscale • We’ll weight red, green, and blue based on how light we perceive them to be, based on laboratory experiments. Starting to talk about more than one way to do the same/similar thing. def greyScaleNew(picture): for px in getPixels(picture): newRed = getRed(px) * 0.299 newGreen = getGreen(px) * 0.587 newBlue = getBlue(px) * 0.114 luminance = newRed+newGreen+newBlue setColor(px,makeColor(luminance,luminance,luminance))