MP 1: Audio/ Video Recorder and Player

MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: KlaraNahrstedt February 7th, 2012

Learning Goals • Capturing a video and audio using a webcam • Storing the audio and video data in PC using compression • Playing a video from the stored file with fast forward, rewind, pause and play functionalities • Playing an audio from the stored file • Comparing the media frames across different compression

Covered Aspects of Multimedia Audio/Video Presentation Playback Image/Video Capture Audio/Video Perception/ Playback Image/Video Information Representation Transmission Transmission Compression Processing Audio Capture Media Server Storage Audio Information Representation A/V Playback

System Architecture Image/Video Capture Audio/Video Perception/ Playback Image/Video Information Representation Compression Processing Audio Capture Audio Information Representation

System Modules and Data Flow: Recording Frame rate, resolution • You need to create separate threads • Muxer is optional. You may need it to run the video file using standard player type Thread Thread Capture Filter Encoder Store Webcam Player Thread … … … … Encode Muxer Store … … Encode

System Modules and Data Flow: Playback • Display the video on your GUI and place the audio data to the sound card • Demuxer is optional. You may need it run standard video files from Internet Media File type Capture Decode Player Rate control … … Decode … … … … Capture Demuxer Decode

System Modules and Data Flow: Monitoring in Recording • Store the monitoring output to a file • [Optional] You may draw chart showing [time vs. compression ratio (r)], [time vs. compression time (t)] and [time vs. frame size (s)] [bonus point] • X-axis: Time in 5 second interval • Y-axis: Average values (of r, s or t) in last 5 seconds Thread Monitoring Component Monitoring Point Thread Thread Capture Filter Encode Store Webcam Player Thread

System Modules and Data Flow: Monitoring in Playback • Store the monitoring output to a file • [Optional] You may draw plot showing [time vs. decompression time (d)] [bonus point] [Hint: Try JFreeChart library] • X-axis: Time in 5 second interval • Y-axis: Average values (of d) in last 5 seconds Monitoring Point Thread Media File Monitoring Component Thread Thread Capture Decode Player

gstreamer Architecture • Goal: create a multimedia application using gstreamer • gstreamer uses 3rd party plugins for processing (over 150) Multimedia Application gstreamer Core Framework Encoder Decoder Muxer Demuxer Sink source 3rd Party plugins

How does it work (1) ? • You need to create element for each system modules • Elements are equipped with Pads: source and sink pads Source Sink gstreamerelements • The number of source and sink pad varies depending on the element type • You can add/ create additional sink or source pad

Creating gstreamer elements Sink Src Src Src Sink Sink • Java code example to create an element • Elements element_name = ElementFactory.make (“plugin name”, “your defined name”) • Java code example to set element property • element_name.set(“property”, “value”) Sink source Src Capture filter Filter encoder Encode muxer Sink Muxer sink Store

How does it work (2) ? Src Sink • You need to link the elements • Linking define whom is receiving and sending data to whom • This is similar to linking the system modules Capture Filter Src Media Source Media Filter

gstreamer-java pipeline Sink Src Src Src Src Sink Sink Sink • Java example code for creating, and linking a pipeline • Pipeline pipe= new Pipeline(“test”); • pipe.addMany(source, filter,encoder, muxer,sink); • Element.linkMany(source, filter,encoder, muxer,sink); • pipe.play(); Webcam Capture Filter Encode Store Media Encoder Media Muxer Media Filter Media Source Media Sink

How to create multiple pipelining Src Sink Sink Src Src Src Sink Sink • Several pipelines from the same source or towards same sink • Example: recording and playback at the same time Sink Thread Thread Capture Filter Encoder Store Webcam Player Thread Media Encoder Media Muxer Media Sink Media Filter Media Source Media Sink Src

How to create multiple pipelining Src Sink Sink Src Src Src Src Src Src Sink Sink Sink Sink Sink • You can create additional pads • You can use tee elements • Use queue elements after tee Sink muxer Media Sink encoder queue1 tee filter source Src queue2 Sink

How to redirect frames to application Src Src Src Src Sink Sink Sink • How to deliver a frame to your application • Use appsink element Emit Signals Media Encoder AppSink Media Filter Media Source • Java example code for getting frames from AppSink • AppSinkappsink = (AppSink) ElementFactory.make(“appsink”, null); • appsink.set(“emit-signal”, true); • appsink.setSync(false);

Some gstreamer plugins • Video webcam source: v4l2src • Audio webcam source:alsasrc • Video or Audio file source: filesrc • Video Encoder: ffenc_mpeg4 (mpeg4), jpegenc (mjpeg), … • Video Decoder:ffdec_mpeg4 (mpeg4), jpegdec(mjpeg), … • Audio Encoder: vorbisenc, alawenc, mulawenc • Audio Decoder: vorbisdec, alawdec, mulawdec • Muxer [optional]:avimux (avi), matroskamux(mkv), … • Demuxer [optional]: avidemux (avi), • Audio sink: alsasink

Evaluations Required Points: 100, Optional Points: 20 Points are will be considered based on live demo and interview performance

Environment and Equipment • Choose what ever language you like, Gstreamer is compatible with most popular languages

Environment and Equipment • If you choose to use EWS.. • group directories • /team/cs414/G#(# is your group number) • If you choose to use your own machines.. • Windows/Mac • Android/iOS • Feel free to use your own camera/mic, or you can borrow two logitech cameras (with mic embedded) from Engineering IT (Barb Leisner)

MP 1: Audio/ Video Recorder and Player