Audio, Images, and Video

Audio, Images, and Video Basic Sound Concepts Fourier Analysis Analog to Digital Conversion Sampling and Quantization Audio Representations Sound Perception Video Representations Visual Perception Analog Transmission of Signals Digital Audio Examples Audio Compression Voice Compression / Coding Basic Imaging Concepts TV Scanning Pattern (NTSC) RGB Color Video Analog, Traditional TV Standards Image Aspect Ratios Θέµατα Συστηµάτων Πολυµέσων 02-1

Basic Sound Concepts produced by vibration of matter sound is not transmitted in a vacuum represented by waveform period (T) frequency (f) f = 1 / T amplitude analog representation Θέµατα Συστηµάτων Πολυµέσων 02-2

Fourier Analysis Fourier Theorem: any (reasonably behaved) periodic function g(t), with period T, can be expressed as where a n, b n, c n are the Fourier coefficients Θέµατα Συστηµάτων Πολυµέσων 02-3

Fourier Analysis Example Assume identical extension to make periodic harmonics: possibly infinite Root Mean Square (RMS) of Fourier coefficients = represents the power (energy) of the signal at a given harmonic (frequency) Θέµατα Συστηµάτων Πολυµέσων 02-4

Low-pass Filter Approximation Θέµατα Συστηµάτων Πολυµέσων 02-5

Analog Transmission of Signals Modulation Carrier Types of Analog Signal Modulation AM FM Phase Modulation Digital Signal Modulation example digital signal: (a) AM: (b) non-zero amplitude represents 1 FM: (c) higher frequency represents 1 Phase Modulation: (d) Θέµατα Συστηµάτων Πολυµέσων 02-6

Analog to Digital Conversion Θέµατα Συστηµάτων Πολυµέσων 02-7

Sampling and Quantization Sampling convert continuous time signal to discrete time signal still analog (continuous amplitude) Nyquist Theorem we can fully reproduce (fully recover without any loss of information) a sampled signal, if the sampling frequency is at least twice the maximum frequency of the signal Quantization convert analog amplitude (at discrete time instants) to discrete values digital signal sampled sine wave 2 bit quantization (fractional) sampled and quantized sine wave Θέµατα Συστηµάτων Πολυµέσων 02-8

DPCM, Delta Modulation and ADPCM Differential PCM: difference from previous value represented Delta Modulation: 1-bit (per sample) difference Adaptive Differential PCM (ADPCM): variable number of bits per sample 56/64 kb/s approximated well with 8 kb/s Θέµατα Συστηµάτων Πολυµέσων 02-9

Understanding Audio Sound is a time varying signal with frequencies (~20hz, ~20khz) Sound composed of many signals added together Low frequencies are vowels and bass High frequencies are consonants Humans perceive sounds over the entire range, but you are more sensitive to low frequencies: 2khz.. 4khz Hearing very dependent on the room and the environment Sounds masked by overlapping sounds Θέµατα Συστηµάτων Πολυµέσων 02-10

Digital Audio Examples Voice 4 khz voice (telephone) 8 khz sampling rate 8 bits per sample (linear quantization) 64 Kb/s digital signal Stereo CD-quality Audio 22 khz CD-quality audio 44 khz sampling rate actually, 44100 samples 16 bit (linear) quantization (Pulse Code Modulation-PCM) 2 channels (stereo) 1411200 b/s = 1.4 Mb/s Θέµατα Συστηµάτων Πολυµέσων 02-11

Audio Representations Optimal sampling frequency is twice the highest frequency to be sampled (Nyquist Theorem) Θέµατα Συστηµάτων Πολυµέσων 02-12

Critical Bands ELECTRO ACOUSTICS Θέµατα Συστηµάτων Πολυµέσων 02-13

Soundfield Θέµατα Συστηµάτων Πολυµέσων 02-14

Impulse Response Θέµατα Συστηµάτων Πολυµέσων 02-15

Audio Noise Masking Θέµατα Συστηµάτων Πολυµέσων 02-16

Audio Compression A bit rate of 1.411 Mbps for stereo music exceeds most access rates, and even 64 Kbps for speech exceeds the access rate for a dial-up modem user PCM encoded speech and music is rarely used in the Internet. Compression techniques are used to reduce the bit rates of the stream; popular compression techniques for speech include GSM (13 Kbps) G.729 (8 Kbps) G.723 (both 6.4 and 5.3 Kbps) a large number of proprietary techniques, including those used by RealNetworks MP3: popular compression technique for near CD-quality stereo music (MPEG layer 3) bit rate for music to 128 or 112 Kbps very little sound degradation an MP3 file can be broken up into pieces, and each piece is still playable this headerless file format allows MP3 music files to be streamed across the Internet the MP3 compression standard is complex: uses psychoacoustic masking, redundancy reduction and bit reservoir buffering Θέµατα Συστηµάτων Πολυµέσων 02-17

Voice Compression / Coding Source Coding source (voice/speech) characteristics exploited vo-coder 64 Kb/s => ~ 8 Kb/s with no loss of perceived quality Speech Recognition / Synthesis speech recognition i.e., speech to text conversion transmission of text speech synthesis i.e., text to speech conversion Θέµατα Συστηµάτων Πολυµέσων 02-18

Audio Quality as a Function of Quantization... bits/sample Θέµατα Συστηµάτων Πολυµέσων 02-19

Audio Quality as a Function of Technology Kb/s Θέµατα Συστηµάτων Πολυµέσων 02-20

Related Topics not Covered Music MIDI: Music Instrument Digital Interface Speech speech generation (text to speech conversion) speech to text conversion speaker-dependent speaker independent speaker recognition/identification Θέµατα Συστηµάτων Πολυµέσων 02-21

Discussion Higher quality Filter input More bits per sample (i.e., 10,12, 16, etc.) More channels (e.g., stereo, quadraphonic, etc.) Digital processing Reshape impulse response to simulate a different room Move perceived location from which sound comes Cover missing samples Mix multiple signals (i.e., conference) Cancel echos Θέµατα Συστηµάτων Πολυµέσων 02-22

Interactivity Time Constraints Maximum time not to hear own voice (echo): 100 ms Maximum allowable round-trip time: 300 ms Θέµατα Συστηµάτων Πολυµέσων 02-23

Importance of Sound Passive viewing (e.g., film, video playback, etc.) Very sensitive to sound breaks Visual channel more important (ask film makers!) Tolerate occasional frame drops Video conferencing Sound channel is more important Visual channel still conveys information Some people report that video teleconference users turn off video Θέµατα Συστηµάτων Πολυµέσων 02-24

Producing High Quality Audio Eliminate background noise Directional mic gives more control Some audio systems will cancel wind noise One microphone per speaker Keep the sound levels balanced Sweeten the sound track with interesting sound effects Θέµατα Συστηµάτων Πολυµέσων 02-25

Audio vs. Video Some people argue that sound is easy and video is hard because of the disparity in data rates Not true, audio is every bit as hard as video -- it is just different! We will learn about audio and video just as we learned about printing with the introduction of desktop publishing Θέµατα Συστηµάτων Πολυµέσων 02-26

Understanding Video Analog video is a continuous signal that drives a CRT Composite video combines luminance and chrominance into one signal Component video separates luminance and chrominance signals Θέµατα Συστηµάτων Πολυµέσων 02-27

Computer Video Controller Standards Pixels (Horizontal x Vertical) x (Color) Depth Color Graphics Adapter (CGA): 320 x 200 x 2 Enhanced Graphics Adapter (EGA): 640 x 350 x 4 Video Graphics Array (VGA): 640 x 480 x 8 Super VGA (SVGA): 800 x 600 (x 16..24) Extended Graphics Array (XGA): 1024 x 768 x 24 Θέµατα Συστηµάτων Πολυµέσων 02-28

Basic Imaging Concepts We see objects because of reflected light Visible wavelengths: 250-780 nm Red: 700 nm Green: 546 nm Blue: 436 nm Monochromatic and non-monochromatic sources Luminance Chrominance Images vs. Graphics Θέµατα Συστηµάτων Πολυµέσων 02-29

PC Bus Transfer Rates Θέµατα Συστηµάτων Πολυµέσων 02-30

Scan and Retrace Θέµατα Συστηµάτων Πολυµέσων 02-31

RGB Color Video Θέµατα Συστηµάτων Πολυµέσων 02-32

Color TV: RGB to YUV RED Θέµατα Συστηµάτων Πολυµέσων 02-33

Analog, Traditional TV Standards NTSC (National Television Standards Committee) 525 lines, interlaced (2:1), 4:3 aspect ratio 60 fields/s (actually, 59.94 fields/s, ~= 30 frames/s) YIQ: Y = 0.30 R + 0.59 G + 0.11 B I = 0.60 R - 0.28 G - 0.32 B Q = 0.21 R - 0.52 G + 0.31 B 6 MHz analog signal 4.2 MHz luminence, 1.5 MHz chrominance, each; VCR: 0.5 MHz chrominance PAL (Phase Alternating Line) & SECAM (SEquentiel Couleur Avec Memoire) 625 lines, interlaced (2:1), 4:3 aspect ratio 50 fields/s (25 frames/s) YUV: Y = 0.30 R + 0.59 G + 0.11 B U = 0.493 (B-Y) = -0.15 R - 0.29 G + 0.44 B V = 0.877 (R-Y) = 0.62 R - 0.52 G + 0.10 B 8 MHz analog signal Θέµατα Συστηµάτων Πολυµέσων 02-34

Down Sampling Color Video Θέµατα Συστηµάτων Πολυµέσων 02-35

Interlacing Θέµατα Συστηµάτων Πολυµέσων 02-36

TV Scanning Pattern (NTSC) Θέµατα Συστηµάτων Πολυµέσων 02-37

Many Video Formats Analog - NTSC, PAL, SECAM Digital - CCIR 601, D1-D5, HDTV Θέµατα Συστηµάτων Πολυµέσων 02-38

NTSC Video (525-lines, 60-fields/s) 525 scan lines repeated 29.97 times per second (i.e., 33.37 ms/frame) Interlaced scan lines divide frame into 2 fields each 262.5 lines (i.e., 16.68 ms/field) 20 lines reserved for control information at the beginning of each field...so only 485 lines of visible data...laserdisc and S-VHS display around 420 lines...normal broadcast TV displays around 320 lines Each line lasts 63.6 µs (10.9 µs blanked) Θέµατα Συστηµάτων Πολυµέσων 02-39

NTSC Signal Composition Θέµατα Συστηµάτων Πολυµέσων 02-40

PAL Video (625-lines, 50-fields/s) 625 scan lines repeated 25 times per second (i.e., 40 ms/frame) Interlaced scan lines divide frame into 2 fields each 312.5 lines (i.e., 20 ms/field) Approximately 20% more lines than NTSC Θέµατα Συστηµάτων Πολυµέσων 02-41

Digital Video Representations Digital Component Video (D1/D5, SMPTE RP125) Maintain separate signals for luminance and color 27 MB/s data rate, either parallel or serial Subsampled color signals 4:2:2 Each pixel is 2 bytes: (CB0,Y0)(CR0,Y1)(CB2,Y2)... Digital Composite Video (D2/D3, SMPTE 244M) 14.3 MB/s data rate, either parallel or serial Subsampled color signals 4:2:2 Each pixel is 1 byte Θέµατα Συστηµάτων Πολυµέσων 02-42

Image Aspect Ratios Θέµατα Συστηµάτων Πολυµέσων 02-43

TV Systems: Spatial Parameters Θέµατα Συστηµάτων Πολυµέσων 02-44

TV Systems: Temporal Parameters Θέµατα Συστηµάτων Πολυµέσων 02-45

Human Perception Vision system accepts 20fps as smooth motion More sensitive to low frequencies Luminance more important than chrominance Noise is noise Vision emphasizes edge detection Detect horizontal lines better than vertical lines Detect verical lines better than diagonal lines Visual masking by significant luminance changes Θέµατα Συστηµάτων Πολυµέσων 02-46

Producing High Quality Video Need high quality camera S-video (SVHS, Hi8mm) better than composite 3 chip better than 1 chip Lights, lights, lights,... Experiment with filters to change apparent colors Shoot scene from different angles and cut between them to create visual stimulation Study film/video techniques Let person exit the scene without moving camera Keep orientation of images correct Θέµατα Συστηµάτων Πολυµέσων 02-47

Jargon/Standards Here are the actual standards G.711 - A-LAW/µ-LAW encodings G.721 - ADPCM at 32 kbs G.723 - ADPCM at 24kbs and 40kbs GSM 06.10-8kz, 1.65kbs (used in Europe) LPC (FIPS-1015) - Linear Predictive Coding (2.4kbs) CELP (FIPS-1016) - Code excited LPC (4.8kbs) The emerging standards common formats 8khz 8-bit µ-law mono 22khz 8-bit unsinged linear mono and stereo 44khz 16-bit signed mono and stereo Θέµατα Συστηµάτων Πολυµέσων 02-48