Ψηφιακή Επεξεργασία Φωνής

Transcript

1 ΕΛΛΗΝΙΚΗ ΔΗΜΟΚΡΑΤΙΑ ΠΑΝΕΠΙΣΤΗΜΙΟ ΚΡΗΤΗΣ Ψηφιακή Επεξεργασία Φωνής Ενότητα 5η: Ημιτονοειδής Ανάλυση και Τροποποίηση Φωνής Στυλιανού Ιωάννης Τμήμα Επιστήμης Υπολογιστών

2 CS578- Speech Signal Processing Lecture 5: Sinusoidal modeling and modifications Yannis Stylianou University of Crete, Computer Science Dept., Multimedia Informatics Lab Univ. of Crete

3 Outline 1 Sinusoidal Speech Model 2 Estimation of Sinewave Parameters Voiced Speech Unvoiced Speech The Analysis System 3 Synthesis Linear Amplitude Interpolation Cubic Phase Interpolation 4 Examples 5 Sound Examples 6 Shape Invariant Time-Scale Modifications The Model Parameters Estimation Synthesis Sound Examples 7 Shape Invariant Pitch Modifications 8 Acknowledgments 9 References

4 Source-Filter[1] Source: where: K(t) u(t) = Re α k (t) exp [jφ k (t)] φ k (t) = k=1 t 0 Ω k (σ)dσ + φ k Filter: h(t, τ) with Fourier Transform (FT): H(t, Ω) = M(t, Ω) exp [jφ(t, Ω)]

5 Source-Filter[1] Source: where: K(t) u(t) = Re α k (t) exp [jφ k (t)] φ k (t) = k=1 t 0 Ω k (σ)dσ + φ k Filter: h(t, τ) with Fourier Transform (FT): H(t, Ω) = M(t, Ω) exp [jφ(t, Ω)]

6 Output speech where: K(t) s(t) = Re A k (t) exp [jθ k (t)] k=1 A k (t) = α k (t)m [t, Ω k (t)] θ k (t) = φ k (t) + Φ [t, Ω k (t)] = t 0 Ω k (σ)dσ + Φ [t, Ω k (t)] + φ k

7 Outline 1 Sinusoidal Speech Model 2 Estimation of Sinewave Parameters Voiced Speech Unvoiced Speech The Analysis System 3 Synthesis Linear Amplitude Interpolation Cubic Phase Interpolation 4 Examples 5 Sound Examples 6 Shape Invariant Time-Scale Modifications The Model Parameters Estimation Synthesis Sound Examples 7 Shape Invariant Pitch Modifications 8 Acknowledgments 9 References

8 Frame-by-Frame analysis

9 Stationarity Assumption We assume stationarity inside the analysis window: A l k (t) = Al k Ω l k (t) = Ωl k which leads to: and to: θ l k (t) = Ωl k (t t l) + θ l k K l ] s(t) = A l k exp (jθl k [jω ) exp l k (t t l) k=1 t l T 2 t t l + T 2

10 Stationarity Assumption We assume stationarity inside the analysis window: A l k (t) = Al k Ω l k (t) = Ωl k which leads to: and to: θ l k (t) = Ωl k (t t l) + θ l k K l ] s(t) = A l k exp (jθl k [jω ) exp l k (t t l) k=1 t l T 2 t t l + T 2

11 Stationarity Assumption We assume stationarity inside the analysis window: A l k (t) = Al k Ω l k (t) = Ωl k which leads to: and to: θ l k (t) = Ωl k (t t l) + θ l k K l ] s(t) = A l k exp (jθl k [jω ) exp l k (t t l) k=1 t l T 2 t t l + T 2

12 Discrete-Time Formulation Steps to discrete time formula: Time shift: t = t t l Convert to discrete time: s[n] = A l k exp (jθl k ) exp (jωl k n) N w 1 2 K l k=1 n N w 1 2

13 Mean-Squared Error Given the original measured waveform, y[n] and the synthetic speech waveform, s[n], estimate the unknown parameters A l k, ωl k, and θk l by minimizing the MSE criterion: ɛ l = which can be written as: ɛ l = n=(n w 1)/2 n= (N w 1)/2 n=(n w 1)/2 n= (N w 1)/2 which can be reduced further to: ɛ l = y[n] s[n] 2 K l ( Y ) y[n] 2 + N w (ω l k ) γl k 2 Y (ωk l ) 2 n=(n w 1)/2 n= (N w 1)/2 k=1 y[n] 2 N w Y (ωk l ) 2 K l k=1

14 Mean-Squared Error Given the original measured waveform, y[n] and the synthetic speech waveform, s[n], estimate the unknown parameters A l k, ωl k, and θk l by minimizing the MSE criterion: ɛ l = which can be written as: ɛ l = n=(n w 1)/2 n= (N w 1)/2 n=(n w 1)/2 n= (N w 1)/2 which can be reduced further to: ɛ l = y[n] s[n] 2 K l ( Y ) y[n] 2 + N w (ω l k ) γl k 2 Y (ωk l ) 2 n=(n w 1)/2 n= (N w 1)/2 k=1 y[n] 2 N w Y (ωk l ) 2 K l k=1

15 Mean-Squared Error Given the original measured waveform, y[n] and the synthetic speech waveform, s[n], estimate the unknown parameters A l k, ωl k, and θk l by minimizing the MSE criterion: ɛ l = which can be written as: ɛ l = n=(n w 1)/2 n= (N w 1)/2 n=(n w 1)/2 n= (N w 1)/2 which can be reduced further to: ɛ l = y[n] s[n] 2 K l ( Y ) y[n] 2 + N w (ω l k ) γl k 2 Y (ωk l ) 2 n=(n w 1)/2 n= (N w 1)/2 k=1 y[n] 2 N w Y (ωk l ) 2 K l k=1

16 Karhunen-Loève Expansion Karhunen-Loève expansion allows constructing a random process from harmonic sinusoids with uncorrelated complex amplitudes. Estimated power spectrum should not vary too much over consecutive frequencies. Following the above necessary constraints, for unvoiced speech, and for a window width to be at least 20ms, an 100 Hz harmonic structure provides good results.

17 Karhunen-Loève Expansion Karhunen-Loève expansion allows constructing a random process from harmonic sinusoids with uncorrelated complex amplitudes. Estimated power spectrum should not vary too much over consecutive frequencies. Following the above necessary constraints, for unvoiced speech, and for a window width to be at least 20ms, an 100 Hz harmonic structure provides good results.

18 Karhunen-Loève Expansion Karhunen-Loève expansion allows constructing a random process from harmonic sinusoids with uncorrelated complex amplitudes. Estimated power spectrum should not vary too much over consecutive frequencies. Following the above necessary constraints, for unvoiced speech, and for a window width to be at least 20ms, an 100 Hz harmonic structure provides good results.

19 Karhunen-Loève Expansion Karhunen-Loève expansion allows constructing a random process from harmonic sinusoids with uncorrelated complex amplitudes. Estimated power spectrum should not vary too much over consecutive frequencies. Following the above necessary constraints, for unvoiced speech, and for a window width to be at least 20ms, an 100 Hz harmonic structure provides good results.

20 Example

21 Implementation Window width be 2.5 times the average pitch period or 20 ms, whichever is larger. Use Hamming window, normalized to one: n= w[n] = 1 Use zero padding to get enough samples of the underlying spectrum (i.e., 1024-point FFT) Remove linear phase offset Refine your frequency estimates

22 Implementation Window width be 2.5 times the average pitch period or 20 ms, whichever is larger. Use Hamming window, normalized to one: n= w[n] = 1 Use zero padding to get enough samples of the underlying spectrum (i.e., 1024-point FFT) Remove linear phase offset Refine your frequency estimates

23 Implementation Window width be 2.5 times the average pitch period or 20 ms, whichever is larger. Use Hamming window, normalized to one: n= w[n] = 1 Use zero padding to get enough samples of the underlying spectrum (i.e., 1024-point FFT) Remove linear phase offset Refine your frequency estimates

24 Implementation Window width be 2.5 times the average pitch period or 20 ms, whichever is larger. Use Hamming window, normalized to one: n= w[n] = 1 Use zero padding to get enough samples of the underlying spectrum (i.e., 1024-point FFT) Remove linear phase offset Refine your frequency estimates

25 Implementation Window width be 2.5 times the average pitch period or 20 ms, whichever is larger. Use Hamming window, normalized to one: n= w[n] = 1 Use zero padding to get enough samples of the underlying spectrum (i.e., 1024-point FFT) Remove linear phase offset Refine your frequency estimates

26 Showing the process...

27 Block diagram of the Analysis System

28 Outline 1 Sinusoidal Speech Model 2 Estimation of Sinewave Parameters Voiced Speech Unvoiced Speech The Analysis System 3 Synthesis Linear Amplitude Interpolation Cubic Phase Interpolation 4 Examples 5 Sound Examples 6 Shape Invariant Time-Scale Modifications The Model Parameters Estimation Synthesis Sound Examples 7 Shape Invariant Pitch Modifications 8 Acknowledgments 9 References

29 Problem of frequency matching

30 Frame-to-Frame Peak Matching

31 The birth/death process

32 A birth/death process in speech

33 Why not... Why not to estimate the original speech waveform on the lth frame, directly as: s[n] = A l k cos (nωl k + θl k ), n = 0, 1, 2,, L 1 K l k=1

34 A simple solution: OLA

35 Amplitude Interpolation Linear Interpolation: ( ( n ) A l k [n] = Al k + A l+1 k Ak) l L n = 0, 1, 2,, L 1

36 Phase wrapped

37 Cubic Phase model θ(t) = ζ + γt + αt 2 + βt 3

38 About the phase derivative Assuming that vocal tract is slowly varying, and since: θ(t) = t 0 Ω(σ)dσ + φ + Φ[t, Ω(t)] θ(t) Ω(t) So: θ l Ω l θ l+1 Ω l+1

39 About the phase derivative Assuming that vocal tract is slowly varying, and since: θ(t) = t 0 Ω(σ)dσ + φ + Φ[t, Ω(t)] θ(t) Ω(t) So: θ l Ω l θ l+1 Ω l+1

40 About the phase derivative Assuming that vocal tract is slowly varying, and since: θ(t) = t 0 Ω(σ)dσ + φ + Φ[t, Ω(t)] θ(t) Ω(t) So: θ l Ω l θ l+1 Ω l+1

41 Four constraints for phase polynomial There are four constraints θ(0) = θ l θ(0) = Ω l θ(t ) = θ l+1 + 2πM θ(t ) = Ω l+1 and... five unknowns (don t forget M) We need one more constraint!

42 Four constraints for phase polynomial There are four constraints θ(0) = θ l θ(0) = Ω l θ(t ) = θ l+1 + 2πM θ(t ) = Ω l+1 and... five unknowns (don t forget M) We need one more constraint!

43 Four constraints for phase polynomial There are four constraints θ(0) = θ l θ(0) = Ω l θ(t ) = θ l+1 + 2πM θ(t ) = Ω l+1 and... five unknowns (don t forget M) We need one more constraint!

44 Four constraints for phase polynomial There are four constraints θ(0) = θ l θ(0) = Ω l θ(t ) = θ l+1 + 2πM θ(t ) = Ω l+1 and... five unknowns (don t forget M) We need one more constraint!

45 How to choose M

46 Estimating M Find M that minimizes the criterion: f (M) = Using continuous variable: x = 1 2π T M is the nearest integer to x 0 ] 2 [ θ(t; M) dt [ (θ l + Ω l T θ l+1 ) + (Ω l+1 Ωl) T 2 ]

47 Estimating M Find M that minimizes the criterion: f (M) = Using continuous variable: x = 1 2π T M is the nearest integer to x 0 ] 2 [ θ(t; M) dt [ (θ l + Ω l T θ l+1 ) + (Ω l+1 Ωl) T 2 ]

48 Estimating M Find M that minimizes the criterion: f (M) = Using continuous variable: x = 1 2π T M is the nearest integer to x 0 ] 2 [ θ(t; M) dt [ (θ l + Ω l T θ l+1 ) + (Ω l+1 Ωl) T 2 ]

49 Block diagram of the Synthesis System

50 Outline 1 Sinusoidal Speech Model 2 Estimation of Sinewave Parameters Voiced Speech Unvoiced Speech The Analysis System 3 Synthesis Linear Amplitude Interpolation Cubic Phase Interpolation 4 Examples 5 Sound Examples 6 Shape Invariant Time-Scale Modifications The Model Parameters Estimation Synthesis Sound Examples 7 Shape Invariant Pitch Modifications 8 Acknowledgments 9 References

51 Reconstruction Example

52 Reconstruction Example

53 Magnitude-only Reconstruction Example

54 Outline 1 Sinusoidal Speech Model 2 Estimation of Sinewave Parameters Voiced Speech Unvoiced Speech The Analysis System 3 Synthesis Linear Amplitude Interpolation Cubic Phase Interpolation 4 Examples 5 Sound Examples 6 Shape Invariant Time-Scale Modifications The Model Parameters Estimation Synthesis Sound Examples 7 Shape Invariant Pitch Modifications 8 Acknowledgments 9 References

55 Sound Examples Male Female Male Female Original Mixed Min Zero

56 Outline 1 Sinusoidal Speech Model 2 Estimation of Sinewave Parameters Voiced Speech Unvoiced Speech The Analysis System 3 Synthesis Linear Amplitude Interpolation Cubic Phase Interpolation 4 Examples 5 Sound Examples 6 Shape Invariant Time-Scale Modifications The Model Parameters Estimation Synthesis Sound Examples 7 Shape Invariant Pitch Modifications 8 Acknowledgments 9 References

57 Excitation model We have seen that: where: K(t) u(t) = α k (t) exp [jφ k (t)] φ k (t) = k=1 t 0 Ω k (σ)dσ + φ k Assuming voiced speech and constant frequency in the analysis window, then: K(t) u(t) = α k (t) exp [j(t t 0 )Ω k ] t [0, T ] k=1

58 Excitation model We have seen that: where: K(t) u(t) = α k (t) exp [jφ k (t)] φ k (t) = k=1 t 0 Ω k (σ)dσ + φ k Assuming voiced speech and constant frequency in the analysis window, then: K(t) u(t) = α k (t) exp [j(t t 0 )Ω k ] t [0, T ] k=1

59 Speech model[2] Then: where: K(t) s[n] = A k (t) cos [θ k (t)] k=1 A k (t) = α k (t)m k (t) θ k (t) = φ k (t) + Φ k (t) Therefore: Φ k (t) = θ k (t) (t t 0 )Ω k

60 Uniform time-scale, by ρ Let s t represent the original articulation rate and t the transformed rate: t = ρ t Given the source/filter model: System parameters are time-scaled Excitation parameters (phase) are scaled in such a way to maintain fundamental frequency.

61 Uniform time-scale, by ρ Let s t represent the original articulation rate and t the transformed rate: t = ρ t Given the source/filter model: System parameters are time-scaled Excitation parameters (phase) are scaled in such a way to maintain fundamental frequency.

62 Onset-time model for time-scale

63 Excitation function in t Time-scaled pitch period: Modified excitation function ũ(t ) = P(t ) = P(t ρ 1 ) K(t) k=1 [ ] α k (t ) exp j φ k (t ) where: and φ k (t ) = (t ρ 1 t 0)Ω k α k (t ) = α k (t ρ 1 )

64 System function parameters in t M k (t ) = M k (t ρ 1 ) Φ k (t ) = Φ k (t ρ 1 )

65 Waveform in t where K(t) s(t ) = k=1 [ ] Ã k (t ) exp j θ k (t ) Ã k (t ) = α k (t ) M k (t ) θ k (t ) = φ k (t ) + Φ k (t )

66 Onset times estimation

67 Estimating System Phase Let s assume that the onset time n o (l) for the l th frame is known, then: φ l k = ˆn o(l)ω l k where ˆn o (l) = n o (l) ll. Then, the system phase is estimated as: Φ l k = θl k φl k

68 Estimating System Phase Let s assume that the onset time n o (l) for the l th frame is known, then: φ l k = ˆn o(l)ω l k where ˆn o (l) = n o (l) ll. Then, the system phase is estimated as: Φ l k = θl k φl k

69 Estimating Excitation Phase Let s assume we know the onset time in the previous frame l 1, then the current onset time in t, is given by: n o(l) = n o(l 1) + J P l and then: φ l k = (n o(l) ll )ω l k where L = ρl

70 Synthesis Synthesis is performed in the same way as if no modification is applied: Linear interpolation for amplitudes Cubic interpolation for phases

71 Block diagram for Analysis/Synthesis for Time-Scale modification

72 Example of Time-scale modification

73 Sound Examples Male Female Trumpet Orig Orig 1.25

74 Outline 1 Sinusoidal Speech Model 2 Estimation of Sinewave Parameters Voiced Speech Unvoiced Speech The Analysis System 3 Synthesis Linear Amplitude Interpolation Cubic Phase Interpolation 4 Examples 5 Sound Examples 6 Shape Invariant Time-Scale Modifications The Model Parameters Estimation Synthesis Sound Examples 7 Shape Invariant Pitch Modifications 8 Acknowledgments 9 References

75 Reading paper Paper: T. F. Quatieri and R. J. McAulay: Shape Invariant Time-Scale and Pitch Modification of Speech IEEE Trans. Acoust., Speech, Signal Processing, Vol.40, No.3, pp , March 1992

76 Outline 1 Sinusoidal Speech Model 2 Estimation of Sinewave Parameters Voiced Speech Unvoiced Speech The Analysis System 3 Synthesis Linear Amplitude Interpolation Cubic Phase Interpolation 4 Examples 5 Sound Examples 6 Shape Invariant Time-Scale Modifications The Model Parameters Estimation Synthesis Sound Examples 7 Shape Invariant Pitch Modifications 8 Acknowledgments 9 References

77 Acknowledgments Most, if not all, figures in this lecture are coming from the book: T. F. Quatieri: Discrete-Time Speech Signal Processing, principles and practice 2002, Prentice Hall and have been used after permission from Prentice Hall

78 Outline 1 Sinusoidal Speech Model 2 Estimation of Sinewave Parameters Voiced Speech Unvoiced Speech The Analysis System 3 Synthesis Linear Amplitude Interpolation Cubic Phase Interpolation 4 Examples 5 Sound Examples 6 Shape Invariant Time-Scale Modifications The Model Parameters Estimation Synthesis Sound Examples 7 Shape Invariant Pitch Modifications 8 Acknowledgments 9 References

79 R. J. McAulay and T. F. Quatieri, Speech analysis/synthesis based on a sinusoidal representation, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-34, pp , Aug T. F. Quatieri and R. J. McAulay, Shape Invariant Time-Scale and Pitch Modification of Speech, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-40, pp , March 1992.

80

81 Τέλος Ενότητας

82 Χρηματοδότηση Το παρόν εκπαιδευτικό υλικό έχει αναπτυχθεί στα πλαίσια του εκπαιδευτικού έργου του διδάσκοντα. Το έργο «Ανοικτά Ακαδημαϊκά Μαθήματα στο Πανεπιστήμιο Κρήτης» έχει χρηματοδοτήσει μόνο τη αναδιαμόρφωση του εκπαιδευτικού υλικού. Το έργο υλοποιείται στο πλαίσιο του Επιχειρησιακού Προγράμματος «Εκπαίδευση και Δια Βίου Μάθηση» και συγχρηματοδοτείται από την Ευρωπαϊκή Ένωση (Ευρωπαϊκό Κοινωνικό Ταμείο) και από εθνικούς πόρους. 3

83 Σημειώματα

84 Σημείωμα αδειοδότησης Το παρόν υλικό διατίθεται με τους όρους της άδειας χρήσης Creative Commons Αναφορά, Μη Εμπορική Χρήση, Όχι Παράγωγο Έργο 4.0 [1] ή μεταγενέστερη, Διεθνής Έκδοση. Εξαιρούνται τα αυτοτελή έργα τρίτων π.χ. φωτογραφίες, διαγράμματα κ.λ.π., τα οποία εμπεριέχονται σε αυτό και τα οποία αναφέρονται μαζί με τους όρους χρήσης τους στο «Σημείωμα Χρήσης Έργων Τρίτων». [1] Ως Μη Εμπορική ορίζεται η χρήση: που δεν περιλαμβάνει άμεσο ή έμμεσο οικονομικό όφελος από την χρήση του έργου, για το διανομέα του έργου και αδειοδόχο που δεν περιλαμβάνει οικονομική συναλλαγή ως προϋπόθεση για τη χρήση ή πρόσβαση στο έργο που δεν προσπορίζει στο διανομέα του έργου και αδειοδόχο έμμεσο οικονομικό όφελος (π.χ. διαφημίσεις) από την προβολή του έργου σε διαδικτυακό τόπο Ο δικαιούχος μπορεί να παρέχει στον αδειοδόχο ξεχωριστή άδεια να χρησιμοποιεί το έργο για εμπορική χρήση, εφόσον αυτό του ζητηθεί.. 5

85 Σημείωμα Αναφοράς Copyright Πανεπιστήμιο Κρήτης, Στυλιανού Ιωάννης. «Ψηφιακή Επεξεργασία Φωνής. Ημιτονοειδής Ανάλυση και Τροποποίηση Φωνής». Έκδοση: 1.0. Ηράκλειο/Ρέθυμνο Διαθέσιμο από τη δικτυακή διεύθυνση:

86 Διατήρηση Σημειωμάτων Οποιαδήποτε αναπαραγωγή ή διασκευή του υλικού θα πρέπει να συμπεριλαμβάνει: το Σημείωμα Αναφοράς το Σημείωμα Αδειοδότησης τη δήλωση Διατήρησης Σημειωμάτων το Σημείωμα Χρήσης Έργων Τρίτων (εφόσον υπάρχει) μαζί με τους συνοδευόμενους υπερσυνδέσμους.

87 Σημείωμα Χρήσης Έργων Τρίτων Το Έργο αυτό κάνει χρήση των ακόλουθων έργων: Εικόνες/Σχήματα/Διαγράμματα/Φωτογραφίες Εικόνες/σχήματα/διαγράμματα/φωτογραφίες που περιέχονται σε αυτό το αρχείο προέρχονται από το βιβλίο: Τίτλος: Discrete-time Speech Signal Processing: Principles and Practice Prentice-Hall signal processing series, ISSN Συγγραφέας: Thomas F. Quatieri Εκδότης: Prentice Hall PTR, 2002 ISBN: X, Μέγεθος: 781 σελίδες και αναπαράγονται μετά από άδεια του εκδότη.