ABSTRACT

At present, although the human speech separation has achieved fruitful results, it is not ideal for the separation of singing and accompaniment. Based on low-rank and sparse optimization theory, in this paper, we propose a new singing voice separation algorithm called Low-rank, Sparse Representation with pre-learned dictionaries and side Information (LSRi). The algorithm incorporates both the vocal and instrumental spectrograms as sparse matrix and low-rank matrix, meanwhile combines pre-learning dictionary and the reconstructed voice spectrogram form the annotation. Evaluations on the iKala dataset show that the proposed methods are effective and efficient for singing voice separation.

Keywords:

Singing Voice Separation, Low-Rank and Sparse, Dictionary Learning

1. Introduction

Separating singing voice from music recording is very useful in many applications, such as music information retrieval, singer identification and lyrics recognition and alignment [1] . Although the human auditory system can easily distinguish the vocal and instrumental of music recording, it is extremely difficult for computer systems. In this context, researchers are increasingly concerned with the mining of music information. Many algorithms have been proposed to separate singing voice from music recording.

Robust Principal Component Analysis (RPCA) is a matrix factorization algorithm for solving underlying low-rank and sparse matrices [2] . Suppose we are given a large data matrix M, and know that it may be decomposed as $X=A+E$ , where A is a low-rank matrix and E is a sparse matrix. Based on RPCA, Huang et al. [3] have separated singing-voice from music accompaniment. They assumed that the repetitive music accompaniment lies in a low-rank subspace, while the singing voices can be regarded as sparse within songs. The main drawback to this approach is that it is completely unsupervised, just based on the particular properties of each individual components to guide the decomposition. After, Yu et al. [4] utilized any pre-learned information and pre-learned universal voice and music dictionaries from isolated singing voice and background music training data. They proposed Low-rank and Sparse representation with Pre-learned Dictionaries (LSPD) for singing voice separation. Chan et al. [5] proposed a modified RPCA algorithm. This work represented one of the first attempts to incorporate vocal activity information into the RPCA algorithm, then the vocal activity detection was widely studied [6] [7] . Chan et al. [8] proposed to separate singing voice by group-sparse representation with the idea of pitch annotations separation.

In this paper, we present a model named Low-rank, Sparse representation with pre-learned dictionaries and side information (LSRi) under the ADMM framework. First, we pre-learn voice and music dictionaries from isolated singing voice and background music training data, respectively. Then, we use a sparse spectrogram and a low-rank spectrogram to model the singing voice and the background music, respectively. Outside, a residual term is added to capture the components that are not well modeled by either the sparse or the low-rank term. Finally, we combine the reconstructed voice spectrogram from the vocal annotation. Evaluations on the iKala dataset [9] show its better performance than comparison methods.

The rest of this paper is organized as follows. The overview of the music analysis model is presented in Section 2. The description of theoretical knowledge and experimental results are presented in Section 3. Final Section concludes this work.

2. The Proposed Method

Before we come up with our method, let’s review the Low-rank and Sparse representation with Pre-learned Dictionaries (LSPD) method [4] ,

$\begin{array}{l}\underset{Z_1,Z_2}{\min }{\Vert Z_1\Vert }_{*}+{\lambda }_1{\Vert Z_2\Vert }_1+{\lambda }_2{\Vert E\Vert }_1\\ s.t.X=D_1{Z}_1+D_2{Z}_2+E\end{array}$ (1)

where X is the input spectrogram, $D_1\in R^{m\times k_1}$ is a pre-learned dictionary of the music accompaniment, $D_2\in R^{m\times k_2}$ is a pre-learned dictionary of the singing voice, $D_1{Z}_1$ is the separated instrumentals, $D_2{Z}_2$ is the separated voice. E denotes the residual part. ${\lambda }_1,{\lambda }_2$ are two weighting parameters for balancing the different regularization terms in this model.

Compared with the unsupervised RPCA algorithm, the LSPD algorithm adds pre-learning dictionary information and improves the separation quality. To further improve the separation quality of singing voice and music accompaniment, we proposed Low-rank, Sparse Representation with pre-learned dictionaries and side Information (LSRi).

In our model, we considered more prior information i.e., the reconstructed voice spectrogram from the annotation. Model as follows,

$\begin{array}{l}\underset{Z_1,Z_2}{\min }{\Vert Z_1\Vert }_{*}+{\lambda }_1{\Vert Z_2\Vert }_1+{\lambda }_2{\Vert E\Vert }_1+\frac{\gamma }{2}{\Vert D_2{Z}_2-E_0\Vert }_F^2\\ s.t.\mathrm{<mi>\; </mi>}X=D_1{Z}_1+D_2{Z}_2+E\end{array}$ (2)

Here all parameters in model 2 are in accordance with model 1, and $E_0$ denotes the reconstructed voice spectrogram from the annotation. ${\Vert \cdot \Vert }_F$ denotes the Frobenius norm. In the following, we also use the ADMM algorithm [10] to solve the optimization problem, by introducing two auxiliary variables $J_1$ and $J_2$ as well as three equality constraints,

$\begin{array}{l}\underset{Z_1,Z_2,J_1,J_2}{\min }{\Vert J_1\Vert }_{*}+{\lambda }_1{\Vert J_2\Vert }_1+{\lambda }_2{\Vert E\Vert }_1+\frac{\gamma }{2}{\Vert D_2{Z}_2-E_0\Vert }_F^2\\ s.t\mathrm{.\; <mi>\; </mi>\; <mi>\; </mi>}X=D_1{Z}_1+D_2{Z}_2+E,Z_1=J_1,Z_2=J_2\end{array}$ (3)

The unconstrained augmented Lagrangian $\mathcal{L}$ is given by

$\begin{array}{c}\mathcal{L}={\Vert J_1^{\text{T}}\Vert }_{*}+{\lambda }_1{\Vert J_2\Vert }_1+{\lambda }_2{\Vert E\Vert }_1+\frac{\gamma }{2}{\Vert D_2{Z}_2-E_0\Vert }_F^2\\ +\langle Y_1,X-D_1{Z}_1-D_2{Z}_2-E\rangle +\langle Y_2,Z_1-J_1\rangle +\langle Y_3,Z_2-J_2\rangle \\ +\frac{\mu }{2}\left({\Vert X-D_1{Z}_1-D_2{Z}_2-E\Vert }_F^2+{\Vert Z_1-J_1\Vert }_F^2+{\Vert Z_2-J_2\Vert }_F^2\right)\end{array}$ (4)

where $Y_1,Y_2,Y_3$ are the Lagrange multipliers. We then iteratively update the solutions for $J_1,Z_1,J_2$ and $Z_2$ .

1) Update $J_1$ :

$J_1=\arg \underset{J_1}{\min }{\Vert J_1\Vert }_{*}+\frac{\mu }{2}{\Vert J_1-\left(Z_1+{\mu }^{-1}{Y}_2\right)\Vert }_F^2=U{S}_{\frac{1}{\mu }}\left[\Sigma \right]V^{\text{T}}$ (5)

where $U\Sigma V=svd\left(Z_1+{\mu }^{-1}{Y}_2\right).$

2) Update $Z_1$ :

$\frac{\partial \mathcal{L}}{\partial Z_1}=-D_1^{\text{T}}{Y}_1+Y_2-\mu D_1^{\text{T}}\left(X-D_1{Z}_1-D_2{Z}_2-E\right)+\mu \left(Z_1-J_1\right)$ (6)

setting $\frac{\partial \mathcal{L}}{\partial Z_1}=0$ , we have

$Z_1={\left(D_1^{\text{T}}{D}_1+I\right)}^{-1}\left(D_1^{\text{T}}\left(X-D_2{Z}_2-E+{\mu }^{-1}{Y}_1\right)-{\mu }^{-1}{Y}_2+J_1\right)$ (7)

3) Update $J_2$ :

$J_2=\arg \underset{J_2}{\min }{\lambda }_1{\Vert J_2\Vert }_1+\frac{\mu }{2}{\Vert J_1-\left(Z_1+{\mu }^{-1}{Y}_3\right)\Vert }_F^2$ (8)

that can be solve by the soft-threshold operator

$J_2=S_{\frac{{\lambda }_1}{\mu }}\left(Z_2+{\mu }^{-1}{Y}_3\right)$ (9)

since the spectrogram is non-negative

$J_2=\max \left\{S_{\frac{{\lambda }_1}{\mu }}\left(Z_2+{\mu }^{-1}{Y}_3\right),0\right\}$ (10)

where 0 is an all zero matrix of the size as $J_2$ .

4) Update $Z_2$ :

$\frac{\partial \mathcal{L}}{\partial Z_2}=\gamma D_2^{\text{T}}\left(D_2{Z}_2-E_0\right)-D_2^{\text{T}}{Y}_1+Y_3-\mu D_2^{\text{T}}\left(X-D_1{Z}_1-D_2{Z}_2-E\right)+\mu \left(Z_2-J_2\right)$ (11)

setting $\frac{\partial \mathcal{L}}{\partial Z_2}=0$ , we have

$\begin{array}{c}{Z}_2={\left(\left(\gamma +\mu \right)D_2^{\text{T}}{D}_2+\mu I\right)}^{-1}\left(\gamma D_2^{\text{T}}{E}_0+D_2^{\text{T}}{Y}_1-Y_3+\mu D_2^{\text{T}}\left(X-D_1{Z}_1-E\right)+\mu J_2\right)\\ ={\left(\left(\frac{\gamma }{\mu }+1\right)D_2^{\text{T}}{D}_2+I\right)}^{-1}\left(D_2^{\text{T}}\left(X-D_1{Z}_1-E+\frac{\gamma }{\mu }{E}_0+\frac{1}{\mu }{Y}_1\right)-\frac{1}{\mu }{Y}_3+J_2\right)\end{array}$ (12)

5) Update E:

$E=\arg \underset{E}{\min }{\lambda }_2{\Vert E\Vert }_1+\frac{\mu }{2}{\Vert E-\left(X-D_1{Z}_1-D_2{Z}_2+{\mu }^{-1}{Y}_1\right)\Vert }_F^2$ (13)

Similar to $J_2$ ,

$E=\max \left\{S_{\frac{{\lambda }_2}{\mu }}\left(X-D_1{Z}_1-D_2{Z}_2+{\mu }^{-1}{Y}_1\right),0\right\}$ (14)

Finally, we update the Lagrange multipliers as in [11] .

3. Experiment

3.1. Dataset

Our experiment was conducted on the iKala dataset [9] . The iKala dataset contains 252 30-second clips of Chinese popular songs in CD quality. In the following experiments, we randomly select 44 songs for training (i.e., learning the dictionaries D₁ and D₂), leaving 208 songs for testing the performance of separation. To reduce the computational cost and the memory footprint of the proposed algorithm, we down sample all the audio recordings from 44,100 to 22,050 Hz. Then, computed its STFT by sliding a Hamming window of 1411 samples with a 75% overlap to obtain the spectrogram.

3.2. Dictionary and E₀

Our implementation of Online Dictionary Learning for Sparse Coding (ODL) [12] is based on the SPAMS toolbox. Given N signals ( $x_i\in {ℝ}_m$ ), ODL learns a dictionary D by solving the following joint optimization problem,

$\begin{array}{l}\underset{D\ge \mathrm{0,}\alpha }{\min }\frac{1}{N}{\displaystyle \underset{i=1}{\overset{N}{\sum }}}\left(\frac{1}{2}{\Vert x_i-D{\alpha }_i\Vert }_2^2+\lambda {\Vert {\alpha }_i\Vert }_1\right)\\ s\mathrm{.}t\mathrm{.\; <mi>\; </mi>\; <mi>\; </mi>}{d}_j^{\text{T}}{d}_j\le \mathrm{1,}{\alpha }_i\ge 0\end{array}$ (15)

where ${\Vert \cdot \Vert }_2$ denotes the Euclidean and λ is a regularization parameter. The input frames are extracted from the training set after short-time Fourier transform (STFT). Following [8] , we define the dictionary size to be 100 atoms.

To get the reconstructed voice spectrogram from the annotation (E₀), we first transform the human-labeled vocal pitch contours into a time-frequency binary mask. The authors in [13] have proposed a harmonic mask similar to that of [14] , which only passes integral multiples of the vocal fundamental frequencies [15] [16] ,

$M\left(f\mathrm{,}t\right)=\{\begin{array}{l}1,\text{if}\left|f-n{F}_0\left(t\right)\right|<w/2\mathrm{,}\exists n\in N^{+}\\ 0,\text{otherwise}\mathrm{.}\end{array}$ (16)

Here $F_0\left(t\right)$ is the vocal fundamental frequency at time t, n is the order of the harmonic, and w is the width of the mask. Then we simply define the vocal annotations as $E_0=X\circ M$ , where $\circ$ denotes the Hadamard product.

3.3. Evaluation

¹http://bass-db.gforge.inria.fr/.

Separation performance is measured by BSS EVAL toolbox version 3.0¹. We use source-to-interference ratio (SIR), source-to-artifacts ratio (SAR) and source-to-distortion ratio (SDR) provided in the commonly used BSS EVAL toolbox version 3.0. Denotes the singing voice $\hat{v}$ , the original clean singing voice v, the source-to-distortion ratio (SDR) [17] is computed as follows,

$\text{SDR}\left(\hat{v},v\right)=10{\log }_{10}\left[\frac{{\langle \hat{v},v\rangle }^2}{\Vert {\hat{v}}^2\Vert \Vert v^2\Vert -{\langle \hat{v},v\rangle }^2}\right].$ (17)

Normalized SDR (NSDR) is the improvement of SDR from the original mixture x to the separated singing voice $\hat{v}$ [18] [19] , and is commonly used to measure the separation performance for each mixture,

$\text{NSDR}\left(\hat{v},v,x\right)=\text{SDR}\left(\hat{v},v\right)-\text{SDR}\left(x,v\right).$ (18)

For overall performance evaluation, the global NSDR (GNSDR) is calculated as,

$\text{GNSDR}=\frac{{\displaystyle \underset{i=1}{\overset{N}{\sum }}}{w}_i\text{NSDR}\left({\hat{v}}_i,v_i,x_i\right)}{{\displaystyle \underset{i=1}{\overset{N}{\sum }}}{w}_i},$ (19)

where N is the total number of the songs and w_i is the length of the i-th song. Higher values of SIR, SAR, SDR, GSIR, GSAR, GSDR and GNSDR represent better quality of the separation.

3.4. Parameter Selection

During parameter selection, we use the indicator of global normalized source-to-distortion ratio (GNSDR) as the evaluation index. The higher the value is, the better the separation quality is. In our algorithms, we set ${\lambda }_1={\lambda }_2=1/\sqrt{\max \left(m,n\right)}$ for each $X\in R^{m\times n}$ similar to [9] , Here we only adjust γ.

Figure 1 presents the GNSDR for the separated singing voice and background music, using LSPDi. In the vocal part, we can see that, the GNSDR monotonically increases with γ first and then gradually decreases. When $\gamma =5$ , the LSRi achieves the overall highest GNSDR. In the accompaniment part, the values of GNSDR increase first, steady after $\gamma =5$ . Therefore, we set the parameter $\gamma =5$ .

3.5. Comparison Results

We compare three different Low-rank, Sparse algorithms on the iKala dataset,

・ RPCA unsupervised method proposed by Huang et al. [3] , use default parameter values $\lambda =\frac{1}{\sqrt{\max \left(m,n\right)}}$ .

・ LSPD Supervised method proposed by Yu et al. [4] , use default parameter values ${\lambda }_1={\lambda }_2=\frac{1}{\sqrt{\max \left(m,n\right)}}$ .

・ LSRi Proposed LSRi method with Low-Rank representation and the reconstructed voice spectrogram from the annotation, ${\lambda }_1={\lambda }_2=\frac{1}{\sqrt{\max \left(m,n\right)}}$ and $\gamma =5$ .

As shown in Table 1, whether the singing part or the accompaniment, our method has a higher value of global normalized source-to-distortion ratio (GNSDR), which suggests that LSRi algorithm performs well in the overall separation performance, and introduction of prior knowledge improve the separation performance. In the vocal part, our algorithm achieves higher GSIR than RPCA and LSPD, which shows that LSRi has better ability to remove the instrumental sounds than RPCA and LSPD. In the background music part, our algorithm achieves higher GSIR, which suggests that LSRi has better ability to remove the singing, a better performs in limiting artifacts during the separation process. But GSAR values did not improve significantly, this indicates that we need to improve on eliminating the interference of the algorithm.

4. Conclusion

In this paper, we have presented a time-frequency based source separation algorithm for music signals. LSRi considers both the vocal and instrumental spectrograms as sparse matrix and low-rank matrix, respectively. And the components that are not identified parts are specified as a residual term. Note that the dictionaries for the singing voice and background music are pre-learned from isolated singing voice and background music training data, respectively. Furthermore, LSRi incorporates vocal annotations information further, through which prior knowledge of the voice and background music is introduced to the source separation processing. Our approach has successfully exploited relevant useful information. Evaluations on the iKala dataset show the proposed methods better performance for both the separated singing voice and music accompaniment. In future studies, we can consider applying LSRi to the separation of complete songs.

Cite this paper

Yang, C.H. and Zhang, H.J. (2018) Low-Rank Sparse Representation with Pre-Learned Dictionaries and Side Information for Singing Voice Separation. Advances in Pure Mathematics, 8, 419-427. https://doi.org/10.4236/apm.2018.84024

References