Shulou(Shulou.com)06/01 Report--

This article mainly introduces "nlp natural language processing based on SVD dimensionality reduction optimization method". In daily operation, I believe many people have doubts about nlp natural language processing dimensionality reduction optimization method based on SVD. The editor consulted all kinds of data and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts of "nlp natural language processing dimensionality reduction optimization method based on SVD". Next, please follow the editor to study!

Dimensionality reduction Optimization based on SVD

Vector dimension reduction: reduce the vector dimension on the basis of preserving the "important information" of the data as much as possible. We can find important axes (axes with wide data distribution), represent two-dimensional data as one-dimensional data, and use the projection values on the new axis to represent the values of each data point, as shown in the following diagram.

Sparse matrix and dense matrix transformation: a matrix with most elements of zero is called a sparse matrix. Important axes are found from the sparse matrix and rerepresented with fewer dimensions. As a result, sparse matrices are transformed into dense matrices in which most elements are not zero. This dense matrix is the distributed representation of the words we want.

Singular value decomposition (Singular Value Decomposition,SVD): any matrix X is decomposed into the product of U, S, V, where U and V are orthogonal matrices in which the column vectors are orthogonal to each other, and S is a diagonal matrix in which all elements except diagonal elements are zero.

For what's going on with SVD, analyze it from the code:

The svd method in the linalg module of NumPy is used in the code, as follows.

U, S, V = np.linalg.svd (W)

We output C, W, U, S, V, as shown below, C is a co-occurrence matrix and W is a PPMI matrix. You can see that the S matrix is arranged in descending order.

[0 1 0 0 0] [1 0 1 0 1 1 0] [0 1 0 1 0 0 0] [0 0 1 0 1 0 0] [0 1 0 1 0 0 0] [0 1 0 0 0 1] [0 0 0 1 0] [[0. 1.807 0. 0. 0. 0. 0. ] [1.807 0. 0.807 0. 0.807 0.807 0. ] [0. 0.807 0. 1.807 0. 0. 0. ] [0. 0. 1.807 0. 1.807 0. 0. ] [0. 0.807 0. 1.807 0. 0. 0. ] [0. 0.807 0. 0. 0. 0. 2.807] [0. 0. 0. 0. 0. 2.807 0. ] [[- 3.409e-01-1.110e-16-3.886e-16-1.205e-01 0.000e+00 9.323e-01 2.664e-16] [0.000e+00-5.976e-01 1.802e-01 0.000e+00-7.812e-01 0.000e+00 0.000e+00] [- 4.363e-01-4.241e-17-2.172e-16-5.088e-01-1.767e-17-2.253e-01-7.071e-01 [- 2.614e-16-4.978e-01 6.804e-01-4.382e-17 5.378e-01 9.951e-17-3.521e-17] [- 4.363e-01-3.229e-17-1.654e-16-5.088e-01-1.345e-17-2.253e-01 7.071e-01] [- 7.092e-01-3.229e-17-1.654e-16 6.839e-01-1.345e-17- 1.710e-01 9.095e-17] [3.056e-16-6.285e-01-7.103e-01 7.773e-17 3.169e-01-2.847e-16 4.533e-17]] [3.168e+00 3.168e+00 2.703e+00 2.703e+00 1.514e+00 1.514e+00 1.484e-16] [[0.000e+00-5.976e-01-2.296e-16-4.978e-01-1.186e-16 2.145e-16-6.285e-01] ] [- 3.409e-01-1.110e-16-4.363e-01 0.000e+00-4.363e-01-7.092e-01 0.000e+00] [1.205e-01-5.551e-16 5.088e-01 0.000e+00 5.088e-01-6.839e-01 0.000e+00] [- 0.000e+00-1.802e-01-1.586e-16-6.804e-01 6.344e-17 9.119e-17 7.103e-01] [- 9 .323e-01-5.551e-17 2.253e-01 0.000e+00 2.253e-01 1.710e-01 0.000e+00] [- 0.000e+00 7.812e-01 2.279e-16-5.378e-01 3.390e-16-2.717e-16-3.169e-01] [0.000e+00 2.632e-16-7.071e-01 8.043e-18 7.071e-01 9.088e-17 1.831e-17]]

Let's look at what the U, S, V matrices are, and add the following code.

Print ("_") jym = np.dot (V, U) print (jym) print ("_") jym2 = np.dot (U, V) print (jym2) print ("_") V2 = np.transpose (V) jb = np.dot (V, V2) print (jb)

The output is as follows, so the properties of U and V can be understood. From jb = np.dot (V, V2), the output jb matrix is a unit matrix, it is known that V and U are orthogonal matrices. Jym = np.dot (V, U), the output jym main diagonal elements are all 0. U and V are column vectors orthogonal to each other, and V is transposed in the formula, that is to say, the column vector of U is orthogonal to the row vector of V in the code, so multiply U by V, and their diagonal element is 0.

[[- 6.212e-17 1.000e+00 1.015e-08 2.968e-16-5.249e-09 1.712e-16 6.754e-17] [1.000e+00 1.597e-16 3.967e-16-2.653e-08 1.099e-16-1.336e-08-5.293e-09] [2.653e-08 3.025e-16-2.284e-16-1.000e+00 4.270e-16 1.110e-08 5.760e-09] [3. 718e-16-1.015e-08-1.000e+00 1.958e-16 4.416e-10-2.641e-16 2.132e-16] [1.336e-08 1.143e-16 2.378e-16 1.110e-08 3.405e-17-1.000e+00-2.662e-09] [- 1.096e-17 5.249e-09 4.416e-10-4.753e-16-1.000e+00-4.458e-17 8.307e-17] [- 5.293e -09-1.657e-16 7.657e-17-5.760e-09-1.925e-16 2.662e-09 1.000e+00] [[- 8.977e-18 9.539e-01-2.775e-17-2.497e-01 3.879e-16 7.108e-18-1.668e-01] [9.539e-01 9.667e-18 1.764e-01 0.000e+00 1.764e-01 1.670e-01 0.000e+00] [4.757e-18 1 .764e-01 5.000e-01 6.846e-01-5.000e-01 3.262e-17-1.578e-02] [- 2.497e-01-1.105e-16 6.846e-01 1.064e-16 6.846e-01-2.032e-02 1.016e-16] [3.622e-18 1.764e-01-5.000e-01 6.846e-01 5.000e-01 1.192e-16-1.578e-02] [3.622e -18 1.670e-01-1.220e-16-2.032e-02 6.079e-17 9.043e-17 9.857e-01] [- 1.668e-01 2.741e-17-1.578e-02-5.192e-17-1.578e-02 9.857e-01-4.663e-17]] [[1.000e+00 6.620e-17 7.901e-18-1.015e-08-8.632e-18 5.249e-09-9.431e-17] [6.620e-17 1.000e+00 2.653e-08-3.141e-18 1.336e-08-1.414e-16-5.293e-09] [7.901e-18 2.653e-08 1.000e+00-1.074e-17-1.110e-08 4.054e-17 5.760e-09] [- 1.015e-08-3.141e-18-1.074e-17 1.000e+00 4.150e-18-4.416e-10 1.171e-16] [ -8.632e-18 1.336e-08-1.110e-08 4.150e-18 1.000e+00 3.792e-17-2.662e-09] [5.249e-09-1.414e-16 4.054e-17-4.416e-10 3.792e-17 1.000e+00 2.740e-16] [- 9.431e-17-5.293e-09 5.760e-09 1.171e-16-2.662e-09 2.740e-16 1.000e+00]] the intuitive meaning of SVD

U is an orthogonal matrix. This orthogonal matrix forms the base axis (base vector) in some spaces, and the matrix U can be used as the "word space". S is a diagonal matrix, and the singular values are arranged in descending order on the diagonal. the size of the singular values means the importance of the "corresponding base axis". If the singular value is small, the importance of the corresponding base axis is small, so the original matrix can be approximated by removing the redundant column vectors in the U matrix. Thus, the word vector is represented by the reduced dimension matrix. The schematic diagram is as follows.

Sparse vector W is transformed into dense vector U by SVD. If you want to reduce the dimension of this dense vector, such as reducing it to a two-dimensional vector, you can take out the first two elements of U.

Text = 'You say goodbye and I say hello.'corpus, word_to_id, id_to_word = preprocess (text) vocab_size = len (id_to_word) C = create_co_matrix (corpus, vocab_size, window_size=1) W = ppmi (C) # SVDU, S, V = np.linalg.svd (W) np.set_printoptions (precision=3) # significant digits for i in range (7): print (C [I]) print (U) # plotfor word Word_id in word_to_id.items (): plt.annotate (word, (U [word _ id, 0], U [word _ id, 1]) plt.scatter (U [:, 0], U [:, 1], alpha=0.5) plt.show ()

U of output:

[[- 3.409e-01-1.110e-16-3.886e-16-1.205e-01 0.000e+00 9.323e-01 2.664e-16] [0.000e+00-5.976e-01 1.802e-01 0.000e+00-7.812e-01 0.000e+00 0.000e+00] [- 4.363e-01-4.241e-17-2.172e-16-5.088e-01-1.767e-17-2.253e-01-7.071e-01] [- 2.614e-16-4.978e-01 6.804e-01-4.382e-17 5.378e-01 9.951e-17-3.521e-17] [- 4.363e-01-3.229e-17-1.654e-16-5.088e-01-1.345e-17-2.253e-01 7.071e-01] [- 7.092e-01-3.229e-17-1.654e-16 6.839e-01-1.345e-17- 1.710e-01 9.095e-17] [3.056e-16-6.285e-01-7.103e-01 7.773e-17 3.169e-01-2.847e-16 4.533e-17]]

Each word is represented by a two-dimensional vector and drawn on the graph as follows: goodbye is close to hello, you and I, and this result is based on cosine similarity.

At this point, the study of "SVD-based dimensionality reduction optimization method for nlp natural language processing" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.