Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to achieve Isomap in Python". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to achieve Isomap in Python".

Principal component analysis is a powerful method, but it often fails because it assumes that the data can be linearly modeled. PCA represents new features as a linear combination of existing features and multiplies each feature by a coefficient.

In order to solve the limitations of principal component analysis, people have produced a variety of technologies through the application of data with different structures. However, manifold learning seeks a method that can be extended to all data structures.

Different data structures refer to different attributes in the data. For example, it may be linearly divisible or very sparse. The relationships in the data can be tangent, parallel, enveloping, or orthogonal. PCA works well on a very specific subset of data structures because it works under linear assumptions.

To put things in context, consider a 300 x 300 pixel avatar. Under perfect conditions, each image will be perfectly centered, but in fact, many additional degrees of freedom need to be considered, such as lighting or face tilt. If we think of an avatar as a point in 90000-dimensional space, changing various effects, such as tilting the head or looking in different directions, will make it move nonlinearly in space, even if it is the same kind of the same object.

This kind of data often appears in real-world data sets. In addition, when PCA is skewed and extreme, PCA can become very bad (see nonlinear PCA for a solution). We need a method of dimensionality reduction that can be extended.

Manifold learning refers to this task. Many methods in manifold learning may have been seen before, such as t-SNE and local linear embedding (LLE). There are many articles and papers that delve into the technical and mathematical details of these algorithms, but this article will focus on general intuition and implementation.

Note that although some variants of dimensionality reduction are supervised (such as linear / quadratic discriminant analysis), manifold learning usually refers to unsupervised dimensionality reduction, where categories are not provided to the algorithm (although they may exist).

PCA attempts to create several linear hyperplanes to represent dimensions, just like multiple regression construction as an estimate of data, manifold learning tries to learn manifolds, manifolds are smooth surfaces in multidimensional space. As shown in the following figure, these are usually formed by minor transformations of the image.

Local linear patches can then extract the parts that are tangent to the manifold. These patches (patches) are usually sufficient in number to accurately represent the manifold.

These manifolds are not modeled by any one mathematical function, but by several small linear patches, so these linear neighborhoods can model any manifold. Although it may not be clear how some algorithms approach the modeling manifold, the basic idea is very similar.

The following are the basic assumptions or aspects of the manifold learning algorithm:

There is a nonlinear relationship in the data that can be modeled by manifolds-the surface spans multiple dimensions, smooth, and less "rocking" (too complex).

It is not important to maintain the multidimensional shape of the data. Rather than "unfold" or "project" the data (such as PCA) in a specific direction to maintain the general shape of the data, perform more complex operations, such as unfolding a curled strip or flipping the sphere from the inside out.

The best method of manifold modeling is to treat a surface as a surface composed of several neighborhoods. If each data point tries to maintain the distance from all other points, but only the distance between the points adjacent to it, you can maintain a geometric relationship in the data.

This idea can be well understood by studying the different ways to decompose this spiral data set. On the left is a more PCA-like method for saving the shape of the data, where each point is connected to each other. However, on the right is a method that calculates only the distance between the neighbors of the data points.

This relative neglect of points outside the neighborhood can lead to interesting results. For example, consider this Swiss roll data set, which is wound in three-dimensional space and simplified to a two-dimensional bar chart. In some cases, this effect is not ideal. However, if the curve is the result of the tilt of the camera or the external influence on the audio quality in the image, manifold learning is of great help to us by cleverly solving these complex nonlinear relationships.

On the Swiss Roll dataset, even special variants like Kernel-PCA cannot capture the gradient of the value. On the other hand, the manifold learning algorithm local linear embedding (LLE) can also do it.

Let's learn more about three popular manifold learning algorithms: IsoMap, local linear embedding and t-SNE.

One of the earliest explorations of manifold learning is the Isomap algorithm, that is, the abbreviation of isometric mapping. Isomap seeks a low-dimensional representation to maintain the "distance" between points. Distance is a generalization of surface distance. Therefore, Isomap does not use the distance formula derived from the Pythagorean theorem to measure the pure Euclidean distance, but to optimize the distance along the discovered manifold.

When trained on MNIST data sets, Isomap performs better than PCA, showing the correct segmentation of different types of numbers. The proximity and distance between some groups of numbers reveal the structure of the data. For example, the closer "5" and "3" (in the lower left corner) do look similar.

The following is the implementation of Isomap in Python. Because MNIST is a very large dataset, you may only want to use .fit _ transform (X [: 100]) to train Isomap in the first 100 training examples.

From sklearn.datasets import load_digits # mnistfrom sklearn.manifold import IsomapX, _ = load_digits (return_X_y=True) # loading data embedding = Isomap (n_components=2) # result data has two dimensions, that is, "component" X_transformed = embedding.fit_transform (X) # fitting model and transformation

Local linear embedding uses a variety of tangent linear patches (as shown above) to model the manifold. It can be thought of as a local PCA for each neighborhood to generate a linear hyperplane, and then compare the results globally to find the best nonlinear embedding. The goal of LLE is to "unfold" or "unpack" the structure of the data in a distorted way, so LLE usually has a higher density in the center.

Note that LLE performs relatively poorly on MNIST datasets. This is probably because the MNIST dataset consists of multiple manifolds, while LLE is designed to handle simpler datasets such as Swiss Roll. Its strategy of representing a function as several small linear functions may not well handle large and complex dataset structures.

Assuming that the dataset (X) has been loaded, the implementation of LLE is shown below.

From sklearn.manifold import LocallyLinearEmbeddingembedding = LocallyLinearEmbedding (n_components=2) # 2D X_transformed = embedding.fit_transform (X)

T-SNE is one of the most popular choices in high-dimensional visualization, and it is the representative of t-distribution random neighbor embedding. The algorithm transforms the relationship in the original space into t-distribution, that is, the normal distribution of small samples and relatively unknown standard deviation. This makes t-SNE very sensitive to the common local structures in manifold learning. Because of its many advantages, it is considered to be a feasible visualization method. The advantages are as follows:

It can reveal the structure of data on multiple scales.

It reveals data that exists in multiple manifolds and clusters.

The trend in the central gathering point is small.

Isomap and LLE are the best tools for expanding single, continuous, low-dimensional manifolds. On the other hand, t-SNE focuses on the local structure of the data, trying to aggregate parts rather than trying to "expand". This gives t-SNE an advantage in using multiple manifolds to fit high-dimensional data. It uses the gradient descent method to train and tries to minimize the entropy between distributions. In this sense, it is almost like a simplified, unsupervised neural network.

T-SNE is very powerful, Isomap and LLE try to expand the data, while t-SNE tries to aggregate the data. For high-dimensional multi-manifold datasets such as MNIST, rotation and shift lead to non-linear relationships. The performance of t-SNE is even better than LDA, and LDA also requires label information.

However, t-SNE also has some disadvantages:

T-SNE is computationally expensive (compare the runtime in the chart above). For a million sample data sets, it may take several hours, while PCA can be completed in seconds or minutes.

The algorithm uses randomness (randomness) to select seeds. If the seeds are not placed properly, the running time of the algorithm will be increased and the performance will be reduced.

The global structure is not explicitly preserved (that is, more emphasis on clustering rather than displaying the global structure). However, in the implementation of sklearn, this problem can be solved by using PCA initialization points, and PCA is built specifically to maintain the global structure.

T-SNE can also be implemented in sklearn:

From sklearn.manifold import TSNEembedding = TSNE (n_components=2) # # 2D X _ transformed = embedding.fit_transform (X)

Laurens van der Maaten, author of t-SNE, says that when t-SNE results are not good, consider the following:

As a fitness check, try running PCA on the data to reduce it to two dimensions. If this also gives bad results, then your data may not be well structured in the first place. If PCA is good, but t-SNE is not, I'm sure you did something wrong.

Why would he say that? Manifold learning is not another variant of PCA, but a generalization. Things that perform well in PCA are almost guaranteed to perform well in t-SNE or other manifold learning techniques because they are generalizations.

Just as an apple is also an object of fruit (generalization), it is usually wrong if something does not produce a result similar to its generalization. On the other hand, if both methods fail, the data is likely to be difficult to model.

Key points

Because PCA is linear, it is not possible to model nonlinear relationships.

Nonlinear relationships often occur in data sets because lighting or tilting can move the same kind of data points nonlinearly in Euclidean space.

Manifold learning attempts to extend PCA to various data set structures to reduce dimensionality. The main idea is that the modeling of manifolds or continuous surfaces should keep the local distance prior to the global distance.

Isomap tries to maintain the distance measured by the manifold surface, that is, the distance not in Euclidean space.

Local linear embedding can be regarded as representing a manifold as several linear blocks in which PCA executes.

T-SNE uses more "clustering" methods than "expansion" methods, but still like other manifold learning algorithms, it uses probability and t-distribution to maintain the local distance first.

Thank you for your reading, the above is the content of "how to achieve Isomap in Python". After the study of this article, I believe you have a deeper understanding of how to achieve Isomap in Python, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.