Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about how to analyze the decision tree in Apache Spark. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

Decision Tree in Apache Spark

Decision tree is an effective method to classify, predict and promote decision-making in sequential decision-making. The decision tree consists of two parts:

Decision making (Desion)

Results (Outcome)

The decision tree contains three types of nodes:

Root node (Root node): the top node of the tree that contains all the data.

Split node (Splitting node): a node that assigns data to a subgroup (subgroup).

Terminal node (Terminal node): the final decision (that is, the result).

(Splitting node), only in terms of the concept of tree in discrete mathematics, refers to branch node. The following translation sometimes translates "branch" into branch node to emphasize "branch".

In order to reach the terminal node or get the result, the process starts from the root node. Select the branch node according to the decision made on the root node. Based on the decision made on the branch node, the next child branch node is selected. This process continues until we reach the terminal node, and the value of the terminal node is our result.

Decision Tree in Apache Spark

It may sound strange that there is no implementation of a decision tree in Apache Spark. Technically, however, there is. In Apache Spark, you can find an implementation of the random forest algorithm, where the number of trees can be specified by the user. Therefore, Apache Spark uses a tree to invoke the random forest.

In Apache Spark, decision tree is a greedy algorithm that performs recursive binary segmentation on the feature space. The tree predicts the same label for each partition (that is, leaf nodes). In order to gain information at the nodes of the tree, each branch node is greedily selected by selecting a * partition among a set of possible branches.

Node impurity is a measure of label consistency on a node. The current implementation provides two impure classification methods (Gini impurities and entropy (Gini impurity and entropy)).

Stop rule

Stop recursive tree construction at a node if one of the following conditions is satisfied:

The depth of the node is equal to the maxDepth parameter used for training.

No candidate segmentation node leads to greater information gain than minInfoGain.

There are no candidate split nodes to generate (at least have a training minInstancesPerNode instance) child nodes.

Useful parameters

Algo: it can be classified or regressed.

NumClasses: the number of categories.

MaxDepth: defines the depth of the tree based on the node.

MinInstancesPerNode: each child node of a node to be further split must receive at least such a number of training instances (that is, the number of instances must be equal to this parameter).

MinInfoGain: for further splitting of a node, it must be satisfied that at least so much information is increased after the split.

MaxBins: the number of bin used when discretizing continuous features.

Prepare training data for decision tree

You cannot provide any data directly to the decision tree. It needs a special format to provide. You can use HashingTF techniques to convert training data into tagged data so that the decision tree can understand it. This process is also known as data standardization.

(data) supply and obtain results

Once the data is standardized, you can provide the same decision tree algorithm for row classification. But before that, you need to split the data for training and testing purposes; for the accuracy of the test, you need to keep some of the data for testing. You can provide data like this:

Al splits = data.randomSplit (Array (0.7,0.3) val (trainingData, testData) = (splits (0), splits (1)) / / Train a DecisionTree model. / / Empty categoricalFeaturesInfo indicates all features are continuous. Val numClasses = 2 val categoricalFeaturesInfo = Map [Int, Int] () val impurity = "gini" val maxDepth = 5 val maxBins = 32 val model = DecisionTree.trainClassifier (trainingData, numClasses, categoricalFeaturesInfo, impurity, maxDepth, maxBins)

Here, the data is my standardized input, which I divide into 7:3 ratios for training and testing purposes. We are using "gini" impurities of 5 depth ("gini" impurity).

Once the model is generated, you can also try to predict the classification of other data. But before that, we need to verify the classification accuracy of the recently generated model. You can verify its accuracy by calculating "test error".

/ Evaluate model on test instances and compute test error val labelAndPreds = testData.map {point = > val prediction = model.predict (point.features) (point.label, prediction)} val testErr = labelAndPreds.filter (r = > r.room1! = r.room2) .count () .toDouble / testData.count () println ("Test Error =" + testErr) this is how the editor shared how to parse the decision tree in Apache Spark. If you happen to have similar doubts, please refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.