How to transpose RDD or MLLib matrix
How to RDD or MLLib matrix transpose operation, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain in detail for you, people with this need can come to learn, I hope you can gain something.
How to transpose the matrix of Spark Mllib or a RDD. Spark Mllib matrix has a variety of forms, distributed and non-distributed, non-distributed here let alone the wave tip, very simple, because it is based on arrays. While distributed storage is based on RDD, the problem becomes how to transpose a RDD.
First, let's introduce what a transpose operation is:
According to the definition in the encyclopedia, the matrix obtained by exchanging the rows and rows of a matrix is the transpose of the matrix.
To swap the rows and rows of a RDD, the main ideas are as follows:
1. Convert RDD first, and give each line a unique line number (row, rowIndex).
2, for each line of RDD, convert it to (value, colIndex), and organize it to (colIndex.toLong, (rowIndex, value))
3, carry out flatmap
4, after the completion of step 3, we only need to group by 3key and sort by its key to get the order of the transformed columns.
5. After completing step 4, we can build new rows according to the (rowIndex, value) of each row, using the subscript and its value to ensure the converted order of each row.
At this point the conversion is complete.
The specific steps are as follows:
Def transposeRowMatrix (m: RowMatrix): RowMatrix = {
Val transposedRowsRDD = m.rows.zipWithIndex.map {case (row, rowIndex) = > rowToTransposedTriplet (row, rowIndex)}
.flatMap (x = > x) / / (newRowIndex, (newColIndex, value))
.groupByKey
SortByKey () .sortByKey (_. _ 2) / / A pair of row is sorted to remove the index
.map (buildRow) / / use indexes and values to rebuild each row and remove the index
New RowMatrix (transposedRowsRDD)
}
/ / convert each line
Def rowToTransposedTriplet (row: Vector, rowIndex: Long): Array [(Long, (Long, Double))] = {
Val indexedRow = row.toArray.zipWithIndex
IndexedRow.map {case (value, colIndex) = > (colIndex.toLong, (rowIndex, value))}
}
/ / build a new line
Def buildRow (rowWithIndexes: Iterable [(Long, Double)]): Vector = {
Val resArr = new Array [Double] (rowWithIndexes.size)
RowWithIndexes.foreach {case (index, value) = >
ResArr (index.toInt) = value
}
Vectors.dense (resArr)
}
test
Prepare data
Val observations = sc.parallelize (
Seq (
Vectors.dense (1.0,10.0,100.0)
Vectors.dense (2.0,20.0,200.0pm 2.0)
Vectors.dense (3.0,30.0,300.0)
)
)
Generating matrix
Val mat: RowMatrix = new RowMatrix (observations)
You will find that the ranks have been changed.
Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.
