Shulou(Shulou.com)06/03 Report--

Good programmer big data Learning Route sharing function + map Mapping + Yuanzu, language implementation on various platforms of big data

Hadoop is implemented by java. Since 2003, there are three blocks: data processing, data storage and data calculation.

Storage: hbase-- > data into a table

Processing: hive-- > tools for data warehousing

Calculation: mapreduce-- > entry level

Hive java implementation

Flink drain processing scala implementation-- > Computing engine

Scala implementation of kafka data caching

Spark scala implementation\ python

DL 4J scala implementation

Sklearning python implementation

Akka scala implementation-- > Communication

The development process of scala

Scala author:

Martin, a functional programming enthusiast, has been working on the jvm platform

The charm of scala

Scala script programming for spark-shell in spark cluster

Scala conducts app development of spark for maven of IDEA

Big data's thought of Development and programming

1. Load the dataset, (based on local data, win\ hdfs\ Array)

two。 Split dataset, (aggregate operation)

3. Select apache framework (processing-Hive, storage-HBSE, computing-mapreduce,)

4. Debugging, testing code

5. Tuning, code, app, data skew, troubleshooting troubleshooting

6. Upper line

Val and var

Val is modified by final. Variable values cannot be modified.

Var can be modified

Java and scala share jvm, but they think differently.

Scala data type:

In scala, the base class of all classes is Any

The first kind:

AnyRef, a subclass of Any, is the base class for all reference types, and all inheritance comes from AnyRef

The second kind:

AnyVal, a subclass of Any, Byte\ Char\ Short\ int\ Long\ Float\ Double (no wrapper class)-- > seven data types, Unit (no return value type)\ Boolean

Conditional expression

Object test {

Def main (args: Array [String]): Unit = {

Val x = 1

/ / judge the value of x and assign the result to y

Val y = if (x > 0) 1 else-1

Println (y) / / 1

Println ("-")

/ / mixed type expressions are supported

Val z = if (x > 1) 1 else "error"

Println (z) / / error

Println ("-")

/ / if else is missing, it is equivalent to if (x > 2) 1 else ()

Val m = if (x > 2) 1

Println (m) / ()

Println ("- -")

/ / there is a value in every expression in scala. There is a unit class in scala, which is written (), which is equivalent to void in java.

Val n = if (x > 2) 1 else ()

Println (n) / /

Println ("-")

/ / if,else if

Val k = if (x

< 0) 0 else if (x >

= 1) 1 else-1

Println (k) / 1

}

}

Block expression

* Block expressions are formed by including a series of column expressions in "{}" in scala

* here is a block expression

* modified with the lazy keyword, the code office runs only when the call is made.

Object Block {

Def main (args: Array [String]): Unit = {

Val x = 0

Val result = {

If (x

< 0) { -1 } else if (x >

= 1) {

one

} else {

"error"

}

}

The value of println (result) / / result is the result of the whole block expression.

}

}

Printf printing and the use of foreach

Arr.toBuffer-> variable constant array

Scala > val arr = Array (1, 2, 3, 4)

Arr: Array [Int] = Array (1,2,3,4)

Scala > print (arr.toString)

[I@108fa66

Scala > print (arr.toBuffer)

ArrayBuffer (1,2,3,4)

Printf usage-- > is equivalent to formatting

S represents a string

D represents a number

Object game {

Def main (args: Array [String]): Unit = {

/ / readline () allows us to read data entered by the user from the console, similar to System.in and Scanner in java

Val name = readLine ("Welcome to Game House,please tell me your name:\ n")

Print ("Thanks,then please tell me your age:\ n")

Val age = readInt () / / initialize

If (age > 18) {

Printf ("Hi,%s,you are d years old,so you are legal to come here", name,age)

}

Else {

Printf ("Sorry,%s,you are only d years old,you are illegal to come here", name,age)

}

}

}

Three ways of for cycle + filter

Scala > val arr = Array (2, 3, 4, 6)

Arr: Array [Int] = Array (2,3,4,6)

Scala > for (I println (arr.filter (_% 2examples _ 0) .toBuffer) / / filter calls the for loop internally to filter

ArrayBuffer (2,4,6)

Of the three for loops, to contains the last bit and until does not

Advanced for Loop-> nesting

Scala > for (i val value = for (I arr)

Res19: Array [Int] = Array (2,3,4,6)

Scala > arr.map (x = > print (x))

2346res20: Array [Unit] = Array (())

Scala > arr.foreach (print)

2346

Scala > val r = print (2)

2R: Unit = ()

Functions and methods

-- definition method--

No return type will not print the result unless you print in the method or add the default Unit type

Scala > def test (XRU IntMagi YRV Int): Int = {x * y}

Test: (X: Int, y: Int) Int

Scala > test (3pr 4)

Res22: Int = 12

Scala > def test1 (x * y}) {x * y}

Test1: (X: Int, y: Int) Unit

Scala > test1 (2pr 3)

/ / No result, no return value type

Scala > def test3 (XRU IntMagi YRO Int) = {x * y}

/ / if the = sign is added, the system will determine the data type by itself.

Test3: (X: Int, y: Int) Int

Scala > test3 (2pr 3)

Res8: Int = 6

Scala > def test2 (XRU IntMagi YRO Int) {

| | println (x * y)} |

Test2: (X: Int, y: Int) Unit

Scala > test2 (2pr 3)

6 / / No return value is returned. It is from print.

For recursive methods, you must specify a return type

Object digui {

Def main (args: Array [String]): Unit = {

Println (fab (3))

}

Def fab (n:Int): Int = {

If (n val fun1 = (x Int)) = > x * y

Fun1: (Int, Int) = > Int =

Scala > fun1 (2pr 3)

Res9: Int = 6

Scala > val fun2 = () = > 10

Fun2: () = > Int =

Scala > fun2 ()

Res10: Int = 10

Scala > fun2

Res11: () = > Int =

Call function: map provides an array

Object func {

Def main (args: Array [String]): Unit = {

Val arr = Array (2, 3, 4, 5, 6, 7)

/ / val fun = (x:Int) = > Xing20

/ / val res = arr.map (fun) / / the first kind

/ / val res = arr.map ((x:Int) = > Xero20) / / the second kind

Val res = arr.map (_ * 20) / / the third kind

Println (res.toBuffer) / / output result is ArrayBuffer (40,60,80,100,120,140)

}

}

The difference between a method and a function

In functional programming languages, functions are "first-class citizens" and can be passed and manipulated like any other data type

Case: first define a method, define a function, and then pass the function to the method

Object hanshu {

/ / define a method

Def T1 (f: (Int,Int) = > Int): Int = {

F (2pr 3)

}

/ / define function

Val F1 = (XRU IntMagne YRO int) = > x * y

Val f2 = (mvuIntjnjnlnlt) = > m + n

Def main (args: Array [String]): Unit = {

/ / call function method

Println (T1 (F1)) / / 6

Println (T1 (f2)) / / 5

}

}

Array array

Common algorithms of array

Variable string guide package: import scala.collection.mutable.ArrayBuffer

When defining a collection: Array () is immutable, ArrayBuffer () is variable

The string after the package is imported (variable)

Increase data

Scala > val arr2=new ArrayBuffer [Int] (1)

Arr2: scala.collection.mutable.ArrayBuffer [Int] = ArrayBuffer ()

Scala > arr2 + = 2

Res24: arr2.type = ArrayBuffer (2)

Scala > arr2 + = (3pm 4pm 5)

Res25: arr2.type = ArrayBuffer (2,3,4,5)

Scala > arr2 + + = ArrayBuffer (1)

Res27: arr2.type = ArrayBuffer (2,3,4,5,1)

Add data from a specified location

Scala > arr2.insert (0ju 20je 21)

Scala > arr2

Res29: scala.collection.mutable.ArrayBuffer [Int] = ArrayBuffer (20, 21, 2, 3, 4, 5, 5, 1)

Delete element

Scala > arr2.remove (2jue 3) / / which element begins and ends

Scala > arr2

Res33: scala.collection.mutable.ArrayBuffer [Int] = ArrayBuffer (20,21,5,1)

Reverse

Map mapping

First, guide the package: import scala.collection.mutable.Map

Two methods for defining map sets

Scala > val age = Map ("jack"-> 20, "Leo"-> 21) / / Arrow creation

Age: scala.collection.immutable.Map [String,Int] = Map (jack-> 20, Leo-> 21)

Scala > val age = Map (("jack"-> 20), ("Leo"-> 30)) / / meta-ancestor creation

Age: scala.collection.immutable.Map [String,Int] = Map (jack-> 20, Leo-> 30)

View map set

Scala > age ("Leo")

Res0: Int = 30

Scala > age

Res1: scala.collection.immutable.Map [String,Int] = Map (jack-> 20, Leo-> 30)

Modify map set specified parameters

Scala > age ("Leo")-> 20

Res3: (Int, Int) = (30pm 20)

Add map set parameters

Scala > age + = ("Mike"-> 18, "Heb"-> 23)

Res9: age.type = Map (jack-> 20, Mike-> 18, Heb-> 23, Leo-> 21)

Delete map set parameters

Scala > age-"Mike"

Res10: scala.collection.mutable.Map [String,Int] = Map (jack-> 20, Heb-> 23, Leo-> 21)

Loop traversal (LinkedHashMap\ SortedMap)

Scala > for ((key,value) for (key for) (value val tuple = ("zhangsan", 10pyrrine 2.3 true)

Tuple: (String, Int, Double, Boolean) = (zhangsan,10,2.3,true)

Scala > val tuple, (a dint bje c) = ("zhangsan", 2m true)

Tuple: (String, Int, Boolean) = (zhangsan,2,true)

A: String = zhangsan

B: Int = 2

C: Boolean = true

Get Yuanzu: Yuanzu starts with 1, and the array starts with subscript 0.

Scala > tuple._1

Res14: String = zhangsan

Scala > tuple._3

Res15: Boolean = true

Convert an array to a map map

Scala > val arr = Array (("xixi", 2), ("", 1), ("heihei", 3))

Arr: Array [(String, Int)] = Array ((xixi,2), (,1), (heihei,3))

Scala > arr.toMap

Res16: scala.collection.immutable.Map [String,Int] = Map (xixi-> 2, -> 1, he

Ihei-> 3)

Zipper operation: the zip command binds multiple values together

Note: if the number of elements of the two arrays is not the same, the length of the array generated after the zipper operation is the number of elements of the smaller array.

Scala > val arr2 = Array (2mai 3pm 4)

Arr2: Array [Int] = Array (2,3,4)

Scala > arr zip arr2

Res17: Array [(String, Int), Int)] = Array ((xixi,2), 2), ((,1), 3), ((heihei)

3), 4))

Mapping is a set of kv duals, and duality is the simplest form of meta-ancestor, called dual meta-ancestor.

The maximum meta-ancestor length supported by scala is 22 (elements)

-- gather--

The collection of scala has three classes: sequence sep, set set, and mapping Map

There are two types of collections in scala: mutable mutable and immutable immutable (default)

Sequence--

-immutable sequence import scala.collection.immutable._

-in Scala, the list is either empty (Nil stands for empty list) or a head element plus a tail list.

-:: indicates the combination of right, and the list starts from right to left

-when defining List, List () is immutable and ListBuffer () is variable

Package com.qf.collect

Object ImmutListDemo {

Def main (args: Array [String]) {

/ / create an immutable collection

Val lst1 = List (1, 2, 3)

/ / insert 0 in front of lst1 to generate a new List

Val lst2 = 0:: lst1

Val lst3 = lst1.:: (0)

Val lst4 = 0 +: lst1

Val lst5 = lst1.+: (0)

/ / add an element to the end of lst1 to create a new collection

Val lst6 = lst1: + 3

/ / merge 2 list into a new List

Val lst0 = List (4, 5, 6)

Val lst7 = lst1 + + lst0

/ / insert lst0 after lst1 to generate a new collection

Val lst8 = lst1 + +: lst0

/ / insert lst0 in front of lst1 to generate a new collection

Val lst9 = lst1.::: (lst0)

Println (lst9)

}

}

-- Set

Immutable

Object ImmutableSet extends App {

Val set1 = new HashSet [Int] ()

/ / merge the element with set and generate a new set. The original set remains unchanged.

Val set2 = set1 + 4

Println (set2) / / Set (4)

/ / elements in set cannot be repeated and are removed automatically.

/ / val set3 = set2 + + Set (5 record6, 7) / / Set (5, 6, 6, 7 and 4)

Val set3 = Set (2pm 3) + + set2

Println (set3) / / Set (2d3)

}

Mutable set is unordered and unrepeatable

Scala > import scala.collection.mutable._

Import scala.collection.mutable._

Scala > val set1 = HashSet (2mai 3pm 4)

Set1: scala.collection.mutable.HashSet [Int] = Set (2,3,4)

Scala > set1 + = 1

Res0: set1.type = Set (1,2,3,4)

Scala > set1.add (4)

Res1: Boolean = false

Scala > set1.add (6)

Res2: Boolean = true

Scala > set1

Res3: scala.collection.mutable.HashSet [Int] = Set (1,2,6,3,4)

Scala > set1 + + = HashSet (2Power7)

Res4: set1.type = Set (1,2,6,3,7,4)

Scala > set1-= 1

Res5: set1.type = Set (2,6,3,7,4)

Scala > set1.remove (2)

Res6: Boolean = true

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.