Shulou(Shulou.com)06/03 Report--

1. Basic overview of serialization 1. What is serialization?

Serialization is converting objects in memory into byte sequences (or according to other data transfer protocols) so that they can be persisted to disk and transferred over the network.

2. Why serialization is needed

In general, objects are only stored in local memory and only local process calls are allowed. With the emergence of distributed programs, different processes need to call objects on different hosts, which requires objects to be transferred to other hosts through the network. However, the object cannot be transmitted over the network without processing, and after serialization processing, the object can be transmitted over the network.

3. Serialization scheme in java

The serialization scheme is implemented by itself in java. As long as the Serializable interface is implemented when a class is defined, the corresponding serialization will be implemented automatically within java. Such as:

Public class Test implements Serializable {/ / this serialization number is required to identify the class private static final long serialVersionUID = xxxx;}

However, when the serialization interface in Java is implemented, it will be accompanied by a lot of additional information, such as various check information, header, inheritance system and so on. It is not easy to transmit efficiently on the network (low performance). So hadoop itself implements an additional serialization mechanism, which is small in size, low in bandwidth consumption, and fast in serialization and deserialization.

Serialization in hadoop 1. Class basic dependency

Classes in hadoop that implement the interface Writable can be serialized. And hadoop implements many basic types of serializable classes. The dependency graph is as follows:

​ figure 2.1 hadoop serialization dependency graph

You can see that all serializable classes implement the WritableComparable interface, which inherits both the Writable and Comparable interfaces. Let's take a look at these three interfaces:

/ / WritableComparable.javapublic interface WritableComparable extends Writable, Comparable {} / * empty interface * / Writable.javapublic interface Writable {void write (DataOutput var1) throws IOException; void readFields (DataInput var1) throws IOException;} / * mainly contains methods for reading and writing serialized objects * / Comparable.javapublic interface Comparable {public int compareTo (To) } / * provide methods for comparing serialized objects * / 2, hadoop serialized classes and basic types comparison table java type hadoop writable type booleanBooleanWritablebyteByteWritableIntIntWritablefloatFloatWritablelongLongWritabledoubleDoubleWritablestringTextmapMapWritablearrayArrayWritable3, source code implementation of commonly used serialized classes

Let's take a look at the source code of IntWritable, a commonly used serialization class.

Package org.apache.hadoop.io;import java.io.DataInput;import java.io.DataOutput;import java.io.IOException;import org.apache.hadoop.classification.InterfaceAudience.Public;import org.apache.hadoop.classification.InterfaceStability.Stable;@Public@Stablepublic class IntWritable implements WritableComparable {private int value; public IntWritable () {} public IntWritable (int value) {this.set (value);} public void set (int value) {this.value = value } public int get () {return this.value;} / / here is the method that implements the writable interface public void readFields (DataInput in) throws IOException {this.value = in.readInt ();} public void write (DataOutput out) throws IOException {out.writeInt (this.value) } / / equals comparison method for serialized objects public boolean equals (Object o) {if (! (o instanceof IntWritable)) {return false;} else {IntWritable other = (IntWritable) o; return this.value = = other.value;}} public int hashCode () {return this.value } / / the method for comparing object sizes public int compareTo (IntWritable o) {int thisValue = this.value; int thatValue = o.value; return thisValue < thatValue?-1: (thisValue = = thatValue? 0: 1);} public String toString () {return Integer.toString (this.value);} / * here is the key, using the following Comparator inner class as the default comparison method. Because the static code block is used here, as long as the class is loaded, the code block will be executed and the Comparator object will be created directly. Later, there is no need to call the compare method through the external class to create the object, because the object has been created in advance. Compared to the compareTo method on, you have to manually create an external class object to call the compareTo method, which can be called directly here, which is faster. * / static {WritableComparator.define (IntWritable.class, new IntWritable.Comparator ());} / this inner class also implements the compare comparison method public static class Comparator extends WritableComparator {public Comparator () {super (IntWritable.class);} public int compare (byte [] b1, int S1, int L1, byte [] b2, int S2, int L2) {int thisValue = readInt (b1, S1) Int thatValue = readInt (b2, S2); return thisValue < thatValue?-1: (thisValue = = thatValue? 0: 1);}

The implementation of other short,long serialization classes is similar.

4. Custom serialization class

Main points:

(1) Writable interface must be implemented.

(2) there must be a no-parameter constructor because when deserialization requires reflection to call the no-parameter constructor.

(3) override serialization method

The serialization method for each base type is defined in the public void write (DataOutput out) throws IOException {/ / DataOutput interface. Here, take Long as an example: out.writeLong (upFlow); out.writeLong (downFlow); out.writeLong (sumFlow);}

(4) override deserialization method

Public void readFields (DataInput in) throws IOException {upFlow = in.readLong (); downFlow = in.readLong (); sumFlow = in.readLong ();}

(5) when serializing writes and deserializing reads, note that the order of writes and reads must be exactly the same.

(6) the toSting method can be overridden as needed to facilitate the content saved in the file.

(7) if the custom serialization class is used as a key in a key-value pair, because the MapReduce is sorted by key, then key comparison is involved. So you need to implement the Comparable interface. And the interface has to implement the compareTo method

Public int compareTo (Test o) {return (- 1 | 0 | 1); means less than, equal to, and greater than three results} 5. The property contains the serialization of the custom class.

First of all, the custom class in the property also needs to implement the serialization interface. So the following DateDimension and ContactDimension are serialized.

Public class ComDimension extends BaseDimension {private DateDimension dateDimension = new DateDimension (); private ContactDimension contactDimension = new ContactDimension (); / / Serialization can call the write method of the class directly, as follows: @ Override public void write (DataOutput dataOutput) throws IOException {this.dateDimension.write (dataOutput); this.contactDimension.write (dataOutput);} @ Override public void readFields (DataInput dataInput) throws IOException {this.dateDimension.readFields (dataInput) This.contactDimension.readFields (dataInput);}}

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.