Shulou(Shulou.com)05/31 Report--

This article will explain in detail the sample code about WritableUtils in hadoop-common. The editor thinks it is very practical, so I share it with you for reference. I hope you can get something after reading this article.

Hadoop encapsulates the basic type of java and encodes the integer into fixed length format and variable length format. The variable length format uses a more flexible coding method, which can save space storage for smaller numbers (especially negative numbers).

VIntWritablepublic class VIntWritable implements WritableComparable {private int value;//getter / / setter @ Override public void readFields (DataInput in) throws IOException {value = WritableUtils.readVInt (in);} @ Override public void write (DataOutput out) throws IOException {WritableUtils.writeVInt (out, value);}} WritableUtils.writeVLong public static void writeVInt (DataOutput stream, int I) throws IOException {writeVLong (stream, I);} public static void writeVLong (DataOutput stream, long I) throws IOException {if (I > =-112 & I > 8; len-- } stream.writeByte ((byte) len); len = (len

< -120) ? -(len + 120) : -(len + 112);for (int idx = len; idx != 0; idx--) { int shiftbits = (idx - 1) * 8; long mask = 0xFFL >

Shiftbits); stream.writeByte ((byte) ((I & mask) > > shiftbits));}}

If I is between 122and 127, it is directly converted to byte type storage.

If I is less than-112, convert it to a positive number (XOR-1L), set the identification amount len to-120; otherwise, len is-112

Shift the data to be stored, while len subtracts itself (len not only marks the amount, but also counts the number of shifts).

Writes the identification amount to the output stream.

Reset len and set len to the number of shifts.

Carry on the loop, write the data every 8 bits to the output stream (big-end mode), and analyze the for loop in detail.

WritableUtils.readVLong public static long readVLong (DataInput stream) throws IOException {byte firstByte = stream.readByte (); int len = decodeVIntSize (firstByte); if (len = = 1) {return firstByte;} long I = 0 for (int idx = 0; idx)

< len-1; idx++) { byte b = stream.readByte(); i = i = -112) { return 1; } else if (value < -120) { return -119 - value; }return -111 - value; } public static boolean isNegativeVInt(byte value) {return value < -120 || (value >

=-112 & & value < 0);}

Read a byte type

It is determined that if the readout data is greater than-112, it means it is not a sign, and the original data can be returned directly. If it is less than-120 or between [- 112 ~-120], it means that the number of bits returned needs to be determined.

By getting the number of bits to move, 8 bits at a time, XOR shift. Restore the data.

Judging the representation quantity, if it is less than-120 or between [0 ~-112], it is proved to be negative, and the resulting data will be XOR-1L to get the final value.

This is the end of this article on "sample code for WritableUtils in hadoop-common". I hope the above content can be helpful to you, so that you can learn more knowledge. if you think the article is good, please share it out for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.