How to read protobuf data in hadoop

This article mainly introduces "how to read protobuf data in hadoop". In daily operation, I believe many people have doubts about how to read protobuf data in hadoop. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubt of "how to read protobuf data in hadoop". Next, please follow the editor to study!

When reading protobuf data in mapreduce, you will encounter a pit

BytesWritable . When you save the byte [] data to BytesWritable, the data read through BytesWritable.getBytes () is not necessarily the original data, but may be much longer. This is because BytesWritable uses the automatic memory growth algorithm. When the length of the data you save is size, it may save the data to the buffer with the length of capacity (capacity > size). In this case, the last few characters of the data you get through BytesWritable.getBytes () are redundant. If the protobuf serialized data is stored in it, it cannot be deserialized, so you can use BytesWritable.setCapacity (BytesWritable.getLength ()) to remove the extra space behind it, and then you can use protobuf to reverse the sequence to the original data.

The key code is as follows

Conf.setInputFormat (SequenceFileAsBinaryInputFormat.class); public void map (Object key, BytesWritable values, OutputCollector output, Reporter reporter) throws IOException {values.setCapacity (values.getLength ()); / / very important!! Byte [] data = values.getBytes (); / / more code} at this point, the study on "how to read protobuf data in hadoop" is over, hoping to solve everyone's doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!