In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-01-19 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Servers >
Share
Shulou(Shulou.com)06/01 Report--
This article mainly introduces "how to read protobuf data in hadoop". In daily operation, I believe many people have doubts about how to read protobuf data in hadoop. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubt of "how to read protobuf data in hadoop". Next, please follow the editor to study!
When reading protobuf data in mapreduce, you will encounter a pit
BytesWritable . When you save the byte [] data to BytesWritable, the data read through BytesWritable.getBytes () is not necessarily the original data, but may be much longer. This is because BytesWritable uses the automatic memory growth algorithm. When the length of the data you save is size, it may save the data to the buffer with the length of capacity (capacity > size). In this case, the last few characters of the data you get through BytesWritable.getBytes () are redundant. If the protobuf serialized data is stored in it, it cannot be deserialized, so you can use BytesWritable.setCapacity (BytesWritable.getLength ()) to remove the extra space behind it, and then you can use protobuf to reverse the sequence to the original data.
The key code is as follows
Conf.setInputFormat (SequenceFileAsBinaryInputFormat.class); public void map (Object key, BytesWritable values, OutputCollector output, Reporter reporter) throws IOException {values.setCapacity (values.getLength ()); / / very important!! Byte [] data = values.getBytes (); / / more code} at this point, the study on "how to read protobuf data in hadoop" is over, hoping to solve everyone's doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.