How to read and write by SequenceFile
Editor to share with you how to achieve SequenceFile reading and writing, I hope you will learn something after reading this article, let's discuss it together!
SequenceFile read
Public static void main (String [] args) throws IOException {Configuration conf = new Configuration (); FileSystem fs = FileSystem.get (conf); Path seqFile = new Path ("/ user/hive/warehouse/abc/seqfile.seq"); SequenceFile.Reader reader = new SequenceFile.Reader (fs, seqFile, conf) IntWritable key = new IntWritable (); Text value = new Text (); while (reader.next (key, value)) {System.out.println (key); System.out.println (value) } IOUtils.closeStream (reader);}
After declaring the Reader instance of the sequential file, call the next () method to iterate over the record. Finally, you need to shut down the reader instance.
Returns true; if the key-value pair is read successfully, or false if the end of the file has been read.
SequenceFile write
Public static void main (String [] args) throws IOException {Configuration conf = new Configuration (); FileSystem fs = FileSystem.get (conf); Path targetPath = new Path ("/ user/hive/warehouse/test_url"); final Option optPath = SequenceFile.Writer.file (targetPath); final Option optKeyClass = SequenceFile.Writer.keyClass (Text.class) Final Option optValueClass = SequenceFile.Writer.valueClass (BytesWritable.class); final SequenceFile.Writer writer = SequenceFile.createWriter (conf, optPath, optKeyClass, optValueClass); final Collection listFiles = FileUtils.listFiles (new File ("/ data1/url/"), new String [] {"log"}, false); Text key = null; BytesWritable value = null For (File file: listFiles) {key = new Text (file.getPath ()); value = new BytesWritable (FileUtils.readFileToByteArray (file)); writer.append (key, value);} IOUtils.closeStream (writer);}
Create the SequenceFile object through the CreateWriter () static method and return the SequenceFile.Writer instance.
Once you have an instance of SequenceFile.Writer, you can append data to the end of the file through the append () method.
Finally, close the instance.
After reading this article, I believe you have some understanding of "how to read and write SequenceFile". If you want to know more about it, you are welcome to follow the industry information channel. Thank you for reading!
