Shulou(Shulou.com)06/03 Report--

As a project of big data, Avro schema can be regarded as the best of the fine works. People who study or want to learn big data are fascinated by it and keep digging and learning, so what kind of mysterious veil does it have? Let's uncover it together!

To understand Avro, you must first understand serialization. Serialization is a way to express data in memory. It is a series of bytes that can be saved to disk or sent over the network. Deserialization allows you to read the data back to memory.

For a practical example, how do we serialize the number 108125150? You can specify some specific types: (1) 4 bytes; when stored as Java int type (2) 9 bytes when stored as Java string type.

Many programming languages and libraries support serialization, such as Serializable in Java or pickle in Python. But backward compatibility and cross-language support can be a challenge for us, and Avro was developed to address these challenges.

What is Apache Avro?

The Avro data file format is only part of the Avro project. It is an efficient data serialization framework, a top-level Apache project created by Doug Cutting, and is widely supported in Hadoop and its ecosystem. The biggest feature is compatibility that reads and writes data in Java, C, C++, C #, Python, PHP, and other languages without sacrificing performance. Avro also supports RemoteProcedure Calls (RPC), which can be used to build custom network protocols, and Flume uses it for internal communication.

Types supported by AvroSchemas

(1) simple type:

(2) complex types

Here the record type is the most important, and the other types are mainly used to define the fields of record.

Basic schema example

Here we first quote a SQL CREATE TABLE statement.

Then we use Avro schema to express the same statement

Specify default values in Schema

Avro supports setting default values in schema, which is used when no field value is explicitly specified, similar to SQL.

AvroSchemas and null values

(1) Avro checks for null values when serializing data

(2) Null can be used only when it is explicitly specified in schema.

Schema examples of complex types

Example: record with enum and string array types

Comment schema

Annotating schema is a good way to avoid ambiguity

(1) all types support the addition of optional doc attribute

Avro container format

Avro defines a container file format to store Avro records, also known as the Avro data file format. Similar to Hadoop SequenceFile format, supports cross-language data reading and writing. It also supports the compression of block records, and the compressed data can be sliced. In addition, this format is self-describing, each file contains a copy of schema for writing data, and all records must use the same schema in the file.

Check the Avro data file using the Avro tool

Avro data file is an efficient way to store data, however, the binary format makes debug very inconvenient. Using the avro-tools command to manipulate binary files, you can read schema or data in Avro files.

After digging, did you find the treasure? If found, this is definitely your private property, if not, it does not matter, because there is still some good news waiting for you to come! Shanda Internet mogul personally sit in charge, share big data practical information, come together! Follow the Wechat official account "big data cn". Let's communicate together.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.