Shulou(Shulou.com)06/01 Report--

This article mainly explains "what is the bitcoin blk.dat file format and reading tool". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn what the bitcoin blk.dat file format and reading tool is.

The blk.dat file in the ~ / .bitcoin / blocks/ directory of the bitcoin node (Bitcoin Core) contains the original block data received by the node. These data files form the basis of the bitcoin block chain, and are also the most efficient way to analyze and obtain bitcoin block and transaction data.

1. The working mechanism of blk.dat

Each chunk received by a bitcoin node (Bitcoin Core) is written to a blk.dat file. For performance reasons, instead of writing all chunks to a single jumbo file, the node splits it into multiple blk*.dat files.

Bitcoin / blocksblk00000.datblk00001.datblk00002.dat...

The node first writes the received chunk to blk00000.dat, then if the file is full (about 128MB), it writes blk00001.dat, then blk00002.dat..., and so on.

2. Example of blk.dat file content

The data in the blk.dat file is stored in binary format, and the new chunk received by the node is appended to the end of the file. We can take a look at the contents of the Genesis Block. Read the first 293 bytes of blk00000.dat:

F9beb4d91d01000001000000000000000000000000000000000000000000000000 00000000000000000000003ba3edfd7a7b12b27ac72c3e67768f617fc81bc3888a51323a9fb8aa4b1e5e4a29ab5f49ffff001d1dac2b7c0101000000010000000000000000000000000000000000000000000000000000000000000000ffffffff4d04ffff001d0104455468652054696d65732030332f4a616e2f32303039204368616e63656c6c6f72206f6e206272696e6b206f66207365636f6e64206261696c6f757420666f722062616e6b73ffffffff0100f2052a01000000434104678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac000000003, blk file structure

The above data can be divided into five parts:

Magic bytes (Magic Byte) and size (data size) can be used to find out the starting and ending position of each block.

Block header (block header) contains the header information of the block

Tx count represents the number of transactions in the block, and the following transaction data section is the data of transaction by transaction.

The data structure is as follows:

[magic bytes] [size] [block header] [tx count] [transaction data]

Using the size field to analyze the above data, we know that it takes 293 bytes to read the entire block: the binary code of size is 1d010000, which is processed as follows:

Convert the byte order to get 0000011d

Convert to base 10 and get 285.

So we add 4 bytes of magic bytes and 4 bytes of size, and we get 293 bytes of this block.

4. Note 4.1 blocks are not downloaded sequentially

If you are parsing blk.dat files, it is important to remember that blocks are not necessarily sorted in order. For example, you may read blocks in the following order when you are working with a file:

A B C E F D

This is because for performance reasons, bitcoin nodes download blocks in parallel.

4.2 blk.dat file maximum 128MB

This limitation is controlled by the macro Max _ BLOCKFILE_SIZE in the source code.

5. Block file reading tool

As mentioned earlier, the data in the blk.dat file is binary encoded, so if you open the file directly in a text editor, you may not see much valuable information. There are several tools that can help you.

5.1 od-hexadecimal viewing tool

Od is a simple tool that exports the contents of a file to the format of your choice. For example:

Od-x-endian=big-N293-An blk00000.dat

Where:

-x: displayed as hexadecimal

-- endian=big: displays bytes in big endian order

-N 293: declare the number of bytes to read

-An: do not display file offset

Od is usually built into various linux distributions.

5.2 hexdump-hexadecimal and ascii viewing tools

Similar to od, but hexdump can display the ascii text of the data, which is convenient for quickly viewing the information contained in the transaction. For example:

$hexdump-C-s 8-n 285 blk00000.dat00000008 01 000000 000000 000000 |. | 000000180000000000000000000000000000028000000003b a3 ed fd 7a 7b 12b2 7a c7 2c 3e |. ...z {.. z. > | 00000038 67 76 8f 61 7f c8 1b c3 88 8a 51 32 3a 9f b8 aa | gv.a.Q2:... | 00000048 4b 1e 5e 4a 29 ab 5f 49 ff ff 00 1d 1d ac 2b 7c | K ^ J}. _ I. + | 00000058 01 01 01 000 000 000. | | 00000078 000000 000000 ff 4d 04 ff ff 00 1D | .m. | 00000088 01 04 45 54 68 20 20 69 6d 65 73 20 30 33 2f | .000098 4a 61 6e 2f 32 30 39 20 43 68 63 65 6c | Jan/2009 Chancel | 000000a8 6c 6f 72 20 6f 6e 20 62 69 6e 6b 20 6f 66 20 | lor on brink of | 000000b8 73 65 63 6f 6e 64 20 62 61 69 6c 6f 75 74 20 66 | second bailout f | 000000c8 6f 72 20 62 61 6e 73 ff ff ff ff 01 00 f2 05 | or banks. | 000000d8 2a 01 000000 43 41 04 67 8a fd b0 fe 55 48 27 | *.... CA.g....UH' | 000000e8 19 67 f1 a6 71 30 b7 10 5c d6 a8 28 e0 39 09 a6 | .g. Q 0..\. | 000000f8 79 62 e0 ea 1f 61 de b6 49 f6 bc 3f 4c ef 38 c4 | yb... A..I..?L.8. | 00000108 f3 55 04 e5 1e C1 12 de 5c 38 4d f7 ba 0b 8d 57 | .U.\ 8M....W | 00000118 8a 4c 70 2b 6b f1 1d 5f ac 0000 0000 | .LP + k.. ac. |) 0000125

Where:

-C: display bytes and ascii text

-s: declare the offset location of the data to be displayed

-n: declare the number of bytes to read

It can also refer to displaying raw hexadecimal data:

$hexdump-C-s 8-n 285 blk00000.dat | cut-c 11-58 | tr'\ n'| tr-d''

Where:

Cut-c 11-58: keep only 11 to 58 columns of each row

Tr'\ n': translate line breaks into spaces

Tr-d'': delete all spaces

Hexdump is usually built into various linux distributions.

5.3 BlockETL-load block and transaction data into the SQL database

Unlike the first two simple tools, the BlockETL package is used for data extraction, conversion, and loading in Bitcoin blockchain data analysis, that is, BlockETL is a fundamental tool in the larger blockchain data analysis ecology, which is most valuable when you read bitcoin blk.dat files for further analysis of the data.

The main features of BlockETL are as follows:

Read the original block file directly and extract it quickly.

Load the original block and transaction data into the SQL database to facilitate subsequent block chain data analysis

Multi-pipeline parallel processing is supported, and the number of concurrent pipelines can be controlled.

The structure design is clear, and the code is easy to expand.

BlockETL is not only a Java application, but also a development kit that supports a variety of operating systems such as Windows, OSX and Linux. After configuring basic information such as data catalogs, database connections, and starting, you can see the progress of loading chunks and transaction data from the blk file into the SQL database:

Thank you for your reading, the above is the content of "what is the bitcoin blk.dat file format and reading tools". After the study of this article, I believe you have a deeper understanding of what the bitcoin blk.dat file format and reading tools are, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.