Shulou(Shulou.com)06/02 Report--

Metadata, as the source of big data, plays a very important role. But there's a secret hidden in Impala? And metadata has a deep origin, let's trace it together!

I. Impala architecture

(1) Each slave node runs an Impala process that collaborates with HDFS DataNode processes.

(2) Two other processes run on the master node to support query execution.

1. State Store process: provide check ××× for ImpalaDaemon, and periodically check Impala process status

Catalog process: synchronize metadata changes to all ImpalaDaemons

How Impala executes queries

(1) Impala daemon query execution process

The client (Impala-shell or Hue) connects to the Impala daemon, which is the Coordinator. Coordinator requests other Impala daemons from State Store, distributes queries to other Impala daemons, and returns results to clients.

III. Metadata cache

Based on Impala's query capabilities, we know that there is a cache of metadata in Impala. Metadata is stored in Hive Metastore, so what is the role of metadata cache? When Impala is started, metadata is cached to the node of Impala daemon. When the Impala daemon initiates metadata changes, such as creating tables, deleting tables, or modifying table definitions, these are cached. If we query the table information, we can find it from the cache and query it directly in the node, so there is a metadata cache node used as a cache in each Impala daemon.

When the Impala process changes metadata, it prompts the catalog service, which notifies all Impala daemons to update the cache.

IV. External changes and metadata caching

What we talked about above are internal changes, but many times there will be external changes, which are unknown to Impala, such as:

Hive, Hcatalog, or HueMetadata Manager changed metadata

(2) Data added directly to HDFS directory

External changes will cause the Impala metadata cache to become unavailable and we will have to manually refresh or validate the Impala metadata cache.

Of course, there are different ways to update this change depending on the content and nature of the change, such as:

If you have questions, welcome to spit out, pay attention to Weixin Official Accounts "Big Data cn," keep going with you every day, and there are more resources waiting for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.