Shulou(Shulou.com)06/01 Report--

This article focuses on "the use of locks in Hive". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn the use of locks in Hive.

Locks in Hive

When performing insert into or insert overwrite tasks, manually stopping the program halfway will lead to a deadlock (unable to submit MapReduce). You can only perform query operations, but drop insert operations are inoperable, and will remain stuck no matter how long it takes.

To view the deadlock in Hive, you can use show locks [table] to view it.

Clipboard

You can see the EXCLUSIVE under the Type inside, which is a mutex that needs to be resolved, otherwise subsequent queries and insertion tasks will be affected.

There are two kinds of locks in hive, shared lock Shared (S) and mutex Exclusive (X)

Lock SXS whether X No No

The basic mechanism of locking is:

Changes to meta-information and data require mutexes

Data reading requires a shared lock

Operations that trigger a shared lock can be performed concurrently, but if a mutex is triggered, the table and the partition cannot execute jobs concurrently.

Wechat screenshot _ 20201207211428

For the above case, use the unlock command:

Unlock table tableName

Note: table lock and partition lock are two different locks. Unlocking the table is not valid for the partition, and the partition needs to be unlocked separately.

Unlocking method

Check if the table is locked:

Show locks tableName

General unlocking method:

Unlock table table name;-- unlock the table

Unlock table table name partition (dt='2014-04-01');-- unlock a partition

When high-version hive inserts data by default, it cannot be queried because there is a lock

Possible problems

The road to unlocking is usually not plain sailing, and various problems may be encountered. The author tested it under Hive2.1.1, such as:

Clipboard3

This command cannot be executed, saying that LockManager is not specified, so you need to execute the command:

Set hive.support.concurrency=true

Set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager

If you re-execute it in this way, the command can be executed.

If you return it! Yes! No, no! OK, the ultimate method, which can be executed directly to MySQL metadata:

Select * from HIVE_LOCKS where HL_DB=?

Find all the locks, then delete the corresponding locks according to the conditions, and the locked table can be released.

Delete from HIVE_LOCKS where HL_DB = 'cdn' and HL_TABLE =' cdn_log'

Note: both table names and fields need to be capitalized.

In this way, the lock problem can usually be solved completely.

At this point, I believe you have a deeper understanding of "the use of locks in Hive". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.