Shulou(Shulou.com)06/03 Report--

Since becoming an architect (

After that, Li Dafang seems to have less motivation to learn, especially this year (there are some objective factors, of course).

Towards the end of the year, I was really ashamed and decided to learn from big data. Follow the forefront of the industry (in fact, it is no longer at the forefront.

), dreaming of having the power to have a dream one day.

Practice.

Start the virtual machine with CentOS (IP is 172.18.232.181), select hadoop-2.8.5 and hbase-2.0.2 according to the contents of the official document, and download and decompress zookeeper-3.4.11.

The following are the steps of the official documentation, familiar students can skip directly to the "nightmare start" ().

HDFS

Enter the decompressed directory of hadoop and set up the Java directory in etc/hadoop/hadoop-env.sh, as shown in the figure

Execute the. / sbin/start-dfs.sh script, as shown

Jps, and found that the startup is ready, as shown in the figure

Execute the. / bin/hdfs dfs-ls / command and enter the hdfs to have a look, as shown in the figure

Note: the first entry is empty, there is no hbase directory.

Zookeeper

Enter the zookeeper decompression directory and set the data directory in conf/zoo.cfg, as shown in the figure.

Execute the. / bin/zkServer.sh start command to start, as shown in the figure

Jps, and found that the startup is ready, as shown in the figure

Hbase

Enter the decompressed directory of hbase and set up the Java directory in conf/hbase-env.sh, as shown in the figure

Go to conf/hbase-site.xml to modify the content, as shown in the figure

Note: specify hdfs,zookeeper and cluster mode.

Execute the. / bin/start-hbase.sh script to start, as shown in the figure

Jps, start up, as shown in the picture

Execute the. / bin/hbase shell command to enter the shell interaction, as shown in the figure

You can create tables, insert data, delete data, and so on (not demonstrated here).

Finally, go to hdfs and find that hbase has stored data in it, as shown in the figure.

Li Dapang found that following the official documents all the way down very smoothly, he could not help but feel a double sense of achievement (is it a little superficial?

).

The nightmare begins.

As an old code farmer who has written Java for nearly ten years, how can Li Dafang give up his heart if he doesn't need Java to connect Hbase?

The seeds of nightmares are buried in the soil at this moment.

Start by creating a springboot, then introduce related dependencies, get connections, etc. (details will be tweeted next year, next year

? There's nothing wrong with it), it's done according to the official documentation.

Excited heart, trembling hand, clicked the run button. Why, it was reported correctly, and it started normally ().

Li Dapang thought that the time had come to witness a miracle. Click the call button on the page, as if the whole world is quietly waiting for the joy of seeing the result (). 1 second, 2 seconds, 3 seconds passed, no response, bad, estimated that there is a problem, quickly take a look at the Eclipse console, sure enough, wrong report

The seeds of nightmares have sprouted. The error is as follows:

Caused by: org.apache.hadoop.hbase.MasterNotRunningException: java.net.ConnectException: Call to localhost/127.0.0.1:16000 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/127.0.0.1:16000

The error is that Master is not running because access to localhost/127.0.0.1:16000 is denied.

Thanks to a little bit of Hbase, I know that 16000 is the default port of master (node) (the default port of master in the old version is 60000), but why this IP is native? Li Da Pang has some doubts in his heart. Hbase is obviously running in a virtual machine.

"Di Renjie" handling the case

This is like Di Renjie encountered a big case, and it is a strange case. At this time, the boss of Dige said that he must be steady in the face of a major case and should not be in disorder. Especially when the case is complicated and confusing in the early stage and gets into a stalemate, and you don't even know who your opponent is, you must have ignored some details in the course of handling the case. At this time, you need to recall carefully from the beginning. See if you can remember some clues (is it a little too deep in the play?

).

All right, let's review it with Li Dapang. When Hbase starts, it registers some information into zookeeper. We only configure the address of zookeeper in the Java program, so the program reads the address of master from zookeeper before visiting it.

Since the last person to visit is localhost/127.0.0.1:16000, it means that master registered it. Thinking that Dubbo uses IP when registering url with zookeeper, Li Da-fat thinks that master also registers IP with zookeeper, that is, 127.0.0.1 master 16000.

In order to verify my idea, I went into zookeeper and found that there was a master node, but there was no address information for it. Since it can't be verified, let's solve the problem. For now, it seems that all master needs to do is to adopt the actual IP registration of the machine on which it resides.

So I searched with Chinese keywords and found that you can set a parameter of hbase.master in the hbase-site.xml configuration file and add it as soon as possible, as shown in the figure:

After restarting Hbase, it is found that it will not work ().

The conjecture of Yuanfang

Li Dapang continued to think that this is a pseudo-cluster built, and the official document clearly states that the pseudo-cluster means that although there are multiple processes, they are all on the same machine. Could it be that Hbase detected that he was a pseudo-cluster at startup, so he always registered with 127.0.0.1 (please allow Li Dapang to take care of his own wishful thinking.

).

As the saying goes, mules are drawn out for a walk. First, move zookeeper to another virtual machine, restart Hbase, and find that it doesn't work (). Do you also want to move HDFS to the third virtual machine? Think about it, no, Hbase should not check the installation mode, single node / pseudo cluster / true cluster, it won't be so boring (). So give up the follow-up verification.

Li Dalang then thought that since the hbase.master configuration item can be specified in the configuration file of Hbase, I will try this configuration item in the Java program directly.

), it might work. But only if the master in the virtual machine can be accessed locally.

So enter the cmd window, execute telnet 172.18.232.181 16000, I go, unexpectedly can not get through (). Quickly ran to the virtual machine to execute telnet 172.18.232.181 16000, I went to, unexpectedly did not understand (). Now that you are in the virtual machine, try telnet 127.0.0.1 16000. Rub, it turns out that it works. Then I suddenly remembered that when I looked at the master log before, I always found that it was bound to 127.0.0.1, but it didn't attract much attention at that time. As shown in the picture

one disaster after another

Well, before the old problems are solved, new ones have emerged. Then solve the new problem, once again use Chinese keywords to search, there is no good answer.

Suddenly thought, or use English keywords to try, , all of a sudden I get to. It was found that this was a 2010 (

). I caught up eight years later, and I would like to thank the proposer and https://grokbase.com/t/hbase/user/103pq6p14k/master-binds-only-to-loopback of this question very much.

It was pointed out in the reply that the general algorithm for Hbase master binding is like this, which is the same as putting an elephant in a refrigerator.

):

1. Get the hostname (usually use the hostname command on posix systems)

2. Perform a DNS lookup on this hostname

3. Use the found IP as the bound IP

I suddenly remembered that when I was watching video learning a few years ago, I mentioned that after the Java program was deployed to Linux, many network problems were sometimes related to hostname.

Execute the hostname command quickly and find that the result is localhost, and the IP according to localhost is 127.0.0.1, so master is finally bound to 127.0.0.1 IP 16000.

So modify hostname to host1, and map host1 to native actual IP in / etc/hosts file, as shown in the figure:

Restart Hbase and check the master log. Finally, the bound IP has changed, as shown in the figure:

I am happy to think that all the problems have been solved. I hastened to use Java to adjust it again, and found that it was still a mistake in the first place. Because I empty the logs directory every time I restart Hbase, when I reboot after modifying hostname, I find that the log file name of zookeeper has changed. It used to end with localhost, but now it ends with a new hostname, as shown in the figure:

However, I found that the log file of master still ends with localhost. I wondered if it was because there was no reboot after the modification of hostname (I changed it in the file and modified it with the hostname command, but I was too lazy to restart it). Forget it, let's restart it (

). Restart Linux, and then start Hbase. Sure enough, the log file name of master has changed and ends with host1, as shown below:

Excited again, quickly use Java to try again, unfortunately, it is still the original formula, the original taste ().

The tongue ever turns to the aching tooth

To sum up the current situation, master is bound to the correct IP and port at startup, that is, 172.18.232.181 16000. However, the Java call is still the same as the original error, that is, access to 127.0.0.10 16000 is denied. It shows that although the server socket binding of master is correct when starting up, it is wrong to register with zookeeper and still uses 127.0.0.1 zookeeper 16000 (

).

Li Dafang wondered even more at this time that since the binding was all right, there was no reason to register in zookeeper. Why register 127.0.0.1 instead of the actual IP? This problem has been searched with Chinese keywords many times before, but it has not been solved.

Di Gong once said that in some cases, what it looks like on the surface is actually what it looks like. In some cases, the opposite is true, because someone is deliberately blinding your eyes (or maybe you are blinded by yourself.

).

So how to solve this problem? Or, more accurately, the current problem has not been identified at all, and the error we see is just a result (or phenomenon).

Einstein

Einstein once said, "it is more important to ask a problem than to solve it." He also said, "imagination is more important than knowledge." When his famous quotes are translated into Chinese, it feels like "there is nothing impossible, only the unexpected".

)

When Li Dapang was at his wit's end, a flash of inspiration appeared (). Wait, wait,

You can see that a backup master is registered first, and then the backup master is deleted because it is registered as an active master. And use localhost instead of IP when registering. Our Java program gets localhost instead of 127.0.0.1 as we first thought.

So why did it finally become 127.0.0.1, because the hosts file of the Windows system also maps localhost to 127.0.0.1, which results in the final request for a connection to 127.0.0.1 localhost 16000 and is rejected, which is the error you saw at first. At this point, the truth has come out.

The reason why it has always been thought that master uses IP when registering with zookeeper is influenced by Dubbo (of course, you can't blame Dubbo here, only yourself).

Dense willow trees and bright flowers

The question has been found, this time directly use English keywords to search, it is a sudden get to the answer. The solution is simple: add a hbase.master.hostname configuration item (https://stackoverflow.com/questions/9615707/hbase-how-to-specify-hostname-for-hbase-master) to hbase-site.xml, as shown in the figure:

The answers given by foreigners are generally trustworthy. Restart Hbase.

According to Li Dafang's inference, the error that should be reported when using Java calls at this time is that host1 cannot be parsed, because Windows does not know what host1 is. Then try to adjust it, , it is like this, as shown in the picture:

Then modify the hosts file of Windows, add host1, and map it to the IP of the virtual machine, as shown in the figure:

Call it again. It's already through.

Finally, take a look at the log file of maser to verify the content of master when registering with zookeeper, as shown in the figure:

You can see that it has become host1. So far, all the problems have been solved.

PS: although it is said that theory + practice can produce results, do not rush to practice without looking at it, as you will stumble over some mentally retarded problems. I found it easy afterwards, but I spent a lot of time here. But also easy to make their own self-confidence suffered a serious blow, and even doubt life (

).

Related articles

Five minutes for easy understanding of Hbase determinant storage

(end)

A new theory of programming

Talking about technology from a unique perspective

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.