Shulou(Shulou.com)06/03 Report--

Summary of storm issues (continuously updated)

1. Yaml is different from the property configuration file we usually use, and its requirements are more stringent, so we must pay attention to it when adding configuration to conf/storm.yaml.

For example, you must pay attention to the start position and the space after the colon, otherwise the configuration will not take effect. There are a lot of materials related to yaml that can be referenced on the Internet.

To check whether the configuration is in effect, you can use the command: storm localconfvalue configuration keyword

But this command only works on nimbus, and what you see in supervisor is the default value. I don't know why.

two。 When deploying storm nodes, you need to install jzmq and 0qm, and after installing these two dependent packages, you need to execute sudo-u root ldconfig. Otherwise, an exception will occur:

2012-02-24 16:30:30 worker [ERROR] Error on initialization of server mk-worker

Java.lang.UnsatisfiedLinkError: / usr/local/lib/libjzmq.so.0.0.0: libzmq.so.1: cannot open shared object file: No such file or directory

At java.lang.ClassLoader$NativeLibrary.load (Native Method)

At java.lang.ClassLoader.loadLibrary0 (ClassLoader.java:1803)

At java.lang.ClassLoader.loadLibrary (ClassLoader.java:1728)

At java.lang.Runtime.loadLibrary0 (Runtime.java:823)

At java.lang.System.loadLibrary (System.java:1028)

At org.zeromq.ZMQ. (ZMQ.java:34)

Here is a related discussion:

Http://groups.google.com/group/storm-user/browse_thread/thread/656fb21d7166574d

3. Close nimbus-related processes:

Kill `ps aux | egrep'(daemon\ .nimbus) | (storm\ .UI\ .core)'| fgrep-v egrep | awk'{print $2}'`

4. Kill all storm processes on supervisor:

Kill `ps aux | fgrep storm | fgrep-v 'fgrep' | awk' {print $2}'`

5. At present, storm does not seem to support dynamic loading of log4j configuration files, and each supervisor node needs to modify the log4j/storm.log.properties file to adjust the log output.

An exception that cannot serialize topologies occurs when publishing log4j.Logger

The reason is that the log system fails to pay for serialization correctly.

Solution: use slf4j instead of log4j.

6. When packaging a toplogy project, if you use the assembly method, the configuration for related dependencies is generally as follows:

/

True

Storm:storm

Wiki says you can use compile. Then set the storm dependency to runtime, which doesn't seem to work. In addition, all dependent packages will be unzipped and all dependent configurations and class files will be generated into a file. This is controlled by the true parameter.

7. Sometimes the setting of the conf/storm.yaml parameter of supervisor is unreasonable, which will lead to the phenomenon that worker can not get up.

For example, one example I came across is. Worker.childopts is defined (an unreasonable jvm parameter is given). As a result, worker starts repeatedly without success, and the log information here is not clear, just that it cannot be start, but no more detailed context information is given.

Backtype.storm.daemon.worker mytoplogy-12-1330051497 fdd0effd

-0611-427a-8e23-84fe25047dbd 6701 d7f8354d-9ec0-4272-aa0d-9260f7b0042f

2012-02-24 14:24:27 supervisor [INFO]

A17ff23e-00b2-4e51-9001-74664413d2dd still hasn't started

2012-02-24 14:24:28 supervisor [INFO]

A17ff23e-00b2-4e51-9001-74664413d2dd still hasn't started

2012-02-24 14:24:28 supervisor [INFO]

A17ff23e-00b2-4e51-9001-74664413d2dd still hasn't started

2012-02-24 14:24:29 supervisor [INFO]

A17ff23e-00b2-4e51-9001-74664413d2dd still hasn't started

8. If you have any questions, go to storm-user to search for related posts or ask the author of Storm, Nathan Marz. The man was very enthusiastic in answering questions. Judging from the id of the questioner, it seems that a lot of Chinese people ask questions on it.

9. A Nimbus host is not set exception occurred when publishing topologies to a remote cluster

The reason is that Nimbus is not started correctly, maybe the storm.yaml file is not configured, or there is a problem with the configuration.

Solution: open the storm.yaml file and configure it correctly: nimbus.host: "xxx.xxx.xxx.xxx" and restart the nimbus daemon.

The following exception may sometimes occur when submitting a topology:

Exception in thread "main" java.lang.IllegalArgumentException: Nimbus host is not set

At backtype.storm.utils.NimbusClient. (NimbusClient.java:30)

At backtype.storm.utils.NimbusClient.getConfiguredClient (NimbusClient.java:17)

At backtype.storm.StormSubmitter.submitJar (StormSubmitter.java:78)

At backtype.storm.StormSubmitter.submitJar (StormSubmitter.java:71)

At backtype.storm.StormSubmitter.submitTopology (StormSubmitter.java:50)

At com.taobao.kaleidoscope.storm.IcdbTopology.main (IcdbTopology.java:59)

However, there is no problem with starting nimbus. This is mainly because the inner conf_dir path is set incorrectly. You need to add this sentence to the bin/storm script:

CONF_DIR = STORM_DIR + "/ conf"

10. How do you know the source of a tuple?

A:tuple.getSourceStramId ()

11. Masters, who saw @ Love on Taobao Technology Salon, mentioned that there was a memory leak in version 0.6.x of storm, while version 0.7.0 has been solved. It seems that we haven't encountered it yet, so we are considering whether to upgrade to the latest version.

twelve。 AlreadyAliveException (msg: xxx is already active) exception occurred when publishing topologies to a remote cluster

The reason is that the topology provided has the same name as the topology that is already running.

Solution: just change the topology name when you publish

13. A null pointer exception occurred when publishing a drpc type topologies to a remote cluster and failed to connect to the drpc server

The reason is that the drpc server address is not configured correctly.

Solution: add the drpc server configuration to the conf/storm.yaml file and start all the drpc services specified in the configuration file. The contents are as follows:

Drpc.servers:

-"drpc Server ip"

13. When the client invokes the drpc service, Failing message appears in the log of worker, but bolt does not receive data.

The error log is as follows:

2011-12-02 09:59:16 task [INFO] Failing message backtype.storm.drpc.DRPCSpout$DRPCMessageId@3770bdf7: source: 1:27, stream: 1, id: {- 5919451531315711689 task, [foo.com/blog/1, {"port": 3772, "id": "5", "host": "10.0.0.24"}]

Solution: incorrect configuration of hostname, domain name and hosts files will cause such errors.

Check and modify the hostname, domain name and hosts file of storm-related machines. Restart the network service: service network restart. Restart storm and invoke the drpc service again, successfully.

The Hosts file must contain the following:

[nimbus host ip] [nimbus hostname] [nimbus host alias]

[supervisor host ip] [supervisor hostname] [supervisor host alias]

[zookeeper host ip] [zookeeper hostname] [zookeeper host alias]

14. When bolt is processing messages, Failing message appears in the log of worker

The reason may be due to the message processing timeout of Topology.

Solution: set the appropriate message timeout when submitting the Topology, which is longer than the default message timeout (30 seconds).

Conf.setMessageTimeoutSecs (60)

15. Storm startup Times no jzmq in java.library.path error

The reason is that jzmq cannot be found, and by default, those .so files are installed in / usr/local/lib when executing install_zmq.sh, but may be installed in another path during the actual installation.

Workaround: add to storm.yaml:

Java.library.path: "/ opt/storm/jzmq/lib:/opt/storm/zeromq/lib:/usr/local/lib:/opt/local/lib:/usr/lib"

16. Encountered a make error for No rule to make target `classdist_noinst.stamp' while installing jzmq

Specific make error message:

Make [1]: * No rule to make target `classdist_noinst.stamp'

Needed by `org/zeromq/ZMQ.class'. Stop.

Workaround: manually create an empty classdist_noinst.stamp file.

Touch src/classdist_noinst.stamp

17. An make error for cannot access org.zeromq.ZMQ was encountered while installing jzmq

Specific make error message:

Error: cannot access org.zeromq.ZMQ

Class file for org.zeromq.ZMQ not found

Javadoc: error-Class org.zeromq.ZMQ not found.

Solution: compile manually and then re-make to pass.

Cd src

Javac-d. Org/zeromq/*.java

Cd..

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.