Shulou(Shulou.com)05/31 Report--

How to achieve CloudStack High Availability source code analysis, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain in detail for you, people with this need can come to learn, I hope you can gain something.

Let's first look at the inner class PingTask of DirectAgentAttache. First of all, we need to know that every host registered in CS has a corresponding DirectAgentAttache, which means that every HOST has a PingTask thread running in the background, and the time interval is specified by the global variable ping.interval. The default is 60s.

Let's take a look at PingTask's code.

ServerResource resource = _ resource;if (resource! = null) {PingCommand cmd = resource.getCurrentStatus (_ id); int retried = 0; while (cmd = = null & & + + retried

_ id stands for host_id. When getCurrentStatus returns the correct cmd, it means that you can Ping the host, and then execute _ agentMgr.handleCommands

Public void handleCommands (final AgentAttache attache, final long sequence, final Command [] cmds) {for (final Pair listener: _ cmdMonitors) {final boolean processed = listener.second () .processCommands (attache.getId (), sequence, cmds);}}

We care about BehindOnPingListener. Let's take a look at its processCommands method.

@ Overridepublic boolean processCommands (final long agentId, final long seq, final Command [] commands) {final boolean processed = false; for (final Command cmd: commands) {if (cmd instanceof PingCommand) {pingBy (agentId);}} return processed;}

Next up is the pingBy method.

Public void pingBy (final long agentId) {/ / Update PingMap with the latest time if agent entry exists in the PingMap if (_ pingMap.replace (agentId, InaccurateClock.getTimeInSeconds ()) = = null) {s_logger.info ("PingMap for agent:" + agentId + "will not be updated because agent is no longer in the PingMap");}}

The point here is this _ pingMap, we can see that it is actually a ConcurrentHashMap,key is an agentId (such as hostId), and value is a timestamp, that is, when we Ping this time, we will insert the current time into _ pingMap as a value. Let's recall that PingTask is performed every ping.interval interval, so if our host is running normally, then _ pingMap will be updated almost every ping.interval. (of course, there will be a certain delay in the execution of the getCurrentStatus method.) if the network is unable to connect due to a sudden failure of the host, the time in _ pingMap will remain at the timestamp of the last Ping.

So let's sum up the logic of PingTask: Ping our hosts every ping.interval (default, 60s). If we can Ping, update the value in _ pingMap to the current timestamp, otherwise do nothing.

The other background thread we're going to look at is MonitorTask, which is also executed every other ping.interval, starting with the method findAgentsBehindOnPing

Protected List findAgentsBehindOnPing () {final List agentsBehind = new ArrayList (); final long cutoffTime = InaccurateClock.getTimeInSeconds ()-getTimeout (); for (final Map.Entry entry: _ pingMap.entrySet ()) {if (entry.getValue ())

< cutoffTime) { agentsBehind.add(entry.getKey()); } } return agentsBehind; } protected long getTimeout() { return (long) (PingTimeout.value() * PingInterval.value()); } 全局变量ping.timeout默认值是2.5，这段代码的意思就是找出上一次Ping通的时间距离现在超过ping.interval的2.5倍的主机，简单讲就是Ping不通或者Ping通的延时超过我们认为的不合理时间的主机。 正常情况下该方法返回的会是一个空的List，这个时候MonitorTask就结束当前任务。但是如果出现网络延时或者主机故障的时候，就要执行接下来的代码。 final List behindAgents = findAgentsBehindOnPing();for (final Long agentId : behindAgents) { final QueryBuilder sc = QueryBuilder.create(HostVO.class); sc.and(sc.entity().getId(), Op.EQ, agentId); final HostVO h = sc.find(); if (h != null) { final ResourceState resourceState = h.getResourceState(); if (resourceState == ResourceState.Disabled || resourceState == ResourceState.Maintenance || resourceState == ResourceState.ErrorInMaintenance) { disconnectWithoutInvestigation(agentId, Event.ShutdownRequested); } else { final HostVO host = _hostDao.findById(agentId); if (host != null && (host.getType() == Host.Type.ConsoleProxy || host.getType() == Host.Type.SecondaryStorageVM || host.getType() == Host.Type.SecondaryStorageCmdExecutor)) { disconnectWithoutInvestigation(agentId, Event.ShutdownRequested); } else { disconnectWithInvestigation(agentId, Event.PingTimeout); } } }} 我们假设出问题的是一台计算节点，那么一路往下将要执行的将是AgentManagerImpl的handleDisconnectWithInvestigation方法 protected boolean handleDisconnectWithInvestigation(final AgentAttache attache, Status.Event event) { final long hostId = attache.getId(); HostVO host = _hostDao.findById(hostId); if (host != null) { Status nextStatus = null; nextStatus = host.getStatus().getNextStatus(event); if (nextStatus == Status.Alert) { Status determinedState = investigate(attache); if (determinedState == null) { if ((System.currentTimeMillis() >

> 10)-host.getLastPinged () > AlertWait.value () {determinedState = Status.Alert;} else {return false;}} final Status currentStatus = host.getStatus (); if (determinedState = = Status.Down) {event = Status.Event.HostDown } else if (determinedState = = Status.Up) {agentStatusTransitTo (host, Status.Event.Ping, _ nodeId); return false } else if (determinedState = = Status.Disconnected) {if (currentStatus = = Status.Disconnected) {if ((System.currentTimeMillis () > > 10)-host.getLastPinged () > AlertWait.value ()) {event = Status.Event.WaitedTooLong;} else {return false }} else if (currentStatus = = Status.Up) {event = Status.Event.AgentDisconnected;} handleDisconnectWithoutInvestigation (attache, event, true, true); host = _ hostDao.findById (hostId); / / Maybe the host magically reappeared? If (host! = null & & host.getStatus () = = Status.Down) {_ haMgr.scheduleRestartForVmsOnHost (host, true);} return true;}

Let's take a look at the last if of this method, that is, under certain conditions, our ultimate goal is to restart all virtual machines on the host, which is the real purpose of HA. But we have to remember that the premise for us to enter this handleDisconnectWithInvestigation method is actually very simple, that is, as long as we find that it takes more than, say, two and a half minutes since the last time the Ping connected to the host, we should actually execute the HA when we are very sure that the host is indeed down. So the front pile of the method is repeatedly confirming the status of the host, as shown in the method name Inverstigation (survey). We assume that the currentStatus of the host is UP,event and we know it is PingTimeout, so nextStatus is Alert. Next is to execute the investigate method

Protected Status investigate (final AgentAttache agent) {final Long hostId = agent.getId (); final HostVO host = _ hostDao.findById (hostId); if (host! = null & & host.getType ()! = null & &! host.getType (). IsVirtual ()) {final Answer answer = easySend (hostId, new CheckHealthCommand ()); if (answer! = null & & answer.getResult ()) {final Status status = Status.Up; return status } return _ haMgr.investigate (hostId);} return Status.Alert;}

This method first sends a CheckHealthCommand to the hostId, which can be done in two ways:

1. If you can accept the reply that the host is normal at this time, we will directly return the UP status. When we return to handleDisconnectWithInvestigation, we will find that the task is basically over at this time, which means that the method is triggered by a temporary network failure or what situation the host has returned to normal.

two。 Another situation is that CheckHealthCommand did not get an answer, that is to say, I asked your host directly from management-server and you did not respond, which does not mean that you really hung up. What are we going to do? let's go to various investigators to investigate whether you are alive or not.

@ Overridepublic Status investigate (final long hostId) {final HostVO host = _ hostDao.findById (hostId); if (host = = null) {return Status.Alert;} Status hostState = null; for (Investigator investigator: investigators) {hostState = investigator.isAgentAlive (host); if (hostState! = null) {return hostState;} return hostState;}

Well, if our host is a XenServer host, the most important thing is, of course, XenServerInvestigator. Let's take a look at its isAgentAlive method.

Public Status isAgentAlive (Host agent) {CheckOnHostCommand cmd = new CheckOnHostCommand (agent); List neighbors = _ resourceMgr.listAllHostsInCluster (agent.getClusterId ()); for (HostVO neighbor: neighbors) {Answer answer = _ agentMgr.easySend (neighbor.getId (), cmd); if (answer! = null & & answer.getResult ()) {CheckOnHostAnswer ans = (CheckOnHostAnswer) answer; if (! ans.isDetermined ()) {continue } return ans.isAlive ()? Null: Status.Down;}} return null;}

The logic is simple: if I can't find you directly, I'll go to your neighbor in the same Cluster. I'll send a CheckOnHostCommand command to each of your neighbor hosts to see if they can know what's wrong with you. The specific implementation of the CheckOnHostCommand command is explained in detail in the article on the official website at the beginning.

If the network ping investigation returns that it cannot detect the status of the host, CloudStack HA then relies on the hypervisor specific investigation. For VmWare, there is no such investigation as the hypervisor host handles its own HA. For XenServer and KVM, CloudStack HA deploys a monitoring script that writes the current timestamp on to a heartbeat file on shared storage. If the timestamp cannot be written, the hypervisor host self-fences by rebooting itself. For these two hypervisors, CloudStack HA sends a CheckOnHostCommand to a neighboring hypervisor host that shares the same storage. The neighbor then checks on the heartbeat file on shared storage and see if the heartbeat is no longer being written. If the heartbeat is still being written, the host reports that the host in question is still alive. If the heartbeat file's timestamp is lagging behind, after an acceptable timeout value, the host reports that the host in question is down and HA is started on the VMs on that host.

It roughly means that CS runs a monitoring script on each host of XenServer and KVM, which writes the current timestamp to a file in shared storage. If a host finds that it cannot write data to the file, it will force itself to restart. The logic of the above code is to send CheckOnHostCommand commands to other hosts that share storage with the investigated host. The neighboring host receives the command to check whether the investigated host has a continuous update timestamp in the file. If so, it returns the corresponding host is still alive, otherwise it returns the host is down. In this way, only if the host does fail to connect, the determinedState in the handleDisconnectWithInvestigation method will be Status.Down, then the event will become Status.Event.HostDown, then execute the scheduleRestartForVmsOnHost method of HighAvailabilityManagerImpl to restart all the virtual machines on the host, then enter and leave a HaWorkVO in the database, and then wake up the initialized WorkerThread when CS starts, to the very important restart method of HighAvailabilityManagerImpl.

Protected Long restart (final HaWorkVO work) {boolean isHostRemoved = false; Boolean alive = null; if (work.getStep () = = Step.Investigating) {if (! isHostRemoved) {Investigator investigator = null; for (Investigator it: investigators) {investigator = it; try {(1) alive = investigator.isVmAlive (vm, host) Break;} catch (UnknownVM e) {s_logger.info (investigator.getName () + "could not find" + vm);}} boolean fenced = false If (alive = = null) {for (FenceBuilder fb: fenceBuilders) {(2) Boolean result = fb.fenceOff (vm, host); if (result! = null & & result) {fenced = true; break } (3) _ itMgr.advanceStop (vm.getUuid (), true);}} vm = _ itMgr.findById (vm.getId ()); (4) if (! _ forceHA & &! vm.isHaEnabled ()) {return null; / / VM doesn't require HA} try {HashMap params = new HashMap () (5) if (_ haTag! = null) {params.put (VirtualMachineProfile.Param.HaTag, _ haTag);} WorkType wt = work.getWorkType (); if (wt.equals (WorkType.HA)) {params.put (VirtualMachineProfile.Param.HaOperation, true);} (6) try {_ itMgr.advanceStart (vm.getUuid (), params, null) } catch (InsufficientCapacityException e) {s_logger.warn ("Failed to deploy vm" + vmId + "with original planner, sending HAPlanner"); _ itMgr.advanceStart (vm.getUuid (), params, _ haPlanners.get (0));}} return (System.currentTimeMillis () > 10) + _ restartRetryInterval;}

As in the code above, there are 5 key points that I need to pay attention to. The general process is as follows:

(1) call each investigator.isVmAlive method, and do nothing if isAlive, otherwise go down

(2) call the fb.fenceOff method

(3) execute the _ itMgr.advanceStop method

(4) with regard to the _ forceHA variable, because I did not find it in the global variable or in the configuration table of the database, the initialization value is FALSE, that is to say, only the VM whose vm.isHaEnabled is ture will continue to execute, otherwise it will directly return.

(5) the value of _ haTag is specified by the global variable ha.tag, which is empty by default. If you specify this value, it is important to determine the VM allocation host later, remember this line of code params.put (VirtualMachineProfile.Param.HaTag, _ haTag)

(6) are you familiar with this? yes, anyone who has read the process code for CS to create an VM instance knows that this method is to allocate a VM. Then most of the HA execution code of the entire CS is completed here, and the next step is to restart VM. Whether the VM can be restarted depends on various conditions. For example, whether there is a suitable host in the Cluster, whether the physical resources of the host are sufficient, whether the ha.tag is set, whether the VM uses tags, and so on, we will not go into detail here.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.