Shulou(Shulou.com)06/02 Report--

HA (High Available), high availability cluster, is an effective solution to ensure business continuity. It generally has two or more nodes, and is divided into active nodes and standby nodes. The one who is performing the business is usually called the active node, while the one that is a backup of the active node is called the standby node. When there is a problem with the active node, resulting in the running business (task) can not run properly, the standby node will detect and immediately continue the active node to execute the business. In order to achieve uninterrupted or short-term interruption of business.

A highly available cluster with only two nodes is also known as a dual hot backup, even if two servers are used to back up each other. When one server fails, another server can undertake the service task, so as to automatically ensure that the system can provide services continuously without human intervention. Dual-computer hot backup is only one kind of high-availability cluster, and the high-availability cluster system can support more than two nodes, provide more and more advanced functions than dual-computer hot backup, and can better meet the changing needs of users.

This guide uses pfsense2.33 for high availability configuration examples.

There are three key points for high availability cluster configuration:

CARP is used for IP address redundancy

XMLRPC is used to configure synchronization

Pfsync is used for status table synchronization

Through configuration, two nodes act as "active / passive" clusters, one node acts as the primary node, the other secondary node acts as the backup role, and if the primary node fails, the secondary node takes over as needed.

The following chapters are divided into the following chapters:

Components of a high availability cluster

High availability prerequisites

Configure a high availability cluster

Test high availability

High availability troubleshooting

Upgrade pfSense on a high availability cluster

Composition of high availability cluster

Although often mistakenly referred to as "CARP clustering", two or more redundant firewalls running pfSense software are better called "high availability clusters" or "HA clusters" because CARP is only one of several technologies used to achieve high availability, and future CARP can be transformed into different redundancy protocols.

The most common high availability (HA) cluster configuration consists of only two nodes. There can be more nodes in the cluster, but they don't make more sense. This guide assumes that two pfsense nodes are in use, one as the primary node and the other as the secondary node.

It is important to distinguish between the three functions of HA (IP address redundancy, configuration synchronization, and state table synchronization) because these tasks occur in different places. The firewall performs configuration synchronization and state synchronization on the synchronization interface and communicates directly between firewall units. The firewall sends a CARP heartbeat on each interface of the CARP VIP. Failover instructions do not occur on synchronous interfaces, but on each interface that supports CARP.

High availability Cluster exampl

IP address redundancy (CARP)

In order to connect seamlessly through a cluster connection during a failover, traffic to the cluster must use redundant IP addresses. The Common address redundancy Protocol (CARP) is commonly used to accomplish this task.

The CARP type virtual IP address (VIP) is shared among the nodes of the cluster. One node is the primary station and receives traffic from the IP address, while the other nodes maintain the backup status and monitor the heartbeat to see if the previous primary station fails to assume the host role. Because only one cluster member is using the IP address at a time, CARP VIP has no IP address conflicts.

For the failover to work properly, it is important that inbound traffic arriving at the cluster is sent to the CARP VIP and traffic is sent from the CARP VIP. Inbound traffic includes routing uplink traffic, × ×, NAT, local client gateway and DNS requests. Outbound traffic includes outbound NAT and × ×. If the traffic is addressed directly to the node instead of the CARP VIP, the traffic will not be obtained by other nodes.

CARP works like VRRP and HSRP, and conflicts may even occur in some cases. Heartbeats are sent on each interface that contains CARP VIP, and one heartbeat per VIP per interface. At the default values of deviation and benchmark, VIP sends a heartbeat every second. Deviation determines which node is the primary node at a given point in time. No matter which node sends the heartbeat, the fastest is the primary role. A higher deviation value causes the heartbeat to be sent with more delay, so if the heartbeat is delayed or lost due to network or other problems, the node with the lower deviation value will become the host.

Be careful

Do not use CARP VIP to access firewall GUI,SSH or other management mechanisms. For administrative purposes, you can only use the actual IP address on the interface of each individual node, not VIP. Otherwise, you may not be sure which node is being accessed.

PfSense XML-RPC configuration synchronization

To make it easier to maintain virtually the same firewall nodes, you can use XML-RPC for configuration synchronization. When XML-RPC synchronization is enabled, the settings for the support zone are copied to the secondary device and activated each time the configuration changes. XMLRPC synchronization is optional, but without it, maintaining the cluster would be much more difficult.

Some areas cannot be synchronized, such as interface configuration. Parts that can be synchronized include: firewall rules, aliases, users, certificates, × ×, DHCP, routes, gateways, and so on. As a general rule, hardware-specific or installation-specific items (such as interfaces or values under System > General or System > Advanced) are out of sync. The list of supported zones may vary depending on the version of pfSense used. For a list of areas to synchronize, see the check box for System > High Avail Sync in the XMLRPC section. Most plug-ins cannot synchronize, but some plug-ins contain their own synchronization settings.

Configuration synchronization should use the Sync interface, or if there is no dedicated Sync interface, use the same interface configured for pfsync.

In a two-node cluster, the XML-RPC settings can only be enabled on the primary node and must be disabled on the secondary node.

For XML-RPC to run, both nodes must run GUI on the same port and protocol, such as HTTPS on port 443, which is the default. The administrator account cannot be disabled, and both nodes must have the same administrator account password.

Status table synchronization (pfsync)

Pfsync enables synchronization of firewall state tables between cluster nodes. Changes to the primary node state table are sent to the secondary firewall through the Sync interface and vice versa. When pfsync is active and configured correctly, if the primary node fails, the backup node will take over and the client will not notice the switchover.

Pfsync uses multicast by default, or you can define IP addresses to force unicast updates to be used in environments with only two firewalls, and multicast does not work properly. Any active interface can be used to send pfsync updates, but using a dedicated interface is more beneficial to security and performance. Pfsync does not support any authentication method, so any user with local network access can insert status into the status table if any method other than the private interface is used. In a low-throughput environment where security is not critical, the LAN interface can be used. The bandwidth required for this state synchronization will change from one environment to another, but may reach 10% of the firewall throughput, depending on the state insertion and deletion rates in the network.

Failover can still run without pfsync, but cannot be seamlessly connected. Without pfsync, if one node fails and another takes over, the user connection will be discarded. Users can immediately reconnect through other nodes, but they will be interrupted during the conversion. Depending on the usage in a particular environment, this may be ignored or a brief interruption.

When using pfsync, the pfsync setting must be enabled on all nodes participating in state synchronization, including secondary nodes, or it will not function properly.

High availability prerequisites

Before implementing a redundant configuration, some prerequisites must be met.

This guide assumes that:

Use only two cluster nodes

Both cluster nodes use the same pfsense device

The two devices are factory default configuration and have no other settings.

Be careful

Do not connect the LAN ports of two cells to the same LAN switch. Do not connect until the basic settings are applied to each node, otherwise there will be IP address conflicts, resulting in no separate communication with each node.

Determine the synchronization interface

This guide assumes that an interface on each cluster node is dedicated to synchronization tasks. This is often referred to as the "synchronization" interface, and it is used by firewalls to configure synchronization and pfsync state synchronization. You can use any available interface. In this guide, the OPT4 interface (igb5) will be used to synchronize traffic.

Be careful

Some people call it the "CARP" interface, which is wrong and very misleading. The CARP heartbeat occurs CARP VIP on each interface. CARP traffic and failover operations do not use the Sync interface.

Interface assignment

Interfaces must be assigned in the same order on all nodes. If the interface is not aligned, configuration synchronization and other tasks will not function properly. If any adjustments are made to the interface assignments, they must be replicated equally on both nodes.

Interface assignment schematic diagram

IP address requirement

The high availability cluster requires three IP addresses in each subnet and a separate unused subnet for synchronizing the interface. Each node uses an IP address plus a shared CARP VIP address for failover. The synchronization interface requires only one IP address per node.

The IP address used in this guide is shown in the table below. You can change it to your own IP address as needed.

WAN IP address assignment IP address usage 198.51.100.200/24CARP shared IP address 198.51.100.201swap 24 primary node WAN IP address 198.51.100.202 IP address assignment IP address assignment 192.168.1.1/24CARP shared IP address 192.168.1.2 IP 24 primary node LAN IP address 192.168.1.3 LAN IP address synchronization IP Address assignment IP address usage 172.16.1.2Plus 24 Primary Node Sync IP address 172.16.1.3 CPE 24 Secondary Node synchronous IP address

Single address CARP

It is technically possible to configure an interface with CARP VIP as a unique IP address in a given subnet, but this type of configuration is not recommended. When the WAN uses this configuration, it only allows communication from the primary node to the upstream network (such as Internet), which complicates anything such as updates, plug-in installation, gateway monitoring, or requiring an external connection from the secondary node. It is more suitable for internal interfaces, but they are usually not subject to the same IP address restrictions as WAN, so it is still a priority to configure IP addresses on all nodes. Such configurations are not covered in this guide.

Determine CARP VHID availability

If other devices on the network use conflicting identifiers, CARP may interfere with VRRP, HSRP, or other systems that use CARP. To ensure that there is no conflicting traffic on a network segment, perform packet capture on each interface to look for CARP traffic. The VHID on each layer 2 must be unique, so each interface must be checked separately. As long as the network segment is in a separate broadcast domain, you can use the same VHID on different network segments.

If any CARP or VRRP traffic is displayed, pay attention to VHID / VRID and avoid using this identifier when configuring CARP VIP VHID later.

This guide assumes that there are no other traffic that may conflict.

Setting requirement

After using the installation wizard or manual setup, configure each firewall with a unique hostname and a non-conflicting static IP address.

For example, one node can be "firewall-a.example.com" and the other can be "firewall-b.example.com" or a more personalized pair of names.

Avoid naming nodes master and backup, because they are states rather than roles, and can be named primary and secondary.

For IP addresses, the factory default LAN address is 192.168.1.1. In this case, the address will be CARP VIP. Use this subnet to move each node to its own address in the subnet, such as 192.168.1.2 for the primary node and 192.168.1.3 for the secondary node.

On the secondary node, disable the DHCP server for the LAN segment under the LAN tab of Services > DHCP Server. This service is automatically enabled after configuration synchronization, and if enabled before HA is fully configured, IP address conflicts may occur on the primary and secondary nodes.

Once each node has a unique LAN IP address and DHCP on the secondary node is disabled, both nodes can be plugged into the same LAN switch.

Both nodes must run GUI on the same port and protocol. This guide assumes the use of HTTPS on port 443.

The administrator account cannot be disabled, and both nodes must have the same administrator account password.

On pfSense2.4 and later, any user can be used for configuration. The synchronization user must have System-HA node sync privileges. Now we use admin, and then we can change it.

Both nodes must have a static IP address on the same subnet and configure the correct gateway on the WAN interface settings.

Both nodes must be properly configured with DNS, or use a DNS parser that disables forwarding, or set up a DNS server under System > General Setup.

Visit Services > DNS Resolver. Review the settings, and even if there are no changes, click Save once to make sure that the default value is set.

Switch / layer 2 configuration CARP related

CARP heartbeats are delivered using multicast and may require special processing on the switches involved in the cluster. Filtering, rate limiting, or other interference on some switches may cause CARP multicast to fail. In addition, some switches adopt port security methods that may not work properly with CARP.

So the switch must:

Allows CARP VIP to send and receive multicast traffic without affecting the port.

Allows traffic to be sent and received using multiple MAC addresses.

Allow CARP VIP MAC addresses to move between ports.

Almost all problems with CARP that fail to correctly reflect the expected state are switch failures or other layer 2 problems, so make sure the switch is configured correctly before continuing.

Port configuration

The WAN port of each node must be connected to the same WAN switch and then to the WAN CPE / Modem / uplink. The LAN ports will be connected to the same LAN switch, and so on. The Sync interface can be connected directly between two nodes without the need for a switch.

Configure high availability cluster settings synchronization interface

Before continuing, you must configure the Sync interface on the cluster node. Synchronous IP address assignment lists the addresses to be used for the synchronization interface on each node.

Navigate to Interfaces > OPT4

Set Enable Interface (enable interface)

Enter description text for SYNC

Set IPv4 Configuration Type (IPv4 configuration type) to Static IPv4 (static IPv4)

Set the IPv4 address (IPv4 address) to 172.16.1.2, which is in primary node mode. If it is a secondary node, it is set to 172.16.1.3

Select 24 as the subnet mask from the CIDR drop-down list next to IPv4 address

Do not select Block private networks or Block bogon networks

Click Save

Click Apply Changes (apply changes)

After completing this procedure on the primary node, perform the procedure again on the secondary node with the appropriate IPv4 address (address) value.

Add synchronous firewall rules

To complete the Sync interface configuration, add firewall rules to both nodes to allow synchronization.

At a minimum, firewall rules must synchronize traffic (by default, HTTPS on port 443) and pfsync traffic by configuring. In most cases, simple "allow all" style rules are used. For this guide, both will be shown, and it will also be used as an indicator of synchronization.

Make the following settings on the primary node:

Navigate to the SYNC tab of Firewall > Rules

Click to create a new rule at the top of the list

Set Action to Pass

Set Protocol to TCP

Set Source to YNC Net

Set Destination to SYNC Address

Set Destination port range to 443 or select HTTPS from the drop-down selector

Set Description to Allow configuration synchronization

Click Save

Click to create another new rule at the top of the list

Set Action to Pass

Set Protocol to pfsync

Set Source to SYNC Net

Set Destination to any

Set Description to Allow state synchronization

Click Save

Click Apply Changes

When complete, the rules will be shown in the following figure, "example synchronous Interface Firewall rules," which also includes rules that allow ICMP echoes (ping) for diagnostic purposes.

Example synchronous interface firewall rules

Make the following settings on the secondary node:

Navigate to the SYNC tab of Firewall > Rules

Click to create a new rule at the top of the list

Set Action to Pass

Set Protocol to any

Set Source to SYNC Net

Set Destination to any

Set Description to Temp rule for sync

Click Save

Click Apply Changes

Be careful

The rules of the secondary node are different, and once the first configuration synchronization is carried out, the temporary rules on the secondary node will be replaced by the rules of the primary node.

Configure pfsync

State synchronization using pfsync must be configured on the primary and secondary nodes to run.

Do the following first on the primary node and then on the secondary node:

Navigate to System > High Avail Sync

Find the State Synchronization Settings (pfsync) section on the page

Enable Synchronize States

Set Synchronize Interface to SYNC

Set pfsync Synchronize Peer IP to synchronize another node. 172.16.1.3 on the primary node and 172.16.1.2 on the secondary node

Click Save

Configure XMLRPC

Be careful

Synchronization can only be configured on the primary node. Do the following only on the primary node:

Navigate to System > High Avail Sync

Set Synchronize Config to IP to the Sync interface IP address of the secondary node, 172.16.1.3

Enter Remote System Username (remote system user): admin

Be careful

Before pfSense2.3.x and before, this must always be admin, can not fill in other users! On pfSense2.4 and later, any user can be used for configuration. The synchronization user must have System-HA node sync privileges.

Enter Remote System Password (remote system password). Initially, it will be pfsense if it has not changed.

Click Save

Navigate to the Firewall > Rules SYNC tab on the secondary node and you will find that the current rules are synchronized with the primary node.

Be careful

Do not make changes to the secondary node in the area set to sync! The next time the primary node performs synchronization, these changes will be overwritten.

Add CARP VIPs

Now after the configuration synchronization is complete, the CARP virtual IP address only needs to be added to the primary node and automatically replicated to the secondary node through configuration synchronization. Let's set up. This requires the addition of two CARP VIP: one for WAN and one for LAN.

Navigate to the Firewall > Virtual IPs menu of the primary node

Click

Create a new VIP at the top of the list

Select Type as CARP

Select Interface as WAN

Enter the WAN CARP VIP address and subnet mask in the Address (es) box. Enter 198.51.100.200 and 24 in this example

Enter a random password in Virtual IP Password and enter the password again in the confirmation box. This only needs to be matched between the two nodes, which will be handled through synchronization.

Select the unused CARP VHID identified in determine VHID Group availability.

Be careful

A common strategy is to match the VHID to the last octet of the IP address, so in this case, 200 VHID is used in this example.

Set Base = 1 and Skew = 0 in Advertising Frequency. This value is automatically adjusted when copied to the secondary node.

Enter the description WAN CARP VIP in Description

Click Save

Base and Skew work together to determine the frequency of CARP heartbeats. The value of Base increases by an entire second and should match between the two nodes. The Skew value increases by 1tick for 25 seconds. The primary node should always be offset by 0 or 1, and the secondary node must be higher, usually 100 +. This adjustment is handled automatically by the configuration synchronization process.

Be careful

If CARP is too sensitive to latency on a given network, we recommend adding one second at a time to adjust the Base until it is stable.

Repeat the above process for LAN CARP VIP.

Navigate to Firewall > Virtual IPs

Click

Create a new VIP at the top of the list

Select Type as CARP

Select Interface as LAN

Enter the LAN CARP VIP address and subnet mask in the Address (es) box, and 192.168.1.1 and 24 in this example

Enter a random password in Virtual IP Password and enter the password again in the confirmation box.

Select the unused CARP VHID identified in determine VHID Group availability.

Set Base = 1 and Skew = 0 in Advertising Frequency.

Enter the description LAN CARP VIP in Description

Click Save

Click Apply Changes

If there are other IP addresses in the WAN subnet that are used for 1:1 NAT, port forwarding, × ×, and so on, you can now add them as CARP VIP.

Check Firewall > Virtual IPs on the secondary node to ensure that VIP is synchronized as expected.

If the settings are synchronized successfully, the virtual IP addresses on both nodes will look like CARP virtual IP lists.

CARP Virtual IP list

Check CARP status

Now access Status > CARP on both nodes to confirm the correct state. The primary node should display the MASTER status of all VIP (primary CARP VIP status) and the secondary node should display the BACKUP status of all VIP (CARP VIP status on the secondary node). "if the status is incorrect, see High availability troubleshooting."

CARP VIP status of primary node

Secondary node CARP VIP status

Set manual outbound NAT

The settings for NAT will be automatically synchronized, so these changes only need to be made to the primary node.

Navigate to the Firewall > NAT, Outbound tab on the primary node

Change Mode (mode) to Manual Outbound NAT rule generation (manual outbound NAT rule generation)

Click Save, and the list of rules will be populated with the same rules used by the default auto outbound NAT.

Be careful

If no rules appear in the list, make sure that WAN selects a gateway under Interfaces > WAN.

Click to edit the rule for the LAN subnet

Set Translation Address to WAN CARP VIP, in this case 198.51.100.200.

Click Save

Repeat the edit for each rule in the list, except for those with a source of 127.0.0.0swap 8.

Click Apply Changes

Access the Firewall > NAT, Outbound tab of the secondary node to ensure that outbound rule changes are reflected there.

Be careful

If you later add another local interface, such as a second LAN,DMZ, and so on, and the interface uses a private IP address, you must add additional manual outbound NAT rules at this time.

When complete, the rule changes will be similar to those in CARP VIP's LAN outbound NAT rules.

LAN outbound NAT rules of CARP VIP

Related to other NAT

If any port forwarding wants to use WAN CARP VIP, you must use Firewall > NAT, the Port Forward tab to add. Port forwarding will work as usual, but the Destination (destination) will be WAN CARP VIP.

Set up DHCP

The DHCP server daemons on the cluster nodes need to be adjusted so that they can work together. These changes are automatically synchronized from the primary server to the secondary server, so along with VIP and outbound NAT, these changes only need to be made on the primary node.

Navigate to the Services > DHCP Server, LAN tab

Set the first entry in the list of DNS servers to LAN CARP VIP, here is 192.168.1.1

Set Gateway to LAN CARP VIP, this is 192.168.1.1

Set Failover Peer IP (failover peer IP) to the actual LAN IP address of the secondary node, here is 192.168.1.3

Click Save

Setting the DNS server and gateway to CARP VIP ensures that the local client is communicating with the failover address rather than connecting directly to either node. In this way, if the primary node fails, the local client will continue to talk to the secondary node.

The failover peer IP allows the daemon to communicate directly with the peer in the subnet to exchange data such as lease information. When the setting is synchronized with the secondary device, this value is automatically adjusted so that the secondary node returns to the primary station.

On both nodes, visit Status > DHCP Leases to see the status. It contains the failover pool status, and a row is displayed for each local interface pool. When both nodes are working properly, both will indicate the normal status, as shown in the following figure.

DHCP failover statu

* × × s and other services

When configuring × × (such as Open × × or IPsec), select WAN CARP VIP as the interface of × ×, and ensure that the remote peer also uses CARP VIP as the peer address to build the other side of the tunnel.

For other local services, plug-ins, and so on, similarly, if the service will work on two nodes, we recommend using CARP VIP for binding.

High availability support for various plug-ins varies. Check the documentation for the plug-in for support for high availability.

Additional interface

Repeat several of the same steps required to configure other local interfaces:

Assign interfaces on two nodes

Enable interfaces on both nodes, using different IP addresses within the same subnet

Add CARP VIP to the new subnet (primary node only)

Add firewall rules (primary node only)

Add manual outbound NAT for the new subnet and use CARP VIP for conversion (primary node only)

Use CARP VIP to configure DHCP servers for new subnets for DNS and gateway roles (optional, primary node only)

Test high availability

After all the configurations are complete, tests must now be conducted to verify the validity. The tests of all aspects of the system are listed below. If the test fails, review the configuration and follow the help below to find a solution.

Verify general functionality

Set up the client on LAN, ensure that you receive the DHCP IP address, and display LAN CARP VIP as its gateway and DNS server. Verify that the client can resolve the hostname, access the Internet, and run as expected.

Verify that XMLRPC synchronization is working properly

XMLRPC configuration synchronization can be tested in several ways. The easiest way is to make changes to any major supported areas, such as firewall rules, and then see if the changes are reflected in the secondary node (usually only a few seconds).

The manual way to force the configuration of a synchronous test XMLRPC is to access Status > Filter Reload on the primary node, and then click Force Config Sync (force configuration synchronization). The status changes briefly, and then if everything is all right, a message is displayed indicating that the synchronization has completed successfully.

Verify that CARP is working properly

Access Status > CARP on both nodes to check that CARP is working properly. All primary nodes of CARP VIP will display "MASTER", while all secondary nodes of CARP VIP will display "BACKUP". If the status screen indicates that CARP is disabled, click the Enable CARP (enable CARP) button.

Verify that status synchronization is working properly

The Status > CARP page lists the pfsync nodes that indicate the state synchronization status. These values may not always be the same on both nodes, but they will be close. If the two are very different, it may prompt the problem of state synchronization. If they are the same or almost the same, state synchronization is working.

Test failover

Failover testing can be done in four ways:

Click Enter Persistent CARP Maintenance Mode (persistent CARP maintenance Mode) on Status > CARP on the primary node. This will permanently disable CARP, even if the primary node is restarted. To exit maintenance mode, click Leave Persistent CARP Maintenance Mode (keep CARP maintenance mode) and enable CARP again. This is an easier and faster way to change than disabling CARP.

Unplug the network cable from the CARP VIP interface, such as WAN or LAN. This triggers a failover and reinserts the cable to restore.

Click Temporarily Disable CARP (temporarily disable CARP) on the primary node Status > CARP. This will temporarily disable CARP, and if the primary node restarts, it will restart. Click enable CARP to reopen it.

Shut down or restart the primary node.

During any of the above tests, visit the Status > CARP of the secondary node to confirm that the CARP VIP has taken over and display the MASTER status.

Test the connection from the client on LAN to Internet before, during, and after triggering the failover to ensure that the connection works at each step. Downloading files, streaming is likely to continue. But VoIP-based phones may have a slight outage because they don't buffer data like other phones.

There is also a client trying to obtain an IP address from DHCP through the run time of the secondary node.

If xxx or other services are already configured, they should also be checked during testing to ensure that the xxx established on the secondary node continues to pass through traffic.

Once the primary node returns to the "MASTER" state, make sure everything continues to work.

High availability troubleshooting

If the test fails, check the following settings.

View configuration

Before diving into the technical details below, check the configuration to make sure that all the steps are correct.

CARP troubleshooting to check the status of each interface

If the interface shows "INIT" for the CARP status, as shown in the following figure, the most common case is that this indicates that the interface on which the VIP is located is not connected to anything. If you are not connected to the switch or another port, the interface is down and VIP cannot be fully initialized. If the cable is plugged in, reconfigure it.

CARP status and disconnect interface on the primary interface

If the interface goes offline and will remain offline, such as an interface that is not already used by the firewall, remove the CARP VIP from the interface until it is ready. Configuring CARP VIP on a dropped interface can cause the firewall itself to be degraded, resulting in unstable failover.

Conflicting VHID

VHID determines the virtual MAC address used by CARP IP. Input validation in pfSense will not allow conflicting VHID to be used on a pair of firewalls, but conflicts may occur if there are multiple clusters running CARP in the same broadcast domain. VRRP also uses the same virtual MAC address scheme, so VRRP IP that uses the same VRID as CARP IP VHID produces the same MAC address conflicts.

When using CARP on the WAN interface, it also means that the VRRP or CARP used by the upstream network / ISP may also conflict. Make sure that there is no VHID in use for ISP on this broadcast domain.

In addition to creating MAC conflicts that may interfere with traffic, it may also interfere with CARP VIP state.

Incorrect subnet mask

The subnet mask of the CARP VIP must match the subnet mask on the interface IP address of the same subnet. For example, if the interface IP address is 192.168.1.2 CIDR 24, CARP VIP must use the CIDR mask of / 24.

Layer 2 switch issu

Typically, the layer 2 switch problem itself is represented by two units that show the "MASTER" status of one or more CARP VIP. If this occurs, check the following items:

Ensure that both interfaces (WAN,LAN, etc.) are connected to the VLAN of the correct layer 2 switch. For example, make sure that the LAN ports of both devices are connected to the VLAN of the same switch.

Verify that two nodes can pass through each other on each segment (for example, via ICMP echo). Firewall rules may need to be added to WAN to accommodate this test.

If plugged into a separate switch, ensure that the switch trunks and passes broadcast / multicast traffic correctly.

If you are using the switch on the back of the modem / CPE, try using a real switch. These built-in switches usually do not handle CARP traffic correctly. Typically, inserting a firewall into the appropriate switch and then uploading it to CPE will eliminate the problem.

Disable IGMP Snooping or other multicast restrictions and checks. If they have been turned off, try enabling the feature and disabling it again.

Troubleshooting XMLRPC

If the XMLRPC synchronization attempt fails, a notification is generated in GUI to attract attention, as shown in the following figure. Notifications usually contain information about why they failed and point to fixes, and if it is not enough to troubleshoot, check the other items in this section.

Notification of XMLRPC synchronization problem

Check the system log

XMLRPC fault details are logged to the primary system log (Status > System Logs, General tab). Usually errors are clearly displayed, for example, an authentication failure indicates that the password entered for the administrator user on the synchronization settings is incorrect. As shown in the following figure, a timeout occurred during the synchronization attempt, in this case due to the lack of firewall rules.

XMLRPC synchronization problem logging

Check the firewall log

Access the Status > System Logs, Firewall tab on the secondary node. Check that the entries in the log fail to reach the secondary synchronization interface on the GUI port, as shown in the following figure. If the traffic appears to be blocked, adjust the synchronization interface rules as needed.

XMLRPC synchronization problem Firewall Log access entry

Check administrator user

Access System > User Manager and ensure that the administrator user is enabled on both systems and that the administrator password is the same on both systems. Access System > High Avail Sync and re-check that the administrator username and password have been entered correctly.

In pfSense2.4 and later versions, ensure that the synchronization user exists on the secondary node and has "System-HA node sync" permissions.

Verify the connection

Check Status > Interfaces and make sure that the synchronization interface displays links on both units. If there is no link, make sure that a cable is connected between the two devices. If the link is still not available, try using a small switch or VLAN between the two nodes.

Add firewall rules to the Sync interface to allow ICMP to echo requests, and then try to ping from one firewall to another to ensure that they can meet each other at layer 3. If not, carefully check the interface IP address and subnet mask settings, as well as cabling.

Troubleshooting pfsync

If the pfsync node is not queued under Status > CARP, the prompt state is not synchronized.

Check firewall rules

Check the firewall logs on the Status > System Logs, Firewall tabs of both nodes. If there is any pfsync protocol traffic, the firewall rules on the synchronization interface may be incorrect.

View the rules of the Sync interface tab of Firewall > Rules. Ensure that the rules pass pfsync protocol traffic or any protocol traffic to any destination. Adjust the rule accordingly and check the log and CARP status again to see if it is working.

Verify the connection

See verify connections above to check connections between nodes.

Check the interface

If the state appears to be synchronized, but the failover still does not complete successfully, check Interfaces > (Assign) and make sure that the interfaces are all arranged by physical name. In pfSense2.2 and later, the state is bound to the interface, so if the LAN interface is igb0 on one node and igb3 on the other, the state will not be queued. Adjust the interfaces so that they are the same on both nodes.

Local service troubleshooting DNS resolution

If the local client cannot get the DNS response from the CARP VIP in the cluster, check the following items:

If you are using the default DNS parser (unbound), visit Services > DNS Resolver, and then click Save on the primary node to make sure that the default values are fully respected.

If you are using a DNS parser or DNS transponder, make sure that the daemon is configured to listen on all interfaces or at least the local host and internal CARP VIP.

Ensure that the local interface firewall rules go through TCP and UDP port 53 to the CARP VIP for the local DNS.

Make sure that the firewall itself is configured with a DNS server under System > General, especially if you are using DNS transponders (dnsmasq) instead of DNS parsers (unbound).

DHCP

If the DHCP failover pool status does not show as normal, the following items need to be checked:

Ensure that two devices are connected to the same switch / subnet on the correct interface

Verify the connection between the two devices on the interface

Ensure that the failover peer IP address is configured correctly

Make sure there is CARP VIP on the interface in question

Ensure that the skew of the CARP VIP on the primary node is 0 or 1, and that the skew of the secondary node is 100 or higher.

If everything fails:

You can click to stop the DHCP service on the Status > Services of both nodes

Access Diagnostics > Command on both nodes

Run the following command in the "Shell Execute" box on both nodes: rm / var/dhcpd/var/db/dhcpd.leases*

Click on the Status > Services of both nodes to start the DHCP service

Upgrade pfSense on a high availability cluster

View the update log and upgrade guide

Before starting any part of the upgrade, check the pfSense blog and update log to determine version changes and upgrade considerations.

Backup

Before you begin, back up your configuration under Diagnostics > Backup/Restore on both nodes.

Be careful

Backup is the best way to solve all kinds of problems quickly.

Upgrade secondary node

First perform the upgrade on the secondary node. In this way, if the upgrade fails, there will be no interruption, and if a reinstallation is needed, it can be done effortlessly.

Test auxiliary node

Once the secondary node system is booted, log in and confirm that it is running. If all services are active and the CARP status is OK, it's time to test. Force a failover from the master node by placing the primary node in maintenance mode (see testing failover) and observing what happens on the secondary node. If the secondary takes over the OK and the traffic continues to flow, it can continue.

Upgrade the primary node

When the primary node is in maintenance mode, it can be upgraded without additional interference. Start the operating system upgrade and have the firewall restart. Once restarted, make sure that the local service is running as expected, and then take the node out of maintenance mode.

Test master node

Using the current operating system and the two active nodes, run the final test to ensure that the service is running properly, traffic is flowing, CARP, DHCP, and other state areas are running properly.

Translated from pfsense book

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.