Setting Up Redundancy

Enabling Redundancy

Before You Set Up Redundancy

In redundancy, both nodes will share the exact same configuration state. When a Backup node connects to a Master node, the Backup will attempt to synchronize itself with the Master. Therefore, before you set up for redundancy the following should be considered:

  1. Start with a fresh install for the Backup node.
    Because the current configuration of the Backup node will be overwritten, make sure that it does not contain anything valuable. It is a good idea to export any projects that are unique to the Backup before enabling redundancy.

  2. All system configurations relative to the Master node must also resolve on the Backup node.
    For example, OPC-UA connections and database connections must use addresses that resolve from both nodes, or any OPC-COM servers must be installed and configured identically on both nodes. This means using "localhost" in any of the database connections won't work. You should use the IP Address of the computer instead.

  3. Configure firewalls between the redundancy nodes.
    Redundant systems need TCP connectivity between each other on a variety of ports. Turning off software firewalls or adding special exception rules for each others' addresses is required. Specifically, the Master node must be able to receive data on TCP/IP port 8750 (you can change this in the Settings), and the Backup node must be able to send outgoing data on that port.

The OS platform (i.e. Windows, OS X, Linux) for the Master and Backup must be the same for Redundancy. This means that if your Master Ignition is installed on Windows 8, your Backup can be on a Windows 10 machine, but not on OS X.

To Set Up Redundancy

Once the above points are done, setting up redundancy is fairly simple:

On the Master Gateway

  1. Go to the Configure section of the Master Gateway webpage.

  2. Select Configuration > Redundancy.
    The Redundancy and Network Configuration page is displayed showing different sections and settings.

  3. Change the following Settings:

    • Redundancy Settings, set Mode to Master

    • Backup Mode Settings, type in the Master Node Address (for example, 10.20.8.12)

  4. Click Save Changes.
    Confirm change to Redundancy Settings     page is displayed.

  5. Click Confirm to apply your settings.

  6. Select System > Status to ensure the redundancy mode and state is properly set.

On the Backup Gateway

  1. Do the exact same steps 1-6 above on the Backup Gateway webpage, except in step 3 set the Redundancy Settings Mode to Backup.

  2. Verify the correct port setting in Network Settings.

  3. To verify the redundancy setup, that is, to ensure the Master and the Backup Gateways are connected, go the Status tab of the Gateway webpage and see the System Map. The System Map should show the connected nodes and their current states.

    After saving, the system connects to the Master and downloads a system Backup, and restarts. Once the restart is complete, the Backup node is synchronized and in communication with the Master.

On this page ...

Redundancy Settings

All redundancy settings are configured in the Ignition configuration Gateway under Configuration > Redundancy. Most settings are used by both the Master and Backup nodes, with their individual settings broken out into separate categories.

It is important to know that while the full system configuration is shared between nodes, redundancy settings are not shared between nodes. Therefore, it is perfectly acceptable to have different values for the same settings on the two nodes. For example, it is possible to have a different Standby Activity Level on both nodes, and, of course, the network settings will often be different.

The Master node shares all configuration with the Backup node, and this means that changes cannot be made to your project from the Backup. In fact, the Designer can never be opened from a Backup node, even if the Master is currently offline.

Redundancy Settings

Mode

Independent - Redundancy is not enabled and this Ignition system runs as an independent node.
Master - This is the Master node, who listens for a connection from the Backup node, and is in
charge of managing system synchronization.
Backup - This is the Backup node, who will connect to the Master and receive system updates.

Standby Activity Level

How the node operates when it is not the Active node.
Cold - Performs minimal activities, does not connect to devices, and so on. The purpose is to minimize the load on
the network and on devices.
Warm - Connect to devices, subscribe to tags and set up all executing objects. The purpose is to minimize fail-over time.

Fail-over Timeout

The time, in milliseconds, before the opposite node is determined to be unavailable and this node takes over.

Startup Connection Allowance

The time, in milliseconds, to wait on initial startup for a connection to be established before making a decision on the node's activity level. This is used to prevent unnecessary switch over caused by a node starting as active, only to connect and find that the other node is active, resulting in one of the nodes being de-activated.

It is important to note that this setting can interfere with the Master Recovery Mode - if the Master is active, it will always request the Backup to de-activate. If this setting is low, or 0, the Master will always become active before connecting to the Backup, and thus "manual recovery" will not be possible.

Network Settings

Port

For the Master, the port to listen on. For the Backup, the port to connect to on the Master.

Auto-detect Network Interface

If true, the system will automatically select which network interface to use on the machine. If false, the system will bind itself to the interface of the specified address.

Network Bind Interface

The address to bind to if Auto-detect is false.

Auto-detect HTTP Address

When clients are launched, they are provided with a list of addresses that they may connect to. If this option is true, the list will be generated automatically. If false, they will be provided with the list specified.

HTTP Addresses

The list of addresses to give to the clients if auto-detect is turned off. These are the addresses that the clients will attempt to connect to, so the HTTP and HTTPS ports must match the configuration of the system in the Gateway Control Utility

Master Node Settings

Recovery Mode

How the Master acts when it connects to a Backup node that is currently active.
Automatic - The Master automatically takes back responsibility, and becomes active. The Backup node goes to standby.
Manual - The Backup node is allowed to stay active. The Master will become active if the Backup node fails, or if the user requests a switchover from the Gateway configuration page.

Runtime Update Buffer Size

How many "runtime status updates" can be queued up in memory before the system stops tracking them and forces a full refresh. These updates represent information that the other node should have in order to have the same running state as the Master when it's forced to take over.

This is most often the values of static tags and the current alarm state. Given that the update buffer is only used once the nodes are connected, the default value is usually fine, and only needs to be increased on systems that may have many alarms that change together, or many static tag writes.

Backup Node Settings

Master Node Address

The address where the Master is located.

Ping Rate

The time, in milliseconds, between messages from the Backup to the Master.

Reconnect Period

How often in millisecond, to re-attempt connection when the Backup node is not connected to the Master.

History Mode

How history is treated by the Backup system. If Full, history will be stored normally, as it would be on the Master system. If Partial, history will be cached until the Master is available again and the Backup node is able to determine the exact time that the Master was down.

Troubleshooting

Redundancy Connectivity

When the two redundant nodes are connected, you will be able to see their state details in the Status section of the Gateway webpage. There are also various other places where the redundancy state is shown as connected.

If the two nodes cannot connect, check the following:

  • Verify that the Master address is correct in the Backup. Try to ping the Master machine from the Backup machine, and verify that you're using the correct address for the network card that the Master is connected through.

  • If using system names (or domain names), verify that the name is resolving to the correct address by performing a ping.

  • Verify that the firewall on the Master is set to allow TCP traffic to the designated port.

  • Verify that the Backup is not connecting and then immediately disconnected for some reason.

  • Viewing the error log in the gateway console section should show this. If errors are occurring at regular intervals, look at the message for an indication of what is happening. An example of a potential problem is when the failover time is set too low for the given network, which results in many socket read timeout exceptions, which in turn leads to many disconnect/reconnect attempts.

  • If errors are occurring, but the cause isn't clear, contact Inductive Automation Support.

Advanced Troubleshooting

A variety of loggers can be found under the Gateway console section by going to "Levels" and searching for "Redundancy". By setting these loggers to a finer level, more information will be logged to the console. This is generally only useful under the guidance of Inductive Automation support personnel, though more advanced users may find the additional logged information helpful.

Similar Topics ...