8.1 Creating or Mirroring an SBD Partition

If a single node (or group of nodes) somehow becomes isolated from other nodes, a condition called split brain results. Each side believes the other has failed, and forms its own cluster view that excludes the nodes it cannot see. Neither side is aware of the existence of the other. If the split brain is allowed to persist, each cluster will fail over the resources of the other. Since both clusters retain access to shared disks, corruption will occur when both clusters mount the same volumes.

OES Cluster Services provides a split-brain detector (SBD) function to detect a split-brain condition and resolve it, thus preventing resources from being loaded concurrently on multiple nodes. The SBD partition contains information about the cluster, nodes, and resources that helps to resolve the split brain condition.

OES Cluster Services requires an SBD partition for a cluster if its nodes use physically shared storage. Typically, you create the SBD when you configure the cluster on the first node. You can alternatively configure an SBD for the cluster after you configure the first node, but before you configure OES Cluster Services on the second node of the cluster. You might also need to delete and re-create an SBD partition if the SBD becomes corrupted or its device fails.

An SBD must exist and the cluster must be enabled for shared disk access before you attempt to create shared storage objects such as pools and volumes in a cluster. NLVM and other NSS management tools need the SBD to detect whether a node is a member of the cluster and to get exclusive locks on physically shared storage.

Typically, you use the OES Cluster Services SBD Utility (sbdutil) to create or delete an SBD partition for a cluster, as described in Creating or Deleting Cluster SBD Partitions in the OES 2018 SP2: OES Cluster Services for Linux Administration Guide. However, you can also use NLVM commands in OES 11 SP1 and later to create or delete SBD partitions.

Use the procedures in this section to create a non-mirrored or mirrored SBD partition:

8.1.1 Requirements and Guidelines for Creating an SBD Partition

Consider the requirements and guidelines in this section when you create a OES Cluster Services SBD (split-brain detector) partition.

Preparing OES Cluster Services

Before you create an SBD partition for an existing cluster, you must take the cluster down and stop OES Cluster Services software on all nodes. Do not restart OES Cluster Services and rejoin nodes to the cluster until after you create the new SBD and configure the Shared Disks flag attribute for the Cluster object.

You can mirror an existing SBD while the cluster is up and running.

Using a Shared Disk System

You must have a shared disk system (such as a Fibre Channel SAN or an iSCSI SAN) connected to your cluster nodes before you create a split-brain-detector (SBD) partition. For information, see Shared Disk Configuration Requirements in the OES 2018 SP2: OES Cluster Services for Linux Administration Guide.

Preparing a SAN Device

Use the SAN storage array software to carve a LUN to use exclusively for the SBD partition. The device should have at least 20 MB of free available space. Connect the LUN device to all nodes in the cluster.

For device fault tolerance, you can use the nlvm create raid command to mirror the SBD partition on another SAN device. Before you mirror the device, you must carve a second LUN of the same size, and connect the LUN device to all nodes in the cluster.

The device you use to create the SBD must not be a software RAID device. A hardware RAID configured in a SAN array is seen as a regular device by the server.

If you attach new devices to the server while it is running, you should scan for new devices on each cluster node to ensure that the devices are recognized by all nodes. Log in as the root user, launch a terminal console, then enter

nlvm -s rescan

Working with NLVM Commands in a Cluster

If an SBD does not exist in the cluster, NLVM cannot detect whether a node is a member of the cluster, and therefore, it cannot get exclusive locks to the physically shared storage. In this state, you must use the -s NLVM option to override the shared locking requirement and force NLVM to execute the commands you use to create the SBD partition. To minimize the risk of possible corruption, you are responsible for ensuring that you have exclusive access to the shared storage at this time.

Initializing and Sharing a Device for the SBD

When you use sbdutil to create an SBD, you must initialize the SAN device that you created for the SBD, and mark it as Shareable for Clustering before you create the SBD partition. When you mark the device as Shareable for Clustering, share information is added to the disk in a free-space partition that is about 4 MB in size. This space becomes part of the SBD partition.

When you use NLVM to create an SBD, the nlvm create partition command can accept an initialized or uninitialized device when you use the type=1ad option. NLVM checks the specified device to see if it is initialized, and takes the following actions:

Uninitialized device: NLVM initializes the device, marks it as Shareable for Clustering, and creates the requested SBD partition.
Initialized and shared device: NLVM creates the requested SBD partition.
Initialized and unshared device: NLVM creates the requested SBD partition, but does not alter the shared state. It returns an error warning that the SBD partition is not shared.

You must manually mark the device as Shareable for Clustering after the partition is created. You can use the nlvm share command to share the device.

Determining the SBD Partition Size

When you create the SBD partition by using the nlvm create partition command, you can specify how much free space to use for the SBD, or you can specify the max option to use the entire device. If you specify a device to use as a mirror, the same amount of space is used. If you specify to use the maximum size and the mirror device is bigger than the SBD device, you will not be able to use the excess free space on the mirror for other purposes.

Because an SBD partition must end on a cylinder boundary, the partition size might be slightly smaller than the size you specify. When you use an entire device for the SBD partition, you can use the max option as the size, and let the software determine the size of the partition.

Replacing an Existing SBD Partition

To replace an existing SBD partition, you must first delete the old SBD partition, and then create the new one. To reuse the SBD partition’s device, you must remove the SBD partition, and then re-initialize and share the device.

You must take the cluster down and stop OES Cluster Services on all nodes before you delete the existing SBD partition. Do not restart OES Cluster Services and rejoin nodes to the cluster until after you create the new SBD.

8.1.2 Creating a Non-Mirrored SBD Partition with NLVM

Use the procedure in this section to create a new SBD partition. If an SBD partition already exists, you must first delete the SBD as described in Section 8.3, Deleting an SBD Partition with NLVM.

Ensure that nobody else is changing any storage on any nodes at this time.

Until the SBD exists and the cluster is set up for shared disk access, you are responsible for ensuring that you have exclusive access to the shared storage.
Take the cluster down:
1. Log in to any node in the cluster as the root user, then open a terminal console.
2. At the command prompt, enter
```
cluster down
```
On each cluster node, stop OES Cluster Services:
1. Log in to the cluster node as the root user, then open a terminal console.
2. At the command prompt, enter
```
rcnovell-ncs stop
```
3. After you have stopped OES Cluster Services on all nodes, continue with the next step.
Prepare a SAN device to use for the SBD partition:
1. Use the SAN storage array software to carve a device to use exclusively for the SBD partition.
2. Attach the device to all nodes in the cluster.
3. On each node, log in as the root user and rescan for devices:
```
nlvm -s rescan
```
  Use the -s NLVM option to override the shared locking requirement and force the command to execute.
Log in to any node in the cluster as the root user, then open a terminal console.
View a list of the devices and identify the leaf node name (such as sdc) of the SAN device that you want to use for the SBD partition. At the command prompt, enter
```
nlvm -s list devices --terse
```
Use the -s NLVM option to override the shared locking requirement and force the command to execute.

The device information shows the leaf node name, the size, the amount of free available space, the partitioning format (such as MSDOS or GPT), the shared state (whether it is marked as Shareable for Clustering), and the RAID state (whether the device is an NSS software RAID device). Do not use an NSS software RAID for the device.

For example, the uninitialized device sdc reports a used and free size of 0 KB, a format of None, and a shared state of No:
```
 Name        Size      Used      Free  Format Shared RAID Enabled
 sda       20.00GB   19.99GB    1008KB  MSDOS   No    No
 sdb        1.00GB  400.01MB  623.98MB  MSDOS   No    No
 sdc      102.00MB       0KB       0KB   None   No    No
 sdd        8.00GB   50.01MB    7.95GB  MSDOS  Yes    No
```
Initialize and share the device. At the command prompt, enter
```
nlvm -s init <device_name> format=msdos shared
```
WARNING:Initializing a device destroys all data on the device.

Replace device_name with the leaf node name (such as sdc) of the SAN device you want to use as the SBD partition.

Specify a partitioning format of msdos.

Specify the shared option to mark the device as Shareable for Clustering.

Use the -s NLVM option to override the shared locking requirement and force the command to execute.

You can list the devices to visually verify that the device is formatted and shared:
```
nlvm -s list devices
```
For example, the formatted device sdc reports values for used and free size, a format of MSDOS, and a shared state of Yes:
```
 Name        Size      Used      Free  Format Shared RAID Enabled
 sda       20.00GB   19.99GB    1008KB  MSDOS   No    No
 sdb        1.00GB  400.01MB  623.98MB  MSDOS   No    No
 sdc      102.00MB      16KB  101.98MB  MSDOS  Yes    No
 sdd        8.00GB   50.01MB    7.95GB  MSDOS  Yes    No
```
Create the SBD partition. At the command prompt, enter (all on the same line):
```
nlvm -s create partition 
        type=1ad 
        device=<device_name> 
        size=<value[K|M|G|T]|max>
        label="<cluster_name>"
```
Specify a type of 1ad to create the SBD partition type.

Replace device_name with the leaf node name (such as sdc) of the SAN device you want to use as the SBD partition.

Replace value with the amount of space to use for the SBD partition and select a unit of measure as its multiplier, or specify max to use the entire device. If you specify a value without a multiplier, gigabytes (G) is assumed.

Replace cluster_name with the name of the cluster, such as cluster1. This name must match the name of an existing cluster that has a Cluster object in eDirectory. The name is case sensitive.

Use the -s NLVM option to override the shared locking requirement and force the command to execute.

For example, to create an SBD partition for a cluster named cluster1 on device sdc that has already been initialized and shared, enter
```
nlvm -s create partition type=1ad device=sdc size=max label="cluster1"
```
A partition is created named cluster1.sbd. It uses all available free space on the specified device.
View a list of partitions and verify that the new partition appears in the list. At the command prompt, enter
```
nlvm -s list partitions
```
Use the -s NLVM option to override the shared locking requirement and force the command to execute.

The partition information shows the partition name, the leaf node name of the device, the partition type (1AD), the starting location, and the partition size. Because an SBD partition must end at a cylinder boundary, the partition size might be slightly smaller than the device size, or the size you specified for the partition.

For example, for device sdc that is 102 MB in size, the partition created is 99.59 MB in size:
```
Name                  Type   Start   Size   Device
sda1             83(Linux)    2048 297.00MB sda
sda2         82Linux_Swap)  610304   1.00GB sda
sda3             83(Linux) 2715648   7.99GB sda
cluster1.sbd  1AD(Cluster)      32  99.59MB sdc
```
If you specified the maximum size for the SBD partition, you can list devices again to see that all space on the device is used for the SBD partition:
```
 Name        Size      Used      Free  Format Shared RAID Enabled
 sda       20.00GB   19.99GB    1008KB  MSDOS   No    No
 sdb        1.00GB  400.01MB  623.98MB  MSDOS   No    No
 sdc      102.00MB  102.00MB       0KB  MSDOS  Yes    No
 sdd        8.00GB   50.01MB    7.95GB  MSDOS  Yes    No
```
Modify the Cluster object in eDirectory to enable its NCS: Shared Disk Flag attribute.

This step is required only if the cluster has never had an SBD partition. However, it does no harm to verify that the NCS: Shared Disk Flag attribute is enabled.
1. In a web browser, open iManager, then log in to the eDirectory tree that contains the cluster you want to manage.
  
  IMPORTANT:Log in as an administrator user who has sufficient rights in eDirectory to delete and modify eDirectory objects.
2. Select Directory Administration, then select Modify Object.
3. Browse to locate and select the Cluster object of the cluster you want to manage, then click OK.
4. Under Valued Attributes, select the NCS: Shared Disk Flag, then click Edit.
5. Select (enable) the NCS: Shared Disk Flag check box, then click OK.
6. Click Apply to save changes.
On each cluster node, start OES Cluster Services:
1. Log in to the cluster node as the root user, then open a terminal console.
2. At the command prompt, enter
```
rcnovell-ncs start
```
3. After you have restarted OES Cluster Services on all nodes, continue with the next step.
On each cluster node, join the cluster. At the command prompt, enter
```
cluster join
```
(Optional) Continue with Section 8.1.3, Mirroring an Existing SBD Partition with NLVM.

8.1.3 Mirroring an Existing SBD Partition with NLVM

You can mirror an existing OES Cluster Services SBD partition to provide device fault tolerance. It is not necessary to take the cluster down or stop the cluster software.

Prepare a SAN device to use as the mirror segment for the SBD partition:
1. Use the SAN storage array software to carve a device that is at least the size of the existing SBD partition’s device.
2. Attach the device to all nodes in the cluster.
3. On each node, log in as the root user and rescan for devices:
```
nlvm rescan
```
Log in to any member node of the cluster as the root user, then open a terminal console.

View a list of the devices and identify the leaf node name (such as sde) of the SAN device that you want to use as the mirror for the existing SBD partition. At the command prompt, enter

nlvm list devices

For example, the uninitialized device sde reports a used and free size of 0 KB, a format of None and a shared state of No:

 Name        Size      Used      Free  Format Shared RAID Enabled
 sda       20.00GB   19.99GB    1008KB  MSDOS   No    No
 sdb        1.00GB  400.01MB  623.98MB  MSDOS   No    No
 sdc      102.00MB  102.00MB       0KB  MSDOS  Yes    No
 sdd        8.00GB   50.01MB    7.95GB  MSDOS  Yes    No
 sde      102.00MB       0KB       0KB   None   No    No

Initialize and share the device. At the command prompt, enter
```
nlvm init <device_name> format=msdos shared
```
WARNING:Initializing a device destroys all data on the device.

Replace device_name with the leaf node name (such as sde) of the SAN device you want to use as the mirror for the existing SBD partition.

Specify a partitioning format of msdos.

Specify the shared option to mark the device as Shareable for Clustering.

You can list the devices to visually verify that the device is formatted and shared:
```
nlvm list devices
```
For example, the formatted device sde reports a format of MSDOS and a shared state of Yes:
```
 Name        Size      Used      Free  Format Shared RAID Enabled
 sda       20.00GB   19.99GB    1008KB  MSDOS   No    No
 sdb        1.00GB  400.01MB  623.98MB  MSDOS   No    No
 sdc      102.00MB  102.00MB       0KB  MSDOS  Yes    No
 sdd        8.00GB   50.01MB    7.95GB  MSDOS  Yes    No
 sde      102.00MB      16KB  101.98MB  MSDOS  Yes    No
```
Mirror the SBD partition. At the command prompt, enter (all on the same line):
```
nlvm create raid
     raid=1
     name=<cluster_name>
     type=sbd 
     part=<partition_name>
     device=<device_name> 
```
Specify a RAID type of 1 for mirroring.

Replace cluster_name with the name of the SBD’s cluster, such as cluster1. This name must match the name of an existing cluster that has a Cluster object in eDirectory. The name is case sensitive.

Specify a type of sbd to create SBD partitions on the RAID1 device. The type option must precede the part option in the command.

Replace partition_name with the partition name of the existing SBD partition.

Replace device_name with the leaf node name (such as sde) of the SAN device you want to use as the mirror for the existing SBD partition. The device must be at least the same size as the partition you want to mirror.

You do not specify a size in the command. The size of the existing partition determines the amount of space that is used for the mirrored segment.

For example, to mirror the SBD partition cluster1.sbd with device sde for a cluster named cluster1, enter
```
nlvm create raid raid=1 type=sbd name=cluster1 part=cluster1.sbd device=sde
```
For our example, a RAID1 (mirror) device is created named cluster1.sbd that is made up of device sdc and device sde. The existing SBD partition is renamed from cluster1.sbd to cluster1.msbd0. A new partition named cluster1.msbd1 is created on device sde.

View a list of devices to verify the current state of both devices and to verify that a RAID1 device named cluster1.sbd was created. At the command prompt, enter

nlvm list devices

The entries of interest in the devices list are the devices that you use for the SBD partition (such as sdc and sde) and the newly created RAID1 device:

 Name             Size      Used      Free  Format Shared RAID Enabled
 sda            20.00GB   19.99GB    1008KB  MSDOS   No    No
 sdb             1.00GB  400.01MB  623.98MB  MSDOS   No    No
 sdc           102.00MB  102.00MB       0KB  MSDOS  Yes    No
 sdd             8.00GB   50.01MB    7.95GB  MSDOS  Yes    No
 sde           102.00MB  102.00MB       0KB  MSDOS  Yes    No
 cluster1.sbd   99.57MB   97.57MB       0KB   None  Yes     1    Yes

View a list of partitions to verify the status of mirrored SBD partitions cluster1.msbd0 and cluster1.msbd1. At the command prompt, enter

nlvm list partitions

The entries of interest in the list are cluster1.msbd0 and cluster1.msbd1:

Name                    Type   Start   Size   Device
sda1               83(Linux)    2048 297.00MB sda
sda2           82Linux_Swap)  610304   1.00GB sda
sda3               83(Linux) 2715648   7.99GB sda
cluster1.msbd0  1AD(Cluster)      32  99.59MB sdc
cluster1.msbd1  1AD(Cluster)      32  99.59MB sde

8.1.4 Creating a Mirrored SBD Partition with NLVM

You can create a mirrored OES Cluster Services SBD partition to provide device fault tolerance for the SBD. You must take the cluster down and stop the cluster software. If an SBD partition already exists, you must first delete the SBD as described in Section 8.3, Deleting an SBD Partition with NLVM.

Use the procedure in this section to create a new mirrored SBD partition by using NLVM commands.

Ensure that nobody else is changing any storage on any nodes at this time.

Until the SBD exists and the cluster is set up for shared disk access, you are responsible for ensuring that you have exclusive access to the shared storage.
Take the cluster down:
1. Log in to any node in the cluster as the root user, then open a terminal console.
2. At the command prompt, enter
```
cluster down
```
On each cluster node, stop OES Cluster Services:
1. Log in to the cluster node as the root user, then open a terminal console.
2. At the command prompt, enter
```
rcnovell-ncs stop
```
3. After you have stopped OES Cluster Services on all nodes, continue with the next step.
Prepare two SAN devices to use for the mirrored SBD partition:
1. Use the SAN storage array software to carve two devices of equal size to use exclusively for the mirrored SBD partition.
2. Attach the devices to all nodes in the cluster.
3. On each node, log in as the root user and rescan for devices:
```
nlvm -s rescan
```
  Use the -s NLVM option to override the shared locking requirement and force the command to execute.
Log in to any node in the cluster as the root user, then open a terminal console.
View a list of the devices and identify the leaf node name (such as sdc) of the two SAN devices that you want to use for the mirrored SBD partition. At the command prompt, enter
```
nlvm -s list devices
```
Use the -s NLVM option to override the shared locking requirement and force the command to execute.

The device information shows the leaf node name, the size, the amount of free available space, the partitioning format (such as MSDOS or GPT), the shared state (whether it is marked as Shareable for Clustering), and the RAID state (whether the device is an NSS software RAID device). Do not use an NSS software RAID for the device.
Initialize and share the two devices. At the command prompt, enter
```
nlvm -s init <device_name1>,<device_name2> format=msdos shared
```
WARNING:Initializing a device destroys all data on the device.

Replace device_name1 and device_name2 with the leaf node names (such as sdc and sdd) of the two SAN devices you want to use for the mirrored SBD partition.

Specify a partitioning format of msdos.

Specify the shared option to mark the devices as Shareable for Clustering.

Use the -s NLVM option to override the shared locking requirement and force the command to execute.

For example, to initialize devices sdc and sdd, enter
```
nlvm -s init sdc,sdd format=msdos shared
```
You can list the devices to visually verify that the device is formatted and shared:
```
nlvm -s list devices
```
Create the mirrored SBD partition. At the command prompt, enter (all on the same line):
```
nlvm -s create raid 
     raid=1 
     type=sbd 
     name=<cluster_name> 
     device=<device_name1> size=max 
     device=<device_name2>
```
Specify a RAID type of 1 for mirroring.

Specify a type of sbd to create SBD partitions on the RAID1 device.

Replace cluster_name with the name of the cluster, such as cluster1. This name must match the name of an existing cluster that has a Cluster object in eDirectory. The name is case sensitive.

Replace device_name1 and device_name2 with the leaf node names (such as sdc and sdd) of the two SAN devices you want to use for the mirrored SBD partition. The cluster1.msbd0 mirrored SBD partition is created on the first device option instance in the command. The cluster1.msbd1 mirrored SBD partition is created on the second device option instance in the command.

Specify a size of max to use all of the available space. Specify the size only once. Both devices should be the same size; however, if they are not, the size of the RAID segments is determined by the size of the smaller device.

Use the -s NLVM option to override the shared locking requirement and force the command to execute.

For example, to create a mirrored SBD for a cluster named cluster1 with devices sdc and sdd that have already been initialized and shared, enter
```
nlvm -s create raid raid=1 type=sbd name="cluster1" device=sdc size=max device=sdd
```
A RAID1 device is created named cluster1.sbd. The cluster1.msbd0 partition is created on device sdc. The cluster1.msbd1 partition is created on device sdd.
View a list of devices to verify the current state of both devices and to verify that a RAID1 device named cluster1.sbd was created. At the command prompt, enter
```
nlvm -s list devices
```
View a list of partitions and verify that the new partitions appear in the list. At the command prompt, enter
```
nlvm -s list partitions
```
The partition information shows the partition name, the leaf node name of the device, the partition type (1AD), the starting location, and the partition size. Because an SBD partition must end at a cylinder boundary, the partition size might be slightly smaller than the device size, or the size you specified for the partition.

You can list devices again to see the amount of space that is unused beyond the cylinder boundary. Our example devices show 2.39 MB of free space after the partition is created, as shown in Step 9.
Modify the Cluster object in eDirectory to enable its NCS: Shared Disk Flag attribute.

This step is required only if the cluster has never had an SBD partition. However, it does no harm to verify that the NCS: Shared Disk Flag attribute is enabled.
1. In a web browser, open iManager, then log in to the eDirectory tree that contains the cluster you want to manage.
  
  IMPORTANT:Log in as an administrator user who has sufficient rights in eDirectory to delete and modify eDirectory objects.
2. Select Directory Administration, then select Modify Object.
3. Browse to locate and select the Cluster object of the cluster you want to manage, then click OK.
4. Under Valued Attributes, select the NCS: Shared Disk Flag, then click Edit.
5. Select (enable) the NCS: Shared Disk Flag check box, then click OK.
6. Click Apply to save changes.
On each cluster node, start OES Cluster Services:
1. Log in to the cluster node as the root user, then open a terminal console.
2. At the command prompt, enter
```
rcnovell-ncs start
```
3. After you have restarted OES Cluster Services on all nodes, continue with the next step.
On each cluster node, join the cluster. At the command prompt, enter
```
cluster join
```