Exadata Database Servers Administration

Management Server (MS) running on database servers provides monitoring, alerting, and other administrative capabilities. It also provides the DBMCLI command-line administration tool.

Maintaining the Hard Disks of Oracle Database Servers

Repair of the hard disks does not require the database server in Oracle Exadata Database Machine to be shut down.

No downtime of the rack is required, however individual servers may require downtime, and be taken outside of the cluster temporarily.

Verifying the Database Server Configuration

Oracle recommends verifying the status of the database server RAID devices to avoid possible performance impact, or an outage.

The impact of validating the RAID devices is minimal. The impact of corrective actions will vary depending on the specific issue uncovered, and may range from simple reconfiguration to an outage.

RAID Disk Configuration

The disks are configured RAID-5 configuration.

The disk drives in each database server are controlled by a MegaRAID SAS disk controller.

Disk Configurations for Exadata Database Machine Two-Socket Systems

Server TypeRAID ControllerDisk Configuration
Oracle Exadata Database Machine X8-2MegaRAID SAS 9361-16i4 disk drives in each database server
Oracle Exadata Database Machine X7-2MegaRAID SAS 9361-16i4 disk drives in each database server
Oracle Exadata Database Machine X6-2MegaRAID SAS 9361-8i4 disk drives in each database server
Oracle Exadata Database Machine X5-2MegaRAID SAS 9361-8i4 disk drives in each database server
Oracle Exadata Database Machine X4-2MegaRAID SAS 9261-8i4 disk drives in each database server
Oracle Exadata Database Machine X3-2MegaRAID SAS 9261-8i4 disk drives in each database server
Oracle Exadata Database Machine X2-2MegaRAID SAS 9261-8i4 disk drives in each database server

Disk Configurations for Exadata Database Machine Eight-Socket Systems

Server TypeRAID ControllerDisk Configuration
Oracle Exadata Database Machine X8-8N/ATwo NVMe flash accelerator cards in each database server
Oracle Exadata Database Machine X7-8N/ATwo NVMe flash accelerator cards in each database server
Oracle Exadata Database Machine X5-8MegaRAID SAS 9361-8i8 disk drives in each database server with one virtual drive created across the RAID set
Oracle Exadata Database Machine X4-8MegaRAID SAS 9261-8i7 disk drives in each database server configured as one 6-disk RAID-5 with one global hot spare drive by default
Oracle Exadata Database Machine X3-8MegaRAID SAS 9261-8i8 disk drives in each database server with one virtual drive created across the RAID set

Verifying Disk Controller Configuration on Oracle Exadata Database Machine X7-8 or Later Systems

  • Query mdstat to view the database server disk controller configuration on Oracle Exadata Database Machine X7-8 or later systems.

[[email protected] ~]# cat /proc/mdstat

Personalities : [raid1]

md34 : active raid1 nvme3n1[1] nvme1n1[0]

      3125613568 blocks super external:/md126/0 [2/2] [UU]

md24 : active raid1 nvme2n1[1] nvme0n1[0]

      262144000 blocks super external:/md127/0 [2/2] [UU]

md25 : active raid1 nvme2n1[1] nvme0n1[0]

      2863467520 blocks super external:/md127/1 [2/2] [UU]

md126 : inactive nvme3n1[1](S) nvme1n1[0](S)

      6306 blocks super external:imsm

md127 : inactive nvme2n1[1](S) nvme0n1[0](S)

      6306 blocks super external:imsm

       unused devices: <none>

If the output you see is different, then investigate and correct the problem. Degraded virtual drives usually indicate absent or failed physical disks. Disks that show [1/2] and [U_] or [_U] for the state indicate that one of the NVME disks is down. Failed disks should be replaced quickly.

Verifying Physical Drive Configuration

Check your system for critical, degraded, or failed disks.

To verify physical drive configuration, use the following command to verify the database server physical drive configuration:

/opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL | grep “Firmware state”

The following is an example of the output for Oracle Exadata Database Machine X4-2, Oracle Exadata Database Machine X3-2, and Oracle Exadata Database Machine X2-2:

Firmware state: Online, Spun Up

Firmware state: Online, Spun Up

Firmware state: Online, Spun Up

Firmware state: Online, Spun Up

The drives should show a state of Online, Spun Up. The order of the output is not important. The output for Oracle Exadata Database Machine X3-8 Full Rack or Oracle Exadata Database Machine X2-8 Full Rack should be eight lines of output showing a state of Online, Spun Up.

If your output is different, then investigate and correct the problem.

Degraded virtual drives usually indicate absent or failed physical disks. Critical disks should be replaced immediately to avoid the risk of data loss if the number of failed disks in the node exceed the count needed to sustain the operations of the system. Failed disks should be replaced quickly.

Monitoring a Database Server RAID Set Rebuilding

If a drive in a database server RAID set is replaced, then the progress of the RAID set rebuild should be monitored.

Use the following command on the database server that has the replaced disk. The command is run as the root user.

/opt/MegaRAID/MegaCli/MegaCli64 -pdrbld -showprog -physdrv \

[disk_enclosure:slot_number] -a0

In the preceding command, disk_enclosure and slot_number indicate the replacement disk identified by the MegaCli64 -PDList command. The following is an example of the output from the command:

Rebuild Progress on Device at Enclosure 252, Slot 2 Completed 41% in 13 Minutes.

Reclaiming a Hot Spare Drive After Upgrading to Oracle Exadata System Software Release 12.1.2.1.0 or Later

Oracle Exadata Database Machines upgraded to Oracle Exadata System Software release 12.1.2.1.0 or later that have a hot spare drive cannot use the reclaimdisks.sh script to reclaim the drive. The following procedure describes how to manually reclaim the drive:

The sample output shows Oracle Exadata Database Machine X2-2 database server with four disks. The enclosure identifier, slot number, and such may be different for your system.

  1. Identify the hot spare drive.

# /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL

The following is an example of the output from the command for the hot spare drive:

Enclosure Device ID: 252

Slot Number: 3

Enclosure position: N/A

Device Id: 8

WWN: 5000CCA00A9FAA5F

Sequence Number: 2

Media Error Count: 0

Other Error Count: 0

Predictive Failure Count: 0

Last Predictive Failure Event Seq Number: 0

PD Type: SAS

Hotspare Information:

Type: Global, with enclosure affinity, is revertible

Raw Size: 279.396 GB [0x22ecb25c Sectors]

Non Coerced Size: 278.896 GB [0x22dcb25c Sectors]

Coerced Size: 278.464 GB [0x22cee000 Sectors]

Sector Size: 0

Logical Sector Size: 0

Physical Sector Size: 0

Firmware state: Hotspare, Spun down

Device Firmware Level: A2A8

Shield Counter: 0

Successful diagnostics completion on : N/A

The command identified the hot spare drive on enclosure identifier 252, slot 3.

  1. Obtain the virtual drive information.

# /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -Aall

The following is an example of the output from the command:

Adapter 0 — Virtual Drive Information:

Virtual Drive: 0 (Target Id: 0)

Name :DBSYS

RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3

Size : 556.929 GB

Sector Size : 512

Is VD emulated : No

Parity Size : 278.464 GB

State : Optimal

Strip Size : 1.0 MB

Number Of Drives : 3

Span Depth : 1

Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU

Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU

Default Access Policy: Read/Write

Current Access Policy: Read/Write

Disk Cache Policy : Disabled

Encryption Type : None

Is VD Cached: No

The command identified a RAID 5 configuration for virtual drive 0 on adapter 0.

  1. Remove the hot spare drive.

# /opt/MegaRAID/MegaCli/MegaCli64 -PDHSP -Rmv -PhysDrv[252:3] -a0

  1. Add the drive as an active RAID 5 drive.

# /opt/MegaRAID/MegaCli/MegaCli64 -LDRecon -Start -r5     \

  -Add -PhysDrv[252:3] -L0 -a0

Start Reconstruction of Virtual Drive Success.

Exit Code: 0x00

  1. Monitor the progress of the RAID reconstruction.

# /opt/MegaRAID/MegaCli/MegaCli64 -LDRecon -ShowProg -L0 -a0

Reconstruction on VD #0 (target id #0) Completed 1% in 2 Minutes.

The following output shows the output of the command after the hot spare drive is added to the RAID 5 configuration, and the reconstruction is finished:

Reconstruction on VD #0 is not in Progress.

  1. Verify the number of drives.

# /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -Aall

The following is an example of the output from the command:

Adapter 0 — Virtual Drive Information:

Virtual Drive: 0 (Target Id: 0)

Name :DBSYS

RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3

Size : 835.394 GB

Sector Size : 512

Is VD emulated : No

Parity Size : 278.464 GB

State : Optimal

Strip Size : 1.0 MB

Number Of Drives : 4

Span Depth : 1

Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU

Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU

Default Access Policy: Read/Write

Current Access Policy: Read/Write

Disk Cache Policy : Disabled

Encryption Type : None

Is VD Cached: No

  1. Check the size of the RAID.

# parted /dev/sda print

Model: LSI MR9261-8i (scsi)

Disk /dev/sda: 598GB

Sector size (logical/physical): 512B/4096B

Partition Table: msdos

Number Start End Size Type File system Flags

1 32.3kB 132MB 132MB primary ext3 boot

2 132MB 598GB 598GB primary lvm

  1. Restart the server in order for the changes to take effect.
  2. Check the size of the RAID again.

# parted /dev/sda print

Model: LSI MR9261-8i (scsi)

Disk /dev/sda: 897GB

Sector size (logical/physical): 512B/4096B

Partition Table: msdos

Number Start End Size Type File system Flags

1 32.3kB 132MB 132MB primary ext3 boot

2 132MB 598GB 598GB primary lvm

The increased RAID size allows for extending the volume group. To extend the volume group, you must add an additional partition to the drive.

  1. Obtain the new size, in sectors.

# parted /dev/sda

GNU Parted 2.1

Using /dev/sda

Welcome to GNU Parted! Type ‘help’ to view a list of commands.

(parted) unit s

(parted) print

Model: LSI MR9261-8i (scsi)

Disk /dev/sda: 1751949312s

Sector size (logical/physical): 512B/4096B

Partition Table: msdos

Number Start End Size Type File system Flags

1 63s 257039s 256977s primary ext3 boot

2 257040s 1167957629s 1167700590s primary lvm

In the preceding example, a third partition can be created starting at sector 1167957630, and ending at the end of the disk at sector 1751949311.

  1. Create an additional partition on the drive.

# parted /dev/sda

GNU Parted 2.1

Using /dev/sda

Welcome to GNU Parted! Type ‘help’ to view a list of commands.

(parted) unit s

(parted) mkpart

Partition type? primary/extended? primary

File system type? [ext2]? ext2

Start? 1167957630

End? 1751949311

Warning: The resulting partition is not properly aligned for best performance.

Ignore/Cancel? Ignore

Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy). As a

result, it may not reflect all of your changes until after reboot.

(parted)

(parted) print

Model: LSI MR9261-8i (scsi)

Disk /dev/sda: 1751949312s

Sector size (logical/physical): 512B/4096B

Partition Table: msdos

Number Start End Size Type File system Flags

1 63s 257039s 256977s primary ext3 boot

2 257040s 1167957629s 1167700590s primary lvm

3 1167957630s 1751949311s 583991682s primary

(parted) set 3 lvm on

Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy). As a

result, it may not reflect all of your changes until after reboot.

(parted) print

Model: LSI MR9261-8i (scsi)

Disk /dev/sda: 1751949312s

Sector size (logical/physical): 512B/4096B

Partition Table: msdos

Number Start End Size Type File system Flags

1 63s 257039s 256977s primary ext3 boot

2 257040s 1167957629s 1167700590s primary lvm

3 1167957630s 1751949311s 583991682s primary lvm

  1. Restart the database server.
  2. Create the physical volume.

# pvcreate /dev/partition_name

  1. Add the physical volume to the existing volume group.

In the following example, substitute the actual names for the volume_grouppartition_name, and volume_name.

# vgextend volume_group /dev/partition_name

Volume group “volume_name” successfully extended

  1. Resize the logical volume and file systems.

Understanding Automated File Deletion Policy

Management Server (MS) includes a file deletion policy for the / (root) directory on the database servers that is triggered when file system utilization is high. Deletion of files is triggered when file utilization is 80 percent, and an alert is sent before the deletion begins. The alert includes the name of the directory, and space usage for the subdirectories. In particular, the deletion policy is as follows:

Files in the following directories are deleted using a policy based on the file modification time stamp.

  • /opt/oracle/dbserver/log
  • /opt/oracle/dbserver/dbms/deploy/config/metrics
  • /opt/oracle/dbserver/dbms/deploy/log

Files that are older than the number of days set by the metricHistoryDays attribute are deleted first, then successive deletions occur for earlier files, down to files with modification time stamps older than or equal to 10 minutes, or until file system utilization is less than 75 percent. The metricHistoryDays attribute applies to files in /opt/oracle/dbserver/dbms/deploy/config/metrics. For the other log and trace files, use the diagHistoryDays attribute.

Starting with Oracle Exadata System Software release 12.1.2.2.0, the maximum amount of space for ms-odl.trc and ms-odl.log files is 100 MB (twenty 5 MB files) for *.trc files and 100 MB (twenty 5 MB files) for *.log files. Previously, it was 50 MB (ten 5 MB files) for both *.trc and *.log files.

ms-odl generation files are renamed when they reach 5 MB, and the oldest are deleted when the files use up 100 MB of space.

Maintaining Flash Disks on Exadata Database Servers

Flash disks should be monitored and replaced when necessary.

Starting with Exadata Database Machine X7-8, the database servers contain flash devices instead of hard disks. These flash devices can be replaced without shutting down the server.

Monitoring the Status of Flash Disks

You can monitor the status of a flash disk on the Exadata Database Machine by checking its attributes with the DBMCLI LIST PHYSICALDISK command.

For example, a flash disk status equal to failed is probably having problems and needs to be replaced.

  • Use the DBMCLI command LIST PHSYICALDISK to determine the status of a flash disk:

DBMCLI> LIST PHYSICALDISK WHERE disktype=flashdisk AND status!=normal DETAIL

         name:               FLASH_1_1

         deviceName:         /dev/nvme0n1

         diskType:           FlashDisk

         luns:               1_1

         makeModel:          “Oracle Flash Accelerator F640 PCIe Card”

         physicalFirmware:   QDV1RD09

         physicalInsertTime: 2017-08-11T12:25:00-07:00

         physicalSerial:     PHLE6514003R6P4BGN-1

         physicalSize:       2.910957656800747T

         slotNumber:         “PCI Slot: 1; FDOM: 1”

         status:             failed – dropped for replacement

Performing a Hot-Pluggable Replacement of a Flash Disk

For Oracle Exadata Database Machine X7-8 and X8-8 models, the database server uses hot-pluggable flash disks instead of hard disk drives.

  1. Determine if the flash disk is ready to be replaced.

When performing a hot-pluggable replacement of a flash device on Oracle Exadata Database Machine X7-8 and X8-8 database servers, the disk status should be Dropped for replacement, which indicates the flash disk is ready for online replacement.

DBMCLI> LIST PHYSICALDISK WHERE DISKTYPE=flashdisk AND STATUS LIKE ‘.*dropped

for replacement.*’ DETAIL

         name:               FLASH_1_1

         deviceName:         /dev/nvme0n1

         diskType:           FlashDisk

         luns:               1_1

         makeModel:          “Oracle Flash Accelerator F640 PCIe Card”

         physicalFirmware:   QDV1RD09

         physicalInsertTime: 2017-08-11T12:25:00-07:00

         physicalSerial:     PHLE6514003R6P4BGN-1

         physicalSize:       2.910957656800747T

         slotNumber:         “PCI Slot: 1; FDOM: 1

         status:             failed – dropped for replacement

  1. Locate the failed flash disk based on the PCI number and FDOM number.

A white Locator LED is lit to help locate the affected database server. An amber Fault-Service Required LED is lit to identify the affected flash card.

  1. Make sure the DPCC OK LED is off on the card.
  2. Remove and replace the failed flash disk.
    1. Slide out the DPCC and replace the flash card inside.
    2. Slide the DPCC carrier back to the slot.
  3. Use a stylus to press both ATTN buttons on the front of the DPCC.
  1. If only a single PCIe card is present, press only the corresponding ATTN button.
  2. If you are not performing a hot-pluggable replacement, then this step is not necessary.

The buttons alert the system to a request to bring the devices online. When the system acknowledges the request, it lights the DPCC OK LED indicators on the DPCC. If you do not press the ATTN buttons, then the flash disk will not be detected by the operating system.

Viewing the Network Interfaces

To view the network interfaces, you can run the ipconf.pl command.

Viewing the default network interfaces for an Oracle Exadata Database Machine X8M-2 database server

The following example shows the output for an Oracle Exadata Database Machine X8M-2 database server without the additional network card. In addition to the RDMA Network Fabric interfaces, the output shows the interfaces for three network cards:

  • A single port 1/10Gb card, eth0
  • A dual port 10 or 25Gb card, on eth1 and eth2
  • A dual port 10 or 25Gb card, on eth3 and eth4

[email protected] ibdiagtools]# /opt/oracle.cellos/ipconf.pl

[Info]: ipconf command line: /opt/oracle.cellos/ipconf.pl

Logging started to /var/log/cellos/ipconf.log

Interface re0   is      Linked.    hca: mlx5_0

Interface re1   is      Linked.    hca: mlx5_0

Interface eth0  is      Linked.    driver/mac: igb/00:10:e0:c3:b7:9c

Interface eth1  is      Unlinked.  driver/mac: bnxt_en/00:10:e0:c3:b7:9d (slave of bondeth0)

Interface eth2  is      Linked.    driver/mac: bnxt_en/00:10:e0:c3:b7:9d (slave of bondeth0)

Interface eth3  is      Unlinked.  driver/mac: bnxt_en/00:0a:f7:c3:28:30

Interface eth4  is      Unlinked.  driver/mac: bnxt_en/00:0a:f7:c3:28:38

Viewing the default network interfaces for an Oracle Exadata Database Machine X7-2 or X8-2 database server

The following example shows the output for an Oracle Exadata Database Machine X7-2 or X8-2 database server without the additional network card. In addition to the RDMA Network Fabric interfaces, the output shows the interfaces for three network cards:

  • A single port 10Gb card, on eth0
  • A dual port 10 or 25Gb card, on eth1 and eth2
  • A dual port 25Gb card, on eth3 and eth4

# /opt/oracle.cellos/ipconf.pl

Logging started to /var/log/cellos/ipconf.log

Interface ib0   is          Linked.    hca: mlx4_0

Interface ib1   is          Linked.    hca: mlx4_0

Interface eth0  is          Linked.    driver/mac: igb/00:

10:e0:c3:ba:72

Interface eth1  is          Linked.    driver/mac: bnxt_en

/00:10:e0:c3:ba:73

Interface eth2  is          Linked.    driver/mac: bnxt_en

/00:10:e0:c3:ba:74

Interface eth3  is          Linked.    driver/mac: bnxt_en

/00:0a:f7:c3:14:a0 (slave of bondeth0)

Interface eth4  is          Linked.    driver/mac: bnxt_en

/00:0a:f7:c3:14:a0 (slave of bondeth0)

Viewing the default network interfaces for an Oracle Exadata Database Machine X6-2 database server

The following example shows the output for an Oracle Exadata Database Machine X6-2 database server without the additional network card. In addition to the RDMA Network Fabric interfaces, the output shows the interfaces for two network cards:

  • A quad port 10Gb card, on eth0 to eth3
  • A dual port 10Gb card, on eth4 and eth5

# cd /opt/oracle.cellos/

# ./ipconf.pl

Logging started to /var/log/cellos/ipconf.log

Interface ib0   is          Linked.    hca: mlx4_0

Interface ib1   is          Linked.    hca: mlx4_0

Interface eth0  is          Linked.    driver/mac: ixgbe/00:10:e0:8b:24:b6

Interface eth1  is …..    Linked.    driver/mac: ixgbe/00:10:e0:8b:24:b7

Interface eth2  is …..    Linked.    driver/mac: ixgbe/00:10:e0:8b:24:b8

Interface eth3  is …..    Linked.    driver/mac: ixgbe/00:10:e0:8b:24:b9

Interface eth4  is          Linked.    driver/mac: ixgbe/90:e2:ba:ac:20:ec (slave of bondeth0)

Interface eth5  is           Linked.    driver/mac: ixgbe/90:e2:ba:ac:20:ec (slave of bondeth0)

Creating a Snapshot-Based Backup of Oracle Linux Database Server

A backup should be made before and after every significant change to the software on the database server. For example, a backup should be made before and after the following procedures:

  • Application of operating system patches
  • Application of Oracle patches
  • Reconfiguration of significant operating parameters
  • Installation or reconfiguration of significant non Oracle software

Starting with Oracle Exadata System Software release 19.1.0, the SSHD ClientAliveInterval defaults to 600 seconds. If the time needed for completing backups exceeds 10 minutes, then you can specify a larger value for ClientAliveInterval in the /etc/ssh/sshd_config file. You must restart the SSH service for changes to take effect. After the long running operation completes, remove the modification to the ClientAliveInterval parameter and restart the SSH service.

Creating a Snapshot-Based Backup of Exadata Database Servers X8M with Uncustomized Partitions

This procedure describes how to take a snapshot-based backup of an Oracle Exadata Database Machine X8M database server with uncustomized storage partitions.

Starting with Oracle Exadata Database Machine X8M and Oracle Exadata System Software release 19.3, the database servers use the following storage partitions:

File System Mount PointLogical Volume Name
/ (root)LVDbSys1 or LVDbSys2, whichever is active.
/u01LVDbOra1
/homeLVDbHome
/varLVDbVar1 or LVDbVar2, whichever is active.
/var/logLVDbVarLog
/var/log/auditLVDbVarLogAudit
/tmpLVDbTmp
  1. Prepare a destination to hold the backup.

The destination can be a large, writable NFS location. The NFS location should be large enough to hold the backup tar files. For uncustomized partitions, 145 GB is adequate.

  1. Create a mount point for the NFS share.

For example:

# mkdir /root/remote_FS

  1. Mount the NFS location.

In the following command, ip_address is the IP address of the NFS server, and nfs_location is the location of the NFS share.

# mount -t nfs -o rw,intr,soft,proto=tcp,nolock ip_address:/nfs_location/ /root/remote_FS

  1. Determine the active system volume.

You can use the imageinfo command and examine the device hosting the active system partition.

# imageinfo

Kernel version: 4.14.35-1902.5.1.4.el7uek.x86_64 #2 SMP Wed Oct 9 19:29:16 PDT 2019 x86_64

Image kernel version: 4.14.35-1902.5.1.4.el7uek

Image version: 19.3.1.0.0.191018

Image activated: 2019-11-04 19:18:32 -0800

Image status: success

Node type: KVMHOST

System partition on device: /dev/mapper/VGExaDb-LVDbSys1

In the imageinfo output, the system partition device ends with the name of the logical volume supports the active root (/) file system. Depending on the system image that is in use, the logical volume name is LVDbSys1 or LVDbSys2. Likewise, the logical volume for the /var file system is either LVDbVar1 or LVDbVar2.

You can also confirm the active devices by using the df -hT command and examining the output associated with the root (/) and /var file systems. For example:

# df -hT

Filesystem                          Type      Size  Used Avail Use% Mounted on

devtmpfs                            devtmpfs  378G     0  378G   0% /dev

tmpfs                               tmpfs     755G  1.0G  754G   1% /dev/shm

tmpfs                               tmpfs     378G  4.8M  378G   1% /run

tmpfs                               tmpfs     378G     0  378G   0% /sys/fs/cgroup

/dev/mapper/VGExaDb-LVDbSys1        xfs        15G  7.7G  7.4G  52% /

/dev/sda1                           xfs       510M  112M  398M  22% /boot

/dev/sda2                           vfat      254M  8.5M  246M   4% /boot/efi

/dev/mapper/VGExaDb-LVDbHome        xfs       4.0G   33M  4.0G   1% /home

/dev/mapper/VGExaDb-LVDbVar1        xfs       2.0G  139M  1.9G   7% /var

/dev/mapper/VGExaDb-LVDbVarLog      xfs        18G  403M   18G   3% /var/log

/dev/mapper/VGExaDb-LVDbVarLogAudit xfs      1014M  143M  872M  15% /var/log/audit

/dev/mapper/VGExaDb-LVDbTmp         xfs       3.0G  148M  2.9G   5% /tmp

/dev/mapper/VGExaDb-LVDbOra1        xfs       100G   32G   69G  32% /u01

tmpfs                               tmpfs      76G     0   76G   0% /run/user/0

The remaining examples in the procedure use LVDbSys1 and LVDbVar1, which is consistent with the above imageinfo and df output. However, if the active image is using LVDbSys2, then modify the examples in the following steps to use LVDbSys2 instead of LVDbSys1, and LVDbVar2 instead of LVDbVar1.

  1. Take snapshots of the logical volumes on the server.

Depending on the active system partition identified in the previous step, remember to use either LVDbSys1 or LVDbSys2 to identify the logical volume for the root (/) file system, and likewise use either LVDbVar1 or LVDbVar2 to identify the logical volume for the /var file system.

# lvcreate -L1G -s -c 32K -n root_snap /dev/VGExaDb/LVDbSys1

# lvcreate -L5G -s -c 32K -n u01_snap /dev/VGExaDb/LVDbOra1

# lvcreate -L1G -s -c 32K -n home_snap /dev/VGExaDb/LVDbHome

# lvcreate -L1G -s -c 32K -n var_snap /dev/VGExaDb/LVDbVar1

# lvcreate -L1G -s -c 32K -n varlog_snap /dev/VGExaDb/LVDbVarLog

# lvcreate -L1G -s -c 32K -n audit_snap /dev/VGExaDb/LVDbVarLogAudit

# lvcreate -L1G -s -c 32K -n tmp_snap /dev/VGExaDb/LVDbTmp

  1. Label the snapshots.

# xfs_admin -L DBSYS_SNAP /dev/VGExaDb/root_snap

# xfs_admin -L DBORA_SNAP /dev/VGExaDb/u01_snap

# xfs_admin -L HOME_SNAP /dev/VGExaDb/home_snap

# xfs_admin -L VAR_SNAP /dev/VGExaDb/var_snap

# xfs_admin -L VARLOG_SNAP /dev/VGExaDb/varlog_snap

# xfs_admin -L AUDIT_SNAP /dev/VGExaDb/audit_snap

# xfs_admin -L TMP_SNAP /dev/VGExaDb/tmp_snap

  1. Mount the snapshots.

Mount all of the snapshots under a common directory location; for example, /root/mnt.

# mkdir -p /root/mnt

# mount -t xfs -o nouuid /dev/VGExaDb/root_snap /root/mnt

# mkdir -p /root/mnt/u01

# mount -t xfs -o nouuid /dev/VGExaDb/u01_snap /root/mnt/u01

# mkdir -p /root/mnt/home

# mount -t xfs -o nouuid /dev/VGExaDb/home_snap /root/mnt/home

# mkdir -p /root/mnt/var

# mount -t xfs -o nouuid /dev/VGExaDb/var_snap /root/mnt/var

# mkdir -p /root/mnt/var/log

# mount -t xfs -o nouuid /dev/VGExaDb/varlog_snap /root/mnt/var/log

# mkdir -p /root/mnt/var/log/audit

# mount -t xfs -o nouuid /dev/VGExaDb/audit_snap /root/mnt/var/log/audit

# mkdir -p /root/mnt/tmp

# mount -t xfs -o nouuid /dev/VGExaDb/tmp_snap /root/mnt/tmp

  1. Back up the snapshots.

Use the following commands to write a backup file as a compressed tar file to your prepared NFS backup destination.

# cd /root/mnt

# tar -pjcvf /root/remote_FS/mybackup.tar.bz2 * /boot > /tmp/backup_tar.stdout 2> /tmp/backup_tar.stderr

  1. Check the /tmp/backup_tar.stderr file for any significant errors.

Errors about failing to tar open sockets, and other similar errors, can be ignored.

  1. Unmount and remove all of the snapshots.

# cd /

# umount /root/mnt/tmp

# umount /root/mnt/var/log/audit

# umount /root/mnt/var/log

# umount /root/mnt/var

# umount /root/mnt/home

# umount /root/mnt/u01

# umount /root/mnt

# lvremove /dev/VGExaDb/tmp_snap

# lvremove /dev/VGExaDb/audit_snap

# lvremove /dev/VGExaDb/varlog_snap

# lvremove /dev/VGExaDb/var_snap

# lvremove /dev/VGExaDb/home_snap

# lvremove /dev/VGExaDb/u01_snap

# lvremove /dev/VGExaDb/root_snap

  1. Unmount the NFS backup destination.

# umount /root/remote_FS

  1. Remove the mount point directories that you created during this procedure.

# rm -r /root/mnt

# rmdir /root/remote_FS

Creating a Snapshot-Based Backup of Oracle Linux Database Server with Customized Partitions

When you have customized the partitions, the procedure to back up is the same as the procedure used for non-customized database servers, with the following exceptions:

  • You must add any additional partitions similar to /u01 to the backup.
  • If any partitions were renamed, then use the names defined for your environment. For example, if /u01 was renamed to /myown_u01, then use /myown_u01 in the commands.

Recovering Oracle Linux Database Servers Using a Snapshot-Based Backup

You can recover a database server file systems running Oracle Linux using a snapshot-based backup after severe disaster conditions happen for the database server, or when the server hardware is replaced to such an extent that it amounts to new hardware.

For example, replacing all hard disks leaves no trace of original software on the system. This is similar to replacing the complete system as far as the software is concerned. In addition, it provides a method for disaster recovery of the database servers using an LVM snapshot-based backup taken when the database server was healthy before the disaster condition.

Overview of Snapshot-Based Recovery of Database Servers

The recovery process consists of a series of tasks.

The recovery procedures use the diagnostics.iso image as a virtual CD-ROM to restart the database server in rescue mode using the ILOM.

The general work flow includes the following tasks:

  1. Recreate the following:
    • Boot partitions
    • Physical volumes
    • Volume groups
    • Logical volumes
    • File system
    • Swap partition
  2. Activate the swap partition.
  3. Ensure the /boot partition is the active boot partition.
  4. Restore the data.
  5. Reconfigure GRUB.
  6. Restart the server.

If you use quorum disks, then after recovering the database servers from backup, you must manually reconfigure the quorum disk for the recovered server.

Recovering Oracle Linux Database Server with Uncustomized Partitions

You can recover the Oracle Linux database server from a snapshot-based backup when using uncustomized partitions.

This procedure is applicable when the layout of the partitions, logical volumes, file systems, and their sizes are equal to the layout when the database server was initially deployed.

  1. Prepare an NFS server to host the backup archive file (mybackup.tar.bz2).

The NFS server must be accessible by IP address.

For example, on an NFS server with the IP address nfs_ip, where the directory /export is exported as an NFS mount, put the backup file (mybackup.tar.bz2) in the /export directory.

  1. Restart the recovery target system using the diagnostics.iso file.
  2. Answer as indicated in these examples when prompted by the system. The responses are shown in bold.

Note that for Oracle Exadata System Software release 12.1.2.2.0 or later, DHCP is used and you do not have to manually set up the network.

  1. If you are using Oracle Exadata System Software release 18.1 or later, running on Oracle Exadata Database Machine X7 or later, then the prompt looks like the following:
  1. If you are using Oracle Exadata System Software release 18.1 or later and restoring through one of the 10GbE Ethernet SFP+ ports on Oracle Exadata Database Machine X3-2 or later, then the prompt looks like the following:

——————————————————————————

         Choose from the following by typing letter in ‘()’:

           (e)nter interactive diagnostics shell.

             Use diagnostics shell password to login as root user

             (reboot or power cycle to exit the shell),

           (r)estore system from NFS backup archive,

 Select: r

 Continue (y/n) [n]: y

 Rescue password:

 [INFO     ] Enter path to the backup file on the NFS server in format:

         Enter path to the backup file on the NFS server in format:

         <ip_address_of_the_NFS_share>:/<path>/<archive_file>

         For example, 10.124.1.15:/export/nfs/share/backup.2010.04.08.tar.bz2

 NFS line: <nfs_ip>:/export/mybackup.tar.bz2

 [INFO     ] The backup file could be created either from LVM or non-LVM

based COMPUTE node

 [INFO     ] Versions below 11.2.1.3.0 do not support LVM based partitioning

 Use LVM based scheme. (y/n) [y]: y

 Configure network settings on host via DHCP. (y/n) [y]: n

 Configure bonded network interface. (y/n) [y]: y

 IP Address of bondeth0 on this host: <IP address of the DB host>

Netmask of bondeth0 on this host: <netmask for the above IP address>

 Bonding mode:active-backup or 802.3ad [802.3ad]: active-backup

 Slave interface1 for bondeth0 (ethX) [eth4]: eth4

 Slave interface2 for bondeth0 (ethX) [eth5]: eth5

 [  354.619610] bondeth0: first active interface up!

 [  354.661427] ixgbe 0000:13:00.1 eth5: NIC Link is Up 10 Gbps, Flow Control: RX/TX

 [  354.724414] bondeth0: link status definitely up for interface eth5, 10000 Mbps full duplex

 Default gateway: <Gateway for the above IP address>

  1. If you are using Oracle Exadata System Software release 12.1.x or 12.2.x, then the prompts look like the following:

——————————————————————————

 Use diagnostics shell password to login as root user

            (reboot or power cycle to exit the shell),

          (r)estore system from NFS backup archive.

Select: r

Continue (y/n) [n]: y

Rescue password:

[INFO: ] Enter path to the backup file on the NFS server in format:

       Enter path to the backup file on the NFS server in format:

       <ip_address_of_the_NFS_share>:/<path>/<archive_file>

       For example, 10.124.1.15:/export/nfs/share/backup.2010.04.08.tar.bz2

NFS line: <nfs_ip>:/export/mybackup.tar.bz2

[INFO: ] The backup file could be created either from LVM or non-LVM based COMPUTE node

[INFO: ] Versions below 11.2.1.3.0 do not support LVM based partitioning

Use LVM based scheme. (y/n) [y]: y

  1. If you are using Oracle Exadata System Software release earlier than 12.1.2.2.0, then the prompts look like the following

——————————————————————————

      Choose from following by typing letter in ‘()’:

   (e)nter interactive diagnostics shell. Must use credentials from Oracle

      support to login (reboot or power cycle to exit the shell),

   (r)estore system from NFS backup archive,

Select:r

Are you sure (y/n) [n]:y

The backup file could be created either from LVM or non-LVM based compute node

versions below 11.2.1.3.1 and 11.2.2.1.0 or higher do not support LVM based partitioning

use LVM based scheme(y/n):y

Enter path to the backup file on the NFS server in format:

ip_address_of_the_NFS_share:/path/archive_file

For example, 10.10.10.10:/export/operating_system.tar.bz2

NFS line:<nfs_ip>:/export/mybackup.tar.bz2

IP Address of this host:IP address of the DB host

Netmask of this host:netmask for the above IP address

Default gateway:Gateway for the above IP address. If there is no default gateway in your network, enter 0.0.0.0.

When the recovery completes, the log in screen appears.

  1. Log in as the root user.

If you do not have the password for the root user, then contact Oracle Support Services.

  1. Restart the system.

# shutdown -r now

The restoration process is complete.

  1. Verify that all Oracle software can start and function by logging in to the database server.

The /usr/local/bin/imagehistory command indicates that the database server was reconstructed.

The following is an example of the output:

# imagehistory

Version                  : 11.2.2.1.0

Image activation date    : 2010-10-13 13:42:58 -0700

Imaging mode             : fresh

Imaging status           : success

Version                  : 11.2.2.1.0

Image activation date    : 2010-10-30 19:41:18 -0700

Imaging mode             : restore from nfs backup

Imaging status           : success

  1. Run reclaimdisks.sh on restored database server.

/opt/oracle.SupportTools/reclaimdisks.sh -free -reclaim

Re-Imaging the Oracle Exadata Database Server

The re-image procedure is necessary when a database server needs to be brought to an initial state for various reasons.

Some examples scenarios for re-imaging the database server are:

  • You want to install a new server and need to use an earlier release than is in the image already installed on the server.
  • You need to replace a damaged database server with a new database server.
  • Your database server had multiple disk failures causing local disk storage failure and you do not have a database server backup.
  • You want to repurpose the server to a new rack.

During the re-imaging procedure, the other database servers on Oracle Exadata Database Machine are available. When the new server is added to the cluster, the software is copied from an existing database server to the new server. It is your responsibility to restore scripting, CRON jobs, maintenance actions, and non-Oracle software.

Starting with Oracle Exadata System Software release 19.1.0, Secure Eraser is automatically started during re-imaging if the hardware supports Secure Eraser. This significantly simplifies the re-imaging procedure while maintaining performance. Now, when re-purposing a rack, you only have to image the rack and the secure data erasure is taken care of transparently as part of the process.

Contact Oracle Support Services

If a failed server is being replaced, open a support request with Oracle Support Services.

The support engineer will identify the failed server, and send a replacement. The support engineer will ask for the output from the imagehistory command run from a surviving database server. The output provides a link to the computeImageMaker file that was used to image the original database server, and provides a means to restore the system to the same level.

Download Latest Release of Cluster Verification Utility

The latest release of the cluster verification utility (cluvfy) is available from My Oracle Support.

Remove the Database Server from the Cluster

If you are reimaging a failed server or repurposing a server, follow the steps in this task to remove the server from the cluster before you reimage it. If you are reimaging the server for a different reason, skip this task and proceed with the reimaging task next.

The steps in this task are performed using a working database server in the cluster. In the following commands, working_server is a working database server, and failed_server is the database server you are removing, either because it failed or it is being repurposed.

  1. Log in as the oracle or grid user on a database server in the cluster.

Log in as the user that owns the Oracle Grid Infrastructure software installation.

  1. Disable the listener that runs on the failed server.

$ srvctl disable listener -n failed_server

$ srvctl stop listener -n failed_server

  1. Delete the Oracle home from the Oracle inventory.

In the following command, list_of_working_servers is a list of the servers that are still working in the cluster, such as dm01db02, dm01db03, and so on.

In the following command, replace /u01/app/oracle/product/12.1.0.2/dbhome_1 with the location of your Oracle Database home directory.

$ cd $ORACLE_HOME/oui/bin

$ ./runInstaller -updateNodeList ORACLE_HOME= \

/u01/app/oracle/product/12.1.0.2/dbhome_1 “CLUSTER_NODES=list_of_working_servers

  1. Log in as the grid user on the database server.

The grid user refers to the operating system user that owns the Oracle Grid Infrastructure software installation. The $ORACLE_HOME variable should point to the location of the Grid home.

  1. Verify the failed server is unpinned.

$ olsnodes -s -t

The following is an example of the output from the command:

dm01adm05        Inactive        Unpinned

dm01adm06        Active          Unpinned

dm01adm07        Active          Unpinned

dm01adm08        Active          Unpinned

  1. Log in as the root user on the database server.
  2. Stop and delete the VIP resources for the failed database server.

# srvctl stop vip -i failed_server-vip

PRCC-1016 : failed_server-vip.example.com was already stopped

# srvctl remove vip -i failed_server-vip

Please confirm that you intend to remove the VIPs failed_server-vip (y/[n]) y

  1. Delete the server from the cluster.

# crsctl delete node -n failed_server

CRS-4661: Node dm01db01 successfully deleted.

If you receive an error message similar to the following, then relocate the voting disks.

CRS-4662: Error while trying to delete node dm01db01.

CRS-4000: Command Delete failed, or completed with errors.

To relocate the voting disks use the following steps:

  1. Determine the current location of the voting disks.

# crsctl query css votedisk

The following is an example of the output from the command. The current location is DBFS_DG.

##  STATE    File Universal Id          File Name                Disk group

—  —–    —————–          ———                ———-

1. ONLINE   123456789abab (o/192.168.73.102/DATA_CD_00_dm01cel07) [DBFS_DG]

2. ONLINE   123456789cdcd (o/192.168.73.103/DATA_CD_00_dm01cel08) [DBFS_DG]

3. ONLINE   123456789efef (o/192.168.73.100/DATA_CD_00_dm01cel05) [DBFS_DG]

Located 3 voting disk(s).

  1. Relocate the voting disks to another disk group.

# ./crsctl replace votedisk +DATA

Successful addition of voting disk 2345667aabbdd.

CRS-4266: Voting file(s) successfully replaced

  1. Relocate the voting disks to the original location using a command similar to the following:

# ./crsctl replace votedisk +DBFS_DG

  1. Delete the server from the cluster.
  2. Log in as the grid user on the database server.

The grid user refers to the operating system user that owns the Oracle Grid Infrastructure software installation. The $ORACLE_HOME variable should point to the location of the Grid home.

  1. Update the Oracle inventory.

In the following command, replace /u01/app/12.1.0.2/grid with the location of your Oracle Grid Infrastructure home directory.

$ cd $ORACLE_HOME/oui/bin

$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/12.1.0.2/grid \

  “CLUSTER_NODES=list_of_working_servers” CRS=TRUE

  1. Verify the server was deleted successfully.

$ cluvfy stage -post nodedel -n failed_server -verbose

The following is an example of the output from the command:

Performing post-checks for node removal

Checking CRS integrity…

The Oracle clusterware is healthy on node “dm01db02”

CRS integrity check passed

Result:

Node removal check passed

Post-check for node removal was successful.

Image the Database Server

After the database server has been installed or replaced, you can image the new database server.

You can use installation media on a USB thumb drive, or a touchless option using PXE or ISO attached to the ILOM.

Configure the Re-imaged Database Server

The re-imaged database server does not have any host names, IP addresses, DNS or NTP settings. The steps in this task describe how to configure the re-imaged database server.

You need the following information prior to configuring the re-imaged database server:

  • Name servers
  • Time zone, such as Americas/Chicago
  • NTP servers
  • IP address information for the management network
  • IP address information for the client access network
  • IP address information for the RDMA Network Fabric
  • Canonical host name
  • Default gateway

The information should be the same on all database servers in Oracle Exadata Database Machine. The IP addresses can be obtained from DNS. In addition, a document with the information should have been provided when Oracle Exadata Database Machine was installed.

The following procedure describes how to configure the re-imaged database server:

  1. Power on the replacement database server. When the system boots, it automatically runs the Configure Oracle Exadata routine, and prompts for information.
  2. Enter the information when prompted, and confirm the settings. The start up process will continue.

Prepare the Re-imaged Database Server for the Cluster

This task describes how to ensure the changes made during initial installation are done to the re-imaged, bare metal database server.

  1. Copy or merge the contents of the following files using files on a working database server as reference:
    1. Copy the contents of the /etc/security/limits.conf file.
    2. Merge the contents of the /etc/hosts files.
    3. Copy the /etc/oracle/cell/network-config/cellinit.ora file.
    4. Update the /etc/oracle/cell/network-config/cellinit.ora file with the IP_ADDRESS of the ifcfg-bondib0 interface (in case of active/passive bonding) or ib0 and ib1 interfaces (in case of active/active bonding) of the replacement server.
    5. Copy the /etc/oracle/cell/network-config/cellip.ora file.

The content of the cellip.ora file should be the same on all database servers.

  1. Configure additional network requirements, such as 10 GbE.
  2. Copy the modprobe configuration.

The contents of the configuration file should be the same on all database servers.

  1. Oracle Linux 5 or 6: The file is located at /etc/modprobe.conf.
  2. Oracle Linux 7: The file is located at /etc/modprobe.d/exadata.conf.
  3. Copy the /etc/sysctl.conf file.

The contents of the file should be the same on all database servers.

  1. Update the cellroute.ora.

Make a copy of the /etc/oracle/cell/network-config/cellroute.ora file. Modify the contents on the replacement server to use the local InfiniBand interfaces on the new node.

  1. Restart the database server so the network changes take effect.
  2. Set up the users for the software owners on the replacement database server by adding groups.

If you are using role-separated management, then the users are usually oracle and grid. If you use a single software owner, then the user is usually oracle. The group information is available on a working database server.

  1. Obtain the current group information from a working database server.

# id oracle

uid=1000(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(dba),1003(oper),1004(asmdba)

  1. Use the groupadd command to add the group information to the replacement database server.

# groupadd -g 1001 oinstall

# groupadd -g 1002 dba

# groupadd -g 1003 oper

# groupadd -g 1004 asmdba

  1. Obtain the current user information from a working database server.

# id oracle uid=1000(oracle) gid=1001(oinstall) \

  groups=1001(oinstall),1002(dba),1003(oper),1004(asmdba)

  1. Add the user information to the replacement database server.

# useradd -u 1000 -g 1001 -G 1001,1002,1003,1004 -m -d /home/oracle -s \

  /bin/bash oracle

  1. Create the Oracle Base and Grid home directories, such as /u01/app/oracle and /u01/app/12.2.0.1/grid.

# mkdir -p /u01/app/oracle

# mkdir -p /u01/app/12.2.0.1/grid

# chown -R oracle:oinstall /u01/app

  1. Change the ownership on the cellip.ora and cellinit.ora files.

The ownership is usually oracle:oinstall.

# chown -R oracle:oinstall /etc/oracle/cell/network-config

  1. Secure the restored database server.

# chmod u+x /opt/oracle.SupportTools/harden_passwords_reset_root_ssh

# /opt/oracle.SupportTools/harden_passwords_reset_root_ssh

The database server restarts. Log in as the root user when prompted by the system. You are prompted for a new password. Set the password to match the root password of the other database servers.

  1. Set the password for the Oracle software owner.

The owner is usually oracle.

# passwd oracle

  1. Set up SSH for the oracle account.
    1. Log in to the oracle account on the replacement database server.

# su – oracle

  1. Create the dcli group file on the replacement database server listing the servers in the Oracle cluster.
  2. Run the following command on the replacement database server.

$ dcli -g dbs_group -l oracle -k

  1. Exit and log in again as the oracle user.

$ exit

# su – oracle

  1. Verify SSH equivalency.

$ dcli -g dbs_group -l oracle date

  1. Set up or copy any custom login scripts from a working database server to the replacement database server.

In the following command, replacement_server is the name of the new server, such as dm01db01.

$ scp .bash* [email protected]replacement_server:.

Managing Quorum Disks for High Redundancy Disk Groups

Quorum disks allow for the Oracle RAC voting files to be stored in a high redundancy disk group on an Oracle Exadata Rack with fewer than five storage servers due to the presence of two extra failure groups.

Using Quorum Disks to Increase Fault Tolerance

Quorum disks are used to meet the minimum requirement of five failure groups for a high redundancy disk group on a system that does not have five storage servers.

failure group is a subset of the disks in a disk group, which could fail at the same time because they share hardware. Oracle recommends a minimum of three failure groups for normal redundancy disk groups and five failure groups for high redundancy disk groups to maintain the necessary number of copies of the Partner Status Table (PST) and to ensure robustness with respect to storage hardware failures. On Engineered Systems, these recommendations are enforced to ensure the highest availability of the system.

The PST contains status information about the Oracle Automatic Storage Management (Oracle ASM) disks in a disk group, such as the disk number, status (either online or offline), partner disk number, failure group info, and heartbeat info. To tolerate a single hardware failure, you need 3 total copies of the PST available to form a 2 of 3 majority. If there are two hardware failures, then you need a total for 5 copies of the PST so that after a double failure you still have a 3 of 5 majority.

quorum failure group is a special type of failure group that does not contain user data. Quorum failure groups are used for storing the PST. A quorum failure group can also be used to store a copy of the voting file for Oracle Clusterware. Because disks in quorum failure groups do not contain user data, a quorum failure group is not considered when determining redundancy requirements in respect to storing user data.

In the event of a system failure, three failure groups in a normal redundancy disk group allow a comparison among three PSTs to accurately determine the most up to date and correct version of the PST, which could not be done with a comparison between only two PSTs. Similarly with a high redundancy disk group, if two failure groups are offline, then Oracle ASM would be able to make a comparison among the three remaining PSTs.

You can create quorum failure groups with Oracle Exadata Deployment Assistant (OEDA) when deploying Exadata or you can add them later using the Quorum Disk Manager Utility. The iSCSI quorum disks are created on database nodes and a voting file is created on those quorum disks. These additional quorum failure groups are used to meet the minimum requirement of five voting files and PSTs for a high redundancy disk group. Quorum disks are required when the following conditions exist:

  • The Oracle Exadata Rack has fewer than five storage servers.
  • The Oracle Exadata Rack has at least two database nodes.
  • The Oracle Exadata Rack has at least one high redundancy disk group.

Quorum failure groups allow for a high redundancy disk group to exist on Oracle Exadata Racks with fewer than five storage servers by creating two extra failure groups. Without this feature, a disk group is vulnerable to a double partner storage server failure that results in the loss of the PST or voting file quorum, which can cause a complete cluster and database outage. Refer to My Oracle Support note 1339373.1 for how to restart the clusterware and databases in this scenario.

The iSCSI quorum disk implementation has high availability because the IP addresses on the RDMA Network Fabric are highly available using RDS.

Each iSCSI device shown in the figure below corresponds to a particular path to the iSCSI target. Each path corresponds to an RDMA Network Fabric port on the database node. For each multipath quorum disk device in an active–active system, there are two iSCSI devices, one for ib0 or re0 and the other for ib1 or re1.

Multipath Device Connects to Both iSCSI Devices in an Active-Active System

Quorum disks can be used with bare metal Oracle Real Application Clusters (Oracle RAC) clusters and Oracle VM Oracle RAC clusters. For Oracle VM Oracle RAC clusters, the quorum disk devices reside in the Oracle RAC cluster nodes which are Oracle VM user domains as shown in the following figure.

Quorum Disk Devices on Oracle VM Oracle RAC Cluster

For pkey-enabled environments, the interfaces used for discovering the targets should be the pkey interfaces used for the Oracle Clusterware communication. These interfaces are listed using the following command:

Grid_home/bin/oifcfg getif | grep cluster_interconnect | awk ‘{print $1}’

Overview of Quorum Disk Manager

The Quorum Disk Manager utility, introduced in Oracle Exadata System Software release 12.1.2.3.0, helps you to manage the quorum disks.

This utility enables you to create an iSCSI quorum disk on two of the database nodes and store a voting file on those two quorum disks. These two additional voting files are used to meet the minimum requirement of five voting files for a high redundancy disk group.

The Quorum Disk Manager utility (quorumdiskmgr) is used to create and manage all the necessary components including the iSCSI configuration, the iSCSI targets, the iSCSI LUNs, and the iSCSI devices for implementing quorum disks.

Syntax for the Quorum Disk Manager Utility

The quorum disk manager utility is a command-line tool. It has the following syntax:

quorumdiskmgr —verbobject [–options]

verb is an action performed on an object. It is one of: alter, create, delete, list.

object is an object on which the command performs an action.

options extend the use of a command combination to include additional parameters for the command.

When using the quorumdiskmgr utility, the following rules apply:

  • Verbs, objects, and options are case-sensitive except where explicitly stated.
  • Use the double quote character around the value of an option that includes spaces or punctuation.

quorumdiskmgr Objects

ObjectDescription
configThe quorum disk configurations include the owner and group of the ASM instance to which the iSCSI quorum disks will be added, and the list of network interfaces through which local and remote iSCSI quorum disks will be discovered.
targetA target is an endpoint on each database server that waits for an iSCSI initiator to establish a session and provides required IO data transfer.
deviceA device is an iSCSI device created by logging into a local or remote target.

Creating a Quorum Disk Configuration (–create –config)

The –create –config action creates a quorum disk configuration.

The configuration must be created before any targets or devices can be created.

quorumdiskmgr –create –config [–owner owner –group group]

  –network-iface-list network-iface-list

Parameters

The following table lists the parameters for the –create –config action:

ParameterDescription
–ownerSpecifies the owner of the Oracle ASM instance to which the iSCSI quorum disks will be added. This is an optional parameter. The default value is grid.
–groupSpecifies the group of the Oracle ASM instance to which the iSCSI quorum disks will be added. This is an optional parameter. The default value is dba.
–network-iface-listSpecifies the list of RDMA Network Fabric interface names through which the local and remote targets will be discovered.

Create a Quorum Disk Configuration for a System with InfiniBand Network Fabric

quorumdiskmgr –create –config –owner=oracle –group=dba –network-iface-list=”ib0, ib1″

Create a Quorum Disk Configuration for a System with RoCE Network Fabric

quorumdiskmgr –create –config –owner=oracle –group=dba –network-iface-list=”re0, re1″

Creating a Target (–create –target)

The –create –target action creates a target that will be used to create the devices to add to the specified Oracle ASM disk group.

The –create –target action creates a target that can be accessed by database servers with an RDMA Network Fabric IP address in the specified IP address list.

After a target is created, the asm-disk-group, host-name, and size attributes cannot be changed.

quorumdiskmgr –create –target –asm-disk-group asm_disk_group –visible-to ip_list

   [–host-name host_name] [–size size]

Parameters

ParameterDescription
–asm-disk-groupSpecifies the Oracle ASM disk group to which the device created from the target will be added. The value of asm-disk-group is not case-sensitive.
–visible-toSpecifies a list of RDMA Network Fabric IP addresses. Database servers with an RDMA Network Fabric IP address in the list will have access to the target.
–host-nameSpecifies the host name of the database server on which quorumdiskmgr runs. The total length of the values for asm-disk-group and host-name cannot exceed 26 characters. If the host name is too long, a shorter host name can be specified as long as a different host name is specified for each database server in the rack. This is an optional parameter. The default value is the host name of the database server on which quorumdiskmgr runs. The value of host-name is not case-sensitive.
–sizeSpecifies the size of the target. This is an optional parameter. The default value is 128 MB.

Creating a Target For Oracle ASM Disk Group Devices

This example shows how to create a target for devices added to the DATAC1 disk group. That target is only visible to database servers that have an RDMA Network Fabric IP address of 192.168.10.45 or 192.168.10.46.

quorumdiskmgr –create –target –asm-disk-group=datac1 –visible-to=”192.168.10.45, 192.168.10.46″

 –host-name=db01

Creating a Device (–create –device)

The –create –device action creates devices by discovering and logging into targets on database servers with an RDMA Network Fabric IP address in the specified list of IP addresses.

The created devices will be automatically discovered by the Oracle ASM instance with the owner and group specified during configuration creation.

quorumdiskmgr –create –device –target-ip-list target_ip_list

Parameters

  • –target-ip-list: Specifies a list of RDMA Network Fabric IP addresses.

quorumdiskmgr discovers targets on database servers that have an IP address in the list, then logs in to those targets to create devices.

Creating Devices From a Target For an Oracle ASM Disk Group

This example shows how to create devices using targets on database servers that have an IP address of 192.168.10.45 or 192.168.10.46.

quorumdiskmgr –create –device –target-ip-list=”192.168.10.45, 192.168.10.46″

Listing Quorum Disk Configurations (–list –config)

The –list –config action lists the quorum disk configurations.

quorumdiskmgr –list –config

Sample Output

Listing the quorum disk configuration on rack with InfiniBand Network Fabric

$ quorumdiskmgr –list –config

Owner: grid

Group: dba

ifaces: exadata_ib1 exadata_ib0

Listing the quorum disk configuration on rack with RoCE Network Fabric

$ quorumdiskmgr –list –config

Owner: grid

Group: dba

ifaces: exadata_re1 exadata_re0

Listing Targets (–list –target)

The –list –target action lists the attributes of targets.

The target attributes listed include target name, size, host name, Oracle ASM disk group name, the list of IP addresses (a visible-to IP address list) indicating which database servers have access to the target, and the list of IP addresses (a discovered-by IP address list) indicating which database servers have logged into the target.

If an Oracle ASM disk group name is specified, the action lists all local targets created for the specified Oracle ASM disk group. Otherwise, the action lists all local targets created for quorum disks.

quorumdiskmgr –list –target [–asm-disk-group asm_disk_group]

Parameters

  • –asm-disk-group: Specifies the Oracle ASM disk group. quorumdiskmgr displays all local targets for this Oracle ASM disk group. The value of asm-disk-group is not case-sensitive.

Listing the Target Attributes for a Specific Oracle ASM Disk Group

This example shows how to list the attributes of the target for the DATAC1 disk group.

quorumdiskmgr –list –target –asm-disk-group=datac1

Name: iqn.2015-05.com.oracle:qd–datac1_db01

Size: 128 MB

Host name: DB01

ASM disk group name: DATAC1

Visible to: iqn.1988-12.com.oracle:192.168.10.23, iqn.1988-12.com.oracle:192.168.10.24,

 iqn.1988-12.com.oracle:1b48248af770, iqn.1988-12.com.oracle:7a4a399566

Discovered by: 192.168.10.47, 192.168.10.46

Listing Devices (–list –device)

The –list –device action lists the attributes of devices, including device path, size, host name and ASM disk group name.

  • If only the Oracle ASM disk group name is specified, then the output includes all the devices that have been added to the Oracle ASM disk group.
  • If only the host name is specified, then the output includes all the devices created from the targets on the host.
  • If both an Oracle ASM disk group name and a host name are specified, then the output includes a single device created from the target on the host that has been added to the Oracle ASM disk group.
  • If neither an Oracle ASM disk group name or a host name is specified, then the output includes all quorum disk devices.

quorumdiskmgr –list –device [–asm-disk-group asm_disk_group] [–host-name host_name]

Parameters

ParameterDescription
–asm-disk-groupSpecifies the Oracle ASM disk group to which devices have been added. The value of asm-disk-group is not case-sensitive.
–host-nameSpecifies the host name of the database server from whose targets devices are created. The value of host-name is not case-sensitive.

Listing Device Attributes for an Oracle ASM Disk Group

This example shows how to list the attributes for devices used by the DATAC1 disk group.

$ quorumdiskmgr –list –device –asm-disk-group datac1

Device path: /dev/exadata_quorum/QD_DATAC1_DB01

Size: 128 MB

Host name: DB01

ASM disk group name: DATAC1

Device path: /dev/exadata_quorum/QD_DATAC1_DB02

Size: 128 MB

Host name: DB02

ASM disk group name: DATAC1

Deleting Configurations (–delete –config)

The –delete –config action deletes quorum disk configurations.

The configurations can only be deleted when there are no targets or devices present.

quorumdiskmgr –delete –config

Deleting Targets (–delete –target)

The –delete –target action deletes the targets created for quorum disks on database servers.

If an Oracle ASM disk group name is specified, then this command deletes all the local targets created for the specified Oracle ASM disk group. Otherwise, this command deletes all local targets created for quorum disks.

quorumdiskmgr –delete –target [–asm-disk-group asm_disk_group]

Parameters

  • –asm-disk-group: Specifies the Oracle ASM disk group. Local targets created for this disk group will be deleted.

The value of asm-disk-group is not case-sensitive.

Deleting Targets Created for an Oracle ASM Disk Group

This example shows how to delete targets created for the DATAC1 disk group.

quorumdiskmgr –delete –target –asm-disk-group=datac1

Deleting Devices (–delete –device)

The –delete –device command deletes quorum disk devices.

  • If only an Oracle ASM disk group name is specified, then the command deletes all the devices that have been added to the Oracle ASM disk group.
  • If only a host name is specified, then the command deletes all the devices created from the targets on the host.
  • If both an Oracle ASM disk group name and a host name are specified, then the command deletes a single device created from the target on the host and that has been added to the Oracle ASM disk group.
  • If neither an Oracle ASM disk group name nor a host name is specified, then the command deletes all quorum disk devices.

quorumdiskmgr –delete –device [–asm-disk-group asm_disk_group] [–host-name host_name]

Parameters

ParameterDescription
–asm-disk-groupSpecifies the Oracle ASM disk group whose device you want to delete. The value of asm-disk-group is not case-sensitive.
–host-nameSpecifies the host name of the database server. Devices created from targets on this host will be deleted. The value of host-name is not case-sensitive.

Deleting Quorum Disk Devices Created from Targets on a Specific Host

This example shows how to delete all the quorum disk devices that were created from the targets on the host DB01.

quorumdiskmgr –delete –device –host-name=db01

Changing Owner and Group Values (–alter –config)

The –alter –config action changes the owner and group configurations.

quorumdiskmgr –alter –config –owner owner –group group

Parameters

ParameterDescription
–ownerSpecifies the new owner for the quorum disk configuration. This parameter is optional. If not specified, the owner is unchanged.
–groupSpecifies the new group for the quorum disk configuration. This parameter is optional. If not specified, the group is unchanged.

Changes the Owner and Group Configuration for Quorum Disk Devices

This example shows how to change the assigned owner and group for quorum disk devices.

quorumdiskmgr –alter –config –owner=grid –group=dba

Changing the RDMA Network Fabric IP Addresses (–alter –target)

The –alter –target command changes the RDMA Network Fabric IP addresses of the database servers that have access to the local target created for the specified Oracle ASM disk group.

quorumdiskmgr –alter –target –asm-disk-group asm_disk_group –visible-to ip_list

Parameters

ParameterDescription
–asm-disk-groupSpecifies the Oracle ASM disk group to which the device created from the target will be added. The value of asm-disk-group is not case-sensitive.
–visible-toSpecifies a list of RDMA Network Fabric IP addresses. Database servers with an RDMA Network Fabric IP address in the list will have access to the target.

Changing the RDMA Network Fabric IP Addresses for Accessing Targets

This example shows how to change the RDMA Network Fabric IP address list that determines which database servers have access to the local target created for DATAC1 disk group

quorumdiskmgr –alter –target –asm-disk-group=datac1 –visible-to=”192.168.10.45, 192.168.10.47

Add Quorum Disks to Database Nodes

You can add quorum disks to database nodes on an Oracle Exadata Rack with fewer than 5 storage servers that contains a high redundancy disk group.

The example in this section creates quorum disks for an Oracle Exadata Rack that has two database nodes: db01 and db02.

This is an active-active system. On both db01 and db02 there are two RDMA Network Fabric ports:

  • InfiniBand Network Fabric: ib0 and ib1
  • RoCE Network Fabric: re0 and re1

The network interfaces to be used for communication with the iSCSI devices can be found using the following command:

$ oifcfg getif | grep cluster_interconnect | awk ‘{print $1}’

The IP address of each interface can be found using the following command:

ip addr show interface_name

The RDMA Network Fabric IP addresses for this example are as follows:

On db01:

  • Network interface: ib0 or re0, IP address: 192.168.10.45
  • Network interface: ib1 or re1, IP address: 192.168.10.46

On db02:

  • Network interface: ib0 or re0, IP address: 192.168.10.47
  • Network interface: ib1 or re1, IP address: 192.168.10.48

The Oracle ASM disk group to which the quorum disks will be added is DATAC1. The Oracle ASM owner is grid, and the user group is dba.

Initially, the voting files reside on a normal redundancy disk group RECOC1:

$ Grid_home/bin/crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

—  —–    —————–                ——— ———

 1. ONLINE   21f5507a28934f77bf3b7ecf88b26c47 (o/192.168.76.187;192.168.76.188/RECOC1_CD_00_celadm12) [RECOC1]

 2. ONLINE   387f71ee81f14f38bfbdf0693451e328 (o/192.168.76.189;192.168.76.190/RECOC1_CD_00_celadm13) [RECOC1]

 3. ONLINE   6f7fab62e6054fb8bf167108cdbd2f64 (o/192.168.76.191;192.168.76.192/RECOC1_CD_00_celadm14) [RECOC1]

Located 3 voting disk(s).

  1. Log into db01 and db02 as the root user.
  2. Run the quorumdiskmgr command with the –create –config options to create quorum disk configurations on both db01 and db02.
    • For InfiniBand Network Fabric:

# /opt/oracle.SupportTools/quorumdiskmgr –create –config –owner=grid –group=dba –network-iface-list=”ib0, ib1″

  1. For RoCE Network Fabric:

# /opt/oracle.SupportTools/quorumdiskmgr –create –config –owner=grid –group=dba –network-iface-list=”re0, re1″

  1. Run the quorumdiskmgr command with the –list –config options to verify that the configurations have been successfully created on both db01 and db02.

# /opt/oracle.SupportTools/quorumdiskmgr –list –config

Your output should resemble one of the following:

  1. For Oracle Exadata System Software release 18.x or earlier, the output should look like this:

Owner: grid

Group: dba

ifaces: exadata_ib1 exadata_ib0

  1. If you have upgraded to Oracle Exadata System Software release 19.1.0 or later from an earlier release, then the output should look like this:

Owner: grid

Group: dba

ifaces: exadata_ib0

Initiatior name: iqn.1988-12.com.oracle:7a4a399566

  1. If you have a system that was imaged with Oracle Exadata System Software release 19.1.0 or later (not upgraded), then the output should look like this:

Owner: grid

Group: dba

ifaces: exadata_ib0

Initiatior name: iqn.1988-12.com.oracle:192.168.18.205

  1. If your rack uses RoCE Network Fabric, then the output should look like this:

Owner: grid

Group: dba

ifaces: exadata_re0

Initiatior name: iqn.1988-12.com.oracle:192.168.18.205

  1. Run the quorumdiskmgr command with the –create –target options to create a target on both db01 and db02 for Oracle ASM disk group DATAC1 and make the target visible to both db01 and db02.

For example, you would use a command similar to this:

# /opt/oracle.SupportTools/quorumdiskmgr –create –target –asm-disk-group=datac1

–visible-to=”192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48″

  1. Run the quorumdiskmgr command with the –list –target options to verify the target has been successfully created on both db01 and db02.

# /opt/oracle.SupportTools/quorumdiskmgr –list –target

The output shows both IP addresses and initiator names in the Visible to list only if you have a system that was upgraded from a release older than Oracle Exadata System Software release 19.1.0. Otherwise, the Visible to list shows only IP addresses in it.

  1. If you are running Oracle Exadata System Software release 18.x or earlier, then the output should look like this for each node:

Name: iqn.2015-05.com.oracle:QD_DATAC1_DB01

Size: 128 MB

Host name: DB01

ASM disk group name: DATAC1

Visible to: 192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48

Discovered by:

Name: iqn.2015-05.com.oracle:QD_DATAC1_DB02

Size: 128 MB

Host name: DB02

ASM disk group name: DATAC1

Visible to: 192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48

Discovered by:

  1. If you are running Oracle Exadata System Software release 19.x or later, then the output should look like this for each node:

Name: iqn.2015-05.com.oracle:QD_DATAC1_DB01

Size: 128 MB

Host name: DB01

ASM disk group name: DATAC1

Visible  to: 192.168.10.45, 192.168.10.46, 192.168.10.47,

192.168.10.48, iqn.1988-12.com.oracle:ee657eb81b53,

iqn.1988-12.com.oracle:db357ba82b24

Name: iqn.2015-05.com.oracle:QD_DATAC1_DB02

Size: 128 MB

Host name: DB02

ASM disk group name: DATAC1

Visible to: 192.168.10.45, 192.168.10.46, 192.168.10.47,

192.168.10.48, iqn.1988-12.com.oracle:ee657eb81b53,

iqn.1988-12.com.oracle:db357ba82b24

  1. Run the quorumdiskmgr command with the –create –device options to create devices on both db01 and db02 from targets on both db01 and db02.

For example, you would use a command similar to this:

# /opt/oracle.SupportTools/quorumdiskmgr –create –device –target-ip-list=”192.168.10.45, 192.168.10.46,

 192.168.10.47, 192.168.10.48″

  1. Run the quorumdiskmgr command with the –list –device options to verify the devices have been successfully created on both db01 and db02.

# /opt/oracle.SupportTools/quorumdiskmgr –list –device

On both db01 and db02, the output should look like:

Device path: /dev/exadata_quorum/QD_DATAC1_DB01

Size: 128 MB

Host name: DB01

ASM disk group name: DATAC1

Device path: /dev/exadata_quorum/QD_DATAC1_DB02

Size: 128 MB

Host name: DB02

ASM disk group name: DATAC1

  1. Switch to the grid user on either db01 or db02.
  2. Set up Oracle ASM environments.
  3. Alter the asm_diskstring initialization parameter and add /dev/exadata_quorum/* to the existing string

SQL> ALTER SYSTEM SET asm_diskstring=’o/*/DATAC1_*’,’o/*/RECOC1_*’,’/dev/exadata_quorum/*’ scope=both sid=’*’;

  1. Verify the two quorum disk devices have been automatically discovered by Oracle ASM.

SQL> set linesize 200

SQL> col path format a50

SQL> SELECT inst_id, label, path, mode_status, header_status

FROM gv$asm_disk WHERE path LIKE ‘/dev/exadata_quorum/%’;

The output should look like:

INST_ID LABEL          PATH                                MODE_STATUS HEADER_STATUS

——- ————– ———————————-  ———– ———

      1 QD_DATAC1_DB01 /dev/exadata_quorum/QD_DATAC1_DB01  ONLINE      CANDIDATE

      1 QD_DATAC1_DB02 /dev/exadata_quorum/QD_DATAC1_DB02  ONLINE      CANDIDATE

      2 QD_DATAC1_DB01 /dev/exadata_quorum/QD_DATAC1_DB01  ONLINE      CANDIDATE

      2 QD_DATAC1_DB02 /dev/exadata_quorum/QD_DATAC1_DB02  ONLINE      CANDIDATE

  1. Add the two quorum disk devices to a high redundancy Oracle ASM disk group.

If there is no high redundancy disk group, create a high redundancy disk group and include the two new quorum disks. For example:

SQL> CREATE DISKGROUP DATAC1 HIGH REDUNDANCY ADD QUORUM FAILGROUP db01 DISK ‘/dev/exadata_quorum/QD_ DATAC1_DB01’

QUORUM FAILGROUP db02 DISK ‘/dev/exadata_quorum/QD_ DATAC1_DB02’ …

If a high redundancy disk group already exists, add the two new quorum disks. For example:

SQL> ALTER DISKGROUP datac1 ADD QUORUM FAILGROUP db01 DISK ‘/dev/exadata_quorum/QD_DATAC1_DB01’

QUORUM FAILGROUP db02 DISK ‘/dev/exadata_quorum/QD_DATAC1_DB02’;

  1. Relocate the existing voting files from the normal redundancy disk group to the high redundancy disk group.

$ Grid_home/bin/crsctl replace votedisk +DATAC1

  1. Verify the voting disks have been successfully relocated to the high redundancy disk group and that five voting files exist.

crsctl query css votedisk

The output should show 3 voting disks from storage servers and 2 voting disks from database nodes:

## STATE File Universal Id File Name Disk group

— —– —————– ——— ———

1. ONLINE ca2f1b57873f4ff4bf1dfb78824f2912 (o/192.168.10.42/DATAC1_CD_09_celadm12) [DATAC1]

2. ONLINE a8c3609a3dd44f53bf17c89429c6ebe6 (o/192.168.10.43/DATAC1_CD_09_celadm13) [DATAC1]

3. ONLINE cafb7e95a5be4f00bf10bc094469cad9 (o/192.168.10.44/DATAC1_CD_09_celadm14) [DATAC1]

4. ONLINE 4dca8fb7bd594f6ebf8321ac23e53434 (/dev/exadata_quorum/QD_ DATAC1_DB01) [DATAC1]

5. ONLINE 4948b73db0514f47bf94ee53b98fdb51 (/dev/exadata_quorum/QD_ DATAC1_DB02) [DATAC1]

Located 5 voting disk(s).

  1. Move the Oracle ASM password file to the high redundancy disk group.
  2. Get the source Oracle ASM password file location.

$ asmcmd pwget –asm

  1. Move the Oracle ASM password file to the high redundancy disk group.

$ asmcmd pwmove –asm full_path_of_source_file full_path_of_destination_file

Example:

asmcmd pwmove –asm +recoc1/ASM/PASSWORD/pwdasm.256.898960531 +datac1/asmpwdfile

  1. Move the Oracle ASM SPFILE to the high redundancy disk group.

These commands should be run from any one Oracle VM cluster node for Oracle Real Application Clusters (Oracle RAC).

  1. Get the Oracle ASM SPFILE in use.

$ asmcmd spget

  1. Copy the Oracle ASM SPFILE to the high redundancy disk group.

$ asmcmd spcopy full_path_of_source_file full_path_of_destination_file

  1. Modify the Oracle Grid Infrastructure configuration to use the relocated SPFILE upon next restart.

$ asmcmd spset full_path_of_destination_file

  1. At this point if a downtime can be afforded, restart Oracle Grid Infrastructure.

# Grid_home/bin/crsctl stop crs

# Grid_home/bin/crsctl start crs

If a downtime is not permitted, repeat step 16 every time an initialization parameter modification to the Oracle ASM SPFILE is required until Oracle Grid Infrastructure is restarted.

  1. Relocate the MGMTDB to the high redundancy disk group.

Configure the MGMTDB to not use hugepages using the steps below:

export ORACLE_SID=-MGMTDB

export ORACLE_HOME=$GRID_HOME

sqlplus ”sys as sysdba”

SQL> ALTER SYSTEM SET use_large_pages=false scope=spfile  sid=’*’;

  1. Optional: Restart Oracle Grid Infrastructure.

# Grid_home/bin/crsctl stop crs

# Grid_home/bin/crsctl start crs

Use Cases

The following topics describe various configuration cases when using the quorum disk manager utility.

New Deployments on Oracle Exadata 12.1.2.3.0 or Later

For new deployments on Oracle Exadata release 12.1.2.3.0 and above, OEDA implements this feature by default when all of the following requirements are satisfied:

  • The system has at least two database nodes and fewer than five storage servers.
  • You are running OEDA release February 2016 or later.
  • You meet the software requirements.
  • Oracle Database is 11.2.0.4 and above.
  • The system has at least one high redundancy disk group.

If the system has three storage servers in place, then two quorum disks will be created on the first two database nodes of the cluster picked by OEDA.

If the system has four storage servers in place, then one quorum disk will be created on the first database node picked by OEDA.

Upgrading to Oracle Exadata Release 12.1.2.3.0 or Later

If the target Exadata system has fewer than five storage servers, at least one high redundancy disk group, and two or more database nodes, you can implement this feature manually using quorumdiskmgr.

Downgrading to a Pre-12.1.2.3.0 Oracle Exadata Release

Rolling back to a pre-12.1.2.3.0 Oracle Exadata release, which does not support quorum disks, from a release that supports quorum disks, which is any release 12.1.2.3.0 and later, requires quorum disk configuration to be removed if the environment has quorum disk implementation in place. You need to remove the quorum disk configuration before performing the Exadata software rollback.

To remove quorum disk configuration, perform these steps:

  1. Ensure there is at least one normal redundancy disk group in place. If not, create one.
  2. Relocate the voting files to a normal redundancy disk group:

$GI_HOME/bin/crsctl replace votedisk +normal_redundancy_diskgroup

  1. Drop the quorum disks from ASM. Run the following command for each quorum disk:

SQL> alter diskgroup diskgroup_name drop quorum disk quorum_disk_name force;

Wait for the rebalance operation to complete. You can tell it is complete when v$asm_operation returns no rows for the disk group.

  1. Delete the quorum devices. Run the following command from each database node that has quorum disks in place:

/opt/oracle.SupportTools/quorumdiskmgr –delete –device [–asm-disk-group asm_disk_group] [–host-name host_name]

  1. Delete the targets. Run the following command from each database node that has quorum disks in place:

/opt/oracle.SupportTools/quorumdiskmgr –delete –target [–asm-disk-group asm_disk_group]

  1. Delete the configuration. Run the following command from each database node that has quorum disks in place:

/opt/oracle.SupportTools/quorumdiskmgr –delete –config