Overview Database Machines

Data warehousing:

Oracle database machine will support large and complex queries. Since the storage is connected to high speed infiniband networks, you will be getting more than enough I/O throughput to support massive scans. Using the smart scan feature, it reduces unproductive I/O. It aslo supports the parallel processing to improve the system performance. It uses the hybrid column compression to reduce the storage space.


Database machines are completely eliminating the OLTP issues. It supports large user populations and transactions by providing the enough I/Os per second and caching frequently accessed data. It provides the consistent performance across all the tables and minimizing the IO latency.


To reduce the datacenter space, most of the companies are behind the virtualization technologies for small and mid range operations. How this database machine is address the consolidation issues? In database machine, you can accommodate multiple workloads on same box instead of having the multiple database servers on your environment. You can also prioritize the workloads but it requires proper analysis and planning prior to the implementation.


Database eliminates the configuration issues completely. Since only oracle is going to provide the support for complete database machine components , all the hardwares  and  firmwares will be compatibility  with the oracle database software.It also creates the well-balanced configuration across the database machine to eliminates the bottlenecks.

Database machine consists following components on it.

  1. Exadata Storage servers (cell)
  2. Computing Nodes (Database servers – Typical oracle Linux x86_64 servers )
  3. Infiniband Switches (Internal networking )
  4. Cisco switches. (External networking )
  5. Power distribution units

Architecture of Exadata Database Machine

Exadata Database machine provides a high performance,high availability and plenty of storage space platform for oracle database .The high availability clustering is is provided by Oracle RAC and ASM will be responsible for storage mirroring .Infiniband technology provides high bandwidth and low latency cluster interconnect and storage networking.  The powerful compute nodes joins in the RAC cluster to offers the great performance.

In this section, we will see the

  • Exadata Database Machine Network architecture
  • Exadata Database Machine Storage architecture
  • Exadata Database Machine Software architecture.
  • How to scale up the Exadata Database Machine

Key components of the Exadata Database Machine

Shared storage: Exadata Storage servers 

Database Machine provides intelligent, high-performance shared storage to both single-instance and RAC implementations of Oracle Database using Exadata Storage Server technology.The Exadata storage servers is designed to provide the storage to oracle database using the ASM (Automatic Storage Management). ASM keeps the redundant copies of data on separate Exadata Storage Servers and it protects against the data loss if you lost the disk or entire storage server.

Shared Network – Infiniband 

Database machine uses the infiniband network for interconnect between database servers and exadata storage servers. The infiniband network provides 40Gb/s speed.So the latency is very low and offers the high bandwidth. In Exadata Database machine , multiple infiniband switches  and interface boning will be used to provide the network redundancy.

Shared cache:

The database machine’s RAC environment, the database instance buffer cache are shared. If one instance has kept some data on cache and that required by another instance,the data will be provided to the required node via infiniband cluster interconnect. It increases the performance since the data is happening between memory to memory via cluster interconnect.

Database Server cluster:

The Exadata database machine’s full rack consists , 8 compute nodes and you can able to build the 8-n0de cluster using the oracle RAC. The each  compute nodes has up to 80 CPU cores and 256GB memory .

Cluster interconnect:

By default, the database machine is configured use the infiniband storage network as cluster interconnect.

Database Machine – Network Architecture

There are three different networks has been shown on the above diagram.

Management Network –  ILOM:

ILOM(Integrated lights out manager) is the default remote hardware management on all oracle servers.It uses the traditional Ethernet network to manage the exadata database machine remotely. ILOM provides the graphical remote administration facility and   it also helps the system administrators to monitor the hardware remotely.

Client Access:

The database servers will be accessed by application servers via Ethernet network. Bonding will be created using multiple ethernet adapters for network redundancy and load balancing. Note: This database machine consists Cisco switch to provide the connectivity to ethernet networks.

InfiniBand Network Architecture

The below diagrams shows that how the infiniband links are connected to different components on X3-2 Half/Full Rack setup.

infiniband switch x3-2 half-full rack

The spine switch will be exists only on half rack and full rack  exadata database configuration only. The spine switch will help you to scale the environment by providing the Inifiniband links to multiple racks. In the quarter rack of X3-2 model, you will get leaf switches . You can scale up to 18 rack by adding the infiniband cables to the infiniband switches.

How we can interconnect two racks ? Have a look at the below diagram closely.Single InfiniBand network formed based on a Fat Tree topology

Scale two Racks

Six ports on each leaf switch are reserved for external connectivity.These ports are used for Connecting to media servers for tape backup,Connecting to external ETL servers,Client or application access Including Oracle Exalogic Elastic Cloud

Database Machine Software Architecture

Software architecture- exadata

CELLSRV, MS,RS & IORM are the important process of the exadata storage cell servers. In the DB servers , these storage’s griddisks are used to create the ASM diskgroup.In the database server, there will be special library called LIBCELL. In combination with the database kernel and ASM, LIBCELL transparently maps database I/O to exadata storage server.

There is no other filesystems are allowed to create in Exadata storage cell. Oracle Database must use the ASM for volume manager and filesystem.

Customers has option to choose the database servers operating system between oracle Linux and oracle Solaris x86 . Exadata will support the oracle database 11g release 2 and laster versions of database.

Database Machine Storage  Architecture

Exadata Storage cell

Exadata storage servers has above mentioned software components.  Oracle Linux is the default operating system for exadata storage cell software . CELLSRV is the core exadata storage component which provides the most of the services. Management Server (MS) provides Exadata cell management and configuration.MS is responsible for sending alerts and collects some statistics in addition to those collected by CELLSRV.Restart Server (RS) is used to start up/shut down the CELLSRV and MS services and monitors these services to automatically restart them if required.

How the disks are mapped to  Database from the Exadata storage servers ?

Exadata Disks overview

If you look the below image , you can observe that database servers is considering   the each cell nodes as failure group.

Exadata DG

Exploring the Exadata Storage Cell Processes

Exadata storage cell is new to the industry and only oracle is offering such a customized storage for oracle database. Unlike the traditional SAN storage, Exadata data storage will help to reduce the processing at the DB node level. Since the exadata storage cell has its own processors and 64GB physical memory, it can easily offload the DB nodes. It has huge amount of Flash storage to speed up the I/O. The default Flash cache settings is write through. These flash can also be used as storage (like hard drive). Flash devices can give 10x better performance than normal harddrive.

Examine the Exadata Storage cell Processes

1. Login to Exadata storage cell .

login as: root
[email protected]'s password:
Last login: Sat Nov 15 01:50:58 2014
[[email protected] ~]#
[[email protected] ~]# uname -a
Linux uaexacell1 2.6.39-300.26.1.el5uek #1 SMP Thu Jan 3 18:31:38 PST 2013 x86_64 x86_64 x86_64 GNU/Linux
[[email protected] ~]#
  • List the exadata cell restart server process.(RS)
[[email protected] ~]# ps -ef |grep cellrs
root     10001     1  0 14:23 ?        00:00:00 /opt/oracle/cell11. -ms 1 -cellsrv 1
root     10009 10001  0 14:23 ?        00:00:00 /opt/oracle/cell11. -ms 1 -cellsrv 1
root     10010 10001  0 14:23 ?        00:00:00 /opt/oracle/cell11. -ms 1 -cellsrv 1
root     10011 10001  0 14:23 ?        00:00:00 /opt/oracle/cell11. -ms 1 -cellsrv 1
root     10012 10011  0 14:23 ?        00:00:00 /opt/oracle/cell11. -rs_conf /opt/oracle/cell11. -ms_conf /opt/oracle/cell11. -cellsrv_conf /opt/oracle/cell11. -debug 0
root     10022 10012  0 14:23 ?        00:00:00 /opt/oracle/cell11. -rs_conf /opt/oracle/cell11. -ms_conf /opt/oracle/cell11. -cellsrv_conf /opt/oracle/cell11. -debug 0
root     12992 12945  0 14:48 pts/2    00:00:00 grep cellrs
[[email protected] ~]#

RS – Restart server process is responsible to make the cellsrv & ms process up for all the time. If these process are not responding or terminated, automatically RS(restart server) , will restart the cellsrv & ms process.

  • List the MS process. (Management Server process). MS maintains the cell configuration with the help of cellcli(command line utility). It also responsible for sending alerts and collecting the exadata cell statistics.
[[email protected] ~]# ps -ef | grep ms.err
root     10013 10009  1 14:23 ?        00:00:21 /usr/java/jdk1.5.0_15/bin/java -Xms256m -Xmx512m -Djava.library.path=/opt/oracle/cell11. -Ddisable.checkForUpdate=true -jar /opt/oracle/cell11. -out /opt/oracle/cell11. -err /opt/oracle/cell11.
root     13945 12945  0 14:56 pts/2    00:00:00 grep ms.err
[[email protected] ~]#

MS(Management server) process’s parent process id belongs to RS (restart server).RS will restart the MS when it crashes or terminated abnormally.

  • CELLSRV is multi-threaded  process which provides the storage services to the database nodes. CELLSRV communicates with oracle database to serve simple block requests,such as database buffer cache reads and smart scan requests. You list the cellsrv process using below mentioned command.
[[email protected] ~]# ps -ef | grep "/cellsrv "
root      5705  10010 8 19:13 ?        00:08:20 /opt/oracle/cell11. 100 5000 9 5042
1000      8390  4457  0 20:57 pts/1    00:00:00 grep /cellsrv
[[email protected] ~]#

CELLSRV process’s parent process id belongs to RS process(restart server).RS will restart the CELLRSV when it crashes or terminated abnormally.

  • Let me  kill the MS process and see if it restarts automatically.
[[email protected] ~]# ps -ef |grep ms.err
root     10013 10009  0 14:23 ?        00:00:23 /usr/java/jdk1.5.0_15/bin/java -Xms256m -Xmx512m -Djava.library.path=/opt/oracle/cell11. -Ddisable.checkForUpdate=true -jar /opt/oracle/cell11. -out /opt/oracle/cell11. -err /opt/oracle/cell11.
root     15220 12945  0 15:06 pts/2    00:00:00 grep ms.err
[[email protected] ~]# kill -9 10013
[[email protected] ~]# ps -ef |grep ms.err
root     15245 12945  0 15:07 pts/2    00:00:00 grep ms.err
[[email protected] ~]# ps -ef |grep ms.err
root     15249 12945  0 15:07 pts/2    00:00:00 grep ms.err
[[email protected] ~]#

within few seconds another MS process has started with new PID.

[[email protected] ~]# ps -ef |grep ms.err
root     15366 10009 74 15:07 ?        00:00:00 /usr/java/jdk1.5.0_15/bin/java -Xms256m -Xmx512m -Djava.library.path=/opt/oracle/cell11. -Ddisable.checkForUpdate=true -jar /opt/oracle/cell11. -out /opt/oracle/cell11. -err /opt/oracle/cell11.
root     15379 12945  0 15:07 pts/2    00:00:00 grep ms.err
[[email protected] ~]#
  • How to stop and start the services on exadata storage cell  using the init scripts ? Its like other start up scripts will be located on /etc/init.d and link has been added to /etc/rc3.d to bring up the cell process on the start-up.
[[email protected] ~]# cd /etc/init.d
[[email protected] init.d]# ls -lrt |grep cell
lrwxrwxrwx 1 root root    50 Nov 15 01:15 celld -> /opt/oracle/cell/cellsrv/deploy/scripts/unix/celld
[[email protected] init.d]# cd /etc/rc3.d
[[email protected] rc3.d]# ls -lrt |grep cell
lrwxrwxrwx 1 root root 15 Nov 15 01:15 S99celld -> ../init.d/celld
[[email protected] rc3.d]#

This script can be used to start, stop, restart the exadata cell software.

To stop the cell software

[[email protected] rc3.d]# ./S99celld stop
Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
[[email protected] rc3.d]#

To start the cell software

[[email protected] rc3.d]# ./S99celld start
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services...  running
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...
The STARTUP of MS services was successful.
[[email protected] rc3.d]#

TO restart the cell software,

[[email protected] rc3.d]# ./S99celld restart
Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services...  running
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...
The STARTUP of MS services was successful.
[[email protected] rc3.d]#

Cell software services will be managed using celladmin user and cellcli utility. You can also start,stop,restart the services using cellcli utility.We will see the cellcli in next article.