Avoid OFMW application outage by Implementing Redundant FMW Binary!!!

In any enterprise wherein SOA middleware application been implemented getting Outage for SOA application patching is big nightmare. SOA application outage effect whole business as many projects get deployed in SOA platform, if SOA is down then whole business is down. This problem become more sever over the period of time when more and more application keep migrating to SOA Infrastructure.

Oracle has a solution to this problem. For maximum availability, Oracle recommends using redundant binary installations on shared storage.

However, EDG guide does not have details steps information that how we can achieve this recommendation, it just have recommendation. Details of this recommendation can be read using below link-

http://docs.oracle.com/cd/E28280_01/core.1111/e12036/file_sys.htm#SOEDG2014

Moreover, I have often seen many implementation wherein this recommendation always overlooked as it needs lots more work effort during design, implementation and maintenance. 

In this post we will go through more detailed steps how we can achieve redundant binary concept to avoid SOA application outage.

Below are the some of the design principal we need to adopt.

1. Install two identical Oracle homes for your Oracle Fusion Middleware software on two different shared volume
2. If separate volumes are not available on shared storage, Oracle recommends simulating separate volumes using different directories within the same volume

I will explain above guideline using below examples-

Let’s assume we have a domain (test_prd_domain) which has 2 SCA (WLS_SCA1, WLS_SCA2), 2 OSB (WLS_OSB1, WLS_OSB2) and 1 AdminServer. These WLS servers been deployed in two different physical or virtual servers  as stated in below table.

ZFS Share Name	Size	Mounted Folder Name	vServer Name ( WLS Server Names)
192.168.1.1:/export/fmw_share0	20G	/u01/app/oracle/product/fmw	soaserver1.domain.com.au (AdminServer, WLS_SCA1, WLS_OSB1)
192.168.1.1:/export/fmw_share1	20G	/u01/app/oracle/product/fmw	soaserver2.domain.com.au (WLS_SCA1, WLS_OSB1)

Step to implement above design-

Step 1:

Create two NFS share on your NFS or ZFS device of 20 gb or more as per your requirement which will contain Oracle fusion middleware binaries. e.g.

192.168.1.1:/export/fmw_share0 (20 GB)

192.168.1.1:/export/fmw_share1 (20 GB)

Note: Instruction how to create NFS is out of scope of this article.

Step 2:

Update /etc/fstab file in both virtual or physical sever e.g.

soaserver1.domain.com.au

192.168.1.1:/export/fmw_share0 /u01/app/oracle/product/fmw nfs rw,bg,hard,nointr,rsize=1048576,wsize=1048576,tcp,nfsvers=3,timeo=600 0 0

soaserver2.domain.com.au

192.168.1.1:/export/fmw_share1 /u01/app/oracle/product/fmw nfs rw,bg,hard,nointr,rsize=1048576,wsize=1048576,tcp,nfsvers=3,timeo=600 0 0

Sample snap from different environment – 

Note: In above configuration the mount folder path ‘/u01/app/oracle/product/fmw’ would be same in both virtual or physical servers. 

Step 3: 

Create this folder path ‘/u01/app/oracle/product/fmw’ to both virtual or physical severs e.g.

mkdir –p /u01/app/oracle/product/fmw

Step 4:

Mount the ‘/u01/app/oracle/product/fmw´ directory to both virtual or physical servers using below command – 

‘mount –a’

or

‘mount /u01/app/oracle/product/fmw’

-a indicate - Mount all filesystems (of the given types) mentioned in fstab.

Step 5:

Install the Oracle fusion middleware binary in one of the server soaserver1.domain.com.au at this path /u01/app/oracle/product/fmw

Step 6:

Now copy the content of ‘/u01/app/oracle/product/fmw’ directory from one server to another server using scp command

scp -rp /u01/app/oracle/product/fmw/* oracle@ soaserver2.domain.com.au:/u01/app/oracle/product/fmw

-r = indicate copy recursive

-p =indicate preserves modification times, access times, and modes from the original file.

Step 7:

Run the ‘df –h’ command at both virtual or physical servers to confirm the share location of ‘/u01/app/oracle/product/fmw’ folder. Also check the used, free size of file system.

Both servers should have different NFS or ZFS share location.

Step 8:

For additional verification run the below command to both servers for ‘fmw’ location. The result should match in both virtual or physical servers.    

find /u01/app/oracle/product/fmw -maxdepth 1 -type d -print0 | xargs -0 -I {} sh -c 'echo -e $(find {} | wc -l) {}' | sort –n

Step 9:

Create a domain and start all WLS server instance.

WLS servers instance (AdminServer, WLS_SCA1, WLS_OSB1) which are running on this server soaserver1.domain.com.au server will be using Oracle fusion middlware binary from this location 192.168.1.1:/export/fmw_share0 share

Sample Snap from different environment – 

WLS servers instance (WLS_SCA2, WLS_OSB2) which are running on this server soaserver2.domain.com.au server will be using Oracle fusion middlware binary from this location 192.168.1.1:/export/fmw_share1 share

Sample Snap from different environment – 

Pros:

1. Having redundant binary for fmw folder helps a lot during patching process to avoid outage of SOA application and give major relief to business. Business gains more confidence of SOA platform.
2. In case one of NFS share become corrupted or unavailable, only half server capacity are affected, remain half WLS servers still runs using different NFS share. For additional protection, it has been recommends to mirror disk for these volumes or shares.

Cons:

1. Increase little bit overhead to the SOA administrator while doing patching. In case patching SOA administrator has to do patching for two different FWM NFS share location

I hope this article will help you to implement OFMW redundant binary for your SOA environment where you can enjoy freedom from SOA application outage.

Thanks for reading. Have a good day :)