JVM Parameter SurvivorRatio

This post is very small but useful, I published this coz during my search I faced quite trouble to get the meaning of SurvivorRatio JVM parameters which is been used to segregate the memory between Eden space and Survivor spaces.

In JVM you can segregate the Young generation from total heap using these parameters e.g. -XX:NewSize=400m -XX:MaxNewSize=400m or -XX:NewRatio=3

In ideal condition 40% vs 60% ratio will be good between Young Generation and Old Generation respectively; alternatively it will depend on what type of application you have deployed on JVM and based on GC.log output values can be tuned further..  

To defined explicit memory area for young generation used the below settings. For example if you have 1gb heap size define using this parameter –Xms1024m –Xmx1024m or –Xms1g –Xmx1g then give this “XX:NewSize=400m -XX:MaxNewSize=400m” to define young generation area.

Now there in one more important configuration which you need to consider while segregating young generation that’s called ‘SurvivorRatio’.

SurvivorRatio been used to further split the memory areas in Young Generation and controls the size of the two survivor spaces, young generation contains three sub area 1) Eden Space 2) Survivor One 3) Survivor Two

Eden Space has two Survivor spaces. One survivor space is empty at any given time. These Survivor Spaces serves as the destination of the next copying collection of any living objects in Eden and the other survivor space. If survivor spaces are too small copying collection overflows directly into the tenured generation.

During my search for survivor ratio I found various blogs, each blogs were saying different things, some of them saying, if you set the SurvivorRatio=6 it will set the ratio between each survivor space and eden to be 1:6, other were saying “each survivor space will be 1/8 the young generation” etc but I didn’t get any such blog which gave me crystal clear formula which will help to drive the Survivor space size.

I decided to do some POC and build a formula for that, so that Survivor ratio can be calculated easily.

Here is the formula which I have derived –

Formula to calculate S1 or S2 -

Total Young Generation / (SurvivorRatio+2)*2

E.g. If you have Young generation 400 Mb, SurvivorRatio set to 8 below will be output-

400 / (8+2) *2 = 20 MB

Similarly here is the formula to derived Eden space –

Formula for Eden Space –

(SR +1) * Young Gen / (SR +2)

E.g. if you have Young generation 400 Mb, SurvivorRatio set to 8 below will be output-

8+1*400/8+2 = 3600/10 = 360 MB

Result of few values of SurvivorRatio which I have tested in Test Environment -

Young Generation	S1	S2	Eden	SurvivorRatio
400	33.31	33.31	333.38	4
400	25	25	350	6
400	20	20	360	8

Note:

As per above result, if you decrease the SurvivorRatio value then you will get more Survivor Space, if you Increase SurvivorRatio you will get less Survivor Space.

Weblogic Muxer Thread

1         Weblogic Muxer Socket Thread

Weblogic has Thread group called ‘Weblogic Muxer’, once you take thread dumps you could see always 4-8 running Muxer Threads e.g. ExecuteThread: '3' for queue: 'weblogic.socket.Muxer

Muxer threads plays very important role when your Production environment struggling with performance problems.

In this article I just want to give some insight about Muxer threads which I gained from my own experience in my current project. 

1.1       What is Muxer Threads:

Muxer thread is software components which weblogic uses to read/write the data from the socket on which external system communicating, once Muxer thread reads data from socket its pass on that message to Weblogic Execute queue and from there Weblogic Self Tunning Thread pool threads pick the message and process the request.

You can unable/disable “Enable Native IO” option from Admin console for a specific servers –

1.2       Types of Muxer Threads-

There are three types of Muxer threads –

1.2.1     Native Muxers

Native muxers use platform-specific native binaries to read data from sockets. The majority of all platforms provide some mechanism to poll a socket for data. For example, Unix systems use the poll system call and the Windows architecture uses completion ports. Native muxers provide superior scalability because they implement a non-blocking thread model. When a native muxer is used, the server creates a fixed number of threads dedicated to reading incoming requests. Oracle recommends using the default setting of true for the Enable Native IO parameter which allows the server to automatically select the appropriate muxer to use.

Configuration:

Checked "Enable Native IO" only from Admin console.

Count Of Muxer Threads:

4

Stack Trace from Thread Dump:

"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=3 tid=0x00000001042f4800 nid=0x1e waiting for monitor entry [0xffffffff5e6ff000]

   java.lang.Thread.State: BLOCKED (on object monitor)

                at weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:92)

                - waiting to lock <0xfffffffe20485978> (a java.lang.String)

                at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)

                at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)

                at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)

                at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

Comment/Remark:

In this case Weblogic will use native class "DevPollSocketMuxer" for Sun hardware.

1.2.2     Java Muxer

A Java muxer has the following characteristics:

·         Uses pure Java to read data from sockets.

·         It is also the only muxer available for RMI clients.

·         Blocks on reads until there is data to be read from a socket. This behavior does not scale well when there are a large number of sockets and/or when data arrives infrequently at sockets. This is typically not an issue for clients, but it can create a huge bottleneck for a server.

If the Enable Native IO parameter is not selected, the server instance exclusively uses the Java muxer. This maybe acceptable if there are a small number of clients and the rate at which requests arrive at the server is fairly high. Under these conditions, the Java muxer performs as well as a native muxer and eliminate Java Native Interface (JNI) overhead. Unlike native muxers, the number of threads used to read requests is not fixed and is tunable for Java muxers by configuring the Percent Socket Readers parameter setting in the Administration Console. Ideally, you should configure this parameter so the number of threads roughly equals the number of remote concurrently connected clients up to 50% of the total thread pool size. Each thread waits for a fixed amount of time for data to become available at a socket. If no data arrives, the thread moves to the next socket.

Configuration:

1) UnChecked "Enable Native IO" option from Admin Console

2) Removed"-Dweblogic.MuxerClass=weblogic.socket.NIOSocketMuxer" if exist in setDomainEnv.sh or at Admin server console http://docs.oracle.com/cd/E24329_01/apirefs.1211/e24401/taskhelp/tuning/EnableNIOSocketMuxer.html

Count Of Muxer Threads:

4

Stack Trace from Thread Dump:

"[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x00000001035fb800 nid=0x2c runnable [0xffffffff6a5fe000]

   java.lang.Thread.State: RUNNABLE

 at java.net.SocketInputStream.socketRead0(Native Method)

 at java.net.SocketInputStream.read(SocketInputStream.java:129)

 at weblogic.socket.SocketMuxer.readFromSocket(SocketMuxer.java:988)

 at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:922)

 at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:889)

 at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:339)

 at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)

 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:252)

 at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Comment/Remark:

Weblogic using JAVA Muxer class to read/write the data from sockets.

1.2.3     Non-Blocking IO Muxer

WebLogic Server provides a non-blocking IO implementation that may provide enhanced performance for some applications
To enable non-blocking IO:

1. If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit (see Use the Change Center).
2. In the left pane of the console, expand Environment > Servers.
3. On the Summary of Servers page, select the server instance for which you will enable native IO.
4. Expand the Configuration > Tuning tab.
5. If necessary, click Advanced to access advanced tuning parameters.
6. Enter weblogic.socket.NIOSocketMuxer in the Muxer Class field.
7. Click Save.
8. To activate these changes, in the Change Center of the Administration Console, click Activate Changes.
   Not all changes take effect immediately—some require a restart

Configuration:

1)      Checked "Enable Native IO" option from Admin console.

2)       Added "-Dweblogic.MuxerClass=weblogic.socket.NIOSocketMuxer" in setDomainEnv.sh to override the Muxer class.

Count Of Muxer Threads: 4

Stack Trace from Thread Dump:

"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" daemon prio=3 tid=0x000000010360f800 nid=0x31 runnable [0xffffffff5c2ff000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)

        at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:171)

        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

        - locked <0xfffffffe23f121d8> (a sun.nio.ch.Util$2)

        - locked <0xfffffffe23f121c8> (a java.util.Collections$UnmodifiableSet)

        - locked <0xfffffffe23ebc5b8> (a sun.nio.ch.DevPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

        at weblogic.socket.NIOSocketMuxer.processSockets(NIOSocketMuxer.java:320)

        at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)

        at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)

        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)

        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

Comment/Remark:

This time since we explicitly gave Muxerclass name at JVM level it’s using now "NIOSocketMuxer" class for socket reading.

Snap from Thread dump -

Refer the below URL for details understanding-

http://docs.oracle.com/cd/E24329_01/web.1211/e24390/wls_tuning.htm#i1151673

http://docs.oracle.com/cd/E24329_01/apirefs.1211/e24401/taskhelp/tuning/EnableNIOSocketMuxer.html

1.3       Error Related to Socket Problems –

In Weblogic .out log file you might see lots of IO Exception and Socket Exception mentioned below

<Nov 16, 2012 12:42:28 PM EST> <Error> <WliSbTransports> <BEA-381304> <Exception in HttpInboundMessageContext.close: java.io.IOException: Broken pipe

java.io.IOException: Broken pipe

        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)

        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)

        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)

        at sun.nio.ch.IOUtil.write(IOUtil.java:60)

        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)

        Truncated. see log file for complete stacktrace

> 

<Dec 6, 2012 1:58:11 AM EST> <Error> <WliSbTransports> <BEA-381304> <Exception in HttpOutboundMessageContext.RetrieveHttpResponseWork.run: java.io.IOException: Connection reset by peer

java.io.IOException: Connection reset by peer

                at sun.nio.ch.FileDispatcherImpl.read0(Native Method)

                at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)

                at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)

                at sun.nio.ch.IOUtil.read(IOUtil.java:191)

                at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)

                Truncated. see log file for complete stacktrace

1.4       General Question which rises in Mind in regard to Socket Configuration

Question1: If the configuration contains checked “Enable Native IO” and “-Dweblogic.SocketReaders=8

-Dweblogic.MuxerClass=weblogic.socket.NIOSocketMuxer”

In that case will weblogic uses native IO or non native IO.

Ans:  Above configuration will override default configuration and will use weblogic.socket.NIOSocketMuxer class which is Non blocking class been added later to provide improved performance. Refer this doc for details

http://docs.oracle.com/cd/E24329_01/web.1211/e24390/wls_tuning.htm#i1151673

Question2:  Can we use both Native IO and JAVA IO socket threads together?

Ans: no we can have only one sort of implementation, though your server configuration can have two configurations but one must be suppressed by another configuration.

Question3 : If we use Socket Reader count up to 8 using this configuration “Dweblogic.SocketReaders=8”, what will happen if we disabled the “Enable Native IO” Option, Will weblogic by default allow to use 33% of Execute Threads as Socket Reader threads?

Ans:  No, again default configuration will be override, weblogic will use 8 Muxer thread only to do socked read/write.

Question4: Which one out of Native IO and JAVA IO socket threads we should use in Production with high traffic volume?

Ans:

In our production previously we were having checked “Enable Native IO” and specified “NIOMuxerclass” and were facing many problem e.g. OSB transaction were hanging, were not getting timeout, socket timeout exception was coming in logs, later point of time as per Oracle recommendation we switch back to default Weblogic configuration which checked “Enabled Native IO”, removed NIOMuxer class and after that we witnessed lot more improvement in Production environment.

Question: What should be no of socket reader threads?

Ans:  We have tested various configuration in our Test environment and found that weblogic by default mostly use 4 Muxer threads only regardless of Native or Non Native Muxer Type. For Java Muxer document says it can use 33% threads from Execute Thread groups but we realized even with that configuration in 30 TPS load Weblogic initializing only 4 threads.

2         Summary:

As per my experience I would always suggest to go with default configuration which comes with Weblogic for Socket Reader Muxer threads, previously we have modified the configuration and introduced NIO Muxer class and also increased no of socket Reader threads but it didn’t help for any performance, even we notice lots of strange behaviour start occurring, later point of time we switch back to Default Weblogic Configuration and realized improvement. So go with default configuration. Also when you look at Thread dumps you might see Muxer Thread for Block state for longer time etc so don't be scared for that, Muxer threads are designed in such way only, its normal to see Muxer Thread Blocked state till long time.

OFMW11.1.1.6.1 Timeout...

OFMW11.1.1.6.1 Timeout...

At server level there are some parameters which are quite important and useful; one of them is related to Timeout.

Now definition of timeout is quite simple when you are making any http/https/rmi etc connection you can specify the timeout parameter which means after establishing a connection, if transaction is taking more time than configured timeout value then transaction should get timeout. This is important to implement because it’s prevent long-running processes from holding resources over an extended period of time.

I am going to talk about timeout parameters which are existing in OFM11g 11.1.1.6.1 at various places and their meaning and relations.

SOA server’s timeout configuration –

There are multiple places where you can set the timeout for SOA servers.

1         SyncMaxWaitTime:

The SyncMaxWaitTime property applies to durable synchronous processes having breakpoint activities which forces dehydration.

For this parameter you have concentrate more in details because I have seen lots of audience that have wrong understanding for this parameter.

First of all this parameter only for the Synchronous process with breakpoint activities e.g. receive (except first one), pick (onAlarm, onMessage), wait, reply and checkpoint etc. This configuration is not for the asynchronous process since in case of async BPEL no receiver thread is waiting to timeout.

Another mistake I have noticed the users trying to co-relate this parameter for the synchronous transaction to the external system from Asynchronous BPEL/composite process. Users expect a Sync invocation to external system Webservice should timeout if it’s taking more time than configure for SyncMaxWaitTime period.  This parameter is not applicable for this scenario, try to implement httpReadTimeout and httpConnTimeout property within composite itself, later describe in this post.

Example of Sync BPEL Process with Breakpoint Activities -

<receive name="receiveInput" partnerLink="client" variable="input"

createInstance="yes" />

<assign>

...

</assign>

<wait for="'PT10S'" />

<reply name="replyOutput" partnerLink="client" variable="output" />

For details understanding for this parameter, how engine level its works, please refer the below URL

http://docs.oracle.com/cd/E17904_01/integration.1111/e10224/bp_sync.htm#BABCAGIA

1.1       Steps to modify the SyncMaxWaitTime –

1)      Login into EM console

2)      Expand SOA and right click on "soa-infra" and select: SOA Administration -> BPEL Properties

3)      Click on "More BPEL Configuration Properties" link and Locate syncMaxWaitTime(default 45 second) and change it.

 

2         BPEL Engine Time: 

There are many BPEL engine EJB’s exist in “soa-infra” Enterprise application which process the message during runtime, each EJB has its own timeout parameter which need to configure and should be in relation with JTA and SyncMaxWaitTime.

2.1       Steps to modify the BPEL Engine EJBs timeout–

1)      Log into Oracle WebLogic Administration Console

2)      Click Deployment and find out ‘soa-infra” Enterprise application, stop this application forcefully.

3)      Click on this application soa-infra -> list of EJBs will be appeared.

4)      Following EJBs need to be updated

Ø  BPELActivityManagerBean

Ø  BPELDeliveryBean

Ø  BPELDispatcherBean

Ø  BPELEngineBean

Ø  BPELFinderBean

Ø  BPELInstanceManagerBean

Ø  BPELProcessManagerBean

Ø  BPELSensorValuesBean

Ø  BPELServerManagerBean

5)      Change the parameter in the Configuration tab for the Transaction Timeout setting(default 300 sec for most of EJBs).

6)      Click Save.

7)      Save the Plan.xml to some known location - Ensure no other file is named Plan.xml -.

8)      Restart the  SOA servers and start this application back.

e.g.

Home >Summary of Servers >Summary of Deployments >soa-infra >BPELDispatcherBean

3         JTA:

This is timeout which a weblogic server allows to wait for a transaction to be complete in first phase for 2 phase transaction which is getting used for distributed transactions.

3.1       Steps to modify the JTA Timeout in Weblogic–

1)      Log into Oracle WebLogic Administration Console.

2)      Click on your ‘domain name’ in console

3)      Go to JTA tab page

4)      Change the value of Timeout Seconds (the default is 30).

5)      Click Save.

6)      Restart Oracle WebLogic Server.

4         Webservice Read and connection timeout:

When a http request been made to external system using web service 1) httpReadTimeout 2) httpConnTimeout and 3) optimization properties can be used to timeout that transaction, values are in milliseconds and this configuration only applicable for http protocol based web service invocation.

Modify the composite.xml for the service for which timeout has to set-

<reference name="Service1"
ui:wsdlLocation="http://localhost:8001/soa-infra/services/default/Child!1.0/Child.wsdl">
<interface.wsdl interface="http://xmlns.oracle.com/test/Child/Child#wsdl.interface(Child)"/>
<binding.ws port="http://xmlns.oracle.com/test/Child/Child#wsdl.endpoint(child_client_ep/Child_pt)"
location="http://localhost:8001/soa-infra/services/default/Child!1.0/child_client_ep?WSDL">
<property name="oracle.webservices.local.optimization">false</property>
<property name="oracle.webservices.httpReadTimeout" type="xs:string" many="false">10000</property>

<property name="oracle.webservices.httpConnTimeout" type="xs:string" many="false">5000</property>


</binding.ws>
</reference>

oracle.webservices.local.optimization parameter value could be true/false, make it false if service which user is going to invoke reside in same Weblogic Instance to avoid execution of SOAP stack

In the BPEL process you can add a Catch branch for a RemoteFault if you want to catch this error.

Note when using this parameter it will return a "bpelx:timeout" Fault, and not a RemoteFault. It's possible to use this setting for Invoke or Receive activities.

Refer the Oracle Note ID 979683.1 for more details

5         Relation between timeouts

If the invocation pattern is like this-

Request   - Consumer invokes >> SOA Load balancer >> SOA Managed Server >> Provider System

Response - Consumer invokes << SOA Load balancer << SOA Managed Server << Provider System

Consumer system/API Timeout >> SOA Load Balancer Timeout (If Present) >> Global Transaction Timeout JTA >> BPEL EJB's transaction timeout >> syncMaxWaitTime OR composite Level Timeout >> Provider System/API timeout

6         Weblogic Server Timeout parameters 10.3.6.

There are few more parameters related to timeout which probably you might consider to revisit during problem or to improve your performance.

6.1       Weblogic Console>> DomainName>>Environment >>Servers>>click on particular server>>Protocal tab page >>General tab page:

6.1.1    Complete Message Timeout:       

The maximum number of seconds that this server waits for a complete message to be received.

6.1.2     Idle Connection Timeout:

The maximum number of seconds that a connection is allowed to be idle before it is closed by the server.

6.1.3     Tunneling Client Timeout:

The amount of time (in seconds) after which a missing tunneled client is considered dead.

6.2       Weblogic Console>> DomainName>>Environment >>Servers>>click on particular server>>Protocal tab page >>HTTP tab page:

6.2.1     Post Timeout:

Timeout (in seconds) for reading HTTP POST data in a servlet request. If the POST data is chunked, the amount of time the server waits between the end of receiving the last chunk of data and the end of receiving the next chunk of data in an HTTP POST before it times out.

6.2.2     Duration:

The amount of time this server waits before closing an inactive HTTP connection.

6.2.3     Tunneling Client Timeout:

The amount of time (in seconds) after which a missing tunneled client is considered dead.

6.3       Home >Summary of JDBC Data Sources >SOADataSource>> Transaction Tab page

6.3.1     XA Transaction Timeout:

The number of seconds to set as the transaction branch timeout.
If set, this value is passed as the transaction timeout value in the XAResource.setTransactionTimeout() call on the XA resource manager, typically the JDBC driver.
When this value is set to 0, the WebLogic Server Transaction Manager passes the global WebLogic Server transaction timeout in seconds in the method.
If set, this value should be greater than or equal to the global WebLogic Server transaction timeout.
Note: You must enable XaSetTransactionTimeout to enable setting the transaction branch timeout

6.4       Home >Summary of JDBC Data Sources >SOADataSource >> Connection Pool Tab page

6.4.1     Inactive Connection Timeout:

The number of inactive seconds on a reserved connection before WebLogic Server reclaims the connection and releases it back into the connection pool.
You can use the Inactive Connection Timeout feature to reclaim leaked connections - connections that were not explicitly closed by the application. Note that this feature is not intended to be used in place of properly closing connections.
When set to 0, the feature is disabled.

6.4.2     Connection Reserve Timeout:

The number of seconds after which a call to reserve a connection from the connection pool will timeout.
When set to 0, a call will never timeout.
When set to -1, a call will timeout immediately.

6.4.3     Statement Timeout:

The time after which a statement currently being executed will time out.
StatementTimeout relies on underlying JDBC driver support. WebLogic Server passes the time specified to the JDBC driver using the java.sql.Statement.setQueryTimeout() method. If your JDBC driver does not support this method, it may throw an exception and the timeout value is ignored.
A value of -1 disables this feature.
A value of 0 means that statements will not time out

6.5       Connection Factory Timeout:

If you have implemented the JMS in your project, then there must be a connection factory exist to locate JMS queue, each connection factory has its own timeout-

Weblogic Admin Console>>Home>>JMS Module>>(Your JMS Module)>>(connection Factory Name)>> Transaction Tab page

6.5.1     Transaction Timeout:

The timeout value (in seconds) for all transactions on connections created with this connection factory.

If a transacted session is still active after the timeout has elapsed, the transaction is rolled back. A value of 0 indicates that the default value will be used. If you have long-running transactions, you might want to adjust the value of this attribute to allow transactions to complete.