Weblogic Muxer Thread

1         Weblogic Muxer Socket Thread

Weblogic has Thread group called ‘Weblogic Muxer’, once you take thread dumps you could see always 4-8 running Muxer Threads e.g. ExecuteThread: '3' for queue: 'weblogic.socket.Muxer

Muxer threads plays very important role when your Production environment struggling with performance problems.

In this article I just want to give some insight about Muxer threads which I gained from my own experience in my current project. 

1.1       What is Muxer Threads:

Muxer thread is software components which weblogic uses to read/write the data from the socket on which external system communicating, once Muxer thread reads data from socket its pass on that message to Weblogic Execute queue and from there Weblogic Self Tunning Thread pool threads pick the message and process the request.

You can unable/disable “Enable Native IO” option from Admin console for a specific servers –

1.2       Types of Muxer Threads-

There are three types of Muxer threads –

1.2.1     Native Muxers

Native muxers use platform-specific native binaries to read data from sockets. The majority of all platforms provide some mechanism to poll a socket for data. For example, Unix systems use the poll system call and the Windows architecture uses completion ports. Native muxers provide superior scalability because they implement a non-blocking thread model. When a native muxer is used, the server creates a fixed number of threads dedicated to reading incoming requests. Oracle recommends using the default setting of true for the Enable Native IO parameter which allows the server to automatically select the appropriate muxer to use.

Configuration:

Checked "Enable Native IO" only from Admin console.

Count Of Muxer Threads:

4

Stack Trace from Thread Dump:

"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=3 tid=0x00000001042f4800 nid=0x1e waiting for monitor entry [0xffffffff5e6ff000]

   java.lang.Thread.State: BLOCKED (on object monitor)

                at weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:92)

                - waiting to lock <0xfffffffe20485978> (a java.lang.String)

                at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)

                at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)

                at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)

                at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

Comment/Remark:

In this case Weblogic will use native class "DevPollSocketMuxer" for Sun hardware.

1.2.2     Java Muxer

A Java muxer has the following characteristics:

·         Uses pure Java to read data from sockets.

·         It is also the only muxer available for RMI clients.

·         Blocks on reads until there is data to be read from a socket. This behavior does not scale well when there are a large number of sockets and/or when data arrives infrequently at sockets. This is typically not an issue for clients, but it can create a huge bottleneck for a server.

If the Enable Native IO parameter is not selected, the server instance exclusively uses the Java muxer. This maybe acceptable if there are a small number of clients and the rate at which requests arrive at the server is fairly high. Under these conditions, the Java muxer performs as well as a native muxer and eliminate Java Native Interface (JNI) overhead. Unlike native muxers, the number of threads used to read requests is not fixed and is tunable for Java muxers by configuring the Percent Socket Readers parameter setting in the Administration Console. Ideally, you should configure this parameter so the number of threads roughly equals the number of remote concurrently connected clients up to 50% of the total thread pool size. Each thread waits for a fixed amount of time for data to become available at a socket. If no data arrives, the thread moves to the next socket.

Configuration:

1) UnChecked "Enable Native IO" option from Admin Console

2) Removed"-Dweblogic.MuxerClass=weblogic.socket.NIOSocketMuxer" if exist in setDomainEnv.sh or at Admin server console http://docs.oracle.com/cd/E24329_01/apirefs.1211/e24401/taskhelp/tuning/EnableNIOSocketMuxer.html

Count Of Muxer Threads:

4

Stack Trace from Thread Dump:

"[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x00000001035fb800 nid=0x2c runnable [0xffffffff6a5fe000]

   java.lang.Thread.State: RUNNABLE

 at java.net.SocketInputStream.socketRead0(Native Method)

 at java.net.SocketInputStream.read(SocketInputStream.java:129)

 at weblogic.socket.SocketMuxer.readFromSocket(SocketMuxer.java:988)

 at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:922)

 at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:889)

 at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:339)

 at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)

 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:252)

 at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Comment/Remark:

Weblogic using JAVA Muxer class to read/write the data from sockets.

1.2.3     Non-Blocking IO Muxer

WebLogic Server provides a non-blocking IO implementation that may provide enhanced performance for some applications
To enable non-blocking IO:

1. If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit (see Use the Change Center).
2. In the left pane of the console, expand Environment > Servers.
3. On the Summary of Servers page, select the server instance for which you will enable native IO.
4. Expand the Configuration > Tuning tab.
5. If necessary, click Advanced to access advanced tuning parameters.
6. Enter weblogic.socket.NIOSocketMuxer in the Muxer Class field.
7. Click Save.
8. To activate these changes, in the Change Center of the Administration Console, click Activate Changes.
   Not all changes take effect immediately—some require a restart

Configuration:

1)      Checked "Enable Native IO" option from Admin console.

2)       Added "-Dweblogic.MuxerClass=weblogic.socket.NIOSocketMuxer" in setDomainEnv.sh to override the Muxer class.

Count Of Muxer Threads: 4

Stack Trace from Thread Dump:

"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" daemon prio=3 tid=0x000000010360f800 nid=0x31 runnable [0xffffffff5c2ff000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)

        at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:171)

        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

        - locked <0xfffffffe23f121d8> (a sun.nio.ch.Util$2)

        - locked <0xfffffffe23f121c8> (a java.util.Collections$UnmodifiableSet)

        - locked <0xfffffffe23ebc5b8> (a sun.nio.ch.DevPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

        at weblogic.socket.NIOSocketMuxer.processSockets(NIOSocketMuxer.java:320)

        at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)

        at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)

        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)

        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

Comment/Remark:

This time since we explicitly gave Muxerclass name at JVM level it’s using now "NIOSocketMuxer" class for socket reading.

Snap from Thread dump -

Refer the below URL for details understanding-

http://docs.oracle.com/cd/E24329_01/web.1211/e24390/wls_tuning.htm#i1151673

http://docs.oracle.com/cd/E24329_01/apirefs.1211/e24401/taskhelp/tuning/EnableNIOSocketMuxer.html

1.3       Error Related to Socket Problems –

In Weblogic .out log file you might see lots of IO Exception and Socket Exception mentioned below

<Nov 16, 2012 12:42:28 PM EST> <Error> <WliSbTransports> <BEA-381304> <Exception in HttpInboundMessageContext.close: java.io.IOException: Broken pipe

java.io.IOException: Broken pipe

        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)

        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)

        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)

        at sun.nio.ch.IOUtil.write(IOUtil.java:60)

        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)

        Truncated. see log file for complete stacktrace

> 

<Dec 6, 2012 1:58:11 AM EST> <Error> <WliSbTransports> <BEA-381304> <Exception in HttpOutboundMessageContext.RetrieveHttpResponseWork.run: java.io.IOException: Connection reset by peer

java.io.IOException: Connection reset by peer

                at sun.nio.ch.FileDispatcherImpl.read0(Native Method)

                at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)

                at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)

                at sun.nio.ch.IOUtil.read(IOUtil.java:191)

                at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)

                Truncated. see log file for complete stacktrace

1.4       General Question which rises in Mind in regard to Socket Configuration

Question1: If the configuration contains checked “Enable Native IO” and “-Dweblogic.SocketReaders=8

-Dweblogic.MuxerClass=weblogic.socket.NIOSocketMuxer”

In that case will weblogic uses native IO or non native IO.

Ans:  Above configuration will override default configuration and will use weblogic.socket.NIOSocketMuxer class which is Non blocking class been added later to provide improved performance. Refer this doc for details

http://docs.oracle.com/cd/E24329_01/web.1211/e24390/wls_tuning.htm#i1151673

Question2:  Can we use both Native IO and JAVA IO socket threads together?

Ans: no we can have only one sort of implementation, though your server configuration can have two configurations but one must be suppressed by another configuration.

Question3 : If we use Socket Reader count up to 8 using this configuration “Dweblogic.SocketReaders=8”, what will happen if we disabled the “Enable Native IO” Option, Will weblogic by default allow to use 33% of Execute Threads as Socket Reader threads?

Ans:  No, again default configuration will be override, weblogic will use 8 Muxer thread only to do socked read/write.

Question4: Which one out of Native IO and JAVA IO socket threads we should use in Production with high traffic volume?

Ans:

In our production previously we were having checked “Enable Native IO” and specified “NIOMuxerclass” and were facing many problem e.g. OSB transaction were hanging, were not getting timeout, socket timeout exception was coming in logs, later point of time as per Oracle recommendation we switch back to default Weblogic configuration which checked “Enabled Native IO”, removed NIOMuxer class and after that we witnessed lot more improvement in Production environment.

Question: What should be no of socket reader threads?

Ans:  We have tested various configuration in our Test environment and found that weblogic by default mostly use 4 Muxer threads only regardless of Native or Non Native Muxer Type. For Java Muxer document says it can use 33% threads from Execute Thread groups but we realized even with that configuration in 30 TPS load Weblogic initializing only 4 threads.

2         Summary:

As per my experience I would always suggest to go with default configuration which comes with Weblogic for Socket Reader Muxer threads, previously we have modified the configuration and introduced NIO Muxer class and also increased no of socket Reader threads but it didn’t help for any performance, even we notice lots of strange behaviour start occurring, later point of time we switch back to Default Weblogic Configuration and realized improvement. So go with default configuration. Also when you look at Thread dumps you might see Muxer Thread for Block state for longer time etc so don't be scared for that, Muxer threads are designed in such way only, its normal to see Muxer Thread Blocked state till long time.