1 Weblogic Muxer Socket Thread
Weblogic has Thread group called ‘Weblogic Muxer’, once you
take thread dumps you could see always 4-8 running Muxer Threads e.g. ExecuteThread:
'3' for queue: 'weblogic.socket.Muxer
Muxer threads plays very important role when your Production
environment struggling with performance problems.
In this article I just want to give some insight about Muxer
threads which I gained from my own experience in my current project.
1.1 What is Muxer Threads:
Muxer thread is software components which weblogic uses to
read/write the data from the socket on which external system communicating,
once Muxer thread reads data from socket its pass on that message to Weblogic
Execute queue and from there Weblogic Self Tunning Thread pool threads pick the
message and process the request.
You can unable/disable “Enable Native IO” option from Admin
console for a specific servers –
1.2 Types of Muxer Threads-
There are three types of Muxer threads –
1.2.1 Native Muxers
Native muxers use platform-specific native binaries to read data from sockets. The majority of all platforms provide some mechanism to poll a socket for data. For example, Unix systems use the poll system call and the Windows architecture uses completion ports. Native muxers provide superior scalability because they implement a non-blocking thread model. When a native muxer is used, the server creates a fixed number of threads dedicated to reading incoming requests. Oracle recommends using the default setting of true for the Enable Native IO parameter which allows the server to automatically select the appropriate muxer to use.
Configuration:
Checked "Enable Native IO" only from Admin
console.
Count Of Muxer
Threads:
4
Stack Trace from
Thread Dump:
"ExecuteThread:
'2' for queue: 'weblogic.socket.Muxer'" daemon prio=3
tid=0x00000001042f4800 nid=0x1e waiting for monitor entry [0xffffffff5e6ff000]
java.lang.Thread.State: BLOCKED (on object
monitor)
at
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:92)
- waiting to lock
<0xfffffffe20485978> (a java.lang.String)
at
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
at
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
at
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
Comment/Remark:
In this case Weblogic will use native class "DevPollSocketMuxer" for Sun
hardware.
1.2.2 Java Muxer
A Java muxer has the following characteristics:
·
Uses pure Java to
read data from sockets.
·
It is also the only
muxer available for RMI clients.
·
Blocks on reads
until there is data to be read from a socket. This behavior does not scale well when there are a large number of
sockets and/or when data arrives infrequently at sockets. This is typically
not an issue for clients, but it can create a huge bottleneck for a server.
If the Enable Native IO parameter is not selected, the server instance exclusively uses the
Java muxer. This maybe acceptable if there are a small number of clients and
the rate at which requests arrive at the server is fairly high. Under these
conditions, the Java muxer performs as well as a native muxer and eliminate
Java Native Interface (JNI) overhead. Unlike native muxers, the number of
threads used to read requests is not fixed and is tunable for Java muxers by configuring the Percent Socket Readers
parameter setting in the Administration Console. Ideally, you should
configure this parameter so the number of threads roughly equals the number of
remote concurrently connected clients up to 50% of the total thread pool size.
Each thread waits for a fixed amount of time for data to become available at a
socket. If no data arrives, the thread moves to the next socket.
Configuration:
1) UnChecked "Enable Native IO" option from Admin
Console
2) Removed"-Dweblogic.MuxerClass=weblogic.socket.NIOSocketMuxer"
if exist in setDomainEnv.sh or at Admin server console http://docs.oracle.com/cd/E24329_01/apirefs.1211/e24401/taskhelp/tuning/EnableNIOSocketMuxer.html
Count Of Muxer
Threads:
4
Stack Trace from
Thread Dump:
"[ACTIVE]
ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'"
daemon prio=3 tid=0x00000001035fb800 nid=0x2c runnable [0xffffffff6a5fe000]
java.lang.Thread.State: RUNNABLE
at
java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at
weblogic.socket.SocketMuxer.readFromSocket(SocketMuxer.java:988)
at
weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:922)
at
weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:889)
at
weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:339)
at
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
at
weblogic.work.ExecuteThread.execute(ExecuteThread.java:252)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Comment/Remark:
Weblogic using JAVA Muxer class to read/write the data from
sockets.
1.2.3 Non-Blocking IO Muxer
WebLogic Server provides a non-blocking IO implementation that may provide enhanced performance for some applicationsTo enable non-blocking IO:
- If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit (see Use the Change Center).
- In the left pane of the console, expand Environment > Servers.
- On the Summary of Servers page, select the server instance for which you will enable native IO.
- Expand the Configuration > Tuning tab.
- If necessary, click Advanced to access advanced tuning parameters.
- Enter weblogic.socket.NIOSocketMuxer in the Muxer Class field.
- Click Save.
- To
activate these changes, in the Change Center of the Administration
Console, click Activate Changes.
Not all changes take effect immediately—some require a restart
Configuration:
1)
Checked "Enable Native IO" option
from Admin console.
2)
Added
"-Dweblogic.MuxerClass=weblogic.socket.NIOSocketMuxer" in
setDomainEnv.sh to override the Muxer class.
Count Of Muxer
Threads: 4
Stack Trace from
Thread Dump:
"ExecuteThread:
'3' for queue: 'weblogic.socket.Muxer'" daemon prio=3
tid=0x000000010360f800 nid=0x31 runnable [0xffffffff5c2ff000]
java.lang.Thread.State: RUNNABLE
at
sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at
sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:171)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)
at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0xfffffffe23f121d8> (a
sun.nio.ch.Util$2)
- locked <0xfffffffe23f121c8> (a
java.util.Collections$UnmodifiableSet)
- locked <0xfffffffe23ebc5b8> (a
sun.nio.ch.DevPollSelectorImpl)
at
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
weblogic.socket.NIOSocketMuxer.processSockets(NIOSocketMuxer.java:320)
at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
at
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
at
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
Comment/Remark:
This time since we explicitly gave
Muxerclass name at JVM level it’s using now "NIOSocketMuxer" class for socket reading.Snap from Thread dump -
Refer the below URL for details understanding-
1.3 Error Related to Socket Problems –
In Weblogic .out log file you might see lots of IO Exception
and Socket Exception mentioned below
<Nov 16, 2012 12:42:28 PM EST> <Error>
<WliSbTransports> <BEA-381304> <Exception in
HttpInboundMessageContext.close: java.io.IOException: Broken pipe
java.io.IOException: Broken pipe
at
sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at
sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at
sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
at
sun.nio.ch.IOUtil.write(IOUtil.java:60)
at
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
Truncated. see
log file for complete stacktrace
>
<Dec 6, 2012 1:58:11 AM EST> <Error>
<WliSbTransports> <BEA-381304> <Exception in
HttpOutboundMessageContext.RetrieveHttpResponseWork.run: java.io.IOException:
Connection reset by peer
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native
Method)
at
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
at
sun.nio.ch.IOUtil.read(IOUtil.java:191)
at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
Truncated.
see log file for complete stacktrace
1.4 General Question which rises in Mind in regard to Socket Configuration
Question1: If the
configuration contains checked “Enable Native IO” and “-Dweblogic.SocketReaders=8
-Dweblogic.MuxerClass=weblogic.socket.NIOSocketMuxer”
In that case will weblogic uses native IO or non native IO.
Ans: Above configuration will override default
configuration and will use weblogic.socket.NIOSocketMuxer class which is Non
blocking class been added later to provide improved performance. Refer this doc
for details
http://docs.oracle.com/cd/E24329_01/web.1211/e24390/wls_tuning.htm#i1151673
Question2: Can we use both Native IO and JAVA IO socket
threads together?
Ans: no we can
have only one sort of implementation, though your server configuration can have
two configurations but one must be suppressed by another configuration.
Question3 : If we
use Socket Reader count up to 8 using this configuration “Dweblogic.SocketReaders=8”,
what will happen if we disabled the “Enable Native IO” Option, Will weblogic by
default allow to use 33% of Execute Threads as Socket Reader threads?
Ans: No, again default configuration will be
override, weblogic will use 8 Muxer thread only to do socked read/write.
Question4: Which
one out of Native IO and JAVA IO socket threads we should use in Production
with high traffic volume?
Ans:
In our production previously we were having checked “Enable
Native IO” and specified “NIOMuxerclass” and were facing many problem e.g. OSB
transaction were hanging, were not getting timeout, socket timeout exception
was coming in logs, later point of time as per Oracle recommendation we switch
back to default Weblogic configuration which checked “Enabled Native IO”,
removed NIOMuxer class and after that we witnessed lot more improvement in
Production environment.
Question: What
should be no of socket reader threads?
Ans: We have
tested various configuration in our Test environment and found that weblogic by
default mostly use 4 Muxer threads only regardless of Native or Non Native
Muxer Type. For Java Muxer document says it can use 33% threads from Execute
Thread groups but we realized even with that configuration in 30 TPS load
Weblogic initializing only 4 threads.
2 Summary:
As per my experience I would always suggest to go with
default configuration which comes with Weblogic for Socket Reader Muxer
threads, previously we have modified the configuration and introduced NIO Muxer
class and also increased no of socket Reader threads but it didn’t help for any
performance, even we notice lots of strange behaviour start occurring, later
point of time we switch back to Default Weblogic Configuration and realized improvement.
So go with default configuration. Also when you look at Thread dumps you might
see Muxer Thread for Block state for longer time etc so don't be scared for that,
Muxer threads are designed in such way only, its normal to see Muxer Thread Blocked state till long time.
