I recently ran into a rather worrisome situation where my Admin Server suddenly lost connection with my SOA Managed Server.  Luckily my users were blissfully unaffected and looked at me strangely when I expressed my alarm that Enterprise Manager couldn’t see any of the composites.  They said they’d never met the Enterprise Manager and that they weren’t aware the company made composites.  At that point, sadly, I realized I was on my own.

When I looked at an old screenshot I had of a normal Enterprise Manager from the same 11g environment, I found that not only could I not see my composites, but I found that it claimed the entire soa_cluster was down – which it clearly was not since the users were happily motoring along.

I realized at this point that a mystery was afoot.  Although part of me suspected foul play, I decided to investigate further before pointing any bony fingers of blame.  Since Enterprise Manager was clearly behaving badly, I logged into the WebLogic Console and clicked on the Servers tab.  My suspicions grew deeper when I saw that the health check for soa_server1 was …. blank. (queue dramatic music)

With much trepidation, I clicked into the link for soa_server_1 and switched on my flashlight as I crept from tab to tab until I came upon one that froze my blood – the Monitoring tab.  Within it, I found a small, cryptic note that read “The server is not currently reachable.”  What could it mean, I asked myself? 

There was only one place left to go posthaste … the logs.  It was in the AdminServer.out log where I found the villain: MaxMessageSizeExceededException.  He was sinister and sly, cutting off my ability to see my managed servers but luckily now that I saw him for what he was I knew immediately what to do.

In both the Admin server and the soa_server_1 managed server (as well as all other managed servers in your domain), you will need to increase the message size.  Before doing this, be aware that this size limit is in place according to Oracle to “guard against a denial of service attack in which a caller attempts to force the server to allocate more memory than is available thereby keeping the server from responding quickly to other requests.”  Therefore, it is probably not advisable to simply set this value to the maximum and hope for the best.

To set this value, you will need to visit 2 places in any server you update.  I will review these steps for soa_server_1 and I leave it to you, faithful reader, to complete these steps as well for the Admin Server and your other managed servers … since those steps are exactly the same for each server.  You must also forgive me, but I am going to spell out these instructions a bit for those in my audience who are not as familiar with WebLogic as I know you are. 

First, of course, we must click on the Lock & Edit button in the upper left corner of the screen and then go back to our Servers tab and click soa_server_1.

Now we must go to the Protocols Tab.  Stay on the General Sub Tab.  On this page, increase your Maximum Message Size.  In this example, I’m changing it from “10000000” to “20000000”.  Be sure to click Save … or all is for naught.

As a quick aside:

• Minimum = 4096 bytes

• Default value = 10,000,000 bytes

• Maximum = 2,000,000,000 bytes

Our next and last stop is to the Configuration Tab.  Then find the Server Start Sub Tab.  Inside this, you will need to update your JVM Arguments with the following that matches the message size you updated in the Protocol tab.  You need to set this value here as well so that it doesn’t override the other one at run time.

In this example I am updating my Arguments with the following: “-Dweblogic.MaxMessageSize=20000000”.  As always, please be sure to click Save (unless you like doing things over and over and over).

As we discussed earlier, be sure to repeat these steps not only for the AdminServer but also for any other server that might have fallen victim to our nefarious foe.

Once you have completed all of this you must be sure to click “Activate Changes” in the upper left corner of your screen … or none of your changes will be activated.

And so, dear faithful reader, the solution to this most vexing problem was … in a word … elementary.  In order to fix the MaxMessageSizeExceededException, we simply had to increase the Maximum Message Size.  The oversized messages were responsible for breaking the connection between the Admin Server and the Managed Server.

While this solution should immediately fix your issue and reestablish communication between the admin server and the managed servers, you might want to research which message(s) grew and why by using a tcpdump.  To be on the safe side, I would further recommend applying this fix to ALL managed servers in your domain and restarting the admin and managed servers after making these changes.  For additional information, please refer to Oracle Tuning WebLogic JMS Doc.

In upcoming blogs, I’ll discuss other puzzling predicaments that I have found myself in which typically include BPM and SOA process patterns, software development best practices and development methodologies.