Option to define a maximum number of threads for all receivers.

Details

  • Type: Improvement Improvement
  • Status: Closed Closed
  • Priority: Minor Minor
  • Resolution: Fixed
  • Affects Version/s: 1.2
  • Fix Version/s: 1.4.0
  • Component/s: Core: Containers
  • Labels:
    None
  • Similar Issues:
    None

Description

I was sending a Mule Message to an http connector. The message was created from a url which had different parameters for each message. For example:

http://127.0.0.1:80/mysite/gateway.cfm?site=icm6100&action=UPDATE&itemtype=Article&itemid=5397
http://127.0.0.1:80/mysite/gateway.cfm?site=icm6100&action=UPDATE&itemtype=Article&itemid=5398
http://127.0.0.1:80/mysite/gateway.cfm?site=icm6100&action=UPDATE&itemtype=Article&itemid=5399

Mule looks at the endpoint and creates a thread pool for each unique url. My url's are different every time so a new thread pool was being created for each message.

The system created more and more threads until it ran out of heap memory.

See the mailing list archive:
http://www.nabble.com/Heap-Memory-Usage%2C-Out-Of-Memory-Error-t961635.html

Ross suggests an option to define a maximum number of threads for all receivers.

Issue Links

relates to

Improvement - An improvement or enhancement to an existing feature or task. MULE-966 Improve receiver/dispatcher state handling for better thread safety

  • Critical - Crashes, loss of data, severe memory leak.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Improvement - An improvement or enhancement to an existing feature or task. MULE-1241 More flexible polling/scheduling support for receivers

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Holger Hoffstaette added a comment -

I think a single pool will be a abtter solution because endpoints that are not being accessed just linger around anyway. Creating too many threads (numCPUs * 4..8) and/or pools usually does not increase performance or responsiveness any more.

Show
Holger Hoffstaette added a comment - I think a single pool will be a abtter solution because endpoints that are not being accessed just linger around anyway. Creating too many threads (numCPUs * 4..8) and/or pools usually does not increase performance or responsiveness any more.
Hide
Permalink
Holger Hoffstaette added a comment -

would be nice to fix this for 1.3.

Show
Holger Hoffstaette added a comment - would be nice to fix this for 1.3.
Hide
Permalink
Holger Hoffstaette added a comment -

Looks to be almost the same as MULE-966.

Show
Holger Hoffstaette added a comment - Looks to be almost the same as MULE-966.
Hide
Permalink
Holger Hoffstaette added a comment -

This turns out to be a bit more difficult than originally estimated. Previously all receivers for a connector had their own pools; those were unified to a single pool. This turned out to be a problem because if the number of endpoints (receivers) exceeded the max. pool size (which is always bounded to prevent runaway), the entire startup would stall/block because PollingMessageReceivers are using individual threads for waiting, blocking others from even being started. This can be demostrated reliably by starting the VMLoanBrokerSynchronousTestCase; by upping the receiver pool size, everything immediately starts to work.

The "correct" way to fix this would be to use a ScheduledExecutor specifically for PollingReceivers. Currently a receiver submits itself to a WorkManager and then runs(), never relinquishing the thread until exit. This would need to be revised to let the ScheduledExecutor do the scheduling and only submit a single periodic poll() when necessary.

Show
Holger Hoffstaette added a comment - This turns out to be a bit more difficult than originally estimated. Previously all receivers for a connector had their own pools; those were unified to a single pool. This turned out to be a problem because if the number of endpoints (receivers) exceeded the max. pool size (which is always bounded to prevent runaway), the entire startup would stall/block because PollingMessageReceivers are using individual threads for waiting, blocking others from even being started. This can be demostrated reliably by starting the VMLoanBrokerSynchronousTestCase; by upping the receiver pool size, everything immediately starts to work. The "correct" way to fix this would be to use a ScheduledExecutor specifically for PollingReceivers. Currently a receiver submits itself to a WorkManager and then runs(), never relinquishing the thread until exit. This would need to be revised to let the ScheduledExecutor do the scheduling and only submit a single periodic poll() when necessary.
Hide
Permalink
Holger Hoffstaette added a comment -

A plain ScheduledExecutor in itself might not be enough since that would not use a provided container WorkManager. java.util.Timer is not subclassable or customizable so we'd have at least one free-floating scheduler Thread per connector, submitting items to the WorkManager; if this is acceptable it is probably the easiest thing to do.
Otherwise a somewhat overridden ScheduledExecutor which itself runs in a WorkManager-managed thread and delegates back to its parent might be required.

Show
Holger Hoffstaette added a comment - A plain ScheduledExecutor in itself might not be enough since that would not use a provided container WorkManager. java.util.Timer is not subclassable or customizable so we'd have at least one free-floating scheduler Thread per connector, submitting items to the WorkManager; if this is acceptable it is probably the easiest thing to do. Otherwise a somewhat overridden ScheduledExecutor which itself runs in a WorkManager-managed thread and delegates back to its parent might be required.
Hide
Permalink
Holger Hoffstaette added a comment -

I have a per-connector Scheduler service ready and working for 1.4 but it does not solve the original problem of this issue. Socket-based receivers like tcp or http need to continuously run in their own thread since they hang in accept() waiting for connections; unless we get NIO support this is not easy to fix.
One possibility would be to set the accept timeout (via ServerSocket.setSoTimeout) and treat accepting receivers like periodically scheduled polling.

Show
Holger Hoffstaette added a comment - I have a per-connector Scheduler service ready and working for 1.4 but it does not solve the original problem of this issue. Socket-based receivers like tcp or http need to continuously run in their own thread since they hang in accept() waiting for connections; unless we get NIO support this is not easy to fix. One possibility would be to set the accept timeout (via ServerSocket.setSoTimeout) and treat accepting receivers like periodically scheduled polling.
Hide
Permalink
Holger Hoffstaette added a comment -

Due to the related rework of the transports all receivers now share a single pool.

Show
Holger Hoffstaette added a comment - Due to the related rework of the transports all receivers now share a single pool.
Hide
Permalink
Holger Hoffstaette added a comment -

reopen for fix-version

Show
Holger Hoffstaette added a comment - reopen for fix-version

People

Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: