APM - WebSphere node(s) failing to start after enabling the JavaAgent
search cancel

APM - WebSphere node(s) failing to start after enabling the JavaAgent

book

Article ID: 93003

calendar_today

Updated On:

Products

CA Application Performance Management Agent (APM / Wily / Introscope) INTROSCOPE

Issue/Introduction

Below examples of some of the error messages reported in the Agent or application server log:

Example 1:
SRVE0232E: Internal Server Error. 
Exception Message: [com.wily.introscope.agent.trace.IMethodTracer]

Example 2:
java.lang.NoClassDefFoundError: com.wily.introscope.agent.blame.VirtualStack$TransactionCache (initialization failure)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:140)
at com.wily.introscope.agent.blame.VirtualStack$VirtualStackCursor.<init>(VirtualStack.java:694)
at com.wily.introscope.agent.blame.VirtualStack$1.initialValue(VirtualStack.java:723)
at com.wily.introscope.agent.blame.VirtualStack$1.initialValue(VirtualStack.java:721)

Example 3:
SystemErr R Exception in thread "Thread-47" java.lang.NoClassDefFoundError: com.wily.introscope.agent.blame.VirtualStack$VirtualStackCursor (initialization failure)
SystemErr R at java.lang.J9VMInternals.initializationAlreadyFailed(J9VMInternals.java:140)
SystemErr R at com.wily.introscope.agent.blame.VirtualStack$1.initialValue(VirtualStack.java:806)
OR
Request processing failed; nested exception is java.lang.NoClassDefFoundError: com/wily/introscope/agent/AgentShim

Example 4:
java.lang.NoSuchMethodError: com/wily/introscope/agent/connection/IsengardServerConnectionManager$LoadBalancerNotificationListener.<init>(Lcom/wily/introscope/agent/connection/IsengardServerConnectionManager;Lcom/wily/introscope/agent/connection/IsengardServerConnectionManager$LoadBalancerNotificationListener;)V

Example 5:
2/22/19 03:03:44 PM CET [ERROR] [IntroscopeAgent.Agent] A problem occurred while attempting to create the delegate agent

Environment

  • All Java Agents
  • IBM Websphere

Cause

By default the IBM JVM creates a shared class cache persisted to the disk in a /tmp directory named as "javasharedresources." This directory typically contains all the core Java API classes, as well as the core WAS classes. This cache is reused by other JVMs started on the same machine and intended to reduce the virtual memory usage with multiple WAS servers running on the same box and also as reduces the startup time for subsequent VMs created, however, since Wily inserts probes into the native/core Java classes (for example, threads, sockets), as well the core WAS classes, this could cause the above issues.

For example, if the java.lang.Thread class is substituted with a call to the ManagedThread class from the Wily code (which is how we get all the thread level information) and this class was persisted to the cache, any other JVM starting on the same machine trying to use the same Thread Class from the shared cache will fail since this core class now has references to the Wily codebase. 

JVMs make use of this shared cache feature through Inter Process Communication (IPC) and shared memory segments.

Resolution

 Use one of the following options to resolve this issue:

-Xshareclasses:none starts the application without using the cache and shared classes. This can be a useful short-term solution to confirm that it is a cache related problem.

-Xshareclasses:reset starts the application, with shared classes, then clears the cache and rebuilds it. This is the recommended solution to fix a problem with the cache.