We Keep Coding

Is it possible to instrument a program that also uses dynamic bytecode generation?

I am writing a Java instrumentation program that uses the built-in Instrumentation API with Javassist (v3.26.0-GA) to intercept all the method calls in the target program. Also, I have implemented a REST API service inside this program using Java Spark to send requests for starting/stopping instrumentation by adding/removing transformers, and also for fetching intercepted methods during the instrumentation time.
Now, while I was trying to run WebGoat (an open source Spring Boot application) with my Java agent attached from premain, I was not able to intercept all the methods successfully and in the log, there was a NotFoundException being thrown by Javassist.
This error happened for several classes in WebGoat all had a similar common fact that they had something to do with SpringCGLIB. A few of the errors are shown below.
javassist.NotFoundException: org.owasp.webgoat.hijacksession.cas.HijackSessionAuthenticationProvider$$FastClassBySpringCGLIB$$88f1f22d
at javassist.ClassPool.get(ClassPool.java:430)
at com.sparrow.sptracer.core.transformer.AbstractMethodTransformer.transform(AbstractMethodTransformer.java:87)
at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:244)
at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:541)
at java.base/java.lang.ClassLoader.defineClass0(Native Method)
at java.base/java.lang.System$2.defineClass(System.java:2307)
at java.base/java.lang.invoke.MethodHandles$Lookup$ClassDefiner.defineClass(MethodHandles.java:2439)
at java.base/java.lang.invoke.MethodHandles$Lookup$ClassDefiner.defineClass(MethodHandles.java:2416)
at java.base/java.lang.invoke.MethodHandles$Lookup.defineClass(MethodHandles.java:1843)
at java.base/jdk.internal.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.cglib.core.ReflectUtils.defineClass(ReflectUtils.java:507)
at org.springframework.cglib.core.AbstractClassGenerator.generate(AbstractClassGenerator.java:363)
at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:110)
at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:108)
at org.springframework.cglib.core.internal.LoadingCache$2.call(LoadingCache.java:54)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.springframework.cglib.core.internal.LoadingCache.createEntry(LoadingCache.java:61)
at org.springframework.cglib.core.internal.LoadingCache.get(LoadingCache.java:34)
at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData.get(AbstractClassGenerator.java:134)
at org.springframework.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:319)
at org.springframework.cglib.reflect.FastClass$Generator.create(FastClass.java:65)
at org.springframework.cglib.proxy.MethodProxy.helper(MethodProxy.java:135)
at org.springframework.cglib.proxy.MethodProxy.init(MethodProxy.java:76)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:216)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
at org.owasp.webgoat.hijacksession.cas.HijackSessionAuthenticationProvider$$EnhancerBySpringCGLIB$$6ae99c75.authenticate(<generated>)
at org.owasp.webgoat.hijacksession.HijackSessionAssignment.login(HijackSessionAssignment.java:72)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:517)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:219)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:213)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:501)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:833)
javassist.NotFoundException: org.owasp.webgoat.hijacksession.cas.HijackSessionAuthenticationProvider$$EnhancerBySpringCGLIB$$6ae99c75$$FastClassBySpringCGLIB$$8c045873
at javassist.ClassPool.get(ClassPool.java:430)
at com.sparrow.sptracer.core.transformer.AbstractMethodTransformer.transform(AbstractMethodTransformer.java:87)
at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:244)
at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:541)
at java.base/java.lang.ClassLoader.defineClass0(Native Method)
at java.base/java.lang.System$2.defineClass(System.java:2307)
... similar stacktrace as above
javassist.NotFoundException: org.owasp.webgoat.session.UserSessionData$$FastClassBySpringCGLIB$$2b6b54bc
at javassist.ClassPool.get(ClassPool.java:430)
at com.sparrow.sptracer.core.transformer.AbstractMethodTransformer.transform(AbstractMethodTransformer.java:87)
at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:244)
at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:541)
at java.base/java.lang.ClassLoader.defineClass0(Native Method)
at java.base/java.lang.System$2.defineClass(System.java:2307)
... similar stacktrace as above
[IMPACTRACER-CORE] ERROR [2021-12-24 10:34:20]: NotFoundException on class 'org/owasp/webgoat/session/UserSessionData$$EnhancerBySpringCGLIB$$bbb61fe2$$FastClassBySpringCGLIB$$34cb52d3': org.owasp.webgoat.session.UserSessionData$$EnhancerBySpringCGLIB$$bbb61fe2$$FastClassBySpringCGLIB$$34cb52d3
javassist.NotFoundException: org.owasp.webgoat.session.UserSessionData$$EnhancerBySpringCGLIB$$bbb61fe2$$FastClassBySpringCGLIB$$34cb52d3
at javassist.ClassPool.get(ClassPool.java:430)
at com.sparrow.sptracer.core.transformer.AbstractMethodTransformer.transform(AbstractMethodTransformer.java:87)
at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:244)
at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
... similar stacktrace as above
My guess is that WebGoat uses cglib in its Spring Boot environment to dynamically generate bytecode and the corresponding class is not included in the WebGoat's classpath originally, so javassist.ClassPool.get(className) is throwing the error.
My code for creating ClassPool object is as below.
String name = Descriptor.toJavaName(className);
try {
ClassPool cp = ClassPool.getDefault();
cp.childFirstLookup = true;
cp.appendClassPath(new LoaderClassPath(loader));
CtClass cc = cp.get(name);
Logger.debug("Checking class %s", name);
CtMethod[] methods = cc.getDeclaredMethods();
Logger.debug("Altering %d methods in %s", methods.length, wut);
for (CtMethod m : methods) {
// do some code insertion
}
bytecode = cc.toBytecode();
cc.detach();
} catch (NotFoundException e) {
Logger.error("NotFoundException on class '%s': %s", className, e.getMessage());
e.printStackTrace();
} catch (CannotCompileException e) {
Logger.error("Cannot compile class '%s': %s", className, e.getMessage());
e.printStackTrace(System.out);
} catch (IOException e) {
Logger.error("IOException while transforming class '%s': %s", className, e.getMessage());
} catch (Exception ex) {
Logger.error("Generic exception occurred while transforming class '%s': %s", className, ex.getMessage());
ex.printStackTrace(System.out);
}
The above errors occured when I tried interacting with the WebGoat application on localhost:8080 tried to send a "start istrumentation" request with a specific id using Java Spark REST API, so whenever I send this request, it triggers inst.addTransformer method to create a new Transformer and adds it to the instrumentation object.
What could be the cause of this error? I assume that WebGoat itself is using some instrumentation which means I am instrumenting an instrumented application and I don't know if that is even possible.
Any insight would be much appreciated.

From previous comments:
The unfound classes are dynamic proxies which are heavily used by the Spring Framework in order to implement AOP. Spring can use both JDK dynamic interface proxies and CGLIB proxies, the latter of which is what we are seeing here. Maybe you should simply ignore those types of classes. They are in fact created dynamically, hence the name. But they are rather a result of dynamic (sub-)class generation than of bytecode transformation.
Yes, I have considered just ignoring those dynamically generated classes, but the whole point of my application was to capture every single method invocation as a user interacts with the web application (such as clicking on a button, etc). In this case, would it be okay to ignore these types of dynamically generated classes? I want to make sure I do not miss any method calls.
As those classes are just dynamic proxies, they will either forward the calls to the original methods or call some AOP or interceptor logic first/instead. Either way, you would not miss anything essential, those proxies are more like switchboards or routers, the actual show happens somewhere else. I recommend you to simply try in a little playgrounds project with an aspect or two.
You also asked how to detect and ignore dynamic proxies by their names:
CGLIB proxies: Spring's CGLIB proxies contain substrings like $$FastClassBySpringCGLIB$$ or $$EnhancerBySpringCGLIB$$, followed by 8 characters representing 4 hexadecimal bytes. You could either match with a regular expression of just keep it simple and match the substring BySpringCGLIB$$. If non-Spring CGLIB proxies are also in use somewhere in your application, you would have to watch for other naming patterns. But probably you would get similar errors as before when not filtering them, so you would notice automatically.
JDK proxies: If your Spring application also happens to use JDK proxies, you can identify them easily using JRE API call Proxy.isProxyClass(Class). Thanks to Johannes Kuhn for his comment.
JDK proxies (old answer): You can filter class names beginning with $Proxy, usually something like com.sun.proxy.$Proxy2 (the trailing number being different). According to the JDK documentation: "The unqualified name of a proxy class is unspecified. The space of class names that begin with the string "$Proxy" is, however, to be reserved for proxy classes." At least for Oracle and probably OpenJDK, you can match for that naming pattern. If that holds true for all JVMs, is up to you to test, if chances are that in your environments others are being used. I quickly tried with Semeru OpenJ9, and the proxy naming pattern is identical, even the package name com.sun.proxy. Pleasae note that in more recent JDK versions, JDK proxies will have fully qualified names like jdk.proxy2.$Proxy25, so in e.g. Java 16 or 17 you should not rely on package name com.sun.proxy. Either add more cases or limit matching to the leading $Proxy in the simple class name.
Update 2022-02-26: Because there was activity on this question, I decided to add some more information about Spring-specific tools which can determine whether an object (or a class) is an AOP proxy (class) and, more specifically, if it is a CGLIB or JDK proxy:
Take a look at tool class AopUtils and its handy methods
isAopProxy(Object),
isCglibProxy(Object),
isJdkDynamicProxy(Object).
No more String matching, simply ask Spring.
BTW, there is also a method net.sf.cglib.proxy.Proxy.isProxyClass(Class) directly in CGLIB, which is supposed to do the same, but within Spring it does not work, probably because Spring uses CGLIB in a non-canonical way. Because Spring embeds a package-relocated CGLIB in its core, the corresponding method org.springframework.cglib.proxy.Proxy.isProxyClass(Class) yields the same faulty result. So if you are working within Spring, please do not use those methods, better use AopUtils.
Here is some example code for your convenience, showing how to determine Spring AOP proxy types (JDK vs. CGLIB proxies) using AopUtils. See also my answer here for how to configure Spring in order to use both proxy types.
BTW, instead of Javassist you could also use AspectJ for your purpose. It sounds like a pretty typical use case.

Solr custom UpdateRequestProcessorFactory fails with "Error Instantiating UpdateRequestProcessorFactory"

I have a custom class extending UpdateRequestProcessorFactory doing some work on a document when it gets added to the index. This was working fine in v4.10.3 in standalone Solr. I moved to SolrCloud v5.2 and it throws this error when adding the Collection (node):
ERROR - 2015-06-14 12:25:11.071; [ docs_shard1_replica1] org.apache.solr.common.SolrException; org.apache.solr.common.SolrException: Error CREATEing SolrCore 'docs_shard1_replica1': Unable to create core [docs_shard1_replica1] Caused by: class com.example.solr.update.processor.SelfTaggerUpdateProcessorFactory
at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:661)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:213)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:193)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:646)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:417)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unable to create core [docs_shard1_replica1]
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:651)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:611)
at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:628)
... 27 more
Caused by: org.apache.solr.common.SolrException: Error Instantiating UpdateRequestProcessorFactory, com.example.solr.update.processor.SelfTaggerUpdateProcessorFactory failed to instantiate org.apache.solr.update.processor.UpdateRequestProcessorFactory
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:815)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:658)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:637)
... 29 more
Caused by: org.apache.solr.common.SolrException: Error Instantiating UpdateRequestProcessorFactory, com.example.solr.update.processor.SelfTaggerUpdateProcessorFactory failed to instantiate org.apache.solr.update.processor.UpdateRequestProcessorFactory
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:587)
at org.apache.solr.core.SolrCore.createInitInstance(SolrCore.java:622)
at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2281)
at org.apache.solr.update.processor.UpdateRequestProcessorChain.init(UpdateRequestProcessorChain.java:126)
at org.apache.solr.core.SolrCore.createInitInstance(SolrCore.java:624)
at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2265)
at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2259)
at org.apache.solr.core.SolrCore.loadUpdateProcessorChains(SolrCore.java:1069)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:766)
... 31 more
Caused by: java.lang.ClassCastException: class com.example.solr.update.processor.SelfTaggerUpdateProcessorFactory
at java.lang.Class.asSubclass(Class.java:3208)
at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:475)
at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:422)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:566)
... 39 more
In solrconfig.xml I have configured like this:
<updateRequestProcessorChain>
<processor class="com.example.solr.update.processor.SelfTaggerUpdateProcessorFactory" lib="custom-libs" version="1">
<arr name="source">
<str>title</str>
<str>desc</str>
<str>subject</str>
<str>content</str>
</arr>
<str name="dest">category</str>
</processor>
...
</updateRequestProcessorChain>
I have tried to add my custom jar in two ways:
Add it to the server/lib/ folder where all the Jetty jars are located. This fails with Error Instantiating UpdateRequestProcessorFactory
Add it to the .system collection as suggested and then make it available to the <processor /> using the lib and version attributes, which fails with a Caused by: java.lang.ClassNotFoundException: com.example.solr.update.processor.SelfTaggerUpdateProcessorFactory
Does this Error Instantiating UpdateRequestProcessorFactory exception tell something I don't see? Does the move from standalone to Solrcloud require something for my custom code I missed?

I think the issue is moving from standalone Solr to a Cloud one. Specifically, about where you store your jar file with the custom component. It is probably not visible to all the nodes and therefore fails.
However, Solr 5.2 does have new functionality related to that you may want to use. It allows to use the Config API to add the library, assuming you enable that functionality.

The issue is a classloader issue. I had added my custom jar into the server/lib/ folder.
When I added the Collection, it would instantiate my custom class which needs the UpdateRequestProcessorFactory class but is not available in that classloader.
I solved this by rmoving my jar from server/lib/ and adding this line to solrconfig.xml:
<lib dir="${solr.install.dir:../../..}/custom-libs/" regex="solr-.*\.jar" />
and putting my custom jar in a custom-libs/ folder in ${SOLR_HOME}. This way I make sure that my custom classes are only loaded when the Collection is added, and not before, so they are in the correct classloader.

SpringBoot webapp under Java SecurityManager throws exceptions when granted AllPermission

We have a Spring Boot (1.1.6) application deployed as a .war file to a Tomcat 7.0.52 / OpenJDK 7 server. We need to run this server under a SecurityManager.
Even when we operate the server with a policy that allows AllPermission to all code (essentially the same as not running under a SecurityManager) we get exceptions thrown due to reflection activity. Here's a full stack trace of one of the exceptions.
java.lang.IllegalAccessException: Class org.apache.catalina.security.SecurityUtil$1 can not access a member of class org.springframework.boot.context.web.ErrorPageFilter with modifiers "public"
at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:109) ~[na:1.7.0_65]
at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:261) ~[na:1.7.0_65]
at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:253) ~[na:1.7.0_65]
at java.lang.reflect.Method.invoke(Method.java:599) ~[na:1.7.0_65]
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:277) ~[catalina.jar:7.0.52]
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:274) ~[catalina.jar:7.0.52]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_65]
at javax.security.auth.Subject.doAsPrivileged(Subject.java:536) ~[na:1.7.0_65]
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309) ~[catalina.jar:7.0.52]
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:249) ~[catalina.jar:7.0.52]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237) ~[catalina.jar:7.0.52]
at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55) ~[catalina.jar:7.0.52]
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191) ~[catalina.jar:7.0.52]
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187) ~[catalina.jar:7.0.52]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_65]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186) ~[catalina.jar:7.0.52]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) ~[catalina.jar:7.0.52]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) [catalina.jar:7.0.52]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) [catalina.jar:7.0.52]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) [catalina.jar:7.0.52]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) [catalina.jar:7.0.52]
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) [catalina.jar:7.0.52]
at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:178) [logback-access-1.0.13.jar:na]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) [catalina.jar:7.0.52]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) [catalina.jar:7.0.52]
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) [tomcat-coyote.jar:7.0.52]
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) [tomcat-coyote.jar:7.0.52]
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313) [tomcat-coyote.jar:7.0.52]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
The policy file which is applied to the server is shown below:
grant {
permission java.security.AllPermission;
permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
};
.. we added the explicit ReflectPermission just in case AllPermission didn't include it but we tried it both ways.
Any pointers as to why the exception gets thrown only when we run under a SecurityManager with the above policy?

The error is caused not by Java Security, but by Java Language access rules.
The ErrorPageFilter class is package-visible [1]. Tomcat tries to call public method of that class via reflection, but cannot because by Java Language rules the class itself is not accessible.
[1] https://github.com/spring-projects/spring-boot/blob/master/spring-boot/src/main/java/org/springframework/boot/context/web/ErrorPageFilter.java
(I am linking to master branch, but you are actually using version 1.1.6, so content of the above link may change over time)
A non-public Filter is unusual. In old times when all filters were declared in web.xml it was not possible to use such non-public filters. I do not know Spring Boot internals, but I guess that this filter was added to the web application programmatically, using Servlet 3.0 APIs.
Possible ways to proceed:
a) Ask Spring Boot developers to make the class public
b) Change Apache Tomcat so that instead of looking for method declared by specific class it were looking for method declared by specific interface. A java.lang.reflect.Method declared by interface can be invoked on a class implementing that interface. I expect that this solution will pass those checks in sun.reflect.Reflection.ensureMemberAccess(), but actual testing is needed.
This needs changing internal Tomcat APIs to pass the interface class (in this case, javax.servlet.Filter) as an additional argument to those methods.
Technically, this needs:
File an issue into Tomcat Bugzilla - DONE.
Provide a simple reproduction recipe, so that this can be tested
Wait for the next release
Update: I filed this into Bugzilla,
https://issues.apache.org/bugzilla/show_bug.cgi?id=57281

SpringMvc Instantiating Beans - Singleton

In the under mentioned code i created this because i was getting java.lang.NullPointerException see my error log under. When i implemented this it was fixed and it worked perfectly. However i was told by one of the members here that -
"you are creating new beans for every call to getOfficerRegistrationValidation. this includes recreating the database objects which are every expensive and can cost error if some object are supposed to be created only once (singleton). you should find a way to make OfficerRegistrationValidation reusable and threadsafe so that you can have only one of it and inject this to wherever you want using only IoC"
and i think hes right can someone tell me whats the best way to do this without having to instantiate all my beans again. I just want to instantiate only the certain beans and not run into this problem.
public final class BeanFactory() {
private static ClassPathXmlApplicationContext context;
private static ClassPathXmlApplicationContext getContext() {
if (context== null) {
context = new ClassPathXmlApplicationContext("applicationContext.xml");
}
return context;
}
public static OfficerRegistrationValidation getOfficerRegistrationValidation() {
return (OfficerRegistrationValidation) getContext().getBean("officerRegistrationValidation");
}
}
In the controller make a call like : `BeanFactory.getOfficerRegistrationValidation().validate(....)
Error Log:
java.lang.NullPointerException
at com.crimetrack.service.OfficerRegistrationValidation.validate(OfficerRegistrationValidation.java:51)
at org.springframework.validation.DataBinder.validate(DataBinder.java:725)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.doBind(HandlerMethodInvoker.java:815)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.resolveHandlerArguments(HandlerMethodInvoker.java:367)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:171)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Unknown Source)

I'm going to hazard a guess at what you are asking about (as your question doesn't provide your motivation for using a beanfactory and doesn't provide enough code to give a complete picture)...
Spring beans are by default instantiated as singletons. So you should be able to configure Spring to create your OfficerRegistrationValidation object using either XML or annotation (#Component) and then get it injected into your controller by using the #Inject annotation.
Annotations are the way to go - just make sure you have the following in your Spring config XML (applicationContext.xml):
<context:component-scan base-package="com.acme.project" />
<mvc:annotation-driven />
Then simply add the #Component annotation to the class declaration of your OfficerRegistrationValidation class:
#Component
public class OfficerRegistrationValidation {
// ...
}
and then in your controller, declare a field for your OfficerRegistrationValidation and annotate it thus:
#Controller
public class MyController {
#Inject
private OfficerRegistrationValidation officerRegistrationValidation;
// ...
}
Now you will have access to the OfficerRegistrationValidation singleton bean in your controller - only ~4 lines of code needed!

java.lang.IllegalArgumentException: setAttribute: Non-serializable attribute

I have inherited some code and am getting an error when I try to run it. The error is below:
10:08:32,093 ERROR [MyServlet]:260 - Servlet.service() for servlet MyServlet threw exception
java.lang.IllegalArgumentException: setAttribute: Non-serializable attribute
at org.apache.catalina.session.StandardSession.setAttribute(StandardSession.java:1270)
at org.apache.catalina.session.StandardSession.setAttribute(StandardSession.java:1248)
at org.apache.catalina.session.StandardSessionFacade.setAttribute(StandardSessionFacade.java:130)
at uk.co.my.servlet.MyServlet.doPost(MyServlet.java:121)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at com.my.ddi.security.SecurityContextServletFilter.doFilter(SecurityContextServletFilter.java:55)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:835)
at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:640)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1286)
at java.lang.Thread.run(Unknown Source)
I imagine that the main problem is where it is hitting my code at uk.co.my.servlet.MyServlet.doPost(MyServlet.java:172). Line 121 of MyServlet is
session.setAttribute("LISTENER", myListener);
This myListener object is from the following class:
public class myListener {
static Log log = LogFactory.getLog(TerminationListener.class.getName());
private boolean shouldCancel = false;
How would I go about getting rid of this error? Would I need to use the transient keyword?

If you want an instance of myListener to be placed in a session, you need to make it serializable. That might involve making some fields transient, but then those fields won't be saved in the session. You'll also need to make the class implement Serializable.
I suggest you look at the documentation for Serializable, which goes into a fair amount of detail about serialization.