I am presently developing a Java 11 Spring boot application.
Within a CompletableFuture I am firing a load of REST requests to a webservice. Only one of them (randomly it seems) results in a TimeoutException though it is followed by a successful status 200 OK received response in the logging. Following the (anonymized) log entries:
[Thread-6] INFO n.d.b.a.b.i.RESTInterceptor - Request : <url value>?code=abcd&begindatum=2018-10-01&einddatum=2018-10-02
[http-nio-8080-exec-1] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [/xxx] threw exception [Request processing failed; nested exception is java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException] with root cause
java.util.concurrent.TimeoutException: null
at java.base/java.util.concurrent.CompletableFuture$Timeout.run(CompletableFuture.java:2792)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
[Thread-6] DEBUG n.d.b.a.b.i.RESTInterceptor - Response statuscode : 200 OK
[Thread-6] DEBUG n.d.b.a.b.i.RESTInterceptor - Response headers : [X-Application-Context:"yyyy", Cache-Control:"no-cache, no-store, max-age=0, must-revalidate", Pragma:"no-cache", Expires:"0", Server:"CD", X-Powered-By:"Unknown", Content-Type:"application/json;charset=UTF-8", Date:"Tue, 06 Apr 2021 15:23:16 GMT", Connection:"close", X-Frame-Options:"SAMEORIGIN", X-XSS-Protection:"1;mode=block", X-Content-Type-Options:"nosniff"]
[Thread-6] INFO n.d.b.a.b.i.RESTInterceptor - Response : [{"begindatum":"1993-01-01","id":"aaaaa","vanInstellingserkenningId":"bbbbb","naarInstellingserkenningId":"ccccc","soort":"HIERARCHISCH","naarInstellingserkenning":{"id":"ddddd","type":"ERKENDE_ONDERWIJSINSTELLING","code":"abcd"},"vanInstellingserkenning":{"id":"kkkk","type":"BEVOEGD_GEZAG","code":"12345"}}]
Is there someone having the same experience? I cannot see any difference in the successfully completed requests and the 'failing' one.
After deeper analysis of the problem I finally found out that the TimeoutException is thrown from another Future's handling, elsewhere in the application. The confusion I had that the handling of futures (asynchronous tasks, each in their own Thread) is not what it looks like in a log-file. The shown TimeoutException in the log-snippet is not thrown in the request-response interaction that is shown.
Sorry for the confusion, but maybe others may be helped by this answer.
Related
I'm trying to improve the error handling in a controller by using the #Error annotation.
#Get(uri = "/{id}")
#Secured("ROLE_VIEW")
HttpResponse<AuthorResource> show(Long id) {
try {
ok(authorService.get(id).get())
} catch (Exception e) {
e.printStackTrace()
throw e
}
}
The get() method will either return the AuthorResource (io.vavr.control.Try), or throw a custom exception, subclass of our base ErrorContext exception class.
I then added the following errorhandler in the controller:
#Error(exception = ErrorContext, global = true)
HttpResponse<ErrorContext> onErrorContext(HttpRequest request, ErrorContext error) {
HttpResponse.<ErrorContext>status(HttpStatus.valueOf(error.code)).body(error)
}
Unfortunately, the exception handler does not seem to be invoked, and an INTERNAL_SERVER_ERROR is triggered (instead of a NOT_FOUND in this example). The output of the gradle test task can be found below.
I have tried the following:
Using a global handler (global=true)
Annotating the handler with the actual subclass that is being thrown (ErrorContextWithoutStacktrace)
Am I missing anything?
I have created a working application in github here
Controller class
Test class
Output of test (including partial stacktraces due to length limitation in SO):
10:50:31.455 [ForkJoinPool.commonPool-worker-7] INFO i.m.t.e.TestResourcesResolverLoader - Loaded 1 test resources resolvers: io.micronaut.testresources.testcontainers.GenericTestContainerProvider
10:50:31.579 [main] INFO i.m.testresources.server.Application - A Micronaut Test Resources server is listening on port 43207, started in 182ms
10:50:33.164 [default-nioEventLoopGroup-1-2] INFO i.m.t.e.TestResourcesResolverLoader - Loaded 1 test resources resolvers: io.micronaut.testresources.testcontainers.GenericTestContainerProvider
> Task :test
AuthorControllerSpec STANDARD_OUT
10:50:32.581 [Test worker] INFO i.m.context.env.DefaultEnvironment - Established active environments: [test]
10:50:33.131 [multithreadEventLoopGroup-1-1] DEBUG i.m.h.client.netty.DefaultHttpClient - Sending HTTP GET to http://localhost:43207/requirements/entries
10:50:33.190 [multithreadEventLoopGroup-1-1] DEBUG i.m.h.client.netty.DefaultHttpClient - Received response 200 from http://localhost:43207/requirements/entries
10:50:33.216 [multithreadEventLoopGroup-1-2] DEBUG i.m.h.client.netty.DefaultHttpClient - Sending HTTP POST to http://localhost:43207/list
10:50:33.246 [multithreadEventLoopGroup-1-2] DEBUG i.m.h.client.netty.DefaultHttpClient - Received response 200 from http://localhost:43207/list
10:50:33.249 [multithreadEventLoopGroup-1-3] DEBUG i.m.h.client.netty.DefaultHttpClient - Sending HTTP GET to http://localhost:43207/requirements/entries
10:50:33.254 [multithreadEventLoopGroup-1-3] DEBUG i.m.h.client.netty.DefaultHttpClient - Received response 200 from http://localhost:43207/requirements/entries
10:50:33.258 [multithreadEventLoopGroup-1-4] DEBUG i.m.h.client.netty.DefaultHttpClient - Sending HTTP POST to http://localhost:43207/list
10:50:33.262 [multithreadEventLoopGroup-1-4] DEBUG i.m.h.client.netty.DefaultHttpClient - Received response 200 from http://localhost:43207/list
10:50:33.367 [Test worker] INFO i.m.c.h.g.HibernateDatastoreFactory - Starting GORM for Hibernate
10:50:33.883 [Test worker] INFO org.hibernate.Version - HHH000412: Hibernate ORM core version 5.6.11.Final
10:50:33.984 [Test worker] INFO o.h.validator.internal.util.Version - HV000001: Hibernate Validator 6.2.5.Final
10:50:34.074 [Test worker] INFO o.h.annotations.common.Version - HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
10:50:34.139 [Test worker] INFO org.hibernate.dialect.Dialect - HHH000400: Using dialect: org.hibernate.dialect.H2Dialect
AuthorControllerSpec > It fails to get an non-existing Author > example.controller.AuthorControllerSpec.It fails to get an non-existing Author [badId: 546252431, #0] STANDARD_OUT
10:50:34.879 [default-nioEventLoopGroup-3-2] DEBUG i.m.h.client.netty.DefaultHttpClient - Sending HTTP GET to http://localhost:37361/author/546252431
AuthorControllerSpec > It fails to get an non-existing Author > example.controller.AuthorControllerSpec.It fails to get an non-existing Author [badId: 546252431, #0] STANDARD_ERROR
error.ErrorContextWithoutStacktrace: Author with id 546252431 not found.
AuthorControllerSpec > It fails to get an non-existing Author > example.controller.AuthorControllerSpec.It fails to get an non-existing Author [badId: 546252431, #0] STANDARD_OUT
10:50:34.999 [default-nioEventLoopGroup-3-3] ERROR i.m.http.server.RouteExecutor - Unexpected error occurred: Required argument [HttpRequest request] not specified
io.micronaut.web.router.exceptions.UnsatisfiedRouteException: Required argument [HttpRequest request] not specified
at io.micronaut.web.router.exceptions.UnsatisfiedRouteException.create(UnsatisfiedRouteException.java:76)
at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:297)
at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111)
at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103)
at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659)
at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49)
[Skipped a lot of reactive frames]
at reactor.core.publisher.Flux.subscribe(Flux.java:8522)
at io.micronaut.http.server.netty.RoutingInBoundHandler.handleRouteMatch(RoutingInBoundHandler.java:601)
at io.micronaut.http.server.netty.RoutingInBoundHandler.channelRead0(RoutingInBoundHandler.java:457)
at io.micronaut.http.server.netty.RoutingInBoundHandler.channelRead0(RoutingInBoundHandler.java:147)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
[Skipped a lot of netty channel stuff]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
10:50:35.031 [default-nioEventLoopGroup-3-2] DEBUG i.m.h.client.netty.DefaultHttpClient - Received response 500 from http://localhost:37361/author/546252431
10:50:35.049 [Test worker] ERROR i.m.r.intercept.RecoveryInterceptor - Type [example.controller.Api$Intercepted] executed with error: error.ErrorResource#475fb7
io.micronaut.http.client.exceptions.HttpClientResponseException: error.ErrorResource#475fb7
at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.makeErrorFromRequestBody(DefaultHttpClient.java:2226)
at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.buildResponse(DefaultHttpClient.java:2199)
at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.buildResponse(DefaultHttpClient.java:2122)
at io.micronaut.http.client.netty.DefaultHttpClient$BaseHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2097)
at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2158)
at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2122)
at io.micronaut.http.client.netty.SimpleChannelInboundHandlerInstrumented.channelRead0(SimpleChannelInboundHandlerInstrumented.java:49)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
[Skipped a lot of netty channel stuff]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Suppressed: java.lang.Exception: #block terminated with an error
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
at reactor.core.publisher.Flux.blockFirst(Flux.java:2600)
at io.micronaut.http.client.netty.DefaultHttpClient$1.exchange(DefaultHttpClient.java:498)
at io.micronaut.http.client.netty.DefaultHttpClient$1.retrieve(DefaultHttpClient.java:505)
at io.micronaut.http.client.interceptor.HttpClientIntroductionAdvice.lambda$intercept$5(HttpClientIntroductionAdvice.java:409)
at io.micronaut.http.client.interceptor.HttpClientIntroductionAdvice.handleBlockingCall(HttpClientIntroductionAdvice.java:508)
at io.micronaut.http.client.interceptor.HttpClientIntroductionAdvice.intercept(HttpClientIntroductionAdvice.java:408)
at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:137)
at io.micronaut.retry.intercept.RecoveryInterceptor.intercept(RecoveryInterceptor.java:92)
at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:137)
at example.controller.Api$Intercepted.getAuthor(Unknown Source)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrap.invoke(PojoMetaMethodSite.java:203)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148)
at example.controller.AuthorControllerSpec.$spock_feature_0_0(AuthorControllerSpec.groovy:33)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.spockframework.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:198)
[Skipped a lot of gradle, junit, and spock stuff]
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
AuthorControllerSpec > It fails to get an non-existing Author > example.controller.AuthorControllerSpec.It fails to get an non-existing Author [badId: 546252431, #0] FAILED
org.spockframework.runtime.SpockComparisonFailure at AuthorControllerSpec.groovy:37
You imported the wrong HttpRequest
Instead of import java.net.http.HttpRequest you should import import io.micronaut.http.HttpRequest.
But even then you will not catch HttpClientResponseException because you don't use any Client in your test. If you want to receive this exception you should write something like this:
#Inject
#Client("/author")
HttpClient client
#IgnoreRest
def "It fails to get an non-existing Author with an HttpClient"() {
given:
def token = viewer()
when:
def author = client.toBlocking().exchange(HttpRequest.create(
HttpMethod.GET,
"/" + badId
).bearerAuth(token))
then:
def ex = thrown(HttpClientResponseException)
ex.status == NOT_FOUND
ex.getResponse().getBody(ErrorResource).map {
assert it.message == "Author with id ${badId} not found."
it
}.isPresent()
where:
badId = anyInt()
}
Am invoking an REST endpoint from another service using restTemplate.exchange.
The endpoint that receives the request invokes DB and fetches around 1.5 mil records and stores them in another DB.
Now am getting below x_cf_routererror:"endpoint_failure (context canceled)" after invoking the DB. I get this error in about 120+ seconds and process continues as is.
After this error I see another call being made the same endpoint and this is resulting duplicates in target DB.
Not sure why this is happening, I do not have any retry mechanism in place and the restTempalte timeout is set to 300 at client service that invokes.
Has someone faced this issue? whats causing this endpoint_failure (context canceled) and duplicate invocation of endpoint.
Appreciate your help in this.
Log snippet:
2022-05-12T08:57:18.840-04:00 [APP/PROC/WEB/0] [OUT] 2022-05-12 12:57:18.840 INFO 28 --- [nio-8080-exec-4]
Controller1 : Request received to load all timecard information::RequestedTime=12:57:18.840
2022-05-12T08:59:21.530-04:00 [RTR/17] [OUT] - [2022-05-12T12:57:18.829182975Z] "GET HTTP/1.1" 499 0 22 "-" "Java/1.8.0_332" "" "1" x_forwarded_for:"" x_forwarded_proto:"https" vcap_request_id:"" response_time:122.701301 gorouter_time:0.000164 app_id:"" app_index:"0" instance_id:"" x_cf_routererror:"endpoint_failure (context canceled)" x_b3_traceid:"" x_b3_spanid:"" x_b3_parentspanid:"-" b3:"599552bb012c2adc60adef7187a865e7-60adef7187a865e7"
**Below is the duplicate call**
2022-05-12T08:59:21.777-04:00 [APP/PROC/WEB/0] [OUT] 2022-05-12 12:59:21.777 INFO 28 --- [nio-8080-exec-2]
Controller1 : Request received to load all timecard information::RequestedTime=12:59:21.777
Thanks,
S
The error 499 (or 502) is returned by PCF gorouter, instead of your web app [APP/PROC/WEB/0].
PCF gorouter will retry with same HTTP request for particular error cases.
For more details please refer to: https://docs.cloudfoundry.org/adminguide/troubleshooting-router-error-responses.html
Error 499 usually mean that call is taking too much of time and client closed the connection. It has been noticed many times when your app is not able to get a response within specific time (usually 2min time limit set). You might need to see if there is any DB issue or something else that delaying the expected response.
I am migrating from a self-hosted Elasticsearch FusionAuth search to an AWS Elasticsearch Service solution.
I have a new FusionAuth app EC2 instance reading from the in-use database that is configured to use the new Elasticsearch service.
On triggering a reindex from the new app instance I see that only around 60k or 62.5k documents are being written to the new index when I am expecting roughly 6mil.
I see no errors from AWS's Elasticsearch Service and in the app's logs I can see: (endpoint intentionally omitted)
Feb 13, 2020 10:18:46.116 AM INFO io.fusionauth.api.service.search.ElasticSearchClientProvider - Connecting to FusionAuth Search Engine at [https://vpc-<<omitted>>.eu-west-1.es.amazonaws.com]
13-Feb-2020 11:19:55.176 INFO [http-nio-9011-exec-3] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:430)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:684)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:808)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
"/usr/local/fusionauth/logs/fusionauth-app.log" [readonly] 43708L, 4308629C 42183,1 96%
at io.fusionauth.api.service.search.client.domain.documents.IndexUser.<init>(IndexUser.java:79)
at io.fusionauth.api.service.search.ElasticsearchSearchEngine.lambda$index$1(ElasticsearchSearchEngine.java:140)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at io.fusionauth.api.service.search.ElasticsearchSearchEngine.index(ElasticsearchSearchEngine.java:140)
at io.fusionauth.api.service.user.ReindexRunner$ReindexWorker.run(ReindexRunner.java:101)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-14" java.lang.NullPointerException
at io.fusionauth.api.service.search.client.domain.documents.IndexUser.<init>(IndexUser.java:79)
at io.fusionauth.api.service.search.ElasticsearchSearchEngine.lambda$index$1(ElasticsearchSearchEngine.java:140)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at io.fusionauth.api.service.search.ElasticsearchSearchEngine.index(ElasticsearchSearchEngine.java:140)
at io.fusionauth.api.service.user.ReindexRunner$ReindexWorker.run(ReindexRunner.java:101)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-13" java.lang.NullPointerException
Exception in thread "Thread-11" java.lang.NullPointerException
Exception in thread "Thread-12" java.lang.NullPointerException
Feb 18, 2020 10:23:29.064 AM INFO io.fusionauth.api.service.user.ReindexRunner - Reindex completed in [86797] ms or [86] seconds.
Although there are some exceptions there is also an "Reindex completed" INFO log at the end.
Not knowing the ins-and-outs of Elasticsearch I'm also not sure where to start in investigating a NullPointerException.
It looks like the re-index operation is taking an exception which is likely the cause of the truncated index.
Exception in thread "Thread-14" java.lang.NullPointerException
at io.fusionauth.api.service.search.client.domain.documents.IndexUser.<init>(IndexUser.java:79)
This code makes an assumption that you have a username or email address. This should be enforced by the FusionAuth APIs. But in this case, for this exception to occur the email and username are both NULL.
How did you get users into the db, using the Import API, User API, or something else?
In theory you should find at least one user with a NULL value for the email and username.
This query - or similar - should find the offending user, then we need to identify how this user was added to FusionAuth.
SELECT email, username from identities WHERE email IS NULL OR username IS NULL
I am trying to execute the sample producer and consumer code on the Kinesis Streams website: http://docs.aws.amazon.com/streams/latest/dev/learning-kinesis-module-one-download.html
I've downloaded the source, and I am using Eclipse to run it. I've included the necessary jar files, so I would think that everything would be setup to run.
When I run the processor code that consumes the records from Kinesis, however, I get this error:
Aug 02, 2016 8:35:14 PM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker initialize SEVERE: Caught exception when initializing LeaseCoordinator
Does anyone think they could tell me what is causing this error?
EDIT: Here is the full stack trace from the error on Eclipse:
Aug 02, 2016 9:02:27 PM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker initialize
SEVERE: Caught exception when initializing LeaseCoordinator
com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User: arn:aws:sts::500238854089:assumed-role/NORD-NONPROD-a0121-Team/AEXM is not authorized to perform: dynamodb:CreateTable on resource: arn:aws:dynamodb:us-west-2:500238854089:table/amazon-kinesis-learning (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID: BGME094FRUAEK2KFCPQIAM5U8VVV4KQNSO5AEMVJF66Q9ASUAAJG)
at com.amazonaws.services.kinesis.leases.impl.LeaseManager.createLeaseTableIfNotExists(LeaseManager.java:124)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibLeaseCoordinator.initialize(KinesisClientLibLeaseCoordinator.java:172)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:380)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:324)
at com.amazonaws.services.kinesis.samples.stocktrades.processor.StockTradesProcessor.main(StockTradesProcessor.java:96)
Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User: arn:aws:sts::500238854089:assumed-role/NORD-NONPROD-a0121-Team/AEXM is not authorized to perform: dynamodb:CreateTable on resource: arn:aws:dynamodb:us-west-2:500238854089:table/amazon-kinesis-learning (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID: BGME094FRUAEK2KFCPQIAM5U8VVV4KQNSO5AEMVJF66Q9ASUAAJG)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1401)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:945)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:723)
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:475)
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:437)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:386)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2074)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2044)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:899)
at com.amazonaws.services.kinesis.leases.impl.LeaseManager.createLeaseTableIfNotExists(LeaseManager.java:117)
... 4 more
Your stack trace is telling you exactly what the problem is:
User: arn:aws:sts::500238854089:assumed-role/NORD-NONPROD-a0121-Team/AEXM is not authorized to perform: dynamodb:CreateTable on resource: arn:aws:dynamodb:us-west-2:500238854089:table/amazon-kinesis-learning
Make sure you've provided credentials to the DynamoDBClient that has CreateTable permissions - LeaseCoordinator attempts to create the leasing table in Dynamo.
It is actually possible to configure logging for Scala Kinesis Enrich by running the jar file like this:
java -jar -Dorg.slf4j.simpleLogger.defaultLogLevel=debug snowplow-kinesis-enrich-0.5.0 --config enrich.conf --resolver resolver.json
This should print all debug messages from the Kinesis Client Library. (Watch out because the output will become very verbose.) Could you try rerunning with this change to logging? Hopefully that will provide more clues about what's going wrong.
I want to better understand how Tomcat handles requests and why my specific issue may be happening.
I am trying to enhance my existing Spring MVC web application with the Netflix Hystrix framework. This involves adding a hystrix-specific servlet to handle requests ending with /hystrix.stream.
Based on the logs (below), this mapping works and the request is being forwarded to the correct servlet. However, the exception that is thrown does not occure in the servlet. I have also tried extending the servlet to add additional logging - but it appears that no methods in the servlet are called. The exception seems to be happening because of some inner working on Tomcat that I am not aware of.
Here is a snippet from my web.xml (note that the rest of my application works fine - this is just concerning requests for /hystrix.stream):
<servlet>
<display-name>HystrixMetricsStreamServlet</display-name>
<servlet-name>HystrixMetricsStreamServlet</servlet-name>
<servlet-class>com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>HystrixMetricsStreamServlet</servlet-name>
<url-pattern>/hystrix.stream</url-pattern>
</servlet-mapping>
When I navigate to loalhost:8080/web-app/hystrix.stream I am forwarded to the Tomcat 404 Error. The following exception appears in tomcat's log file:
2014-04-16 15:53:22 DEBUG AuthenticatorBase:419 - Security checking request GET /web-app/hystrix.stream
2014-04-16 15:53:22 DEBUG WebappClassLoader:1582 - loadClass(com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet, false)
2014-04-16 15:53:22 DEBUG WebappClassLoader:1598 - Returning class from cache
2014-04-16 15:53:22 DEBUG RealmBase:617 - No applicable constraints defined
2014-04-16 15:53:22 DEBUG AuthenticatorBase:501 - Not subject to any constraint
2014-04-16 15:53:22 DEBUG [localhost]:449 - Processing ErrorPage[errorCode=404, location=/404.htm]
2014-04-16 15:53:22 DEBUG Http11Processor:986 - Error parsing HTTP request header
java.io.EOFException: Unexpected EOF read on the socket
at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:99)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:952)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2014-04-16 15:53:22 DEBUG Http11Protocol:645 - Socket: [org.apache.tomcat.util.net.SocketWrapper#56a7cbf2:129bf96a[TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: Socket[addr=/0:0:0:0:0:0:0:1,port=53783,localport=8443]]], Status in: [OPEN_READ], State out: [CLOSED]
2014-04-16 15:53:22 DEBUG LimitLatch:126 - Counting down[http-bio-8443-exec-1] latch=4
2014-04-16 15:53:22 DEBUG Http11Processor:986 - Error parsing HTTP request header
java.io.EOFException: Unexpected EOF read on the socket
at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:99)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:952)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Has anyone encountered something like this before? As I mentioned, my application works fine - this only occures for /hystrix.stream. I've used the Chrome Postman plugin to add the Accept and Content-Type = application/json headers to the request - both without success. As far as I know, requests to this servlet do not have any header requirements
Thanks for the help.
issue message is very clear, your request head is not corrent.
byte chr = 0;
do {
// Read new bytes if needed
if (pos >= lastValid) {
if (!fill())//read request content
throw new EOFException(sm.getString("iib.eof.error"));
}
// Set the start time once we start reading data (even if it is
// just skipping blank lines)
if (request.getStartTime() < 0) {
request.setStartTime(System.currentTimeMillis());
}
chr = buf[pos++];
} while ((chr == Constants.CR) || (chr == Constants.LF));
above is the source code of tomcat in InternalInputBuffer.parseRequestLine
- *. first time in the while function will be read content (8*1024).
[usually your header won't be so long]
*. after you read request content, the code will be check the first charactor of '\r\n'
*. so if your request content won't be contains this charactors, will be throw EOF exception
*. then the header paser is faill, so the web container don't know the Servlet method, so your servlet won't be called.
i think the server is ok, the problem is appeared in Chrome Postman plugin, please check the header