lambda execution on s3 bucket throws gateway timeout error - java
I am trying to create a lambda funciton through serverless framewrok and want to expose a REST endpoint that will list all the s3 buckets in a region mentioned if the lambda gets triggered through api gateway call.
I successfully created a REST api with hard coded values as mentioned below -
SUCCESS SCENARIO
lambda code:
#RestController
public class LanguageResource {
#RequestMapping(path = "/languages", method = RequestMethod.GET)
public List<Language> listLambdaLanguages() {
return Arrays.asList(new Language("node"), new Language("java"), new Language("python"));
}
}
SAM.yml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Lambda with spring boot.
Resources:
LambdaSpringBootFunction:
Type: AWS::Serverless::Function
Properties:
Handler: com.example.test.lambda.LambdaHandler::handleRequest
Runtime: java8
CodeUri: target/spring-boot-lambda-1.0.0-SNAPSHOT.jar
MemorySize: 1512
Role: arn:aws:iam::XXXXXX:role/rol-lambda-servicenow
# Policies: AWSLambdaBasicExecutionRole
Timeout: 60
Events:
GetResource:
Type: Api
Properties:
Path: /{proxy+}
Method: any
Outputs:
LambdaSpringBootApi:
Description: URL for application
Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Stage/languages'
Export:
Name: LambdaSpringBootApi
endpoint:
/languages
output:
Mon Aug 31 14:13:55 UTC 2020 : Endpoint request body after transformations: {"resource":"/{proxy+}","path":"/languages","httpMethod":"GET","headers":null,"multiValueHeaders":null,"queryStringParameters":null,"multiValueQueryStringParameters":null,"pathParameters":{"proxy":"languages"},"stageVariables":null,"requestContext":{"resourceId":"b04re4","resourcePath":"/{proxy+}","httpMethod":"GET","extendedRequestId":"SI3RfEFWDoEFltg=","requestTime":"31/Aug/2020:14:13:55 +0000","path":"/{proxy+}","accountId":"XXXXXX","protocol":"HTTP/1.1","stage":"test-invoke-stage","domainPrefix":"testPrefix","requestTimeEpoch":1598883235028,"requestId":"caa5dcc6-60ca-4a35-9a6b-a0f3c2a2555a","identity":{"cognitoIdentityPoolId":null,"cognitoIdentityId":null,"apiKey":"test-invoke-api-key","principalOrgId":null,"cognitoAuthenticationType":null,"userArn":"arn:aws:sts::XXXXXX:assumed-role/rol-mgt-dta-administrators/aniruddha.ghosh","apiKeyId":"test-invoke-api-key-id","userAgent":"aws-internal/3 aws-sdk-java/1.11.829 Linux/4.9.217-0.1.ac.205.84.332.metal1. [TRUNCATED]
Mon Aug 31 14:13:55 UTC 2020 : Sending request to https://lambda.eu-west-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:eu-west-1:XXXXXX:function:ssc-mgt-dta-servicenow-la-LambdaSpringBootFunction-DI8EHLE9ME1U/invocations
Mon Aug 31 14:14:05 UTC 2020 : Received response. Status: 200, Integration latency: 10595 ms
Mon Aug 31 14:14:05 UTC 2020 : Endpoint response headers: {Date=Mon, 31 Aug 2020 14:14:05 GMT, Content-Type=application/json, Content-Length=188, Connection=keep-alive, x-amzn-RequestId=1338096f-3eec-4bf4-bd0f-9873250b4e8b, x-amzn-Remapped-Content-Length=0, X-Amz-Executed-Version=$LATEST, X-Amzn-Trace-Id=root=1-5f4d05a3-789c6e282a4f7f92e19a77ff;sampled=0}
Mon Aug 31 14:14:05 UTC 2020 : Endpoint response body before transformations: **{"statusCode":200,"multiValueHeaders":{"Content-Type":["application/json; charset=UTF-8"]},"body":"[{\"name\":\"node\"},{\"name\":\"java\"},{\"name\":\"python\"}]","isBase64Encoded":false}**
Mon Aug 31 14:14:05 UTC 2020 : Method response body after transformations: [{"name":"node"},{"name":"java"},{"name":"python"}]
Mon Aug 31 14:14:05 UTC 2020 : Method response headers: {Content-Type=application/json; charset=UTF-8, X-Amzn-Trace-Id=Root=1-5f4d05a3-789c6e282a4f7f92e19a77ff;Sampled=0}
Mon Aug 31 14:14:05 UTC 2020 : Successfully completed execution
Mon Aug 31 14:14:05 UTC 2020 : Method completed with status: 200
However, when i try to list s3 buckets what i understood that i need to use aws s3 SDK and while used it as mentioned below it is throwing me gateway timeout error with no specific details. I am very new into these things that is why need help
FAILURE SCENARIO
lambda code:
#RequestMapping(path = "/languages", method = RequestMethod.GET)
public ListBucketsResponse listLambdaLanguages() {
Region region = Region.EU_WEST_1;
S3Client s3 = S3Client.builder().region(region).build();
ListBucketsRequest listBucketsRequest = ListBucketsRequest.builder().build();
ListBucketsResponse listBucketsResponse = s3.listBuckets(listBucketsRequest);
listBucketsResponse.buckets().stream().forEach(x -> System.out.println(x.name()));
return listBucketsResponse;
}
SAM.yml
endpoint:
/languages
Output:
Mon Aug 31 14:17:40 UTC 2020 : Endpoint request body after transformations: {"resource":"/{proxy+}","path":"/languages","httpMethod":"GET","headers":null,"multiValueHeaders":null,"queryStringParameters":null,"multiValueQueryStringParameters":null,"pathParameters":{"proxy":"languages"},"stageVariables":null,"requestContext":{"resourceId":"2n2pvy","resourcePath":"/{proxy+}","httpMethod":"GET","extendedRequestId":"SI30xFwRDoEFRPA=","requestTime":"31/Aug/2020:14:17:40 +0000","path":"/{proxy+}","accountId":"XXXXXX","protocol":"HTTP/1.1","stage":"test-invoke-stage","domainPrefix":"testPrefix","requestTimeEpoch":1598883460839,"requestId":"df87b0f3-57de-45f8-9c11-d64000be3d61","identity":{"cognitoIdentityPoolId":null,"cognitoIdentityId":null,"apiKey":"test-invoke-api-key","principalOrgId":null,"cognitoAuthenticationType":null,"userArn":"arn:aws:sts::XXXXXX:assumed-role/rol-mgt-dta-administrators/aniruddha.ghosh","apiKeyId":"test-invoke-api-key-id","userAgent":"aws-internal/3 aws-sdk-java/1.11.829 Linux/4.9.217-0.1.ac.205.84.332.metal1. [TRUNCATED]
Mon Aug 31 14:17:40 UTC 2020 : Sending request to https://lambda.eu-west-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:eu-west-1:XXXXXX:function:ssc-mgt-dta-servicenow-la-LambdaSpringBootFunction-U14LQ9SEJQ4I/invocations
Mon Aug 31 14:17:57 UTC 2020 : Received response. Status: 200, Integration latency: 16221 ms
Mon Aug 31 14:17:57 UTC 2020 : Endpoint response headers: {Date=Mon, 31 Aug 2020 14:17:57 GMT, Content-Type=application/json, Content-Length=143, Connection=keep-alive, x-amzn-RequestId=1c9e34f3-fa10-4db0-b825-5e40e3ac9e94, x-amzn-Remapped-Content-Length=0, X-Amz-Executed-Version=$LATEST, X-Amzn-Trace-Id=root=1-5f4d0684-a47151cdb1421b4494698ebf;sampled=0}
Mon Aug 31 14:17:57 UTC 2020 : Endpoint response body before transformations: {"statusCode":502,"multiValueHeaders":{"Content-Type":["application/json"]},"body":"{\"message\":\"Gateway timeout\"}","isBase64Encoded":false}
Mon Aug 31 14:17:57 UTC 2020 : Method response body after transformations: {"message":"Gateway timeout"}
Mon Aug 31 14:17:57 UTC 2020 : Method response headers: {Content-Type=application/json, X-Amzn-Trace-Id=Root=1-5f4d0684-a47151cdb1421b4494698ebf;Sampled=0}
Mon Aug 31 14:17:57 UTC 2020 : Successfully completed execution
Mon Aug 31 14:17:57 UTC 2020 : Method completed with status: 502
Note:
this role rol-lambda-servicenow has full access on s3
Your lambda does not have permission to do that so it's blocked. You can read more here https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-policy-templates.html
Also try to go to CloudWatch log groups in your AWS console. Logs should tell you clearly about that problem
Related
AWS lambda :Malformed Lambda proxy response
This is a typical issue while Lambda sitting behind API gateway. I've googled almost all the posts but still can't not figure out why it fails. I've read the instruction on what the response should be look like https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-output-format It is looking for { "isBase64Encoded": true|false, "statusCode": httpStatusCode, "headers": { "headerName": "headerValue", ... }, "multiValueHeaders": { "headerName": ["headerValue", "headerValue2", ...], ... }, "body": "..." } Here is the log of my test from API gateway console (I have double checked make sure the name of fields are exactly the same) Thu Mar 19 10:07:58 UTC 2020 : Endpoint response body before transformations: "{\"statusCode\":200,\"body\":\"RESULTS\",\"isBase64Encoded\":false}" Thu Mar 19 10:07:58 UTC 2020 : Execution failed due to configuration error: Malformed Lambda proxy response Thu Mar 19 10:07:58 UTC 2020 : Method completed with status: 502 Can anyone help? EDIT I've removed "body" from response and still get the same error. The following is the actual log Thu Mar 19 23:29:54 UTC 2020 : Received response. Status: 200, Integration latency: 24 ms Thu Mar 19 23:29:54 UTC 2020 : Endpoint response headers: {Date=Thu, 19 Mar 2020 23:29:54 GMT, Content-Type=application/json, Content-Length=48, Connection=keep-alive, x-amzn-RequestId=f2c2c752-a5e0-45e4-9ff0-d91826b51c7b, x-amzn-Remapped-Content-Length=0, X-Amz-Executed-Version=$LATEST, X-Amzn-Trace-Id=root=1-5e740072-46cee9045a56e04b8023816d;sampled=0} Thu Mar 19 23:29:54 UTC 2020 : Endpoint response body before transformations: "{\"statusCode\":200,\"isBase64Encoded\":false}" Thu Mar 19 23:29:54 UTC 2020 : Execution failed due to configuration error: Malformed Lambda proxy response Thu Mar 19 23:29:54 UTC 2020 : Method completed with status: 502 The actual JSON response catptured in java app & lambda is {"statusCode":200,"isBase64Encoded":false}
After comparing to Node.js example, seems that it is to do with the handler. When using a RequestHandler, the response is a String public class MyLambda implements RequestHandler<Object, String> { #Override public String handleRequest(Object input, Context context) { From API gateway, The response is showing as follows, note that the whole response is a String, and API gateway complains Thu Mar 19 23:29:54 UTC 2020 : Endpoint response body before transformations: "{\"statusCode\":200,\"isBase64Encoded\":false}" Thu Mar 19 23:29:54 UTC 2020 : Execution failed due to configuration error: Malformed Lambda proxy response Changed that to RequestStreamHandler , the error is gone public class APIgatewayTest implements RequestStreamHandler { #Override public void handleRequest(InputStream input, OutputStream output, Context context) throws IOException { From API gateway, the response is as follows Thu Apr 09 09:13:08 UTC 2020 : Endpoint response body before transformations: {"statusCode":200,"body":"\"Hello from Lambda!\""} Thu Apr 09 09:13:08 UTC 2020 : Method response body after transformations: "Hello from Lambda!"
ARIBA - ServerThread.run() caught an exception while running ariba.base.tool.checkmeta.CheckMeta. java.lang.NullPointerException
We are getting below error while trying to connect to DB2 system. ServerThread.run() caught an exception while running ariba.base.tool.checkmeta.CheckMeta. java.lang.NullPointerException Please see more details from the log. Wed Sep 13 12:29:55 BST 2017 () (startup.base:INFO) [ID9393]: Base phase start 'Purge shared temp diretory' Wed Sep 13 12:29:55 BST 2017 () (startup.base:INFO) [ID9394]: Base phase stop 'Purge shared temp diretory' (trial #1) duration=0.0s - Mem. in KB: 44858->44858 (0) Wed Sep 13 12:29:55 BST 2017 () (startup.base:INFO) [ID9393]: Base phase start 'Init BaseService' Wed Sep 13 12:29:55 BST 2017 () (startup.base:INFO) [ID9394]: Base phase stop 'Init BaseService' (trial #1) duration=0.0s - Mem. in KB: 44856->44728 (128) Wed Sep 13 12:29:55 BST 2017 () (startup.base:INFO) [ID9393]: Base phase start 'Verify database connectivity' Wed Sep 13 12:29:56 BST 2017 () (sqlio:WARN) [ID3960]: Expecting JDBC Driver : [09.01.0000]. Getting JDBC Driver: 09.07.0011 Wed Sep 13 12:29:56 BST 2017 () (sqlio:WARN) [ID3961]: The JDBC Driver is the Wrong Version. Wed Sep 13 12:29:56 BST 2017 () (util:WARN) [ID2886]: ServerThread.run() caught an exception while running ariba.base.tool.checkmeta.CheckMeta. java.lang.NullPointerException at ariba.base.server.BaseServer.run(BaseServer.java:7 93) at ariba.base.server.SimpleBaseServer.run(SimpleBaseS erver.java:137) at ariba.base.tool.checkmeta.CheckMeta.run(CheckMeta. java:146) at ariba.rpc.server.ServerThread.run(ServerThread.jav a:479) at ariba.base.server.ObjectServerThread.run(ObjectSer verThread.java:488) at java.lang.Thread.run(Thread.java:790) Wed Sep 13 12:29:56 BST 2017 () (util:WARN) [ID2935]: Server shutting down with exit code: 1. Shutdown message: Problem on main server thread: java.lang.NullPointerException at ariba.base.server.BaseServer.run(BaseServer.java:7 93) at ariba.base.server.SimpleBaseServer.run(SimpleBaseS erver.java:137) at ariba.base.tool.checkmeta.CheckMeta.run(CheckMeta. java:146) at ariba.rpc.server.ServerThread.run(ServerThread.jav a:479) at ariba.base.server.ObjectServerThread.run(ObjectSer verThread.java:488) at java.lang.Thread.run(Thread.java:790) (NodesDefault) Wed Sep 13 12:29:57 BST 2017 () (org.jgroups.protocols.pbcast.NAKACK:WARN)
SOAP UI displays faultstring even after error 500 is received. need to view the same in JAVA
Here's the story: Sending a request from SOAPUI (5.2.1) gives me either correct response or, in case of invalid parameters, the error in form of xml. Looking at SoapUI log this is what i see: Wed Jan 18 16:32:39 EST 2017:DEBUG:Attempt 1 to execute request Wed Jan 18 16:32:39 EST 2017:DEBUG:Sending request: POST/XBC/services/TranslateGeometryService HTTP/1.1 Wed Jan 18 16:32:48 EST 2017:DEBUG:Receiving response: HTTP/1.1 401 Unauthorized Wed Jan 18 16:32:48 EST 2017:DEBUG:Connection can be kept alive for 10000 MILLISECONDS Wed Jan 18 16:32:48 EST 2017:DEBUG:Target requested authentication Wed Jan 18 16:32:48 EST 2017:DEBUG:Authorization challenge processed Wed Jan 18 16:32:48 EST 2017:DEBUG:Authentication scope: BASIC 'Spring Security Application'#mywebservice:80 Wed Jan 18 16:32:48 EST 2017:INFO:mywebservice:80 requires authentication with the realm 'Spring Security Application' Wed Jan 18 16:32:48 EST 2017:DEBUG:Found credentials Wed Jan 18 16:32:48 EST 2017:DEBUG:Attempt 2 to execute request Wed Jan 18 16:32:48 EST 2017:DEBUG:Sending request: POST/XBC/services/TranslateGeometryService HTTP/1.1 Wed Jan 18 16:33:14 EST 2017:DEBUG:Receiving response: HTTP/1.1 500 Internal Server Error Wed Jan 18 16:33:14 EST 2017:DEBUG:Connection shut down Wed Jan 18 16:33:14 EST 2017:INFO:Got response for [TranslateGeometrySoap11.getLocationFromTramPoleTrackOffset:Request 1] in 35794ms (312 bytes) (Note that those 312 bytes come after error 500 and connection shut down messages) Response window, at the same time, in SOAPUI shows this: <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"> <SOAP-ENV:Header/> <SOAP-ENV:Body> <SOAP-ENV:Fault> <faultcode>SOAP-ENV:Server</faultcode> <faultstring xml:lang="en">Error: No Track exists with Track Code: CHS-CSK-UP-TW</faultstring> </SOAP-ENV:Fault> </SOAP-ENV:Body> This is perfect and works as i expect it to work. Now for my problem that is seriously driving me mad. When i attempt to send exactly the same request from my java code that is supposed to return the error above, i receive IOException that says: Server returned HTTP response code: 500 for URL: http://mywebservice/XBC/services/TranslateGeometryService So, pretty much same as what SOAPUI shows in its log. But... No matter what i do in my code i cannot get that message in faultstring that SOAPUI displays: "No Track exists with Track Code: CHS-CSK-UP-TW". How is SOAPUI doing it after it receives error 500? Or, more importantly - how do i get it in JAVA after receiving error 500? Tried playing with SoapFaultException class and SoapFault, but i could not find anything more anywhere. Thanks to anyone who tries to answer this question. Cheers, Pierre
The next code in Java handler uses SOAPMessageContext TO detect SOAP fault and extract fault string from SOAPBody: public bool handleFault(SOAPMessageContext context) { try { if (context.getMessage().getSOAPBody().hasFault()) { SOAPFault fa = context.getMessage().getSOAPBody().getFault(); System.err.println(fa.getFaultCode()); System.err.println(fa.getFaultString()); } return true; } catch (SOAPException ex) { System.err.println("Fault parsing Exception:" + ex.getMessage()); return true; } return false; }
yoda-time 2.3 toDateTimeAtStartOfDay() unexpected behaviour
Could you please explain the conditions under which Yoda time gives different result while working with same input data: LocalDate value = LocalDate.now() Here is what debug shows in Junit test value 2015-10-16 ((LocalDate) value).toDateTimeAtStartOfDay().toDate() Fri Oct 16 00:00:00 EEST 2015 ((LocalDate) value).toDate() Fri Oct 16 00:00:00 EEST 2015 Here is what real environment debug gives (Karaf) value 2015-10-16 ((LocalDate) value).toDateTimeAtStartOfDay().toDate() Thu Oct 15 20:00:00 EDT 2015 ((LocalDate) value).toDate() Fri Oct 16 00:00:00 EDT 2015 The question is why we have different results when toDateTimeAtStartOfDay() is used?
How to use Twitter4j to show tweets
I have followed this tutorial http://www.javacodegeeks.com/2012/03/twitter-api-on-your-java-application.html and did the exact same thing. But I am getting an error which is shown below. Am I missing something? Should I be setting some permissions in any file? I have checked in many places but still couldn't fix this. Please help Exception in thread "main" 401:Authentication credentials (https://dev.twitter.com/docs/auth) were missing or incorrect. Ensure that you have set valid conumer key/secret, access token/secret, and the system clock in in sync. error - Not authorized request - /1/statuses/user_timeline.json? TwitterException{exceptionCode=[ced778ef-115a04e4], statusCode=401, retryAfter=-1, rateLimitStatus=RateLimitStatusJSONImpl{remainingHits=145, hourlyLimit=150, resetTimeInSeconds=1362901, secondsUntilReset=1235, resetTime=Sun Mar 10 13:15:51 IST 2013}, featureSpecificRateLimitStatus=null, version=2.2.5} at twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:185) at twitter4j.internal.http.HttpClientWrapper.request(HttpClientWrapper.java:65) at twitter4j.internal.http.HttpClientWrapper.get(HttpClientWrapper.java:85) at twitter4j.TwitterImpl.get(TwitterImpl.java:1895) at twitter4j.TwitterImpl.getUserTimeline(TwitterImpl.java:254) at com.jdwb.twitterapi.tweet.main(tweet.java:33) public class tweet { public static void main(String[] args) throws TwitterException { Twitter twitter = new TwitterFactory().getInstance(); ResponseList < Status > a = twitter.getUserTimeline(new Paging(1, 5)); for (Status b: a) { System.out.println(b.getText()); } } } -------------------------Properties file------------- debug=true oauth.consumerKey=*************** oauth.consumerSecret=*********** oauth.accessToken=******************* oauth.accessTokenSecret=*********************** [Sun Mar 10 14:41:26 IST 2013]status: 401 Unauthorized [Sun Mar 10 14:41:26 IST 2013]x-ratelimit-remaining: 143 [Sun Mar 10 14:41:26 IST 2013]content-encoding: gzip [Sun Mar 10 14:41:26 IST 2013]date: Sun, 10 Mar 2013 09:11:26 GMT [Sun Mar 10 14:41:26 IST 2013]x-ratelimit-reset: 1362908664 [Sun Mar 10 14:41:26 IST 2013]x-transaction: 58f6c286bb1aa6de [Sun Mar 10 14:41:26 IST 2013]pragma: no-cache [Sun Mar 10 14:41:26 IST 2013]cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0 [Sun Mar 10 14:41:26 IST 2013]{"request":"\/1\/statuses\/user_timeline.json?include_rts=true&include_entities=true&include_my_retweet=1&count=5&page=1","error":"Not authorized"} Exception in thread "main" 401:Authentication credentials (https://dev.twitter.com/docs/auth) were missing or incorrect. Ensure that you have set valid conumer key/secret, access token/secret, and the system clock in in sync. error - Not authorized
I presume that you have not got your authorization set up correctly have a look here for how to set up the properties file. you need to sign up to the twitter dev to get your API keys so that you can fill in the information that you need.