We Keep Coding

AWS API Gateway and Lambda to return image

Say I have this HTML:
<img src="http://example.com/pic"/>
What I would like to do is have example.com/pic map to an AWS API Gateway endpoint.
That endpoint would then call a lambda function.
That lambda function would read a random image from an s3 bucket and return it.
So my aim is to use a STANDARD HTML image tag and end up with an image from an s3 bucket but going via some decision code in the lambda to decide the image to return.
I know you can use s3 to serve static content directly (hence the lambda to make the decision about what image). I also know I could do stuff in the lambda like b64 encode the response and then handle it on the client but I am aiming to use the standard HTML IMG tag.
Is this possible?
I've tried using the ResponseStreamHandler (Java SDK) for the lambda and returning the byte array of the image and also added the API gateway config to not map the output to JSON, but nothing seems to work!

It seems AWS has simplified this process, so that many answers are outdated and/or overly complicated.
This is how I got Lambda to return an image through the API Gateway, as of June 2018:
1) In API Gateway, enable Use Lambda Proxy integration for your API. (This setting is located on the Integration Request section, where you had set the type to Lambda.)
2) In API Gateway, select your API and click Settings. In Binary Media Types add */*. (Note: I tried adding simply 'image/jpeg', but it seems to require */* to get all of this to work)
3) Be sure to deploy your API, otherwise your changes will not be live. (In API Gateway, select your API, then Actions > Deploy API).
4) In your Lambda code, return your image in Base64 encoding (this example is C# code):
// set the Content-type header
// set to whichever image type you're returning
var headersDic = new Dictionary<string, string>();
headersDic.Add("Content-type", "image/jpeg");
// return the response object that APIGateway requires
return new APIGatewayProxyResponse
{
StatusCode = 200,
Headers = headersDic,
// return the image in Base64 encoding
Body = Convert.ToBase64String(...your image data...),
IsBase64Encoded = true
};
Done.
If you've setup your API to not require authentication, simply type your API link into your browser, and it will display the image. Or put the API link into an IMG tag. e.g. <img src="https://asdf.execute-api.us-east-1.amazonaws.com/live/myapi" />
Note: Even though in step 2 you set the Binary Media Types to */*, API Gateway will still return text if that is what your Lambda is returning.

Luckily, now AWS API Gateway supports binary data, though you also need to update your resource method through the CLI as it is not yet implemented in the Console. This is what you need to do:
In the Method Response of your method
Set Content-Type as image/jpeg in HTTP 200 Status Response
Header
In the Integration Response of your method
Set Content-Type as 'image/jpeg' in Header Mappings. Mind the quotes!
With the AWS CLI, set contentHandling attribute to CONVERT_TO_BINARYon your Integration Response
Check to entire process in this great step-by step guide: https://stackoverflow.com/a/41434295/720665

I've run in to a similar problem. As mentioned you currently can't directly return your image in binary format from your API Gateway endpoint, which would be required for the browser to display it correctly.
However, I solved this by instead having API Gateway return a 302 Redirect, pointing to the correct file in S3. You can have your Lambda function return the url to the file, which is later on mapped to the Location header in API Gateway. The browser will follow the redirect and display the image properly.
There are several ways to implement the redirect, but I did as follow:
Lambda returns an object with the target image like so:
function handler(event, context) {
context.succeed({
location: "https://[bucket-name].s3-eu-west-1.amazonaws.com/myimage.png" });
});
}
Remove the normal '200' Method Response Status from The Integration Response in API Gateway. Replace it with a '302' response status and add the 'Location' header mapped to value 'integration.response.body.location'
Add the 302 status to the Method Response as well

Just to be clear, the client does two different requests:
The first to get the HTML (including the image url).
The second to fetch the image data from the url.
In other words, the image data is not inlined in the HTML.
Based on this knowledge you can have a Lambda (behind the API gateway) as you suggest. The Lambda implementation can have some logic that determines the url to the image stored in S3. However, the Lambda returns JSON data and not HTML (there are workarounds such as return the html in a variable) which makes things trickier, especially for large HTML pages.
I suggest a slightly different approach, since just receiving an image tag will not get you far. I assume you will inline the image tag in a HTML document, probably by using JavaScript. Then you might as well let the API Gateway / Lambda request return a JSON document with the image url and let the JavaScript either update an existing image tag with the new url or generate the tag for you.

It currently isn't possible because you cannot return binary data through the AWS API Gateway.
For this to work, the lambda function would need to return the image data as binary blob, and some meta-information such as the image content type. Then, AWS API Gateway would need to be able to map this to the HTTP response. For example:-
lambda returns:
{
contentType: 'image/png',
image: "encoded binary data"
}
then API gateway would need to map contentType to the 'content-type' header of the response, and put the image data in the body of the response with the right encoding and length.
Unfortunately, it doesn't do this right now. It only maps text encodings like application/json or application/xml as the response type (it is designed for APIs after all).
You could very easily achieve this using ElasticBeanstalk where you have a lot more control over the http response.

I resolved this problem be reconsidering my design. The underlying thinking is that Lambdas are good computing units and bad file servers.
I changed
Client > APIGW > Lambda > Image
into
Client > APIGW > Lambda > SignedURL
then Client(SignedURL) > Image
Since my client was web based, everything was easier after that shift

RESTService returns HTML when writer does not contain data

I'm currently developing RESservices using the customService bean. One thing I've noticed is that for some reason when I dont use the responsewriter object but only set the response status using engine.getHttpResponse().setStatus(404) (for instance). The header is correctly set ( 404 ) but there is still some html generated.
I've already tried to set the rendered property on the view tag (of the xpage) to false but that doesn't seem to do the trick. Is there some documentation on how to use the CustomServiceBean and not returning any data?

Setting the Status is a good approach, so the API is easy to handle. However you want to consider:
set the content type to what would actually be rendered if you had a result. If you don't set response.contenttype it defaults to HTML
404 is meant for navigation/urls. Since your XPage renders a result, the request found a valid URL. That based on user and/or parameters there's no result isn't therefore a 4xx class of error, but rather a 5xx. Pick one of them.
when testing use wget or curl, so you can be sure the browser doesn't do the HTML
Let us know how it goes

I think returning a 404 response causes always HTML data as it normally gives you the error message etc.
If you don't want to response with any data just return an empty JSON object - as I assume you return JSON when you HAVE data, right?

dont set 404 status ,set null for this case,ant error status like 404 means creating html for that

Spring REST tutorial [duplicate]

I'm building a REST API, but I've encountered a problem.
It seems that accepted practice in designing a REST API is that if the resource requested doesn't exist, a 404 is returned.
However, to me, this adds unnecessary ambiguity. HTTP 404 is more traditionally associated with a bad URI. So in effect we're saying "Either you got to the right place, but that specific record does not exist, or there's no such location on the Internets! I'm really not sure which one..."
Consider the following URI:
http://mywebsite/api/user/13
If I get a 404 back, is that because User 13 does not exist? Or is it because my URL should have been:
http://mywebsite/restapi/user/13
In the past, I've just returned a NULL result with an HTTP 200 OK response code if the record doesn't exist. It's simple, and in my opinion very clean, even if it's not necessarily accepted practice. But is there a better way to do this?

404 is just the HTTP response code. On top of that, you can provide a response body and/or other headers with a more meaningful error message that developers will see.

Use 404 if the resource does not exist. Don't return 200 with an empty body.
This is akin to undefined vs empty string (e.g. "") in programming. While very similar, there is definitely a difference.
404 means that nothing exists at that URI (like an undefined variable in programming). Returning 200 with an empty body means that something does exist there and that something is just empty right now (like an empty string in programming).
404 doesn't mean it was a "bad URI". There are special HTTP codes that are intended for URI errors (e.g. 414 Request-URI Too Long).

As with most things, "it depends". But to me, your practice is not bad and is not going against the HTTP spec per se. However, let's clear some things up.
First, URI's should be opaque. Even if they're not opaque to people, they are opaque to machines. In other words, the difference between http://mywebsite/api/user/13, http://mywebsite/restapi/user/13 is the same as the difference between http://mywebsite/api/user/13 and http://mywebsite/api/user/14 i.e. not the same is not the same period. So a 404 would be completely appropriate for http://mywebsite/api/user/14 (if there is no such user) but not necessarily the only appropriate response.
You could also return an empty 200 response or more explicitly a 204 (No Content) response. This would convey something else to the client. It would imply that the resource identified by http://mywebsite/api/user/14 has no content or is essentially nothing. It does mean that there is such a resource. However, it does not necessarily mean that you are claiming there is some user persisted in a data store with id 14. That's your private concern, not the concern of the client making the request. So, if it makes sense to model your resources that way, go ahead.
There are some security implications to giving your clients information that would make it easier for them to guess legitimate URI's. Returning a 200 on misses instead of a 404 may give the client a clue that at least the http://mywebsite/api/user part is correct. A malicious client could just keep trying different integers. But to me, a malicious client would be able to guess the http://mywebsite/api/user part anyway. A better remedy would be to use UUID's. i.e. http://mywebsite/api/user/3dd5b770-79ea-11e1-b0c4-0800200c9a66 is better than http://mywebsite/api/user/14. Doing that, you could use your technique of returning 200's without giving much away.

That is an very old post but I faced to a similar problem and I would like to share my experience with you guys.
I am building microservice architecture with rest APIs. I have some rest GET services, they collect data from back-end system based on the request parameters.
I followed the rest API design documents and I sent back HTTP 404 with a perfect JSON error message to client when there was no data which align to the query conditions (for example zero record was selected).
When there was no data to sent back to the client I prepared an perfect JSON message with internal error code, etc. to inform the client about the reason of the "Not Found" and it was sent back to the client with HTTP 404. That works fine.
Later I have created a rest API client class which is an easy helper to hide the HTTP communication related code and I used this helper all the time when I called my rest APIs from my code.
BUT I needed to write confusing extra code just because HTTP 404 had two different functions:
the real HTTP 404 when the rest API is not available in the given url, it is thrown by the application server or web-server where the rest API application runs
client get back HTTP 404 as well when there is no data in database based on the where condition of the query.
Important: My rest API error handler catches all the exceptions appears in the back-end service which means in case of any error my rest API always returns with a perfect JSON message with the message details.
This is the 1st version of my client helper method which handles the two different HTTP 404 response:
public static String getSomething(final String uuid) {
String serviceUrl = getServiceUrl();
String path = "user/" + , uuid);
String requestUrl = serviceUrl + path;
String httpMethod = "GET";
Response response = client
.target(serviceUrl)
.path(path)
.request(ExtendedMediaType.APPLICATION_UTF8)
.get();
if (response.getStatus() == Response.Status.OK.getStatusCode()) {
// HTTP 200
return response.readEntity(String.class);
} else {
// confusing code comes here just because
// I need to decide the type of HTTP 404...
// trying to parse response body
try {
String responseBody = response.readEntity(String.class);
ObjectMapper mapper = new ObjectMapper();
ErrorInfo errorInfo = mapper.readValue(responseBody, ErrorInfo.class);
// re-throw the original exception
throw new MyException(errorInfo);
} catch (IOException e) {
// this is a real HTTP 404
throw new ServiceUnavailableError(response, requestUrl, httpMethod);
}
// this exception will never be thrown
throw new Exception("UNEXPECTED ERRORS, BETTER IF YOU DO NOT SEE IT IN THE LOG");
}
BUT, because my Java or JavaScript client can receive two kind of HTTP 404 somehow I need to check the body of the response in case of HTTP 404. If I can parse the response body then I am sure I got back a response where there was no data to send back to the client.
If I am not able to parse the response that means I got back a real HTTP 404 from the web server (not from the rest API application).
It is so confusing and the client application always needs to do extra parsing to check the real reason of HTTP 404.
Honestly I do not like this solution. It is confusing, needs to add extra bullshit code to clients all the time.
So instead of using HTTP 404 in this two different scenarios I decided that I will do the following:
I am not using HTTP 404 as a response HTTP code in my rest application anymore.
I am going to use HTTP 204 (No Content) instead of HTTP 404.
In that case client code can be more elegant:
public static String getString(final String processId, final String key) {
String serviceUrl = getServiceUrl();
String path = String.format("key/%s", key);
String requestUrl = serviceUrl + path;
String httpMethod = "GET";
log(requestUrl);
Response response = client
.target(serviceUrl)
.path(path)
.request(ExtendedMediaType.APPLICATION_JSON_UTF8)
.header(CustomHttpHeader.PROCESS_ID, processId)
.get();
if (response.getStatus() == Response.Status.OK.getStatusCode()) {
return response.readEntity(String.class);
} else {
String body = response.readEntity(String.class);
ObjectMapper mapper = new ObjectMapper();
ErrorInfo errorInfo = mapper.readValue(body, ErrorInfo.class);
throw new MyException(errorInfo);
}
throw new AnyServerError(response, requestUrl, httpMethod);
}
I think this handles that issue better.
If you have any better solution please share it with us.

404 Not Found technically means that uri does not currently map to a resource. In your example, I interpret a request to http://mywebsite/api/user/13 that returns a 404 to imply that this url was never mapped to a resource. To the client, that should be the end of conversation.
To address concerns with ambiguity, you can enhance your API by providing other response codes. For example, suppose you want to allow clients to issue GET requests the url http://mywebsite/api/user/13, you want to communicate that clients should use the canonical url http://mywebsite/restapi/user/13. In that case, you may want to consider issuing a permanent redirect by returning a 301 Moved Permanently and supply the canonical url in the Location header of the response. This tells the client that for future requests they should use the canonical url.

So in essence, it sounds like the answer could depend on how the request is formed.
If the requested resource forms part of the URI as per a request to http://mywebsite/restapi/user/13 and user 13 does not exist, then a 404 is probably appropriate and intuitive because the URI is representative of a non-existent user/entity/document/etc. The same would hold for the more secure technique using a GUID http://mywebsite/api/user/3dd5b770-79ea-11e1-b0c4-0800200c9a66 and the api/restapi argument above.
However, if the requested resource ID was included in the request header [include your own example], or indeed, in the URI as a parameter, eg http://mywebsite/restapi/user/?UID=13 then the URI would still be correct (because the concept of a USER does exits at http://mywebsite/restapi/user/); and therefore the response could reasonable be expected to be a 200 (with an appropriately verbose message) because the specific user known as 13 does not exist but the URI does. This way we are saying the URI is good, but the request for data has no content.
Personally a 200 still doesn't feel right (though I have previously argued it does). A 200 response code (without a verbose response) could cause an issue not to be investigated when an incorrect ID is sent for example.
A better approach would be to send a 204 - No Contentresponse. This is compliant with w3c's description *The server has fulfilled the request but does not need to return an entity-body, and might want to return updated metainformation.*1 The confusion, in my opinion is caused by the Wikipedia entry stating 204 No Content - The server successfully processed the request, but is not returning any content. Usually used as a response to a successful delete request. The last sentence is highly debateable. Consider the situation without that sentence and the solution is easy - just send a 204 if the entity does not exist. There is even an argument for returning a 204 instead of a 404, the request has been processed and no content has been returned! Please be aware though, 204's do not allow content in the response body
Sources
http://en.wikipedia.org/wiki/List_of_HTTP_status_codes
1. http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

This old but excellent article... http://www.infoq.com/articles/webber-rest-workflow says this about it...
404 Not Found - The service is far too lazy (or secure) to give us a real reason why our request failed, but whatever the reason, we need to deal with it.

This recently came up with our team.
We use both 404 Not found with a message body and 204 No Content based on the following rational.
If the request URI indicates the location of a single resource, we use 404 Not found. When the request queries a URI, we use 204 No Content
http://mywebsite/api/user/13 would return 404 when user 13 does not exist
http://mywebsite/api/users?id=13 would return 204 no content
http://mywebsite/api/users?firstname=test would return 204 no content
The idea here being, 'query routes' are expected to be able to return 1, many or no content.
Whatever pattern you choose, the most important things is to be consistent - so get buy in from your team.

The Uniform Resource Identifier is a unique pointer to the resource. A poorly form URI doesn't point to the resource and therefore performing a GET on it will not return a resource. 404 means The server has not found anything matching the Request-URI. If you put in the wrong URI or bad URI that is your problem and the reason you didn't get to a resource whether a HTML page or IMG.

Since this discussion seems to be able to survive the end of time I'll throw in the JSON:API Specifications
404 Not Found
A server MUST respond with 404 Not Found when processing a request to fetch a single resource that does not exist, except when the request warrants a 200 OK response with null as the primary data (as described above).
HTTP/1.1 200 OK
Content-Type: application/vnd.api+json
{
"links": {
"self": "http://example.com/articles/1/author"
},
"data": null
}
Also please see this Stackoverflow question

For this scenario HTTP 404 is response code for the response from the REST API
Like 400, 401, 404 , 422 unprocessable entity
use the Exception handling to check the full exception message.
try{
// call the rest api
} catch(RestClientException e) {
//process exception
if(e instanceof HttpStatusCodeException){
String responseText=((HttpStatusCodeException)e).getResponseBodyAsString();
//now you have the response, construct json from it, and extract the errors
System.out.println("Exception :" +responseText);
}
}
This exception block give you the proper message thrown by the REST API

Open an authenticated image served by django from java using Apache http client

I Am serving an authenticated image using django. The image is behind a view which require login, and in the end I have to check more things than just the authentication.
Because of a reason to complicated to explain here, I cannot use the real url to the image, but I Am serving it with a custom url leading to the authenticated view.
From java the image must be reachable, to save or display. For this part I use Apache httpclient.
In Apacahe I tried a lot of things (every example and combination of examples...) but can't seem to get it working.
For other parts of the webapp I use django-rest-framwork, which I succesfully connected to from java (and c and curl).
I use the login_reuired decorator in django, which makes the attempt to get to the url redirect to a login page first.
Trying the link and the login in a webviewer, I see the 200 code (OK) in the server console.
Trying the link with the httpclient, I get a 302 Found in the console.... (looking up 302, it means a redirect..)
this is what I do in django:
in urls.py:
url(r'^photolink/(?P<filename>.*)$', 'myapp.views.photolink',name='photolink'),
in views.py:
import mimetypes
import os
#login_required
def photolink(request, filename):
# from the filename I get the image object, for this question not interesting
# there is a good reason for this complicated way to reach a photo, but not the point here
filename_photo = some_image_object.url
base_filename=os.path.basename(filename_photo)
# than this is the real path and filename to the photo:
path_filename=os.path.join(settings.MEDIA_ROOT,'photos',mac,base_filename)
mime = mimetypes.guess_type(filename_photot)[0]
logger.debug("mimetype response = %s" % mime)
image_data = open(path_filename, 'rb').read()
return HttpResponse(image_data, mimetype=mime)
by the way, if i get this working i need another decorator to pass some other tests....
but i first need to get this thing working....
for now it's not a secured url.... plain http.
in java i tried a lot of things... using apache's httpclient 4.2.1
proxy, cookies, authentication negociation, with follow redirects... and so on...
Am I overlooking some basic thing here?...
it seems the login of the website client is not suitable for automated login...
so the problem can be in my code in django....or in the java code....

In the end the problem was, using HTTP authorization.
Which is not by default used in the login_required decorator.
adding a custom decorator that checks for HTTP authorization did the trick:
see this example: http://djangosnippets.org/snippets/243/

serving GWT permutations from appengine blob store - XSRF not found

In trying to serve GWT permutations out of the blob store in order to escape the AppEngine hard limit of 150 mb for static files, I've succeed in doing so for "html" and image files "jpeg, png, .etc" and other .rpc calls, but am hung up on XSRF calls.
In the server logs, I see:
The serialization policy file '/theapplication/CCA65B31464BDB27545C23C142FEEEF8.gwt.rpc' was not found;
My upload log shows it was uploaded /CCA65B31464BDB27545C23C142FEEEF8.gwt.rpc : HTTP/1.1 200 OK
The request url shows http://14.applicationXYZ.appspot.com/xsrf
the RequestPayload shows: http://14.applicationXYZ.appspot.com/theapplication/|CCA65B31464BDB27545C23C142FEEEF8|com.google.gwt.user.client.rpc.XsrfTokenService|getNewXsrfToken|1|2|3|4|0|
Other rpc calls are resolving (via a server filter is looking for /theapplication and mapping the requests to a blob to serve) as in the following case where an rpc call is made without an Xsrf request (as the user is not logged in yet)
req url -- http://14.applicationXYZ.appspot.com/someRPCCall
RequestPayload -- http://14.applicationXYZ.appspot.com/theapplication/|62D7E6737056C685E10947B640409549|com.abc.client.rpc.Service|doWork|java.lang.String/2004016611|java.lang.Boolean/476441737|wwwerr|1|2|3|4|3|5|5|6|7|7|6|0|
So, I have two questions:
1) why is XSRF call failing to return the appropriate blob, ie. why doesn't the xrsf call get handled by the filter the way other url calls to /theapplication/* do?
2) What can I do about it?
3) Also, I tried setting the content type to "text/x-gwt-rpc; charset=UTF-8 and also as unspecified when I uploaded the blob. Anyone know what the content type should be for *.gwt.rpc in case I do get the xrsf working? Could having the wrong content type be causing the trouble?
***note applicationXYZ is not the real name so no the links won't work.

OK /xsrf is mapped to a servlet as well, so if the filter returns a blob without passing on the filter, it seems it won't reach the servlet.
Anyway, it's easy enough just to upload the few .rpc files as normal and not serve them as blobs.