We Keep Coding

Unable to get Dataflow steps information using Java API

I am trying to get the steps & jobMetrics information of an active streaming job using the following code:
val jobMetrics = dataflowClient.projects().jobs().getMetrics(projectId, jobId).execute().getMetrics
val steps = dataflowClient.projects().jobs().get(projectId, jobId).execute().getSteps
But, even though I am able to get the jobMetrics information, the steps is always returning null.
Any pointers on what I am doing wrong?

Apologies I'm Java illiterate but with regards to the details that you need I can point you to which endpoint in Dataflow API to get them. My examples are done by sending HTTP requests to the Dataflow API using curl.
To retrieve steps using Dataflow API use projects.jobs.get endpoint and specify what JobView you want. JOB_VIEW_ALL is required to return job.steps.
Sample curl command used:
curl -X GET -H "Content-Type: application/json" -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) https://dataflow.googleapis.com/v1b3/projects/my-project/locations/us-central1/jobs/2022-01-13_19_11_56-999999999999?view=JOB_VIEW_ALL
Response snippet with steps:
But if following Dataflow Java API. You need to build your GetJobRequest to include setView() with Enum JobView as the parameter. Using code snippet from getJob() documentation:
try (JobsV1Beta3Client jobsV1Beta3Client = JobsV1Beta3Client.create()) {
GetJobRequest request =
GetJobRequest.newBuilder()
.setProjectId("your-project")
.setJobId("jobId101296568")
.setView(JobView.forNumber(2)) // try 2 as value for JOB_VIEW_ALL as per Enum JobView documentation provided above
.setLocation("us-central1")
.build();
Job response = jobsV1Beta3Client.getJob(request);
}
Code snippet above should return Job object where you can try doing getSteps() and see if it will return something.

Spring rest with Spring Boot: Upload MultipartFile and Json object as parameters

I know this question was asked on stackoverflow before. However, I haven't found an answer that actually tests the service using, for example, Postman.
I have followed the documentation and my service is pretty much the same as described on the documentation:
#PostMapping("/someUrl")
public ResponseEntity<Void> uploadFile(#RequestPart(name="foo", required = false) Foo foo, #RequestPart("file") MultipartFile file) {
return new ResponseEntity<>(OK);
}
When I try to heat my service using Postman, I got the following error:
org.springframework.web.multipart.MultipartException: Current request is not a multipart request
And on postman, I got the following return
The question is: Is it possible to call my service using Postman? If it is, what parameters am I missing?

Yes it is possible to call your service using Postman.
Instead of sending JSON as text, create a file (e.g. test.json) containing JSON data that you want to send:
{
"teste": "abc"
}
Then in form-data section change the type of your foo field from Text to File and choose JSON file that you've created.
And that's it. Click Send and your request should be processed without any error.
Note: Header should be left empty i.e. it should not contain any custom Content-Type value.

Looks like there is some issue with Postman while using multipart file with json data. https://github.com/postmanlabs/postman-app-support/issues/576
You can use curl to test your service like below.
curl -v -H "Content-Type:multipart/form-data" -F "foo={\"teste\":\"abc\"};type=application/json" -F "file=#amazon.png" http://localhost:8080/v1/files/someURL

Converting Curl to POST api for java

I would like to post some data to a rest api.
The API documentation (See page 38) asks for the following:
curl -u "USERNAME:PASSWORD" -H "Content-type: text/xml" -X "POST"
--data-binary #-
"https://qualysapi.qualys.com/qps/rest/3.0/create/was/webapp/" <
file.xml
Note: “file.xml” contains the request POST data.
Request POST data:
<ServiceRequest>
<data>
<WebApp>
<name><![CDATA[My Web Application]]></name>
<url><![CDATA[http://mywebapp.com]]></url>
</WebApp>
</data>
</ServiceRequest>
I have confirmed that the call works on the command line using curl.
I then began to write a small app in Java and found UniRest.
Thats where the problem starts. I do not know how to convert the curl request into Unirest.
I have this much so far:
Unirest.post("http://apiurl).basicAuth("user","pass").field(name, file).asBinary();
the latter half
.field(name, file).asBinary();
doesnt make sense to me. What is the intent behind giving the file a name. Isn't suppose to retrieve the data from the file?
Furthermore, I would like to avoid writing data to file. How can I create the same xml with UniRest.
If not xml, could I do the same with JSON? The API attached above (appendix C) also accepts JSON. However, how can I nest fields with the builder pattern of the Unirest api

According to the UniRest documentation, it looks like you can write any array of bytes to a field in the request. You just have to encode the string into a byte array.
Unirest.post("http://apiroot")
.field(name, xmlString.getBytes(StandardCharsets.UTF_8))
.asBinary();
Alternatively, you can use any InputStream,
Unirest.post("http://apiroot")
.field(name, new CharSequenceInputStream(xmlString, StandardCharsets.UTF_8))
.asBinary();
Usually data is the body of the request though (not as a field). If you do want to send the data as the request body and not a form field, you should use the body(String body) method instead of the field(String name, Object object) method, for instance:
String data = "<ServiceRequest>... etc...</ServiceRequest>";
Unirest.post("http://apiroot")
.body(xmlString)
.asBinary();

AWS API Gateway and Lambda to return image

Say I have this HTML:
<img src="http://example.com/pic"/>
What I would like to do is have example.com/pic map to an AWS API Gateway endpoint.
That endpoint would then call a lambda function.
That lambda function would read a random image from an s3 bucket and return it.
So my aim is to use a STANDARD HTML image tag and end up with an image from an s3 bucket but going via some decision code in the lambda to decide the image to return.
I know you can use s3 to serve static content directly (hence the lambda to make the decision about what image). I also know I could do stuff in the lambda like b64 encode the response and then handle it on the client but I am aiming to use the standard HTML IMG tag.
Is this possible?
I've tried using the ResponseStreamHandler (Java SDK) for the lambda and returning the byte array of the image and also added the API gateway config to not map the output to JSON, but nothing seems to work!

It seems AWS has simplified this process, so that many answers are outdated and/or overly complicated.
This is how I got Lambda to return an image through the API Gateway, as of June 2018:
1) In API Gateway, enable Use Lambda Proxy integration for your API. (This setting is located on the Integration Request section, where you had set the type to Lambda.)
2) In API Gateway, select your API and click Settings. In Binary Media Types add */*. (Note: I tried adding simply 'image/jpeg', but it seems to require */* to get all of this to work)
3) Be sure to deploy your API, otherwise your changes will not be live. (In API Gateway, select your API, then Actions > Deploy API).
4) In your Lambda code, return your image in Base64 encoding (this example is C# code):
// set the Content-type header
// set to whichever image type you're returning
var headersDic = new Dictionary<string, string>();
headersDic.Add("Content-type", "image/jpeg");
// return the response object that APIGateway requires
return new APIGatewayProxyResponse
{
StatusCode = 200,
Headers = headersDic,
// return the image in Base64 encoding
Body = Convert.ToBase64String(...your image data...),
IsBase64Encoded = true
};
Done.
If you've setup your API to not require authentication, simply type your API link into your browser, and it will display the image. Or put the API link into an IMG tag. e.g. <img src="https://asdf.execute-api.us-east-1.amazonaws.com/live/myapi" />
Note: Even though in step 2 you set the Binary Media Types to */*, API Gateway will still return text if that is what your Lambda is returning.

Luckily, now AWS API Gateway supports binary data, though you also need to update your resource method through the CLI as it is not yet implemented in the Console. This is what you need to do:
In the Method Response of your method
Set Content-Type as image/jpeg in HTTP 200 Status Response
Header
In the Integration Response of your method
Set Content-Type as 'image/jpeg' in Header Mappings. Mind the quotes!
With the AWS CLI, set contentHandling attribute to CONVERT_TO_BINARYon your Integration Response
Check to entire process in this great step-by step guide: https://stackoverflow.com/a/41434295/720665

I've run in to a similar problem. As mentioned you currently can't directly return your image in binary format from your API Gateway endpoint, which would be required for the browser to display it correctly.
However, I solved this by instead having API Gateway return a 302 Redirect, pointing to the correct file in S3. You can have your Lambda function return the url to the file, which is later on mapped to the Location header in API Gateway. The browser will follow the redirect and display the image properly.
There are several ways to implement the redirect, but I did as follow:
Lambda returns an object with the target image like so:
function handler(event, context) {
context.succeed({
location: "https://[bucket-name].s3-eu-west-1.amazonaws.com/myimage.png" });
});
}
Remove the normal '200' Method Response Status from The Integration Response in API Gateway. Replace it with a '302' response status and add the 'Location' header mapped to value 'integration.response.body.location'
Add the 302 status to the Method Response as well

Just to be clear, the client does two different requests:
The first to get the HTML (including the image url).
The second to fetch the image data from the url.
In other words, the image data is not inlined in the HTML.
Based on this knowledge you can have a Lambda (behind the API gateway) as you suggest. The Lambda implementation can have some logic that determines the url to the image stored in S3. However, the Lambda returns JSON data and not HTML (there are workarounds such as return the html in a variable) which makes things trickier, especially for large HTML pages.
I suggest a slightly different approach, since just receiving an image tag will not get you far. I assume you will inline the image tag in a HTML document, probably by using JavaScript. Then you might as well let the API Gateway / Lambda request return a JSON document with the image url and let the JavaScript either update an existing image tag with the new url or generate the tag for you.

It currently isn't possible because you cannot return binary data through the AWS API Gateway.
For this to work, the lambda function would need to return the image data as binary blob, and some meta-information such as the image content type. Then, AWS API Gateway would need to be able to map this to the HTTP response. For example:-
lambda returns:
{
contentType: 'image/png',
image: "encoded binary data"
}
then API gateway would need to map contentType to the 'content-type' header of the response, and put the image data in the body of the response with the right encoding and length.
Unfortunately, it doesn't do this right now. It only maps text encodings like application/json or application/xml as the response type (it is designed for APIs after all).
You could very easily achieve this using ElasticBeanstalk where you have a lot more control over the http response.

I resolved this problem be reconsidering my design. The underlying thinking is that Lambdas are good computing units and bad file servers.
I changed
Client > APIGW > Lambda > Image
into
Client > APIGW > Lambda > SignedURL
then Client(SignedURL) > Image
Since my client was web based, everything was easier after that shift

POST curl request returns syantactic error on Spring tc server

I'm new to Spring, STS, MVC, and web development.
I have a working Spring REST-based web application. I also have a java client which works. I followed these 2 tutorials:
RESTful webservices with Spring
Get started with Spring MVC
What I was able to do is use a java client within the same project and use the RestTemplate postForLocation method. It works! However, now I would like the client to be an iPhone and I don't know how to do that. In the java client, the RestTemplate did a post using the com.project.Transaction class.
Please correct me if I am wrong here. In the XML file, the restTemplate "messageConverter" attribute gets set to jaxbMarshaller which is a Jaxb2Marshaller class with "Transaction" as one of the classes bounded. I don't understand the details of it but I am assuming the RestTemplate takes the Transaction object and marshalls it into XML which gets sent to the server as a POST request.
First question:
Is there any way I can see what the marshalled object (ie: the output) looks like? I'm using STS and please be specific as I am new. For example, maybe the data sent is like <xml ...?
Second question:
I am trying to use curl to make a similar POST request as the java client. This is my curl request:
curl -X POST -H 'Accept:application/xml' -H 'Content-Type: application/xml' http://localhost:8080/BarcodePayment/transactions/ --data "<?xml version="1.0" encoding="UTF-8"?><transaction><amount>3.1</amount><id>5</id><paid>true</id></transaction>"
When I do that, I receive a STATUS 400: syntactically error message.
Third queston:
I would love to be able to understand the details a little bit better. I looked into the source code for RestTemplate from here. In it, the postForLocation method uses HttpEntityRequestCallback which I cannot find any info in Google. I found HttpHeaders in java API doc. However, in RestTemplate, they call getLocation() method which does not exist in the Java API doc under javax.ws.rs.core -> HttpHeaders. How can I understand this stuff better?
I know it's a lot of question. Thanks for the help! Let me know if you need more code snippets and I'll be happy to provide!
Details:
TransactionController
#RequestMapping(method = RequestMethod.POST)
public View addTransaction(#RequestBody Transaction transaction)
{
transactionService.saveTransaction(transaction);
return new RedirectView("/transactionsView/"+ transaction.getId());
}

Your xml is incorrect:
<transaction><amount>3.1</amount><id>5</id><paid>true</id></transaction>
^^^ should be /paid

It turns out I have to use single quotes for the data. After --data "< xml .... ", I change it to ' < xml ... ' and it works.