I am creating a Spring Cloud Function that I want to give two inputs, an id and a Multipart file (CSV file) but I am having trouble.
If I choose to send a post with a multipart file the function won't recognise this and gives an error like Failed to determine input for function call with parameters:
With the Postman request being this:
#Bean
public Function<MultipartFile, String> uploadWatchlist() {
return body -> {
try {
return service.convert(body);
}
}
}
I have tried using something more akin to Spring MVC like a request entity object but no luck.
The backup I have (other than Python haha) will be using the binary data post so it will just be a string that has the contents of the file which does work, but requires me to append the id inside to each row of the csv which is a bit messy.
There are other solutions but trying to get this working as Java lambdas are what we want to try and use as first choice.
The infrastructure will be to fix up a manual file upload/verification process that is tedious at the moment and looks like: postman -> load balancer -> lambda -> ecs
The postman/load balancer part will be replaced in future. Ideally have the lambda sorted in Java taking in a file and id.
Thanks for any help :)
Related
I use Java (with Quarkus) for the backend.
Vue3 for the frontend.
API and Websockets to transition data between the two.
I am building a chat message application.
So far I was able to send a message inside the backend under a JSON format, do stuff in Java with this message (which holds the message inside a MessageJson class that I have created, hence the data type is now type of MessageJson), transform it into a String, then and send it back to the frontend. I wish to transform the String in the frontend, into a Json.
Here is my function which send the message back to the front (using Websockets):
public MessageResponseForm processBackend(MessageJson messageJson) {
MessageResponseForm messageResponseForm = new MessageResponseForm();
messages(messageJson);
webSocketConfig.sendMessage("U001", messageJson.toString());
return messageResponseForm;
}
The frontend retrieve it without issue, in the String format.
But I need it back into Json format.
I use the function JSON.parse() in Vue.js but it gives me this:
Which is normal in my opinion, because I should receive something like this from the backend:
"{"toRecipient":"U002","fromSender":"U00555","messageContent":"ewww555"}" .....
But instead I got this:
"MessageJson(toRecipient=U002, fromSender=U00555, messageContent=ewww555, dateCreated=2022-9-23 14:15:57, companyName=test)"
My Class MessageJson does not seems to keep the correct Json format.
How to prevent this?
Do I need to use the #Produces annotations in Quarkus? If yes, where exactly?
I found no other method related to ToString()...
EDIT: Do I need to use the dependency GSON to do the job, or can I do this natively?
I'm looking for a way to access the name of the file being processed during the data transformation within a DoFn.
My pipeline is as shown below:
Pipeline p = Pipeline.create(options);
p.apply(FileIO.match()
.filepattern(options.getInput())
.continuously(Duration.standardSeconds(5),
Watch.Growth.<String>never()))
.apply(FileIO.readMatches()
.withCompression(Compression.GZIP))
.apply(XmlIO.<MyString>readFiles()
.withRootElement("root")
.withRecordElement("record")
.withRecordClass(MyString.class))//<-- This only returns the contents of the file
.apply(ParDo.of(new ProcessRecord()))//<-- I need to access file name here
.apply(ParDo.of(new FormatRecord()))
.apply(Window.<String>into(FixedWindows.of(Duration.standardSeconds(5))))
.apply(new CustomWrite(options));
Each file that is processed is an XML document. While processing the content, I need access to the name of the file being processed too to include in the transformed record.
Is there a way to achieve this?
This post has a similar question, but since i'm trying to use XmlIO I havent found a way to access the file metadata.
Below is the approach I found online, but not sure if there is a way to use it in the pipeline described above.
p.apply(FileIO.match()
.filepattern(options.getInput())
.continuously(Duration.standardSeconds(5),
Watch.Growth.<String>never()))//File Metadata
.apply(FileIO.readMatches()
.withCompression(Compression.GZIP))//Readable Files
.apply(MapElements
.into(TypeDescriptors.kvs(TypeDescriptors.strings(),new TypeDescriptor<ReadableFile>() {} ))
.via((ReadableFile file) -> {
return KV.of(file.getMetadata().resourceId().getFilename(),file);
})
);
Any suggestions are highly appreciated.
Thank you for your time reviewing this.
EDIT:
I took Alexey's advice and implemented a custom XmlIO. It would be nice if we could just extend the class we need and override the appropriate method. However, in this specific case, there was a reference to one method which was protected within the sdk because of which I couldn't easily override what i needed and instead ended up copying a whole bunch of files. While this works for now, I hope in future there is a more straighforward way to access the file metadata in these IO implementations.
I don't think it's possible to do "out-of-box" with a current implementation of of XmlIO since it returns a PCollection<T> where T is a type of your xml record and, if I'm not mistaken, there is no way to add a file name there. Though, you still can try to "reimplement" a ReadFiles and XmlSource in a way that it will return parsed payload and input file metadata.
I am using ng-file-upload AngularJs API to upload multiple files to server.But this is the traditional way to do it.But my requirement is that i dont need to store files in a server as it is.I have a REST end point that responsible for store user input data to DB.Along with the REST request i pass the file Array object with other forms values.When data comes to REST end point it access each attributes and store data.When it tried to read File Array obj i can not read the file content for each file.
Sample File Upload Code
jsfiddle
Note that i just want to pass only $scope.files along with the REST request.Please let me know how can i read file content values from server side reading file Array in Java.If you guys know any better way to do this please share your ideas.
REST Service Code Snippet
#POST
#Path("/manual")
#Produces(MediaType.APPLICATION_JSON)
public boolean insertResults(testVO testResult) {
for(Object o:testVO.getFiles()){
LinkedHashMap<String, String> l=(LinkedHashMap<String, String>) o;
System.out.println(l.get("result"));
}
}
Note: testVO.getFiles() type is Object[] array.
In my preceding code i convert object into LinkedHashMap and access the necessary fields like size,type,etc.But my requirement is that how can i get the content belong to that file.
Using JQuery and Spring's #ModelAndView annotation for the controller.
I'm trying to code a process in which the user clicks an icon and if a certain criteria on the DB is met, a zip file will be produced on the server containing a bunch of files, then this zip file should be sent to the browser for saving.
If the criteria isn't met, then an error message should be sent to the browser telling there isn't any file to be created and produced.
However if I use JQuery' .post method, I can receive the error message (if that is the case) but never the zip binary file.
If I use a regular Href Link I can receive the file (if that is the case) but don't know how to receive the message when the file cannot be produced.
Is there an alternative or a standard way to do this?
Thanks for your support!
-Gabriel.
You should probably split your server-side method in two:
the first one validates the criteria. If unsuccessful, it notifies of an exception, otherwise it returns a URL to the method in next point
the second one actually returns the zip file
In your frontend, the code will look something like this:
$.post(urlToPoint1, data, function(response) {
if (response.success) {
// download the file using the url provided
// (pointing to method described in point 2)
window.location.href = response.url;
}
else {
alert('whatever');
}
});
I asked something like this previously, but upon re-reading my original post, it was not easy to understand what I was really asking. I have the following situation. We have (or at least I'm trying to get working) a custom file upload procedure that will take in the file, a set number of 'known' metadata values (and they will always be there), as well as potentially an unknown number of additional metadata values. The service that exists currently uses the Jersey framework (1.16)
I currently have both client and server code that handles dealing with the file upload portion and the known metadata values (server code below)
#POST
#Path("asset/{obfuscatedValue0}/")
#Consumes(MediaType.MULTIPART_FORM_DATA)
public UUID uploadBlob(#PathParam("obfuscatedValue0") Integer obfuscatedValue0,
#FormDataParam("obfuscatedValue1") String obfuscatedValue1,
#FormDataParam("obfuscatedValue2") String obfuscatedValue2,
#FormDataParam("obfuscatedValue3") String obfuscatedValue3,
#FormDataParam("obfuscatedValue4") String obfuscatedValue4,
#FormDataParam("obfuscatedValue5") String obfuscatedValue5,
#FormDataParam("file") InputStream uploadedInputStream) {
.....
}
...and excerpt of client code:
Builder requestBuilder = _storageService
.path("asset")
.path(obfuscatedValue0.toString())
.type(MediaType.MULTIPART_FORM_DATA)
.accept(MediaType.APPLICATION_JSON);
FormDataMultiPart part = new FormDataMultiPart()
.field("file", is, MediaType.TEXT_PLAIN_TYPE) // 'is' is an inputstream from earlier in code.
.field("obfuscatedValue1", obfuscatedValue1)
.field("obfuscatedValue2", obfuscatedValue2)
.field("obfuscatedValue3", obfuscatedValue3)
.field("obfuscatedValue4", obfuscatedValue4)
.field("obfuscatedValue5", obfuscatedValue5);
storedAsset = requestBuilder.post(UUID.class, part);
However, I need to pass a map of additional parameters that will have an unknown number of values/names. From what I've seen, there is no easy way to do this using the FormDataParam annotation like my previous example.
Based upon various internet searches related to Jersey file uploads, I've attempted to convert it to use MultivaluedMap with the content type set to "application/x-www-form-urlencoded" so it resembles this:
#POST
#Path("asset/{value}/")
#Consumes("application/x-www-form-urlencoded")
public UUID uploadBlob(#PathParam(value), MultivaluedMap<String,String> formParams) {
....
}
It's my understanding that MultivaluedMap is intended to obtain a general map of form parameters (and as such, cannot play nicely together in the same method bearing #FormDataParam annotations.) If I can pass all this information from the Client inside some sort of map, I think I can figure out how to handle parsing the map to grab and 'doMagic()' on the data to get what I want done; I don't think I'll have a problem there.
What I AM fairly confused about is how to format the request client-side code when using this second method within the jersey framework. Can anyone provide some guidance for the situation, or some suggestions on how to proceed? I'm considering trying the solution proposed here and developing a custom xml adapter to deal with this situation, and sending xml instead of multipart-form-data but I'm still confused how this would interact with the InputStream value that will need to be passed. It appears the examples with MultivaluedMap that I've seen only deal with String data.