I'm working in a Spring Boot api that can receive very large objects and try to save it in a MongoDB database. Because of this the program sometimes throws me the next error:
org.bson.BsonMaximumSizeExceededException: Payload document size is larger than maximum of 16793600.
I'd read that MongoDB only permits objects of size below 16MB, this is very inconvenient for my system because an object can easily surpass this gap. To solve this I had read about GridFS, technology that allows to surpass the 16MB files gap.
Now I'm trying to implement GridFS in my system but I only had seen examples using files to save in the database, something like this:
gridFsOperations.store(new FileInputStream("/Users/myuser/Desktop/text.txt"), "myText.txt", "text/plain", metaData);
But I want to do is not to take the data from a file, but to the api to receive a object and save it, something like this:
#PostMapping
public String save(#RequestBody Object object){
DBObject metaData = new BasicDBObject();
metaData.put("type", "data");
gridFsOperations.store(object, metaData);
return "Stored successfully...";
}
Is it a posible way to doing this?
Get an InputStream from the request and pass it to a GridFSBucket. Here's a rough example:
In your controller:
#PostMapping
public ResponseEntity<String> uploadFile(MultipartHttpServletRequest request)
{
Iterator<String> iterator = request.getFilenames();
String filename = iterator.next();
MultipartFile mf = request.getFile(filename);
// I always have a service layer between controller and repository but for purposes of this example...
myDao.uploadFile(filename, mf.getInputStream());
}
In your DAO/repository:
private GridFSBucket bucket;
#Autowired
void setMongoDatabase(MongoDatabase db)
{
bucket = GridFSBuckets.create(db);
}
public ObjectId uploadFile(String filename, InputStream is)
{
Document metadata = new Document("type", "data");
GridFSUploadOptions opts = new GridFSUploadOptions().metadata(metadata);
ObjectId oid = bucket.uploadFromStream(filename, is, opts);
try
{
is.close();
}
catch (IOException ioe)
{
throw new UncheckedIOException(ioe);
}
return oid;
}
I paraphrased this from existing code so it may not be perfect but will be good enough to point you in the right direction.
Related
I would like to write a Spring controller method which returns an image from storage. Below is my current version, but it has two problems:
The #GetMapping annotation requires the 'produces' parameter which is a string array of media types. The program does not work if that parameter is not present; it just displays the image data as text. The problem is that if I want to support an additional media type then I have to recompile the program. Is there a way to set up the 'produces' media type from inside the viewImg method?
The code below will display any image type except svg, which will display only the message "The image cannot be displayed because it contains errors". The web browser (Firefox) identifies it as media type "webp". However, if I remove all media types from the 'produces' string array except the "image/svg+xml" entry, the image is displayed.
Please advise how to write a controller method that is more general (so that it works with any media type) and does not have issues with svg media type.
Here is my test code:
#GetMapping(value = "/pic/{id}",
produces = {
"image/bmp",
"image/gif",
"image/jpeg",
"image/png",
"image/svg+xml",
"image/tiff",
"image/webp"
}
)
public #ResponseBody
byte[] viewImg(#PathVariable Long id) {
byte[] data = new byte[0];
String inputFile = "/path/to/image.svg";
try {
InputStream inputStream = new FileInputStream(inputFile);
long fileSize = new File(inputFile).length();
data = new byte[(int) fileSize];
inputStream.read(data);
} catch (IOException e) {
e.printStackTrace();
}
return data;
}
I recommend FileSystemResource for handling file contents. You can avoid .contentType(..) started line if you don't want to send Content-Type value.
#GetMapping("/pic/{id}")
public ResponseEntity<Resource> viewImg(#PathVariable Long id) throws IOException {
String inputFile = "/path/to/image.svg";
Path path = new File(inputFile).toPath();
FileSystemResource resource = new FileSystemResource(path);
return ResponseEntity.ok()
.contentType(MediaType.parseMediaType(Files.probeContentType(path)))
.body(resource);
}
In my code currently, I get data from the database and then I write a file out of the data. I have this kind of camel route and working solution:-
private static final String INPUT_FILE_DIRECTORY_URI = "file:" + System.getProperty("user.home")
+ "/data/cdr/?noop=false";
private static final String SFTP_SERVER = "sftp://" +System.getProperty("user.name")
+ "#sftp_server_url/data/cdr/?privateKeyFile=~/.ssh/id_rsa&passiveMode=true";
from(INPUT_FILE_DIRECTORY_URI)
.streamCaching()
.log("Sending file to local sftp")
.to(SFTP_SERVER);
I don't want to write a file in the local disk. Instead, I want to write file data directly to the SFTP server. I don't know how to do it? But I imagine it should be possible to do it. Can you tell me is it possible? If yes, how to do it?
I managed to solve this problem in another way. It is more suitable for my particular problem.
byte[] csvData = csvStringBuilder.toString().getBytes();
Routes.withProducer(producer)
.withHeader(Exchange.FILE_NAME, myCsvFile.csv)
.withBody(csvData)
.to(SFTP_SERVER).request(byte[].class);
You shouldn't use streamCaching unless you really using it. It store your file in memory, use it if you need to consume multiples times your input.
You can use Jpa component or a custom bean getting your data. Load it from database and then send it to your ftp server.
With Jpa :
#Entity
#NamedQuery(name = "data", query = "select x from Data x where x.id = 1")
public class Data { ... }
After that you can define a consumer uri like this one:
from("jpa://org.examples.Data?consumer.namedQuery=data")
.to("SFTP_SERVER");
EDIT : to convert a list to csv and send it to ftp :
from("jpa://org.examples.Data?consumer.namedQuery=data")
.marshal()
.csv()
.to("sftp://" +System.getProperty("user.name") +
"#sftp_server_url/data/cdr/myFile.csv?" +"privateKeyFile=~/.ssh/id_rsa&passiveMode=true");
See CSV component who convert a list to a csv file.
Yes it is possible :) To do this send the file inputStream in a camel DIRECT component and in the associated route make the copy to FTP. I use this case, to upload a file and directly copy it to ftp with from(directInputStreamName).to(yourFtpUri). This is an sample code :
Your service
#Service
public class FileService {
#Produce(uri = PfnumDownloadConstants.CAMEL_DIRECT_UPLOAD)
private ProducerTemplate producer;
public void sendFileToFtp(File fileToSend, String ftpDestinationUri) throws IOException {
Map<String, Object> headers = new HashMap<>();
//In this variable you can init the ftp destination uri or you can hard code it in route
headers.put("destinationUri", ftpDestinationUri);
//set filename to name your file in ftp
headers.put(Exchange.FILE_NAME_ONLY, file.getName());
InputStream targetStream = new FileInputStream(file);
//send stream as body and list of headers to direct
producer.sendBodyAndHeaders(targetStream, headers);
}
}
Your Camel route
#Component
public class FileUploadRoute extends RouteBuilder {
#Override
public void configure() throws Exception {
//Manage camel exception in a dedicated processor
onException(Exception.class).process(exceptionProcessor).log("error :: ${exception}");
from(CAMEL_DIRECT_UPLOAD)
.log("file copy to ftp '${header.CamelFileNameOnly}' in process")
.toD("file:/mnt?fileName=${header.CamelFileNameOnly}&delete=false")
.log("copy done");
}
}
I am using a document converter api called cloudconvert. They don't have an official java library, but a third party java option. I needed a little customization so I cloned the github project and added it to my project. I am sending cloudconvert a .epub file and getting a .pdf file in return. If I use the default settings it works without issue and properly converts my .epub to a .pdf. Here is the code that makes it happen.
Here is what triggers the conversion:
// Create service object
CloudConvertService service = new CloudConvertService("api-key");
// Create conversion process
ConvertProcess process = service.startProcess(convertFrom, convertTo);
// Perform conversion
//convertFromFile is a File object with a .epub extension
process.startConversion(convertFromFile);
// Wait for result
ProcessStatus status;
waitLoop:
while (true) {
status = process.getStatus();
switch (status.step) {
case FINISHED:
break waitLoop;
case ERROR:
throw new RuntimeException(status.message);
}
// Be gentle
Thread.sleep(200);
}
//Download result
service.download(status.output.url, convertToFile);
//lean up
process.delete();
startConversion() calls:
public void startConversion(File file) throws ParseException, FileNotFoundException, IOException {
if (!file.exists()) {
throw new FileNotFoundException("File not found: " + file);
}
startConversion(new FileDataBodyPart("file", file));
}
Which calls this to actually send the POST request using jersey:
private void startConversion(BodyPart bodyPart) {
if (args == null) {
throw new IllegalStateException("No conversion arguments set.");
}
MultiPart multipart = new FormDataMultiPart()
.field("input", "upload")
.field("outputformat", args.outputformat)
.bodyPart(bodyPart);
//root is a class level WebTarget object
root.request(MediaType.APPLICATION_JSON).post(Entity.entity(multipart, multipart.getMediaType()));
}
Up to this point everything is working. My problem is that the when the conversion happens the .pdf that returns has very small margins. cloudconvert provides a way to change those margins. You can send in an optional json param converteroptions and set the margins manually. I have tested this out using postman and it works without issue, I was able to get a properly formatted margin document. So know this is possible. Here is the POSTMAN info I used:
#POST : https://host123d1qo.cloudconvert.com/process/WDK9Yq0z1xso6ETgvpVQ
Headers: 'Content-Type' : 'application/json'
Body:
{
"input": "base64",
"file": "0AwAAIhMAAAAA", //base64 file string that is much longer than this
"outputformat": "pdf",
"converteroptions": {
"margin_bottom": 75,
"margin_top": 75,
"margin_right": 50,
"margin_left": 50
}
}
Here are my attempts at getting the POST request formatted properly, I'm just not very experienced with jersey and the couple of answers I did find on stackoverflow didn't work for me.
Attempt 1, I tried adding the json string as a Multipart.field. It didn't give me any errors and still returned a converted .pdf file, but the margins didn't get changed so I must not be sending it back right.
private void startConversion(BodyPart bodyPart) {
String jsonString = "{\"margin_bottom\":75,\"margin_top\":75,\"margin_right\":50,\"margin_left\":50}";
MultiPart multipart = new FormDataMultiPart()
.field("input", "upload")
.field("outputformat", args.outputformat)
.field("converteroptions", jsonString)
.bodyPart(bodyPart);
root.request(MediaType.APPLICATION_JSON).post(Entity.entity(multipart, multipart.getMediaType()));
}
Attempt 2, when I had it working in POSTMAN it was using the 'input' type as 'base64' so I tried changing it to that but it this time it doesn't return anything at all, no request errors, just a timeout error at the 5 minute mark.
//I pass in a File object rather than the bodypart object.
private void startConversion(File file) {
byte[] encoded1 = Base64.getEncoder().encode(FileUtils.readFileToByteArray(file));
String encoded64 = new String(encoded1, StandardCharsets.US_ASCII);
String jsonString = "{\"margin_bottom\":75,\"margin_top\":75,\"margin_right\":50,\"margin_left\":50}";
MultiPart multipart = new FormDataMultiPart()
.field("input", "base64")
.field("outputformat", args.outputformat)
.field("file", encoded64)
.field("converteroptions", jsonString);
root.request(MediaType.APPLICATION_JSON).post(Entity.entity(multipart, multipart.getMediaType()));
}
Attempt 3, after some googling on how to properly send jersey json post requests I changed the format. This time it returned a 400 bad request error.
private void startConversionPDF(File file) throws IOException {
byte[] encoded1 = Base64.getEncoder().encode(FileUtils.readFileToByteArray(file));
String encoded64 = new String(encoded1, StandardCharsets.US_ASCII);
String jsonString = "{\"input\":\"base64\",\"file\":\"" + encoded64 + "\",\"outputformat\":\"pdf\",\"converteroptions\":{\"margin_bottom\":75,\"margin_top\":75,\"margin_right\":50,\"margin_left\":50}}";
root.request(MediaType.APPLICATION_JSON).post(Entity.json(jsonString));
}
Attempt 4, Someone said you don't need to manually use a jsonString you should use serializable java beans. So I created the corresponding classes and made the request like shown below. Same 400 bad request error.
#XmlRootElement
public class PDFConvert implements Serializable {
private String input;
private String file;
private String outputformat;
private ConverterOptions converteroptions;
//with the a default constructor and getters/setters for all
}
#XmlRootElement
public class ConverterOptions implements Serializable {
private int margin_bottom;
private int margin_top;
private int margin_left;
private int margin_right;
//with the a default constructor and getters/setters for all
}
private void startConversionPDF(File file) throws IOException {
byte[] encoded1 = Base64.getEncoder().encode(FileUtils.readFileToByteArray(file));
String encoded64 = new String(encoded1, StandardCharsets.US_ASCII);
PDFConvert data = new PDFConvert();
data.setInput("base64");
data.setFile(encoded64);
data.setOutputformat("pdf");
ConverterOptions converteroptions = new ConverterOptions();
converteroptions.setMargin_top(75);
converteroptions.setMargin_bottom(75);
converteroptions.setMargin_left(50);
converteroptions.setMargin_right(50);
data.setConverteroptions(converteroptions);
root.request(MediaType.APPLICATION_JSON).post(Entity.json(data));
}
I know this is quite the wall of text, but I wanted to show all the different things I tried so that I wouldn't waste anyone's time. Thank you for any help or ideas you might have to make this work. I really want to make it work with jersey because I have several other conversions I do that work perfectly, they just don't need any converteroptions. Also I know its possible because it works when manually running the process through POSTMAN.
Cloudconvert api documentation for starting a conversion
Github repo with the recommended 3rd party java library I am using/modifying
I finally figured it out. Hours of trial and error. Here is the code that did it:
private void startConversionPDF(File file) throws IOException {
if (args == null) {
throw new IllegalStateException("No conversion arguments set.");
}
PDFConvert data = new PDFConvert();
data.setInput("upload");
data.setOutputformat("pdf");
ConverterOptions converteroptions = new ConverterOptions();
converteroptions.setMargin_top(60);
converteroptions.setMargin_bottom(60);
converteroptions.setMargin_left(30);
converteroptions.setMargin_right(30);
data.setConverteroptions(converteroptions);
MultiPart multipart = new FormDataMultiPart()
.bodyPart(new FormDataBodyPart("json", data, MediaType.APPLICATION_JSON_TYPE))
.bodyPart(new FileDataBodyPart("file", file));
root.request(MediaType.APPLICATION_JSON).post(Entity.entity(multipart, multipart.getMediaType()));
}
I have a webapp which make REST call for respective functionalities.
Say, I have a webapp in which a user registers and a KeyValue pair is generated which is required for authentication during ssh. Now this KeyValue pair is unique.
And the other web page of the same application I want those KeyValue pair to be downloaded as a ppk file to ssh. "So that particular object is required".
So is there a way to store the object.
I do have a Postgres DB.
What are my possible options?
Edit: The Object I want to store, is a third-party implemented library JSch (which is an full implementation of SSH-2), so simple Key-Value pair might not be solution I guess!)
REST is stateless so is not possible to mantain it on the server side!
You can create a token that is the encryption of the couple key, value on the server side and send it to the client (browser).
The browser is responsible to mantain the token and send it to the server if needed.
The server can decrypt the token and retrieve the original key value couple.
This will grant you that the client can't access to the couple key value because it is encrypted in a manner that the client can't read it.
Note If you need to mantain those values only for a limited quantity of time you can encrypt in the token also a value corresponding to the creation of token or to last access to the token.
This technique is generally used to mantain "sessions" on client side in an application that use only REST calls, but needs to maintain informations related to the current user.
If you only want to store a single key/value pair, then you can store it in the session. If you want to store multiple pairs, it works similar, but you'd store a collection of pairs in the session. Have a look at HttpServletRequest.getSession() for details.
If you want to share the key/value pair across all sessions (other users, multiple browsers) and you only have a single web server, just use a static variable.
If you want to share the key/value pair across all sessions and you have multiple servers, then you'll have to store it in the database (unless you'd consider build some data exchange protocol between servers, but I doubt that would be the simplest thing you could do).
You should store the key value pair in DB. You can use redis for faster access to this pair similar to a cache and auto delete based on expiration time you want to set.
This is the same way data is cached by servers in sessions. Since REST APIs are dateless, you have this alternate which should work just fine.
As I wanted a key for ssh'ing into my machine which is provided by KeyPair Object of JSch. I did stored it in my database by serializable for future applications like this,
#Entity
#Table(name ="keypairs")
public class KeyPairDomain {
private Long id;
private String username;
private byte[] byteObject;
private KeyPair keypair;
public KeyPairDomain(KeyPair kpair, String username) {
super();
this.setUsername(username);
this.keypair = kpair;
}
#Transient
public KeyPair getKeypair() {
return keypair;
}
public void setKeypair(KeyPair complexObject) {
this.setKeypair(complexObject);
ByteArrayOutputStream baos;
ObjectOutputStream out;
baos = new ByteArrayOutputStream();
try {
out = new ObjectOutputStream(baos);
out.writeObject(complexObject);
out.close();
} catch (IOException e) {
e.printStackTrace();
}
this.byteObject = baos.toByteArray();
}
#Column(columnDefinition = "bytea")
public byte[] getByteObject() {
return byteObject;
}
public void setByteObject(byte[] byteObject) {
ByteArrayInputStream bais;
ObjectInputStream in;
try {
bais = new ByteArrayInputStream(byteObject);
in = new ObjectInputStream(bais);
setKeypair((KeyPair) in.readObject());
in.close();
} catch (IOException ex) {
ex.printStackTrace();
} catch (ClassNotFoundException ex) {
ex.printStackTrace();
}
this.byteObject = byteObject;
}
public String getUsername() {
return username;
}
#Column(name = "username")
public void setUsername(String username) {
this.username = username;
}
#Id
#GeneratedValue
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
}
but I stored a ppk file in a tmp folder of any machine type. And when required I use that location to fetch that key.
#Consumes(MediaType.APPLICATION_JSON)
#Produces(MediaType.APPLICATION_OCTET_STREAM)
#POST
#Path("/getCredentials")
public Response getCredentials(HashMap<String, String> userDetails) {
String username = userDetails.get("username");
File homedir = new File(System.getProperty("user.home"));
File file = new File(homedir, "credentials/" + username + "/Key.ppk");
System.out.println("File Path: "+file.getAbsolutePath());
System.out.println("File Name: "+file.getName());
return Response.ok(file, MediaType.APPLICATION_OCTET_STREAM)
.header("Content-Disposition", "attachment; filename=" +file.getName()).build();
}
I have the following code:
restTemplate.getForObject("http://img.championat.com/news/big/l/c/ujejn-runi_1439911080563855663.jpg", File.class);
I especially took image which doesn't require authorization and available absolutely for all.
when following code executes I see the following stacktrace:
org.springframework.web.client.RestClientException: Could not extract response: no suitable HttpMessageConverter found for response type [class java.io.File] and content type [image/jpeg]
at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:108)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:559)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:512)
at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:243)
at com.terminal.controller.CreateCompanyController.handleFileUpload(CreateCompanyController.java:615)
what do I wrong?
Image is a byte array, so you need to use byte[].class object as a second argument for RestTemplate.getForObject:
String url = "http://img.championat.com/news/big/l/c/ujejn-runi_1439911080563855663.jpg";
byte[] imageBytes = restTemplate.getForObject(url, byte[].class);
Files.write(Paths.get("image.jpg"), imageBytes);
To make it work, you will need to configure a ByteArrayHttpMessageConverter in your application config:
#Bean
public RestTemplate restTemplate(List<HttpMessageConverter<?>> messageConverters) {
return new RestTemplate(messageConverters);
}
#Bean
public ByteArrayHttpMessageConverter byteArrayHttpMessageConverter() {
return new ByteArrayHttpMessageConverter();
}
I've tested this in a Spring Boot project and the image is saved to a file as expected.
If you simply need to get an image from a URL, Java comes with the javax.imageio.ImageIO class, which contains this method signature:
public static BufferedImage read(URL var0) throws IOException;
example use:
try {
BufferedImage image = ImageIO.read(new URL("http://www.foo.com/icon.png"));
int height = image.getHeight();
int width = image.getWidth();
} catch (IOException e) {}
The RestTemplate is expecting a class (e.g. some in-memory representation) to convert the response from the server into. For example, it could convert a response like:
{id: 1, name: "someone"}
into a class like:
class NamedThing {
private int id;
private String name;
// getters/setters go here
}
By calling:
NamedThing thing = restTemplate.getForObject("/some/url", NamedThing.class);
But, what you seem to really want to do is to take the response from the server and stream it directly to a file. Various methods exist to get the response body of your HTTP request as something like an InputStream that you can read incrementally, and then write out to an OutputStream (e.g. your file).
This answer shows how to use IOUtils.copy() from commons-io to do some the dirty work. But you need to get an InputStream of your file... A simple way is to use an HttpURLConnection. There's a tutorial with more information.