I have a class which has a method that is receiving an object as a parameter.
This method is invoked via RMI.
public RMIClass extends Serializable {
public RMIMethod(MyFile file){
// do stuff
}
}
MyFile has a property called "body", which is a byte array.
public final class MyFile implements Serializable {
private byte[] body = new byte[0];
//....
public byte[] getBody() {
return body;
}
//....
}
This property holds the gzipped data of a file that was parsed by another application.
I need to decompress this byte array before performing further actions with it.
All the examples I see of decompressing gzipped data assume that I want to write it to the disk and create a physical file, which I do not.
How do I do this?
Thanks in advance.
Wrap your byte array with a ByteArrayInputStream and feed it into a GZipInputStream
Look at those samples, and wherever they're using FileOutputStream, use ByteArrayOutputStream instead. Wherever they're using FileInputStream, use ByteArrayInputStream instead. The rest should be simple.
JDK 9+
private byte[] gzipUncompress(byte[] compressedBytes) throws IOException {
try (InputStream inputStream = new GZIPInputStream(new ByteArrayInputStream(compressedBytes))) {
try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) {
inputStream.transferTo(outputStream);
return outputStream.toByteArray();
}
}
}
Why don't you create your own class that extends OutputStream or , whatever is the archive writing to ?
If you want to write to a ByteBuffer you can do this.
private static void uncompress(final byte[] input, final ByteBuffer output) throws IOException
{
final GZIPInputStream inputGzipStream = new GZIPInputStream(new ByteArrayInputStream(input));
Channels.newChannel(inputGzipStream).read(output);
}
Related
I am trying to send objects to different parts of the environment using base64 encoding, but I get an "Illegal base64 character b7" error.
The encoding code:
public static String serialize(Serializable object) throws IOException {
try (ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(SIZE);
ObjectOutputStream outputStream = new ObjectOutputStream(byteArrayOutputStream)) {
outputStream.writeObject(object);
return Base64.getEncoder().encodeToString(byteArrayOutputStream.toByteArray());
}
}
The decoding code:
public static Serializable deserialize(String base64String) throws ClassNotFoundException, IOException {
byte[] decoded = Base64.getDecoder().decode(base64String);
try (ObjectInputStream inputStream = new ObjectInputStream(new ByteArrayInputStream(decoded))) {
return (Serializable) inputStream.readObject();
}
}
I found the answer, it was in the code that decoded the object - it did not just receive encoded objects, but also regular objects x.x
After putting a check on that, my code worked.
I have a problem that I have not been able to solve and it does not occur to me that it could be.
I have a class to which I am passing an InputStream from the main method, the problem is that when transforming the InputString to String with the class IOUtils.toString of AWS, or with the IOUtils of commons-io, they return
an empty String
No matter what the problem may be, since inside the main class, it works correctly and returns the String it should, but when I use it inside the other class (without having done anything), it returns the empty String to me.
these are my classes:
public class Main {
public static void main(String [] args) throws IOException {
InputStream inputStream = new ByteArrayInputStream("{\"name\":\"Camilo\",\"functionName\":\"hello\"}".getBytes());
OutputStream outputStream = new ByteArrayOutputStream();
LambdaExecutor lambdaExecutor = new LambdaExecutor();
String test = IOUtils.toString(inputStream); //this test variable have "{\"name\":\"Camilo\",\"functionName\":\"hello\"}"
lambdaExecutor.handleRequest(inputStream,outputStream);
}
}
and this:
public class LambdaExecutor{
private FrontController frontController;
public LambdaExecutor(){
this.frontController = new FrontController();
}
public void handleRequest(InputStream inputStream, OutputStream outputStream) throws IOException {
//Service service = frontController.findService(inputStream);
String test = IOUtils.toString(inputStream); //this test variable have "" <-empty String
System.exit(0);
//service.execute(inputStream, outputStream, context);
}
}
I used the debug tool, and the InputStream object is the same in both classes
By the time that you've passed the stream into handleRequest(), you've already consumed the stream:
public static void main(String [] args) throws IOException {
InputStream inputStream = new ByteArrayInputStream("{\"name\":\"Camilo\",\"functionName\":\"hello\"}".getBytes());
OutputStream outputStream = new ByteArrayOutputStream();
LambdaExecutor lambdaExecutor = new LambdaExecutor();
String test = IOUtils.toString(inputStream); //this consumes the stream, and nothing more can be read from it
lambdaExecutor.handleRequest(inputStream,outputStream);
}
When you took that out, the method worked as, as you said in the comments.
If you want the data to be re-useable, you'll have to use the reset() method if you want the same data again, or close and re-open the stream to re-use the object with different data.
// have your data
byte[] data = "{\"name\":\"Camilo\",\"functionName\":\"hello\"}".getBytes();
// open the stream
InputStream inputStream = new ByteArrayInputStream(data);
...
// do something with the inputStream, and reset if you need the same data again
if(inputStream.markSupported()) {
inputStream.reset();
} else {
inputStream.close();
inputStream = new ByteArrayInputStream(data);
}
...
// close the stream after use
inputStream.close();
Always close the stream after you use it, or use a try block to take advantage of AutoCloseable; you can do the same with the output stream:
try (InputStream inputStream = new ByteArrayInputStream(data);
OutputStream outputStream = new ByteArrayOutputStream()) {
lambdaExecutor.handleRequest(inputStream, outputStream);
} // auto-closed the streams
The reason you can't is because you can only read from a stream once.
To be able to read twice, you must call the reset() method for it to return to the beginning. After reading, call reset() and you can read it again!
Some sources don't support resetting it so you would actually have to create the stream again. To check if the source supports it, use the markSupported() method of the stream!
I am trying to write JUnit using MockitoJUnitRunner.
I am passing file id to my function which is downloading file from cloud and returning zip file as download.
here is my code
public void getLogFile(HttpServletResponse response, String id) throws IOException {
response.setContentType("Content-type: application/zip");
response.setHeader("Content-Disposition", "attachment; filename=LogFiles.zip");
ServletOutputStream out = response.getOutputStream();
ZipOutputStream zos = new ZipOutputStream(new BufferedOutputStream(out));
zos.putNextEntry(new ZipEntry(id));
InputStream inputStream = someDao.getFile(id);
BufferedInputStream fif = new BufferedInputStream(inputStream);
int data = 0;
while ((data = fif.read()) != -1) {
zos.write(data);
}
fif.close();
zos.closeEntry();
zos.close();
}
And my JUnit function is
#Mock
private MockHttpServletResponse mockHttpServletResponse;
anyInputStream = new ByteArrayInputStream("test data".getBytes());
#Test
public void shouldDownloadFile() throws IOException {
ServletOutputStream outputStream = mock(ServletOutputStream.class);
when(mockHttpServletResponse.getOutputStream()).thenReturn(outputStream);
=> when(someDao.download(anyString())).thenReturn(anyInputStream);
controller.getLogFile(mockHttpServletResponse, id);
verify(mockHttpServletResponse).setContentType("Content-type: application/zip");
verify(mockHttpServletResponse).setHeader("Content-Disposition","attachment; filename=LogFiles.zip");
verify(atmosdao).download(atmosFilePath);
}
This unit test is passing but I want to verify what is written on outputStream, how can I do it ? as I am writing "test data" to mocked outputStream like
anyInputStream = new ByteArrayInputStream("test data".getBytes());
when(someDao.download(anyString())).thenReturn(anyInputStream);
mockHttpServletResponse.getContentAsString() is giving me null !
Is it possible to assert MockHttpServletResponse which is written using zipoutputStream ? if yes then how can i do it ?
Thanks.
Instead of mocking your OutputStream, you could create a custom one:
public class CustomOutputStream extends ServletOutputStream {
private ByteArrayOutputStream out = new ByteArrayOutputStream();
private String content;
#Override
public void write(int b) throws IOException {
out.write(b);
}
#Override
public void close() throws IOException {
content = new String(out.toByteArray());
out.close();
super.close();
}
public String getContentAsString() {
return this.content;
}
}
This class will store all the bytes written to it and keep them in the content field.
Then you replace this:
ServletOutputStream outputStream = mock(ServletOutputStream.class);
by this:
CustomOutputStream outputStream = new CustomOutputStream();
When your servlet calls getOutputStream(), it will use the custom one, and in the end getContentAsString() will return you the output that was written to your servlet.
Note: the output is zipped, so the String will contain strange characters. If you want the original string, you'll have to unzip it (and in this case I'd use the byte array returned by out.toByteArray() instead of the String, because when you create a String this way you can have encoding problems when calling string.getBytes())
I got what I was looking for...
This is what I did to assert data written to zipoutputStream using powerMockito.
#Test
public void ShouldAttemptToWriteDownloadedFileToZipOutputStream() throws Exception {
InputStream anyInputStream = new ByteArrayInputStream("test data".getBytes());
ServletOutputStream outputStream = mock(ServletOutputStream.class);
BufferedOutputStream bufferedOutputStream = Mockito.mock(BufferedOutputStream.class);
PowerMockito.whenNew(BufferedOutputStream.class).withArguments(outputStream).thenReturn(bufferedOutputStream);
ZipOutputStream zipOutputStream = Mockito.mock(ZipOutputStream.class);
PowerMockito.whenNew(ZipOutputStream.class).withArguments(bufferedOutputStream).thenReturn(zipOutputStream);
BufferedInputStream bufferedInputStream = new BufferedInputStream(anyInputStream);
PowerMockito.whenNew(BufferedInputStream.class).withArguments(anyInputStream).thenReturn(bufferedInputStream);
subjectUnderTest.getLogFile(mockHttpServletResponse, "12345");
int data = 0;
while ((data = bufferedInputStream.read()) != -1) {
verify(zipOutputStream).write(data);
}
}
Thanks Hugo for your help !
Anybody have any idea about,How to handle unstructured data like Audio,Video and Images using Hbase.I tried for this alot but i didn't get any idea.please any help is appreciated.
Option 1: convert image to byte array and you can prepare put request and insert to table. Similarly audio and video files also can be achieved.
See https://docs.oracle.com/javase/7/docs/api/javax/imageio/package-summary.html
import javax.imageio.ImageIO;
/* * Convert an image to a byte array
*/
private byte[] convertImageToByteArray (String ImageName)throws IOException {
byte[] imageInByte;
BufferedImage originalImage = ImageIO.read(new File(ImageName));
// convert BufferedImage to byte array
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ImageIO.write(originalImage, "jpg", baos);
imageInByte = baos.toByteArray();
baos.close();
return imageInByte;
}
Option 2 : You can do that in below way using Apache commons lang API. probably this is best option than above which will be applicable to all objects including image/audio/video etc.. This can be used NOT ONLY for hbase you can save it in hdfs as well
See my answer for more details.
For ex : byte[] mediaInBytes = org.apache.commons.lang.SerializationUtils.serialize(Serializable obj)
for deserializing, you can do this static Object deserialize(byte[] objectData)
see the doc in above link..
Example usage of the SerializationUtils
import java.io.FileInputStream;
import java.io.FileOutputStream;
import org.apache.commons.lang.SerializationUtils;
public class SerializationUtilsTest {
public static void main(String[] args) {
try {
// File to serialize object to it can be your image or any media file
String fileName = "testSerialization.ser";
// New file output stream for the file
FileOutputStream fos = new FileOutputStream(fileName);
// Serialize String
SerializationUtils.serialize("SERIALIZE THIS", fos);
fos.close();
// Open FileInputStream to the file
FileInputStream fis = new FileInputStream(fileName);
// Deserialize and cast into String
String ser = (String) SerializationUtils.deserialize(fis);
System.out.println(ser);
fis.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
Note :jar of apache commons lang always available in hadoop cluster.(not external dependency)
After receiving the uploaded file i want to return a byte[ ] representing the uploaded file i overrode the receiveUpload methode:
/**
* Invoked when a new upload arrives.
*
* #param filename
* the desired filename of the upload, usually as specified
* by the client.
* #param mimeType
* the MIME type of the uploaded file.
* #return Stream to which the uploaded file should be written.
*/
public OutputStream receiveUpload(String filename, String mimeType);
But it returns an OutputStream
Here's the full implementation :
class FileUploaderReceiver implements Receiver{
public File file;
#Override
public OutputStream receiveUpload(String filename,
String mimeType) {
// Create upload stream
OutputStream fos = null; // Stream to write to
try {
// Open the file for writing.
file = new File("/tmp/uploads/" + filename);
fos = new FileOutputStream(file);
} catch (final java.io.FileNotFoundException e) {
new Notification("Could not open file<br/>",
e.getMessage(),
Notification.Type.ERROR_MESSAGE)
.show(Page.getCurrent());
return null;
}
return fos; // Return the output stream to write to
}
So how to get the byte[ ], i know that i can retrieve it using the ByteArrayOutputStream class, but i'am blocked.
Any idea will be appreciated,
Thank you
Wrap the OutputStream with a ByteArrayOutputStream, then use toByteArray().
As kostyan mentioned, you need to use an InputStream (with respect to your method intention). From the InputStream, you can get the bytes, using something like this: http://lasanthals.blogspot.de/2012/09/get-byte-array-from-inputstream.html.
Do note, I provide this as a quick answer, from a quick search, have not tried this one myself.
The problem is how to be notified when the data is fully written to the returned stream.
You can return a ByteArrayOutputStream with overriden close() method. When the stream gets closed, you'll know that the upload was fully written to that stream.
public OutputStream receiveUpload(String filename, String mimeType) {
return new ByteArrayOutputStream() {
#Override
public void close() throws IOException {
byte[] uploadData = toByteArray();
//....
}
};
}