I'm trying to write and read a gzip to/from Redis. The problem is that I tried saving the read bytes to a file and opening it with gzip - it's invalid. The strings are also different when looking at them in the Eclipse console.
Here's my code:
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import redis.clients.jedis.Jedis;
public class TestRedis
{
public static void main(String[] args) throws IOException
{
String fileName = "D:/temp/test_write.gz";
String jsonKey = fileName;
Jedis jedis = new Jedis("127.0.0.1");
byte[] jsonContent = ReadFile(new File(fileName).getPath());
// test-write data we're storing in redis
FileOutputStream fostream = new FileOutputStream("D:/temp/test_write_before_redis.gz"); // looks ok
fostream.write(jsonContent);
fostream.close();
jedis.set(jsonKey.getBytes(), jsonContent);
System.out.println("writing, key: " + jsonKey + ",\nvalue: " + new String(jsonContent)); // looks ok
byte[] readJsonContent = jedis.get(jsonKey).getBytes();
String readJsonContentString = new String(readJsonContent);
FileOutputStream fos = new FileOutputStream("D:/temp/test_read.gz"); // invalid gz file :(
fos.write(readJsonContent);
fos.close();
System.out.println("n\nread json content from redis: " + readJsonContentString);
}
private static byte[] ReadFile(String aFilePath) throws IOException
{
Path path = Paths.get(aFilePath);
return Files.readAllBytes(path);
}
}
You are using Jedis.get(String) to read which includes an inner UTF-8 conversion. But using Jedis.set(byte[], byte[]) to write does not include such conversion. The mismatch could be because of this reason. If so, you can try Jedis.get(byte[]) to read from redis to skip UTF-8 conversion. E.g.
byte[] readJsonContent = jedis.get(jsonKey.getBytes());
Related
I'm having a terrible time dealing with image en/de-coding in TensorFlow Java. I need to handle the B64 because I have a saved model from Google AutoML vision that expects that input format. Just to be explicit the Maven import is:
<dependency>
<groupId>org.tensorflow</groupId>
<artifactId>tensorflow-core-platform</artifactId>
<version>0.4.0</version>
</dependency>
and the following minimal example shows the root issue:
import java.io.File;
import java.io.InputStream;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import javax.activation.MimetypesFileTypeMap;
import org.apache.commons.codec.binary.Base64;
import org.tensorflow.Graph;
import org.tensorflow.Output;
import org.tensorflow.Session;
import org.tensorflow.op.image.DecodeJpeg;
import org.tensorflow.op.image.DecodeJpeg.Options;
import org.tensorflow.types.TString;
import org.tensorflow.types.TUint8;
public class tensorflowLoadMinimal{
public static void main(String[] args) throws Exception{
// Get a public JPG locally for example purposes
String imgUrl = "https://file-examples-com.github.io/"
+ "uploads/2017/10/file_example_JPG_100kB.jpg";
String localPath = "/tmp/imgFile.jpg";
InputStream in = new URL(imgUrl).openStream();
Files.copy(in, Paths.get(localPath), StandardCopyOption.REPLACE_EXISTING);
// Sanity checking the JPG; base64 encode
File f = new File(localPath);
System.out.println("Mime Type of " + f.getName() + " is " +
new MimetypesFileTypeMap().getContentType(f));
byte[] fileBytes = Files.readAllBytes(Paths.get(localPath));
String encodedString = Base64.encodeBase64String(fileBytes);
// Make b64 string a tensor; wrap in TF structs
Graph graph = new Graph();
Session s = new Session(graph);
TString tensor = TString.scalarOf(encodedString);
Output<TString> tensorAsOut = graph
.opBuilder("Const", "imgPixels", graph.baseScope())
.setAttr("dtype", tensor.dataType())
.setAttr("value", tensor)
.build()
.<TString> output(0);
// Try to decode b64 as Jpeg... and fail
Options[] opts = new Options[1];
opts[0] = DecodeJpeg.channels(3L);
DecodeJpeg dJpg = DecodeJpeg.create(graph.baseScope(), tensorAsOut, opts);
Output<TUint8> jpgOut = dJpg.image();
s.run(jpgOut);
s.close();
}
}
It confirms I have a JPG file, and then fails to do the decoding, complaining the input format is not an image file, with succinct output:
Mime Type of imgFile.jpg is image/jpeg
...
Exception in thread "main" org.tensorflow.exceptions.TFInvalidArgumentException: Unknown image file format. One of JPEG, PNG, GIF, BMP required.
[[{{node DecodeJpeg}}]]
at org.tensorflow.internal.c_api.AbstractTF_Status.throwExceptionIfNotOK(AbstractTF_Status.java:87)
...
at orc.tensorflowLoadMinimal.main(tensorflowLoadMinimal.java:55)
Where am I going wrong?
It reads:
Unknown image file format. One of JPEG, PNG, GIF, BMP required.
Which can be fixed either by removing this superfluous part:
// Try to decode b64 as Jpeg... and fail
Options[] opts = new Options[1];
opts[0] = DecodeJpeg.channels(3L);
DecodeJpeg dJpg = DecodeJpeg.create(graph.baseScope(), tensorAsOut, opts);
Output<TUint8> jpgOut = dJpg.image();
s.run(jpgOut);
s.close();
... or by passing the expected parameter JPEG, likely into DecodeJpeg.create() or opts.
Given that it is now possible to handle binary data in Amazon Api Gateway and Amazon Lambda, I wanted to try to make an Amazon Lambda endpoint which returned an Excel spreadsheet. It is entirely possible to do so using node/js, as demonstrated here. Unfortunately, any time I try to do this using Java, it falls to pieces.
My initial attempt was to create a simple workbook using apache XSSFWorkbook, write it to the output stream provided by RequestStreamHandler, and done.
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestStreamHandler;
import org.apache.poi.ss.usermodel.Workbook;
import org.apache.poi.xssf.usermodel.XSSFWorkbook;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
public class FileRequestHandler implements RequestStreamHandler {
public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context)
throws IOException {
Workbook wb = new XSSFWorkbook();
String sheetName = "Problem sheet";
wb.createSheet(sheetName);
wb.write(outputStream);
}
}
When tested locally, the output stream can be piped to a file resulting in a valid output excel file.
import com.amazonaws.util.StringInputStream;
import org.junit.Test;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
public class FileRequestHandlerTest {
#Test
public void shouldCreateExcelFile() throws IOException {
FileRequestHandler fileRequestHandler = new FileRequestHandler();
InputStream inputStream = new StringInputStream("hello world");
String fileName = "FileRequestLambda";
String path = fileName + ".xlsx";
FileOutputStream fileOutputStream = new FileOutputStream(path);
fileRequestHandler.handleRequest(inputStream, fileOutputStream, TestUtils.createContext());
fileOutputStream.close();
}
}
But when I run it in Amazon Lambda, I get malformed binary output:
PKn��I_rels/.rels���j�0��}
�{㴃1F�^Ơ�2��l%1I,c�[��3�l
l�����H��4��R�l��·����q}*�2�������;�*��
t"�^�l;1W)�N�iD)ejuD�cKz[:}g����#:�
�3����4�7N�s_ni�G�M*7�����2R�+�� �2�/�����b��mC�Pp�ֱ$POyQ�抒�DsZ��IС�'un���~�PK����OPKn��I[Content_Types].xml�SMO1��+6��m��1���G%��β
�J[���MDL0�S;yo�{3i�Ӎ5�c��5lć�B'��nѰ��S}˪��)0�aÜg��`<�L��.�p'D�ZH�t��>Z�Tƅ ��#q=��]F��\4�=`+���P�!-!S.�v�#��+�����N�tEV=nHe7���S,;K]_h7Q+�W8߶Z��re��c�U�����}�����g�&A��,���H�$�B<��`�"�Jb���"���I�N�1���A���CI�#��܂v��?|\�{��`�b������$�c�D��|2�PKKB�>'PKn��IdocProps/app.xmlM��
�0D�~EȽ��ADҔ���A? ��6�lB�J?ߜ���0���ͯ��)�#��H6���V>��$;�SC
;̢(�ra�g�l�&�e��L!y�%��49��`_���4G���F��J��Wg
�GS�b����
~�PK�|wؑ�PKn��IdocProps/core.xmlm��J�0F��!�m�V����(���Ż��m��!�v}{ӺVP/g��a��wG5�wp~4��4�1-�u���n��c�גOFC����6��e�888c��<�홰
B��/P�g��q�b��!��'��W�)��"
�<p�S��I)Ŧ�onZR�#��Ќ�6�S�߅u��G?n�<��\�\����ۛ���t���p|��f� Q4��ac&ߓ��������i��"�UG+vV��z�ɯ���U�^�H#�����IM�$�&�PK����PKn��Ixl/sharedStrings.xml=�A� ﾂ��.z0Ɣ�`������,�����q2��o�ԇ���N�E��x5�z>�W���(R�K���^4{�����ŀ�5��y�V����y�m�XV�\�.��j�����
8�PKp��&x�PKn��I
xl/styles.xml���n� ��>bop2TQ��P)U��RWb�6*�����ӤS�Nw�s���3ߍ���t��(l��������ҝx�!N=#$ɀ��}��3c���ʰr`:i��2��w,�
�d
�T��R#�voc �;c�iE���Û��E<|��4Iɣ�����F#��n���B�z�F���y�j3y��yҥ�jt>���2��Lژ�!6��2F��OY��4#M�!���G��������1�t��y��p��" n����u�����a�ΦDi�9�&#��%I��9��}���cK��T��$?������`J������7���o��f��M|PK�1X#C�PKn��Ixl/workbook.xml���N�0��<��wj�E�8��J��P�;�����hmZ'Q�#����~;���;vCJ6 �Fà���"��|x|�}���#]����C�0�<֜'=�WiG��#y���O#�2i#������+`!��F�{��-�O�!/B�r)�;&h�����zOz�o����xO��I2����YuĔ��s�u��<J8Q�z6��Qm�:�,�c��Z�����PK1����dPKn��Ixl/_rels/workbook.xml.rels��Mk1#���0�nv-�R�^����0$����$dƯo���R�OC�ރ�-��������#Sՠ(�����ܼ?��b��p�����d�AJ�¾O�
#�/�f�iD�b�P6m�#Jy�N'�[�HO��E�k����3�W���ܑ`���Zri㪐����?�ض��e�������7p�wj�W5r���]������=�|���<:�[p��7�O�PK��4��9PKn��Ixl/worksheets/sheet1.xmleP�N�0���މ�V��THU$���$��j���[��c�����3��-v�nT���/a����7�Zߗ��z���]uQ���0 ��zJD�[�C3�3!� }|鈝�H��ab4�br�^���v�z���:�)P1v%ܭ#W�"|�8�?X�ܚ���C[B�'�~��ȅO������Tyb�bgN�<�|��$��ƙ��{#&����h��>��D�Ű�z�#��6��8�LF�dQ����,4�xS����/PK�_�Y�lPKn��I����O_rels/.relsPKn��IKB�>'[Content_Types].xmlPKn��I�|wؑ��docProps/app.xmlPKn��I����mdocProps/core.xmlPKn��Ip��&x��xl/sharedStrings.xmlPKn��I�1X#C�
nxl/styles.xmlPKn��I1����d�xl/workbook.xmlPKn��I��4��9xl/_rels/workbook.xml.relsPKn��I�_�Y�l$ xl/worksheets/sheet1.xmlPK ?Z
The output is about 5KB in size, while the output on my local computer is about 3KB in size. This appears to be a problem with binary output in general for Java on Amazon Lambda. When I do run some code that writes an image to the output string, it also works locally, but results in an image twice the size and garbled when run from Amazon Lambda.
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestStreamHandler;
import java.io.*;
import java.net.URL;
public class ImageRequestHandler implements RequestStreamHandler {
public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context)
throws IOException {
String address = "https://upload.wikimedia.org/wikipedia/commons/thumb/1/1d/AmazonWebservices_Logo.svg/580px-AmazonWebservices_Logo.svg.png";
URL url = new URL(address);
InputStream in = new BufferedInputStream(url.openStream());
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
int n;
while (-1!=(n=in.read(buf)))
{
out.write(buf, 0, n);
}
out.close();
in.close();
byte[] response = out.toByteArray();
outputStream.write(response);
}
}
The types of the was input and output streams are:
lambdainternal.util.NativeMemoryAsInputStream
lambdainternal.util.LambdaByteArrayOutputStream
Help?
I had the same problem with returning JPG image from Amazon Lambda and I found a work-around.
You need to encode an output stream with base64 encoding:
OutputStream encodedStream = Base64.getEncoder().wrap(outputStream);
encodedStream.write(response);
encodedStream.close();
Then you need to update Method Response and Integration Response of your function as described here: AWS Gateway API base64Decode produces garbled binary?
Am using a YaHP-Converter to convert HTML File to Pdf. Here is the code example i have used for converting. The code works me fine. But i want open Pdf file after this conversion.
Any idea please.
CYaHPConverter converter = new CYaHPConverter();
FileOutputStream out = new FileOutputStream(pdfOut);
Map properties = new HashMap();
List headerFooterList = new ArrayList();
properties.put(IHtmlToPdfTransformer.PDF_RENDERER_CLASS,IHtmlToPdfTransformer.FLYINGSAUCER_PDF_RENDERER);
converter.convertToPdf(htmlContents,
IHtmlToPdfTransformer.LEGALL,
headerFooterList,
"file:///D:/temp/",
out,
properties);
Thanks in advance
I think this helps:
import java.io.File;
import java.io.FileOutputStream;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Scanner;
// http://www.allcolor.org/YaHPConverter/
import org.allcolor.yahp.converter.CYaHPConverter;
import org.allcolor.yahp.converter.IHtmlToPdfTransformer;
public class HtmlToPdf_yahp_2 {
public static void main(String ... args ) throws Exception {
String root = "c:/temp/html";
String input = "file_1659686.htm"; // need to be charset utf-8
htmlToPdfFile(new File(root, input),
new File(root, input + ".pdf"));
System.out.println("Done");
}
public static void htmlToPdfFile(File htmlIn, File pdfOut) throws Exception {
Scanner scanner =
new Scanner(htmlIn).useDelimiter("\\Z");
String htmlContents = scanner.next();
CYaHPConverter converter = new CYaHPConverter();
FileOutputStream out = new FileOutputStream(pdfOut);
Map properties = new HashMap();
List headerFooterList = new ArrayList();
properties.put(IHtmlToPdfTransformer.PDF_RENDERER_CLASS,
IHtmlToPdfTransformer.FLYINGSAUCER_PDF_RENDERER);
//properties.put(IHtmlToPdfTransformer.FOP_TTF_FONT_PATH, fontPath);
converter.convertToPdf(htmlContents,
IHtmlToPdfTransformer.A4P,
headerFooterList,
"file:///temp/html/",
out,
properties);
out.flush();
out.close();
}
}
See this for futher info:
http://www.rgagnon.com/javadetails/java-convert-html-to-pdf-using-yahp.html
I have an XML file which have a node called "CONTENIDO", in this node I have a PDF file encoded in base64 string.
I'm trying to read this node, decode the string in base64 and download the PDF file to my computer.
The problem is that the file is downloaded with the same size (in kb) as the original PDF and has the same number of pages, but... all the pages are in blank without any content and when I open the downloaded file a popup appears with an error saying "unknown distinctive 806.6n". I don't know what that means.
I've tried to find a solution in the internet, with diferents ways to decode the string, but always get the same result... The XML is Ok I've checked the base64 string and is Ok.
I've also debugged the code and I've seen that the content of the var "fichero" where I'm reading the base64 string is also Ok, so I don't know what can be the problem.
This is my code:
package prueba.sap.com;
import java.io.ByteArrayOutputStream;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerFactory;
import org.w3c.dom.Document;
import org.w3c.dom.NodeList;
import sun.misc.BASE64Decoder;
import javax.xml.bind.DatatypeConverter;
public class anexoPO {
public static void main(String[] args) throws Exception {
FileInputStream inFile =
new FileInputStream("C:/prueba/prueba_attach_b64.xml");
FileOutputStream outFile =
new FileOutputStream("C:/prueba/salida.pdf");
anexoPO myMapping = new anexoPO();
myMapping.execute(inFile, outFile);
System.out.println("Success");
System.out.println(inFile);
}
public void execute(InputStream in, OutputStream out)
throws com.sap.aii.mapping.api.StreamTransformationException {
try {
//************************Code To Generate The XML Parsing Objects*****************************//
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder db = dbf.newDocumentBuilder();
Document doc = db.parse(in);
Document docout = db.newDocument();
NodeList CONTENIDO = doc.getElementsByTagName("CONTENIDO");
String fichero = CONTENIDO.item(0).getChildNodes().item(0).getNodeValue();
//************** decode *************/
//import sun.misc.BASE64Decoder;
//BASE64Decoder decoder = new BASE64Decoder();
//byte[] decoded = decoder.decodeBuffer(fichero);
//import org.apache.commons.codec.binary.*;
//byte[] decoded = Base64.decode(fichero);
//import javax.xml.bind.DatatypeConverter;
byte[] decoded = DatatypeConverter.parseBase64Binary(fichero);
//************** decode *************/
String str = new String(decoded);
out.write(str.getBytes());
} catch (Exception e) {
System.out.print("Problem parsing the file");
e.printStackTrace();
}
}
}
Thanks in advance.
Definitely:
out.write(decoded);
out.close();
Strings cannot represent all bytes, and PDF is binary.
Also remove the import of sun.misc.BASE64Decoder, as this package does not exist everywhere. It might be removed by the compiler, however I would not bet on it.
I'm using PDF-Renderer to view PDF files within my java application. It's working perfectly for normal PDF files.
However, i want the application to be able to display encrypted PDF files. The ecrypted file will be decrypted with CipherInputStream, but i do not want to save the decrypted data on disk. Am trying to figure a way i can pass the decryted data from CipherInputStream to the PDFFile constructor without having to write the decryted data to file.
I will also appreciate if someone can help with a link to PDF-Renderer tutorial, so that i can read up more on it.
Thanks.
Try the following class:
import com.sun.pdfview.PDFFile;
import java.io.IOException;
import java.io.InputStream;
import java.nio.ByteBuffer;
import java.nio.channels.Channels;
import java.nio.channels.ReadableByteChannel;
public class PDFFileUtility {
private static final int READ_BLOCK = 8192;
public static PDFFile getPDFFile(InputStream in) throws IOException {
ReadableByteChannel bc = Channels.newChannel(in);
ByteBuffer bb = ByteBuffer.allocate(READ_BLOCK);
while (bc.read(bb) != -1) {
bb = resizeBuffer(bb); //get new buffer for read
}
return new PDFFile(bb);
}
private static ByteBuffer resizeBuffer(ByteBuffer in) {
ByteBuffer result = in;
if (in.remaining() < READ_BLOCK) {
result = ByteBuffer.allocate(in.capacity() * 2);
in.flip();
result.put(in);
}
return result;
}
}
So call:
PDFFileUtility.getPDFFile(myCipherInputStream);