We Keep Coding

Zip Archives get corrupted when uploading to Azure Blob Store using REST API

I have been really banging my head against the wall with this one, uploading text files is fine, but when I upload a zip archive into my blob store -> it gets corrupted, and cannot be opened once downloaded.
Doing a hex compare (image below) of the original versus file that has been through Azure shows some subtle replacements have happened, but I cannot find the source of the change/corruption.
I have tried forcing UTF-8/Ascii/UTF-16, but found UTF-8 is probably correct, none have resolved the issue.
I have also tried different http libraries but got the same result.
Deployment environment is forcing unirest, and cannot use the Microsoft API (Which seems to work fine).
package blobQuickstart.blobAzureApp;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import java.util.Base64;
import org.junit.Test;
import kong.unirest.HttpResponse;
import kong.unirest.Unirest;
public class StackOverflowExample {
#Test
public void uploadSmallZip() throws Exception {
File testFile = new File("src/test/resources/zip/simple.zip");
String blobStore = "secretstore";
UploadedFile testUploadedFile = new UploadedFile();
testUploadedFile.setName(testFile.getName());
testUploadedFile.setFile(testFile);
String contentType = "application/zip";
String body = readFileContent(testFile);
String url = "https://" + blobStore + ".blob.core.windows.net/naratest/" + testFile.getName() + "?sv=2020-02-10&ss=b&srt=o&sp=c&se=2021-09-07T20%3A10%3A50Z&st=2021-09-07T18%3A10%3A50Z&spr=https&sig=xvQTkCQcfMTwWSP5gXeTB5vHlCh2oZXvmvL3kaXRWQg%3D";
HttpResponse<String> response = Unirest.put(url)
.header("x-ms-blob-type", "BlockBlob").header("Content-Type", contentType)
.body(body).asString();
if (!response.isSuccess()) {
System.out.println(response.getBody());
throw new Exception("Failed to Upload File! Unexpected response code: " + response.getStatus());
}
}
private static String readFileContent(File file) throws Exception {
InputStream is = new FileInputStream(file);
ByteArrayOutputStream answer = new ByteArrayOutputStream();
byte[] byteBuffer = new byte[8192];
int nbByteRead;
while ((nbByteRead = is.read(byteBuffer)) != -1)
{
answer.write(byteBuffer, 0, nbByteRead);
}
is.close();
byte[] fileContents = answer.toByteArray();
String s = Base64.getEncoder().encodeToString(fileContents);
byte[] resultBytes = Base64.getDecoder().decode(s);
String encodedContents = new String(resultBytes);
return encodedContents;
}
}
Please help!

byte[] resultBytes = Base64.getDecoder().decode(s);
String encodedContents = new String(resultBytes);
You are creating a String from a byte array containing binary data. String is only for printable characters. You do multiple pointless encoding/decoding just taking more memory.
If the content is in a ZIP format, it's binary, just return the byte array. Or you can encode the content, but then you should return the content encoded. As a weakness, you're doing it all in memory, limiting potential size of the content.

Unirest file handlers will by default force a multipart body - not supported by Azure.
A Byte Array can be provided directly as per this: https://github.com/Kong/unirest-java/issues/248
Unirest.put("http://somewhere")
.body("abc".getBytes())

Lotus Notes Java API mistake with encoding

I'm newble in Java and I need to extract data from NSF files to DXL xml files. I tried it with python, but OLE wrapper doesn't have a few major functionality. I have two different formates of DXL files: with rawitemdata and with richtext.
When I switched to Java Lotus Notes API I receive mistakes in rawitemdata fields. I use doc.convertToMIME(2); procedure to provide mail body in the HTML format for getting sameness for each type of documents. As result all fields stored as rawitemdata and it's great. It's what I want. But some rawitemdata fields have crashed encoding.
It's look like
<rawitemdata type="19">
AgAPAAAAAgAmAj4AhQEAAAAAAAANCi0tMF9fPUNDQkIwRTFFREZBRjk4Mjg4ZjllOGE5M2RmOTM4NjkwOTE4Y0NDQkIwRTFFREZBRjk4
MjgNCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IGJpbmFyeQ0KQ29udGVudC10eXBlOiBhcHBsaWNhdGlvbi9vY3RldC1zdHJlYW07
IA0KCW5hbWU9Ij0/S09JOC1SP0I/OFBMdjlPL3I3K3d1Wkc5amVBPT0/PSINCkNvbnRlbnQtRGlzcG9zaXRpb246IGF0dGFjaG1lbnQ7
IGZpbGVuYW1lPSI9P0tPSTgtUj84UEx2OU8vcjcrd3VaRzlqZUE9PT89Ig0KQ29udGVudC1JRDogPDNfXz1DQ0JCMEUxRURGQUY5ODI4
OGY5ZThhOTNkZjkzODY5MDkxQGxvY2FsPg0KDQoFzwXQBc4F0gXOBcoFzgXLLmRvY3g=
</rawitemdata>
After decode from base64 I got this
\x02\x00\x0f\x00\x00\x00\x02\x00&\x02>\x00\x85\x01\x00\x00\x00\x00\x00\x00\r\n-
-0__=CCBB0E1EDFAF98288f9e8a93df938690918cCCBB0E1EDFAF9828\r\nContent-Transfer-Encoding:
binary\r\nContent-type: application/octet-stream; \r\n\tname="=?KOI8-R?B?8PLv9O/r7+wuZG9jeA==?
="\r\nContent-Disposition: attachment; filename="=?KOI8-R?8PLv9O/r7+wuZG9jeA==?="\r\nContent-ID:
<3__=CCBB0E1EDFAF98288f9e8a93df93869091#local>\r\n\r\n\x05\xcf\x05\xd0\x05\xce\x05\xd2\x05\xce\x05\xca\x
05\xce\x05\xcb.docx
It's easy to explain:
first 20 bytes it's a header: \x02\x00\x0f\x00\x00\x00\x02\x00&\x02>\x00\x85\x01\x00\x00\x00\x00\x00\x00
to unpack it I use next code struct.unpack("<hhhh hhLL", data[:20])
it returns tuple like this: (2, 15, 0, 2, 550, 62, 389, 0) where the 5th element it is a length of body. But I changed the body and lazed to calc current header for new body
the body with rfc822 headers
After base64 and then koi8-r header Content-Disposition decoded we could see field name with content ПРОТОКОЛ.docx. And it's correct content.
If I try decode the last part of the body which contains byte encoding \x05\xcf\x05\xd0\x05\xce\x05\xd2\x05\xce\x05\xca\x
05\xce\x05\xcb.docx I got exception. This is part can't being decoded correctly.
It's look like cp1251 encoding under unicode. Because the content ПРОТОКОЛ.docx in cp1251 codepage has '\xcf\xd0\xce\xd2\xce\xca\xce\xcb.docx' bytes.
My java code is quite simple:
import lotus.domino.Database;
import lotus.domino.Session;
import lotus.domino.NotesFactory;
import java.io.File;
import java.io.FileWriter;
import java.io.BufferedWriter;
import lotus.domino.*;
public class ExportDXL {
public static void main(String[] args) throws Exception {
String server = "";
String dbPath = "C:\\share\\test.nsf";
NotesThread.sinitThread();
Session session = NotesFactory.createSession((String)null, (String)null, "test");
Database db = session.getDatabase(server, dbPath);
System.out.println("Db Title: " + db.getTitle());
DxlExporter exporter = session.createDxlExporter();
exporter.setConvertNotesBitmapsToGIF(true);
View view = db.getView("$ALL");
Document doc = view.getFirstDocument();
while(doc != null){
doc.convertToMIME(2); // or set 1 for get plain text
String xmldoc = doc.generateXML();
FileWriter fw = null;
String id = doc.getUniversalID();
fw = new FileWriter("C:\\lotus_test\\"+ id +".xml");
fw.write(xmldoc);
fw.flush();
fw.close();
doc = view.getNextDocument(doc);
}
}
}
How can I provide correct encoding in this case? Or how to set code page for Lotus Notes API ?

How to properly open a png file

I am trying to attach a png file. Currently when I sent the email, the attachment is 2x bigger than the file should be and an invalid png file. Here is the code I currently have:
import com.sendgrid.*;
Attachments attachments = new Attachments();
String filePath = "/Users/david/Desktop/screenshot5.png";
String data = "";
try {
data = new String(Files.readAllBytes(Paths.get(filePath)));
} catch (IOException e) {
}
byte[] encoded = Base64.encodeBase64(data.getBytes());
String encodedString = new String(encoded);
attachments.setContent(encodedString);
Perhaps I am encoding the data incorrectly? What would be the correct way to 'get' the data to attach it?

With respect, this is why Python presents a problem to modern developers. It abstracts away important concepts that you can't fully understand in interpreted languages.
First, and this is a relatively basic concept, but you can't convert arbitrary byte sequences to a string and hope it works out. The following line is your first problem:
data = new String(Files.readAllBytes(Paths.get(filePath)));
EDIT: It looks like the library you are using expects the file to be base64 encoded. I have no idea why. Try changing your code to this:
Attachments attachments = new Attachments();
String filePath = "/Users/david/Desktop/screenshot5.png";
try {
byte[] encoded = Base64.encodeBase64(Files.readAllBytes(Paths.get(filePath)));
String encodedString = new String(encoded);
attachments.setContent(encodedString);
} catch (IOException e) {
}
The only issue you were having is that you were trying to represent arbitrary bytes as a string.

Take a look at the Builder class in the repository here. Example:
FileInputStream fileContent = new FileInputStream(filePath);
Attachments.Builder builder = new Attachments.Builder(fileName, fileContent);
mail.addAttachments(builder.build());

reading unicode characters from properties file java

Please help me to read the UNICODE characters as it is from the properties file in java. For example : if I pass the key "Account.label.register" it should return to me as "\u5BC4\u5B58\u5668" but not its character representation like "寄存器" . Here is my sample properties file
file_ch.properties
Account.label.register = \u5BC4\u5B58\u5668
Account.label.login = \u767B\u5F55
Account.label.username = \u7528\u6237\u540D
Account.label.password = \u5BC6\u7801
Thank you.
Hi , I am reading properties file using the following java code
#Override
public ResourceBundle getTexts(String bundleName) {
ResourceBundle myResources = null;
try {
myResources = ResourceBundle.getBundle(bundleName, getLocale());
} catch (Exception e) {
myResources = ResourceBundle.getBundle(getDefaultBundleKey(), getLocale());
}
return myResources;
}
Using the above approach it's ok fine, I am getting chinese characters. But for some of the ajax requests in my application I need to pass the chinese text in X-JSON header. Sample code is given below
HashMap<String, List<String>> map = new HashMap<String, List<String>>();
List<String> errors = new ArrayList<String>();
errors.add(str); /*ex: str = "无效的代码" , value taken from properties file through resource bundle*/
map.put("ERROR", errors);
JSONObject json = JSONObject.fromObject(map);
response.setCharacterEncoding("UTF-8");
response.setHeader("X-JSON", json.toString());
response.setStatus(500);
I am passing english for example str="Invalid Code" X-JSON header is carrying the information as it is. But if the str="无效的代码" (chinese or any other text) X-JSON header is carrying the text as empty like below is the response I am getting
response :
connection:close
Content-Encoding:gzip
Content-Type:text/html;charset=UTF-8
Date:Wed, 08 Jun 2016 10:17:43 GMT
Server:Apache-Coyote/1.1
Transfer-Encoding:chunked
Vary:Accept-Encoding
X-JSON:{"ERROR":["Invalid Code"]}
However if the "error" contains "chinese" text for ex:"无效的代码"
response :
connection:close
Content-Encoding:gzip
Content-Type:text/html;charset=UTF-8
Date:Wed, 08 Jun 2016 10:17:43 GMT
Server:Apache-Coyote/1.1
Transfer-Encoding:chunked
Vary:Accept-Encoding
**X-JSON:{"ERROR":[" "]}** /*expecting the response X-JSON:{"ERROR":["无效的代码"]}*/
As the chinese text is coming as empty , I thought of sending unicode through X-JSON header like below
{"ERROR":["\u65E0\u6548\u7684\u4EE3\u7801"]}
After that want to parse the Unicode characters using Javascript code after evaluating X-JSON header like below
var json;
try {
json = xhr.getResponseHeader('X-Json');
} catch (e) {
alert(e);
}
if (json) {
var data = eval('(' + json + ')');
decodeMsg(data);
}
function decodeMsg(message) {
var mssg = message;
var r = /\\u([\d\w]{4})/gi;
mssg = mssg.replace(r, function (match, grp) {
return String.fromCharCode(parseInt(grp, 16)); } );
mssg = unescape(mssg);
return mssg;
}
Please give suggestions. Thank you.

Update of answer:
The original encoding of .properties was in Latin-1, ISO-8859-1 (éö).
This needed u-escaping for the full Unicode range of characters.
However the newer java versions try UTF-8 first. So you can keep the .properties file in UTF-8! Which is a tremendous improvement.
Original answer: .properties in ISO-8859-1 as of java 1.
The error is that in HTTP the header lines are in ISO-8859-1, basic Latin-1.
The solution there is to use %XX conversion of UTF-8 bytes (in this case).
However you are better served in case of JSON simply doing as you intended.
So you want to send u-escaped Unicode, using \uXXXX. As not only Java, but also JavaScript/JSON knows this convention, you only need this u-escaping in java on the server.
static String uescape(String s) {
StringBuilder sb = new StringBuilder(s.length() * 6);
for (int i = 0; i < chars.length; ++i) {
char ch = s.charAt(i);
if (ch < 128) {
sb.append(ch);
} else {
sb.append(String.format("\\u%04X", (int) ch));
}
}
return sb.toString();
}
errors.add(uescape(str));
This zero-pads every non-ASCII (>=128) char as 4 digit hex, the exact format.
Or use apache-commons StringEscapeUtils.escapeJava which also does quotes and \n and such - much safer.

Escape the backslashes in your properties file by doubling them:
Account.label.register = \\u5BC4\\u5B58\\u5668
Account.label.login = \\u767B\\u5F55
Account.label.username = \\u7528\\u6237\\u540D
Account.label.password = \\u5BC6\\u7801

Java mail is not supporting foreign languages

I need to send HTML having content in different languages. my configuration are :
MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_18_19002270.1337852743826"
------=_Part_18_19002270.1337852743826 Content-Type: text/html; charset=Cp1252 Content-Transfer-Encoding: quoted-printable
and in mail am getting all characters as ?. Can anybody suggest me how to set encoding so that i could get mail in proper language.
Thanks
found the solution:)
MimeBodyPart htmlPart = new MimeBodyPart();
htmlPart.setContent(bodyText, "UTF-8");
htmlPart.setText(bodyText, "utf-8");
htmlPart.setHeader("Content-Type","text/html; charset=\"utf-8\"");
htmlPart.setHeader("Content-Transfer-Encoding", "quoted-printable");
Still am not able to encode Subject line.

This piece of code might help you. :)
MimeMessage msg = new MimeMessage(session);
msg.setSubject("yourSubject", "UTF-8"); // here you specify your subject encoding
msg.setContent("yourBody", "text/plain; charset=utf-8");
msg.setFrom("senderAddress");
msg.addRecipient(Message.RecipientType.TO, "recieverAddress");
Transport.send(msg);
EDIT:
Here is the method I've used to set the encoding of subject line.

Convert whole html File read string to UniCode
I used the below code to convert Whole HTML File read as String using FileUtils
String contents = FileUtils.readFileToString(new File("/path/to/the/HTMLfile"), "UTF-8")
I was trying to send an Email in Chinese, Arabic, Japanese and Spanish. So while sending I was unable to figure it out while setting Content in MimeMessage and MimeMultipart
So, I read the whole file as a string and then check whether the character lies between 1-128 i.e. ASCII equivalent characters (Numeric Special Characters Space etc..) that belong to ASCII. I retained them in the string and rest all characters are converted to Unicode
Function to convert html file string to Unicode string and is then set in mime body part and mime multipart. Hope it is very clear
public String convertToUniCode(String yourHTMLBodyStr) {
String str = "<div>Chinese 你好嗎 English How are you Japanese お元気ですか Arabic كيف حالك Spanish cómo estás</div>";
String[] codePointAt = new String[str.length()];
for (int j = 0; j < str.length(); j++) {
int charactercode = Character.codePointAt(str, j);
if (charactercode > 0 && charactercode < 128) {
codePointAt[j] = String.valueOf(str.charAt(j));
} else {
codePointAt[j] = "&#" + String.valueOf(charactercode) + ";";
}
}
StringBuilder strBuilder = new StringBuilder();
for (int i = 0; i < codePointAt.length; i++) {
strBuilder.append(codePointAt[i]);
}
System.out.println("New String :: " + strBuilder.toString());
}
Output(Unicode String) with &# Like &#16003 :
New String :: Chinese 你好嗎 English How are you Japanese お元気ですか Arabic كيف حالك Spanish cómo estás
MimeMessage msg = new MimeMessage(session);
String htmlUnicodeStr = convertToUniCode(yourHTMLBodyStr);
msg.setSubject(UniCodeSubject, "UTF-8"); // here you specify your subject encoding
Multipart emailMimeMultipart;
MimeBodyPart messageBody = new MimeBodyPart();
messageBody.setDataHandler(new DataHandler(new ByteArrayDataSource(htmlUnicodeStr, "text/html")));
emailMimeMultipart.addBodyPart(messageBody);
//Add UTF-8 In MimeMultiPart While Setting Content and Transport.Send()

Try to switch charset=Cp1252 to charset=UTF-8.