Previously my code was just this:
objJson = gson.toJson(objList);
return objJson;
And I got the string JSON with that return value.
But then I started getting famous Out Of Memory errors when the JSON becomes too large.
Then I followed the following post that converts the list of objects into JSON String in an efficient way that OOM errors will be no more:
https://sites.google.com/site/gson/streaming
So, I have followed the above approach here:
public String writeJsonStream(List<MyObj> objList) throws IOException {
ByteArrayOutputStream baos=new ByteArrayOutputStream();
OutputStreamWriter outputStreamWriter=new OutputStreamWriter(baos,"UTF-8");
JsonWriter writer = new JsonWriter(outputStreamWriter);
writer.setIndent(" ");
writer.beginArray();
Gson gson = new GsonBuilder().excludeFieldsWithoutExposeAnnotation().setPrettyPrinting().create();
for (MyObj myobj : objList) {
gson.toJson(myobj, MyObj.class, writer);
}
String objJson = writer.toString();
writer.endArray();
writer.close();
return objJson;
}
But this returning object com.google.gson.stream.JsonWriter#6a9454cd.
The method gson.toJson(myobj, MyObj.class, writer); is of type void and doesn't returns JSON string. So how can I get the JSON string in this case?
That's because you are getting the JSON String from String objJson = writer.toString();. This is not how you should retrieve it: this code will just call the toString() method on the writer instance. Since toString() is not overriden for this object, the result is the default com.google.gson.stream.JsonWriter#6a9454cd.
What you want to do instead is get the bytes that the JsonWriter wrote to the output stream. In your case, you are using a ByteArrayOutputStream so you can call toString(charsetName) to get the content as a String:
public String writeJsonStream(List<MyObj> objList) throws IOException {
ByteArrayOutputStream baos=new ByteArrayOutputStream();
OutputStreamWriter outputStreamWriter=new OutputStreamWriter(baos,"UTF-8");
JsonWriter writer = new JsonWriter(outputStreamWriter);
writer.setIndent(" ");
writer.beginArray();
Gson gson = new GsonBuilder().excludeFieldsWithoutExposeAnnotation().setPrettyPrinting().create();
for (MyObj myobj : objList) {
gson.toJson(myobj, MyObj.class, writer);
}
writer.endArray();
writer.close();
return baos.toString("UTF-8");
}
As a side note, you could just use a StringWriter instead and get the content with toString():
StringWriter stringWriter = new StringWriter();
JsonWriter writer = new JsonWriter(stringWriter);
// rest of code
writer.close();
String json = stringWriter.toString();
Well, converting stream to a static object unfortunately does not avoid OOM problem. In either way, you are trying to allocate/request memory to construct/write that string. You must pipe the resulting stream to another one, or simply consume all the data in that stream by using BufferedWriter in some loop or else.
Try this. Change ByteArrayOutputStream to OutputStream and get object from System.out
public String writeJsonStream(List<MyObj> objList) throws IOException {
OutputStream baos = System.out;
OutputStreamWriter outputStreamWriter=new OutputStreamWriter(baos,"UTF-8");
JsonWriter writer = new JsonWriter(outputStreamWriter);
writer.setIndent(" ");
writer.beginArray();
Gson gson = new GsonBuilder().excludeFieldsWithoutExposeAnnotation().setPrettyPrinting().create();
for (MyObj myobj : objList) {
gson.toJson(myobj, MyObj.class, writer);
}
String objJson = writer.toString();
writer.endArray();
writer.close();
return objJson;
Related
I have the below piece of code where I write a string to an output stream with Jackson's ObjectMapper.
OutputStream outputStream = new PrintStream(System.out);
ObjectMapper objectMapper = new ObjectMapper();
String output = "{\"test\":\"mytest/123\"}";
objectMapper.writeValue(outputStream, output);
When I run this code, the output stream prints the following string, where there are additional escape characters.
"{\"vfi\":\"test/mytest/123\"}"
How can I avoid the escape characters here? I am not using ObjectMapper.writeValueAsString but still they get printed. Appreciate any help.
You are serializing a Java string as JSON string. JSON strings must start and end with " and any embedded quotes must be escaped. (ref: https://json.org)
You have two options:
Print the string directly, without serializing via ObjectMapper
OutputStream outputStream = new PrintStream(System.out);
String output = "{\"test\":\"mytest/123\"}";
outputStream.println(output);
Serialize an object, e.g. Map, to a JSON string
OutputStream outputStream = new PrintStream(System.out);
ObjectMapper objectMapper = new ObjectMapper();
Map<String, String> output = Map.of("test", "mytest/123");
objectMapper.writeValue(outputStream, output);
It's also possible to define a custom class to represent your JSON structure. Depending on the complexity of your structure and other requirements, this could be the preferrable option.
class MyOutput {
private final String test;
public MyOutput(final String test) { this.test = test; }
public String getTest() { return test; }
}
OutputStream outputStream = new PrintStream(System.out);
ObjectMapper objectMapper = new ObjectMapper();
MyOutput output = new MyOutput("mytest/123");
objectMapper.writeValue(outputStream, output);
I would like to read the content of a css file stored in my NSF to include it in the style element in a (HTML) file that I create.
FacesContext.getCurrentInstance().getExternalContext().getResourceAsStream
does not seem to be the way.
Anyone a suggestion?
I tried
InputStream input = FacesContext.getCurrentInstance().getExternalContext().getResourceAsStream("file.css");
and then
StringBuilder sb = new StringBuilder();
sb.append("<style>").append("\n");
sb.append(convertStreamToString(input)).append("\n");
sb.append("</style>").append("\n");
static String convertStreamToString(java.io.InputStream is) {
java.util.Scanner s = new java.util.Scanner(is).useDelimiter("\\A");
return s.hasNext() ? s.next() : "";
}
which works fine
You can use Apache Commons IOUtils to copy from the InputStream to a StringWriter and then use toString() on the StringWriter. Here's a simple method that does that:
private static String getResourceFile(final String filename) {
final InputStream input = FacesContext.getCurrentInstance().getExternalContext().getResourceAsStream(filename);
final StringWriter writer = new StringWriter();
try {
IOUtils.copy(input, writer, "utf-8");
} catch (final IOException e) {
System.out.println("Error reading: " + filename);
}
return writer.toString();
}
I am trying to read content from SequenceInputStream
but it return me null but when i inspect it contains value please refer the screenshot
following is my code
InputStream is = m.getContent(InputStream.class);
SequenceInputStream si = new SequenceInputStream(is, null);
int j;
int i=0;
while((i=si.read())!=-1) {
System.out.print((char)i);
}
I also tryid to read it from InputStream
but the same output
follwoing is my input stream read code
CachedOutputStream bos = new CachedOutputStream();
IOUtils.copy(is,bos);
String soapMessage = new String(bos.getBytes());
System.out.println("-------------------------------------------");
System.out.println("incoming message is " + soapMessage);
System.out.println("-------------------------------------------");
bos.flush();
It is mandatory to use two streams should be not null. If anyone of the stream is null you will get null pointer exception after reading the first one.
you change your code to SequenceInputStream si = new SequenceInputStream(null, is); you get NPE at first place
Why you have to use sequenceInputStream to read the content. You can use below code or any other stream reader
StringWriter writer = new StringWriter();
IOUtils.copy(inputStream, writer, encoding);
String theString = writer.toString();
Thanks,
Gowtham
I'm converting Json to avro. I have json data in JSONArray. So while converting it into byte array i'm facing the problem.
below is my code:
static byte [] fromJsonToAvro(JSONArray json, String schemastr) throws Exception {
ExcelToJson ejj = new ExcelToJson();
List<String> list = new ArrayList<String>();
if (json != null) {
int len = json.length();
for (int i=0;i<len;i++){
list.add(json.get(i).toString());
}
}
InputStream input = new ByteArrayInputStream(list.getBytes()); //json.toString().getBytes()
DataInputStream din = new DataInputStream(input);
.
.
.//rest of the logic
So how can i do it? How to convert JsonArray object to bytes(i.e., how to use getBytes() method for JsonArray objects). The above code giving an error at list.getBytes() and saying getBytes() is undifined for list.
Avro works at the record level, bound to a schema. I don't think there's such a concept as "convert this JSON fragment to bytes for an Avro field independent of any schema or record".
Assuming the array is part of a larger JSON record, if you're starting with a string of the record, you could do
public static byte[] jsonToAvro(String json, String schemaStr) throws IOException {
InputStream input = null;
DataFileWriter<GenericRecord> writer = null;
Encoder encoder = null;
ByteArrayOutputStream output = null;
try {
Schema schema = new Schema.Parser().parse(schemaStr);
DatumReader<GenericRecord> reader = new GenericDatumReader<GenericRecord>(schema);
input = new ByteArrayInputStream(json.getBytes());
output = new ByteArrayOutputStream();
DataInputStream din = new DataInputStream(input);
writer = new DataFileWriter<GenericRecord>(new GenericDatumWriter<GenericRecord>());
writer.create(schema, output);
Decoder decoder = DecoderFactory.get().jsonDecoder(schema, din);
GenericRecord datum;
while (true) {
try {
datum = reader.read(null, decoder);
} catch (EOFException eofe) {
break;
}
writer.append(datum);
}
writer.flush();
return output.toByteArray();
} finally {
try { input.close(); } catch (Exception e) { }
}
}
For an on-line json to avro converter check the following URL
http://avro4s-ui.landoop.com
It is using the library avro4s that offers a lot of conversions including json=>avro
This discussion is likely useful:
http://mail-archives.apache.org/mod_mbox/avro-user/201209.mbox/%3CCALEq1Z8s1sfaAVB7YE2rpZ=v3q1V_h7Vm39h0HsOzxJ+qfQRSg#mail.gmail.com%3E
The gist is that there is a special Json schema and you can use JsonReader/Writer to get to and from that. The Json schema you should use is defined here:
https://github.com/apache/avro/blob/trunk/share/schemas/org/apache/avro/data/Json.avsc
i like to encode a java map of strings as a single base 64 encoded string. The encoded string will be transmitted to a remote endpoint and maybe manipulated by a not nice person. So the worst thing that should happen are invaild key,value-tuples, but should not bring any other security risks aside.
Example:
Map<String,String> map = ...
String encoded = Base64.encode(map);
// somewhere else
Map<String,String> map = Base64.decode(encoded);
Yes, must be Base64. Not like that or that or any other of these. Is there an existing lightweight solution (Single Utils-Class prefered) out there? Or do i have to create my own?
Anything better than this?
// marshalling
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(map);
oos.close();
String encoded = new String(Base64.encodeBase64(baos.toByteArray()));
// unmarshalling
byte[] decoded = Base64.decodeBase64(encoded.getBytes());
ByteArrayInputStream bais = new ByteArrayInputStream(decoded);
ObjectInputStream ois = new ObjectInputStream(bais);
map = (Map<String,String>) ois.readObject();
ois.close();
Thanks,
my primary requirements are: encoded string should be as short as possible and contain only latin characters or characters from the base64 alphabet (not my call). there are no other reqs.
Use Google Gson to convert Map to JSON. Use GZIPOutputStream to compress the JSON string. Use Apache Commons Codec Base64 or Base64OutputStream to encode the compressed bytes to a Base64 string.
Kickoff example:
public static void main(String[] args) throws IOException {
Map<String, String> map = new HashMap<String, String>();
map.put("key1", "value1");
map.put("key2", "value2");
map.put("key3", "value3");
String serialized = serialize(map);
Map<String, String> deserialized = deserialize(serialized, new TypeToken<Map<String, String>>() {}.getType());
System.out.println(deserialized);
}
public static String serialize(Object object) throws IOException {
ByteArrayOutputStream byteaOut = new ByteArrayOutputStream();
GZIPOutputStream gzipOut = null;
try {
gzipOut = new GZIPOutputStream(new Base64OutputStream(byteaOut));
gzipOut.write(new Gson().toJson(object).getBytes("UTF-8"));
} finally {
if (gzipOut != null) try { gzipOut.close(); } catch (IOException logOrIgnore) {}
}
return new String(byteaOut.toByteArray());
}
public static <T> T deserialize(String string, Type type) throws IOException {
ByteArrayOutputStream byteaOut = new ByteArrayOutputStream();
GZIPInputStream gzipIn = null;
try {
gzipIn = new GZIPInputStream(new Base64InputStream(new ByteArrayInputStream(string.getBytes("UTF-8"))));
for (int data; (data = gzipIn.read()) > -1;) {
byteaOut.write(data);
}
} finally {
if (gzipIn != null) try { gzipIn.close(); } catch (IOException logOrIgnore) {}
}
return new Gson().fromJson(new String(byteaOut.toByteArray()), type);
}
Another possible way would be using JSON which is a very ligthweight lib.
The the encoding then would look like this:
JSONObject jso = new JSONObject( map );
String encoded = new String(Base64.encodeBase64( jso.toString( 4 ).toByteArray()));
Your solution works. The only other approach would be to serialize the map yourself (iterate over the keys and values). That would mean you'd have to make sure you handle all the cases correctly (for example, if you transmit the values as key=value, you must find a way to allow = in the key/value and you must separate the pairs somehow which means you must also allow this separation character in the name, etc).
All in all, it's hard to get right, easy to get wrong and would take a whole lot more code and headache. Plus don't forget that you'd have to write a lot of error handling code in the parser (receiver side).