We Keep Coding

How to generate UUID per message in log4j's JSON Layout?

I need to generate a random string for a correlation key for each log message and I found that there is a way to generate UUID in log4j configuration file.
There is a mention of UUID on https://logging.apache.org/log4j/2.x/manual/layouts.html, but it doesn't say anything about how to use it.
I am trying to set this as a value for a key in my JsonLayout.
appender.rolling.layout.external-correlation-id.type = KeyValuePair
appender.rolling.layout.external-correlation-id.key = external-correlation-id
appender.rolling.layout.external-correlation-id.value = %u{"RANDOM"}
But that doesn't do anything. It just adds the literal string in the log message ... "external-correlation-id":"%u{\"RANDOM\"}" ....
How can I get a random string to set it in the log message? Is there a way for me to atleast directly call the UUID.randomUUID() in thelog4j properties file?
I don't want to use MDC for this and am looking for a way to do it directly from the log4j configuration file.
Any other help with this will be very appreciated.

I've done something similar in a JsonLayout in a KeyValuePair using a custom StrLookup.
The code below shows a simple example of returning a UUID every time a message is logged using a custom Lookup. One would add the ability to have different operations per key etc. but this works.
import java.util.UUID;
import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.config.plugins.Plugin;
import org.apache.logging.log4j.core.lookup.StrLookup;
#Plugin(name = "dyn", category = "Lookup")
public class DynamicLookup implements StrLookup {
#Override
public String lookup(String key) {
if("uuid".equals(key)) {
return UUID.randomUUID().toString();
} else {
return null;
}
}
#Override
public String lookup(LogEvent event, String key) {
if("uuid".equals(key)) {
return UUID.randomUUID().toString();
} else {
return null;
}
}
}
Then reference the Lookup using double $$ to have the value evaluated every message instead of just once.
<KeyValuePair key="event_id" value="$${dyn:uuid:-}"/>

java parse tweet corpus json

I have a problem: I need to parse a JSON file in Java where each line represents a tweet and follows the standard JSON of Twitter. I do not need all the information, I attach two photos to show you which fields I need. I would do it without using any support library. Thank you!
This is what I did for now. I do not think it's the best way to do it, especially going ahead I'll be in trouble because the names of many fields repeat
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
public class TweetCorpus implements Iterable<Tweet>
{
private List<Tweet> tweets;
public static TweetCorpus parseFile(File file)
{
List<Tweet> tweets = new ArrayList<>();
try(BufferedReader br = Files.newBufferedReader(file.toPath()))
{
while(br.ready())
{
String tweet = br.readLine();
//System.out.println(tweet);
if(!tweet.isEmpty())
{
long l = Long.parseLong(tweet.substring(tweet.indexOf("\"id\":") + 5, tweet.indexOf(",\"id_str\":")));
String t = tweet.substring(tweet.indexOf(",\"text\":\"") + 9, tweet.indexOf(",\"source\":"));
tweets.add(new Tweet(l, t));
}
}
}
catch(IOException e)
{
e.printStackTrace();
}
return new TweetCorpus(tweets);
}
public int getTweetCount() { return tweets.size(); }
public TweetCorpus(List<Tweet> tweets)
{
this.tweets = tweets;
}
#Override
public Iterator<Tweet> iterator()
{
return tweets.iterator();
}
public static void main(String[] args)
{
TweetCorpus t = parseFile(new File("C:\\Users\\acer\\Desktop\\Moroder\\Uni\\1 Anno - 2 Semestre\\Metodologie Di Programmazione\\Progetto\\HM4Test\\tweetsCorpus.js"));
t.getTweetCount();
}
}
json media/retweet tweet
json "normal" tweet

You can use Gson or Jackson java library to parse json to Tweet object. Their are tools online which generates pojo from json, which you can use with jackson to parse your json string to object.
Once you have json values in an object, you can use getters/setters to extract/modify the values you are interested in from input json.
Well writing your own parser would be a reinventing the wheel kind of task. But if your need is to write your own parser, refer to jackson project on github for inspiration on design and maintenance.
This would help you in making a generic application.
Quick reference for jackson parser ,
https://dzone.com/articles/processing-json-with-jackson

Re-inventing a JSON parser using only readLine() is a really bad idea. If you don't have experience writing parsers by hand, you will end up with a lot of bad code that is really hard to understand. Just use a library. There are tons of good JSON libraries for Java.
Jackson
GSON
Boon
Example code:
static class User {
String id, name;
}
static class MyTweet {
String id, text;
User user;
}
// if the entire file is a JSON array:
void parse(Reader r) {
List<MyTweet> tweets = objectMapper.readValue(
r, new TypeReference<List<MyTweet>>(){});
}
// if each line is a single JSON object:
void parse(BufferedReader r) {
while (r.ready()) {
String line = r.readLine();
MyTweet tweet = objectMapper.readValue(line, MyTweet.class);
}
}

How can I tell gson which fields to add to json in Java?

I am using gson to produce json of a collection of objects in Java (Some objects have other collections too). This json will be used to populate the web page for users with different clearance levels. Therefore the detail which users can see differs. Web page only shows what it needs to show however if I use the same json for two different pages then html source code will have more data than it should have. Is there a way to inform gson which variables in which class should be added to the json? As far as I search I could not find an easy way. Either I will produce json myself or clear extra data from the json which gson produced.

I need to use same classes for different clearance levels and get different json.
You are trying to use Gson to generate multiple different JSON outputs of the same objects in the same JVM, which is going to be difficult, both in Gson and any good serialization library, because their express goal is essentially the opposite of what you're looking for.
The right thing to do would be to instead represent these different clearance levels with different classes, and simply serialize those different classes with Gson as normal. This way you separate the security model from the serialization, letting you safely pass this information around.
/**
* Core data class, contains all information the application needs.
* Should never be serialized for display to any end user, no matter their level.
*/
public class GlobalData {
private final String username;
private final String private_data;
private final String secure_data;
}
/** Interface for all data display operations */
public interface DisplayData {
/** Returns a JSON representation of the data to be displayed */
public String toJson();
}
/**
* Class for safe display to an untrusted user, only holds onto public
* data anyone should see.
*/
public class UserDisplayData implements DisplayData {
private final String username;
public UserDisplayData(GlobalData gd) {
username = gd.username;
}
public String toJson() {
return gson.toJson(this);
}
}
/**
* Class for safe display to a trusted user, holds private information but
* does not display secure content (passwords, credit cards, etc.) that even
* admins should not see.
*/
public class AdminDisplayData implements DisplayData {
private final String username;
private final String private_data;
public AdminDisplayData(GlobalData gd) {
username = gd.username;
private_data = gd.private_data;
}
public String toJson() {
// these could be different Gson instances, for instance
// admin might want to see nulls, while users might not.
return gson.toJson(this);
}
}
Now you can sanitize and serialize your data as two separate steps, and use type safety to ensure your GlobalData is never displayed.
public void getDisplayData(GlobalData gd, User user) {
if(user.isAdmin()) {
return new AdminDisplayData(gd);
} else {
return new UserDisplayData(gd);
}
}
public void showData(DisplayData data) {
String json = data.toJson();
// display json however you want
}
If you erroneously tried to call showData(gd) you'd get a clear compilation error that you've done something wrong, and it's a quick fix to get the correct result by calling showData(getDisplayData(gd, user)) which safely and clearly does exactly what you want.

you can add a Expose annotations like this on the filed you don't want:
#Expose(serialize = false, deserialize = false)
private String address;
some more information here:
https://sites.google.com/site/gson/gson-user-guide#TOC-Gson-s-Expose

How to serialize ANY Object into a URI?

My basic question: is there anything built that already does this automatically (doesn't have to be part of a popular library/package)? The main things I'm working with are Spring (MVC) and Jackson2.
I understand there are a few manual ways to do this:
Create a method in each class that serializes its specific properties into property=value& form (kind of stinks because it's a bunch of logic duplication, I feel).
Create a function that accepts an object, and uses reflection to dynamically read all the properties (I guess the getters), and build the string by getting each. I'm assuming this is how Jackson works for serialization/deserialization in general, but I really don't know.
Use some feature of Jackson to customly serialize the object. I've researched custom serializers, but it seems they are specific to a class (so I'd have to create one for each Class I'm trying to serialize), while I was hoping for a generic way. I'm just having trouble understanding how to apply one universally to objects. A few of the links:
http://techtraits.com/Programming/2011/11/20/using-custom-serializers-with-jackson/
http://wiki.fasterxml.com/JacksonHowToCustomSerializers
Use ObjectMapper.convertValue(object, HashMap.class);, iterate over the HashMap's key/value pairs, and build the string (which is what I'm using now, but I feel the conversions are excessive?).
I'm guessing there's others I'm not thinking of.
The main post I've looked into is Java: Getting the properties of a class to construct a string representation
My point is that I have several classes that I want to be able to serialize without having to specify something specific for each. That's why I'm thinking a function using reflection (#2 above) is the only way to handle this (if I have to do it manually).
If it helps, an example of what I mean is with, say, these two classes:
public class C1 {
private String C1prop1;
private String C1prop2;
private String C1prop3;
// Getters and setters for the 3 properties
}
public class C2 {
private String C2prop1;
private String C2prop2;
private String C2prop3;
// Getters and setters for the 3 properties
}
(no, the properties names and conventions are not what my actual app is using, it's just an example)
The results of serializing would be C1prop1=value&C1prop2=value&C1prop3=value and C2prop1=value&C2prop2=value&C2prop3=value, but there's only one place that defines how the serialization happens (already defined somewhere, or created manually by me).
So my idea is that I will have to end up using a form of the following (taken from the post I linked above):
public String toString() {
StringBuilder sb = new StringBuilder();
try {
Class c = Class.forName(this.getClass().getName());
Method m[] = c.getDeclaredMethods();
Object oo;
for (int i = 0; i < m.length; i++)
if (m[i].getName().startsWith("get")) {
oo = m[i].invoke(this, null);
sb.append(m[i].getName().substring(3) + ":"
+ String.valueOf(oo) + "\n");
}
} catch (Throwable e) {
System.err.println(e);
}
return sb.toString();
}
And modify it to accept an object, and change the format of the items appended to the StringBuilder. That works for me, I don't need help modifying this now.
So again, my main question is if there's something that already handles this (potentially simple) serialization instead of me having to (quickly) modify the function above, even if I have to specify how to deal with each property and value and how to combine each?
If it helps, the background of this is that I'm using a RestTemplate (Spring) to make a GET request to a different server, and I want to pass a specific object's properties/values in the URL. I understand I can use something like:
restTemplate.getForObject("URL?C1prop1={C1Prop1}&...", String.class, C1Object);
I believe the properties will be automatically mapped. But like I said, I don't want to have to make a different URL template and method for each object type. I'm hoping to have something like the following:
public String getRequest(String url, Object obj) {
String serializedUri = SERIALIZE_URI(obj);
String response = restTemplate.getForObject("URL?" + serializedUri, String.class);
return response;
}
where SERIALIZE_URI is where I'd handle it. And I could call it like getRequest("whatever", C1Object); and getRequest("whateverElse", C2Object);.

I think, solution number 4 is OK. It is simple to understand and clear.
I propose similar solution in which we can use #JsonAnySetter annotation. Please, see below example:
import com.fasterxml.jackson.annotation.JsonAnySetter;
import com.fasterxml.jackson.databind.ObjectMapper;
public class JacksonProgram {
public static void main(String[] args) throws Exception {
C1 c1 = new C1();
c1.setProp1("a");
c1.setProp3("c");
User user = new User();
user.setName("Tom");
user.setSurname("Irg");
ObjectMapper mapper = new ObjectMapper();
System.out.println(mapper.convertValue(c1, UriFormat.class));
System.out.println(mapper.convertValue(user, UriFormat.class));
}
}
class UriFormat {
private StringBuilder builder = new StringBuilder();
#JsonAnySetter
public void addToUri(String name, Object property) {
if (builder.length() > 0) {
builder.append("&");
}
builder.append(name).append("=").append(property);
}
#Override
public String toString() {
return builder.toString();
}
}
Above program prints:
prop1=a&prop2=null&prop3=c
name=Tom&surname=Irg
And your getRequest method could look like this:
public String getRequest(String url, Object obj) {
String serializedUri = mapper.convertValue(obj, UriFormat.class).toString();
String response = restTemplate.getForObject(url + "?" + serializedUri, String.class);
return response;
}

Lets we have c1.
c1.setC1prop1("C1prop1");
c1.setC1prop2("C1prop2");
c1.setC1prop3("C1prop3");
Converts c1 into URI
UriComponentsBuilder.fromHttpUrl("http://test.com")
.queryParams(new ObjectMapper().convertValue(c1, LinkedMultiValueMap.class))
.build()
.toUri());
After we will have
http://test.com?c1prop1=C1prop1&c1prop2=C1prop2&c1prop3=C1prop3

What libs can I use to bind POJOs to external files for TDD without much overhead?

I need a way to bind POJO objects to an external entity, that could be XML, YAML, structured text or anything easy to write and maintain in order to create Mock data for unit testing and TDD. Below are some libraries I tried, but the main problems with them were that I am stuck (for at least more 3 months) to Java 1.4. I'd like any insights on what I could use instead, with as low overhead and upfront setup (like using Schemas or DTDs, for instance) as possible and without complex XML. Here are the libraries I really like (but that apparently doesn't work with 1.4 or doesn't support constructors - you gotta have setters):
RE-JAXB (or Really Easy Java XML Bindings)
http://jvalentino.blogspot.com/2008/07/in-response-to-easiest-java-xml-binding.html
http://sourceforge.net/projects/rejaxb/
Seamlessy binds this:
<item>
<title>Astronauts' Dirty Laundry</title>
<link>http://liftoff.msfc.nasa.gov/news/2003/news-laundry.asp</link>
<description>Compared to earlier spacecraft, the International Space
Station has many luxuries, but laundry facilities are not one of them.
Instead, astronauts have other options.</description>
<pubDate>Tue, 20 May 2003 08:56:02 GMT</pubDate>
<guid>http://liftoff.msfc.nasa.gov/2003/05/20.html#item570</guid>
</item>
To this:
#ClassXmlNodeName("item")
public class Item {
private String title;
private String link;
private String description;
private String pubDate;
private String guid;
//getters and settings go here...
}
Using:
Rss rss = new Rss();
XmlBinderFactory.newInstance().bind(rss, new File("Rss2Test.xml"));
Problem: It relies on annotations, so no good for Java 1.4
jYaml
http://jyaml.sourceforge.net/
Seamlessly binds this:
--- !user
name: Felipe Coury
password: felipe
modules:
- !module
id: 1
name: Main Menu
admin: !user
name: Admin
password: password
To this:
public class User {
private String name;
private String password;
private List modules;
}
public class Module {
private int id;
private String name;
private User admin;
}
Using:
YamlReader reader = new YamlReader(new FileReader("example.yaml"));
reader.getConfig().setClassTag("user", User.class);
reader.getConfig().setClassTag("module", Module.class);
User user = (User) reader.read(User.class);
Problem: It won't work with constructors (so no good for immutable objects). I'd have to either change my objects or write custom code por handling the YAML parsing.
Remember that I would like to avoid - as much as I can - writing data descriptors, I'd like something that "just works".
Do you have any suggestions?

If the objects to be populated are simple beans it may be a good idea to look at apache common's BeanUtils class. The populate() method might suit the described cases. Generally dependency injection frameworks like Spring can be very useful, but that might not be answer for the current problem. For input in form of xml, jibx might be a good alternative, so would be jaxb 1.0.

Just use XStream (for XML or you could give a try to JSON).
But...
Man, I just can't avoid to think that put the test data outside the unit test itself will leads you to unreadable tests. You will need look two files when reading a test case, you will lose refactoring tools (when changing property's name). Jay Fields can explain it better than me:
http://blog.jayfields.com/2007/06/testing-inline-setup.html
Kind Regards

You may give it a try to the deefault XMLEncoder/XMLDecoder that was added to the platform in Java1.4
Here's the way I use it.
import java.beans.XMLEncoder;
import java.beans.XMLDecoder;
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
public class ToXml {
/**
* Write an object to a file in XML format.
* #param o - The object to serialize.
* #param file - The file where to write the object.
*/
public static void writeObject( Object o, String file ) {
XMLEncoder e = null;
try {
e = new XMLEncoder( new BufferedOutputStream( new FileOutputStream(file)));
e.writeObject(o);
}catch( IOException ioe ) {
throw new RuntimeException( ioe );
}finally{
if( e != null ) {
e.close();
}
}
}
/**
* Read a xml serialized object from the specified file.
* #param file - The file where the serialized xml version of the object is.
* #return The object represented by the xmlfile.
*/
public static Object readObject( String file ){
XMLDecoder d = null;
try {
d = new XMLDecoder( new BufferedInputStream( new FileInputStream(file)));
return d.readObject();
}catch( IOException ioe ) {
throw new RuntimeException( ioe );
}finally{
if( d != null ) {
d.close();
}
}
}
}
It's easy, is simple, is in the core libraries.
You just have to write the load mechanism.
I have this swing app that loads data from a remote EJB in 5 - 10 secs. What I do is to store the previous session in XML like this and when the app loads it has all the data from the previous session in less than 1 sec.
While the user start to work with the app, a background thread fetches those elements that have changed since the last session.